xdp: hints via kfuncs
Subsystems (15)
v8 cover xdp: hints via kfuncs
Please see the first patch in the series for the overall design and use-cases.
See the following email from Toke for the per-packet metadata overhead:
https://lore.kernel.org/bpf/[email protected]/T/#m49d48ea08d525ec88360c7d14c4d34fb0e45e798
Recent changes:
- Keep new functions in en/xdp.c, do 'extern mlx5_xdp_metadata_ops' (Tariq)
- Remove mxbuf pointer and use xsk_buff_to_mxbuf (Tariq)
- Clarify xdp_buff vs 'XDP frame' (Jesper)
- Explicitly mention that AF_XDP RX descriptor lacks metadata size (Jesper)
- Drop libbpf_flags/xdp_flags from selftests and use ifindex instead of ifname (due to recent xsk.h refactoring)
Prior art (to record pros/cons for different approaches):
- Stable UAPI approach:
https://lore.kernel.org/bpf/[email protected]/
- Metadata+BTF_ID appoach:
https://lore.kernel.org/bpf/166256538687.1434226.15760041133601409770.stgit@firesoul/
- v7:
https://lore.kernel.org/bpf/[email protected]/
- v6:
https://lore.kernel.org/bpf/[email protected]/
- v5:
https://lore.kernel.org/bpf/[email protected]/
- v4:
https://lore.kernel.org/bpf/[email protected]/
- v3:
https://lore.kernel.org/bpf/[email protected]/
- v2:
https://lore.kernel.org/bpf/[email protected]/
- v1:
https://lore.kernel.org/bpf/[email protected]/
- kfuncs v2 RFC:
https://lore.kernel.org/bpf/[email protected]/
- kfuncs v1 RFC:
https://lore.kernel.org/bpf/[email protected]/
Stanislav Fomichev (13):
bpf: Document XDP RX metadata
bpf: Rename bpf_{prog,map}_is_dev_bound to is_offloaded
bpf: Move offload initialization into late_initcall
bpf: Reshuffle some parts of bpf/offload.c
bpf: Introduce device-bound XDP programs selftests/bpf: Update expected test_offload.py messages
bpf: XDP metadata RX kfuncs
veth: Introduce veth_xdp_buff wrapper for xdp_buff
veth: Support RX XDP metadata
selftests/bpf: Verify xdp_metadata xdp->af_xdp path net/mlx4_en: Introduce wrapper for xdp_buff net/mlx4_en: Support RX XDP metadata selftests/bpf: Simple program to dump XDP RX metadata
Toke Høiland-Jørgensen (4):
bpf: Support consuming XDP HW metadata from fext programs
xsk: Add cb area to struct xdp_buff_xsk net/mlx5e: Introduce wrapper for xdp_buff net/mlx5e: Support RX XDP metadata
Documentation/networking/index.rst | 1 + Documentation/networking/xdp-rx-metadata.rst | 110 +++++ drivers/net/ethernet/mellanox/mlx4/en_clock.c | 13 +- .../net/ethernet/mellanox/mlx4/en_netdev.c | 6 + drivers/net/ethernet/mellanox/mlx4/en_rx.c | 63 ++- drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 5 + drivers/net/ethernet/mellanox/mlx5/core/en.h | 5 +- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 5 + .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 31 +- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 10 +- .../ethernet/mellanox/mlx5/core/en/xsk/rx.c | 47 +- .../ethernet/mellanox/mlx5/core/en/xsk/rx.h | 2 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 1 + .../net/ethernet/mellanox/mlx5/core/en_rx.c | 99 ++-- drivers/net/netdevsim/bpf.c | 4 - drivers/net/veth.c | 87 ++-- include/linux/bpf.h | 61 ++- include/linux/netdevice.h | 8 + include/net/xdp.h | 21 + include/net/xsk_buff_pool.h | 5 + include/uapi/linux/bpf.h | 5 + kernel/bpf/core.c | 12 +- kernel/bpf/offload.c | 425 ++++++++++++------ kernel/bpf/syscall.c | 34 +- kernel/bpf/verifier.c | 44 +- net/bpf/test_run.c | 3 + net/core/dev.c | 9 +- net/core/filter.c | 2 +- net/core/xdp.c | 64 +++ tools/include/uapi/linux/bpf.h | 5 + tools/testing/selftests/bpf/.gitignore | 1 + tools/testing/selftests/bpf/Makefile | 9 +- .../selftests/bpf/prog_tests/xdp_metadata.c | 410 +++++++++++++++++ .../selftests/bpf/progs/xdp_hw_metadata.c | 81 ++++ .../selftests/bpf/progs/xdp_metadata.c | 64 +++ .../selftests/bpf/progs/xdp_metadata2.c | 23 + tools/testing/selftests/bpf/test_offload.py | 10 +- tools/testing/selftests/bpf/xdp_hw_metadata.c | 403 +++++++++++++++++ tools/testing/selftests/bpf/xdp_metadata.h | 15 + 39 files changed, 1910 insertions(+), 293 deletions(-) create mode 100644 Documentation/networking/xdp-rx-metadata.rst create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_metadata.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_hw_metadata.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_metadata.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_metadata2.c create mode 100644 tools/testing/selftests/bpf/xdp_hw_metadata.c create mode 100644 tools/testing/selftests/bpf/xdp_metadata.h
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 1/17 bpf: Document XDP RX metadata
Document all current use-cases and assumptions.
--- Documentation/networking/index.rst | 1 + Documentation/networking/xdp-rx-metadata.rst | 110 +++++++++++++++++++ 2 files changed, 111 insertions(+) create mode 100644 Documentation/networking/xdp-rx-metadata.rst
Documentation/networking/index.rst +1 -0
@@ -120,6 +120,7 @@ Refer to :ref:`netdev-FAQ` for a guide on netdev development process specifics. xfrm_proc xfrm_sync xfrm_sysctl + xdp-rx-metadata .. only:: subproject and html
Documentation/networking/xdp-rx-metadata.rst new file +110 -0
110 lines not on this page · raw message
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 2/17 bpf: Rename bpf_{prog,map}_is_dev_bound to is_offloaded
In reply to Stanislav FomichevBPF offloading infra will be reused to implement bound-but-not-offloaded bpf programs. Rename existing helpers for clarity. No functional changes.
--- include/linux/bpf.h | 8 ++++---- kernel/bpf/core.c | 4 ++-- kernel/bpf/offload.c | 4 ++-- kernel/bpf/syscall.c | 22 +++++++++++----------- kernel/bpf/verifier.c | 18 +++++++++--------- net/core/dev.c | 4 ++-- net/core/filter.c | 2 +- 7 files changed, 31 insertions(+), 31 deletions(-)
include/linux/bpf.h +4 -4
@@ -2481,12 +2481,12 @@ void unpriv_ebpf_notify(int new_state);#if defined(CONFIG_NET ) && defined(CONFIG_BPF_SYSCALL )int bpf_prog_offload_init (struct bpf_prog *prog,union bpf_attr *attr); -static inline bool bpf_prog_is_dev_bound (const struct bpf_prog_aux *aux) +static inline bool bpf_prog_is_offloaded (const struct bpf_prog_aux *aux) {return aux->offload_requested; } -static inline bool bpf_map_is_dev_bound (struct bpf_map *map) +static inline bool bpf_map_is_offloaded (struct bpf_map *map) {return unlikely (map->ops == &bpf_map_offload_ops); }
@@ -2513,12 +2513,12 @@ static inline int bpf_prog_offload_init(struct bpf_prog *prog,return -EOPNOTSUPP ; } -static inline bool bpf_prog_is_dev_bound (struct bpf_prog_aux *aux) +static inline bool bpf_prog_is_offloaded (struct bpf_prog_aux *aux) {return false; } -static inline bool bpf_map_is_dev_bound (struct bpf_map *map) +static inline bool bpf_map_is_offloaded (struct bpf_map *map) {return false; }
kernel/bpf/core.c +2 -2
@@ -2182,7 +2182,7 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)* valid program, which in this case would simply not * be JITed, but falls back to the interpreter. */ -if (!bpf_prog_is_dev_bound (fp->aux)) { +if (!bpf_prog_is_offloaded (fp->aux)) { *err =bpf_prog_alloc_jited_linfo (fp);if (*err)return fp;
@@ -2553,7 +2553,7 @@ static void bpf_prog_free_deferred(struct work_struct *work)#endif bpf_free_used_maps (aux);bpf_free_used_btfs (aux); -if (bpf_prog_is_dev_bound (aux)) +if (bpf_prog_is_offloaded (aux))bpf_prog_offload_destroy (aux->prog);#ifdef CONFIG_PERF_EVENTS if (aux->prog->has_callchain_buf)
kernel/bpf/offload.c +2 -2
@@ -549,7 +549,7 @@ static bool __bpf_offload_dev_match(struct bpf_prog *prog,struct bpf_offload_netdev *ondev1, *ondev2;struct bpf_prog_offload *offload; -if (!bpf_prog_is_dev_bound (prog->aux)) +if (!bpf_prog_is_offloaded (prog->aux))return false; offload = prog->aux->offload;
@@ -581,7 +581,7 @@ bool bpf_offload_prog_map_match(struct bpf_prog *prog, struct bpf_map *map)struct bpf_offloaded_map *offmap;bool ret; -if (!bpf_map_is_dev_bound (map)) +if (!bpf_map_is_offloaded (map))return bpf_map_offload_neutral (map); offmap =map_to_offmap (map);
kernel/bpf/syscall.c +11 -11
@@ -181,7 +181,7 @@ static int bpf_map_update_value(struct bpf_map *map, struct file *map_file,int err;/* Need to create a kthread, thus must support schedule */ -if (bpf_map_is_dev_bound (map)) { +if (bpf_map_is_offloaded (map)) {return bpf_map_offload_update_elem (map, key, value, flags); }else if (map->map_type ==BPF_MAP_TYPE_CPUMAP || map->map_type ==BPF_MAP_TYPE_STRUCT_OPS ) {
@@ -238,7 +238,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value,void *ptr;int err; -if (bpf_map_is_dev_bound (map)) +if (bpf_map_is_offloaded (map))return bpf_map_offload_lookup_elem (map, key, value);bpf_disable_instrumentation ();
@@ -1483,7 +1483,7 @@ static int map_delete_elem(union bpf_attr *attr, bpfptr_t uattr) goto err_put; } -if (bpf_map_is_dev_bound (map)) { +if (bpf_map_is_offloaded (map)) { err =bpf_map_offload_delete_elem (map, key); goto out; } else if (IS_FD_PROG_ARRAY(map) ||
@@ -1547,7 +1547,7 @@ static int map_get_next_key(union bpf_attr *attr)if (!next_key) goto free_key; -if (bpf_map_is_dev_bound (map)) { +if (bpf_map_is_offloaded (map)) { err =bpf_map_offload_get_next_key (map, key, next_key); goto out; }
@@ -1605,7 +1605,7 @@ int generic_map_delete_batch(struct bpf_map *map, map->key_size))break ; -if (bpf_map_is_dev_bound (map)) { +if (bpf_map_is_offloaded (map)) { err =bpf_map_offload_delete_elem (map, key);break ; }
@@ -1851,7 +1851,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) map->map_type == BPF_MAP_TYPE_PERCPU_HASH || map->map_type == BPF_MAP_TYPE_LRU_HASH || map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH) { -if (!bpf_map_is_dev_bound (map)) { +if (!bpf_map_is_offloaded (map)) {bpf_disable_instrumentation ();rcu_read_lock (); err = map->ops->map_lookup_and_delete_elem (map, key, value, attr->flags);
@@ -1944,7 +1944,7 @@ static int find_prog_type(enum bpf_prog_type type, struct bpf_prog *prog)if (!ops)return -EINVAL ; -if (!bpf_prog_is_dev_bound (prog->aux)) +if (!bpf_prog_is_offloaded (prog->aux)) prog->aux->ops = ops;else prog->aux->ops = &bpf_offload_prog_ops;
@@ -2255,7 +2255,7 @@ bool bpf_prog_get_ok(struct bpf_prog *prog,if (prog->type != *attach_type)return false; -if (bpf_prog_is_dev_bound (prog->aux) && !attach_drv) +if (bpf_prog_is_offloaded (prog->aux) && !attach_drv)return false;return true;
@@ -2598,7 +2598,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr)atomic64_set (&prog->aux->refcnt,1 ); prog->gpl_compatible = is_gpl ?1 :0 ; -if (bpf_prog_is_dev_bound (prog->aux)) { +if (bpf_prog_is_offloaded (prog->aux)) { err =bpf_prog_offload_init (prog, attr);if (err) goto free_prog_sec;
@@ -3997,7 +3997,7 @@ static int bpf_prog_get_info_by_fd(struct file *file,return -EFAULT ; } -if (bpf_prog_is_dev_bound (prog->aux)) { +if (bpf_prog_is_offloaded (prog->aux)) { err =bpf_prog_offload_info_fill (&info, prog);if (err)return err;
@@ -4225,7 +4225,7 @@ static int bpf_map_get_info_by_fd(struct file *file, } info.btf_vmlinux_value_type_id = map->btf_vmlinux_value_type_id; -if (bpf_map_is_dev_bound (map)) { +if (bpf_map_is_offloaded (map)) { err =bpf_map_offload_info_fill (&info, map);if (err)return err;
kernel/bpf/verifier.c +9 -9
@@ -13826,7 +13826,7 @@ static int do_check(struct bpf_verifier_env *env) env->prev_log_len = env->log.len_used; } -if (bpf_prog_is_dev_bound (env->prog->aux)) { +if (bpf_prog_is_offloaded (env->prog->aux)) { err =bpf_prog_offload_verify_insn (env, env->insn_idx, env->prev_insn_idx);if (err)
@@ -14306,7 +14306,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, } } -if ((bpf_prog_is_dev_bound (prog->aux) ||bpf_map_is_dev_bound (map)) && +if ((bpf_prog_is_offloaded (prog->aux) ||bpf_map_is_offloaded (map)) && !bpf_offload_prog_map_match (prog, map)) {verbose (env,"offload device mismatch between prog and map\n" );return -EINVAL ;
@@ -14787,7 +14787,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)unsigned int orig_prog_len = env->prog->len;int err; -if (bpf_prog_is_dev_bound (env->prog->aux)) +if (bpf_prog_is_offloaded (env->prog->aux))bpf_prog_offload_remove_insns (env, off, cnt); err =bpf_remove_insns (env->prog, off, cnt);
@@ -14868,7 +14868,7 @@ static void opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env)else continue ; -if (bpf_prog_is_dev_bound (env->prog->aux)) +if (bpf_prog_is_offloaded (env->prog->aux))bpf_prog_offload_replace_insn (env, i, &ja);memcpy (insn, &ja,sizeof (ja));
@@ -15055,7 +15055,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) } } -if (bpf_prog_is_dev_bound (env->prog->aux)) +if (bpf_prog_is_offloaded (env->prog->aux))return 0 ; insn = env->prog->insnsi + delta;
@@ -15455,7 +15455,7 @@ static int fixup_call_args(struct bpf_verifier_env *env)int err =0 ;if (env->prog->jit_requested && - !bpf_prog_is_dev_bound (env->prog->aux)) { + !bpf_prog_is_offloaded (env->prog->aux)) { err =jit_subprogs (env);if (err ==0 )return 0 ;
@@ -16942,7 +16942,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr)if (ret <0 ) goto skip_full_check; -if (bpf_prog_is_dev_bound (env->prog->aux)) { +if (bpf_prog_is_offloaded (env->prog->aux)) { ret =bpf_prog_offload_verifier_prep (env->prog);if (ret) goto skip_full_check;
@@ -16955,7 +16955,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr) ret =do_check_subprogs (env); ret = ret ?:do_check_main (env); -if (ret ==0 &&bpf_prog_is_dev_bound (env->prog->aux)) +if (ret ==0 &&bpf_prog_is_offloaded (env->prog->aux)) ret =bpf_prog_offload_finalize (env); skip_full_check:
@@ -16990,7 +16990,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr)/* do 32-bit optimization after insn patching has done so those patched * insns could be handled correctly. */ -if (ret ==0 && !bpf_prog_is_dev_bound (env->prog->aux)) { +if (ret ==0 && !bpf_prog_is_offloaded (env->prog->aux)) { ret =opt_subreg_zext_lo32_rnd_hi32 (env, attr); env->prog->aux->verifier_zext =bpf_jit_needs_zext () ? !ret : false;
net/core/dev.c +2 -2
@@ -9224,8 +9224,8 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extackNL_SET_ERR_MSG (extack,"Native and generic XDP can't be active at the same time" );return -EEXIST ; } -if (!offload &&bpf_prog_is_dev_bound (new_prog->aux)) { -NL_SET_ERR_MSG (extack,"Using device-bound program without HW_MODE flag is not supported" ); +if (!offload &&bpf_prog_is_offloaded (new_prog->aux)) { +NL_SET_ERR_MSG (extack,"Using offloaded program without HW_MODE flag is not supported" );return -EINVAL ; }if (new_prog->expected_attach_type ==BPF_XDP_DEVMAP ) {
net/core/filter.c +1 -1
@@ -8760,7 +8760,7 @@ static bool xdp_is_valid_access(int off, int size, }if (type ==BPF_WRITE ) { -if (bpf_prog_is_dev_bound (prog->aux)) { +if (bpf_prog_is_offloaded (prog->aux)) {switch (off) {case offsetof(struct xdp_md , rx_queue_index):return __is_valid_xdp_access (off, size);
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 3/17 bpf: Move offload initialization into late_initcall
In reply to Stanislav FomichevSo we don't have to initialize it manually from several paths.
--- kernel/bpf/offload.c | 22 +++++++--------------- 1 file changed, 7 insertions(+), 15 deletions(-)
kernel/bpf/offload.c +7 -15
@@ -56,7 +56,6 @@ static const struct rhashtable_params offdevs_params = { };static struct rhashtable offdevs; -static bool offdevs_inited;static int bpf_dev_offload_check (struct net_device *netdev) {
@@ -72,8 +71,6 @@ bpf_offload_find_netdev(struct net_device *netdev) {lockdep_assert_held (&bpf_devs_lock); -if (!offdevs_inited) -return NULL ;return rhashtable_lookup_fast (&offdevs, &netdev, offdevs_params); }
@@ -673,18 +670,6 @@ struct bpf_offload_dev * bpf_offload_dev_create(const struct bpf_prog_offload_ops *ops, void *priv) { struct bpf_offload_dev *offdev; -int err; - -down_write (&bpf_devs_lock); -if (!offdevs_inited) { - err =rhashtable_init (&offdevs, &offdevs_params); -if (err) { -up_write (&bpf_devs_lock); -return ERR_PTR (err); - } - offdevs_inited = true; - } -up_write (&bpf_devs_lock); offdev =kzalloc (sizeof (*offdev),GFP_KERNEL );if (!offdev)
@@ -710,3 +695,10 @@ void *bpf_offload_dev_priv(struct bpf_offload_dev *offdev)return offdev->priv; }EXPORT_SYMBOL_GPL (bpf_offload_dev_priv); + +static int __init bpf_offload_init (void ) +{ +return rhashtable_init (&offdevs, &offdevs_params); +} + +late_initcall (bpf_offload_init);
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 4/17 bpf: Reshuffle some parts of bpf/offload.c
In reply to Stanislav FomichevTo avoid adding forward declarations in the main patch, shuffle some code around. No functional changes.
--- kernel/bpf/offload.c | 222 +++++++++++++++++++++++-------------------- 1 file changed, 117 insertions(+), 105 deletions(-)
kernel/bpf/offload.c +117 -105
@@ -74,6 +74,121 @@ bpf_offload_find_netdev(struct net_device *netdev)return rhashtable_lookup_fast (&offdevs, &netdev, offdevs_params); } +static int __bpf_offload_dev_netdev_register (struct bpf_offload_dev *offdev, +struct net_device *netdev) +{ +struct bpf_offload_netdev *ondev; +int err; + + ondev =kzalloc (sizeof (*ondev),GFP_KERNEL ); +if (!ondev) +return -ENOMEM ; + + ondev->netdev = netdev; + ondev->offdev = offdev; +INIT_LIST_HEAD (&ondev->progs); +INIT_LIST_HEAD (&ondev->maps); + +down_write (&bpf_devs_lock); + err =rhashtable_insert_fast (&offdevs, &ondev->l, offdevs_params); +if (err) { +netdev_warn (netdev,"failed to register for BPF offload\n" ); + goto err_unlock_free; + } + +list_add (&ondev->offdev_netdevs, &offdev->netdevs); +up_write (&bpf_devs_lock); +return 0 ; + +err_unlock_free: +up_write (&bpf_devs_lock); +kfree (ondev); +return err; +} + +static void __bpf_prog_offload_destroy (struct bpf_prog *prog) +{ +struct bpf_prog_offload *offload = prog->aux->offload; + +if (offload->dev_state) + offload->offdev->ops->destroy (prog); + +/* Make sure BPF_PROG_GET_NEXT_ID can't find this dead program */ +bpf_prog_free_id (prog, true); + +list_del_init (&offload->offloads); +kfree (offload); + prog->aux->offload =NULL ; +} + +static int bpf_map_offload_ndo (struct bpf_offloaded_map *offmap, +enum bpf_netdev_command cmd) +{ +struct netdev_bpf data = {}; +struct net_device *netdev; + +ASSERT_RTNL (); + + data.command = cmd; + data.offmap = offmap; +/* Caller must make sure netdev is valid */ + netdev = offmap->netdev; + +return netdev->netdev_ops->ndo_bpf (netdev, &data); +} + +static void __bpf_map_offload_destroy (struct bpf_offloaded_map *offmap) +{ +WARN_ON (bpf_map_offload_ndo (offmap,BPF_OFFLOAD_MAP_FREE )); +/* Make sure BPF_MAP_GET_NEXT_ID can't find this dead map */ +bpf_map_free_id (&offmap->map, true); +list_del_init (&offmap->offloads); + offmap->netdev =NULL ; +} + +static void __bpf_offload_dev_netdev_unregister (struct bpf_offload_dev *offdev, +struct net_device *netdev) +{ +struct bpf_offload_netdev *ondev, *altdev; +struct bpf_offloaded_map *offmap, *mtmp; +struct bpf_prog_offload *offload, *ptmp; + +ASSERT_RTNL (); + +down_write (&bpf_devs_lock); + ondev =rhashtable_lookup_fast (&offdevs, &netdev, offdevs_params); +if (WARN_ON (!ondev)) + goto unlock; + +WARN_ON (rhashtable_remove_fast (&offdevs, &ondev->l, offdevs_params)); +list_del (&ondev->offdev_netdevs); + +/* Try to move the objects to another netdev of the device */ + altdev =list_first_entry_or_null (&offdev->netdevs, + struct bpf_offload_netdev, + offdev_netdevs); +if (altdev) { +list_for_each_entry (offload, &ondev->progs, offloads) + offload->netdev = altdev->netdev; +list_splice_init (&ondev->progs, &altdev->progs); + +list_for_each_entry (offmap, &ondev->maps, offloads) + offmap->netdev = altdev->netdev; +list_splice_init (&ondev->maps, &altdev->maps); + }else { +list_for_each_entry_safe (offload, ptmp, &ondev->progs, offloads) +__bpf_prog_offload_destroy (offload->prog); +list_for_each_entry_safe (offmap, mtmp, &ondev->maps, offloads) +__bpf_map_offload_destroy (offmap); + } + +WARN_ON (!list_empty (&ondev->progs)); +WARN_ON (!list_empty (&ondev->maps)); +kfree (ondev); +unlock: +up_write (&bpf_devs_lock); +} +int bpf_prog_offload_init (struct bpf_prog *prog,union bpf_attr *attr) {struct bpf_offload_netdev *ondev;
@@ -206,21 +321,6 @@ bpf_prog_offload_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)up_read (&bpf_devs_lock); } -static void __bpf_prog_offload_destroy (struct bpf_prog *prog) -{ -struct bpf_prog_offload *offload = prog->aux->offload; - -if (offload->dev_state) - offload->offdev->ops->destroy (prog); - -/* Make sure BPF_PROG_GET_NEXT_ID can't find this dead program */ -bpf_prog_free_id (prog, true); - -list_del_init (&offload->offloads); -kfree (offload); - prog->aux->offload =NULL ; -} -void bpf_prog_offload_destroy (struct bpf_prog *prog) {down_write (&bpf_devs_lock);
@@ -340,22 +440,6 @@ int bpf_prog_offload_info_fill(struct bpf_prog_info *info,const struct bpf_prog_ops bpf_offload_prog_ops = { }; -static int bpf_map_offload_ndo (struct bpf_offloaded_map *offmap, -enum bpf_netdev_command cmd) -{ -struct netdev_bpf data = {}; -struct net_device *netdev; - -ASSERT_RTNL (); - - data.command = cmd; - data.offmap = offmap; -/* Caller must make sure netdev is valid */ - netdev = offmap->netdev; - -return netdev->netdev_ops->ndo_bpf (netdev, &data); -} -struct bpf_map *bpf_map_offload_map_alloc (union bpf_attr *attr) {struct net *net = current->nsproxy->net_ns;
@@ -405,15 +489,6 @@ struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr)return ERR_PTR (err); } -static void __bpf_map_offload_destroy (struct bpf_offloaded_map *offmap) -{ -WARN_ON (bpf_map_offload_ndo (offmap,BPF_OFFLOAD_MAP_FREE )); -/* Make sure BPF_MAP_GET_NEXT_ID can't find this dead map */ -bpf_map_free_id (&offmap->map, true); -list_del_init (&offmap->offloads); - offmap->netdev =NULL ; -} -void bpf_map_offload_map_free (struct bpf_map *map) {struct bpf_offloaded_map *offmap =map_to_offmap (map);
@@ -592,77 +667,14 @@ bool bpf_offload_prog_map_match(struct bpf_prog *prog, struct bpf_map *map)int bpf_offload_dev_netdev_register (struct bpf_offload_dev *offdev,struct net_device *netdev) { -struct bpf_offload_netdev *ondev; -int err; - - ondev =kzalloc (sizeof (*ondev),GFP_KERNEL ); -if (!ondev) -return -ENOMEM ; - - ondev->netdev = netdev; - ondev->offdev = offdev; -INIT_LIST_HEAD (&ondev->progs); -INIT_LIST_HEAD (&ondev->maps); - -down_write (&bpf_devs_lock); - err =rhashtable_insert_fast (&offdevs, &ondev->l, offdevs_params); -if (err) { -netdev_warn (netdev,"failed to register for BPF offload\n" ); - goto err_unlock_free; - } - -list_add (&ondev->offdev_netdevs, &offdev->netdevs); -up_write (&bpf_devs_lock); -return 0 ; - -err_unlock_free: -up_write (&bpf_devs_lock); -kfree (ondev); -return err; +return __bpf_offload_dev_netdev_register (offdev, netdev); }EXPORT_SYMBOL_GPL (bpf_offload_dev_netdev_register);void bpf_offload_dev_netdev_unregister (struct bpf_offload_dev *offdev,struct net_device *netdev) { -struct bpf_offload_netdev *ondev, *altdev; -struct bpf_offloaded_map *offmap, *mtmp; -struct bpf_prog_offload *offload, *ptmp; - -ASSERT_RTNL (); - -down_write (&bpf_devs_lock); - ondev =rhashtable_lookup_fast (&offdevs, &netdev, offdevs_params); -if (WARN_ON (!ondev)) - goto unlock; - -WARN_ON (rhashtable_remove_fast (&offdevs, &ondev->l, offdevs_params)); -list_del (&ondev->offdev_netdevs); - -/* Try to move the objects to another netdev of the device */ - altdev =list_first_entry_or_null (&offdev->netdevs, - struct bpf_offload_netdev, - offdev_netdevs); -if (altdev) { -list_for_each_entry (offload, &ondev->progs, offloads) - offload->netdev = altdev->netdev; -list_splice_init (&ondev->progs, &altdev->progs); - -list_for_each_entry (offmap, &ondev->maps, offloads) - offmap->netdev = altdev->netdev; -list_splice_init (&ondev->maps, &altdev->maps); - }else { -list_for_each_entry_safe (offload, ptmp, &ondev->progs, offloads) -__bpf_prog_offload_destroy (offload->prog); -list_for_each_entry_safe (offmap, mtmp, &ondev->maps, offloads) -__bpf_map_offload_destroy (offmap); - } - -WARN_ON (!list_empty (&ondev->progs)); -WARN_ON (!list_empty (&ondev->maps)); -kfree (ondev); -unlock: -up_write (&bpf_devs_lock); +__bpf_offload_dev_netdev_unregister (offdev, netdev); }EXPORT_SYMBOL_GPL (bpf_offload_dev_netdev_unregister);
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 5/17 bpf: Introduce device-bound XDP programs
In reply to Stanislav FomichevNew flag BPF_F_XDP_DEV_BOUND_ONLY plus all the infra to have a way to associate a netdev with a BPF program at load time.
netdevsim checks are dropped in favor of generic check in dev_xdp_attach.
--- drivers/net/netdevsim/bpf.c | 4 -- include/linux/bpf.h | 24 +++++++-- include/uapi/linux/bpf.h | 5 ++ kernel/bpf/core.c | 4 +- kernel/bpf/offload.c | 95 +++++++++++++++++++++++++--------- kernel/bpf/syscall.c | 9 ++-- net/core/dev.c | 5 ++ tools/include/uapi/linux/bpf.h | 5 ++ 8 files changed, 113 insertions(+), 38 deletions(-)
drivers/net/netdevsim/bpf.c +0 -4
@@ -315,10 +315,6 @@ nsim_setup_prog_hw_checks(struct netdevsim *ns, struct netdev_bpf *bpf)NSIM_EA (bpf->extack,"xdpoffload of non-bound program" );return -EINVAL ; } -if (!bpf_offload_dev_match (bpf->prog, ns->netdev)) { -NSIM_EA (bpf->extack,"program bound to different dev" ); -return -EINVAL ; - } state = bpf->prog->aux->offload->dev_priv;if (WARN_ON (strcmp (state->state,"xlated" ))) {
include/linux/bpf.h +20 -4
@@ -1261,7 +1261,8 @@ struct bpf_prog_aux {enum bpf_prog_type saved_dst_prog_type;enum bpf_attach_type saved_dst_attach_type;bool verifier_zext;/* Zero extensions has been inserted by verifier. */ -bool offload_requested; +bool dev_bound;/* Program is bound to the netdev. */ +bool offload_requested;/* Program is bound and offloaded to the netdev. */ bool attach_btf_trace;/* true if attaching to BTF-enabled raw tp */ bool func_proto_unreliable;bool sleepable;
@@ -2451,7 +2452,7 @@ void __bpf_free_used_maps(struct bpf_prog_aux *aux,bool bpf_prog_get_ok (struct bpf_prog *,enum bpf_prog_type *,bool );int bpf_prog_offload_compile (struct bpf_prog *prog); -void bpf_prog_offload_destroy (struct bpf_prog *prog); +void bpf_prog_dev_bound_destroy (struct bpf_prog *prog);int bpf_prog_offload_info_fill (struct bpf_prog_info *info,struct bpf_prog *prog);
@@ -2479,7 +2480,13 @@ bool bpf_offload_dev_match(struct bpf_prog *prog, struct net_device *netdev);void unpriv_ebpf_notify (int new_state);#if defined(CONFIG_NET ) && defined(CONFIG_BPF_SYSCALL ) -int bpf_prog_offload_init (struct bpf_prog *prog,union bpf_attr *attr); +int bpf_prog_dev_bound_init (struct bpf_prog *prog,union bpf_attr *attr); +void bpf_dev_bound_netdev_unregister (struct net_device *dev); + +static inline bool bpf_prog_is_dev_bound (const struct bpf_prog_aux *aux) +{ +return aux->dev_bound; +}static inline bool bpf_prog_is_offloaded (const struct bpf_prog_aux *aux) {
@@ -2507,12 +2514,21 @@ void sock_map_unhash(struct sock *sk);void sock_map_destroy (struct sock *sk);void sock_map_close (struct sock *sk,long timeout);#else -static inline int bpf_prog_offload_init (struct bpf_prog *prog, +static inline int bpf_prog_dev_bound_init (struct bpf_prog *prog,union bpf_attr *attr) {return -EOPNOTSUPP ; } +static inline void bpf_dev_bound_netdev_unregister (struct net_device *dev) +{ +} + +static inline bool bpf_prog_is_dev_bound (const struct bpf_prog_aux *aux) +{ +return false; +} +static inline bool bpf_prog_is_offloaded (struct bpf_prog_aux *aux) {return false;
include/uapi/linux/bpf.h +5 -0
@@ -1156,6 +1156,11 @@ enum bpf_link_type {*/ #define BPF_F_XDP_HAS_FRAGS (1U << 5) +/* If BPF_F_XDP_DEV_BOUND_ONLY is used in BPF_PROG_LOAD command, the loaded +* program becomes device-bound but can access XDP metadata. +*/ +#define BPF_F_XDP_DEV_BOUND_ONLY (1U << 6) +/* link_create.kprobe_multi.flags used in LINK_CREATE command for * BPF_TRACE_KPROBE_MULTI attach type to create return probe. */
kernel/bpf/core.c +2 -2
@@ -2553,8 +2553,8 @@ static void bpf_prog_free_deferred(struct work_struct *work)#endif bpf_free_used_maps (aux);bpf_free_used_btfs (aux); -if (bpf_prog_is_offloaded (aux)) -bpf_prog_offload_destroy (aux->prog); +if (bpf_prog_is_dev_bound (aux)) +bpf_prog_dev_bound_destroy (aux->prog);#ifdef CONFIG_PERF_EVENTS if (aux->prog->has_callchain_buf)put_callchain_buffers ();
kernel/bpf/offload.c +71 -24
189 lines not on this page · raw message
kernel/bpf/syscall.c +5 -4
@@ -2491,7 +2491,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr)BPF_F_TEST_STATE_FREQ |BPF_F_SLEEPABLE |BPF_F_TEST_RND_HI32 | - BPF_F_XDP_HAS_FRAGS)) +BPF_F_XDP_HAS_FRAGS | + BPF_F_XDP_DEV_BOUND_ONLY)) return -EINVAL; if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
@@ -2575,7 +2576,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) prog->aux->attach_btf = attach_btf; prog->aux->attach_btf_id = attr->attach_btf_id; prog->aux->dst_prog = dst_prog; - prog->aux->offload_requested = !!attr->prog_ifindex; + prog->aux->dev_bound = !!attr->prog_ifindex; prog->aux->sleepable = attr->prog_flags &BPF_F_SLEEPABLE ; prog->aux->xdp_has_frags = attr->prog_flags &BPF_F_XDP_HAS_FRAGS ;
@@ -2598,8 +2599,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr)atomic64_set (&prog->aux->refcnt,1 ); prog->gpl_compatible = is_gpl ?1 :0 ; -if (bpf_prog_is_offloaded (prog->aux)) { - err =bpf_prog_offload_init (prog, attr); +if (bpf_prog_is_dev_bound (prog->aux)) { + err =bpf_prog_dev_bound_init (prog, attr);if (err) goto free_prog_sec; }
net/core/dev.c +5 -0
@@ -9228,6 +9228,10 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extackNL_SET_ERR_MSG (extack,"Using offloaded program without HW_MODE flag is not supported" );return -EINVAL ; } +if (bpf_prog_is_dev_bound (new_prog->aux) && !bpf_offload_dev_match (new_prog, dev)) { +NL_SET_ERR_MSG (extack,"Program bound to different device" ); +return -EINVAL ; + }if (new_prog->expected_attach_type ==BPF_XDP_DEVMAP ) {NL_SET_ERR_MSG (extack,"BPF_XDP_DEVMAP programs can not be attached to a device" );return -EINVAL ;
@@ -10830,6 +10834,7 @@ void unregister_netdevice_many_notify(struct list_head *head,dev_shutdown (dev);dev_xdp_uninstall (dev); +bpf_dev_bound_netdev_unregister (dev);netdev_offload_xstats_disable_all (dev);
tools/include/uapi/linux/bpf.h +5 -0
@@ -1156,6 +1156,11 @@ enum bpf_link_type {*/ #define BPF_F_XDP_HAS_FRAGS (1U << 5) +/* If BPF_F_XDP_DEV_BOUND_ONLY is used in BPF_PROG_LOAD command, the loaded +* program becomes device-bound but can access XDP metadata. +*/ +#define BPF_F_XDP_DEV_BOUND_ONLY (1U << 6) +/* link_create.kprobe_multi.flags used in LINK_CREATE command for * BPF_TRACE_KPROBE_MULTI attach type to create return probe. */
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 6/17 selftests/bpf: Update expected test_offload.py messages
In reply to Stanislav FomichevGeneric check has a different error message, update the selftest.
--- tools/testing/selftests/bpf/test_offload.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
tools/testing/selftests/bpf/test_offload.py +5 -5
@@ -1039,7 +1039,7 @@ netns = [] offload = bpf_pinned("/sys/fs/bpf/offload") ret, _, err = sim.set_xdp(offload, "drv", fail=False, include_stderr=True) fail(ret == 0, "attached offloaded XDP program to drv") - check_extack(err, "Using device-bound program without HW_MODE flag is not supported.", args) + check_extack(err, "Using offloaded program without HW_MODE flag is not supported.", args) rm("/sys/fs/bpf/offload") sim.wait_for_flush()
@@ -1088,12 +1088,12 @@ netns = [] ret, _, err = sim.set_xdp(pinned, "offload", fail=False, include_stderr=True) fail(ret == 0, "Pinned program loaded for a different device accepted") - check_extack_nsim(err, "program bound to different dev.", args) + check_extack(err, "Program bound to different device.", args) simdev2.remove() ret, _, err = sim.set_xdp(pinned, "offload", fail=False, include_stderr=True) fail(ret == 0, "Pinned program loaded for a removed device accepted") - check_extack_nsim(err, "xdpoffload of non-bound program.", args) + check_extack(err, "Program bound to different device.", args) rm(pin_file) bpftool_prog_list_wait(expected=0)
@@ -1334,12 +1334,12 @@ netns = [] ret, _, err = simA.set_xdp(progB, "offload", force=True, JSON=False, fail=False, include_stderr=True) fail(ret == 0, "cross-ASIC program allowed") - check_extack_nsim(err, "program bound to different dev.", args) + check_extack(err, "Program bound to different device.", args) for d in simdevB.nsims: ret, _, err = d.set_xdp(progA, "offload", force=True, JSON=False, fail=False, include_stderr=True) fail(ret == 0, "cross-ASIC program allowed") - check_extack_nsim(err, "program bound to different dev.", args) + check_extack(err, "Program bound to different device.", args) start_test("Test multi-dev ASIC cross-dev map reuse...")
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 7/17 bpf: XDP metadata RX kfuncs
In reply to Stanislav FomichevDefine a new kfunc set (xdp_metadata_kfunc_ids) which implements all possible XDP metatada kfuncs. Not all devices have to implement them. If kfunc is not supported by the target device, the default implementation is called instead. The verifier, at load time, replaces a call to the generic kfunc with a call to the per-device one. Per-device kfunc pointers are stored in separate struct xdp_metadata_ops.
--- include/linux/bpf.h | 17 ++++++++++- include/linux/netdevice.h | 8 +++++ include/net/xdp.h | 21 +++++++++++++ kernel/bpf/core.c | 8 +++++ kernel/bpf/offload.c | 44 +++++++++++++++++++++++++++ kernel/bpf/verifier.c | 25 ++++++++++++++- net/bpf/test_run.c | 3 ++ net/core/xdp.c | 64 +++++++++++++++++++++++++++++++++++++++ 8 files changed, 188 insertions(+), 2 deletions(-)
include/linux/bpf.h +16 -1
@@ -2480,6 +2480,9 @@ bool bpf_offload_dev_match(struct bpf_prog *prog, struct net_device *netdev);void unpriv_ebpf_notify (int new_state);#if defined(CONFIG_NET ) && defined(CONFIG_BPF_SYSCALL ) +int bpf_dev_bound_kfunc_check (struct bpf_verifier_log *log, +struct bpf_prog_aux *prog_aux); +void *bpf_dev_bound_resolve_kfunc (struct bpf_prog *prog,u32 func_id);int bpf_prog_dev_bound_init (struct bpf_prog *prog,union bpf_attr *attr);void bpf_dev_bound_netdev_unregister (struct net_device *dev);
@@ -2514,8 +2517,20 @@ void sock_map_unhash(struct sock *sk);void sock_map_destroy (struct sock *sk);void sock_map_close (struct sock *sk,long timeout);#else +static inline int bpf_dev_bound_kfunc_check (struct bpf_verifier_log *log, +struct bpf_prog_aux *prog_aux) +{ +return -EOPNOTSUPP ; +} + +static inline void *bpf_dev_bound_resolve_kfunc (struct bpf_prog *prog, +u32 func_id) +{ +return NULL ; +} +static inline int bpf_prog_dev_bound_init (struct bpf_prog *prog, -union bpf_attr *attr) +union bpf_attr *attr) {return -EOPNOTSUPP ; }
include/linux/netdevice.h +8 -0
@@ -74,6 +74,7 @@ struct udp_tunnel_nic_info;struct udp_tunnel_nic ;struct bpf_prog ;struct xdp_buff ; +struct xdp_md ;void synchronize_net (void ); void netdev_set_default_ethtool_ops(struct net_device *dev,
@@ -1618,6 +1619,11 @@ struct net_device_ops { bool cycles); }; +struct xdp_metadata_ops { +int (*xmo_rx_timestamp)(const struct xdp_md *ctx,u64 *timestamp); +int (*xmo_rx_hash)(const struct xdp_md *ctx,u32 *hash); +}; +/** * enum netdev_priv_flags - &struct net_device priv_flags *
@@ -1801,6 +1807,7 @@ enum netdev_ml_priv_type {* * @netdev_ops: Includes several pointers to callbacks, * if one wants to override the ndo_*() functions +* @xdp_metadata_ops: Includes pointers to XDP metadata callbacks. * @ethtool_ops: Management operations * @l3mdev_ops: Layer 3 master device operations * @ndisc_ops: Includes callbacks for different IPv6 neighbour
@@ -2050,6 +2057,7 @@ struct net_device {unsigned int flags;unsigned long long priv_flags;const struct net_device_ops *netdev_ops; +const struct xdp_metadata_ops *xdp_metadata_ops;int ifindex;unsigned short gflags;unsigned short hard_header_len;
include/net/xdp.h +21 -0
@@ -409,4 +409,25 @@ void xdp_attachment_setup(struct xdp_attachment_info *info,#define DEV_MAP_BULK_SIZE XDP_BULK_QUEUE_SIZE +#define XDP_METADATA_KFUNC_xxx \ + XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP, \ + bpf_xdp_metadata_rx_timestamp) \ + XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ + bpf_xdp_metadata_rx_hash) \ + +enum { +#define XDP_METADATA_KFUNC(name, _) name, +XDP_METADATA_KFUNC_xxx +#undef XDP_METADATA_KFUNC +MAX_XDP_METADATA_KFUNC , +}; + +#ifdef CONFIG_NET +u32 bpf_xdp_metadata_kfunc_id (int id); +bool bpf_dev_bound_kfunc_id (u32 btf_id); +#else +static inline u32 bpf_xdp_metadata_kfunc_id (int id) {return 0 ; } +static inline bool bpf_dev_bound_kfunc_id (u32 btf_id) {return false; } +#endif +#endif /* __LINUX_NET_XDP_H__ */
kernel/bpf/core.c +8 -0
@@ -2096,6 +2096,14 @@ bool bpf_prog_map_compatible(struct bpf_map *map,if (fp->kprobe_override)return false; +/* XDP programs inserted into maps are not guaranteed to run on +* a particular netdev (and can run outside driver context entirely +* in the case of devmap and cpumap). Until device checks +* are implemented, prohibit adding dev-bound programs to program maps. +*/ +if (bpf_prog_is_dev_bound (fp->aux)) +return false; +spin_lock (&map->owner.lock);if (!map->owner.type) {/* There's no owner yet where we could check for
kernel/bpf/offload.c +44 -0
50 lines not on this page · raw message
kernel/bpf/verifier.c +24 -1
@@ -2189,6 +2189,12 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)return -EINVAL ; } +if (bpf_dev_bound_kfunc_id (func_id)) { + err =bpf_dev_bound_kfunc_check (&env->log, prog_aux); +if (err) +return err; + } + desc = &tab->descs[tab->nr_descs++]; desc->func_id = func_id; desc->imm = call_imm;
@@ -15499,12 +15505,25 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, struct bpf_insn *insn_buf, int insn_idx, int *cnt) { const struct bpf_kfunc_desc *desc; +void *xdp_kfunc;if (!insn->imm) {verbose (env,"invalid kernel function call not eliminated in verifier pass\n" );return -EINVAL ; } + *cnt =0 ; + +if (bpf_dev_bound_kfunc_id (insn->imm)) { + xdp_kfunc =bpf_dev_bound_resolve_kfunc (env->prog, insn->imm); +if (xdp_kfunc) { + insn->imm =BPF_CALL_IMM (xdp_kfunc); +return 0 ; + } + +/* fallback to default kfunc when not supported by netdev */ + } +/* insn->imm has the btf func_id. Replace it with * an address (relative to __bpf_call_base). */
@@ -15515,7 +15534,6 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return -EFAULT ; } - *cnt =0 ; insn->imm = desc->imm;if (insn->off)return 0 ;
@@ -16522,6 +16540,11 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,if (tgt_prog) {struct bpf_prog_aux *aux = tgt_prog->aux; +if (bpf_prog_is_dev_bound (tgt_prog->aux)) { +bpf_log (log,"Replacing device-bound programs not supported\n" ); +return -EINVAL ; + } +for (i =0 ; i < aux->func_info_cnt; i++)if (aux->func_info[i].type_id == btf_id) { subprog = i;
net/bpf/test_run.c +3 -0
@@ -1300,6 +1300,9 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,if (kattr->test.flags & ~BPF_F_TEST_XDP_LIVE_FRAMES )return -EINVAL ; +if (bpf_prog_is_dev_bound (prog->aux)) +return -EINVAL ; +if (do_live) {if (!batch_size) batch_size =NAPI_POLL_WEIGHT ;
net/core/xdp.c +64 -0
73 lines not on this page · raw message
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 8/17 bpf: Support consuming XDP HW metadata from fext programs
In reply to Stanislav FomichevFrom: Toke Høiland-Jørgensen <[email protected]>
Instead of rejecting the attaching of PROG_TYPE_EXT programs to XDP programs that consume HW metadata, implement support for propagating the offload information. The extension program doesn't need to set a flag or ifindex, these will just be propagated from the target by the verifier. We need to create a separate offload object for the extension program, though, since it can be reattached to a different program later (which means we can't just inherit the offload information from the target).
An additional check is added on attach that the new target is compatible with the offload information in the extension prog.
--- include/linux/bpf.h | 14 ++++++ kernel/bpf/offload.c | 112 +++++++++++++++++++++++++++++++----------- kernel/bpf/syscall.c | 7 +++ kernel/bpf/verifier.c | 5 +- 4 files changed, 106 insertions(+), 32 deletions(-)
include/linux/bpf.h +14 -0
@@ -2484,6 +2484,7 @@ int bpf_dev_bound_kfunc_check(struct bpf_verifier_log *log, struct bpf_prog_aux *prog_aux);void *bpf_dev_bound_resolve_kfunc (struct bpf_prog *prog,u32 func_id);int bpf_prog_dev_bound_init (struct bpf_prog *prog,union bpf_attr *attr); +int bpf_prog_dev_bound_inherit (struct bpf_prog *new_prog,struct bpf_prog *old_prog);void bpf_dev_bound_netdev_unregister (struct net_device *dev);static inline bool bpf_prog_is_dev_bound (const struct bpf_prog_aux *aux)
@@ -2496,6 +2497,8 @@ static inline bool bpf_prog_is_offloaded(const struct bpf_prog_aux *aux)return aux->offload_requested; } +bool bpf_prog_dev_bound_match (const struct bpf_prog *lhs,const struct bpf_prog *rhs); +static inline bool bpf_map_is_offloaded (struct bpf_map *map) {return unlikely (map->ops == &bpf_map_offload_ops);
@@ -2535,6 +2538,12 @@ static inline int bpf_prog_dev_bound_init(struct bpf_prog *prog,return -EOPNOTSUPP ; } +static inline int bpf_prog_dev_bound_inherit (struct bpf_prog *new_prog, +struct bpf_prog *old_prog) +{ +return -EOPNOTSUPP ; +} +static inline void bpf_dev_bound_netdev_unregister (struct net_device *dev) { }
@@ -2549,6 +2558,11 @@ static inline bool bpf_prog_is_offloaded(struct bpf_prog_aux *aux)return false; } +static inline bool bpf_prog_dev_bound_match (const struct bpf_prog *lhs,const struct bpf_prog *rhs) +{ +return false; +} +static inline bool bpf_map_is_offloaded (struct bpf_map *map) {return false;
kernel/bpf/offload.c +82 -30
156 lines not on this page · raw message
kernel/bpf/syscall.c +7 -0
@@ -2605,6 +2605,13 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) goto free_prog_sec; } +if (type ==BPF_PROG_TYPE_EXT && dst_prog && +bpf_prog_is_dev_bound (dst_prog->aux)) { + err =bpf_prog_dev_bound_inherit (prog, dst_prog); +if (err) + goto free_prog_sec; + } +/* find program type: socket_filter vs tracing_filter */ err =find_prog_type (type, prog);if (err <0 )
kernel/bpf/verifier.c +3 -2
@@ -16540,8 +16540,9 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,if (tgt_prog) {struct bpf_prog_aux *aux = tgt_prog->aux; -if (bpf_prog_is_dev_bound (tgt_prog->aux)) { -bpf_log (log,"Replacing device-bound programs not supported\n" ); +if (bpf_prog_is_dev_bound (prog->aux) && + !bpf_prog_dev_bound_match (prog, tgt_prog)) { +bpf_log (log,"Target program bound device mismatch" );return -EINVAL ; }
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 9/17 veth: Introduce veth_xdp_buff wrapper for xdp_buff
In reply to Stanislav FomichevNo functional changes. Boilerplate to allow stuffing more data after xdp_buff.
--- drivers/net/veth.c | 56 +++++++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 25 deletions(-)
drivers/net/veth.c +31 -25
@@ -116,6 +116,10 @@ static struct { { "peer_ifindex" }, }; +struct veth_xdp_buff { +struct xdp_buff xdp; +}; +static int veth_get_link_ksettings (struct net_device *dev,struct ethtool_link_ksettings *cmd) {
@@ -592,23 +596,24 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq,rcu_read_lock (); xdp_prog =rcu_dereference (rq->xdp_prog);if (likely (xdp_prog)) { -struct xdp_buff xdp; +struct veth_xdp_buff vxbuf; +struct xdp_buff *xdp = &vxbuf.xdp;u32 act; -xdp_convert_frame_to_buff (frame, &xdp); - xdp.rxq = &rq->xdp_rxq; +xdp_convert_frame_to_buff (frame, xdp); + xdp->rxq = &rq->xdp_rxq; - act =bpf_prog_run_xdp (xdp_prog, &xdp); + act =bpf_prog_run_xdp (xdp_prog, xdp);switch (act) {case XDP_PASS : -if (xdp_update_frame_from_buff (&xdp, frame)) +if (xdp_update_frame_from_buff (xdp, frame)) goto err_xdp;break ;case XDP_TX : orig_frame = *frame; - xdp.rxq->mem = frame->mem; -if (unlikely (veth_xdp_tx (rq, &xdp, bq) <0 )) { + xdp->rxq->mem = frame->mem; +if (unlikely (veth_xdp_tx (rq, xdp, bq) <0 )) {trace_xdp_exception (rq->dev, xdp_prog, act); frame = &orig_frame; stats->rx_drops++;
@@ -619,8 +624,8 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq, goto xdp_xmit;case XDP_REDIRECT : orig_frame = *frame; - xdp.rxq->mem = frame->mem; -if (xdp_do_redirect (rq->dev, &xdp, xdp_prog)) { + xdp->rxq->mem = frame->mem; +if (xdp_do_redirect (rq->dev, xdp, xdp_prog)) { frame = &orig_frame; stats->rx_drops++; goto err_xdp;
@@ -801,7 +806,8 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, {void *orig_data, *orig_data_end;struct bpf_prog *xdp_prog; -struct xdp_buff xdp; +struct veth_xdp_buff vxbuf; +struct xdp_buff *xdp = &vxbuf.xdp;u32 act, metalen;int off;
@@ -815,22 +821,22 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, }__skb_push (skb, skb->data -skb_mac_header (skb)); -if (veth_convert_skb_to_xdp_buff (rq, &xdp, &skb)) +if (veth_convert_skb_to_xdp_buff (rq, xdp, &skb)) goto drop; - orig_data = xdp.data; - orig_data_end = xdp.data_end; + orig_data = xdp->data; + orig_data_end = xdp->data_end; - act =bpf_prog_run_xdp (xdp_prog, &xdp); + act =bpf_prog_run_xdp (xdp_prog, xdp);switch (act) {case XDP_PASS :break ;case XDP_TX : -veth_xdp_get (&xdp); +veth_xdp_get (xdp);consume_skb (skb); - xdp.rxq->mem = rq->xdp_mem; -if (unlikely (veth_xdp_tx (rq, &xdp, bq) <0 )) { + xdp->rxq->mem = rq->xdp_mem; +if (unlikely (veth_xdp_tx (rq, xdp, bq) <0 )) {trace_xdp_exception (rq->dev, xdp_prog, act); stats->rx_drops++; goto err_xdp;
@@ -839,10 +845,10 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,rcu_read_unlock (); goto xdp_xmit;case XDP_REDIRECT : -veth_xdp_get (&xdp); +veth_xdp_get (xdp);consume_skb (skb); - xdp.rxq->mem = rq->xdp_mem; -if (xdp_do_redirect (rq->dev, &xdp, xdp_prog)) { + xdp->rxq->mem = rq->xdp_mem; +if (xdp_do_redirect (rq->dev, xdp, xdp_prog)) { stats->rx_drops++; goto err_xdp; }
@@ -862,7 +868,7 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,rcu_read_unlock ();/* check if bpf_xdp_adjust_head was used */ - off = orig_data - xdp.data; + off = orig_data - xdp->data;if (off >0 )__skb_push (skb, off);else if (off <0 )
@@ -871,21 +877,21 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,skb_reset_mac_header (skb);/* check if bpf_xdp_adjust_tail was used */ - off = xdp.data_end - orig_data_end; + off = xdp->data_end - orig_data_end;if (off !=0 )__skb_put (skb, off);/* positive on grow, negative on shrink */ /* XDP frag metadata (e.g. nr_frags) are updated in eBPF helpers * (e.g. bpf_xdp_adjust_tail), we need to update data_len here. */ -if (xdp_buff_has_frags (&xdp)) +if (xdp_buff_has_frags (xdp)) skb->data_len =skb_shinfo (skb)->xdp_frags_size;else skb->data_len =0 ; skb->protocol =eth_type_trans (skb, rq->dev); - metalen = xdp.data - xdp.data_meta; + metalen = xdp->data - xdp->data_meta;if (metalen)skb_metadata_set (skb, metalen); out:
@@ -898,7 +904,7 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,return NULL ; err_xdp:rcu_read_unlock (); -xdp_return_buff (&xdp); +xdp_return_buff (xdp); xdp_xmit:return NULL ; }
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 10/17 veth: Support RX XDP metadata
In reply to Stanislav FomichevThe goal is to enable end-to-end testing of the metadata for AF_XDP.
--- drivers/net/veth.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
drivers/net/veth.c +31 -0
@@ -118,6 +118,7 @@ static struct {struct veth_xdp_buff {struct xdp_buff xdp; +struct sk_buff *skb; }; static int veth_get_link_ksettings(struct net_device *dev,
@@ -602,6 +603,7 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq,xdp_convert_frame_to_buff (frame, xdp); xdp->rxq = &rq->xdp_rxq; + vxbuf.skb =NULL ; act =bpf_prog_run_xdp (xdp_prog, xdp);
@@ -823,6 +825,7 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,__skb_push (skb, skb->data -skb_mac_header (skb));if (veth_convert_skb_to_xdp_buff (rq, xdp, &skb)) goto drop; + vxbuf.skb = skb; orig_data = xdp->data; orig_data_end = xdp->data_end;
@@ -1602,6 +1605,28 @@ static int veth_xdp(struct net_device *dev, struct netdev_bpf *xdp) } } +static int veth_xdp_rx_timestamp (const struct xdp_md *ctx,u64 *timestamp) +{ +struct veth_xdp_buff *_ctx = (void *)ctx; + +if (!_ctx->skb) +return -EOPNOTSUPP ; + + *timestamp =skb_hwtstamps (_ctx->skb)->hwtstamp; +return 0 ; +} + +static int veth_xdp_rx_hash (const struct xdp_md *ctx,u32 *hash) +{ +struct veth_xdp_buff *_ctx = (void *)ctx; + +if (!_ctx->skb) +return -EOPNOTSUPP ; + + *hash =skb_get_hash (_ctx->skb); +return 0 ; +} +static const struct net_device_ops veth_netdev_ops = { .ndo_init = veth_dev_init, .ndo_open = veth_open,
@@ -1623,6 +1648,11 @@ static const struct net_device_ops veth_netdev_ops = { .ndo_get_peer_dev = veth_peer_dev, }; +static const struct xdp_metadata_ops veth_xdp_metadata_ops = { + .xmo_rx_timestamp = veth_xdp_rx_timestamp, + .xmo_rx_hash = veth_xdp_rx_hash, +}; +#define VETH_FEATURES (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_HW_CSUM | \ NETIF_F_RXCSUM | NETIF_F_SCTP_CRC | NETIF_F_HIGHDMA | \ NETIF_F_GSO_SOFTWARE | NETIF_F_GSO_ENCAP_ALL | \
@@ -1639,6 +1669,7 @@ static void veth_setup(struct net_device *dev) dev->priv_flags |=IFF_PHONY_HEADROOM ; dev->netdev_ops = &veth_netdev_ops; + dev->xdp_metadata_ops = &veth_xdp_metadata_ops; dev->ethtool_ops = &veth_ethtool_ops; dev->features |=NETIF_F_LLTX ; dev->features |=VETH_FEATURES ;
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 11/17 selftests/bpf: Verify xdp_metadata xdp->af_xdp path
In reply to Stanislav Fomichev- create new netns
- create veth pair (veTX+veRX)
- setup AF_XDP socket for both interfaces
- attach bpf to veRX
- send packet via veTX
- verify the packet has expected metadata at veRX
--- tools/testing/selftests/bpf/Makefile | 2 +- .../selftests/bpf/prog_tests/xdp_metadata.c | 410 ++++++++++++++++++ .../selftests/bpf/progs/xdp_metadata.c | 64 +++ .../selftests/bpf/progs/xdp_metadata2.c | 23 + tools/testing/selftests/bpf/xdp_metadata.h | 15 + 5 files changed, 513 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_metadata.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_metadata.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_metadata2.c create mode 100644 tools/testing/selftests/bpf/xdp_metadata.h
tools/testing/selftests/bpf/Makefile +1 -1
@@ -527,7 +527,7 @@ TRUNNER_BPF_PROGS_DIR := progs TRUNNER_EXTRA_SOURCES := test_progs.c cgroup_helpers.c trace_helpers.c \ network_helpers.c testing_helpers.c \ btf_helpers.c flow_dissector_load.h \ - cap_helpers.c test_loader.c + cap_helpers.c test_loader.c xsk.c TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read $(OUTPUT)/bpf_testmod.ko \ $(OUTPUT)/liburandom_read.so \ $(OUTPUT)/xdp_synproxy \
tools/testing/selftests/bpf/prog_tests/xdp_metadata.c new file +410 -0
410 lines not on this page · raw message
tools/testing/selftests/bpf/progs/xdp_metadata.c new file +64 -0
64 lines not on this page · raw message
tools/testing/selftests/bpf/progs/xdp_metadata2.c new file +23 -0
@@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <vmlinux.h> +#include "xdp_metadata.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> + +extern int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, + __u32 *hash) __ksym; + +int called; + +SEC("freplace/rx") +int freplace_rx (struct xdp_md *ctx) +{ +u32 hash =0 ; +/* Call _any_ metadata function to make sure we don't crash. */ +bpf_xdp_metadata_rx_hash (ctx, &hash); + called++; +return XDP_PASS ; +} + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/xdp_metadata.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#pragma once + +#ifndef ETH_P_IP +#define ETH_P_IP 0x0800 +#endif + +#ifndef ETH_P_IPV6 +#define ETH_P_IPV6 0x86DD +#endif + +struct xdp_meta { +__u64 rx_timestamp; +__u32 rx_hash; +};
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v8 11/17] selftests/bpf: Verify xdp_metadata xdp->af_xdp path
Quoted 20 lines · Stanislav Fomichev
- create new netns
- create veth pair (veTX+veRX)
- setupAF_XDPsocket for both interfaces
- attach bpf to veRX
- send packet via veTX
- verify the packet has expected metadata at veRXCc John Fastabend <[email protected]>Cc David Ahern <[email protected]>Cc Martin KaFai Lau <[email protected]>Cc Jakub Kicinski <[email protected]>Cc Willem de Bruijn <[email protected]>Cc Jesper Dangaard Brouer <[email protected]>Cc Anatoly Burakov <[email protected]>Cc Alexander Lobakin <[email protected]>Cc Magnus Karlsson <[email protected]>Cc Maryam Tahhan <[email protected]>Signed-off-by Stanislav Fomichev <[email protected]>
Quoted 3 lines
--- tools/testing/selftests/bpf/Makefile | 2 +- .../selftests/bpf/prog_tests/xdp_metadata.c | 410 ++++++++++++++++++
The test cannot be run in s390x, so it needs to be in DENYLIST.s390x.
Re: [PATCH bpf-next v8 11/17] selftests/bpf: Verify xdp_metadata xdp->af_xdp path
Quoted 25 lines · Martin KaFai Lau
Quoted 20 lines · Stanislav Fomichev
- create new netns
- create veth pair (veTX+veRX)
- setupAF_XDPsocket for both interfaces
- attach bpf to veRX
- send packet via veTX
- verify the packet has expected metadata at veRXCc John Fastabend <[email protected]>Cc David Ahern <[email protected]>Cc Martin KaFai Lau <[email protected]>Cc Jakub Kicinski <[email protected]>Cc Willem de Bruijn <[email protected]>Cc Jesper Dangaard Brouer <[email protected]>Cc Anatoly Burakov <[email protected]>Cc Alexander Lobakin <[email protected]>Cc Magnus Karlsson <[email protected]>Cc Maryam Tahhan <[email protected]>Signed-off-by Stanislav Fomichev <[email protected]>Quoted 3 lines
--- tools/testing/selftests/bpf/Makefile | 2 +- .../selftests/bpf/prog_tests/xdp_metadata.c | 410 ++++++++++++++++++
Quoted 1 line
The test cannot be run in s390x, so it needs to be in DENYLIST.s390x.
Ahh, good point, will add, thanks!
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v8 11/17] selftests/bpf: Verify xdp_metadata xdp->af_xdp path
In reply to Stanislav FomichevQuoted 6 lines · Stanislav Fomichev
- create new netns
- create veth pair (veTX+veRX)
- setupAF_XDPsocket for both interfaces
- attach bpf to veRX
- send packet via veTX
- verify the packet has expected metadata at veRX
I have seen this in CI for a couple of times. Could you help to take a look?
https://github.com/kernel-patches/bpf/actions/runs/4019879316/jobs/6907358876
#281 xdp_metadata:FAIL
Caught signal #11!
Stack trace:
./test_progs-no_alu32(crash_handler+0x38)[0x563debb469b2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7faaf98a2140]
./test_progs-no_alu32(bpf_object__destroy_skeleton+0[ 122.480620]
new_name[134]: segfault at 563df12a8970 ip 0000563debb71a23 sp 00007ffffb5ad3d0 error 4 in test_progs-no_alu32[563deb94b000+254000] likely on CPU 4 (core 0, socket 4)
x1b)[0x563debb71[ 122.481715] Code: 8b 45 e8 8b 40 38 39 45 f4 7c b0 90 90
c9 c3 f3 0f 1e fa 55 48 89 e5 48 83 ec 10 48 89 7d f8 48 83 7d f8 00 74 67 48 8b 45 f8 <48> 8b 40 40 48 85 c0 74 0c 48 8b 45 f8 48 89 c7 e8 63 ff ff ff 48
a23]
./test_progs-no_alu32(+0x3c192)[0x563deb966192]
./test_progs-no_alu32(test_xdp_metadata+0x1a32)[0x563deb969bc8]
./test_progs-no_alu32(+0x21ce50)[0x563debb46e50]
./test_progs-no_alu32(main+0x54b)[0x563debb4892d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7faaf96ddd0a]
./test_progs-no_alu32(_start+0x2e)[0x563deb94cdce]
Re: [PATCH bpf-next v8 11/17] selftests/bpf: Verify xdp_metadata xdp->af_xdp path
Quoted 10 lines · Martin KaFai Lau
Quoted 6 lines · Stanislav Fomichev
- create new netns
- create veth pair (veTX+veRX)
- setupAF_XDPsocket for both interfaces
- attach bpf to veRX
- send packet via veTX
- verify the packet has expected metadata at veRXI have seen this in CI for a couple of times. Could you help to take a look?
Interesting, sure, will try to take a look, thanks!
Quoted 22 lines
https://github.com/kernel-patches/bpf/actions/runs/4019879316/jobs/6907358876
#281 xdp_metadata:FAIL Caught signal #11! Stack trace: ./test_progs-no_alu32(crash_handler+0x38)[0x563debb469b2] /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7faaf98a2140] ./test_progs-no_alu32(bpf_object__destroy_skeleton+0[ 122.480620]new_name[134]: segfault at
563df12a8970ip0000563debb71a23sp00007ffffb5ad3d0error 4 in test_progs-no_alu32[563deb94b000+254000] likely on CPU 4 (core 0, socket 4)
x1b)[0x563debb71[ 122.481715] Code: 8b 45 e8 8b 40 38 39 45 f4 7c b0 90 90
c9 c3 f3 0f 1e fa 55 48 89 e5 48 83 ec 10 48 89 7d f8 48 83 7d f8 00 74 67 48 8b 45 f8 <48> 8b 40 40 48 85 c0 74 0c 48 8b 45 f8 48 89 c7 e8 63 ff ff ff 48
a23]
./test_progs-no_alu32(+0x3c192)[0x563deb966192] ./test_progs-no_alu32(test_xdp_metadata+0x1a32)[0x563deb969bc8] ./test_progs-no_alu32(+0x21ce50)[0x563debb46e50] ./test_progs-no_alu32(main+0x54b)[0x563debb4892d] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7faaf96ddd0a] ./test_progs-no_alu32(_start+0x2e)[0x563deb94cdce]
PATCH v8 12/17 net/mlx4_en: Introduce wrapper for xdp_buff
In reply to Stanislav FomichevNo functional changes. Boilerplate to allow stuffing more data after xdp_buff.
--- drivers/net/ethernet/mellanox/mlx4/en_rx.c | 26 +++++++++++++--------- 1 file changed, 15 insertions(+), 11 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_rx.c +15 -11
@@ -661,9 +661,14 @@ static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va,#define MLX4_CQE_STATUS_IP_ANY (MLX4_CQE_STATUS_IPV4)#endif +struct mlx4_en_xdp_buff { +struct xdp_buff xdp; +}; +int mlx4_en_process_rx_cq (struct net_device *dev,struct mlx4_en_cq *cq,int budget) {struct mlx4_en_priv *priv =netdev_priv (dev); +struct mlx4_en_xdp_buff mxbuf = {};int factor = priv->cqe_factor;struct mlx4_en_rx_ring *ring;struct bpf_prog *xdp_prog;
@@ -671,7 +676,6 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int budbool doorbell_pending;bool xdp_redir_flush;struct mlx4_cqe *cqe; -struct xdp_buff xdp;int polled =0 ;int index;
@@ -681,7 +685,7 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud ring = priv->rx_ring[cq_ring]; xdp_prog =rcu_dereference_bh (ring->xdp_prog); -xdp_init_buff (&xdp, priv->frag_info[0 ].frag_stride, &ring->xdp_rxq); +xdp_init_buff (&mxbuf.xdp, priv->frag_info[0 ].frag_stride, &ring->xdp_rxq); doorbell_pending = false; xdp_redir_flush = false;
@@ -776,24 +780,24 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud priv->frag_info[0 ].frag_size, DMA_FROM_DEVICE); -xdp_prepare_buff (&xdp, va - frags[0 ].page_offset, +xdp_prepare_buff (&mxbuf.xdp, va - frags[0 ].page_offset, frags[0 ].page_offset, length, false); - orig_data = xdp.data; + orig_data = mxbuf.xdp.data; - act =bpf_prog_run_xdp (xdp_prog, &xdp); + act =bpf_prog_run_xdp (xdp_prog, &mxbuf.xdp); - length = xdp.data_end - xdp.data; -if (xdp.data != orig_data) { - frags[0 ].page_offset = xdp.data - - xdp.data_hard_start; - va = xdp.data; + length = mxbuf.xdp.data_end - mxbuf.xdp.data; +if (mxbuf.xdp.data != orig_data) { + frags[0 ].page_offset = mxbuf.xdp.data - + mxbuf.xdp.data_hard_start; + va = mxbuf.xdp.data; }switch (act) {case XDP_PASS :break ;case XDP_REDIRECT : -if (likely (!xdp_do_redirect (dev, &xdp, xdp_prog))) { +if (likely (!xdp_do_redirect (dev, &mxbuf.xdp, xdp_prog))) { ring->xdp_redirect++; xdp_redir_flush = true; frags[0 ].page =NULL ;
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 13/17 net/mlx4_en: Support RX XDP metadata
In reply to Stanislav FomichevRX timestamp and hash for now. Tested using the prog from the next patch.
Also enabling xdp metadata support; don't see why it's disabled, there is enough headroom..
--- drivers/net/ethernet/mellanox/mlx4/en_clock.c | 13 +++++--- .../net/ethernet/mellanox/mlx4/en_netdev.c | 6 ++++ drivers/net/ethernet/mellanox/mlx4/en_rx.c | 33 ++++++++++++++++++- drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 5 +++ 4 files changed, 52 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_clock.c +9 -4
@@ -58,9 +58,7 @@ u64 mlx4_en_get_cqe_ts(struct mlx4_cqe *cqe)return hi | lo; } -void mlx4_en_fill_hwtstamps (struct mlx4_en_dev *mdev, -struct skb_shared_hwtstamps *hwts, -u64 timestamp) +u64 mlx4_en_get_hwtstamp (struct mlx4_en_dev *mdev,u64 timestamp) {unsigned int seq;u64 nsec;
@@ -70,8 +68,15 @@ void mlx4_en_fill_hwtstamps(struct mlx4_en_dev *mdev, nsec =timecounter_cyc2time (&mdev->clock, timestamp); }while (read_seqretry (&mdev->clock_lock, seq)); +return ns_to_ktime (nsec); +} + +void mlx4_en_fill_hwtstamps (struct mlx4_en_dev *mdev, +struct skb_shared_hwtstamps *hwts, +u64 timestamp) +{memset (hwts,0 ,sizeof (struct skb_shared_hwtstamps )); - hwts->hwtstamp =ns_to_ktime (nsec); + hwts->hwtstamp =mlx4_en_get_hwtstamp (mdev, timestamp); }/**
drivers/net/ethernet/mellanox/mlx4/en_netdev.c +6 -0
@@ -2889,6 +2889,11 @@ static const struct net_device_ops mlx4_netdev_ops_master = { .ndo_bpf = mlx4_xdp, }; +static const struct xdp_metadata_ops mlx4_xdp_metadata_ops = { + .xmo_rx_timestamp = mlx4_en_xdp_rx_timestamp, + .xmo_rx_hash = mlx4_en_xdp_rx_hash, +}; +struct mlx4_en_bond {struct work_struct work;struct mlx4_en_priv *priv;
@@ -3310,6 +3315,7 @@ int mlx4_en_init_netdev(struct mlx4_en_dev *mdev, int port, dev->netdev_ops = &mlx4_netdev_ops_master; else dev->netdev_ops = &mlx4_netdev_ops; + dev->xdp_metadata_ops = &mlx4_xdp_metadata_ops; dev->watchdog_timeo =MLX4_EN_WATCHDOG_TIMEOUT ;netif_set_real_num_tx_queues (dev, priv->tx_ring_num[TX ]);netif_set_real_num_rx_queues (dev, priv->rx_ring_num);
drivers/net/ethernet/mellanox/mlx4/en_rx.c +32 -1
@@ -663,8 +663,35 @@ static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va,struct mlx4_en_xdp_buff {struct xdp_buff xdp; +struct mlx4_cqe *cqe; +struct mlx4_en_dev *mdev; +struct mlx4_en_rx_ring *ring; +struct net_device *dev; }; +int mlx4_en_xdp_rx_timestamp (const struct xdp_md *ctx,u64 *timestamp) +{ +struct mlx4_en_xdp_buff *_ctx = (void *)ctx; + +if (unlikely (_ctx->ring->hwtstamp_rx_filter !=HWTSTAMP_FILTER_ALL )) +return -EOPNOTSUPP ; + + *timestamp =mlx4_en_get_hwtstamp (_ctx->mdev, +mlx4_en_get_cqe_ts (_ctx->cqe)); +return 0 ; +} + +int mlx4_en_xdp_rx_hash (const struct xdp_md *ctx,u32 *hash) +{ +struct mlx4_en_xdp_buff *_ctx = (void *)ctx; + +if (unlikely (!(_ctx->dev->features &NETIF_F_RXHASH ))) +return -EOPNOTSUPP ; + + *hash =be32_to_cpu (_ctx->cqe->immed_rss_invalid); +return 0 ; +} +int mlx4_en_process_rx_cq (struct net_device *dev,struct mlx4_en_cq *cq,int budget) {struct mlx4_en_priv *priv =netdev_priv (dev);
@@ -781,8 +808,12 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud DMA_FROM_DEVICE);xdp_prepare_buff (&mxbuf.xdp, va - frags[0 ].page_offset, - frags[0 ].page_offset, length, false); + frags[0 ].page_offset, length, true); orig_data = mxbuf.xdp.data; + mxbuf.cqe = cqe; + mxbuf.mdev = priv->mdev; + mxbuf.ring = ring; + mxbuf.dev = dev; act =bpf_prog_run_xdp (xdp_prog, &mxbuf.xdp);
drivers/net/ethernet/mellanox/mlx4/mlx4_en.h +5 -0
@@ -796,10 +796,15 @@ void mlx4_en_update_pfc_stats_bitmap(struct mlx4_dev *dev,int mlx4_en_netdev_event (struct notifier_block *this,unsigned long event,void *ptr); +struct xdp_md ; +int mlx4_en_xdp_rx_timestamp (const struct xdp_md *ctx,u64 *timestamp); +int mlx4_en_xdp_rx_hash (const struct xdp_md *ctx,u32 *hash); +/* * Functions for time stamping */ u64 mlx4_en_get_cqe_ts (struct mlx4_cqe *cqe); +u64 mlx4_en_get_hwtstamp (struct mlx4_en_dev *mdev,u64 timestamp);void mlx4_en_fill_hwtstamps (struct mlx4_en_dev *mdev,struct skb_shared_hwtstamps *hwts,u64 timestamp);
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 14/17 xsk: Add cb area to struct xdp_buff_xsk
In reply to Stanislav FomichevFrom: Toke Høiland-Jørgensen <[email protected]>
Add an area after the xdp_buff in struct xdp_buff_xsk that drivers can use to stash extra information to use in metadata kfuncs. The maximum size of 24 bytes means the full xdp_buff_xsk structure will take up exactly two cache lines (with the cb field spanning both). Also add a macro drivers can use to check their own wrapping structs against the available size.
--- include/net/xsk_buff_pool.h | 5 +++++ 1 file changed, 5 insertions(+)
include/net/xsk_buff_pool.h +5 -0
@@ -19,8 +19,11 @@ struct xdp_sock;struct device ;struct page ; +#define XSK_PRIV_MAX 24 +struct xdp_buff_xsk {struct xdp_buff xdp; +u8 cb[XSK_PRIV_MAX ];dma_addr_t dma;dma_addr_t frame_dma;struct xsk_buff_pool *pool;
@@ -28,6 +31,8 @@ struct xdp_buff_xsk {struct list_head free_list_node; }; +#define XSK_CHECK_PRIV_TYPE (t) BUILD_BUG_ON(sizeof(t) > offsetofend(struct xdp_buff_xsk, cb)) +struct xsk_dma_map {dma_addr_t *dma_pages;struct device *dev;
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
PATCH v8 15/17 net/mlx5e: Introduce wrapper for xdp_buff
In reply to Stanislav FomichevFrom: Toke Høiland-Jørgensen <[email protected]>
Preparation for implementing HW metadata kfuncs. No functional change.
--- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 3 +- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 6 +- .../ethernet/mellanox/mlx5/core/en/xsk/rx.c | 33 +++++++---- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 58 +++++++++---------- 4 files changed, 57 insertions(+), 43 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +2 -1
@@ -158,8 +158,9 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq,/* returns true if packet was consumed by xdp */ bool mlx5e_xdp_handle (struct mlx5e_rq *rq,struct page *page, -struct bpf_prog *prog,struct xdp_buff *xdp) +struct bpf_prog *prog,struct mlx5e_xdp_buff *mxbuf) { +struct xdp_buff *xdp = &mxbuf->xdp;u32 act;int err;
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +5 -1
@@ -44,10 +44,14 @@ (MLX5E_XDP_INLINE_WQE_MAX_DS_CNT *MLX5_SEND_WQE_DS - \ sizeof(struct mlx5_wqe_inline_seg)) +struct mlx5e_xdp_buff { +struct xdp_buff xdp; +}; +struct mlx5e_xsk_param ;int mlx5e_xdp_max_mtu (struct mlx5e_params *params,struct mlx5e_xsk_param *xsk);bool mlx5e_xdp_handle (struct mlx5e_rq *rq,struct page *page, -struct bpf_prog *prog,struct xdp_buff *xdp); +struct bpf_prog *prog,struct mlx5e_xdp_buff *mlctx);void mlx5e_xdp_mpwqe_complete (struct mlx5e_xdpsq *sq);bool mlx5e_poll_xdpsq_cq (struct mlx5e_cq *cq);void mlx5e_free_xdpsq_descs (struct mlx5e_xdpsq *sq);
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +21 -12
@@ -8,6 +8,14 @@/* RX data path */ +static struct mlx5e_xdp_buff *xsk_buff_to_mxbuf (struct xdp_buff *xdp) +{ +/* mlx5e_xdp_buff shares its layout with xdp_buff_xsk +* and private mlx5e_xdp_buff fields fall into xdp_buff_xsk->cb +*/ +return (struct mlx5e_xdp_buff *)xdp; +} +int mlx5e_xsk_alloc_rx_mpwqe (struct mlx5e_rq *rq,u16 ix) {struct mlx5e_mpw_info *wi =mlx5e_get_mpw_info (rq, ix);
@@ -22,6 +30,7 @@ int mlx5e_xsk_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix) goto err;BUILD_BUG_ON (sizeof (wi->alloc_units[0 ]) !=sizeof (wi->alloc_units[0 ].xsk)); +XSK_CHECK_PRIV_TYPE (struct mlx5e_xdp_buff ); batch =xsk_buff_alloc_batch (rq->xsk_pool, (struct xdp_buff **)wi->alloc_units, rq->mpwqe.pages_per_wqe);
@@ -233,7 +242,7 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, u32 head_offset, u32 page_idx) { -struct xdp_buff *xdp = wi->alloc_units[page_idx].xsk; +struct mlx5e_xdp_buff *mxbuf =xsk_buff_to_mxbuf (wi->alloc_units[page_idx].xsk);struct bpf_prog *prog;/* Check packet size. Note LRO doesn't use linear SKB */
@@ -249,9 +258,9 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,*/ WARN_ON_ONCE (head_offset); -xsk_buff_set_size (xdp, cqe_bcnt); -xsk_buff_dma_sync_for_cpu (xdp, rq->xsk_pool); -net_prefetch (xdp->data); +xsk_buff_set_size (&mxbuf->xdp, cqe_bcnt); +xsk_buff_dma_sync_for_cpu (&mxbuf->xdp, rq->xsk_pool); +net_prefetch (mxbuf->xdp.data);/* Possible flows: * - XDP_REDIRECT to XSKMAP:
@@ -269,7 +278,7 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,*/ prog =rcu_dereference (rq->xdp_prog); -if (likely (prog &&mlx5e_xdp_handle (rq,NULL , prog, xdp))) { +if (likely (prog &&mlx5e_xdp_handle (rq,NULL , prog, mxbuf))) {if (likely (__test_and_clear_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq->flags)))__set_bit (page_idx, wi->xdp_xmit_bitmap);/* non-atomic */ return NULL ;/* page/packet was consumed by XDP */
@@ -278,14 +287,14 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,/* XDP_PASS: copy the data from the UMEM to a new SKB and reuse the * frame. On SKB allocation failure, NULL is returned. */ -return mlx5e_xsk_construct_skb (rq, xdp); +return mlx5e_xsk_construct_skb (rq, &mxbuf->xdp); }struct sk_buff *mlx5e_xsk_skb_from_cqe_linear (struct mlx5e_rq *rq,struct mlx5e_wqe_frag_info *wi,u32 cqe_bcnt) { -struct xdp_buff *xdp = wi->au->xsk; +struct mlx5e_xdp_buff *mxbuf =xsk_buff_to_mxbuf (wi->au->xsk);struct bpf_prog *prog;/* wi->offset is not used in this function, because xdp->data and the
@@ -295,17 +304,17 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq *rq,*/ WARN_ON_ONCE (wi->offset); -xsk_buff_set_size (xdp, cqe_bcnt); -xsk_buff_dma_sync_for_cpu (xdp, rq->xsk_pool); -net_prefetch (xdp->data); +xsk_buff_set_size (&mxbuf->xdp, cqe_bcnt); +xsk_buff_dma_sync_for_cpu (&mxbuf->xdp, rq->xsk_pool); +net_prefetch (mxbuf->xdp.data); prog =rcu_dereference (rq->xdp_prog); -if (likely (prog &&mlx5e_xdp_handle (rq,NULL , prog, xdp))) +if (likely (prog &&mlx5e_xdp_handle (rq,NULL , prog, mxbuf)))return NULL ;/* page/packet was consumed by XDP */ /* XDP_PASS: copy the data from the UMEM to a new SKB. The frame reuse * will be handled by mlx5e_free_rx_wqe. * On SKB allocation failure, NULL is returned. */ -return mlx5e_xsk_construct_skb (rq, xdp); +return mlx5e_xsk_construct_skb (rq, &mxbuf->xdp); }
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +29 -29
141 lines not on this page · raw message
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
Re: [PATCH bpf-next v8 15/17] net/mlx5e: Introduce wrapper for xdp_buff
Quoted 20 lines · Stanislav Fomichev
From: Toke Høiland-Jørgensen <[email protected]>
Preparation for implementing HW metadata kfuncs. No functional change.
Cc Tariq Toukan <[email protected]>Cc Saeed Mahameed <[email protected]>Cc John Fastabend <[email protected]>Cc David Ahern <[email protected]>Cc Martin KaFai Lau <[email protected]>Cc Jakub Kicinski <[email protected]>Cc Willem de Bruijn <[email protected]>Cc Jesper Dangaard Brouer <[email protected]>Cc Anatoly Burakov <[email protected]>Cc Alexander Lobakin <[email protected]>Cc Magnus Karlsson <[email protected]>Cc Maryam Tahhan <[email protected]>Signed-off-by Toke Høiland-Jørgensen <[email protected]>Signed-off-by Stanislav Fomichev <[email protected]>
Quoted 7 lines
--- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 3 +- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 6 +- .../ethernet/mellanox/mlx5/core/en/xsk/rx.c | 33 +++++++---- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 58 +++++++++---------- 4 files changed, 57 insertions(+), 43 deletions(-)
PATCH v8 16/17 net/mlx5e: Support RX XDP metadata
In reply to Stanislav FomichevFrom: Toke Høiland-Jørgensen <[email protected]>
Support RX hash and timestamp metadata kfuncs. We need to pass in the cqe pointer to the mlx5e_skb_from* functions so it can be retrieved from the XDP ctx to do this.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 5 +- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 5 ++ .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 28 ++++++++++ .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 4 ++ .../ethernet/mellanox/mlx5/core/en/xsk/rx.c | 14 +++++ .../ethernet/mellanox/mlx5/core/en/xsk/rx.h | 2 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 1 + .../net/ethernet/mellanox/mlx5/core/en_rx.c | 51 ++++++++++--------- 8 files changed, 84 insertions(+), 26 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +3 -2
@@ -626,10 +626,11 @@ struct mlx5e_rq;typedef void (*mlx5e_fp_handle_rx_cqe )(struct mlx5e_rq *,struct mlx5_cqe64 *);typedef struct sk_buff * (*mlx5e_fp_skb_from_cqe_mpwrq )(struct mlx5e_rq *rq,struct mlx5e_mpw_info *wi, -u16 cqe_bcnt,u32 head_offset,u32 page_idx); +struct mlx5_cqe64 *cqe,u16 cqe_bcnt, +u32 head_offset,u32 page_idx);typedef struct sk_buff * (*mlx5e_fp_skb_from_cqe )(struct mlx5e_rq *rq,struct mlx5e_wqe_frag_info *wi, -u32 cqe_bcnt); +struct mlx5_cqe64 *cqe,u32 cqe_bcnt);typedef bool (*mlx5e_fp_post_rx_wqes )(struct mlx5e_rq *rq);typedef void (*mlx5e_fp_dealloc_wqe )(struct mlx5e_rq *,u16 );typedef void (*mlx5e_fp_shampo_dealloc_hd )(struct mlx5e_rq *,u16 ,u16 ,bool );
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +5 -0
@@ -73,6 +73,11 @@ int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget);void mlx5e_free_rx_descs (struct mlx5e_rq *rq);void mlx5e_free_rx_in_progress_descs (struct mlx5e_rq *rq); +static inline bool mlx5e_rx_hw_stamp (struct hwtstamp_config *config) +{ +return config->rx_filter ==HWTSTAMP_FILTER_ALL ; +} +/* TX */ netdev_tx_t mlx5e_xmit (struct sk_buff *skb,struct net_device *dev);bool mlx5e_poll_tx_cq (struct mlx5e_cq *cq,int napi_budget);
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +28 -0
@@ -156,6 +156,34 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq,return true; } +static int mlx5e_xdp_rx_timestamp (const struct xdp_md *ctx,u64 *timestamp) +{ +const struct mlx5e_xdp_buff *_ctx = (void *)ctx; + +if (unlikely (!mlx5e_rx_hw_stamp (_ctx->rq->tstamp))) +return -EOPNOTSUPP ; + + *timestamp =mlx5e_cqe_ts_to_ns (_ctx->rq->ptp_cyc2time, + _ctx->rq->clock,get_cqe_ts (_ctx->cqe)); +return 0 ; +} + +static int mlx5e_xdp_rx_hash (const struct xdp_md *ctx,u32 *hash) +{ +const struct mlx5e_xdp_buff *_ctx = (void *)ctx; + +if (unlikely (!(_ctx->xdp.rxq->dev->features &NETIF_F_RXHASH ))) +return -EOPNOTSUPP ; + + *hash =be32_to_cpu (_ctx->cqe->rss_hash_result); +return 0 ; +} + +const struct xdp_metadata_ops mlx5e_xdp_metadata_ops = { + .xmo_rx_timestamp = mlx5e_xdp_rx_timestamp, + .xmo_rx_hash = mlx5e_xdp_rx_hash, +}; +/* returns true if packet was consumed by xdp */ bool mlx5e_xdp_handle (struct mlx5e_rq *rq,struct page *page,struct bpf_prog *prog,struct mlx5e_xdp_buff *mxbuf)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +4 -0
@@ -46,6 +46,8 @@struct mlx5e_xdp_buff {struct xdp_buff xdp; +struct mlx5_cqe64 *cqe; +struct mlx5e_rq *rq; };struct mlx5e_xsk_param ;
@@ -60,6 +62,8 @@ void mlx5e_xdp_rx_poll_complete(struct mlx5e_rq *rq);int mlx5e_xdp_xmit (struct net_device *dev,int n,struct xdp_frame **frames,u32 flags); +extern const struct xdp_metadata_ops mlx5e_xdp_metadata_ops; + INDIRECT_CALLABLE_DECLARE(bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd, struct skb_shared_info *sinfo,
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +14 -0
@@ -52,25 +52,30 @@ int mlx5e_xsk_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix)if (likely (rq->mpwqe.umr_mode ==MLX5E_MPWRQ_UMR_MODE_ALIGNED )) {for (i =0 ; i < batch; i++) { +struct mlx5e_xdp_buff *mxbuf =xsk_buff_to_mxbuf (wi->alloc_units[i].xsk);dma_addr_t addr =xsk_buff_xdp_get_frame_dma (wi->alloc_units[i].xsk); umr_wqe->inline_mtts[i] = (struct mlx5_mtt ) { .ptag =cpu_to_be64 (addr |MLX5_EN_WR ), }; + mxbuf->rq = rq; } }else if (unlikely (rq->mpwqe.umr_mode ==MLX5E_MPWRQ_UMR_MODE_UNALIGNED )) {for (i =0 ; i < batch; i++) { +struct mlx5e_xdp_buff *mxbuf =xsk_buff_to_mxbuf (wi->alloc_units[i].xsk);dma_addr_t addr =xsk_buff_xdp_get_frame_dma (wi->alloc_units[i].xsk); umr_wqe->inline_ksms[i] = (struct mlx5_ksm ) { .key = rq->mkey_be, .va =cpu_to_be64 (addr), }; + mxbuf->rq = rq; } }else if (likely (rq->mpwqe.umr_mode ==MLX5E_MPWRQ_UMR_MODE_TRIPLE )) {u32 mapping_size =1 << (rq->mpwqe.page_shift -2 );for (i =0 ; i < batch; i++) { +struct mlx5e_xdp_buff *mxbuf =xsk_buff_to_mxbuf (wi->alloc_units[i].xsk);dma_addr_t addr =xsk_buff_xdp_get_frame_dma (wi->alloc_units[i].xsk); umr_wqe->inline_ksms[i << 2] = (struct mlx5_ksm) {
@@ -89,6 +94,7 @@ int mlx5e_xsk_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix) .key = rq->mkey_be, .va = cpu_to_be64(rq->wqe_overflow.addr), }; + mxbuf->rq = rq; } }else { __be32 pad_size = cpu_to_be32((1 << rq->mpwqe.page_shift) -
@@ -96,6 +102,7 @@ int mlx5e_xsk_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix)__be32 frame_size =cpu_to_be32 (rq->xsk_pool->chunk_size);for (i =0 ; i < batch; i++) { +struct mlx5e_xdp_buff *mxbuf =xsk_buff_to_mxbuf (wi->alloc_units[i].xsk);dma_addr_t addr =xsk_buff_xdp_get_frame_dma (wi->alloc_units[i].xsk); umr_wqe->inline_klms[i <<1 ] = (struct mlx5_klm ) {
@@ -108,6 +115,7 @@ int mlx5e_xsk_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix) .va = cpu_to_be64(rq->wqe_overflow.addr), .bcount = pad_size, }; + mxbuf->rq = rq; } }
@@ -238,6 +246,7 @@ static struct sk_buff *mlx5e_xsk_construct_skb(struct mlx5e_rq *rq, struct xdp_bstruct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear (struct mlx5e_rq *rq,struct mlx5e_mpw_info *wi, +struct mlx5_cqe64 *cqe,u16 cqe_bcnt,u32 head_offset,u32 page_idx)
@@ -258,6 +267,8 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,*/ WARN_ON_ONCE (head_offset); +/* mxbuf->rq is set on allocation, but cqe is per-packet so set it here */ + mxbuf->cqe = cqe;xsk_buff_set_size (&mxbuf->xdp, cqe_bcnt);xsk_buff_dma_sync_for_cpu (&mxbuf->xdp, rq->xsk_pool);net_prefetch (mxbuf->xdp.data);
@@ -292,6 +303,7 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,struct sk_buff *mlx5e_xsk_skb_from_cqe_linear (struct mlx5e_rq *rq,struct mlx5e_wqe_frag_info *wi, +struct mlx5_cqe64 *cqe,u32 cqe_bcnt) {struct mlx5e_xdp_buff *mxbuf =xsk_buff_to_mxbuf (wi->au->xsk);
@@ -304,6 +316,8 @@ struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq *rq,*/ WARN_ON_ONCE (wi->offset); +/* mxbuf->rq is set on allocation, but cqe is per-packet so set it here */ + mxbuf->cqe = cqe;xsk_buff_set_size (&mxbuf->xdp, cqe_bcnt);xsk_buff_dma_sync_for_cpu (&mxbuf->xdp, rq->xsk_pool);net_prefetch (mxbuf->xdp.data);
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h +2 -0
@@ -13,11 +13,13 @@ int mlx5e_xsk_alloc_rx_wqes_batched(struct mlx5e_rq *rq, u16 ix, int wqe_bulk);int mlx5e_xsk_alloc_rx_wqes (struct mlx5e_rq *rq,u16 ix,int wqe_bulk);struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear (struct mlx5e_rq *rq,struct mlx5e_mpw_info *wi, +struct mlx5_cqe64 *cqe,u16 cqe_bcnt,u32 head_offset,u32 page_idx);struct sk_buff *mlx5e_xsk_skb_from_cqe_linear (struct mlx5e_rq *rq,struct mlx5e_wqe_frag_info *wi, +struct mlx5_cqe64 *cqe,u32 cqe_bcnt);#endif /* __MLX5_EN_XSK_RX_H__ */
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +1 -0
@@ -5053,6 +5053,7 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev)SET_NETDEV_DEV (netdev, mdev->device); netdev->netdev_ops = &mlx5e_netdev_ops; + netdev->xdp_metadata_ops = &mlx5e_xdp_metadata_ops;mlx5e_dcbnl_build_netdev (netdev);
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +27 -24
156 lines not on this page · raw message
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v8 16/17] net/mlx5e: Support RX XDP metadata
Quoted 5 lines · Stanislav Fomichev
From: Toke Høiland-Jørgensen <[email protected]>
Support RX hash and timestamp metadata kfuncs. We need to pass in the cqe pointer to the mlx5e_skb_from* functions so it can be retrieved from the XDP ctx to do this.
Tariq, I believe Stan and Toke have addressed your comments in v7 on the mlx5 changes. I have also tested in my mlx5 setup with the xdp_hw_metadata in patch 17. Please help to take a look at patch 15, 16, and ack them if they look good to you also.
Re: [PATCH bpf-next v8 16/17] net/mlx5e: Support RX XDP metadata
In reply to Stanislav FomichevQuoted 22 lines · Stanislav Fomichev
From: Toke Høiland-Jørgensen <[email protected]>
Support RX hash and timestamp metadata kfuncs. We need to pass in the cqe pointer to the mlx5e_skb_from* functions so it can be retrieved from the XDP ctx to do this.
Cc Tariq Toukan <[email protected]>Cc Saeed Mahameed <[email protected]>Cc John Fastabend <[email protected]>Cc David Ahern <[email protected]>Cc Martin KaFai Lau <[email protected]>Cc Jakub Kicinski <[email protected]>Cc Willem de Bruijn <[email protected]>Cc Jesper Dangaard Brouer <[email protected]>Cc Anatoly Burakov <[email protected]>Cc Alexander Lobakin <[email protected]>Cc Magnus Karlsson <[email protected]>Cc Maryam Tahhan <[email protected]>Signed-off-by Toke Høiland-Jørgensen <[email protected]>Signed-off-by Stanislav Fomichev <[email protected]>
Quoted 11 lines
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 5 +- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 5 ++ .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 28 ++++++++++ .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 4 ++ .../ethernet/mellanox/mlx5/core/en/xsk/rx.c | 14 +++++ .../ethernet/mellanox/mlx5/core/en/xsk/rx.h | 2 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 1 + .../net/ethernet/mellanox/mlx5/core/en_rx.c | 51 ++++++++++--------- 8 files changed, 84 insertions(+), 26 deletions(-)
Thanks for your patches.
PATCH v8 17/17 selftests/bpf: Simple program to dump XDP RX metadata
In reply to Stanislav FomichevTo be used for verification of driver implementations. Note that the skb path is gone from the series, but I'm still keeping the implementation for any possible future work.
$ xdp_hw_metadata <ifname>
On the other machine:
$ echo -n xdp | nc -u -q1 <target> 9091 # for AF_XDP $ echo -n skb | nc -u -q1 <target> 9092 # for skb
Sample output:
# xdp xsk_ring_cons__peek: 1 0x19f9090: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000 rx_timestamp_supported: 1 rx_timestamp: 1667850075063948829 0x19f9090: complete idx=8 addr=8000
# skb found skb hwtstamp = 1668314052.854274681
Decoding:
# xdp rx_timestamp=1667850075.063948829
$ date -d @1667850075 Mon Nov 7 11:41:15 AM PST 2022 $ date Mon Nov 7 11:42:05 AM PST 2022
# skb $ date -d @1668314052 Sat Nov 12 08:34:12 PM PST 2022 $ date Sat Nov 12 08:37:06 PM PST 2022
--- tools/testing/selftests/bpf/.gitignore | 1 + tools/testing/selftests/bpf/Makefile | 7 +- .../selftests/bpf/progs/xdp_hw_metadata.c | 81 ++++ tools/testing/selftests/bpf/xdp_hw_metadata.c | 403 ++++++++++++++++++ 4 files changed, 491 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_hw_metadata.c create mode 100644 tools/testing/selftests/bpf/xdp_hw_metadata.c
tools/testing/selftests/bpf/.gitignore +1 -0
@@ -47,3 +47,4 @@ test_cpp xskxceiver xdp_redirect_multi xdp_synproxy +xdp_hw_metadata
tools/testing/selftests/bpf/Makefile +6 -1
@@ -83,7 +83,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \ - xskxceiver xdp_redirect_multi xdp_synproxy veristat + xskxceiver xdp_redirect_multi xdp_synproxy veristat xdp_hw_metadata TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read $(OUTPUT)/sign-file TEST_GEN_FILES += liburandom_read.so
@@ -383,6 +383,7 @@ test_subskeleton.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib test_subskeleton_lib.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib.bpf.o test_usdt.skel.h-deps := test_usdt.bpf.o test_usdt_multispec.bpf.o xsk_xdp_progs.skel.h-deps := xsk_xdp_progs.bpf.o +xdp_hw_metadata.skel.h-deps := xdp_hw_metadata.bpf.o LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps)))
@@ -580,6 +581,10 @@ $(OUTPUT)/xskxceiver: xskxceiver.c $(OUTPUT)/xsk.o $(OUTPUT)/xsk_xdp_progs.skel. $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@ +$(OUTPUT)/xdp_hw_metadata: xdp_hw_metadata.c $(OUTPUT)/network_helpers.o $(OUTPUT)/xsk.o $(OUTPUT)/xdp_hw_metadata.skel.h | $(OUTPUT) + $(call msg,BINARY,,$@) + $(Q)$(CC) $(CFLAGS) -static $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@ + # Make sure we are able to include and link libbpf against c++. $(OUTPUT)/test_cpp: test_cpp.cpp $(OUTPUT)/test_core_extern.skel.h $(BPFOBJ) $(call msg,CXX,,$@)
tools/testing/selftests/bpf/progs/xdp_hw_metadata.c new file +81 -0
81 lines not on this page · raw message
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +403 -0
403 lines not on this page · raw message
Signature · 1 line
2.39.0.246.g2a6d74b583-goog
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v8 17/17] selftests/bpf: Simple program to dump XDP RX metadata
Quoted 22 lines · Stanislav Fomichev
tools/testing/selftests/bpf/Makefile +2
-1@@ -83,7 +83,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \ - xskxceiver xdp_redirect_multi xdp_synproxy veristat + xskxceiver xdp_redirect_multi xdp_synproxy veristat xdp_hw_metadata TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read $(OUTPUT)/sign-file TEST_GEN_FILES += liburandom_read.so@@ -383,6 +383,7 @@ test_subskeleton.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib test_subskeleton_lib.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib.bpf.o test_usdt.skel.h-deps := test_usdt.bpf.o test_usdt_multispec.bpf.o xsk_xdp_progs.skel.h-deps := xsk_xdp_progs.bpf.o +xdp_hw_metadata.skel.h-deps := xdp_hw_metadata.bpf.o LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps)))
tools/testing/selftests/bpf/Makefile $(OUTPUT)/xskxceiver: xskxceiver.c $(OUTPUT)/xsk.o $(OUTPUT)/xsk_xdp_progs.skel. · line 586
Quoted 6 lines
tools/testing/selftests/bpf/Makefile +3
-0@@ -580,6 +581,10 @@ $(OUTPUT)/xskxceiver: xskxceiver.c $(OUTPUT)/xsk.o $(OUTPUT)/xsk_xdp_progs.skel. $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@ +$(OUTPUT)/xdp_hw_metadata: xdp_hw_metadata.c $(OUTPUT)/network_helpers.o $(OUTPUT)/xsk.o $(OUTPUT)/xdp_hw_metadata.skel.h | $(OUTPUT) + $(call msg,BINARY,,$@) + $(Q)$(CC) $(CFLAGS) -static $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@
My dev machine fails on '-static' :(. A few machines that I got also don't have those static libraries, so likely the default environment that I got here.
It seems to be the only binary using '-static' in this Makefile. Can it be removed or at least not the default?
Re: [PATCH bpf-next v8 17/17] selftests/bpf: Simple program to dump XDP RX metadata
Quoted 34 lines · Martin KaFai Lau
Quoted 22 lines · Stanislav Fomichev
tools/testing/selftests/bpf/Makefile +2
-1@@ -83,7 +83,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \ - xskxceiver xdp_redirect_multi xdp_synproxy veristat + xskxceiver xdp_redirect_multi xdp_synproxy veristat xdp_hw_metadata TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read $(OUTPUT)/sign-file TEST_GEN_FILES += liburandom_read.so@@ -383,6 +383,7 @@ test_subskeleton.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib test_subskeleton_lib.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib.bpf.o test_usdt.skel.h-deps := test_usdt.bpf.o test_usdt_multispec.bpf.o xsk_xdp_progs.skel.h-deps := xsk_xdp_progs.bpf.o +xdp_hw_metadata.skel.h-deps := xdp_hw_metadata.bpf.o LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps)))tools/testing/selftests/bpf/Makefile $(OUTPUT)/xskxceiver: xskxceiver.c $(OUTPUT)/xsk.o $(OUTPUT)/xsk_xdp_progs.skel. · line 586
Quoted 6 lines
tools/testing/selftests/bpf/Makefile +3
-0@@ -580,6 +581,10 @@ $(OUTPUT)/xskxceiver: xskxceiver.c $(OUTPUT)/xsk.o $(OUTPUT)/xsk_xdp_progs.skel. $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@ +$(OUTPUT)/xdp_hw_metadata: xdp_hw_metadata.c $(OUTPUT)/network_helpers.o $(OUTPUT)/xsk.o $(OUTPUT)/xdp_hw_metadata.skel.h | $(OUTPUT) + $(call msg,BINARY,,$@) + $(Q)$(CC) $(CFLAGS) -static $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@My dev machine fails on '-static' :(. A few machines that I got also don't have those static libraries, so likely the default environment that I got here.
Quoted 2 lines
It seems to be the only binary using '-static' in this
Makefile. Can it be removed or at least not the default?
Sure, I can leave it out. It's mostly here due to G's environment where it is easier to work with static binaries.
Re: [PATCH bpf-next v8 17/17] selftests/bpf: Simple program to dump XDP RX metadata
In reply to Stanislav FomichevTesting this on mlx5 and I'm not getting the RX-timestamp. See command details below.
Quoted 5 lines · Stanislav Fomichev
To be used for verification of driver implementations. Note that the skb path is gone from the series, but I'm still keeping the implementation for any possible future work.
$ xdp_hw_metadata <ifname>
sudo ./xdp_hw_metadata mlx5p1
Output:
[...cut ...]
open bpf program...
load bpf program...
prepare skb endpoint...
XXX timestamping_enable(): setsockopt(SO_TIMESTAMPING) ret:0 prepare xsk map...
map[0] = 3
map[1] = 4
map[2] = 5
map[3] = 6
map[4] = 7
map[5] = 8
attach bpf program...
poll: 0 (0)
poll: 0 (0)
poll: 0 (0)
poll: 1 (0)xsk_ring_cons__peek: 10x1821788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000rx_timestamp: 0rx_hash: 27733558070x1821788: complete idx=8 addr=8000
poll: 0 (0)
The trace_pipe:
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
<idle>-0 [005] ..s2. 2722.884762: bpf_trace_printk:
forwarding UDP:9091 to AF_XDP
<idle>-0 [005] ..s2. 2722.884771: bpf_trace_printk:
populated rx_hash with 2773355807Quoted 3 lines
On the other machine:
$ echo -n xdp | nc -u -q1 <target> 9091 # for AF_XDP
Fixing the source-port to see if RX-hash remains the same.
$ echo xdp | nc --source-port=2000 --udp 198.18.1.1 9091
Quoted 10 lines
$ echo -n skb | nc -u -q1 <target> 9092 # for skbSample output:
# xdp xsk_ring_cons__peek: 1 0x19f9090: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000 rx_timestamp_supported: 1 rx_timestamp: 1667850075063948829 0x19f9090: complete idx=8 addr=8000
xsk_ring_cons__peek: 10x1821788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000rx_timestamp: 0rx_hash: 27733558070x1821788: complete idx=8 addr=8000
It doesn't look like hardware RX-timestamps are getting enabled.
[... cut to relevant code ...]
Quoted 6 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +0
-0@@ -0,0 +1,403 @@
[...]
Quoted 11 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +11
-0+static void timestamping_enable (int fd,int val) +{ +int ret; + + ret =setsockopt (fd,SOL_SOCKET ,SO_TIMESTAMPING , &val,sizeof (val)); +if (ret <0 ) +error (-1 , errno,"setsockopt(SO_TIMESTAMPING)" ); +} + +int main (int argc,char *argv[]) +{
[...]
tools/testing/selftests/bpf/xdp_hw_metadata.c line 378
Quoted 8 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +8
-0+printf ("prepare skb endpoint...\n" ); + server_fd =start_server (AF_INET6 ,SOCK_DGRAM ,NULL ,9092 ,1000 ); +if (server_fd <0 ) +error (-1 , errno,"start_server" ); +timestamping_enable (server_fd, +SOF_TIMESTAMPING_SOFTWARE | +SOF_TIMESTAMPING_RAW_HARDWARE ); +
I don't think this timestamping_enable() with these flags are enough to enable hardware timestamping.
--Jesper
Re: [PATCH bpf-next v8 17/17] selftests/bpf: Simple program to dump XDP RX metadata
Quoted 4 lines · Jesper Dangaard Brouer
Testing this on mlx5 and I'm not getting the RX-timestamp. See command details below.
CC'ed Toke since I've never tested mlx5 myself.
I was pretty close to getting the setup late last week, let me try to see whether it's ready or not.
Quoted 107 lines · Stanislav Fomichev
Quoted 5 lines
To be used for verification of driver implementations. Note that the skb path is gone from the series, but I'm still keeping the implementation for any possible future work.
$ xdp_hw_metadata <ifname>sudo ./xdp_hw_metadata mlx5p1
Output:
[...cut ...]
open bpf program...
load bpf program...
prepare skb endpoint...
XXXtimestamping_enable():setsockopt(SO_TIMESTAMPING)ret:0 prepare xsk map...
map[0] = 3
map[1] = 4
map[2] = 5
map[3] = 6
map[4] = 7
map[5] = 8
attach bpf program...
poll: 0 (0)
poll: 0 (0)
poll: 0 (0)
poll: 1 (0)xsk_ring_cons__peek: 10x1821788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000rx_timestamp: 0rx_hash: 27733558070x1821788: complete idx=8 addr=8000
poll: 0 (0)The
trace_pipe:$ sudo cat /sys/kernel/debug/tracing/trace_pipe <idle>-0 [005] ..s2. 2722.884762: bpf_trace_printk: forwarding UDP:9091 to AF_XDP <idle>-0 [005] ..s2. 2722.884771: bpf_trace_printk: populated rx_hash with 2773355807Quoted 3 lines
On the other machine:
$ echo -n xdp | nc -u -q1 <target> 9091 # for AF_XDPFixing the source-port to see if RX-hash remains the same.
$ echo xdp | nc --source-port=2000 --udp 198.18.1.1 9091Quoted 10 lines
$ echo -n skb | nc -u -q1 <target> 9092 # for skbSample output:
# xdp xsk_ring_cons__peek: 1 0x19f9090: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000 rx_timestamp_supported: 1 rx_timestamp: 1667850075063948829 0x19f9090: complete idx=8 addr=8000
xsk_ring_cons__peek: 10x1821788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000rx_timestamp: 0rx_hash: 27733558070x1821788: complete idx=8 addr=8000It doesn't look like hardware RX-timestamps are getting enabled.
[... cut to relevant code ...]
Quoted 6 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +0
-0@@ -0,0 +1,403 @@[...]
Quoted 11 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +11
-0+static void timestamping_enable (int fd,int val) +{ +int ret; + + ret =setsockopt (fd,SOL_SOCKET ,SO_TIMESTAMPING , &val,sizeof (val)); +if (ret <0 ) +error (-1 , errno,"setsockopt(SO_TIMESTAMPING)" ); +} + +int main (int argc,char *argv[]) +{[...]
tools/testing/selftests/bpf/xdp_hw_metadata.c line 378
Quoted 8 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +8
-0+printf ("prepare skb endpoint...\n" ); + server_fd =start_server (AF_INET6 ,SOCK_DGRAM ,NULL ,9092 ,1000 ); +if (server_fd <0 ) +error (-1 , errno,"start_server" ); +timestamping_enable (server_fd, +SOF_TIMESTAMPING_SOFTWARE | +SOF_TIMESTAMPING_RAW_HARDWARE ); +I don't think this
timestamping_enable()with these flags are enough to enable hardware timestamping.--Jesper
Re: [PATCH bpf-next v8 17/17] selftests/bpf: Simple program to dump XDP RX metadata
Quoted 107 lines · Stanislav Fomichev
Quoted 4 lines · Jesper Dangaard Brouer
Testing this on mlx5 and I'm not getting the RX-timestamp. See command details below.
CC'ed Toke since I've never tested mlx5 myself.
I was pretty close to getting the setup late last week, let me try to see whether it's ready or not.Quoted 43 lines · Stanislav Fomichev
Quoted 5 lines
To be used for verification of driver implementations. Note that the skb path is gone from the series, but I'm still keeping the implementation for any possible future work.
$ xdp_hw_metadata <ifname>sudo ./xdp_hw_metadata mlx5p1
Output:
[...cut ...]
open bpf program...
load bpf program...
prepare skb endpoint...
XXXtimestamping_enable():setsockopt(SO_TIMESTAMPING)ret:0 prepare xsk map...
map[0] = 3
map[1] = 4
map[2] = 5
map[3] = 6
map[4] = 7
map[5] = 8
attach bpf program...
poll: 0 (0)
poll: 0 (0)
poll: 0 (0)
poll: 1 (0)xsk_ring_cons__peek: 10x1821788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000rx_timestamp: 0rx_hash: 27733558070x1821788: complete idx=8 addr=8000
poll: 0 (0)The
trace_pipe:$ sudo cat /sys/kernel/debug/tracing/trace_pipe <idle>-0 [005] ..s2. 2722.884762: bpf_trace_printk: forwarding UDP:9091 to AF_XDP <idle>-0 [005] ..s2. 2722.884771: bpf_trace_printk: populated rx_hash with 2773355807Quoted 15 lines
Quoted 3 lines
On the other machine:
$ echo -n xdp | nc -u -q1 <target> 9091 # for AF_XDPFixing the source-port to see if RX-hash remains the same.
$ echo xdp | nc --source-port=2000 --udp 198.18.1.1 9091Quoted 7 lines
$ echo -n skb | nc -u -q1 <target> 9092 # for skbSample output:
# xdp xsk_ring_cons__peek: 1 0x19f9090: rx_desc[0]->addr=100000000008000 addr=8100comp_addr=8000
Quoted 10 lines
Quoted 3 lines
rx_timestamp_supported: 1 rx_timestamp: 1667850075063948829 0x19f9090: complete idx=8 addr=8000
xsk_ring_cons__peek: 10x1821788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000rx_timestamp: 0rx_hash: 27733558070x1821788: complete idx=8 addr=8000Quoted 5 lines
It doesn't look like hardware RX-timestamps are getting enabled.
[... cut to relevant code ...]
Review
Quoted 1 line
(quoted diff) +0
-0
b/tools/testing/selftests/bpf/xdp_hw_metadata.cQuoted 7 lines
Quoted 5 lines
new file mode 100644
index 000000000000..0008f0f239e8tools/testing/selftests/bpf/xdp_hw_metadata.c new file +0
-0@@ -0,0 +1,403 @@[...]
Quoted 5 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c
Quoted 5 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +5
-0+static void timestamping_enable (int fd,int val) +{ +int ret; + + ret = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val,sizeof(val));
Quoted 8 lines
Quoted 6 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +6
-0+if (ret <0 ) +error (-1 , errno,"setsockopt(SO_TIMESTAMPING)" ); +} + +int main (int argc,char *argv[]) +{[...]
Quoted 12 lines
Quoted 2 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c
Quoted 2 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +2
-0+printf ("prepare skb endpoint...\n" ); + server_fd = start_server(AF_INET6, SOCK_DGRAM, NULL, 9092,1000);
Quoted 9 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c
Quoted 6 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +6
-0+if (server_fd <0 ) +error (-1 , errno,"start_server" ); +timestamping_enable (server_fd, +SOF_TIMESTAMPING_SOFTWARE | +SOF_TIMESTAMPING_RAW_HARDWARE ); +I don't think this
timestamping_enable()with these flags are enough to enable hardware timestamping.
Yeah, agreed, looks like that's the issue. timestamping_enable() has been used for the xdp->skb path that I've eventually removed from the series, so it's mostly a noop here..
Maybe you can try the following before I send a proper patch?
tools/testing/selftests/bpf/xdp_hw_metadata.c +28 -1
@@ -24,6 +24,7 @@#include <linux/net_tstamp.h> #include <linux/udp.h> #include <linux/sockios.h> +#include <linux/net_tstamp.h> #include <sys/mman.h> #include <net/if.h> #include <poll.h>
@@ -278,13 +279,37 @@ static int rxq_num(const char *ifname) ret =ioctl (fd,SIOCETHTOOL , &ifr);if (ret <0 ) -error (-1 , errno,"socket" ); +error (-1 , errno,"ioctl(SIOCETHTOOL)" );close (fd);return ch.rx_count + ch.combined_count; } +static void hwtstamp_enable (const char *ifname) +{ +struct hwtstamp_config cfg = { + .rx_filter =HWTSTAMP_FILTER_ALL , + + }; + +struct ifreq ifr = { + .ifr_data = (void *)&cfg, + }; +strcpy (ifr.ifr_name, ifname); +int fd, ret; + + fd =socket (AF_UNIX ,SOCK_DGRAM ,0 ); +if (fd <0 ) +error (-1 , errno,"socket" ); + + ret =ioctl (fd,SIOCSHWTSTAMP , &ifr); +if (ret <0 ) +error (-1 , errno,"ioctl(SIOCSHWTSTAMP)" ); + +close (fd); +} +static void cleanup (void ) {LIBBPF_OPTS (bpf_xdp_attach_opts, opts);
@@ -341,6 +366,8 @@ int main(int argc, char *argv[])printf ("rxq: %d\n" , rxq); +hwtstamp_enable (ifname); + rx_xsk =malloc (sizeof (struct xsk ) * rxq);if (!rx_xsk)error (-1 ,ENOMEM ,"malloc" );
Quoted 2 lines
Quoted 2 lines
--Jesper
Re: [PATCH bpf-next v8 17/17] selftests/bpf: Simple program to dump XDP RX metadata
Quoted 118 lines · On 24/01/2023 19.48, [email protected] wrote:
Quoted 117 lines · Stanislav Fomichev
Quoted 4 lines · Jesper Dangaard Brouer
Testing this on mlx5 and I'm not getting the RX-timestamp. See command details below.
CC'ed Toke since I've never tested mlx5 myself.
I was pretty close to getting the setup late last week, let me try to see whether it's ready or not.Quoted 43 lines · Stanislav Fomichev
Quoted 5 lines
To be used for verification of driver implementations. Note that the skb path is gone from the series, but I'm still keeping the implementation for any possible future work.
$ xdp_hw_metadata <ifname>sudo ./xdp_hw_metadata mlx5p1
Output:
[...cut ...]
open bpf program...
load bpf program...
prepare skb endpoint...
XXXtimestamping_enable():setsockopt(SO_TIMESTAMPING)ret:0 prepare xsk map...
map[0] = 3
map[1] = 4
map[2] = 5
map[3] = 6
map[4] = 7
map[5] = 8
attach bpf program...
poll: 0 (0)
poll: 0 (0)
poll: 0 (0)
poll: 1 (0)xsk_ring_cons__peek: 10x1821788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000rx_timestamp: 0rx_hash: 27733558070x1821788: complete idx=8 addr=8000
poll: 0 (0)The
trace_pipe:$ sudo cat /sys/kernel/debug/tracing/trace_pipe <idle>-0 [005] ..s2. 2722.884762: bpf_trace_printk: forwarding UDP:9091 to AF_XDP <idle>-0 [005] ..s2. 2722.884771: bpf_trace_printk: populated rx_hash with 2773355807Quoted 15 lines
Quoted 3 lines
On the other machine:
$ echo -n xdp | nc -u -q1 <target> 9091 # for AF_XDPFixing the source-port to see if RX-hash remains the same.
$ echo xdp | nc --source-port=2000 --udp 198.18.1.1 9091Quoted 7 lines
$ echo -n skb | nc -u -q1 <target> 9092 # for skbSample output:
# xdp xsk_ring_cons__peek: 1 0x19f9090: rx_desc[0]->addr=100000000008000 addr=8100comp_addr=8000
Quoted 10 lines
Quoted 3 lines
rx_timestamp_supported: 1
rx_timestamp: 1667850075063948829
0x19f9090: complete idx=8 addr=8000
xsk_ring_cons__peek: 10x1821788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000rx_timestamp: 0rx_hash: 27733558070x1821788: complete idx=8 addr=8000Quoted 5 lines
It doesn't look like hardware RX-timestamps are getting enabled.
[... cut to relevant code ...]
Review
Quoted 1 line
(quoted diff) +0
-0
b/tools/testing/selftests/bpf/xdp_hw_metadata.cQuoted 7 lines
Quoted 5 lines
new file mode 100644
index 000000000000..0008f0f239e8tools/testing/selftests/bpf/xdp_hw_metadata.c new file +0
-0@@ -0,0 +1,403 @@[...]
Quoted 5 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c
Quoted 5 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +5
-0+static void timestamping_enable (int fd,int val) +{ + int ret; + + ret = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val,sizeof(val));
Quoted 10 lines
Quoted 6 lines
+ if (ret < 0)
+error(-1, errno, "setsockopt(SO_TIMESTAMPING)");tools/testing/selftests/bpf/xdp_hw_metadata.c new file +4
-0+} + +int main (int argc,char *argv[]) +{[...]
Quoted 2 lines
+
printf("prepare skb endpoint...\n");
+server_fd= start_server(AF_INET6,SOCK_DGRAM, NULL, 9092,1000);
Quoted 7 lines
Quoted 6 lines
+ if (
server_fd< 0)
+error(-1, errno, "start_server"); + timestamping_enable(server_fd,
+SOF_TIMESTAMPING_SOFTWARE| +SOF_TIMESTAMPING_RAW_HARDWARE); +
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
I don't think this
timestamping_enable()with these flags are enough to enable hardware timestamping.Yeah, agreed, looks like that's the issue.
timestamping_enable()has been used for thexdp->skbpath that I've eventually removed from the series, so it's mostly a noop here..Maybe you can try the following before I send a proper patch?
Yes, below patch fixed the issue, thx.
Now I get HW timestamps, plus I added some software CLOCK_TAI timestamps to compare against.
Output is now:
poll: 1 (0)
xsk_ring_cons__peek: 1
0xf64788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000
rx_hash: 3697961069
rx_timestamp: 1674657672142214773 (sec:1674657672.1422) XDP RX-time: 1674657709561774876 (sec:1674657709.5618) delta sec:37.4196 AF_XDP time: 1674657709561871034 (sec:1674657709.5619) delta
sec:0.0001 (96.158 usec)
0xf64788: complete idx=8 addr=8000
My NIC hardware clock is clearly not synced with system time, as above delta say 37.4 seconds between HW and XDP timestamps (using bpf_ktime_get_tai_ns()).
Time between XDP and AF_XDP wakeup is reported to be 96 usec, which is also higher than I expected. As explained in [1] this is caused by CPU sleep states.
My /dev/cpu_dma_latency was set to 2000000000. Applying tuned-adm profile latency-performance this value change to 2.
$ sudo hexdump --format '"%d\n"' /dev/cpu_dma_latency 2000000000 $ sudo hexdump --format '"%d\n"' /dev/cpu_dma_latency 2
Now the time between XDP and AF_XDP wakeup is reduced to approx 12 usec.
rx_timestamp: 1674659206344977544 (sec:1674659206.3450) XDP RX-time: 1674659243776087765 (sec:1674659243.7761) delta sec:37.4311 AF_XDP time: 1674659243776099841 (sec:1674659243.7761) delta
sec:0.0000 (12.076 usec)
[1] https://github.com/xdp-project/bpf-examples/tree/master/AF_XDP-interaction
Quoted 62 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c +28
-1@@ -24,6 +24,7 @@ #include <linux/net_tstamp.h> #include <linux/udp.h> #include <linux/sockios.h> +#include <linux/net_tstamp.h> #include <sys/mman.h> #include <net/if.h> #include <poll.h>@@ -278,13 +279,37 @@ static int rxq_num(const char *ifname) ret = ioctl(fd, SIOCETHTOOL, &ifr); if (ret < 0) - error(-1, errno, "socket"); + error(-1, errno, "ioctl(SIOCETHTOOL)"); close(fd); return ch.rx_count + ch.combined_count; } +static void hwtstamp_enable (const char *ifname) +{ + struct hwtstamp_config cfg = { + .rx_filter = HWTSTAMP_FILTER_ALL, + + }; + + struct ifreq ifr = { + .ifr_data = (void *)&cfg, + }; + strcpy(ifr.ifr_name, ifname); + int fd, ret; + + fd = socket(AF_UNIX, SOCK_DGRAM, 0); + if (fd < 0) + error(-1, errno, "socket"); + + ret = ioctl(fd, SIOCSHWTSTAMP, &ifr); + if (ret < 0) + error(-1, errno, "ioctl(SIOCSHWTSTAMP)"); + + close(fd); +} + static void cleanup(void) { LIBBPF_OPTS(bpf_xdp_attach_opts, opts);@@ -341,6 +366,8 @@ int main(int argc, char *argv[]) printf("rxq: %d\n", rxq); + hwtstamp_enable(ifname); + rx_xsk = malloc(sizeof(struct xsk) * rxq); if (!rx_xsk) error(-1, ENOMEM, "malloc");
Re: [PATCH bpf-next v8 17/17] selftests/bpf: Simple program to dump XDP RX metadata
Quoted 165 lines · Jesper Dangaard Brouer
Quoted 118 lines · On 24/01/2023 19.48, [email protected] wrote:
Quoted 117 lines · Stanislav Fomichev
Quoted 4 lines · Jesper Dangaard Brouer
Testing this on mlx5 and I'm not getting the RX-timestamp. See command details below.
CC'ed Toke since I've never tested mlx5 myself.
I was pretty close to getting the setup late last week, let me try to see whether it's ready or not.Quoted 43 lines · Stanislav Fomichev
Quoted 5 lines
To be used for verification of driver implementations. Note that the skb path is gone from the series, but I'm still keeping the implementation for any possible future work.
$ xdp_hw_metadata <ifname>sudo ./xdp_hw_metadata mlx5p1
Output:
[...cut ...]
open bpf program...
load bpf program...
prepare skb endpoint...
XXXtimestamping_enable():setsockopt(SO_TIMESTAMPING)ret:0 prepare xsk map...
map[0] = 3
map[1] = 4
map[2] = 5
map[3] = 6
map[4] = 7
map[5] = 8
attach bpf program...
poll: 0 (0)
poll: 0 (0)
poll: 0 (0)
poll: 1 (0)xsk_ring_cons__peek: 10x1821788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000rx_timestamp: 0rx_hash: 27733558070x1821788: complete idx=8 addr=8000
poll: 0 (0)The
trace_pipe:$ sudo cat /sys/kernel/debug/tracing/trace_pipe <idle>-0 [005] ..s2. 2722.884762: bpf_trace_printk: forwarding UDP:9091 to AF_XDP <idle>-0 [005] ..s2. 2722.884771: bpf_trace_printk: populated rx_hash with 2773355807Quoted 15 lines
Quoted 3 lines
On the other machine:
$ echo -n xdp | nc -u -q1 <target> 9091 # for AF_XDPFixing the source-port to see if RX-hash remains the same.
$ echo xdp | nc --source-port=2000 --udp 198.18.1.1 9091Quoted 7 lines
$ echo -n skb | nc -u -q1 <target> 9092 # for skbSample output:
# xdp xsk_ring_cons__peek: 1 0x19f9090: rx_desc[0]->addr=100000000008000 addr=8100comp_addr=8000
Quoted 10 lines
Quoted 3 lines
rx_timestamp_supported: 1 rx_timestamp: 1667850075063948829 0x19f9090: complete idx=8 addr=8000
xsk_ring_cons__peek: 10x1821788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000rx_timestamp: 0rx_hash: 27733558070x1821788: complete idx=8 addr=8000Quoted 5 lines
It doesn't look like hardware RX-timestamps are getting enabled.
[... cut to relevant code ...]
Review
Quoted 1 line
(quoted diff) +0
-0
b/tools/testing/selftests/bpf/xdp_hw_metadata.cQuoted 7 lines
Quoted 5 lines
new file mode 100644
index 000000000000..0008f0f239e8tools/testing/selftests/bpf/xdp_hw_metadata.c new file +0
-0@@ -0,0 +1,403 @@[...]
Quoted 5 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c
Quoted 5 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +5
-0+static void timestamping_enable (int fd,int val) +{ +int ret; + + ret = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val,sizeof(val));
Quoted 10 lines
Quoted 6 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +6
-0+if (ret <0 ) +error (-1 , errno,"setsockopt(SO_TIMESTAMPING)" ); +} + +int main (int argc,char *argv[]) +{[...]
tools/testing/selftests/bpf/xdp_hw_metadata.c
Quoted 2 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +2
-0+printf ("prepare skb endpoint...\n" ); + server_fd = start_server(AF_INET6, SOCK_DGRAM, NULL, 9092,1000);
Quoted 7 lines
Quoted 6 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c new file +6
-0+if (server_fd <0 ) +error (-1 , errno,"start_server" ); +timestamping_enable (server_fd, +SOF_TIMESTAMPING_SOFTWARE | +SOF_TIMESTAMPING_RAW_HARDWARE ); +Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
I don't think this
timestamping_enable()with these flags are enough to enable hardware timestamping.Yeah, agreed, looks like that's the issue.
timestamping_enable()has been used for thexdp->skbpath that I've eventually removed from the series, so it's mostly a noop here..Maybe you can try the following before I send a proper patch?
Yes, below patch fixed the issue, thx.
Now I get HW timestamps, plus I added some software
CLOCK_TAItimestamps to compare against.Output is now:
poll: 1 (0)
xsk_ring_cons__peek: 1
0xf64788: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000
rx_hash: 3697961069rx_timestamp: 1674657672142214773 (sec:1674657672.1422) XDP RX-time: 1674657709561774876 (sec:1674657709.5618) delta sec:37.4196 AF_XDP time: 1674657709561871034 (sec:1674657709.5619) deltasec:0.0001 (96.158 usec)
0xf64788: complete idx=8 addr=8000My NIC hardware clock is clearly not synced with system time, as above delta say 37.4 seconds between HW and XDP timestamps (using
bpf_ktime_get_tai_ns()).Time between XDP and
AF_XDPwakeup is reported to be 96 usec, which is also higher than I expected. As explained in [1] this is caused by CPU sleep states.My
/dev/cpu_dma_latencywas set to 2000000000. Applying tuned-adm profile latency-performance this value change to 2.$ sudo hexdump --format '"%d\n"' /dev/cpu_dma_latency 2000000000 $ sudo hexdump --format '"%d\n"' /dev/cpu_dma_latency 2Now the time between XDP and
AF_XDPwakeup is reduced to approx 12 usec.
Quoted 8 lines
rx_timestamp: 1674659206344977544 (sec:1674659206.3450) XDP RX-time: 1674659243776087765 (sec:1674659243.7761) delta sec:37.4311 AF_XDP time: 1674659243776099841 (sec:1674659243.7761) deltasec:0.0000 (12.076 usec)
[1]
https://github.com/xdp-project/bpf-examples/tree/master/AF_XDP-interaction
Great, thank you for testing and investigating the clock discrepancy! Will send it as a patch later today, will add your Tested-by (if you don't mind).
Quoted 64 lines
Quoted 63 lines
tools/testing/selftests/bpf/xdp_hw_metadata.c +28
-1@@ -24,6 +24,7 @@#include <linux/net_tstamp.h> #include <linux/udp.h> #include <linux/sockios.h> +#include <linux/net_tstamp.h> #include <sys/mman.h> #include <net/if.h> #include <poll.h> @@ -278,13 +279,37 @@ static int rxq_num(const char *ifname) ret =ioctl (fd,SIOCETHTOOL , &ifr);if (ret <0 ) -error (-1 , errno,"socket" ); +error (-1 , errno,"ioctl(SIOCETHTOOL)" );close (fd);return ch.rx_count + ch.combined_count; } +static void hwtstamp_enable (const char *ifname) +{ +struct hwtstamp_config cfg = { + .rx_filter =HWTSTAMP_FILTER_ALL , + + }; + +struct ifreq ifr = { + .ifr_data = (void *)&cfg, + }; +strcpy (ifr.ifr_name, ifname); +int fd, ret; + + fd =socket (AF_UNIX ,SOCK_DGRAM ,0 ); +if (fd <0 ) +error (-1 , errno,"socket" ); + + ret =ioctl (fd,SIOCSHWTSTAMP , &ifr); +if (ret <0 ) +error (-1 , errno,"ioctl(SIOCSHWTSTAMP)" ); + +close (fd); +} +static void cleanup (void ) {LIBBPF_OPTS (bpf_xdp_attach_opts, opts);@@ -341,6 +366,8 @@ int main(int argc, char *argv[])printf ("rxq: %d\n" , rxq); +hwtstamp_enable (ifname); + rx_xsk =malloc (sizeof (struct xsk ) * rxq);if (!rx_xsk)error (-1 ,ENOMEM ,"malloc" );
This series was applied to bpf/bpf-next.git (master) by Martin KaFai Lau <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Martin KaFai Lau <[email protected]>:
Quoted 10 lines · On Thu, 19 Jan 2023 14:15:19 -0800 you wrote:
Please see the first patch in the series for the overall design and use-cases.
See the following email from Toke for the per-packet metadata overhead:
https://lore.kernel.org/bpf/[email protected]/T/#m49d48ea08d525ec88360c7d14c4d34fb0e45e798Recent changes:
- Keep new functions inen/xdp.c, do 'externmlx5_xdp_metadata_ops' (Tariq)[...]
Here is the summary with links:
- [bpf-next,v8,01/17] bpf: Document XDP RX metadata
(no matching commit)
- [bpf-next,v8,02/17] bpf: Rename bpf_{prog,map}_is_dev_bound to is_offloaded (no matching commit)
- [bpf-next,v8,03/17] bpf: Move offload initialization into late_initcall (no matching commit)
- [bpf-next,v8,04/17] bpf: Reshuffle some parts of bpf/offload.c (no matching commit)
- [bpf-next,v8,05/17] bpf: Introduce device-bound XDP programs
(no matching commit)
- [bpf-next,v8,06/17] selftests/bpf: Update expected test_offload.py messages (no matching commit)
- [bpf-next,v8,07/17] bpf: XDP metadata RX kfuncs
(no matching commit)
- [bpf-next,v8,08/17] bpf: Support consuming XDP HW metadata from fext programs (no matching commit)
- [bpf-next,v8,09/17] veth: Introduce veth_xdp_buff wrapper for xdp_buff (no matching commit)
- [bpf-next,v8,10/17] veth: Support RX XDP metadata
(no matching commit)
- [bpf-next,v8,11/17] selftests/bpf: Verify xdp_metadata xdp->af_xdp path (no matching commit)
- [bpf-next,v8,12/17] net/mlx4_en: Introduce wrapper for xdp_buff (no matching commit)
- [bpf-next,v8,13/17] net/mlx4_en: Support RX XDP metadata
(no matching commit)
- [bpf-next,v8,14/17] xsk: Add cb area to struct xdp_buff_xsk
(no matching commit)
- [bpf-next,v8,15/17] net/mlx5e: Introduce wrapper for xdp_buff
https://git.kernel.org/bpf/bpf-next/c/384a13ca8a5d - [bpf-next,v8,16/17] net/mlx5e: Support RX XDP metadata https://git.kernel.org/bpf/bpf-next/c/bc8d405b1ba9
- [bpf-next,v8,17/17] selftests/bpf: Simple program to dump XDP RX metadata (no matching commit)
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Martin KaFai Lau
reviewerQuoted 9 lines · Stanislav Fomichev
Please see the first patch in the series for the overall design and use-cases.
See the following email from Toke for the per-packet metadata overhead:
https://lore.kernel.org/bpf/[email protected]/T/#m49d48ea08d525ec88360c7d14c4d34fb0e45e798Recent changes:
- Keep new functions inen/xdp.c, do 'externmlx5_xdp_metadata_ops' (Tariq)
Quoted 8 lines
- Remove mxbuf pointer and use
xsk_buff_to_mxbuf(Tariq)- Clarify
xdp_buffvs 'XDP frame' (Jesper)- Explicitly mention that
AF_XDPRX descriptor lacks metadata size (Jesper)- Drop libbpf_flags/xdp_flags from selftests and use ifindex instead of ifname (due to recent
xsk.hrefactoring)
Applied with the minor changes in the selftests discussed in patch 11 and 17. Thanks!
Quoted 15 lines · Martin KaFai Lau
Quoted 13 lines · Stanislav Fomichev
Please see the first patch in the series for the overall design and use-cases.
See the following email from Toke for the per-packet metadata overhead:
https://lore.kernel.org/bpf/[email protected]/T/#m49d48ea08d525ec88360c7d14c4d34fb0e45e798Recent changes:
- Keep new functions inen/xdp.c, do 'externmlx5_xdp_metadata_ops' (Tariq)- Remove mxbuf pointer and use
xsk_buff_to_mxbuf(Tariq)- Clarify
xdp_buffvs 'XDP frame' (Jesper)
Quoted 7 lines
Quoted 4 lines
- Explicitly mention that
AF_XDPRX descriptor lacks metadata size (Jesper)- Drop libbpf_flags/xdp_flags from selftests and use ifindex instead of ifname (due to recent
xsk.hrefactoring)Applied with the minor changes in the selftests discussed in patch 11 and 17. Thanks!
Awesome, thanks! I was gonna resend around Wed, but thank you for taking care of that!
From: Stanislav Fomichev <[email protected]>
Date: Mon, 23 Jan 2023 10:55:52 -0800
Quoted 18 lines · Martin KaFai Lau
Quoted 17 lines · Stanislav Fomichev
Quoted 15 lines
Please see the first patch in the series for the overall design and use-cases.
See the following email from Toke for the per-packet metadata overhead:
https://lore.kernel.org/bpf/[email protected]/T/#m49d48ea08d525ec88360c7d14c4d34fb0e45e798Recent changes:
- Keep new functions inen/xdp.c, do 'externmlx5_xdp_metadata_ops' (Tariq)- Remove mxbuf pointer and use
xsk_buff_to_mxbuf(Tariq)- Clarify
xdp_buffvs 'XDP frame' (Jesper)- Explicitly mention that
AF_XDPRX descriptor lacks metadata size (Jesper)
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
- Drop libbpf_flags/xdp_flags from selftests and use ifindex instead of ifname (due to recent
xsk.hrefactoring)Applied with the minor changes in the selftests discussed in patch 11 and 17. Thanks!
Awesome, thanks! I was gonna resend around Wed, but thank you for taking care of that!
Great stuff, congrats! :)
Olek
Quoted 24 lines · Alexander Lobakin
From: Stanislav Fomichev <[email protected]>
Date: Mon, 23 Jan 2023 10:55:52 -0800Quoted 21 lines · Martin KaFai Lau
Quoted 20 lines · Stanislav Fomichev
Quoted 17 lines
Please see the first patch in the series for the overall design and use-cases.
See the following email from Toke for the per-packet metadata overhead:
https://lore.kernel.org/bpf/[email protected]/T/#m49d48ea08d525ec88360c7d14c4d34fb0e45e798Recent changes:
- Keep new functions inen/xdp.c, do 'externmlx5_xdp_metadata_ops' (Tariq)- Remove mxbuf pointer and use
xsk_buff_to_mxbuf(Tariq)- Clarify
xdp_buffvs 'XDP frame' (Jesper)- Explicitly mention that
AF_XDPRX descriptor lacks metadata size (Jesper)- Drop libbpf_flags/xdp_flags from selftests and use ifindex instead of ifname (due to recent
xsk.hrefactoring)
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Applied with the minor changes in the selftests discussed in patch 11 and 17. Thanks!
Awesome, thanks! I was gonna resend around Wed, but thank you for taking care of that!
Great stuff, congrats! :)
Yeah! Thanks for carrying this forward, Stanislav! :)
-Toke
Quoted 26 lines · Toke Høiland-Jørgensen
Quoted 24 lines · Alexander Lobakin
From: Stanislav Fomichev <[email protected]>
Date: Mon, 23 Jan 2023 10:55:52 -0800Quoted 21 lines · Martin KaFai Lau
Quoted 20 lines · Stanislav Fomichev
Quoted 17 lines
Please see the first patch in the series for the overall design and use-cases.
See the following email from Toke for the per-packet metadata overhead:
https://lore.kernel.org/bpf/[email protected]/T/#m49d48ea08d525ec88360c7d14c4d34fb0e45e798Recent changes:
- Keep new functions inen/xdp.c, do 'externmlx5_xdp_metadata_ops' (Tariq)- Remove mxbuf pointer and use
xsk_buff_to_mxbuf(Tariq)- Clarify
xdp_buffvs 'XDP frame' (Jesper)- Explicitly mention that
AF_XDPRX descriptor lacks metadata size (Jesper)- Drop libbpf_flags/xdp_flags from selftests and use ifindex instead of ifname (due to recent
xsk.hrefactoring)
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Applied with the minor changes in the selftests discussed in patch 11 and 17. Thanks!
Awesome, thanks! I was gonna resend around Wed, but thank you for taking care of that!
Great stuff, congrats! :)
Yeah! Thanks for carrying this forward, Stanislav! :)
+1000 -- great work everybody! :-)
To Alexander (Cc Jesse and Tony), do you think someone from Intel could look at extending drivers:
drivers/net/ethernet/intel/igb/ - chip i210 drivers/net/ethernet/intel/igc/ - chip i225 drivers/net/ethernet/stmicro/stmmac - for CPU integrated LAN ports
We have a customer that have been eager to get hardware RX-timestamping for their AF_XDP use-case (PoC code[1] use software timestamping via bpf_ktime_get_ns() today). Getting driver support will qualify this hardware as part of their HW solution.
--Jesper
[1] https://github.com/xdp-project/bpf-examples/blob/master/AF_XDP-interaction/af_xdp_kern.c#L77
Quoted 29 lines · Jesper Dangaard Brouer
Quoted 26 lines · Toke Høiland-Jørgensen
Quoted 24 lines · Alexander Lobakin
From: Stanislav Fomichev <[email protected]>
Date: Mon, 23 Jan 2023 10:55:52 -0800Quoted 21 lines · Martin KaFai Lau
Quoted 20 lines · Stanislav Fomichev
Quoted 17 lines
Please see the first patch in the series for the overall design and use-cases.
See the following email from Toke for the per-packet metadata overhead:
https://lore.kernel.org/bpf/[email protected]/T/#m49d48ea08d525ec88360c7d14c4d34fb0e45e798Recent changes:
- Keep new functions inen/xdp.c, do 'externmlx5_xdp_metadata_ops' (Tariq)- Remove mxbuf pointer and use
xsk_buff_to_mxbuf(Tariq)- Clarify
xdp_buffvs 'XDP frame' (Jesper)- Explicitly mention that
AF_XDPRX descriptor lacks metadata size (Jesper)- Drop libbpf_flags/xdp_flags from selftests and use ifindex instead of ifname (due to recent
xsk.hrefactoring)
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Applied with the minor changes in the selftests discussed in patch 11 and 17. Thanks!
Awesome, thanks! I was gonna resend around Wed, but thank you for taking care of that!
Great stuff, congrats! :)
Yeah! Thanks for carrying this forward, Stanislav! :)
Thank you all as well for the valuable feedback and reviews!
Quoted 18 lines
+1000 -- great work everybody! :-)
To Alexander (Cc Jesse and Tony), do you think someone from Intel could look at extending drivers:
drivers/net/ethernet/intel/igb/- chip i210drivers/net/ethernet/intel/igc/- chip i225drivers/net/ethernet/stmicro/stmmac- for CPU integrated LAN portsWe have a customer that have been eager to get hardware RX-timestamping for their
AF_XDPuse-case (PoC code[1] use software timestamping viabpf_ktime_get_ns()today). Getting driver support will qualify this hardware as part of their HW solution.--Jesper
[1]
https://github.com/xdp-project/bpf-examples/blob/master/AF_XDP-interaction/af_xdp_kern.c#L77
From: Jesper Dangaard Brouer <[email protected]>
Date: Tue, 24 Jan 2023 13:23:49 +0100
Quoted 42 lines · Toke Høiland-Jørgensen
Quoted 34 lines
Quoted 32 lines · Alexander Lobakin
From: Stanislav Fomichev <[email protected]>
Date: Mon, 23 Jan 2023 10:55:52 -0800Quoted 29 lines
Quoted 26 lines · Martin KaFai Lau
Quoted 13 lines · Stanislav Fomichev
Please see the first patch in the series for the overall design and use-cases.
See the following email from Toke for the per-packet metadata overhead:
https://lore.kernel.org/bpf/[email protected]/T/#m49d48ea08d525ec88360c7d14c4d34fb0e45e798Recent changes:
- Keep new functions inen/xdp.c, do 'externmlx5_xdp_metadata_ops' (Tariq)- Remove mxbuf pointer and use
xsk_buff_to_mxbuf(Tariq)Quoted 7 lines
- Clarify
xdp_buffvs 'XDP frame' (Jesper)- Explicitly mention that
AF_XDPRX descriptor lacks metadata size (Jesper)- Drop libbpf_flags/xdp_flags from selftests and use ifindex instead of ifname (due to recent
xsk.hrefactoring)Applied with the minor changes in the selftests discussed in patch 11 and 17.
Thanks!Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Awesome, thanks! I was gonna resend around Wed, but thank you for taking care of that!
Great stuff, congrats! :)
Yeah! Thanks for carrying this forward, Stanislav! :)
Quoted 8 lines
+1000 -- great work everybody! :-)
To Alexander (Cc Jesse and Tony), do you think someone from Intel could look at extending drivers:
drivers/net/ethernet/intel/igb/ - chip i210
drivers/net/ethernet/intel/igc/- chip i225drivers/net/ethernet/stmicro/stmmac- for CPU integrated LAN ports
Sorry, just noticed :s
I work with ice only, but seems like the responsible teams started some work already. At least for i225. They may write some follow-ups soon.
Quoted 10 lines
We have a customer that have been eager to get hardware RX-timestamping for their
AF_XDPuse-case (PoC code[1] use software timestamping viabpf_ktime_get_ns()today). Getting driver support will qualify this hardware as part of their HW solution.--Jesper
[1]
https://github.com/xdp-project/bpf-examples/blob/master/AF_XDP-interaction/af_xdp_kern.c#L77
Thanks,
Olek
Quoted 54 lines
From: Jesper Dangaard Brouer <[email protected]>
Date: Tue, 24 Jan 2023 13:23:49 +0100Quoted 50 lines · Toke Høiland-Jørgensen
Quoted 37 lines
Quoted 35 lines · Alexander Lobakin
From: Stanislav Fomichev <[email protected]>
Date: Mon, 23 Jan 2023 10:55:52 -0800Quoted 32 lines
Quoted 29 lines · Martin KaFai Lau
Quoted 16 lines · Stanislav Fomichev
Please see the first patch in the series for the overall design and use-cases.
See the following email from Toke for the per-packet metadata overhead:
https://lore.kernel.org/bpf/[email protected]
om/T/#m49d48ea08d525ec88360c7d14c4d34fb0e45e798Recent changes:
- Keep new functions inen/xdp.c, do 'externmlx5_xdp_metadata_ops' (Tariq)- Remove mxbuf pointer and use
xsk_buff_to_mxbuf(Tariq)- Clarify
xdp_buffvs 'XDP frame' (Jesper)Quoted 7 lines
- Explicitly mention that
AF_XDPRX descriptor lacks metadata size
(Jesper)- Drop libbpf_flags/xdp_flags from selftests and use ifindex instead
of ifname (due to recentxsk.hrefactoring)Applied with the minor changes in the selftests discussed in patch 11 and 17.
Thanks!Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Awesome, thanks! I was gonna resend around Wed, but thank you for taking care of that!
Great stuff, congrats! :)
Yeah! Thanks for carrying this forward, Stanislav! :)
+1000 -- great work everybody! :-)
To Alexander (Cc Jesse and Tony), do you think someone from Intel could look at extending drivers:
Quoted 9 lines
Quoted 3 lines
drivers/net/ethernet/intel/igb/ - chip i210
drivers/net/ethernet/intel/igc/- chip i225drivers/net/ethernet/stmicro/stmmac- for CPU integrated LAN portsSorry, just noticed :s
I work with ice only, but seems like the responsible teams started some work already. At least for i225. They may write some follow-ups soon.
Hi Jesper,
FYI, me and my team will enable Rx metadata on stmmac and igc for Rx HWTS. Will submit the patches for review after done.
Quoted 7 lines
Quoted 7 lines
We have a customer that have been eager to get hardware RX-timestamping for their
AF_XDPuse-case (PoC code[1] use software timestamping viabpf_ktime_get_ns()today). Getting driver support will qualify this hardware as part of their HW solution.
Quoted 8 lines
Quoted 5 lines
--Jesper
[1]
https://github.com/xdp-project/bpf-examples/blob/master/AF_XDP-interac
tion/af_xdp_kern.c#L77Thanks,
Olek
Thanks & Regards
Siang