bpf: Netdev TX metadata
Subsystems (10)
v3 cover bpf: Netdev TX metadata
--- Changes since RFC v2 ---
- Separate skb & xdp path
- XSK is still weird in mlx5 driver; I have to create xdp_frame on the stack for af_xdp
- Add TX checksum kfunc
- In a separate patch for now to show extensibility
- Same API as we have in skb world
- mlx5 patches with tx timestamp/checksum support
- xdp_hw_metadata is extended with appropriate bits
- report sw completion timestamp as well
--- Use cases ---
The goal of this series is to add two new standard-ish places in the transmit path:
1. Right before the packet is transmitted (with access to TX descriptors)
2. Right after the packet is actually transmitted and we've received the completion (again, with access to TX completion descriptors)
Accessing TX descriptors unlocks the following use-cases:
- Setting device hints at TX: XDP/AF_XDP might use these new hooks to use device offloads. The existing case implements TX timestamp.
- Observability: global per-netdev hooks can be used for tracing the packets and exploring completion descriptors for all sorts of device errors.
Accessing TX descriptors also means that the hooks have to be called from the drivers.
The hooks are a light-weight alternative to XDP at egress and currently don't provide any packet modification abilities. However, eventually, can expose new kfuncs to operate on the packet (or, rather, the actual descriptors; for performance sake).
--- UAPI ---
The hooks are implemented in a HID-BPF style. Meaning they don't expose any UAPI and are implemented as tracing programs that call a bunch of kfuncs. The attach/detach operation happen via regular global fentry points. Network namespace and ifindex are exposed to allow filtering out particular netdev.
--- skb vs xdp ---
The hooks operate on a new light-weight devtx_ctx which contains:
- sinfo (frags)
- netdev
skb and xdp_frame hook points are separate; skb and xdp_frame are passed directly into the hook where appropriate.
--- TODO ---
Things that I'm planning to do for the non-RFC series:
- have some real device support to verify xdp_hw_metadata works
- performance numbers with/without feature enabled (Toke)
- add has_timestamp flag to af_xdp tx_desc (Toke & Jesper)
- freplace
- explore dynptr (Toke)
- Documentation/networking/xdp-rx-metadata.rst - like documentation
Stanislav Fomichev (14):
bpf: Rename some xdp-metadata functions into dev-bound
bpf: Make it easier to add new metadata kfunc
xsk: Support XDP_TX_METADATA_LEN
bpf: Implement devtx hook points
bpf: Implement devtx timestamp kfunc
net: veth: Implement devtx timestamp kfuncs
bpf: Introduce tx checksum devtx kfuncs
net: veth: Implement devtx tx checksum net/mlx5e: Implement devtx kfuncs selftests/xsk: Support XDP_TX_METADATA_LEN selftests/bpf: Add helper to query current netns cookie selftests/bpf: Add csum helpers
selftests/bpf: Extend xdp_metadata with devtx kfuncs selftests/bpf: Extend xdp_hw_metadata with devtx kfuncs
MAINTAINERS | 2 + .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 15 + .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 155 +++++++++- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 4 +- .../ethernet/mellanox/mlx5/core/en/xsk/tx.c | 10 + .../net/ethernet/mellanox/mlx5/core/en_tx.c | 24 ++ .../net/ethernet/mellanox/mlx5/core/main.c | 15 +- drivers/net/veth.c | 126 +++++++- include/linux/netdevice.h | 6 + include/net/devtx.h | 66 +++++ include/net/offload.h | 45 +++ include/net/xdp.h | 18 +- include/net/xdp_sock.h | 1 + include/net/xsk_buff_pool.h | 1 + include/uapi/linux/if_xdp.h | 1 + kernel/bpf/offload.c | 52 +++- kernel/bpf/verifier.c | 4 +- net/core/Makefile | 1 + net/core/dev.c | 1 + net/core/devtx.c | 168 +++++++++++ net/core/xdp.c | 20 +- net/xdp/xsk.c | 35 ++- net/xdp/xsk_buff_pool.c | 1 + net/xdp/xsk_queue.h | 7 +- tools/testing/selftests/bpf/network_helpers.c | 21 ++ tools/testing/selftests/bpf/network_helpers.h | 44 +++ .../selftests/bpf/prog_tests/xdp_metadata.c | 64 ++++- .../selftests/bpf/progs/xdp_hw_metadata.c | 173 +++++++++++ .../selftests/bpf/progs/xdp_metadata.c | 141 +++++++++ tools/testing/selftests/bpf/xdp_hw_metadata.c | 269 +++++++++++++++++- tools/testing/selftests/bpf/xdp_metadata.h | 16 ++ tools/testing/selftests/bpf/xsk.c | 17 ++ tools/testing/selftests/bpf/xsk.h | 1 + 33 files changed, 1451 insertions(+), 73 deletions(-) create mode 100644 include/net/devtx.h create mode 100644 include/net/offload.h create mode 100644 net/core/devtx.c
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 1/14 bpf: Rename some xdp-metadata functions into dev-bound
No functional changes.
To make existing dev-bound infrastructure more generic and be less tightly bound to the xdp layer, rename some functions and move kfunc-related things into kernel/bpf/offload.c
--- include/net/offload.h | 28 ++++++++++++++++++++++++++++ include/net/xdp.h | 18 +----------------- kernel/bpf/offload.c | 26 ++++++++++++++++++++++++-- kernel/bpf/verifier.c | 4 ++-- net/core/xdp.c | 20 ++------------------ 5 files changed, 57 insertions(+), 39 deletions(-) create mode 100644 include/net/offload.h
include/net/offload.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __LINUX_NET_OFFLOAD_H__ +#define __LINUX_NET_OFFLOAD_H__ + +#include <linux/types.h> + +#define XDP_METADATA_KFUNC_xxx \ + NETDEV_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP, \ + bpf_xdp_metadata_rx_timestamp) \ + NETDEV_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ + bpf_xdp_metadata_rx_hash) + +enum { +#define NETDEV_METADATA_KFUNC(name, _) name, +XDP_METADATA_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC +MAX_NETDEV_METADATA_KFUNC , +}; + +#ifdef CONFIG_NET +u32 bpf_dev_bound_kfunc_id (int id); +bool bpf_is_dev_bound_kfunc (u32 btf_id); +#else +static inline u32 bpf_dev_bound_kfunc_id (int id) {return 0 ; } +static inline bool bpf_is_dev_bound_kfunc (u32 btf_id) {return false; } +#endif + +#endif /* __LINUX_NET_OFFLOAD_H__ */
include/net/xdp.h +1 -17
@@ -9,6 +9,7 @@#include <linux/skbuff.h> /* skb_shared_info */ #include <uapi/linux/netdev.h> #include <linux/bitfield.h> +#include <net/offload.h> /** * DOC: XDP RX-queue information
@@ -384,19 +385,6 @@ void xdp_attachment_setup(struct xdp_attachment_info *info,#define DEV_MAP_BULK_SIZE XDP_BULK_QUEUE_SIZE -#define XDP_METADATA_KFUNC_xxx \ - XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP, \ - bpf_xdp_metadata_rx_timestamp) \ - XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ - bpf_xdp_metadata_rx_hash) \ - -enum { -#define XDP_METADATA_KFUNC(name, _) name, -XDP_METADATA_KFUNC_xxx -#undef XDP_METADATA_KFUNC -MAX_XDP_METADATA_KFUNC , -}; -enum xdp_rss_hash_type {/* First part: Individual bits for L3/L4 types */ XDP_RSS_L3_IPV4 =BIT (0 ),
@@ -444,14 +432,10 @@ enum xdp_rss_hash_type { };#ifdef CONFIG_NET -u32 bpf_xdp_metadata_kfunc_id (int id); -bool bpf_dev_bound_kfunc_id (u32 btf_id);void xdp_set_features_flag (struct net_device *dev,xdp_features_t val);void xdp_features_set_redirect_target (struct net_device *dev,bool support_sg);void xdp_features_clear_redirect_target (struct net_device *dev);#else -static inline u32 bpf_xdp_metadata_kfunc_id (int id) {return 0 ; } -static inline bool bpf_dev_bound_kfunc_id (u32 btf_id) {return false; }static inline void xdp_set_features_flag (struct net_device *dev,xdp_features_t val)
kernel/bpf/offload.c +24 -2
@@ -844,9 +844,9 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id)if (!ops) goto out; -if (func_id ==bpf_xdp_metadata_kfunc_id (XDP_METADATA_KFUNC_RX_TIMESTAMP )) +if (func_id ==bpf_dev_bound_kfunc_id (XDP_METADATA_KFUNC_RX_TIMESTAMP )) p = ops->xmo_rx_timestamp; -else if (func_id ==bpf_xdp_metadata_kfunc_id (XDP_METADATA_KFUNC_RX_HASH )) +else if (func_id ==bpf_dev_bound_kfunc_id (XDP_METADATA_KFUNC_RX_HASH )) p = ops->xmo_rx_hash; out:up_read (&bpf_devs_lock);
@@ -854,6 +854,28 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id)return p; } +BTF_SET_START (dev_bound_kfunc_ids) +#define NETDEV_METADATA_KFUNC(name, str) BTF_ID(func, str) +XDP_METADATA_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC +BTF_SET_END (dev_bound_kfunc_ids) + +BTF_ID_LIST (dev_bound_kfunc_ids_unsorted) +#define NETDEV_METADATA_KFUNC(name, str) BTF_ID(func, str) +XDP_METADATA_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC + +u32 bpf_dev_bound_kfunc_id(int id) +{ +/* dev_bound_kfunc_ids is sorted and can't be used */ +return dev_bound_kfunc_ids_unsorted[id]; +} + +bool bpf_is_dev_bound_kfunc (u32 btf_id) +{ +return btf_id_set_contains (&dev_bound_kfunc_ids, btf_id); +} +static int __init bpf_offload_init (void ) {return rhashtable_init (&offdevs, &offdevs_params);
kernel/bpf/verifier.c +2 -2
@@ -2721,7 +2721,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) } } -if (bpf_dev_bound_kfunc_id (func_id)) { +if (bpf_is_dev_bound_kfunc (func_id)) { err =bpf_dev_bound_kfunc_check (&env->log, prog_aux);if (err)return err;
@@ -17923,7 +17923,7 @@ static void specialize_kfunc(struct bpf_verifier_env *env,void *xdp_kfunc;bool is_rdonly; -if (bpf_dev_bound_kfunc_id (func_id)) { +if (bpf_is_dev_bound_kfunc (func_id)) { xdp_kfunc =bpf_dev_bound_resolve_kfunc (prog, func_id);if (xdp_kfunc) { *addr = (unsigned long )xdp_kfunc;
net/core/xdp.c +2 -18
@@ -741,9 +741,9 @@ __bpf_kfunc int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, u32 *hash,__diag_pop ();BTF_SET8_START (xdp_metadata_kfunc_ids) -#define XDP_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, 0) +#define NETDEV_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, 0) XDP_METADATA_KFUNC_xxx -#undef XDP_METADATA_KFUNC +#undef NETDEV_METADATA_KFUNCBTF_SET8_END (xdp_metadata_kfunc_ids) static const struct btf_kfunc_id_set xdp_metadata_kfunc_set = {
@@ -751,22 +751,6 @@ static const struct btf_kfunc_id_set xdp_metadata_kfunc_set = { .set = &xdp_metadata_kfunc_ids, }; -BTF_ID_LIST (xdp_metadata_kfunc_ids_unsorted) -#define XDP_METADATA_KFUNC(name, str) BTF_ID(func, str) -XDP_METADATA_KFUNC_xxx -#undef XDP_METADATA_KFUNC - -u32 bpf_xdp_metadata_kfunc_id(int id) -{ -/* xdp_metadata_kfunc_ids is sorted and can't be used */ -return xdp_metadata_kfunc_ids_unsorted[id]; -} - -bool bpf_dev_bound_kfunc_id (u32 btf_id) -{ -return btf_id_set8_contains (&xdp_metadata_kfunc_ids, btf_id); -} -static int __init xdp_metadata_init (void ) {return register_btf_kfunc_id_set (BPF_PROG_TYPE_XDP , &xdp_metadata_kfunc_set);
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 2/14 bpf: Make it easier to add new metadata kfunc
In reply to Stanislav FomichevNo functional changes.
Instead of having hand-crafted code in bpf_dev_bound_resolve_kfunc, move kfunc <> xmo handler relationship into XDP_METADATA_KFUNC_xxx. This way, any time new kfunc is added, we don't have to touch bpf_dev_bound_resolve_kfunc.
--- include/net/offload.h | 8 +++++--- kernel/bpf/offload.c | 13 +++++++------ net/core/xdp.c | 2 +- 3 files changed, 13 insertions(+), 10 deletions(-)
include/net/offload.h +5 -3
@@ -6,12 +6,14 @@#define XDP_METADATA_KFUNC_xxx \ NETDEV_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP, \ - bpf_xdp_metadata_rx_timestamp) \ + bpf_xdp_metadata_rx_timestamp, \ + xmo_rx_timestamp) \ NETDEV_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ - bpf_xdp_metadata_rx_hash) + bpf_xdp_metadata_rx_hash, \ + xmo_rx_hash)enum { -#define NETDEV_METADATA_KFUNC(name, _) name, +#define NETDEV_METADATA_KFUNC(name, _, __) name, XDP_METADATA_KFUNC_xxx #undef NETDEV_METADATA_KFUNCMAX_NETDEV_METADATA_KFUNC ,
kernel/bpf/offload.c +7 -6
@@ -844,10 +844,11 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id)if (!ops) goto out; -if (func_id ==bpf_dev_bound_kfunc_id (XDP_METADATA_KFUNC_RX_TIMESTAMP )) - p = ops->xmo_rx_timestamp; -else if (func_id ==bpf_dev_bound_kfunc_id (XDP_METADATA_KFUNC_RX_HASH )) - p = ops->xmo_rx_hash; +#define NETDEV_METADATA_KFUNC (name, _, xmo) \ + if (func_id == bpf_dev_bound_kfunc_id(name)) p = ops->xmo; + XDP_METADATA_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC + out:up_read (&bpf_devs_lock);
@@ -855,13 +856,13 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id) } BTF_SET_START(dev_bound_kfunc_ids) -#define NETDEV_METADATA_KFUNC(name, str) BTF_ID(func, str) +#define NETDEV_METADATA_KFUNC(name, str, _) BTF_ID(func, str) XDP_METADATA_KFUNC_xxx#undef NETDEV_METADATA_KFUNC BTF_SET_END(dev_bound_kfunc_ids) BTF_ID_LIST(dev_bound_kfunc_ids_unsorted) -#define NETDEV_METADATA_KFUNC (name, str) BTF_ID(func, str) +#define NETDEV_METADATA_KFUNC (name, str, _) BTF_ID(func, str)XDP_METADATA_KFUNC_xxx #undef NETDEV_METADATA_KFUNC
net/core/xdp.c +1 -1
@@ -741,7 +741,7 @@ __bpf_kfunc int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, u32 *hash,__diag_pop (); BTF_SET8_START(xdp_metadata_kfunc_ids) -#define NETDEV_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, 0) +#define NETDEV_METADATA_KFUNC(_, name, __) BTF_ID_FLAGS(func, name, 0) XDP_METADATA_KFUNC_xxx #undef NETDEV_METADATA_KFUNC BTF_SET8_END(xdp_metadata_kfunc_ids)
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 3/14 xsk: Support XDP_TX_METADATA_LEN
In reply to Stanislav FomichevFor zerocopy mode, tx_desc->addr can point to the arbitrary offset and carry some TX metadata in the headroom. For copy mode, there is no way currently to populate skb metadata.
Introduce new XDP_TX_METADATA_LEN that indicates how many bytes to treat as metadata. Metadata bytes come prior to tx_desc address (same as in RX case).
--- include/net/xdp_sock.h | 1 + include/net/xsk_buff_pool.h | 1 + include/uapi/linux/if_xdp.h | 1 + net/xdp/xsk.c | 35 +++++++++++++++++++++++++++++++++-- net/xdp/xsk_buff_pool.c | 1 + net/xdp/xsk_queue.h | 7 ++++--- 6 files changed, 41 insertions(+), 5 deletions(-)
include/net/xdp_sock.h +1 -0
@@ -51,6 +51,7 @@ struct xdp_sock {struct list_head flush_node;struct xsk_buff_pool *pool;u16 queue_id; +u8 tx_metadata_len;bool zc;enum {XSK_READY =0 ,
include/net/xsk_buff_pool.h +1 -0
@@ -75,6 +75,7 @@ struct xsk_buff_pool {u32 chunk_size;u32 chunk_shift;u32 frame_len; +u8 tx_metadata_len;/* inherited from xsk_sock */ u8 cached_need_wakeup;bool uses_need_wakeup;bool dma_need_sync;
include/uapi/linux/if_xdp.h +1 -0
@@ -63,6 +63,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_TX_METADATA_LEN 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */
net/xdp/xsk.c +33 -2
@@ -485,6 +485,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,int err; hr =max (NET_SKB_PAD ,L1_CACHE_ALIGN (dev->needed_headroom)); + hr =max (hr,L1_CACHE_ALIGN ((u32 )xs->tx_metadata_len)); tr = dev->needed_tailroom; len = desc->len;
@@ -493,14 +494,21 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,return ERR_PTR (err);skb_reserve (skb, hr); -skb_put (skb, len); +skb_put (skb, len + xs->tx_metadata_len); buffer =xsk_buff_raw_get_data (xs->pool, desc->addr); - err =skb_store_bits (skb,0 , buffer, len); + buffer -= xs->tx_metadata_len; + + err =skb_store_bits (skb,0 , buffer, len + xs->tx_metadata_len);if (unlikely (err)) {kfree_skb (skb);return ERR_PTR (err); } + +if (xs->tx_metadata_len) { +skb_metadata_set (skb, xs->tx_metadata_len); +__skb_pull (skb, xs->tx_metadata_len); + } } skb->dev = dev;
@@ -1137,6 +1145,29 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,mutex_unlock (&xs->mutex);return err; } +case XDP_TX_METADATA_LEN : + { +int val; + +if (optlen <sizeof (val)) +return -EINVAL ; +if (copy_from_sockptr (&val, optval,sizeof (val))) +return -EFAULT ; + +if (val >=256 ) +return -EINVAL ; +if (val %4 ) +return -EINVAL ; + +mutex_lock (&xs->mutex); +if (xs->state !=XSK_READY ) { +mutex_unlock (&xs->mutex); +return -EBUSY ; + } + xs->tx_metadata_len = val; +mutex_unlock (&xs->mutex); +return 0 ; + }default :break ; }
net/xdp/xsk_buff_pool.c +1 -0
@@ -85,6 +85,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,XDP_PACKET_HEADROOM ; pool->umem = umem; pool->addrs = umem->addrs; + pool->tx_metadata_len = xs->tx_metadata_len;INIT_LIST_HEAD (&pool->free_list);INIT_LIST_HEAD (&pool->xsk_tx_list);spin_lock_init (&pool->xsk_tx_list_lock);
net/xdp/xsk_queue.h +4 -3
@@ -133,12 +133,13 @@ static inline bool xskq_cons_read_addr_unchecked(struct xsk_queue *q, u64 *addr)static inline bool xp_aligned_validate_desc (struct xsk_buff_pool *pool,struct xdp_desc *desc) { -u64 offset = desc->addr & (pool->chunk_size -1 ); +u64 addr = desc->addr - pool->tx_metadata_len; +u64 offset = addr & (pool->chunk_size -1 );if (offset + desc->len > pool->chunk_size)return false; -if (desc->addr >= pool->addrs_cnt) +if (addr >= pool->addrs_cnt)return false;if (desc->options)
@@ -149,7 +150,7 @@ static inline bool xp_aligned_validate_desc(struct xsk_buff_pool *pool,static inline bool xp_unaligned_validate_desc (struct xsk_buff_pool *pool,struct xdp_desc *desc) { -u64 addr =xp_unaligned_add_offset_to_addr (desc->addr); +u64 addr =xp_unaligned_add_offset_to_addr (desc->addr) - pool->tx_metadata_len;if (desc->len > pool->chunk_size)return false;
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 4/14 bpf: Implement devtx hook points
In reply to Stanislav Fomichevdevtx is a lightweight set of hooks before and after packet transmission. The hooks are implemented as a tracing program which has access to the XDP-metadata-like kfuncs. The initial set of kfuncs is implemented in the next patch, but the idea is similar to XDP metadata:
the kfuncs have netdev-specific implementation, but common interface. Upon loading, the kfuncs are resolved to direct calls against per-netdev implementation. This can be achieved by marking devtx-tracing programs as dev-bound (largely reusing xdp-dev-bound program infrastructure).
--- MAINTAINERS | 2 ++ include/net/devtx.h | 66 +++++++++++++++++++++++++++++++++++++ kernel/bpf/offload.c | 15 +++++++++ net/core/Makefile | 1 + net/core/dev.c | 1 + net/core/devtx.c | 78 ++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 163 insertions(+) create mode 100644 include/net/devtx.h create mode 100644 net/core/devtx.c
MAINTAINERS +2 -0
@@ -23063,11 +23063,13 @@ L: [email protected] S: Supported F: drivers/net/ethernet/*/*/*/*/*xdp* F: drivers/net/ethernet/*/*/*xdp* +F: include/net/devtx.h F: include/net/xdp.h F: include/net/xdp_priv.h F: include/trace/events/xdp.h F: kernel/bpf/cpumap.c F: kernel/bpf/devmap.c +F: net/core/devtx.c F: net/core/xdp.c F: samples/bpf/xdp* F: tools/testing/selftests/bpf/*/*xdp*
include/net/devtx.h new file +66 -0
66 lines not on this page · raw message
kernel/bpf/offload.c +15 -0
@@ -25,6 +25,7 @@#include <linux/rhashtable.h> #include <linux/rtnetlink.h> #include <linux/rwsem.h> +#include <net/devtx.h> /* Protects offdevs, members of bpf_offload_netdev and offload members * of all progs.
@@ -228,6 +229,7 @@ int bpf_prog_dev_bound_init(struct bpf_prog *prog, union bpf_attr *attr)int err;if (attr->prog_type !=BPF_PROG_TYPE_SCHED_CLS && + attr->prog_type !=BPF_PROG_TYPE_TRACING && attr->prog_type !=BPF_PROG_TYPE_XDP )return -EINVAL ;
@@ -242,6 +244,15 @@ int bpf_prog_dev_bound_init(struct bpf_prog *prog, union bpf_attr *attr)if (!netdev)return -EINVAL ; +/* Make sure device-bound tracing programs are being attached +* to the appropriate netdev. +*/ +if (attr->prog_type ==BPF_PROG_TYPE_TRACING && + !devtx_hooks_match (prog->aux->attach_btf_id, netdev->xdp_metadata_ops)) { + err = -EINVAL ; + goto out; + } + err =bpf_dev_offload_check (netdev);if (err) goto out;
@@ -252,6 +263,9 @@ int bpf_prog_dev_bound_init(struct bpf_prog *prog, union bpf_attr *attr) err =__bpf_prog_dev_bound_init (prog, netdev);up_write (&bpf_devs_lock); +if (!err) +devtx_hooks_enable (); + out:dev_put (netdev);return err;
@@ -384,6 +398,7 @@ void bpf_prog_dev_bound_destroy(struct bpf_prog *prog) ondev =bpf_offload_find_netdev (netdev);if (!ondev->offdev &&list_empty (&ondev->progs))__bpf_offload_dev_netdev_unregister (NULL , netdev); +devtx_hooks_disable (); }up_write (&bpf_devs_lock);rtnl_unlock ();
net/core/Makefile +1 -0
@@ -39,4 +39,5 @@ obj-$(CONFIG_FAILOVER) += failover.o obj-$(CONFIG_NET_SOCK_MSG) += skmsg.o obj-$(CONFIG_BPF_SYSCALL) += sock_map.o obj-$(CONFIG_BPF_SYSCALL) += bpf_sk_storage.o +obj-$(CONFIG_BPF_SYSCALL) += devtx.o obj-$(CONFIG_OF) += of_net.o
net/core/dev.c +1 -0
@@ -150,6 +150,7 @@#include <linux/pm_runtime.h> #include <linux/prandom.h> #include <linux/once_lite.h> +#include <net/devtx.h> #include "dev.h" #include "net-sysfs.h"
net/core/devtx.c new file +78 -0
78 lines not on this page · raw message
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 5/14 bpf: Implement devtx timestamp kfunc
In reply to Stanislav FomichevTwo kfuncs, one per hook point:
1. at submit time - bpf_devtx_sb_request_timestamp - to request HW to put TX timestamp into TX completion descriptors
2. at completion time - bpf_devtx_cp_timestamp - to read out TX timestamp
--- include/linux/netdevice.h | 4 +++ include/net/offload.h | 12 +++++++ kernel/bpf/offload.c | 6 ++++ net/core/devtx.c | 73 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 95 insertions(+)
include/linux/netdevice.h +4 -0
@@ -1654,10 +1654,14 @@ struct net_device_ops { bool cycles); }; +struct devtx_ctx ; +struct xdp_metadata_ops {int (*xmo_rx_timestamp)(const struct xdp_md *ctx,u64 *timestamp);int (*xmo_rx_hash)(const struct xdp_md *ctx,u32 *hash,enum xdp_rss_hash_type *rss_type); +int (*xmo_request_tx_timestamp)(const struct devtx_ctx *ctx); +int (*xmo_tx_timestamp)(const struct devtx_ctx *ctx,u64 *timestamp); };/**
include/net/offload.h +12 -0
@@ -12,9 +12,21 @@ bpf_xdp_metadata_rx_hash, \ xmo_rx_hash) +#define DEVTX_SUBMIT_KFUNC_xxx \ + NETDEV_METADATA_KFUNC(DEVTX_KFUNC_REQUEST_TX_TIMESTAMP, \ + bpf_devtx_request_tx_timestamp, \ + xmo_request_tx_timestamp) + +#define DEVTX_COMPLETE_KFUNC_xxx \ + NETDEV_METADATA_KFUNC(DEVTX_KFUNC_TX_TIMESTAMP, \ + bpf_devtx_tx_timestamp, \ + xmo_tx_timestamp) +enum { #define NETDEV_METADATA_KFUNC(name, _, __) name, XDP_METADATA_KFUNC_xxx +DEVTX_SUBMIT_KFUNC_xxx +DEVTX_COMPLETE_KFUNC_xxx #undef NETDEV_METADATA_KFUNCMAX_NETDEV_METADATA_KFUNC , };
kernel/bpf/offload.c +6 -0
@@ -862,6 +862,8 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id)#define NETDEV_METADATA_KFUNC (name, _, xmo) \ if (func_id == bpf_dev_bound_kfunc_id(name)) p = ops->xmo; XDP_METADATA_KFUNC_xxx + DEVTX_SUBMIT_KFUNC_xxx + DEVTX_COMPLETE_KFUNC_xxx#undef NETDEV_METADATA_KFUNC out:
@@ -873,12 +875,16 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id) BTF_SET_START(dev_bound_kfunc_ids) #define NETDEV_METADATA_KFUNC(name, str, _) BTF_ID(func, str) XDP_METADATA_KFUNC_xxx +DEVTX_SUBMIT_KFUNC_xxx +DEVTX_COMPLETE_KFUNC_xxx #undef NETDEV_METADATA_KFUNC BTF_SET_END(dev_bound_kfunc_ids) BTF_ID_LIST(dev_bound_kfunc_ids_unsorted) #define NETDEV_METADATA_KFUNC(name, str, _) BTF_ID(func, str) XDP_METADATA_KFUNC_xxx +DEVTX_SUBMIT_KFUNC_xxx +DEVTX_COMPLETE_KFUNC_xxx #undef NETDEV_METADATA_KFUNC u32 bpf_dev_bound_kfunc_id(int id)
net/core/devtx.c +73 -0
76 lines not on this page · raw message
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 6/14 net: veth: Implement devtx timestamp kfuncs
In reply to Stanislav FomichevHave a software-based example for kfuncs to showcase how it can be used in the real devices and to have something to test against in the selftests.
Both path (skb & xdp) are covered. Only the skb path is really tested though.
--- drivers/net/veth.c | 97 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 95 insertions(+), 2 deletions(-)
drivers/net/veth.c +95 -2
@@ -27,6 +27,7 @@#include <linux/bpf_trace.h> #include <linux/net_tstamp.h> #include <net/page_pool.h> +#include <net/devtx.h> #define DRV_NAME "veth"#define DRV_VERSION "1.0"
@@ -123,6 +124,13 @@ struct veth_xdp_buff {struct sk_buff *skb; }; +struct veth_devtx_ctx { +struct devtx_ctx devtx; +struct xdp_frame *xdpf; +struct sk_buff *skb; +ktime_t xdp_tx_timestamp; +}; +static int veth_get_link_ksettings (struct net_device *dev,struct ethtool_link_ksettings *cmd) {
@@ -313,10 +321,33 @@ static int veth_xdp_rx(struct veth_rq *rq, struct sk_buff *skb)return NET_RX_SUCCESS ; } +DEFINE_DEVTX_HOOKS (veth); +static int veth_forward_skb (struct net_device *dev,struct sk_buff *skb,struct veth_rq *rq,bool xdp) { -return __dev_forward_skb (dev, skb) ?: xdp ? +struct net_device *orig_dev = skb->dev; +int ret; + + ret =__dev_forward_skb (dev, skb); +if (ret) +return ret; + +if (devtx_enabled ()) { +struct veth_devtx_ctx ctx = { + .devtx = { + .netdev = orig_dev, + .sinfo =skb_shinfo (skb), + }, + .skb = skb, + }; + +__skb_push (skb,ETH_HLEN ); +veth_devtx_complete_skb (&ctx.devtx, skb); +__skb_pull (skb,ETH_HLEN ); + } + +return xdp ?veth_xdp_rx (rq, skb) :__netif_rx (skb); }
@@ -356,6 +387,18 @@ static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev) goto drop; } +if (devtx_enabled ()) { +struct veth_devtx_ctx ctx = { + .devtx = { + .netdev = skb->dev, + .sinfo =skb_shinfo (skb), + }, + .skb = skb, + }; + +veth_devtx_submit_skb (&ctx.devtx, skb); + } + rcv_priv =netdev_priv (rcv); rxq =skb_get_queue_mapping (skb);if (rxq < rcv->real_num_rx_queues) {
@@ -509,11 +552,28 @@ static int veth_xdp_xmit(struct net_device *dev, int n,for (i =0 ; i < n; i++) {struct xdp_frame *frame = frames[i];void *ptr =veth_xdp_to_ptr (frame); +struct veth_devtx_ctx ctx;if (unlikely (xdp_get_frame_len (frame) > max_len || -__ptr_ring_produce (&rq->xdp_ring, ptr))) +__ptr_ring_full (&rq->xdp_ring))) +break ; + +if (devtx_enabled ()) { +memset (&ctx,0 ,sizeof (ctx)); + ctx.devtx.netdev = dev; + ctx.devtx.sinfo =xdp_frame_has_frags (frame) ? +xdp_get_shared_info_from_frame (frame) :NULL ; + ctx.xdpf = frame; + +veth_devtx_submit_xdp (&ctx.devtx, frame); + } + +if (unlikely (__ptr_ring_produce (&rq->xdp_ring, ptr)))break ; nxmit++; + +if (devtx_enabled ()) +veth_devtx_complete_xdp (&ctx.devtx, frame); }spin_unlock (&rq->xdp_ring.producer_lock);
@@ -1732,6 +1792,28 @@ static int veth_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash,return 0 ; } +static int veth_devtx_request_tx_timestamp (const struct devtx_ctx *_ctx) +{ +struct veth_devtx_ctx *ctx = (struct veth_devtx_ctx *)_ctx; + +if (ctx->skb) +__net_timestamp (ctx->skb); +else + ctx->xdp_tx_timestamp =ktime_get_real (); + +return 0 ; +} + +static int veth_devtx_tx_timestamp (const struct devtx_ctx *_ctx,u64 *timestamp) +{ +struct veth_devtx_ctx *ctx = (struct veth_devtx_ctx *)_ctx; + +if (ctx->skb) + *timestamp = ctx->skb->tstamp; + +return 0 ; +} +static const struct net_device_ops veth_netdev_ops = { .ndo_init = veth_dev_init, .ndo_open = veth_open,
@@ -1756,6 +1838,8 @@ static const struct net_device_ops veth_netdev_ops = {static const struct xdp_metadata_ops veth_xdp_metadata_ops = { .xmo_rx_timestamp = veth_xdp_rx_timestamp, .xmo_rx_hash = veth_xdp_rx_hash, + .xmo_request_tx_timestamp = veth_devtx_request_tx_timestamp, + .xmo_tx_timestamp = veth_devtx_tx_timestamp, };#define VETH_FEATURES (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_HW_CSUM | \
@@ -2041,11 +2125,20 @@ static struct rtnl_link_ops veth_link_ops = {static __init int veth_init (void ) { +int ret; + + ret =devtx_hooks_register (&veth_devtx_hook_ids, &veth_xdp_metadata_ops); +if (ret) { +pr_warn ("failed to register devtx hooks: %d" , ret); +return ret; + } +return rtnl_link_register (&veth_link_ops); } static __exit void veth_exit(void) { +devtx_hooks_unregister (&veth_devtx_hook_ids);rtnl_link_unregister (&veth_link_ops); }
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 7/14 bpf: Introduce tx checksum devtx kfuncs
In reply to Stanislav FomichevAdd new kfunc that will be used for tx checksum offloading. The API mirrors existing one from sk_buff:
- csum_start - checksum the packet starting from this position
- csum_offset - put checksum at this offset
--- include/linux/netdevice.h | 2 ++ include/net/offload.h | 5 ++++- net/core/devtx.c | 17 +++++++++++++++++ 3 files changed, 23 insertions(+), 1 deletion(-)
include/linux/netdevice.h +2 -0
@@ -1662,6 +1662,8 @@ struct xdp_metadata_ops { enum xdp_rss_hash_type *rss_type);int (*xmo_request_tx_timestamp)(const struct devtx_ctx *ctx);int (*xmo_tx_timestamp)(const struct devtx_ctx *ctx,u64 *timestamp); +int (*xmo_request_l4_checksum)(const struct devtx_ctx *ctx, +u16 csum_start,u16 csum_offset); };/**
include/net/offload.h +4 -1
@@ -15,7 +15,10 @@#define DEVTX_SUBMIT_KFUNC_xxx \ NETDEV_METADATA_KFUNC(DEVTX_KFUNC_REQUEST_TX_TIMESTAMP, \ bpf_devtx_request_tx_timestamp, \ - xmo_request_tx_timestamp) + xmo_request_tx_timestamp) \ + NETDEV_METADATA_KFUNC(DEVTX_KFUNC_REQUEST_L4_CHECKSUM, \ + bpf_devtx_request_l4_csum, \ + xmo_request_l4_checksum)#define DEVTX_COMPLETE_KFUNC_xxx \ NETDEV_METADATA_KFUNC(DEVTX_KFUNC_TX_TIMESTAMP, \
net/core/devtx.c +17 -0
@@ -106,6 +106,23 @@ __bpf_kfunc int bpf_devtx_tx_timestamp(const struct devtx_ctx *ctx, __u64 *timesreturn -EOPNOTSUPP ; } +/** +* bpf_devtx_request_l4_csum - Request TX checksum offload on the packet. +* Callable only from the devtx-submit hook. +* @ctx: devtx context pointer. +* @csum_start: start checksumming from given position +* @csum_offset: add resulting checksum at given offset +* +* Note, this checksum offload doesn't calculate pseudo-header part. +* +* Returns 0 on success or ``-errno`` on error. +*/ +__bpf_kfunc int bpf_devtx_request_l4_csum(const struct devtx_ctx *ctx, +u16 csum_start,u16 csum_offset) +{ +return -EOPNOTSUPP ; +} +__diag_pop ();BTF_SET8_START (devtx_sb_kfunc_ids)
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 8/14 net: veth: Implement devtx tx checksum
In reply to Stanislav FomichevImplement tx checksum kfunc for veth by checksumming the packet in software (since there is nothing to offload). The change mostly exists to make it possible to have software-based selftest.
Probably should instead set csum_start/csum_offset on the skb itself?
--- drivers/net/veth.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
drivers/net/veth.c +29 -0
@@ -1814,6 +1814,34 @@ static int veth_devtx_tx_timestamp(const struct devtx_ctx *_ctx, u64 *timestamp)return 0 ; } +static int veth_devtx_request_l4_csum (const struct devtx_ctx *_ctx, +u16 csum_start,u16 csum_offset) +{ +struct veth_devtx_ctx *ctx = (struct veth_devtx_ctx *)_ctx; +struct sk_buff *skb = ctx->skb; +__wsum csum; +int ret; + +if (!skb) +return -EINVAL ; + +if (skb_transport_header_was_set (skb)) +return -EINVAL ; + +if (csum_start >= skb->len) +return -EINVAL ; + + ret =skb_ensure_writable (skb, csum_offset +sizeof (__sum16)); +if (ret) +return ret; + + csum =csum_partial (skb->data + csum_start, skb->len - csum_start,0 ); + *(__sum16 *)(skb->data + csum_offset) =csum_fold (csum) ?:CSUM_MANGLED_0 ; + skb->ip_summed =CHECKSUM_UNNECESSARY ; + +return 0 ; +} +static const struct net_device_ops veth_netdev_ops = { .ndo_init = veth_dev_init, .ndo_open = veth_open,
@@ -1840,6 +1868,7 @@ static const struct xdp_metadata_ops veth_xdp_metadata_ops = { .xmo_rx_hash = veth_xdp_rx_hash, .xmo_request_tx_timestamp = veth_devtx_request_tx_timestamp, .xmo_tx_timestamp = veth_devtx_tx_timestamp, + .xmo_request_l4_checksum = veth_devtx_request_l4_csum, };#define VETH_FEATURES (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_HW_CSUM | \
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 9/14 net/mlx5e: Implement devtx kfuncs
In reply to Stanislav FomichevTX timestamp:
- requires passing clock, not sure I'm passing the correct one (from cq->mdev), but the timestamp value looks convincing
TX checksum:
- looks like device does packet parsing (and doesn't accept custom start/offset), so I'm using the same approach we do for skb (support specific header offesets)
Some quirks that might need some more thought:
- AF_XDP requires creating on-stack xdp_frame
- AF_XDP passes only head to the completion hook
--- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 15 ++ .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 155 +++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 4 +- .../ethernet/mellanox/mlx5/core/en/xsk/tx.c | 10 ++ .../net/ethernet/mellanox/mlx5/core/en_tx.c | 24 +++ .../net/ethernet/mellanox/mlx5/core/main.c | 15 +- 6 files changed, 217 insertions(+), 6 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +15 -0
@@ -6,6 +6,7 @@#include "en.h" #include <linux/indirect_call_wrapper.h> +#include <net/devtx.h> #define MLX5E_TX_WQE_EMPTY_DS_COUNT (sizeof(struct mlx5e_tx_wqe) / MLX5_SEND_WQE_DS)
@@ -506,4 +507,18 @@ static inline struct mlx5e_mpw_info *mlx5e_get_mpw_info(struct mlx5e_rq *rq, int return (struct mlx5e_mpw_info *)((char *)rq->mpwqe.info + array_size(i, isz)); } + +struct mlx5e_devtx_ctx { +struct devtx_ctx devtx; +union { +struct { +struct mlx5_cqe64 *cqe;/* tx completion */ +struct mlx5e_cq *cq;/* tx completion */ + }; +struct mlx5e_tx_wqe *wqe;/* tx */ + }; +}; + +DECLARE_DEVTX_HOOKS (mlx5e); +#endif
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +151 -4
215 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +3 -1
@@ -82,13 +82,15 @@ enum mlx5e_xdp_xmit_mode {* num, page_1, page_2, ... , page_num. * * MLX5E_XDP_XMIT_MODE_XSK: -* none. +* frame.xsk_head + frame.xsk_head_len for header portion only. */ union mlx5e_xdp_info {enum mlx5e_xdp_xmit_mode mode;union {struct xdp_frame *xdpf;dma_addr_t dma_addr; +void *xsk_head; +u32 xsk_head_len; } frame;union {struct mlx5e_rq *rq;
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c +10 -0
@@ -55,6 +55,10 @@ static void mlx5e_xsk_tx_post_err(struct mlx5e_xdpsq *sq, nopwqe =mlx5e_post_nop (&sq->wq, sq->sqn, &sq->pc);mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, *xdpi); +mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, + (union mlx5e_xdp_info ) { .frame.xsk_head =NULL }); +mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, + (union mlx5e_xdp_info ) { .frame.xsk_head_len =0 }); sq->doorbell_cseg = &nopwqe->ctrl; }
@@ -106,6 +110,12 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget)mlx5e_xsk_tx_post_err (sq, &xdpi); }else {mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, xdpi); +mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, + (union mlx5e_xdp_info ) + { .frame.xsk_head = xdptxd.data }); +mlx5e_xdpi_fifo_push (&sq->db.xdpi_fifo, + (union mlx5e_xdp_info ) + { .frame.xsk_head_len = xdptxd.len }); } flush = true;
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +24 -0
@@ -758,6 +758,18 @@ static void mlx5e_tx_wi_consume_fifo_skbs(struct mlx5e_txqsq *sq, struct mlx5e_tfor (i =0 ; i < wi->num_fifo_pkts; i++) {struct sk_buff *skb =mlx5e_skb_fifo_pop (&sq->db.skb_fifo); +if (devtx_enabled ()) { +struct mlx5e_devtx_ctx ctx = { + .devtx = { + .netdev = skb->dev, + .sinfo =skb_shinfo (skb), + }, + .cqe = cqe, + }; + +mlx5e_devtx_complete_skb (&ctx.devtx, skb); + } +mlx5e_consume_skb (sq, skb, cqe, napi_budget); } }
@@ -826,6 +838,18 @@ bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget) sqcc += wi->num_wqebbs;if (likely (wi->skb)) { +if (devtx_enabled ()) { +struct mlx5e_devtx_ctx ctx = { + .devtx = { + .netdev = wi->skb->dev, + .sinfo =skb_shinfo (wi->skb), + }, + .cqe = cqe, + }; + +mlx5e_devtx_complete_skb (&ctx.devtx, wi->skb); + } +mlx5e_tx_wi_dma_unmap (sq, wi, &dma_fifo_cc);mlx5e_consume_skb (sq, wi->skb, cqe, napi_budget);
drivers/net/ethernet/mellanox/mlx5/core/main.c +14 -1
@@ -48,6 +48,7 @@#include <linux/mlx5/vport.h> #include <linux/version.h> #include <net/devlink.h> +#include <net/devtx.h> #include "mlx5_core.h" #include "thermal.h" #include "lib/eq.h"
@@ -73,6 +74,7 @@#include "sf/dev/dev.h" #include "sf/sf.h" #include "mlx5_irq.h" +#include "en/xdp.h" MODULE_AUTHOR ("Eli Cohen <[email protected]>" );MODULE_DESCRIPTION ("Mellanox 5th generation network adapters (ConnectX series) core driver" );
@@ -2277,6 +2279,8 @@ static void mlx5_core_verify_params(void) } } +DEFINE_DEVTX_HOOKS (mlx5e); +static int __init mlx5_init (void ) {int err;
@@ -2289,9 +2293,15 @@ static int __init mlx5_init(void)mlx5_core_verify_params ();mlx5_register_debugfs (); + err =devtx_hooks_register (&mlx5e_devtx_hook_ids, &mlx5e_xdp_metadata_ops); +if (err) { +pr_warn ("failed to register devtx hooks: %d" , err); + goto err_debug; + } + err =mlx5e_init ();if (err) - goto err_debug; + goto err_devtx; err =mlx5_sf_driver_register ();if (err)
@@ -2307,6 +2317,8 @@ static int __init mlx5_init(void)mlx5_sf_driver_unregister (); err_sf:mlx5e_cleanup (); +err_devtx: +devtx_hooks_unregister (&mlx5e_devtx_hook_ids); err_debug:mlx5_unregister_debugfs ();return err;
@@ -2314,6 +2326,7 @@ static int __init mlx5_init(void) static void __exit mlx5_cleanup(void) { +devtx_hooks_unregister (&mlx5e_devtx_hook_ids);pci_unregister_driver (&mlx5_core_driver);mlx5_sf_driver_unregister ();mlx5e_cleanup ();
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 11 lines · Stanislav Fomichev
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +11
-0+ +static int mlx5e_devtx_request_l4_checksum (const struct devtx_ctx *_ctx, +u16 csum_start,u16 csum_offset) +{ +const struct mlx5e_devtx_ctx *ctx = (void *)_ctx; +struct mlx5_wqe_eth_seg *eseg; + +if (unlikely (!ctx->wqe)) +return -ENODATA ; + + eseg = &ctx->wqe->eth;
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c static int mlx5e_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 301
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +10
-0+ +switch (csum_offset) { +case sizeof (struct ethhdr ) +sizeof (struct iphdr ) + offsetof(struct udphdr , check): +case sizeof (struct ethhdr ) +sizeof (struct ipv6hdr ) + offsetof(struct udphdr , check): +/* Looks like HW/FW is doing parsing, so offsets are largely ignored. */ + eseg->cs_flags =MLX5_ETH_WQE_L3_CSUM |MLX5_ETH_WQE_L4_CSUM ; +break ; +default : +return -EINVAL ; + }
I think this proves my point: csum is not generalizable even across veth and mlx5. Above is a square peg that tries to fit csum_start/offset api (that makes sense from SW pov) into HW that has different ideas about csum-ing.
Here is what mlx5 does:
mlx5e_txwqe_build_eseg_csum(struct mlx5e_txqsq *sq, struct sk_buff *skb,
struct mlx5e_accel_tx_state *accel, struct mlx5_wqe_eth_seg *eseg)
{
if (unlikely(mlx5e_ipsec_txwqe_build_eseg_csum(sq, skb, eseg)))
return;
if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
if (skb->encapsulation) {
eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM |
MLX5_ETH_WQE_L4_INNER_CSUM;
sq->stats->csum_partial_inner++;
} else {
eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM;
sq->stats->csum_partial++;
}How would you generalize that into csum api that will work across NICs ?
My answer stands: you cannot.
My proposal again:
add driver specifc kfuncs and hooks for things like csum.
Kuba,
since you nacked driver specific stuff please suggest a way to unblock this stalemate.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 47 lines · Alexei Starovoitov
Quoted 11 lines · Stanislav Fomichev
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +11
-0+ +static int mlx5e_devtx_request_l4_checksum (const struct devtx_ctx *_ctx, +u16 csum_start,u16 csum_offset) +{ +const struct mlx5e_devtx_ctx *ctx = (void *)_ctx; +struct mlx5_wqe_eth_seg *eseg; + +if (unlikely (!ctx->wqe)) +return -ENODATA ; + + eseg = &ctx->wqe->eth;drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c static int mlx5e_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 301
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +10
-0+ +switch (csum_offset) { +case sizeof (struct ethhdr ) +sizeof (struct iphdr ) + offsetof(struct udphdr , check): +case sizeof (struct ethhdr ) +sizeof (struct ipv6hdr ) + offsetof(struct udphdr , check): +/* Looks like HW/FW is doing parsing, so offsets are largely ignored. */ + eseg->cs_flags =MLX5_ETH_WQE_L3_CSUM |MLX5_ETH_WQE_L4_CSUM ; +break ; +default : +return -EINVAL ; + }I think this proves my point: csum is not generalizable even across veth and mlx5. Above is a square peg that tries to fit csum_start/offset api (that makes sense from SW pov) into HW that has different ideas about csum-ing.
Here is what mlx5 does:
mlx5e_txwqe_build_eseg_csum(structmlx5e_txqsq*sq, structsk_buff*skb,
structmlx5e_accel_tx_state*accel, structmlx5_wqe_eth_seg*eseg){ if (unlikely(mlx5e_ipsec_txwqe_build_eseg_csum(sq, skb, eseg))) return; if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) { eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM; if (skb->encapsulation) { eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM | MLX5_ETH_WQE_L4_INNER_CSUM; sq->stats->csum_partial_inner++; } else { eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM; sq->stats->csum_partial++; }
Quoted 6 lines
How would you generalize that into csum api that will work across NICs ?
My answer stands: you cannot.
My proposal again:
add driver specifc kfuncs and hooks for things like csum.
I do see your point, but to also give you my perspective: I have no clue what those _CSUM tx bits do (as a non-mlx employee). And what kind of packets they support (initial patch doesn't give any info). We can definitely expose mlx5 specific request_l4_checksum(bool encap) which does things similar to mlx5e_txwqe_build_eseg_csum, but then, what does it _actually_ do? It obviously can't checksum arbitrary packet formats (because it has this inner/outer selection bit), so there is really no way for me to provide a per-driver kfunc api. Maybe the vendors can?
So having csum_start/csum_offset abstraction which works with fixed offsets seems like at least it correctly sets the expectation for BPF program writers.
The vendors are already supposed to conform to this start/offset API for skb.
But back to your point: should we maybe try to meet somewhere in the middle?
1. We try to provide "generic" offload kfuncs; for mlx5, we'll have this mlx5e_devtx_request_l4_checksum which works for fixed offsets
2. We also let vendors do device-specific "extensions" where devices deviate too much: bpf_request_RAW_mlx5e_l4_checksum(bool encap) This can give BPF authors opportunity to write somewhat portable programs and also use vendor specific apis when/if needed.
I think we had a similar idea for rx side: have generic kfuncs, but also let vendors experiment with custom kfuncs if they want to differentiate.
WDYT? Can it give us the best things from both sides?
Quoted 2 lines
Kuba,
since you nacked driver specific stuff please suggest a way to unblock this stalemate.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 67 lines · Stanislav Fomichev
Quoted 47 lines · Alexei Starovoitov
Quoted 11 lines · Stanislav Fomichev
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +11
-0+ +static int mlx5e_devtx_request_l4_checksum (const struct devtx_ctx *_ctx, +u16 csum_start,u16 csum_offset) +{ +const struct mlx5e_devtx_ctx *ctx = (void *)_ctx; +struct mlx5_wqe_eth_seg *eseg; + +if (unlikely (!ctx->wqe)) +return -ENODATA ; + + eseg = &ctx->wqe->eth;drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c static int mlx5e_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 301
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +10
-0+ +switch (csum_offset) { +case sizeof (struct ethhdr ) +sizeof (struct iphdr ) + offsetof(struct udphdr , check): +case sizeof (struct ethhdr ) +sizeof (struct ipv6hdr ) + offsetof(struct udphdr , check): +/* Looks like HW/FW is doing parsing, so offsets are largely ignored. */ + eseg->cs_flags =MLX5_ETH_WQE_L3_CSUM |MLX5_ETH_WQE_L4_CSUM ; +break ; +default : +return -EINVAL ; + }I think this proves my point: csum is not generalizable even across veth and mlx5. Above is a square peg that tries to fit csum_start/offset api (that makes sense from SW pov) into HW that has different ideas about csum-ing.
Here is what mlx5 does:
mlx5e_txwqe_build_eseg_csum(structmlx5e_txqsq*sq, structsk_buff*skb,
structmlx5e_accel_tx_state*accel, structmlx5_wqe_eth_seg*eseg){ if (unlikely(mlx5e_ipsec_txwqe_build_eseg_csum(sq, skb, eseg))) return; if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) { eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM; if (skb->encapsulation) { eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM | MLX5_ETH_WQE_L4_INNER_CSUM; sq->stats->csum_partial_inner++; } else { eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM; sq->stats->csum_partial++; }Quoted 6 lines
How would you generalize that into csum api that will work across NICs ?
My answer stands: you cannot.
My proposal again:
add driver specifc kfuncs and hooks for things like csum.I do see your point, but to also give you my perspective: I have no clue what those
_CSUMtx bits do (as a non-mlx employee). And what kind of packets they support (initial patch doesn't give any info). We can definitely expose mlx5 specificrequest_l4_checksum(bool encap)which does things similar tomlx5e_txwqe_build_eseg_csum, but then, what does it _actually_ do? It obviously can't checksum arbitrary packet formats (because it has this inner/outer selection bit), so there is really no way for me to provide a per-driver kfunc api. Maybe the vendors can?
Quoted 8 lines
So having csum_start/csum_offset abstraction which works with fixed offsets seems like at least it correctly sets the expectation for BPF program writers.
The vendors are already supposed to conform to this start/offset API for skb.But back to your point: should we maybe try to meet somewhere in the middle?
1. We try to provide "generic" offload kfuncs; for mlx5, we'll have thismlx5e_devtx_request_l4_checksumwhich works for fixed offsets
But it doesn't.
Even if you add a check for csum_start (that's missing in the patch) there need to be a way to somehow figure out
whether skb->encapsulation is true and set appropriate flags. Otherwise this request csum will do "something" that only the HW vendor knows. That would be even harder to debug for bpf prog writers.
So instead of helping bpf prog devs it will actively hurt them.
Another example. If bpf prog was developed and tested on veth it will work for some values of csum_offset on real HW and will -EINVAL for the other values.
Just horrible user experience comparing to the case where the user knows that each netdev is potentially different and _has_ to develop and test their prog on the given HW NIC and not assume that the kernel can "do the right thing".
This csum exercise is clear example that kernel is not in a position to do so.
For timestamp it's arguable, but for csum there is no generic api that kernel can apply universally to NICs.
Quoted 12 lines
2. We also let vendors do device-specific "extensions" where devices deviate too much:
bpf_request_RAW_mlx5e_l4_checksum(bool encap)This can give BPF authors opportunity to write somewhat portable programs and also use vendor specific apis when/if needed.I think we had a similar idea for rx side: have generic kfuncs, but also let vendors experiment with custom kfuncs if they want to differentiate.
WDYT? Can it give us the best things from both sides?Quoted 2 lines
Kuba,
since you nacked driver specific stuff please suggest a way to unblock this stalemate.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 78 lines · Alexei Starovoitov
Quoted 67 lines · Stanislav Fomichev
Quoted 47 lines · Alexei Starovoitov
Quoted 11 lines · Stanislav Fomichev
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +11
-0+ +static int mlx5e_devtx_request_l4_checksum (const struct devtx_ctx *_ctx, +u16 csum_start,u16 csum_offset) +{ +const struct mlx5e_devtx_ctx *ctx = (void *)_ctx; +struct mlx5_wqe_eth_seg *eseg; + +if (unlikely (!ctx->wqe)) +return -ENODATA ; + + eseg = &ctx->wqe->eth;drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c static int mlx5e_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 301
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +10
-0+ +switch (csum_offset) { +case sizeof (struct ethhdr ) +sizeof (struct iphdr ) + offsetof(struct udphdr , check): +case sizeof (struct ethhdr ) +sizeof (struct ipv6hdr ) + offsetof(struct udphdr , check): +/* Looks like HW/FW is doing parsing, so offsets are largely ignored. */ + eseg->cs_flags =MLX5_ETH_WQE_L3_CSUM |MLX5_ETH_WQE_L4_CSUM ; +break ; +default : +return -EINVAL ; + }I think this proves my point: csum is not generalizable even across veth and mlx5. Above is a square peg that tries to fit csum_start/offset api (that makes sense from SW pov) into HW that has different ideas about csum-ing.
Here is what mlx5 does:
mlx5e_txwqe_build_eseg_csum(structmlx5e_txqsq*sq, structsk_buff*skb,
structmlx5e_accel_tx_state*accel, structmlx5_wqe_eth_seg*eseg){ if (unlikely(mlx5e_ipsec_txwqe_build_eseg_csum(sq, skb, eseg))) return; if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) { eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM; if (skb->encapsulation) { eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM | MLX5_ETH_WQE_L4_INNER_CSUM; sq->stats->csum_partial_inner++; } else { eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM; sq->stats->csum_partial++; }Quoted 6 lines
How would you generalize that into csum api that will work across NICs ?
My answer stands: you cannot.
My proposal again:
add driver specifc kfuncs and hooks for things like csum.I do see your point, but to also give you my perspective: I have no clue what those
_CSUMtx bits do (as a non-mlx employee). And what kind of packets they support (initial patch doesn't give any info). We can definitely expose mlx5 specificrequest_l4_checksum(bool encap)which does things similar tomlx5e_txwqe_build_eseg_csum, but then, what does it _actually_ do? It obviously can't checksum arbitrary packet formats (because it has this inner/outer selection bit), so there is really no way for me to provide a per-driver kfunc api. Maybe the vendors can?Quoted 8 lines
So having csum_start/csum_offset abstraction which works with fixed offsets seems like at least it correctly sets the expectation for BPF program writers.
The vendors are already supposed to conform to this start/offset API for skb.But back to your point: should we maybe try to meet somewhere in the middle?
1. We try to provide "generic" offload kfuncs; for mlx5, we'll have thismlx5e_devtx_request_l4_checksumwhich works for fixed offsets
Quoted 8 lines
But it doesn't.
Even if you add a check forcsum_start(that's missing in the patch) there need to be a way to somehow figure out
whetherskb->encapsulationis true and set appropriate flags. Otherwise this request csum will do "something" that only the HW vendor knows. That would be even harder to debug for bpf prog writers.So instead of helping bpf prog devs it will actively hurt them.
Can we make it more robust? The device can look at the payload (via descriptors or extra payload pointer via devtx_ctx) and verify h_proto/nexthdr.
It won't be perfect, I agree, but we can get it working for the common cases (and have device-specific kfuncs for the rest).
Quoted 7 lines
Another example. If bpf prog was developed and tested on veth it will work for some values of
csum_offseton real HW and will-EINVALfor the other values.
Just horrible user experience comparing to the case where the user knows that each netdev is potentially different and _has_ to develop and test their prog on the given HW NIC and not assume that the kernel can "do the right thing".
For this, I was actually thinking that we need to provide some SW-based fallback mechanism.
Because if I have a program and a nic that doesn't have an offload implemented at all, having a fallback might be useful:
if (bpf_devtx_request_l4_csum(...)) {
/* oops, hw bailed on us, fallback to sw and expose a counter */ bpf_devtx_l4_csum_slowpath(csum_start, csum_offset, data, len); pkt_sw_csum++;
}
This is probably needed regardless of which way we do it?
Regarding veth vs non-veth: we already have similar issues with generic xdp vs non-generic.
I'm not sure we can completely avoid having surprises when switching from sw to hw paths.
It's whether the users will have to debug 10-20% of their program or they'd have to start completely from scratch for every nic.
Quoted 4 lines
This csum exercise is clear example that kernel is not in a position to do so.
For timestamp it's arguable, but for csum there is no generic api that kernel can apply universally to NICs.
Sure, I agree, it's a mix of both. For some offloads, we can have something common, for some we can't.
But I'm not sure why we have to pick one or another. We can try to have common apis (maybe not ideal, yes) and we can expose vendor specific ones if there is need.
If the generic ones get unused - we kill them in the future. If none of the vendors comes up with non-generic ones - the generic ones are good enough.
I'm assuming you favor non-generic ones because it's easier to implement? But most of the netdev-bound infra is already there for rx, so there is really not a lot of extra code to reuse it at tx. (it's two lines to allow tracing progs to be dev-bound and to check whether devices match at attach).
Or is there some other reason I'm missing?
Quoted 12 lines
Quoted 12 lines
2. We also let vendors do device-specific "extensions" where devices deviate too much:
bpf_request_RAW_mlx5e_l4_checksum(bool encap)This can give BPF authors opportunity to write somewhat portable programs and also use vendor specific apis when/if needed.I think we had a similar idea for rx side: have generic kfuncs, but also let vendors experiment with custom kfuncs if they want to differentiate.
WDYT? Can it give us the best things from both sides?Quoted 2 lines
Kuba,
since you nacked driver specific stuff please suggest a way to unblock this stalemate.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 88 lines · Stanislav Fomichev
Quoted 85 lines · Alexei Starovoitov
Quoted 67 lines · Stanislav Fomichev
Quoted 47 lines · Alexei Starovoitov
Quoted 11 lines · Stanislav Fomichev
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +11
-0+ +static int mlx5e_devtx_request_l4_checksum (const struct devtx_ctx *_ctx, +u16 csum_start,u16 csum_offset) +{ +const struct mlx5e_devtx_ctx *ctx = (void *)_ctx; +struct mlx5_wqe_eth_seg *eseg; + +if (unlikely (!ctx->wqe)) +return -ENODATA ; + + eseg = &ctx->wqe->eth;drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c static int mlx5e_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 301
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +10
-0+ +switch (csum_offset) { +case sizeof (struct ethhdr ) +sizeof (struct iphdr ) + offsetof(struct udphdr , check): +case sizeof (struct ethhdr ) +sizeof (struct ipv6hdr ) + offsetof(struct udphdr , check): +/* Looks like HW/FW is doing parsing, so offsets are largely ignored. */ + eseg->cs_flags =MLX5_ETH_WQE_L3_CSUM |MLX5_ETH_WQE_L4_CSUM ; +break ; +default : +return -EINVAL ; + }I think this proves my point: csum is not generalizable even across veth and mlx5. Above is a square peg that tries to fit csum_start/offset api (that makes sense from SW pov) into HW that has different ideas about csum-ing.
Here is what mlx5 does:
mlx5e_txwqe_build_eseg_csum(structmlx5e_txqsq*sq, structsk_buff*skb,
structmlx5e_accel_tx_state*accel, structmlx5_wqe_eth_seg*eseg){ if (unlikely(mlx5e_ipsec_txwqe_build_eseg_csum(sq, skb, eseg))) return; if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) { eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM; if (skb->encapsulation) { eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM | MLX5_ETH_WQE_L4_INNER_CSUM; sq->stats->csum_partial_inner++; } else { eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM; sq->stats->csum_partial++; }Quoted 6 lines
How would you generalize that into csum api that will work across NICs ?
My answer stands: you cannot.
My proposal again:
add driver specifc kfuncs and hooks for things like csum.I do see your point, but to also give you my perspective: I have no clue what those
_CSUMtx bits do (as a non-mlx employee). And what kind of packets they support (initial patch doesn't give any info). We can definitely expose mlx5 specificrequest_l4_checksum(bool encap)which does things similar tomlx5e_txwqe_build_eseg_csum, but then, what does it _actually_ do? It obviously can't checksum arbitrary packet formats (because it has this inner/outer selection bit), so there is really no way for me to provide a per-driver kfunc api. Maybe the vendors can?Quoted 8 lines
So having csum_start/csum_offset abstraction which works with fixed offsets seems like at least it correctly sets the expectation for BPF program writers.
The vendors are already supposed to conform to this start/offset API for skb.But back to your point: should we maybe try to meet somewhere in the middle?
1. We try to provide "generic" offload kfuncs; for mlx5, we'll have thismlx5e_devtx_request_l4_checksumwhich works for fixed offsetsBut it doesn't.
Even if you add a check forcsum_start(that's missing in the patch) there need to be a way to somehow figure out
whetherskb->encapsulationis true and set appropriate flags. Otherwise this request csum will do "something" that only the HW vendor knows. That would be even harder to debug for bpf prog writers.
Quoted 7 lines
Quoted 1 line
So instead of helping bpf prog devs it will actively hurt them.
Can we make it more robust? The device can look at the payload (via descriptors or extra payload pointer via
devtx_ctx) and verify h_proto/nexthdr.
It won't be perfect, I agree, but we can get it working for the common cases (and have device-specific kfuncs for the rest).
More robust with more checks ?
That will slow things down and the main advantage of XDP vs skb layer will be lost.
It's best to stay at skb then when csum and timestamp is available.
Quoted 13 lines
Quoted 7 lines
Another example. If bpf prog was developed and tested on veth it will work for some values of
csum_offseton real HW and will-EINVALfor the other values.
Just horrible user experience comparing to the case where the user knows that each netdev is potentially different and _has_ to develop and test their prog on the given HW NIC and not assume that the kernel can "do the right thing".For this, I was actually thinking that we need to provide some SW-based fallback mechanism.
Because if I have a program and a nic that doesn't have an offload implemented at all, having a fallback might be useful:
Quoted 7 lines
if (
bpf_devtx_request_l4_csum(...)) {
/* oops, hw bailed on us, fallback to sw and expose a counter */bpf_devtx_l4_csum_slowpath(csum_start, csum_offset, data, len); pkt_sw_csum++;
}This is probably needed regardless of which way we do it?
sw fallback? We already have 'generic XDP' and people misuse it thinking it's a layer they should be using.
It's a nice feeling to say that my XDP prog was developed and tested on mlx5, but I can move it to a different server with brand new NIC that doesn't support XDP yet and my prog will still work because of "generic XDP".
I think such devs are playing with fire and will be burned when "generic XDP" NIC will be DDoSed.
Same thing here. If we do HW offload of csum it's better be in HW. Devs have to be 100% certain that HW is offloading it.
Quoted 7 lines
Regarding veth vs non-veth: we already have similar issues with generic xdp vs non-generic.
I'm not sure we can completely avoid having surprises when switching from sw to hw paths.
It's whether the users will have to debug 10-20% of their program or they'd have to start completely from scratch for every nic.
If rewrite for a nic is not acceptable then they should be using skb layer.
Quoted 15 lines
Quoted 5 lines
This csum exercise is clear example that kernel is not in a position to do so.
For timestamp it's arguable, but for csum there is no generic api that kernel can apply universally to NICs.Sure, I agree, it's a mix of both. For some offloads, we can have something common, for some we can't.
But I'm not sure why we have to pick one or another. We can try to have common apis (maybe not ideal, yes) and we can expose vendor specific ones if there is need.
If the generic ones get unused - we kill them in the future. If none of the vendors comes up with non-generic ones - the generic ones are good enough.
Quoted 1 line
I'm assuming you favor non-generic ones because it's easier to implement?
Not only.
yes, it's easier to implement, but the expectations are also clear. The kernel won't be trying to fall back to the slow path. XDP prog will tell HW 'do csum' and HW will do it.
For generality we have an skb layer.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 97 lines · Alexei Starovoitov
Quoted 88 lines · Stanislav Fomichev
Quoted 85 lines · Alexei Starovoitov
Quoted 67 lines · Stanislav Fomichev
Quoted 47 lines · Alexei Starovoitov
Quoted 11 lines · Stanislav Fomichev
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +11
-0+ +static int mlx5e_devtx_request_l4_checksum (const struct devtx_ctx *_ctx, +u16 csum_start,u16 csum_offset) +{ +const struct mlx5e_devtx_ctx *ctx = (void *)_ctx; +struct mlx5_wqe_eth_seg *eseg; + +if (unlikely (!ctx->wqe)) +return -ENODATA ; + + eseg = &ctx->wqe->eth;drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c static int mlx5e_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 301
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +10
-0+ +switch (csum_offset) { +case sizeof (struct ethhdr ) +sizeof (struct iphdr ) + offsetof(struct udphdr , check): +case sizeof (struct ethhdr ) +sizeof (struct ipv6hdr ) + offsetof(struct udphdr , check): +/* Looks like HW/FW is doing parsing, so offsets are largely ignored. */ + eseg->cs_flags =MLX5_ETH_WQE_L3_CSUM |MLX5_ETH_WQE_L4_CSUM ; +break ; +default : +return -EINVAL ; + }I think this proves my point: csum is not generalizable even across veth and mlx5. Above is a square peg that tries to fit csum_start/offset api (that makes sense from SW pov) into HW that has different ideas about csum-ing.
Here is what mlx5 does:
mlx5e_txwqe_build_eseg_csum(structmlx5e_txqsq*sq, structsk_buff*skb,
structmlx5e_accel_tx_state*accel, structmlx5_wqe_eth_seg*eseg){ if (unlikely(mlx5e_ipsec_txwqe_build_eseg_csum(sq, skb, eseg))) return; if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) { eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM; if (skb->encapsulation) { eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM | MLX5_ETH_WQE_L4_INNER_CSUM; sq->stats->csum_partial_inner++; } else { eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM; sq->stats->csum_partial++; }Quoted 6 lines
How would you generalize that into csum api that will work across NICs ?
My answer stands: you cannot.
My proposal again:
add driver specifc kfuncs and hooks for things like csum.I do see your point, but to also give you my perspective: I have no clue what those
_CSUMtx bits do (as a non-mlx employee). And what kind of packets they support (initial patch doesn't give any info). We can definitely expose mlx5 specificrequest_l4_checksum(bool encap)which does things similar tomlx5e_txwqe_build_eseg_csum, but then, what does it _actually_ do? It obviously can't checksum arbitrary packet formats (because it has this inner/outer selection bit), so there is really no way for me to provide a per-driver kfunc api. Maybe the vendors can?Quoted 8 lines
So having csum_start/csum_offset abstraction which works with fixed offsets seems like at least it correctly sets the expectation for BPF program writers.
The vendors are already supposed to conform to this start/offset API for skb.But back to your point: should we maybe try to meet somewhere in the middle?
1. We try to provide "generic" offload kfuncs; for mlx5, we'll have thismlx5e_devtx_request_l4_checksumwhich works for fixed offsetsBut it doesn't.
Even if you add a check forcsum_start(that's missing in the patch) there need to be a way to somehow figure out
whetherskb->encapsulationis true and set appropriate flags. Otherwise this request csum will do "something" that only the HW vendor knows. That would be even harder to debug for bpf prog writers.Quoted 7 lines
Quoted 1 line
So instead of helping bpf prog devs it will actively hurt them.
Can we make it more robust? The device can look at the payload (via descriptors or extra payload pointer via
devtx_ctx) and verify h_proto/nexthdr.
It won't be perfect, I agree, but we can get it working for the common cases (and have device-specific kfuncs for the rest).
Quoted 4 lines
More robust with more checks ?
That will slow things down and the main advantage of XDP vs skb layer will be lost.
It's best to stay at skb then when csum and timestamp is available.
This will slow things down, but not to the point where it's on par with doing sw checksum. At least in theory.
We can't stay at skb when using AF_XDP. AF_XDP would benefit from having the offloads.
Quoted 21 lines
Quoted 13 lines
Quoted 7 lines
Another example. If bpf prog was developed and tested on veth it will work for some values of
csum_offseton real HW and will-EINVALfor the other values.
Just horrible user experience comparing to the case where the user knows that each netdev is potentially different and _has_ to develop and test their prog on the given HW NIC and not assume that the kernel can "do the right thing".For this, I was actually thinking that we need to provide some SW-based fallback mechanism.
Because if I have a program and a nic that doesn't have an offload implemented at all, having a fallback might be useful:Quoted 7 lines
if (
bpf_devtx_request_l4_csum(...)) {
/* oops, hw bailed on us, fallback to sw and expose a counter */bpf_devtx_l4_csum_slowpath(csum_start, csum_offset, data, len); pkt_sw_csum++;
}This is probably needed regardless of which way we do it?
Quoted 8 lines
sw fallback? We already have 'generic XDP' and people misuse it thinking it's a layer they should be using.
It's a nice feeling to say that my XDP prog was developed and tested on mlx5, but I can move it to a different server with brand new NIC that doesn't support XDP yet and my prog will still work because of "generic XDP".
I think such devs are playing with fire and will be burned when "generic XDP" NIC will be DDoSed.
This is user controlled. The users might as well chose to not fallback to sw by not calling another helper and fail hard.
I agree with generic xdp concerns and veth path being deceptive, but I also believe they serve their purpose of getting user's feet wet before they switch to experimenting with real devices:
you get most of you program working and polish it up on the real HW.
Moving to a different NIC will obviously require testing. But the difference with generic APIs is that the programs will mostly work and won't have to be completely rewritten.
Quoted 2 lines
Same thing here. If we do HW offload of csum it's better be in HW. Devs have to be 100% certain that HW is offloading it.
I hope we can both agree that with an api like mlx5_l4_csum_offload(bool encap) we can't be 100% certain that the hw is gonna handle any packet layout? So how is that different from a generic api that also can't work in all cases?
Quoted 8 lines
Quoted 6 lines
Regarding veth vs non-veth: we already have similar issues with generic xdp vs non-generic.
I'm not sure we can completely avoid having surprises when switching from sw to hw paths.
It's whether the users will have to debug 10-20% of their program or they'd have to start completely from scratch for every nic.If rewrite for a nic is not acceptable then they should be using skb layer.
AF_XDP is a generic layer for low-level access and it provides generic descriptor format, so why suddenly we have this requirement where we have to do prog rewrite for every new nic?
Current AF_XDP programs are pretty portable (obviously depend on a bunch of nic features), it seems like a good idea to try to preserve this property? (again, with an asterisk, where we should allow some differentiation, etc, etc)
Quoted 14 lines
Quoted 14 lines
Quoted 4 lines
This csum exercise is clear example that kernel is not in a position to do so.
For timestamp it's arguable, but for csum there is no generic api that kernel can apply universally to NICs.Sure, I agree, it's a mix of both. For some offloads, we can have something common, for some we can't.
But I'm not sure why we have to pick one or another. We can try to have common apis (maybe not ideal, yes) and we can expose vendor specific ones if there is need.
If the generic ones get unused - we kill them in the future. If none of the vendors comes up with non-generic ones - the generic ones are good enough.
Quoted 7 lines
Quoted 1 line
I'm assuming you favor non-generic ones because it's easier to implement?
Not only.
yes, it's easier to implement, but the expectations are also clear. The kernel won't be trying to fall back to the slow path. XDP prog will tell HW 'do csum' and HW will do it.
For generality we have an skb layer.
(seems like I've addressed these points above, I don't see anything extra here)
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 6 lines · Stanislav Fomichev
This will slow things down, but not to the point where it's on par with doing sw checksum. At least in theory.
We can't stay at skb when usingAF_XDP.AF_XDPwould benefit from having the offloads.
To clarify: yes, AF_XDP needs generalized HW offloads.
I just don't see how xdp tx offloads are moving a needle in that direction.
Quoted 4 lines
I hope we can both agree that with an api like
mlx5_l4_csum_offload(bool encap)we can't be 100% certain that the hw is gonna handle any packet layout? So how is that different from a generic api that also can't work in all cases?
If it's hw specific then yes.
Will [mlx5|...]_l4_csum_offload apply to other nics? I doubt.
Quoted 8 lines
AF_XDPis a generic layer for low-level access and it provides generic descriptor format, so why suddenly we have this requirement where we have to do prog rewrite for every new nic?Current
AF_XDPprograms are pretty portable (obviously depend on a bunch of nic features), it seems like a good idea to try to preserve this property? (again, with an asterisk, where we should allow some differentiation, etc, etc)
Agree. AF_XDP needs a generic api that will allow user space request HW to do TSO, csum offload, etc.
xdp tx and af_xdp are too different to pull under the same framework. xdp progs will interact with the kernel via kfuncs.af_xdp needs a different api to express packet geometry and offload requests. The user space cannot do it with bpf programs.
In case of AF_XDP the bpf prog in the kernel is a filter only. For the majority of the cases bpf prog is not necessary and shouldn't be required to express HW offloads.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 10 lines · Alexei Starovoitov
Quoted 6 lines · Stanislav Fomichev
This will slow things down, but not to the point where it's on par with doing sw checksum. At least in theory.
We can't stay at skb when usingAF_XDP.AF_XDPwould benefit from having the offloads.To clarify: yes,
AF_XDPneeds generalized HW offloads.
Great! To reiterate, I'm mostly interested in af_xdp wrt tx timestamps. So if the consensus is not to mix xdp-tx and af_xdp-tx, I'm fine with switching to adding some fixed af_xdp descriptor format to enable offloads on tx.
Quoted 1 line
I just don't see how xdp tx offloads are moving a needle in that direction.
Let me try to explain how both might be similar, maybe I wasn't clear enough on that.
For af_xdp tx packet, the userspace puts something in the af_xdp frame metadata area (headrom) which then gets executed/interpreted by the bpf program at devtx (which calls kfuncs to enable particular offloads).
IOW, instead of defining some fixed layout for the tx offloads, the userspace and bpf program have some agreement on the layout (and bpf program "applies" the offloads by calling the kfuncs).
Also (in theory) the same hooks can be used for xdp-tx. Does it make sense? But, again, happy to scratch that whole idea if we're fine with a fixed layout for af_xdp.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 21 lines · Stanislav Fomichev
Quoted 10 lines · Alexei Starovoitov
Quoted 6 lines · Stanislav Fomichev
This will slow things down, but not to the point where it's on par with doing sw checksum. At least in theory.
We can't stay at skb when usingAF_XDP.AF_XDPwould benefit from having the offloads.To clarify: yes,
AF_XDPneeds generalized HW offloads.Great! To reiterate, I'm mostly interested in
af_xdpwrt tx timestamps. So if the consensus is not to mix xdp-tx and af_xdp-tx, I'm fine with switching to adding some fixedaf_xdpdescriptor format to enable offloads on tx.Quoted 1 line
I just don't see how xdp tx offloads are moving a needle in that direction.
Quoted 12 lines
Let me try to explain how both might be similar, maybe I wasn't clear enough on that.
Foraf_xdptx packet, the userspace puts something in theaf_xdpframe metadata area (headrom) which then gets executed/interpreted by the bpf program at devtx (which calls kfuncs to enable particular offloads).
IOW, instead of defining some fixed layout for the tx offloads, the userspace and bpf program have some agreement on the layout (and bpf program "applies" the offloads by calling the kfuncs).
Also (in theory) the same hooks can be used for xdp-tx. Does it make sense? But, again, happy to scratch that whole idea if we're fine with a fixed layout foraf_xdp.
Checksum offload is an important demonstrator too.
It is admittedly a non-trivial one. Checksum offload has often been discussed as a pain point ("protocol ossification").
In general, drivers can accept every CHECKSUM_COMPLETE skb that matches their advertised feature NETIF_F_[HW|IP|IPV6]_CSUM. I don't see why this would be different for kfuncs for packets coming from userspace.
The problematic drivers are the ones that do not implement CHECKSUM_COMPLETE as intended, but ignore this simple protocol-independent hint in favor of parsing from scratch, possibly zeroing the field, computing multiple layers, etc.
All of which is unnecessary with LCO. An AF_XDP user can be expected to apply LCO and only request checksum insertion for the innermost checksum.
The biggest problem is with these devices that parse in hardware (and possibly also in the driver to identify and fix up hardware limitations) is that they will fail if encountering an unknown protocol. Which brings us to advertising limited typed support:NETIF_F_HW_CSUM vs NETIF_F_IP_CSUM.
The fact that some devices that deviate from industry best practices cannot support more advanced packet formats is unfortunate, but not a reason to hold others back. No different from current kernel path. The BPF program can fallback onto software checksumming on these devices, like the kernel path. Perhaps we do need to pass along with csum_start and csum_off a csum_type that matches the existing NETIF_F_[HW|IP|IPV6]_CSUM, to let drivers return with -EOPNOTSUPP quickly if for the generic case.
For implementation in essence it is just reordering driver code that already exists for the skb case. I think the ice patch series to support rx timestamping is a good indication of what it takes to support XDP kfuncs: not so much new code, but reordering the driver logic.
Which also indicates to me that the driver *is* the right place to implement this logic, rather than reimplement it in a BPF library. It avoids both code duplication and dependency hell, if the library ships independent from the driver.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 36 lines · Willem de Bruijn
Quoted 21 lines · Stanislav Fomichev
Quoted 10 lines · Alexei Starovoitov
Quoted 6 lines · Stanislav Fomichev
This will slow things down, but not to the point where it's on par with doing sw checksum. At least in theory.
We can't stay at skb when usingAF_XDP.AF_XDPwould benefit from having the offloads.To clarify: yes,
AF_XDPneeds generalized HW offloads.Great! To reiterate, I'm mostly interested in
af_xdpwrt tx timestamps. So if the consensus is not to mix xdp-tx and af_xdp-tx, I'm fine with switching to adding some fixedaf_xdpdescriptor format to enable offloads on tx.Quoted 1 line
I just don't see how xdp tx offloads are moving a needle in that direction.
Quoted 12 lines
Let me try to explain how both might be similar, maybe I wasn't clear enough on that.
Foraf_xdptx packet, the userspace puts something in theaf_xdpframe metadata area (headrom) which then gets executed/interpreted by the bpf program at devtx (which calls kfuncs to enable particular offloads).
IOW, instead of defining some fixed layout for the tx offloads, the userspace and bpf program have some agreement on the layout (and bpf program "applies" the offloads by calling the kfuncs).
Also (in theory) the same hooks can be used for xdp-tx. Does it make sense? But, again, happy to scratch that whole idea if we're fine with a fixed layout foraf_xdp.
Quoted 6 lines
Checksum offload is an important demonstrator too.
It is admittedly a non-trivial one. Checksum offload has often been discussed as a pain point ("protocol ossification").
In general, drivers can accept every
CHECKSUM_COMPLETEskb that
Erm.. CHECKSUM_PARTIAL
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
In reply to Willem de BruijnQuoted 13 lines · Willem de Bruijn
Quoted 12 lines · Stanislav Fomichev
Quoted 9 lines · Alexei Starovoitov
Quoted 6 lines · Stanislav Fomichev
This will slow things down, but not to the point where it's on par with doing sw checksum. At least in theory.
We can't stay at skb when usingAF_XDP.AF_XDPwould benefit from having the offloads.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
To clarify: yes,
AF_XDPneeds generalized HW offloads.Great! To reiterate, I'm mostly interested in
af_xdpwrt tx timestamps. So if the consensus is not to mix xdp-tx and af_xdp-tx, I'm fine with switching to adding some fixedaf_xdpdescriptor format to enable offloads on tx.
since af_xdp is a primary user let's figure out what is the best api for that. If any code can be salvaged for xdp tx, great, but let's not start with xdp tx as prerequisite.
Quoted 15 lines
Quoted 15 lines
Quoted 2 lines
I just don't see how xdp tx offloads are moving a needle in that direction.
Let me try to explain how both might be similar, maybe I wasn't clear enough on that.
Foraf_xdptx packet, the userspace puts something in theaf_xdpframe metadata area (headrom) which then gets executed/interpreted by the bpf program at devtx (which calls kfuncs to enable particular offloads).
IOW, instead of defining some fixed layout for the tx offloads, the userspace and bpf program have some agreement on the layout (and bpf program "applies" the offloads by calling the kfuncs).
Also (in theory) the same hooks can be used for xdp-tx. Does it make sense? But, again, happy to scratch that whole idea if we're fine with a fixed layout foraf_xdp.
So instead of defining csum offload format in xsk metadata we'll defining it as a set of arguments to a kfunc and tx-side xsk prog will just copy the args from metadata into kfunc args ?
Seems like an unnecesary step. Such xsk prog won't be doing anything useful. Just copying from one place to another.
It seems the only purpose of such bpf prog is to side step uapi exposure. bpf is not used to program anything. There won't be any control flow. Just odd intermediate copy step.
Instead we can define a metadata struct for csum nic offload outside of uapi/linux/if_xdp.h with big 'this is not an uapi' warning. User space can request it via setsockopt.
And probably feature query the nic via getsockopt.
Error handling is critical here. With xsk tx prog the errors are messy. What to do when kfunc returns error? Store it back into packet metadata ? and then user space needs to check every single packet for errors? Not practical imo.
Feature query via getsockopt would be done once instead and user space will fill in "csum offload struct" in packet metadata and won't check per-packet error. If driver said the csum feature is available it's better work for every packet.
Notice mlx5e_txwqe_build_eseg_csum() returns void.
Quoted 41 lines
Checksum offload is an important demonstrator too.
It is admittedly a non-trivial one. Checksum offload has often been discussed as a pain point ("protocol ossification").
In general, drivers can accept every
CHECKSUM_COMPLETEskb that matches their advertised feature NETIF_F_[HW|IP|IPV6]_CSUM. I don't see why this would be different for kfuncs for packets coming from userspace.The problematic drivers are the ones that do not implement
CHECKSUM_COMPLETEas intended, but ignore this simple protocol-independent hint in favor of parsing from scratch, possibly zeroing the field, computing multiple layers, etc.All of which is unnecessary with LCO. An
AF_XDPuser can be expected to apply LCO and only request checksum insertion for the innermost checksum.The biggest problem is with these devices that parse in hardware (and possibly also in the driver to identify and fix up hardware limitations) is that they will fail if encountering an unknown protocol. Which brings us to advertising limited typed support:
NETIF_F_HW_CSUMvsNETIF_F_IP_CSUM.The fact that some devices that deviate from industry best practices cannot support more advanced packet formats is unfortunate, but not a reason to hold others back. No different from current kernel path. The BPF program can fallback onto software checksumming on these devices, like the kernel path. Perhaps we do need to pass along with
csum_startandcsum_offacsum_typethat matches the existing NETIF_F_[HW|IP|IPV6]_CSUM, to let drivers return with-EOPNOTSUPPquickly if for the generic case.For implementation in essence it is just reordering driver code that already exists for the skb case. I think the ice patch series to support rx timestamping is a good indication of what it takes to support XDP kfuncs: not so much new code, but reordering the driver logic.
Quoted 4 lines
Which also indicates to me that the driver *is* the right place to implement this logic, rather than reimplement it in a BPF library. It avoids both code duplication and dependency hell, if the library ships independent from the driver.
Agree with all of the above.
I think defining CHECKSUM_PARTIAL struct request for af_xdp is doable and won't require much changes in the drivers.
If we do it for more than one driver from the start there is a chance it will work for other drivers too. imo ice+gve+mlx5 would be enough.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 112 lines · Alexei Starovoitov
Quoted 13 lines · Willem de Bruijn
Quoted 12 lines · Stanislav Fomichev
Quoted 9 lines · Alexei Starovoitov
Quoted 6 lines · Stanislav Fomichev
This will slow things down, but not to the point where it's on par with doing sw checksum. At least in theory.
We can't stay at skb when usingAF_XDP.AF_XDPwould benefit from having the offloads.Quoted 6 lines
Quoted 6 lines
Quoted 1 line
To clarify: yes,
AF_XDPneeds generalized HW offloads.Great! To reiterate, I'm mostly interested in
af_xdpwrt tx timestamps. So if the consensus is not to mix xdp-tx and af_xdp-tx, I'm fine with switching to adding some fixedaf_xdpdescriptor format to enable offloads on tx.since
af_xdpis a primary user let's figure out what is the best api for that. If any code can be salvaged for xdp tx, great, but let's not start with xdp tx as prerequisite.Quoted 15 lines
Quoted 15 lines
Quoted 2 lines
I just don't see how xdp tx offloads are moving a needle in that direction.
Let me try to explain how both might be similar, maybe I wasn't clear enough on that.
Foraf_xdptx packet, the userspace puts something in theaf_xdpframe metadata area (headrom) which then gets executed/interpreted by the bpf program at devtx (which calls kfuncs to enable particular offloads).
IOW, instead of defining some fixed layout for the tx offloads, the userspace and bpf program have some agreement on the layout (and bpf program "applies" the offloads by calling the kfuncs).
Also (in theory) the same hooks can be used for xdp-tx. Does it make sense? But, again, happy to scratch that whole idea if we're fine with a fixed layout foraf_xdp.So instead of defining csum offload format in xsk metadata we'll defining it as a set of arguments to a kfunc and tx-side xsk prog will just copy the args from metadata into kfunc args ?
Seems like an unnecesary step. Such xsk prog won't be doing anything useful. Just copying from one place to another.
It seems the only purpose of such bpf prog is to side step uapi exposure. bpf is not used to program anything. There won't be any control flow. Just odd intermediate copy step.
Instead we can define a metadata struct for csum nic offload outside ofuapi/linux/if_xdp.hwith big 'this is not an uapi' warning. User space can request it via setsockopt.
And probably feature query the nic via getsockopt.Error handling is critical here. With xsk tx prog the errors are messy. What to do when kfunc returns error? Store it back into packet metadata ? and then user space needs to check every single packet for errors? Not practical imo.
Feature query via getsockopt would be done once instead and user space will fill in "csum offload struct" in packet metadata and won't check per-packet error. If driver said the csum feature is available it's better work for every packet.
Noticemlx5e_txwqe_build_eseg_csum()returns void.Quoted 41 lines
Checksum offload is an important demonstrator too.
It is admittedly a non-trivial one. Checksum offload has often been discussed as a pain point ("protocol ossification").
In general, drivers can accept every
CHECKSUM_COMPLETEskb that matches their advertised feature NETIF_F_[HW|IP|IPV6]_CSUM. I don't see why this would be different for kfuncs for packets coming from userspace.The problematic drivers are the ones that do not implement
CHECKSUM_COMPLETEas intended, but ignore this simple protocol-independent hint in favor of parsing from scratch, possibly zeroing the field, computing multiple layers, etc.All of which is unnecessary with LCO. An
AF_XDPuser can be expected to apply LCO and only request checksum insertion for the innermost checksum.The biggest problem is with these devices that parse in hardware (and possibly also in the driver to identify and fix up hardware limitations) is that they will fail if encountering an unknown protocol. Which brings us to advertising limited typed support:
NETIF_F_HW_CSUMvsNETIF_F_IP_CSUM.The fact that some devices that deviate from industry best practices cannot support more advanced packet formats is unfortunate, but not a reason to hold others back. No different from current kernel path. The BPF program can fallback onto software checksumming on these devices, like the kernel path. Perhaps we do need to pass along with
csum_startandcsum_offacsum_typethat matches the existing NETIF_F_[HW|IP|IPV6]_CSUM, to let drivers return with-EOPNOTSUPPquickly if for the generic case.For implementation in essence it is just reordering driver code that already exists for the skb case. I think the ice patch series to support rx timestamping is a good indication of what it takes to support XDP kfuncs: not so much new code, but reordering the driver logic.
Quoted 4 lines
Which also indicates to me that the driver *is* the right place to implement this logic, rather than reimplement it in a BPF library. It avoids both code duplication and dependency hell, if the library ships independent from the driver.
Quoted 5 lines
Agree with all of the above.
I think definingCHECKSUM_PARTIALstruct request foraf_xdpis doable and won't require much changes in the drivers.
If we do it for more than one driver from the start there is a chance it will work for other drivers too. imo ice+gve+mlx5 would be enough.
Basically, add to AF_XDP what we already have for its predecessorAF_PACKET: setsockopt PACKET_VNET_HDR?
Possibly with a separate new struct, rather than virtio_net_hdr. As that has dependencies on other drivers, notably virtio and its specification process.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 7 lines · Willem de Bruijn
Basically, add to
AF_XDPwhat we already have for its predecessorAF_PACKET: setsockoptPACKET_VNET_HDR?Possibly with a separate new struct, rather than
virtio_net_hdr. As that has dependencies on other drivers, notably virtio and its specification process.
yeah. Forgot about this one.
That's a perfect fit. I would reuse virtio_net_hdr as-is. Why reinvent the wheel?
It would force uapi, but some might argue it's a good thing.
Jakub Kicinski
maintainerRe: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 11 lines · On Wed, 12 Jul 2023 12:42:35 -0700 Alexei Starovoitov wrote:
Quoted 6 lines
Basically, add to
AF_XDPwhat we already have for its predecessorAF_PACKET: setsockoptPACKET_VNET_HDR?Possibly with a separate new struct, rather than
virtio_net_hdr. As that has dependencies on other drivers, notably virtio and its specification process.yeah. Forgot about this one.
That's a perfect fit. I would reusevirtio_net_hdras-is. Why reinvent the wheel?
It would force uapi, but some might argue it's a good thing.
I was gonna reply on the other leg of the thread but it sounds like we're in agreement now? 👏️ virtio_net_hdr is the kind of generic descriptor I had in mind.
I'd suggest breaking hdr_len into L2len, L3len and L4len, tho. How does virtio do IP length updates during TSO if it doesn't know where L3 starts? HW will want to know, and it's easy to add them together in cases where it doesn't. Which is why I kept saying "packet geometry" rather than pointing at virtio_net_hdr.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 17 lines · Jakub Kicinski
Quoted 11 lines · On Wed, 12 Jul 2023 12:42:35 -0700 Alexei Starovoitov wrote:
Quoted 6 lines
Basically, add to
AF_XDPwhat we already have for its predecessorAF_PACKET: setsockoptPACKET_VNET_HDR?Possibly with a separate new struct, rather than
virtio_net_hdr. As that has dependencies on other drivers, notably virtio and its specification process.yeah. Forgot about this one.
That's a perfect fit. I would reusevirtio_net_hdras-is. Why reinvent the wheel?
It would force uapi, but some might argue it's a good thing.I was gonna reply on the other leg of the thread but it sounds like we're in agreement now? 👏️
virtio_net_hdris the kind of generic descriptor I had in mind.
Quoted 5 lines
I'd suggest breaking
hdr_leninto L2len, L3len and L4len, tho. How does virtio do IP length updates during TSO if it doesn't know where L3 starts? HW will want to know, and it's easy to add them together in cases where it doesn't. Which is why I kept saying "packet geometry" rather than pointing atvirtio_net_hdr.
Perfect, I'll drop the kfuncs and will switch to a more static way of passing this info. We can discuss the particular layout once I have something concrete to show :-)
Thanks, again, everyone for the comments!
Jakub Kicinski
maintainerRe: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
In reply to Alexei StarovoitovQuoted 8 lines · On Tue, 11 Jul 2023 15:56:57 -0700 Alexei Starovoitov wrote:
I think this proves my point: csum is not generalizable even across veth and mlx5. Above is a square peg that tries to fit csum_start/offset api (that makes sense from SW pov) into HW that has different ideas about csum-ing.
Here is what mlx5 does:
mlx5e_txwqe_build_eseg_csum(structmlx5e_txqsq*sq, structsk_buff*skb,
structmlx5e_accel_tx_state*accel, structmlx5_wqe_eth_seg*eseg)
Quoted 7 lines
{
if (unlikely(mlx5e_ipsec_txwqe_build_eseg_csum(sq, skb, eseg)))
return;
if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
if (skb->encapsulation) {This should be irrelevant today, as LCO exists?
Quoted 10 lines
eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM | MLX5_ETH_WQE_L4_INNER_CSUM; sq->stats->csum_partial_inner++; } else { eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM; sq->stats->csum_partial++; }How would you generalize that into csum api that will work across NICs ?
Quoted 7 lines
My answer stands: you cannot.
My proposal again:
add driver specifc kfuncs and hooks for things like csum.Kuba,
since you nacked driver specific stuff please suggest a way to unblock this stalemate.
I hope I'm not misremembering but I think I suggested at the beginning to create a structure describing packet geometry and requested offloads, and for the prog fill that in.
All operating systems I know end up doing that, we'll end up doing that as well. The question is whether we're willing to learn from experience or prefer to go on a wild ride first...
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 14 lines · Jakub Kicinski
Quoted 12 lines · On Tue, 11 Jul 2023 15:56:57 -0700 Alexei Starovoitov wrote:
I think this proves my point: csum is not generalizable even across veth and mlx5. Above is a square peg that tries to fit csum_start/offset api (that makes sense from SW pov) into HW that has different ideas about csum-ing.
Here is what mlx5 does:
mlx5e_txwqe_build_eseg_csum(structmlx5e_txqsq*sq, structsk_buff*skb,
structmlx5e_accel_tx_state*accel, structmlx5_wqe_eth_seg*eseg)
{
if (unlikely(mlx5e_ipsec_txwqe_build_eseg_csum(sq, skb, eseg)))
return;
Quoted 5 lines
Quoted 3 lines
if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) { eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM; if (skb->encapsulation) {This should be irrelevant today, as LCO exists?
Hmm. Maybe. But LCO is an example that prog devs have to be aware of and use it properly.
Meaning for certain protocols compute outer csum LCO way and let inner go through HW csuming.
In this case I have no idea what these mlx5 flags do.
I hope this part of the code was tested with udp tunnels.
Quoted 15 lines
Quoted 15 lines
eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM | MLX5_ETH_WQE_L4_INNER_CSUM; sq->stats->csum_partial_inner++; } else { eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM; sq->stats->csum_partial++; }How would you generalize that into csum api that will work across NICs ?
My answer stands: you cannot.
My proposal again:
add driver specifc kfuncs and hooks for things like csum.
Quoted 6 lines
Quoted 2 lines
Kuba,
since you nacked driver specific stuff please suggest a way to unblock this stalemate.I hope I'm not misremembering but I think I suggested at the beginning to create a structure describing packet geometry and requested offloads, and for the prog fill that in.
hmm. but that's what skb is for. skb == packet geometry == layout of headers, payload, inner vs outer, csum partial, gso params.
bpf at tc layer supposed to interact with that correctly. If the packet is modified skb geometry should be adjusted accordingly. Like BPF_F_RECOMPUTE_CSUM flag in bpf_skb_store_bytes().
Quoted 4 lines
All operating systems I know end up doing that, we'll end up doing that as well. The question is whether we're willing to learn from experience or prefer to go on a wild ride first...
I don't follow. This thread was aimed to add xdp layer knobs. To me XDP is a driver level. 'struct xdp_md' along with BPF_F_XDP_HAS_FRAGS is the best abstraction we could do generalizing dma-buffers (page and multi-page) that drivers operate on. Everything else at driver level is too unique to generalize. skb layer is already doing its job.
In that sense "generic XDP" is a feature for testing only. Trying to make "generic XDP" fast is missing the point of XDP.
AF_XDP is a different concept. Exposing timestamp,
csum, TSO to AF_XDP users is a different design challenge. I'm all for doing that, but trying to combine "timestamp in xdp tx" and "timestamp in AF_XDP" will lead to bad trade-off-s for both. Which I think this patchset demonstrates.
Jakub Kicinski
maintainerRe: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 11 lines · On Tue, 11 Jul 2023 19:37:23 -0700 Alexei Starovoitov wrote:
Quoted 3 lines
I hope I'm not misremembering but I think I suggested at the beginning to create a structure describing packet geometry and requested offloads, and for the prog fill that in.
hmm. but that's what skb is for. skb == packet geometry == layout of headers, payload, inner vs outer, csum partial, gso params.
bpf at tc layer supposed to interact with that correctly. If the packet is modified skb geometry should be adjusted accordingly. Like
BPF_F_RECOMPUTE_CSUMflag inbpf_skb_store_bytes().
Quoted 6 lines
Quoted 3 lines
All operating systems I know end up doing that, we'll end up doing that as well. The question is whether we're willing to learn from experience or prefer to go on a wild ride first...
I don't follow. This thread was aimed to add xdp layer knobs. To me XDP is a driver level.
Driver is not a layer of the networking stack, I don't think it's a useful or meaningful anchor point for the mental model.
We're talking about a set of functionality, we can look at how that functionality was exposed in existing code.
Quoted 3 lines
'struct
xdp_md' along withBPF_F_XDP_HAS_FRAGSis the best abstraction we could do generalizing dma-buffers (page and multi-page) that drivers operate on.
I think you're working against your own claim. xdp frags explicitly reuse struct skb_shared_info.
Quoted 2 lines
Everything else at driver level is too unique to generalize. skb layer is already doing its job.
How can you say that drivers are impossible to generalize and than that the skb layer "is doing its job" ie. generalizes them?
Quoted 2 lines
In that sense "generic XDP" is a feature for testing only. Trying to make "generic XDP" fast is missing the point of XDP.
That's a topic on its own.
Quoted 5 lines
AF_XDPis a different concept. Exposing timestamp,
csum, TSO toAF_XDPusers is a different design challenge. I'm all for doing that, but trying to combine "timestamp in xdp tx" and "timestamp inAF_XDP" will lead to bad trade-off-s for both. Which I think this patchset demonstrates.
Too vague to agree or disagree with.
Re: [RFC bpf-next v3 09/14] net/mlx5e: Implement devtx kfuncs
Quoted 26 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 11 Jul 2023 19:37:23 -0700 Alexei Starovoitov wrote:
Quoted 3 lines
I hope I'm not misremembering but I think I suggested at the beginning to create a structure describing packet geometry and requested offloads, and for the prog fill that in.
hmm. but that's what skb is for. skb == packet geometry == layout of headers, payload, inner vs outer, csum partial, gso params.
bpf at tc layer supposed to interact with that correctly. If the packet is modified skb geometry should be adjusted accordingly. Like
BPF_F_RECOMPUTE_CSUMflag inbpf_skb_store_bytes().Quoted 6 lines
Quoted 3 lines
All operating systems I know end up doing that, we'll end up doing that as well. The question is whether we're willing to learn from experience or prefer to go on a wild ride first...
I don't follow. This thread was aimed to add xdp layer knobs. To me XDP is a driver level.
Driver is not a layer of the networking stack, I don't think it's a useful or meaningful anchor point for the mental model.
We're talking about a set of functionality, we can look at how that functionality was exposed in existing code.
Quoted 6 lines
Quoted 3 lines
'struct
xdp_md' along withBPF_F_XDP_HAS_FRAGSis the best abstraction we could do generalizing dma-buffers (page and multi-page) that drivers operate on.I think you're working against your own claim. xdp frags explicitly reuse struct
skb_shared_info.
Yes they do and it's an implementation detail. skb_shared_info is convenient for drivers to fill in. xdp prog isn't aware of the exact mechanism. skb_shared_info is not exposed to a program. Not sure what point you're trying to make.
Quoted 6 lines
Quoted 3 lines
Everything else at driver level is too unique to generalize. skb layer is already doing its job.
How can you say that drivers are impossible to generalize and than that the skb layer "is doing its job" ie. generalizes them?
Yeah. 'generalize' is a wrong word to use to describe a feature that should have the same look and feel across drivers. What I meant by 'csum impossible to generalize across drivers' is that there is no HW-only path across them.
skb layer is doing it through sw fallback and combination of csum_complete/unnecessary.
I'm saying that your proposal of
"create a structure describing packet geometry and requested offloads" already exists and it's called skb.
I see no point doing this exercise again at XDP layer.
RFC v3 10/14 selftests/xsk: Support XDP_TX_METADATA_LEN
In reply to Stanislav FomichevAdd new config field and call setsockopt.
--- tools/testing/selftests/bpf/xsk.c | 17 +++++++++++++++++ tools/testing/selftests/bpf/xsk.h | 1 + 2 files changed, 18 insertions(+)
tools/testing/selftests/bpf/xsk.c +17 -0
@@ -47,6 +47,10 @@#define PF_XDP AF_XDP#endif +#ifndef XDP_TX_METADATA_LEN +#define XDP_TX_METADATA_LEN 9 +#endif +#define pr_warn (fmt, ...) fprintf(stderr, fmt, ##__VA_ARGS__)#define XSKMAP_SIZE 1
@@ -124,12 +128,14 @@ static int xsk_set_xdp_socket_config(struct xsk_socket_config *cfg, cfg->rx_size =XSK_RING_CONS__DEFAULT_NUM_DESCS ; cfg->tx_size =XSK_RING_PROD__DEFAULT_NUM_DESCS ; cfg->bind_flags =0 ; + cfg->tx_metadata_len =0 ;return 0 ; } cfg->rx_size = usr_cfg->rx_size; cfg->tx_size = usr_cfg->tx_size; cfg->bind_flags = usr_cfg->bind_flags; + cfg->tx_metadata_len = usr_cfg->tx_metadata_len;return 0 ; }
@@ -479,6 +485,17 @@ int xsk_socket__create_shared(struct xsk_socket **xsk_ptr, umem->tx_ring_setup_done = true; } +if (xsk->config.tx_metadata_len) { +int optval = xsk->config.tx_metadata_len; + + err =setsockopt (xsk->fd,SOL_XDP ,XDP_TX_METADATA_LEN , + &optval,sizeof (optval)); +if (err) { + err = -errno; + goto out_put_ctx; + } + } + err =xsk_get_mmap_offsets (xsk->fd, &off);if (err) { err = -errno;
tools/testing/selftests/bpf/xsk.h +1 -0
@@ -212,6 +212,7 @@ struct xsk_socket_config {__u32 rx_size;__u32 tx_size;__u16 bind_flags; +__u8 tx_metadata_len; };/* Set config to NULL to get the default configuration. */
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 11/14 selftests/bpf: Add helper to query current netns cookie
In reply to Stanislav FomichevWill be used to filter out netdev in AF_XDP metadata selftests. The helper returns netns cookie of the current process.
--- tools/testing/selftests/bpf/network_helpers.c | 21 +++++++++++++++++++ tools/testing/selftests/bpf/network_helpers.h | 1 + 2 files changed, 22 insertions(+)
tools/testing/selftests/bpf/network_helpers.c +21 -0
@@ -450,3 +450,24 @@ int get_socket_local_port(int sock_fd)return -1 ; } + +#ifndef SO_NETNS_COOKIE +#define SO_NETNS_COOKIE 71 +#endif + +__u64 get_net_cookie (void ) +{ +socklen_t optlen; +__u64 optval =0 ; +int fd; + + fd =socket (AF_LOCAL ,SOCK_DGRAM ,0 ); +if (fd >=0 ) { + optlen =sizeof (optval); +getsockopt (fd,SOL_SOCKET ,SO_NETNS_COOKIE , &optval, &optlen); +close (fd); + } + +return optval; +} +
tools/testing/selftests/bpf/network_helpers.h +1 -0
@@ -57,6 +57,7 @@ int make_sockaddr(int family, const char *addr_str, __u16 port, struct sockaddr_storage *addr, socklen_t *len);char *ping_command (int family);int get_socket_local_port (int sock_fd); +__u64 get_net_cookie (void );struct nstoken ;/**
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 12/14 selftests/bpf: Add csum helpers
In reply to Stanislav FomichevChecksum helpers will be used to calculate pseudo-header checksum in AF_XDP metadata selftests.
The helpers are mirroring existing kernel ones:
- csum_tcpudp_magic : IPv4 pseudo header csum
- csum_ipv6_magic : IPv6 pseudo header csum
- csum_fold : fold csum and do one's complement
--- tools/testing/selftests/bpf/network_helpers.h | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+)
tools/testing/selftests/bpf/network_helpers.h +43 -0
@@ -68,4 +68,47 @@ struct nstoken;*/ struct nstoken *open_netns (const char *name);void close_netns (struct nstoken *token); + +static __u16 csum_fold (__u32 csum) +{ + csum = (csum &0xffff ) + (csum >>16 ); + csum = (csum &0xffff ) + (csum >>16 ); + +return (__u16 )~csum; +} + +static inline __sum16 csum_tcpudp_magic (__be32 saddr,__be32 daddr, +__u32 len,__u8 proto, +__wsum csum) +{ +__u64 s = csum; + + s += (__u32 )saddr; + s += (__u32 )daddr; + s +=htons (proto + len); + s = (s &0xffffffff ) + (s >>32 ); + s = (s &0xffffffff ) + (s >>32 ); + +return csum_fold ((__u32 )s); +} + +static inline __sum16 csum_ipv6_magic (const struct in6_addr *saddr, +const struct in6_addr *daddr, +__u32 len,__u8 proto, +__wsum csum) +{ +__u64 s = csum; +int i; + +for (i =0 ; i <4 ; i++) + s += (__u32 )saddr->s6_addr32[i]; +for (i =0 ; i <4 ; i++) + s += (__u32 )daddr->s6_addr32[i]; + s +=htons (proto + len); + s = (s &0xffffffff ) + (s >>32 ); + s = (s &0xffffffff ) + (s >>32 ); + +return csum_fold ((__u32 )s); +} +#endif
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 13/14 selftests/bpf: Extend xdp_metadata with devtx kfuncs
In reply to Stanislav FomichevAttach kfuncs that request and report TX timestamp via ringbuf. Confirm on the userspace side that the program has triggered and the timestamp is non-zero.
Also make sure skb has a sensible pointers and data.
In addition, calculate pseudo-header checksum and offload payload checksum calculation to the kfunc.
--- .../selftests/bpf/prog_tests/xdp_metadata.c | 64 +++++++- .../selftests/bpf/progs/xdp_metadata.c | 141 ++++++++++++++++++ tools/testing/selftests/bpf/xdp_metadata.h | 16 ++ 3 files changed, 217 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/prog_tests/xdp_metadata.c +60 -4
@@ -42,6 +42,8 @@ struct xsk {struct xsk_ring_prod tx;struct xsk_ring_cons rx;struct xsk_socket *socket; +int tx_completions; +struct devtx_sample last_sample; };static int open_xsk (int ifindex,struct xsk *xsk)
@@ -51,6 +53,7 @@ static int open_xsk(int ifindex, struct xsk *xsk) .rx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, .tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, .bind_flags = XDP_COPY, + .tx_metadata_len = TX_META_LEN, };const struct xsk_umem_config umem_config = { .fill_size =XSK_RING_PROD__DEFAULT_NUM_DESCS ,
@@ -138,6 +141,7 @@ static void ip_csum(struct iphdr *iph)static int generate_packet (struct xsk *xsk,__u16 dst_port) { +struct xdp_tx_meta *meta;struct xdp_desc *tx_desc;struct udphdr *udph;struct ethhdr *eth;
@@ -151,10 +155,13 @@ static int generate_packet(struct xsk *xsk, __u16 dst_port)return -1 ; tx_desc =xsk_ring_prod__tx_desc (&xsk->tx, idx); - tx_desc->addr = idx % (UMEM_NUM /2 ) *UMEM_FRAME_SIZE ; + tx_desc->addr = idx % (UMEM_NUM /2 ) *UMEM_FRAME_SIZE +TX_META_LEN ;printf ("%p: tx_desc[%u]->addr=%llx\n" , xsk, idx, tx_desc->addr); data =xsk_umem__get_data (xsk->umem_area, tx_desc->addr); + meta = data -TX_META_LEN ; + meta->request_timestamp =1 ; + eth = data; iph = (void *)(eth +1 ); udph = (void *)(iph +1 );
@@ -178,7 +185,8 @@ static int generate_packet(struct xsk *xsk, __u16 dst_port) udph->source =htons (AF_XDP_SOURCE_PORT ); udph->dest =htons (dst_port); udph->len =htons (sizeof (*udph) +UDP_PAYLOAD_BYTES ); - udph->check =0 ; + udph->check = ~csum_tcpudp_magic (iph->saddr, iph->daddr, +ntohs (udph->len),IPPROTO_UDP ,0 );memset (udph +1 ,0xAA ,UDP_PAYLOAD_BYTES );
@@ -192,7 +200,8 @@ static int generate_packet(struct xsk *xsk, __u16 dst_port)return 0 ; } -static void complete_tx (struct xsk *xsk) +static void complete_tx (struct xsk *xsk,struct xdp_metadata *bpf_obj, +struct ring_buffer *ringbuf) {__u32 idx;__u64 addr;
@@ -202,6 +211,14 @@ static void complete_tx(struct xsk *xsk)printf ("%p: complete tx idx=%u addr=%llx\n" , xsk, idx, addr);xsk_ring_cons__release (&xsk->comp,1 ); + +ring_buffer__poll (ringbuf,1000 ); + +ASSERT_EQ (bpf_obj->bss->pkts_fail_tx,0 ,"pkts_fail_tx" ); +ASSERT_GE (xsk->tx_completions,1 ,"tx_completions" ); +ASSERT_EQ (xsk->last_sample.timestamp_retval,0 ,"timestamp_retval" ); +ASSERT_GE (xsk->last_sample.hw_timestamp,0 ,"hw_timestamp" ); +ASSERT_EQ (xsk->last_sample.tx_csum,0x1c72 ,"tx_csum" ); } }
@@ -276,8 +293,23 @@ static int verify_xsk_metadata(struct xsk *xsk)return 0 ; } +static int process_sample (void *ctx,void *data,size_t len) +{ +struct devtx_sample *sample = data; +struct xsk *xsk = ctx; + +printf ("%p: got tx timestamp sample %u %llu\n" , + xsk, sample->timestamp_retval, sample->hw_timestamp); + + xsk->tx_completions++; + xsk->last_sample = *sample; + +return 0 ; +} +void test_xdp_metadata (void ) { +struct ring_buffer *tx_compl_ringbuf =NULL ;struct xdp_metadata2 *bpf_obj2 =NULL ;struct xdp_metadata *bpf_obj =NULL ;struct bpf_program *new_prog, *prog;
@@ -290,6 +322,7 @@ void test_xdp_metadata(void)int retries =10 ;int rx_ifindex;int tx_ifindex; +int syscall_fd;int sock_fd;int ret;
@@ -323,6 +356,14 @@ void test_xdp_metadata(void)if (!ASSERT_OK_PTR (bpf_obj,"open skeleton" )) goto out; + prog =bpf_object__find_program_by_name (bpf_obj->obj,"tx_submit" ); +bpf_program__set_ifindex (prog, tx_ifindex); +bpf_program__set_flags (prog,BPF_F_XDP_DEV_BOUND_ONLY ); + + prog =bpf_object__find_program_by_name (bpf_obj->obj,"tx_complete" ); +bpf_program__set_ifindex (prog, tx_ifindex); +bpf_program__set_flags (prog,BPF_F_XDP_DEV_BOUND_ONLY ); + prog =bpf_object__find_program_by_name (bpf_obj->obj,"rx" );bpf_program__set_ifindex (prog, rx_ifindex);bpf_program__set_flags (prog,BPF_F_XDP_DEV_BOUND_ONLY );
@@ -330,6 +371,18 @@ void test_xdp_metadata(void)if (!ASSERT_OK (xdp_metadata__load (bpf_obj),"load skeleton" )) goto out; + bpf_obj->data->ifindex = tx_ifindex; + bpf_obj->data->net_cookie =get_net_cookie (); + + ret =xdp_metadata__attach (bpf_obj); +if (!ASSERT_OK (ret,"xdp_metadata__attach" )) + goto out; + + tx_compl_ringbuf =ring_buffer__new (bpf_map__fd (bpf_obj->maps.tx_compl_buf), + process_sample, &tx_xsk,NULL ); +if (!ASSERT_OK_PTR (tx_compl_ringbuf,"ring_buffer__new" )) + goto out; +/* Make sure we can't add dev-bound programs to prog maps. */ prog_arr =bpf_object__find_map_by_name (bpf_obj->obj,"prog_arr" );if (!ASSERT_OK_PTR (prog_arr,"no prog_arr map" ))
@@ -364,7 +417,8 @@ void test_xdp_metadata(void) "verify_xsk_metadata")) goto out; -complete_tx (&tx_xsk); +/* Verify AF_XDP TX packet has completion event with a timestamp. */ +complete_tx (&tx_xsk, bpf_obj, tx_compl_ringbuf);/* Make sure freplace correctly picks up original bound device * and doesn't crash.
@@ -402,5 +456,7 @@ void test_xdp_metadata(void)xdp_metadata__destroy (bpf_obj);if (tok)close_netns (tok); +if (tx_compl_ringbuf) +ring_buffer__free (tx_compl_ringbuf);SYS_NOFAIL ("ip netns del xdp_metadata" ); }
tools/testing/selftests/bpf/progs/xdp_metadata.c +141 -0
161 lines not on this page · raw message
tools/testing/selftests/bpf/xdp_metadata.h +16 -0
@@ -18,3 +18,19 @@ struct xdp_meta {__s32 rx_hash_err; }; }; + +struct devtx_sample { +int timestamp_retval; +__u64 hw_timestamp; +__u64 sw_complete_timestamp; +__u16 tx_csum; +}; + +#define TX_META_LEN 8 + +struct xdp_tx_meta { +__u8 request_timestamp; +__u8 padding0; +__u16 padding1; +__u32 padding2; +};
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
RFC v3 14/14 selftests/bpf: Extend xdp_hw_metadata with devtx kfuncs
In reply to Stanislav FomichevWhen we get packets on port 9091, we swap src/dst and send it out. At this point, we also request the timestamp and plumb it back to the userspace. The userspace simply prints the timestamp.
Also print current UDP checksum, rewrite it with the pseudo-header checksum and offload TX checksum calculation to devtx. Upon completion, report TX checksum back (mlx5 doesn't put it back, so I've used tcpdump to confirm that the checksum is correct).
Some other related changes:
- switched to zerocopy mode by default; new flag can be used to force old behavior
- request fixed TX_METADAT_LEN headroom
- some other small fixes (umem size, fill idx+i, etc)
mvbz3:~# ./xdp_hw_metadata eth3 -c mlx5e_devtx_complete_xdp -s mlx5e_devtx_submit_xd attach rx bpf program... poll: 0 (0) skip=1/0 redir=0 ring_full=0 rx_fail=0 tx_fail=0 l4_csum_fail=0 ... xsk_ring_cons__peek: 1 0xeb4cb8: rx_desc[0]->addr=80100 addr=80100 comp_addr=80100 rx_hash: 0x6A1A897A with RSS type:0x2A rx_timestamp: 1688749963628930772 (sec:1688749963.6289) XDP RX-time: 1688749963901850574 (sec:1688749963.9019) delta sec:0.2729 (272919.802 usec) AF_XDP time: 1688749963901967812 (sec:1688749963.9020) delta sec:0.0001 (117.238 usec) 0xeb4cb8: ping_pong with csum=8e3b (want 4b0b) 0xeb4cb8: complete tx idx=0 addr=8 got tx sample: tx_err 0 hw_timestamp 1688749963859814790 sw_timestamp 1688749963902297286 csum 8e3b 0xeb4cb8: complete rx idx=128 addr=80100 poll: 0 (0) skip=7/0 redir=1 ring_full=0 rx_fail=0 tx_fail=0 l4_csum_fail=0
mvbz4:~# nc -Nu -q1 ${MVBZ3_LINK_LOCAL_IP}%eth3 9091mvbz4:~# tcpdump -vvx -i eth3 udp
tcpdump: listening on eth3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:12:43.901436 IP6 (flowlabel 0x7a5d2, hlim 127, next-header UDP (17) payload length: 11) fe80::1270:fdff:fe48:1087.44339 > fe80::1270:fdff:fe48:1077.9091: [bad udp cksum 0x3b8e -> 0x0b4b!] UDP, length 3
0x0000: 6007 a5d2 000b 117f fe80 0000 0000 0000
0x0010: 1270 fdff fe48 1087 fe80 0000 0000 0000
0x0020: 1270 fdff fe48 1077 ad33 2383 000b 3b8e
0x0030: 7864 70
10:12:43.902125 IP6 (flowlabel 0x7a5d2, hlim 127, next-header UDP (17) payload length: 11) fe80::1270:fdff:fe48:1077.9091 > fe80::1270:fdff:fe48:1087.44339: [udp sum ok] UDP, length 3
0x0000: 6007 a5d2 000b 117f fe80 0000 0000 0000
0x0010: 1270 fdff fe48 1077 fe80 0000 0000 0000
0x0020: 1270 fdff fe48 1087 2383 ad33 000b 0b4b
0x0030: 7864 70--- .../selftests/bpf/progs/xdp_hw_metadata.c | 173 +++++++++++ tools/testing/selftests/bpf/xdp_hw_metadata.c | 269 +++++++++++++++++- 2 files changed, 427 insertions(+), 15 deletions(-)
tools/testing/selftests/bpf/progs/xdp_hw_metadata.c +173 -0
@@ -4,6 +4,7 @@#include "xdp_metadata.h" #include <bpf/bpf_helpers.h> #include <bpf/bpf_endian.h> +#include <bpf/bpf_tracing.h> struct { __uint(type, BPF_MAP_TYPE_XSKMAP);
@@ -12,14 +13,30 @@ struct {__type (value, __u32); } xsk SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 4096); +} tx_compl_buf SEC(".maps"); +__u64 pkts_skip =0 ; +__u64 pkts_tx_skip =0 ;__u64 pkts_fail =0 ;__u64 pkts_redir =0 ; +__u64 pkts_fail_tx =0 ; +__u64 pkts_fail_l4_csum =0 ; +__u64 pkts_ringbuf_full =0 ; + +int ifindex =-1 ; +__u64 net_cookie =-1 ; extern int bpf_xdp_metadata_rx_timestamp(const struct xdp_md *ctx, __u64 *timestamp) __ksym; extern int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, __u32 *hash, enum xdp_rss_hash_type *rss_type) __ksym; +extern int bpf_devtx_request_tx_timestamp(const struct devtx_ctx *ctx) __ksym; +extern int bpf_devtx_tx_timestamp(const struct devtx_ctx *ctx, __u64 *timestamp) __ksym; +extern int bpf_devtx_request_l4_csum(const struct devtx_ctx *ctx, + u16 csum_start, u16 csum_offset) __ksym; SEC("xdp")int rx (struct xdp_md *ctx)
@@ -90,4 +107,160 @@ int rx(struct xdp_md *ctx)return bpf_redirect_map (&xsk, ctx->rx_queue_index,XDP_PASS ); } +/* This is not strictly required; only to showcase how to access the payload. */ +static __always_inline bool tx_filter(const struct devtx_ctx *devtx, +const void *data,__be16 *proto) +{ +int port_offset =sizeof (struct ethhdr ) + offsetof(struct udphdr , source); +struct ethhdr eth = {}; +struct udphdr udp = {}; + +bpf_probe_read_kernel (ð.h_proto,sizeof (eth.h_proto), + data + offsetof(struct ethhdr , h_proto)); + + *proto = eth.h_proto; +if (eth.h_proto ==bpf_htons (ETH_P_IP )) { + port_offset +=sizeof (struct iphdr ); + }else if (eth.h_proto ==bpf_htons (ETH_P_IPV6 )) { + port_offset +=sizeof (struct ipv6hdr ); + }else { +__sync_add_and_fetch (&pkts_tx_skip,1 ); +return false; + } + +bpf_probe_read_kernel (&udp.source,sizeof (udp.source), data + port_offset); + +/* Replies to UDP:9091 */ +if (udp.source !=bpf_htons (9091 )) { +__sync_add_and_fetch (&pkts_tx_skip,1 ); +return false; + } + +return true; +} + +static inline bool my_netdev (const struct devtx_ctx *devtx) +{ +static struct net_device *netdev; + +if (netdev) +return netdev == devtx->netdev; + +if (devtx->netdev->ifindex != ifindex) +return false; +if (devtx->netdev->nd_net.net->net_cookie != net_cookie) +return false; + + netdev = devtx->netdev; +return true; +} + +static inline int udpoff (__be16 proto) +{ +if (proto ==bpf_htons (ETH_P_IP )) +return sizeof (struct ethhdr ) +sizeof (struct iphdr ); +else if (proto ==bpf_htons (ETH_P_IPV6 )) +return sizeof (struct ethhdr ) +sizeof (struct ipv6hdr ); +else +return 0 ; +} + +static inline int tx_submit (const struct devtx_ctx *devtx,const void *data,u8 meta_len) +{ +struct xdp_tx_meta meta = {}; +__be16 proto =0 ; +int off, ret; + +if (!my_netdev (devtx)) +return 0 ; +if (meta_len !=TX_META_LEN ) +return 0 ; + +bpf_probe_read_kernel (&meta,sizeof (meta), data -TX_META_LEN ); +if (!meta.request_timestamp) +return 0 ; + +if (!tx_filter (devtx, data, &proto)) +return 0 ; + + ret =bpf_devtx_request_tx_timestamp (devtx); +if (ret <0 ) +__sync_add_and_fetch (&pkts_fail_tx,1 ); + + off =udpoff (proto); +if (!off) +return 0 ; + + ret =bpf_devtx_request_l4_csum (devtx, off, off + offsetof(struct udphdr , check)); +if (ret <0 ) +__sync_add_and_fetch (&pkts_fail_l4_csum,1 ); + +return 0 ; +} + +SEC("?fentry") +int BPF_PROG (tx_submit_xdp ,const struct devtx_ctx *devtx,const struct xdp_frame *xdpf) +{ +return tx_submit (devtx, xdpf->data, xdpf->metasize); +} + +SEC("?fentry") +int BPF_PROG (tx_submit_skb ,const struct devtx_ctx *devtx,const struct sk_buff *skb) +{ +return tx_submit (devtx, skb->data, devtx->sinfo->meta_len); +} + +static inline int tx_complete (const struct devtx_ctx *devtx,const void *data,u8 meta_len) +{ +struct xdp_tx_meta meta = {}; +struct devtx_sample *sample; +struct udphdr udph; +__be16 proto =0 ; +int off; + +if (!my_netdev (devtx)) +return 0 ; +if (meta_len !=TX_META_LEN ) +return 0 ; + +bpf_probe_read_kernel (&meta,sizeof (meta), data -TX_META_LEN ); +if (!meta.request_timestamp) +return 0 ; + +if (!tx_filter (devtx, data, &proto)) +return 0 ; + + off =udpoff (proto); +if (!off) +return 0 ; + +bpf_probe_read_kernel (&udph,sizeof (udph), data + off); + + sample =bpf_ringbuf_reserve (&tx_compl_buf,sizeof (*sample),0 ); +if (!sample) { +__sync_add_and_fetch (&pkts_ringbuf_full,1 ); +return 0 ; + } + + sample->timestamp_retval =bpf_devtx_tx_timestamp (devtx, &sample->hw_timestamp); + sample->sw_complete_timestamp =bpf_ktime_get_tai_ns (); + sample->tx_csum = udph.check; + +bpf_ringbuf_submit (sample,0 ); + +return 0 ; +} + +SEC("?fentry") +int BPF_PROG (tx_complete_xdp ,const struct devtx_ctx *devtx,const struct xdp_frame *xdpf) +{ +return tx_complete (devtx, xdpf->data, xdpf->metasize); +} + +SEC("?fentry") +int BPF_PROG (tx_complete_skb ,const struct devtx_ctx *devtx,const struct sk_buff *skb) +{ +return tx_complete (devtx, skb->data, devtx->sinfo->meta_len); +} + char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/xdp_hw_metadata.c +254 -15
394 lines not on this page · raw message
Signature · 1 line
2.41.0.255.g8b1d071c50-goog