Add support for transmitting packets using XDP in bpf_prog_run()
Subsystems (9)
v4 cover Add support for transmitting packets using XDP in bpf_prog_run()
This series adds support for transmitting packets using XDP in bpf_prog_run(), by enabling a new mode "live packet" mode which will handle the XDP program return codes and redirect the packets to the stack or other devices.
The primary use case for this is testing the redirect map types and the ndo_xdp_xmit driver operation without an external traffic generator. But it turns out to also be useful for creating a programmable traffic generator in XDP, as well as injecting frames into the stack. A sample traffic generator, which was included in previous versions of the series, but now moved to xdp-tools, transmits up to 11.5 Mpps/core on my test machine.
To transmit the frames, the new mode instantiates a page_pool structure in bpf_prog_run() and initialises the pages to contain XDP frames with the data passed in by userspace. These frames can then be handled as though they came from the hardware XDP path, and the existing page_pool code takes care of returning and recycling them. The setup is optimised for high performance with a high number of repetitions to support stress testing and the traffic generator use case; see patch 6 for details.
The series is structured as follows: Patches 1-5 adds a few features to page_pool that are needed for the usage in bpf_prog_run() and perform a couple of preparatory refactorings of the XDP redirect and memory management code. Patch 6 adds the support to bpf_prog_run() itself and patch 7 adds a selftest that exercises all the XDP return codes.
v4:
- Fix a few code style issues (Alexei)
- Also handle the other return codes: XDP_PASS builds skbs and injects them into the stack, and XDP_TX is turned into a redirect out the same interface (Alexei).
- Drop the last patch adding an xdp_trafficgen program to samples/bpf; this will live in xdp-tools instead (Alexei).
- Add a separate bpf_test_run_xdp_live() function to test_run.c instead of entangling the new mode in the existing bpf_test_run().
v3:
- Reorder patches to make sure they all build individually (Patchwork)
- Remove a couple of unused variables (Patchwork)
- Remove unlikely() annotation in slow path and add back John's ACK that I accidentally dropped for v2 (John)
v2:
- Split up up __xdp_do_redirect to avoid passing two pointers to it (John)
- Always reset context pointers before each test run (John)
- Use get_mac_addr() from xdp_sample_user.h instead of rolling our own (Kumar)
- Fix wrong offset for metadata pointer
Toke Høiland-Jørgensen (7):
xdp: Allow registering memory model without rxq reference
page_pool: Add callback to init pages when they are allocated
page_pool: Store the XDP mem id
xdp: Move conversion to xdp_frame out of map functions
xdp: add xdp_do_redirect_frame() for pre-computed xdp_frames
bpf: Add "live packet" mode for XDP in bpf_prog_run() selftests/bpf: Add selftest for XDP_REDIRECT in bpf_prog_run()
include/linux/bpf.h | 20 +- include/linux/filter.h | 4 + include/net/page_pool.h | 11 +- include/net/xdp.h | 3 + include/uapi/linux/bpf.h | 2 + kernel/bpf/Kconfig | 1 + kernel/bpf/cpumap.c | 8 +- kernel/bpf/devmap.c | 32 +- net/bpf/test_run.c | 287 +++++++++++++++++- net/core/filter.c | 73 ++++- net/core/page_pool.c | 6 +- net/core/xdp.c | 94 ++++-- tools/include/uapi/linux/bpf.h | 2 + .../bpf/prog_tests/xdp_do_redirect.c | 118 +++++++ .../bpf/progs/test_xdp_do_redirect.c | 39 +++ 15 files changed, 610 insertions(+), 90 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c
Signature · 1 line
2.34.1
PATCH v4 1/7 xdp: Allow registering memory model without rxq reference
The functions that register an XDP memory model take a struct xdp_rxq as parameter, but the RXQ is not actually used for anything other than pulling out the struct xdp_mem_info that it embeds. So refactor the register functions and export variants that just take a pointer to the xdp_mem_info.
This is in preparation for enabling XDP_REDIRECT in bpf_prog_run(), using a page_pool instance that is not connected to any network device.
--- include/net/xdp.h | 3 ++ net/core/xdp.c | 92 +++++++++++++++++++++++++++++++---------------- 2 files changed, 65 insertions(+), 30 deletions(-)
include/net/xdp.h +3 -0
@@ -260,6 +260,9 @@ bool xdp_rxq_info_is_reg(struct xdp_rxq_info *xdp_rxq);int xdp_rxq_info_reg_mem_model (struct xdp_rxq_info *xdp_rxq,enum xdp_mem_type type,void *allocator);void xdp_rxq_info_unreg_mem_model (struct xdp_rxq_info *xdp_rxq); +int xdp_reg_mem_model (struct xdp_mem_info *mem, +enum xdp_mem_type type,void *allocator); +void xdp_unreg_mem_model (struct xdp_mem_info *mem);/* Drivers not supporting XDP metadata can use this helper, which * rejects any room expansion for metadata as a result.
net/core/xdp.c +62 -30
157 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v4 2/7 page_pool: Add callback to init pages when they are allocated
In reply to Toke Høiland-JørgensenAdd a new callback function to page_pool that, if set, will be called every time a new page is allocated. This will be used from bpf_test_run() to initialise the page data with the data provided by userspace when running XDP programs with redirect turned on.
--- include/net/page_pool.h | 2 ++ net/core/page_pool.c | 2 ++ 2 files changed, 4 insertions(+)
include/net/page_pool.h +2 -0
@@ -80,6 +80,8 @@ struct page_pool_params {enum dma_data_direction dma_dir;/* DMA mapping direction */ unsigned int max_len;/* max DMA sync memory size */ unsigned int offset;/* DMA addr offset */ +void (*init_callback)(struct page *page,void *arg); +void *init_arg; };struct page_pool {
net/core/page_pool.c +2 -0
@@ -217,6 +217,8 @@ static void page_pool_set_pp_info(struct page_pool *pool, { page->pp = pool; page->pp_magic |=PP_SIGNATURE ; +if (pool->p.init_callback) + pool->p.init_callback (page, pool->p.init_arg); }static void page_pool_clear_pp_info (struct page *page)
Signature · 1 line
2.34.1
PATCH v4 3/7 page_pool: Store the XDP mem id
In reply to Toke Høiland-JørgensenStore the XDP mem ID inside the page_pool struct so it can be retrieved later for use in bpf_prog_run().
--- include/net/page_pool.h | 9 +++++++-- net/core/page_pool.c | 4 +++- net/core/xdp.c | 2 +- 3 files changed, 11 insertions(+), 4 deletions(-)
include/net/page_pool.h +7 -2
@@ -96,6 +96,7 @@ struct page_pool {unsigned int frag_offset;struct page *frag_page;long frag_users; +u32 xdp_mem_id;/* * Data structure for allocation side
@@ -170,9 +171,12 @@ bool page_pool_return_skb_page(struct page *page);struct page_pool *page_pool_create (const struct page_pool_params *params); +struct xdp_mem_info ; +#ifdef CONFIG_PAGE_POOL void page_pool_destroy (struct page_pool *pool); -void page_pool_use_xdp_mem (struct page_pool *pool,void (*disconnect)(void *)); +void page_pool_use_xdp_mem (struct page_pool *pool,void (*disconnect)(void *), +struct xdp_mem_info *mem);void page_pool_release_page (struct page_pool *pool,struct page *page);void page_pool_put_page_bulk (struct page_pool *pool,void **data,int count);
@@ -182,7 +186,8 @@ static inline void page_pool_destroy(struct page_pool *pool) }static inline void page_pool_use_xdp_mem (struct page_pool *pool, -void (*disconnect)(void *)) +void (*disconnect)(void *), +struct xdp_mem_info *mem) { } static inline void page_pool_release_page(struct page_pool *pool,
net/core/page_pool.c +3 -1
@@ -693,10 +693,12 @@ static void page_pool_release_retry(struct work_struct *wq)schedule_delayed_work (&pool->release_dw,DEFER_TIME ); } -void page_pool_use_xdp_mem (struct page_pool *pool,void (*disconnect)(void *)) +void page_pool_use_xdp_mem (struct page_pool *pool,void (*disconnect)(void *), +struct xdp_mem_info *mem) {refcount_inc (&pool->user_cnt); pool->disconnect = disconnect; + pool->xdp_mem_id = mem->id; }void page_pool_destroy (struct page_pool *pool)
net/core/xdp.c +1 -1
@@ -320,7 +320,7 @@ static struct xdp_mem_allocator *__xdp_reg_mem_model(struct xdp_mem_info *mem, }if (type ==MEM_TYPE_PAGE_POOL ) -page_pool_use_xdp_mem (allocator, mem_allocator_disconnect); +page_pool_use_xdp_mem (allocator, mem_allocator_disconnect, mem);mutex_unlock (&mem_id_lock);
Signature · 1 line
2.34.1
PATCH v4 4/7 xdp: Move conversion to xdp_frame out of map functions
In reply to Toke Høiland-JørgensenAll map redirect functions except XSK maps convert xdp_buff to xdp_frame before enqueueing it. So move this conversion of out the map functions and into xdp_do_redirect(). This removes a bit of duplicated code, but more importantly it makes it possible to support caller-allocated xdp_frame structures, which will be added in a subsequent commit.
--- include/linux/bpf.h | 20 ++++++++++---------- kernel/bpf/cpumap.c | 8 +------- kernel/bpf/devmap.c | 32 +++++++++++--------------------- net/core/filter.c | 24 +++++++++++++++++------- 4 files changed, 39 insertions(+), 45 deletions(-)
include/linux/bpf.h +10 -10
@@ -1616,17 +1616,17 @@ void bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth);struct btf *bpf_get_btf_vmlinux (void );/* Map specifics */ -struct xdp_buff ; +struct xdp_frame ;struct sk_buff ;struct bpf_dtab_netdev ;struct bpf_cpu_map_entry ;void __dev_flush (void ); -int dev_xdp_enqueue (struct net_device *dev,struct xdp_buff *xdp, +int dev_xdp_enqueue (struct net_device *dev,struct xdp_frame *xdpf,struct net_device *dev_rx); -int dev_map_enqueue (struct bpf_dtab_netdev *dst,struct xdp_buff *xdp, +int dev_map_enqueue (struct bpf_dtab_netdev *dst,struct xdp_frame *xdpf,struct net_device *dev_rx); -int dev_map_enqueue_multi (struct xdp_buff *xdp,struct net_device *dev_rx, +int dev_map_enqueue_multi (struct xdp_frame *xdpf,struct net_device *dev_rx,struct bpf_map *map,bool exclude_ingress);int dev_map_generic_redirect (struct bpf_dtab_netdev *dst,struct sk_buff *skb,struct bpf_prog *xdp_prog);
@@ -1635,7 +1635,7 @@ int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, bool exclude_ingress);void __cpu_map_flush (void ); -int cpu_map_enqueue (struct bpf_cpu_map_entry *rcpu,struct xdp_buff *xdp, +int cpu_map_enqueue (struct bpf_cpu_map_entry *rcpu,struct xdp_frame *xdpf,struct net_device *dev_rx);int cpu_map_generic_redirect (struct bpf_cpu_map_entry *rcpu,struct sk_buff *skb);
@@ -1813,26 +1813,26 @@ static inline void __dev_flush(void) { } -struct xdp_buff ; +struct xdp_frame ;struct bpf_dtab_netdev ;struct bpf_cpu_map_entry ;static inline -int dev_xdp_enqueue (struct net_device *dev,struct xdp_buff *xdp, +int dev_xdp_enqueue (struct net_device *dev,struct xdp_frame *xdpf,struct net_device *dev_rx) {return 0 ; }static inline -int dev_map_enqueue (struct bpf_dtab_netdev *dst,struct xdp_buff *xdp, +int dev_map_enqueue (struct bpf_dtab_netdev *dst,struct xdp_frame *xdpf,struct net_device *dev_rx) {return 0 ; }static inline -int dev_map_enqueue_multi (struct xdp_buff *xdp,struct net_device *dev_rx, +int dev_map_enqueue_multi (struct xdp_frame *xdpf,struct net_device *dev_rx,struct bpf_map *map,bool exclude_ingress) {return 0 ;
@@ -1860,7 +1860,7 @@ static inline void __cpu_map_flush(void) }static inline int cpu_map_enqueue (struct bpf_cpu_map_entry *rcpu, -struct xdp_buff *xdp, +struct xdp_frame *xdpf,struct net_device *dev_rx) {return 0 ;
kernel/bpf/cpumap.c +1 -7
@@ -746,15 +746,9 @@ static void bq_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf)list_add (&bq->flush_node, flush_list); } -int cpu_map_enqueue (struct bpf_cpu_map_entry *rcpu,struct xdp_buff *xdp, +int cpu_map_enqueue (struct bpf_cpu_map_entry *rcpu,struct xdp_frame *xdpf,struct net_device *dev_rx) { -struct xdp_frame *xdpf; - - xdpf =xdp_convert_buff_to_frame (xdp); -if (unlikely (!xdpf)) -return -EOVERFLOW ; -/* Info needed when constructing SKB on remote CPU */ xdpf->dev_rx = dev_rx;
kernel/bpf/devmap.c +11 -21
@@ -467,24 +467,19 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, bq->q[bq->count++] = xdpf; } -static inline int __xdp_enqueue (struct net_device *dev,struct xdp_buff *xdp, +static inline int __xdp_enqueue (struct net_device *dev,struct xdp_frame *xdpf,struct net_device *dev_rx,struct bpf_prog *xdp_prog) { -struct xdp_frame *xdpf;int err;if (!dev->netdev_ops->ndo_xdp_xmit)return -EOPNOTSUPP ; - err =xdp_ok_fwd_dev (dev, xdp->data_end - xdp->data); + err =xdp_ok_fwd_dev (dev, xdpf->len);if (unlikely (err))return err; - xdpf =xdp_convert_buff_to_frame (xdp); -if (unlikely (!xdpf)) -return -EOVERFLOW ; -bq_enqueue (dev, xdpf, dev_rx, xdp_prog);return 0 ; }
@@ -520,27 +515,27 @@ static u32 dev_map_bpf_prog_run_skb(struct sk_buff *skb, struct bpf_dtab_netdevreturn act; } -int dev_xdp_enqueue (struct net_device *dev,struct xdp_buff *xdp, +int dev_xdp_enqueue (struct net_device *dev,struct xdp_frame *xdpf,struct net_device *dev_rx) { -return __xdp_enqueue (dev, xdp, dev_rx,NULL ); +return __xdp_enqueue (dev, xdpf, dev_rx,NULL ); } -int dev_map_enqueue (struct bpf_dtab_netdev *dst,struct xdp_buff *xdp, +int dev_map_enqueue (struct bpf_dtab_netdev *dst,struct xdp_frame *xdpf,struct net_device *dev_rx) {struct net_device *dev = dst->dev; -return __xdp_enqueue (dev, xdp, dev_rx, dst->xdp_prog); +return __xdp_enqueue (dev, xdpf, dev_rx, dst->xdp_prog); } -static bool is_valid_dst (struct bpf_dtab_netdev *obj,struct xdp_buff *xdp) +static bool is_valid_dst (struct bpf_dtab_netdev *obj,struct xdp_frame *xdpf) {if (!obj || !obj->dev->netdev_ops->ndo_xdp_xmit)return false; -if (xdp_ok_fwd_dev (obj->dev, xdp->data_end - xdp->data)) +if (xdp_ok_fwd_dev (obj->dev, xdpf->len))return false;return true;
@@ -586,14 +581,13 @@ static int get_upper_ifindexes(struct net_device *dev, int *indexes)return n; } -int dev_map_enqueue_multi (struct xdp_buff *xdp,struct net_device *dev_rx, +int dev_map_enqueue_multi (struct xdp_frame *xdpf,struct net_device *dev_rx,struct bpf_map *map,bool exclude_ingress) { struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);struct bpf_dtab_netdev *dst, *last_dst =NULL ;int excluded_devices[1 +MAX_NEST_DEV ];struct hlist_head *head; -struct xdp_frame *xdpf;int num_excluded =0 ;unsigned int i;int err;
@@ -603,15 +597,11 @@ int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, excluded_devices[num_excluded++] = dev_rx->ifindex; } - xdpf =xdp_convert_buff_to_frame (xdp); -if (unlikely (!xdpf)) -return -EOVERFLOW ; -if (map->map_type ==BPF_MAP_TYPE_DEVMAP ) {for (i =0 ; i < map->max_entries; i++) { dst =rcu_dereference_check (dtab->netdev_map[i],rcu_read_lock_bh_held ()); -if (!is_valid_dst (dst, xdp)) +if (!is_valid_dst (dst, xdpf))continue ;if (is_ifindex_excluded (excluded_devices, num_excluded, dst->dev->ifindex))
@@ -634,7 +624,7 @@ int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, head =dev_map_index_hash (dtab, i);hlist_for_each_entry_rcu (dst, head, index_hlist,lockdep_is_held (&dtab->index_lock)) { -if (!is_valid_dst (dst, xdp)) +if (!is_valid_dst (dst, xdpf))continue ; if (is_ifindex_excluded(excluded_devices, num_excluded,
net/core/filter.c +17 -7
@@ -3964,12 +3964,24 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp,enum bpf_map_type map_type = ri->map_type;void *fwd = ri->tgt_value;u32 map_id = ri->map_id; +struct xdp_frame *xdpf;struct bpf_map *map;int err; ri->map_id =0 ;/* Valid map id idr range: [1,INT_MAX[ */ ri->map_type =BPF_MAP_TYPE_UNSPEC ; +if (map_type ==BPF_MAP_TYPE_XSKMAP ) { + err =__xsk_map_redirect (fwd, xdp); + goto out; + } + + xdpf =xdp_convert_buff_to_frame (xdp); +if (unlikely (!xdpf)) { + err = -EOVERFLOW ; + goto err; + } +switch (map_type) {case BPF_MAP_TYPE_DEVMAP : fallthrough;
@@ -3977,17 +3989,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, map =READ_ONCE (ri->map);if (unlikely (map)) {WRITE_ONCE (ri->map,NULL ); - err =dev_map_enqueue_multi (xdp, dev, map, + err =dev_map_enqueue_multi (xdpf, dev, map, ri->flags &BPF_F_EXCLUDE_INGRESS ); }else { - err =dev_map_enqueue (fwd, xdp, dev); + err =dev_map_enqueue (fwd, xdpf, dev); }break ;case BPF_MAP_TYPE_CPUMAP : - err =cpu_map_enqueue (fwd, xdp, dev); -break ; -case BPF_MAP_TYPE_XSKMAP : - err =__xsk_map_redirect (fwd, xdp); + err =cpu_map_enqueue (fwd, xdpf, dev);break ;case BPF_MAP_TYPE_UNSPEC :if (map_id ==INT_MAX ) {
@@ -3996,7 +4005,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = -EINVAL ;break ; } - err =dev_xdp_enqueue (fwd, xdp, dev); + err =dev_xdp_enqueue (fwd, xdpf, dev);break ; } fallthrough;
@@ -4004,6 +4013,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = -EBADRQC ; } +out:if (unlikely (err)) goto err;
Signature · 1 line
2.34.1
PATCH v4 5/7 xdp: add xdp_do_redirect_frame() for pre-computed xdp_frames
In reply to Toke Høiland-JørgensenAdd an xdp_do_redirect_frame() variant which supports pre-computed xdp_frame structures. This will be used in bpf_prog_run() to avoid having to write to the xdp_frame structure when the XDP program doesn't modify the frame boundaries.
--- include/linux/filter.h | 4 +++ net/core/filter.c | 65 +++++++++++++++++++++++++++++++++++------- 2 files changed, 58 insertions(+), 11 deletions(-)
include/linux/filter.h +4 -0
@@ -1019,6 +1019,10 @@ int xdp_do_generic_redirect(struct net_device *dev, struct sk_buff *skb,int xdp_do_redirect (struct net_device *dev,struct xdp_buff *xdp,struct bpf_prog *prog); +int xdp_do_redirect_frame (struct net_device *dev, +struct xdp_buff *xdp, +struct xdp_frame *xdpf, +struct bpf_prog *prog);void xdp_do_flush (void );/* The xdp_do_flush_map() helper has been renamed to drop the _map suffix, as
net/core/filter.c +54 -11
95 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v4 6/7 bpf: Add "live packet" mode for XDP in bpf_prog_run()
In reply to Toke Høiland-JørgensenThis adds support for running XDP programs through bpf_prog_run() in a mode that enables live packet processing of the resulting frames. Previous uses of bpf_prog_run() for XDP returned the XDP program return code and the modified packet data to userspace, which is useful for unit testing of XDP programs.
This patch adds a new mode with different semantics. When enabled through the new BPF_F_TEST_XDP_LIVE_FRAMES flag, the XDP program return codes will be honoured: returning XDP_PASS will result in the frame being injected into the networking stack as if it came from the selected networking interface, while returning XDP_TX and XDP_REDIRECT will result in the frame being transmitted out an egress interface. XDP_TX is translated into an XDP_REDIRECT operation, since the real XDP_TX action is only possible from within the network drivers themselves, not from the process context where bpf_prog_run() is executed.
To achieve this new mode of operation, we create a page pool instance while setting up the test run, and feed pages from that into the XDP program. The setup cost of this is amortised over the number of repetitions specified by userspace.
To support performance testing use case, we further optimise the setup step so that all pages in the pool are pre-initialised with the packet data, and pre-computed context and xdp_frame objects stored at the start of each page. This makes it possible to entirely avoid touching the page content on each XDP program invocation, and enables sending up to 11.5 Mpps/core on my test box.
Because the data pages are recycled by the page pool, and the test runner doesn't re-initialise them for each run, subsequent invocations of the XDP program will see the packet data in the state it was after the last time it ran on that particular page. This means that an XDP program that modifies the packet before redirecting it has to be careful about which assumptions it makes about the packet content, but that is only an issue for the most naively written programs.
Enabling the new flag is only allowed when not setting ctx_out and data_out in the test specification, since using it means frames will be redirected somewhere else, so they can't be returned.
--- include/uapi/linux/bpf.h | 2 + kernel/bpf/Kconfig | 1 + net/bpf/test_run.c | 287 ++++++++++++++++++++++++++++++++- tools/include/uapi/linux/bpf.h | 2 + 4 files changed, 284 insertions(+), 8 deletions(-)
include/uapi/linux/bpf.h +2 -0
@@ -1225,6 +1225,8 @@ enum {/* If set, run the test on the cpu specified by bpf_attr.test.cpu */ #define BPF_F_TEST_RUN_ON_CPU (1U << 0) +/* If set, XDP frames will be transmitted after processing */ +#define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1)/* type for BPF_ENABLE_STATS */ enum bpf_stats_type {
kernel/bpf/Kconfig +1 -0
@@ -30,6 +30,7 @@ config BPF_SYSCALL select TASKS_TRACE_RCU select BINARY_PRINTF select NET_SOCK_MSG if NET + select PAGE_POOL if NET default n help Enable the bpf() system call that allows to manipulate BPF programs
net/bpf/test_run.c +279 -8
355 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +2 -0
@@ -1225,6 +1225,8 @@ enum {/* If set, run the test on the cpu specified by bpf_attr.test.cpu */ #define BPF_F_TEST_RUN_ON_CPU (1U << 0) +/* If set, XDP frames will be transmitted after processing */ +#define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1)/* type for BPF_ENABLE_STATS */ enum bpf_stats_type {
Signature · 1 line
2.34.1
PATCH v4 7/7 selftests/bpf: Add selftest for XDP_REDIRECT in bpf_prog_run()
In reply to Toke Høiland-JørgensenThis adds a selftest for the XDP_REDIRECT facility in bpf_prog_run, that redirects packets into a veth and counts them using an XDP program on the other side of the veth pair.
--- .../bpf/prog_tests/xdp_do_redirect.c | 118 ++++++++++++++++++ .../bpf/progs/test_xdp_do_redirect.c | 39 ++++++ 2 files changed, 157 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c
tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c new file +118 -0
@@ -0,0 +1,118 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <test_progs.h> +#include <network_helpers.h> +#include <net/if.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ipv6.h> +#include <linux/in6.h> +#include <linux/udp.h> +#include <bpf/bpf_endian.h> +#include "test_xdp_do_redirect.skel.h" + +#define SYS (fmt, ...) \ + ({ \ + char cmd[1024]; \ + snprintf(cmd, sizeof(cmd), fmt, ##__VA_ARGS__); \ + if (!ASSERT_OK(system(cmd), cmd)) \ + goto fail; \ + }) + +struct udp_packet { +struct ethhdr eth; +struct ipv6hdr iph; +struct udphdr udp; +__u8 payload[64 -sizeof (struct udphdr ) + -sizeof (struct ethhdr ) -sizeof (struct ipv6hdr )]; +} __packed; + +static struct udp_packet pkt_udp = { + .eth.h_proto =__bpf_constant_htons (ETH_P_IPV6 ), + .eth.h_dest = {0x00 ,0x11 ,0x22 ,0x33 ,0x44 ,0x55 }, + .eth.h_source = {0x66 ,0x77 ,0x88 ,0x99 ,0xaa ,0xbb }, + .iph.version =6 , + .iph.nexthdr =IPPROTO_UDP , + .iph.payload_len =bpf_htons (sizeof (struct udp_packet ) + - offsetof(struct udp_packet , udp)), + .iph.hop_limit =2 , + .iph.saddr.s6_addr16 = {bpf_htons (0xfc00 ),0 ,0 ,0 ,0 ,0 ,0 ,bpf_htons (1 )}, + .iph.daddr.s6_addr16 = {bpf_htons (0xfc00 ),0 ,0 ,0 ,0 ,0 ,0 ,bpf_htons (2 )}, + .udp.source =bpf_htons (1 ), + .udp.dest =bpf_htons (1 ), + .udp.len =bpf_htons (sizeof (struct udp_packet ) + - offsetof(struct udp_packet , udp)), + .payload = {0x42 },/* receiver XDP program matches on this */ +}; + +#define NUM_PKTS 3 +void test_xdp_do_redirect (void ) +{ +struct test_xdp_do_redirect *skel =NULL ; +struct xdp_md ctx_in = { .data_end =sizeof (pkt_udp) }; +DECLARE_LIBBPF_OPTS (bpf_test_run_opts, opts, + .data_in = &pkt_udp, + .data_size_in = sizeof(pkt_udp), + .ctx_in = &ctx_in, + .ctx_size_in = sizeof(ctx_in), + .flags = BPF_F_TEST_XDP_LIVE_FRAMES, + .repeat = NUM_PKTS, + ); +int err, prog_fd, ifindex_src, ifindex_dst; +struct bpf_link *link; + + skel =test_xdp_do_redirect__open (); +if (!ASSERT_OK_PTR (skel,"skel" )) +return ; + +/* We setup a veth pair that we can not only XDP_REDIRECT packets +* between, but also route them. The test packet (defined above) has +* address information so it will be routed back out the same interface +* after it has been received, which will allow it to be picked up by +* the XDP program on the destination interface. +* +* The XDP program we run with bpf_prog_run() will cycle through all +* four return codes (DROP/PASS/TX/REDIRECT), so we should end up with +* NUM_PKTS - 1 packets seen on the dst iface. We match the packets on +* the UDP payload. +*/ +SYS ("ip link add veth_src type veth peer name veth_dst" ); +SYS ("ip link set dev veth_src address 00:11:22:33:44:55" ); +SYS ("ip link set dev veth_dst address 66:77:88:99:aa:bb" ); +SYS ("ip link set dev veth_src up" ); +SYS ("ip link set dev veth_dst up" ); +SYS ("ip addr add dev veth_src fc00::1/64" ); +SYS ("ip addr add dev veth_dst fc00::2/64" ); +SYS ("ip neigh add fc00::2 dev veth_src lladdr 66:77:88:99:aa:bb" ); +SYS ("sysctl -w net.ipv6.conf.all.forwarding=1" ); + + ifindex_src =if_nametoindex ("veth_src" ); + ifindex_dst =if_nametoindex ("veth_dst" ); +if (!ASSERT_NEQ (ifindex_src,0 ,"ifindex_src" ) || + !ASSERT_NEQ (ifindex_dst,0 ,"ifindex_dst" )) + goto fail; + +memcpy (skel->rodata->expect_dst, &pkt_udp.eth.h_dest,ETH_ALEN ); + skel->rodata->ifindex_out = ifindex_src; + ctx_in.ingress_ifindex = ifindex_src; + +if (!ASSERT_OK (test_xdp_do_redirect__load (skel),"load" )) + goto fail; + + link =bpf_program__attach_xdp (skel->progs.xdp_count_pkts, ifindex_dst); +if (!ASSERT_OK_PTR (link,"prog_attach" )) + goto fail; + skel->links.xdp_count_pkts = link; + + prog_fd =bpf_program__fd (skel->progs.xdp_redirect_notouch); + err =bpf_prog_test_run_opts (prog_fd, &opts); +if (!ASSERT_OK (err,"prog_run" )) + goto fail; + +/* wait for the packets to be flushed */ +kern_sync_rcu (); + +ASSERT_EQ (skel->bss->pkts_seen,NUM_PKTS -1 ,"pkt_count" ); +fail: +system ("ip link del dev veth_src" ); +test_xdp_do_redirect__destroy (skel); +}
tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> + +#define ETH_ALEN 6 +const volatile int ifindex_out; +const volatile __u8 expect_dst[ETH_ALEN ]; +volatile int pkts_seen =0 ; +volatile int retcode =XDP_DROP ; + +SEC("xdp") +int xdp_redirect_notouch (struct xdp_md *xdp) +{ +if (retcode ==XDP_REDIRECT ) +bpf_redirect (ifindex_out,0 ); +return retcode++; +} + +SEC("xdp") +int xdp_count_pkts (struct xdp_md *xdp) +{ +void *data = (void *)(long )xdp->data; +void *data_end = (void *)(long )xdp->data_end; +struct ethhdr *eth = data; +struct ipv6hdr *iph = (void *)(eth +1 ); +struct udphdr *udp = (void *)(iph +1 ); +__u8 *payload = (void *)(udp +1 ); +int i; + +if (payload +1 > data_end) +return XDP_ABORTED ; + +if (iph->nexthdr ==IPPROTO_UDP && *payload ==0x42 ) + pkts_seen++; + +return XDP_PASS ; +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.34.1