Generic XDP improvements
Subsystems (9)
Kumar Kartikeya Dwivedi
maintainerv3 cover Generic XDP improvements
This small series makes some improvements to generic XDP mode and brings it closer to native XDP. Patch 1 splits out generic XDP processing into reusable parts, patch 2 adds pointer friendly wrappers for bitops (not have to cast back and forth the address of local pointer to unsigned long *), patch 3 implements generic cpumap support (details in commit) and patch 4 allows devmap bpf prog execution before generic_xdp_tx is called.
Patch 5 just updates a couple of selftests to adapt to changes in behavior (in that specifying devmap/cpumap prog fd in generic mode is now allowed).
Changelog:
----------
v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]
* list_for_each_entry -> list_for_each_entry_safe (due to deletion of skb)
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Move __ptr_{set,clear,test}_bit to bitops.h (Toke)
Also changed argument order to match the bit op they wrap.
* Remove map value size checking functions for cpumap/devmap (Toke)
* Rework prog run for skb in cpu_map_kthread_run (Toke)
* Set skb->dev to dst->dev after devmap prog has run
* Don't set xdp rxq that will be overwritten in cpumap prog run
Kumar Kartikeya Dwivedi (5):
net: core: split out code to run generic XDP prog
bitops: add non-atomic bitops for pointers
bpf: cpumap: implement generic cpumap
bpf: devmap: implement devmap prog execution for generic XDP
bpf: update XDP selftests to not fail with generic XDP
include/linux/bitops.h | 19 +++ include/linux/bpf.h | 10 +- include/linux/netdevice.h | 2 + include/linux/skbuff.h | 10 +- include/linux/typecheck.h | 10 ++ kernel/bpf/cpumap.c | 115 +++++++++++++++--- kernel/bpf/devmap.c | 49 ++++++-- net/core/dev.c | 103 ++++++++-------- net/core/filter.c | 6 +- .../bpf/prog_tests/xdp_cpumap_attach.c | 4 +- .../bpf/prog_tests/xdp_devmap_attach.c | 4 +- 11 files changed, 241 insertions(+), 91 deletions(-)
Signature · 1 line
2.31.1
Kumar Kartikeya Dwivedi
maintainerPATCH v3 1/5 net: core: split out code to run generic XDP prog
This helper can later be utilized in code that runs cpumap and devmap programs in generic redirect mode and adjust skb based on changes made to xdp_buff.
When returning XDP_REDIRECT/XDP_TX, it invokes __skb_push, so whenever a generic redirect path invokes devmap/cpumap prog if set, it must __skb_pull again as we expect mac header to be pulled.
It also drops the skb_reset_mac_len call after do_xdp_generic, as the mac_header and network_header are advanced by the same offset, so the difference (mac_len) remains constant.
--- include/linux/netdevice.h | 2 + net/core/dev.c | 84 ++++++++++++++++++++++++--------------- 2 files changed, 55 insertions(+), 31 deletions(-)
include/linux/netdevice.h +2 -0
@@ -3984,6 +3984,8 @@ static inline void dev_consume_skb_any(struct sk_buff *skb)__dev_kfree_skb_any (skb,SKB_REASON_CONSUMED ); } +u32 bpf_prog_run_generic_xdp (struct sk_buff *skb,struct xdp_buff *xdp, +struct bpf_prog *xdp_prog);void generic_xdp_tx (struct sk_buff *skb,struct bpf_prog *xdp_prog);int do_xdp_generic (struct bpf_prog *xdp_prog,struct sk_buff *skb);int netif_rx (struct sk_buff *skb);
net/core/dev.c +53 -31
117 lines not on this page · raw message
Signature · 1 line
2.31.1
Kumar Kartikeya Dwivedi
maintainerPATCH v3 2/5 bitops: add non-atomic bitops for pointers
In reply to Kumar Kartikeya Dwivedicpumap needs to set, clear, and test the lowest bit in skb pointer in various places. To make these checks less noisy, add pointer friendly bitop macros that also do some typechecking to sanitize the argument.
These wrap the non-atomic bitops __set_bit, __clear_bit, and test_bit but for pointer arguments. Pointer's address has to be passed in and it is treated as an unsigned long *, since width and representation of pointer and unsigned long match on targets Linux supports. They are prefixed with double underscore to indicate lack of atomicity.
--- include/linux/bitops.h | 19 +++++++++++++++++++ include/linux/typecheck.h | 10 ++++++++++ 2 files changed, 29 insertions(+)
include/linux/bitops.h +19 -0
@@ -4,6 +4,7 @@#include <asm/types.h> #include <linux/bits.h> +#include <linux/typecheck.h> #include <uapi/linux/kernel.h>
@@ -253,6 +254,24 @@ static __always_inline void __assign_bit(long nr, volatile unsigned long *addr,__clear_bit (nr, addr); } +#define __ptr_set_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __set_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_clear_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __clear_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_test_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + test_bit(nr, (unsigned long *)(addr)); \ + }) +#ifdef __KERNEL__#ifndef set_mask_bits
include/linux/typecheck.h +10 -0
@@ -22,4 +22,14 @@ (void )__tmp; \ }) +/* +* Check at compile that something is a pointer type. +* Always evaluates to 1 so you may use it easily in comparisons. +*/ +#define typecheck_pointer (x) \ +({ typeof(x) __dummy; \ + (void)sizeof(*__dummy); \ + 1; \ +}) +#endif /* TYPECHECK_H_INCLUDED */
Signature · 1 line
2.31.1
Re: [PATCH net-next v3 2/5] bitops: add non-atomic bitops for pointers
Quoted 40 lines · Kumar Kartikeya Dwivedi
cpumap needs to set, clear, and test the lowest bit in skb pointer in various places. To make these checks less noisy, add pointer friendly bitop macros that also do some typechecking to sanitize the argument.
These wrap the non-atomic bitops
__set_bit,__clear_bit, andtest_bitbut for pointer arguments. Pointer's address has to be passed in and it is treated as an unsigned long *, since width and representation of pointer and unsigned long match on targets Linux supports. They are prefixed with double underscore to indicate lack of atomicity.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bitops.h | 19 +++++++++++++++++++ include/linux/typecheck.h | 10 ++++++++++ 2 files changed, 29 insertions(+)include/linux/bitops.h +9
-0@@ -4,6 +4,7 @@#include <asm/types.h> #include <linux/bits.h> +#include <linux/typecheck.h> #include <uapi/linux/kernel.h> @@ -253,6 +254,24 @@ static __always_inline void __assign_bit(long nr, volatile unsigned long *addr,__clear_bit (nr, addr); } +#define __ptr_set_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __set_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_clear_bit (nr, addr) \ + ({ \
include/linux/bitops.h static __always_inline void __assign_bit(long nr, volatile unsigned long *addr, · line 274
Quoted 10 lines
include/linux/bitops.h +10
-0+typecheck_pointer (*(addr)); \ +__clear_bit (nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_test_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + test_bit(nr, (unsigned long *)(addr)); \ + }) +
Before these were functions that returned the modified values, now they are macros that modify in-place. Why the change? :)
-Toke
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH net-next v3 2/5] bitops: add non-atomic bitops for pointers
Quoted 47 lines · Toke Høiland-Jørgensen
Quoted 45 lines · Kumar Kartikeya Dwivedi
cpumap needs to set, clear, and test the lowest bit in skb pointer in various places. To make these checks less noisy, add pointer friendly bitop macros that also do some typechecking to sanitize the argument.
These wrap the non-atomic bitops
__set_bit,__clear_bit, andtest_bitbut for pointer arguments. Pointer's address has to be passed in and it is treated as an unsigned long *, since width and representation of pointer and unsigned long match on targets Linux supports. They are prefixed with double underscore to indicate lack of atomicity.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bitops.h | 19 +++++++++++++++++++ include/linux/typecheck.h | 10 ++++++++++ 2 files changed, 29 insertions(+)include/linux/bitops.h +14
-0@@ -4,6 +4,7 @@#include <asm/types.h> #include <linux/bits.h> +#include <linux/typecheck.h> #include <uapi/linux/kernel.h> @@ -253,6 +254,24 @@ static __always_inline void __assign_bit(long nr, volatile unsigned long *addr,__clear_bit (nr, addr); } +#define __ptr_set_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __set_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_clear_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __clear_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_test_bit (nr, addr) \
Quoted 9 lines
include/linux/bitops.h static __always_inline void __assign_bit(long nr, volatile unsigned long *addr, · line 274
Quoted 5 lines
include/linux/bitops.h +5
-0+ ({ \ +typecheck_pointer (*(addr)); \ +test_bit (nr, (unsigned long *)(addr)); \ + }) +Before these were functions that returned the modified values, now they are macros that modify in-place. Why the change? :)
Given that we're exporting this to all kernel users now, it felt more appropriate to follow the existing convention/argument order for the functions/ops they are wrapping.
I really have no preference here though...
Quoted 2 lines
-Toke
--
Kartikeya
Re: [PATCH net-next v3 2/5] bitops: add non-atomic bitops for pointers
Quoted 58 lines · Kumar Kartikeya Dwivedi
Quoted 47 lines · Toke Høiland-Jørgensen
Quoted 45 lines · Kumar Kartikeya Dwivedi
cpumap needs to set, clear, and test the lowest bit in skb pointer in various places. To make these checks less noisy, add pointer friendly bitop macros that also do some typechecking to sanitize the argument.
These wrap the non-atomic bitops
__set_bit,__clear_bit, andtest_bitbut for pointer arguments. Pointer's address has to be passed in and it is treated as an unsigned long *, since width and representation of pointer and unsigned long match on targets Linux supports. They are prefixed with double underscore to indicate lack of atomicity.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bitops.h | 19 +++++++++++++++++++ include/linux/typecheck.h | 10 ++++++++++ 2 files changed, 29 insertions(+)include/linux/bitops.h +14
-0@@ -4,6 +4,7 @@#include <asm/types.h> #include <linux/bits.h> +#include <linux/typecheck.h> #include <uapi/linux/kernel.h> @@ -253,6 +254,24 @@ static __always_inline void __assign_bit(long nr, volatile unsigned long *addr,__clear_bit (nr, addr); } +#define __ptr_set_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __set_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_clear_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __clear_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_test_bit (nr, addr) \Quoted 9 lines
include/linux/bitops.h static __always_inline void __assign_bit(long nr, volatile unsigned long *addr, · line 274
Quoted 5 lines
include/linux/bitops.h +5
-0+ ({ \ +typecheck_pointer (*(addr)); \ +test_bit (nr, (unsigned long *)(addr)); \ + }) +Before these were functions that returned the modified values, now they are macros that modify in-place. Why the change? :)
Quoted 3 lines
Given that we're exporting this to all kernel users now, it felt more appropriate to follow the existing convention/argument order for the functions/ops they are wrapping.
I wasn't talking about the order of the arguments; swapping those is fine. But before, you had:
static void *__ptr_set_bit(void *ptr, int bit)
with usage (function return is the modified value):
ret = ptr_ring_produce(rcpu->queue, __ptr_set_bit(skb, 0));
now you have:
#define __ptr_set_bit(nr, addr)
with usage (modifies argument in-place):__ptr_set_bit(0, &skb);
ret = ptr_ring_produce(rcpu->queue, skb);
why change from function to macro?
-Toke
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH net-next v3 2/5] bitops: add non-atomic bitops for pointers
Quoted 72 lines · Toke Høiland-Jørgensen
Quoted 58 lines · Kumar Kartikeya Dwivedi
Quoted 47 lines · Toke Høiland-Jørgensen
Quoted 45 lines · Kumar Kartikeya Dwivedi
cpumap needs to set, clear, and test the lowest bit in skb pointer in various places. To make these checks less noisy, add pointer friendly bitop macros that also do some typechecking to sanitize the argument.
These wrap the non-atomic bitops
__set_bit,__clear_bit, andtest_bitbut for pointer arguments. Pointer's address has to be passed in and it is treated as an unsigned long *, since width and representation of pointer and unsigned long match on targets Linux supports. They are prefixed with double underscore to indicate lack of atomicity.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bitops.h | 19 +++++++++++++++++++ include/linux/typecheck.h | 10 ++++++++++ 2 files changed, 29 insertions(+)include/linux/bitops.h +14
-0@@ -4,6 +4,7 @@#include <asm/types.h> #include <linux/bits.h> +#include <linux/typecheck.h> #include <uapi/linux/kernel.h> @@ -253,6 +254,24 @@ static __always_inline void __assign_bit(long nr, volatile unsigned long *addr,__clear_bit (nr, addr); } +#define __ptr_set_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __set_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_clear_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __clear_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_test_bit (nr, addr) \Quoted 9 lines
include/linux/bitops.h static __always_inline void __assign_bit(long nr, volatile unsigned long *addr, · line 274
Quoted 5 lines
include/linux/bitops.h +5
-0+ ({ \ +typecheck_pointer (*(addr)); \ +test_bit (nr, (unsigned long *)(addr)); \ + }) +Before these were functions that returned the modified values, now they are macros that modify in-place. Why the change? :)
Quoted 3 lines
Given that we're exporting this to all kernel users now, it felt more appropriate to follow the existing convention/argument order for the functions/ops they are wrapping.
I wasn't talking about the order of the arguments; swapping those is fine. But before, you had:
static void *__ptr_set_bit(void *ptr, int bit)
with usage (function return is the modified value):
ret =ptr_ring_produce(rcpu->queue, __ptr_set_bit(skb, 0));
Quoted 9 lines
now you have:
#define__ptr_set_bit(nr, addr)with usage (modifies argument in-place):
__ptr_set_bit(0, &skb);
ret =ptr_ring_produce(rcpu->queue, skb);why change from function to macro?
Earlier it just took the pointer value and returned one with the bit set. I changed it to work similar to __set_bit.
So such a function modifying in place doesn't allow seeing through what the type of *addr is, it would have to take void * which would work with any pointer. It's just a little more safe (so we can be sure casting to unsigned long * is ok by inspecting the typeof(*addr) ).
Quoted 2 lines
-Toke
--
Kartikeya
Re: [PATCH net-next v3 2/5] bitops: add non-atomic bitops for pointers
Quoted 80 lines · Kumar Kartikeya Dwivedi
Quoted 79 lines · Toke Høiland-Jørgensen
Quoted 58 lines · Kumar Kartikeya Dwivedi
Quoted 47 lines · Toke Høiland-Jørgensen
Quoted 45 lines · Kumar Kartikeya Dwivedi
cpumap needs to set, clear, and test the lowest bit in skb pointer in various places. To make these checks less noisy, add pointer friendly bitop macros that also do some typechecking to sanitize the argument.
These wrap the non-atomic bitops
__set_bit,__clear_bit, andtest_bitbut for pointer arguments. Pointer's address has to be passed in and it is treated as an unsigned long *, since width and representation of pointer and unsigned long match on targets Linux supports. They are prefixed with double underscore to indicate lack of atomicity.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bitops.h | 19 +++++++++++++++++++ include/linux/typecheck.h | 10 ++++++++++ 2 files changed, 29 insertions(+)include/linux/bitops.h +14
-0@@ -4,6 +4,7 @@#include <asm/types.h> #include <linux/bits.h> +#include <linux/typecheck.h> #include <uapi/linux/kernel.h> @@ -253,6 +254,24 @@ static __always_inline void __assign_bit(long nr, volatile unsigned long *addr,__clear_bit (nr, addr); } +#define __ptr_set_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __set_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_clear_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __clear_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_test_bit (nr, addr) \Quoted 9 lines
include/linux/bitops.h static __always_inline void __assign_bit(long nr, volatile unsigned long *addr, · line 274
Quoted 5 lines
include/linux/bitops.h +5
-0+ ({ \ +typecheck_pointer (*(addr)); \ +test_bit (nr, (unsigned long *)(addr)); \ + }) +Before these were functions that returned the modified values, now they are macros that modify in-place. Why the change? :)
Quoted 3 lines
Given that we're exporting this to all kernel users now, it felt more appropriate to follow the existing convention/argument order for the functions/ops they are wrapping.
I wasn't talking about the order of the arguments; swapping those is fine. But before, you had:
static void *__ptr_set_bit(void *ptr, int bit)
with usage (function return is the modified value):
ret =ptr_ring_produce(rcpu->queue, __ptr_set_bit(skb, 0));now you have:
#define__ptr_set_bit(nr, addr)with usage (modifies argument in-place):
__ptr_set_bit(0, &skb);
ret =ptr_ring_produce(rcpu->queue, skb);
Quoted 5 lines
Quoted 2 lines
why change from function to macro?
Earlier it just took the pointer value and returned one with the bit set. I changed it to work similar to
__set_bit.
Hmm, okay, fair enough I suppose there's something to be said for consistency, even though I personally prefer the function style. Let's keep it as macros, then :)
-Toke
RE: [PATCH net-next v3 2/5] bitops: add non-atomic bitops for pointers
From: Toke Høiland-Jørgensen
Quoted 87 lines
Sent: 23 June 2021 12:09
Quoted 28 lines · Kumar Kartikeya Dwivedi
Quoted 27 lines · Toke Høiland-Jørgensen
Quoted 25 lines · Kumar Kartikeya Dwivedi
Quoted 24 lines · Toke Høiland-Jørgensen
Quoted 22 lines · Kumar Kartikeya Dwivedi
cpumap needs to set, clear, and test the lowest bit in skb pointer in various places. To make these checks less noisy, add pointer friendly bitop macros that also do some typechecking to sanitize the argument.
These wrap the non-atomic bitops
__set_bit,__clear_bit, andtest_bitbut for pointer arguments. Pointer's address has to be passed in and it is treated as an unsigned long *, since width and representation of pointer and unsigned long match on targets Linux supports. They are prefixed with double underscore to indicate lack of atomicity.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bitops.h | 19 +++++++++++++++++++ include/linux/typecheck.h | 10 ++++++++++ 2 files changed, 29 insertions(+)include/linux/bitops.h +0
-0@@ -4,6 +4,7 @@Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
include/linux/bitops.h static __always_inline void __assign_bit(long nr, volatile unsigned long
Quoted 7 lines
include/linux/bitops.h +1
-0#include <asm/types.h> #include <linux/bits.h> +#include <linux/typecheck.h> #include <uapi/linux/kernel.h> @@ -253,6 +254,24 @@ static __always_inline void __assign_bit(long nr, volatile unsigned long*addr,
Quoted 48 lines
Quoted 47 lines
Quoted 26 lines
Quoted 16 lines
Quoted 16 lines
include/linux/bitops.h +13
-0__clear_bit (nr, addr); } +#define __ptr_set_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __set_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_clear_bit (nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __clear_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_test_bit (nr, addr) \Quoted 9 lines
include/linux/bitops.h static __always_inline void __assign_bit(long nr, volatile unsigned long *addr, · line 274
Quoted 5 lines
include/linux/bitops.h +5
-0+ ({ \ +typecheck_pointer (*(addr)); \ +test_bit (nr, (unsigned long *)(addr)); \ + }) +Before these were functions that returned the modified values, now they are macros that modify in-place. Why the change? :)
Quoted 3 lines
Given that we're exporting this to all kernel users now, it felt more appropriate to follow the existing convention/argument order for the functions/ops they are wrapping.
I wasn't talking about the order of the arguments; swapping those is fine. But before, you had:
static void *__ptr_set_bit(void *ptr, int bit)
with usage (function return is the modified value):
ret =ptr_ring_produce(rcpu->queue, __ptr_set_bit(skb, 0));now you have:
#define__ptr_set_bit(nr, addr)with usage (modifies argument in-place):
__ptr_set_bit(0, &skb);
ret =ptr_ring_produce(rcpu->queue, skb);why change from function to macro?
Quoted 6 lines
Quoted 2 lines
Earlier it just took the pointer value and returned one with the bit set. I changed it to work similar to
__set_bit.Hmm, okay, fair enough I suppose there's something to be said for consistency, even though I personally prefer the function style. Let's keep it as macros, then :)
Passing the address of the pointer will trash a lot of optimisations. You do really want to use the return address.
Or, even better, get the whole thing inlined.
So something like:
#define ptr_set_bit(ptr, val) ((typeof (ptr))((unsigned long)(ptr) | (1 << (val))))
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Kumar Kartikeya Dwivedi
maintainerPATCH v3 3/5 bpf: cpumap: implement generic cpumap
In reply to Kumar Kartikeya DwivediThis change implements CPUMAP redirect support for generic XDP programs. The idea is to reuse the cpu map entry's queue that is used to push native xdp frames for redirecting skb to a different CPU. This will match native XDP behavior (in that RPS is invoked again for packet reinjected into networking stack).
To be able to determine whether the incoming skb is from the driver or cpumap, we reuse skb->redirected bit that skips generic XDP processing when it is set. To always make use of this, CONFIG_NET_REDIRECT guard on it has been lifted and it is always available.
From the redirect side, we add the skb to ptr_ring with its lowest bit set to 1. This should be safe as skb is not 1-byte aligned. This allows kthread to discern between xdp_frames and sk_buff. On consumption of the ptr_ring item, the lowest bit is unset.
In the end, the skb is simply added to the list that kthread is anyway going to maintain for xdp_frames converted to skb, and then received again by using netif_receive_skb_list.
Bulking optimization for generic cpumap is left as an exercise for a future patch for now.
Since cpumap entry progs are now supported, also remove check in generic_xdp_install for the cpumap.
--- include/linux/bpf.h | 9 +++- include/linux/skbuff.h | 10 +--- kernel/bpf/cpumap.c | 115 +++++++++++++++++++++++++++++++++++------ net/core/dev.c | 3 +- net/core/filter.c | 6 ++- 5 files changed, 115 insertions(+), 28 deletions(-)
include/linux/bpf.h +8 -1
@@ -1513,7 +1513,8 @@ bool dev_map_can_have_prog(struct bpf_map *map);void __cpu_map_flush (void );int cpu_map_enqueue (struct bpf_cpu_map_entry *rcpu,struct xdp_buff *xdp,struct net_device *dev_rx); -bool cpu_map_prog_allowed (struct bpf_map *map); +int cpu_map_generic_redirect (struct bpf_cpu_map_entry *rcpu, +struct sk_buff *skb);/* Return map's numa specified by userspace */ static inline int bpf_map_attr_numa_node (const union bpf_attr *attr)
@@ -1710,6 +1711,12 @@ static inline int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu,return 0 ; } +static inline int cpu_map_generic_redirect (struct bpf_cpu_map_entry *rcpu, +struct sk_buff *skb) +{ +return -EOPNOTSUPP ; +} +static inline bool cpu_map_prog_allowed (struct bpf_map *map) {return false;
include/linux/skbuff.h +2 -8
@@ -863,8 +863,8 @@ struct sk_buff { __u8 tc_skip_classify:1; __u8 tc_at_ingress:1;#endif -#ifdef CONFIG_NET_REDIRECT __u8 redirected:1; +#ifdef CONFIG_NET_REDIRECT __u8 from_ingress:1;#endif #ifdef CONFIG_TLS_DEVICE
@@ -4664,17 +4664,13 @@ static inline __wsum lco_csum(struct sk_buff *skb)static inline bool skb_is_redirected (const struct sk_buff *skb) { -#ifdef CONFIG_NET_REDIRECT return skb->redirected; -#else -return false; -#endif }static inline void skb_set_redirected (struct sk_buff *skb,bool from_ingress) { -#ifdef CONFIG_NET_REDIRECT skb->redirected =1 ; +#ifdef CONFIG_NET_REDIRECT skb->from_ingress = from_ingress;if (skb->from_ingress) skb->tstamp =0 ;
@@ -4683,9 +4679,7 @@ static inline void skb_set_redirected(struct sk_buff *skb, bool from_ingress)static inline void skb_reset_redirect (struct sk_buff *skb) { -#ifdef CONFIG_NET_REDIRECT skb->redirected =0 ; -#endif }static inline bool skb_csum_is_sctp (struct sk_buff *skb)
kernel/bpf/cpumap.c +99 -16
179 lines not on this page · raw message
net/core/dev.c +1 -2
@@ -5642,8 +5642,7 @@ static int generic_xdp_install(struct net_device *dev, struct netdev_bpf *xdp)* have a bpf_prog installed on an entry */ for (i =0 ; i < new->aux->used_map_cnt; i++) { -if (dev_map_can_have_prog (new->aux->used_maps[i]) || -cpu_map_prog_allowed (new->aux->used_maps[i])) { +if (dev_map_can_have_prog (new->aux->used_maps[i])) {mutex_unlock (&new->aux->used_maps_mutex);return -EINVAL ; }
net/core/filter.c +5 -1
@@ -4038,8 +4038,12 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, goto err;consume_skb (skb);break ; +case BPF_MAP_TYPE_CPUMAP : + err =cpu_map_generic_redirect (fwd, skb); +if (unlikely (err)) + goto err; +break ;default : -/* TODO: Handle BPF_MAP_TYPE_CPUMAP */ err = -EBADRQC ; goto err; }
Signature · 1 line
2.31.1
Kumar Kartikeya Dwivedi
maintainerPATCH v3 4/5 bpf: devmap: implement devmap prog execution for generic XDP
In reply to Kumar Kartikeya DwivediThis lifts the restriction on running devmap BPF progs in generic redirect mode. To match native XDP behavior, it is invoked right before generic_xdp_tx is called, and only supports XDP_PASS/XDP_ABORTED/ XDP_DROP actions.
We also return 0 even if devmap program drops the packet, as semantically redirect has already succeeded and the devmap prog is the last point before TX of the packet to device where it can deliver a verdict on the packet.
This also means it must take care of freeing the skb, as xdp_do_generic_redirect callers only do that in case an error is returned.
Since devmap entry prog is supported, remove the check in generic_xdp_install entirely.
--- include/linux/bpf.h | 1 - kernel/bpf/devmap.c | 49 ++++++++++++++++++++++++++++++++++++--------- net/core/dev.c | 18 ----------------- 3 files changed, 39 insertions(+), 29 deletions(-)
include/linux/bpf.h +0 -1
@@ -1508,7 +1508,6 @@ int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb,int dev_map_redirect_multi (struct net_device *dev,struct sk_buff *skb,struct bpf_prog *xdp_prog,struct bpf_map *map,bool exclude_ingress); -bool dev_map_can_have_prog (struct bpf_map *map);void __cpu_map_flush (void ); int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp,
kernel/bpf/devmap.c +39 -10
67 lines not on this page · raw message
net/core/dev.c +0 -18
@@ -5633,24 +5633,6 @@ static int generic_xdp_install(struct net_device *dev, struct netdev_bpf *xdp)struct bpf_prog *new = xdp->prog;int ret =0 ; -if (new) { -u32 i; - -mutex_lock (&new->aux->used_maps_mutex); - -/* generic XDP does not work with DEVMAPs that can -* have a bpf_prog installed on an entry -*/ -for (i =0 ; i < new->aux->used_map_cnt; i++) { -if (dev_map_can_have_prog (new->aux->used_maps[i])) { -mutex_unlock (&new->aux->used_maps_mutex); -return -EINVAL ; - } - } - -mutex_unlock (&new->aux->used_maps_mutex); - } -switch (xdp->command) {case XDP_SETUP_PROG :rcu_assign_pointer (dev->xdp_prog, new);
Signature · 1 line
2.31.1
Kumar Kartikeya Dwivedi
maintainerPATCH v3 5/5 bpf: update XDP selftests to not fail with generic XDP
In reply to Kumar Kartikeya DwivediGeneric XDP devmaps and cpumaps now allow setting value_size to 8 bytes (so that prog_fd can be specified) and XDP progs using them succeed in SKB mode now. Adjust the checks.
--- tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c | 4 ++-- tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c +2 -2
@@ -29,8 +29,8 @@ void test_xdp_with_cpumap_helpers(void)*/ prog_fd =bpf_program__fd (skel->progs.xdp_redir_prog); err =bpf_set_link_xdp_fd (IFINDEX_LO , prog_fd,XDP_FLAGS_SKB_MODE ); -CHECK (err ==0 ,"Generic attach of program with 8-byte CPUMAP" , -"should have failed\n" ); +CHECK (err,"Generic attach of program with 8-byte CPUMAP" , +"shouldn't have failed\n" ); prog_fd =bpf_program__fd (skel->progs.xdp_dummy_cm); map_fd =bpf_map__fd (skel->maps.cpu_map);
tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c +2 -2
@@ -31,8 +31,8 @@ void test_xdp_with_devmap_helpers(void)*/ dm_fd =bpf_program__fd (skel->progs.xdp_redir_prog); err =bpf_set_link_xdp_fd (IFINDEX_LO , dm_fd,XDP_FLAGS_SKB_MODE ); -CHECK (err ==0 ,"Generic attach of program with 8-byte devmap" , -"should have failed\n" ); +CHECK (err,"Generic attach of program with 8-byte devmap" , +"shouldn't have failed\n" ); dm_fd =bpf_program__fd (skel->progs.xdp_dummy_dm); map_fd =bpf_map__fd (skel->maps.dm_ports);
Signature · 1 line
2.31.1
Re: [PATCH net-next v3 5/5] bpf: update XDP selftests to not fail with generic XDP
Quoted 15 lines · Kumar Kartikeya Dwivedi
Generic XDP devmaps and cpumaps now allow setting
value_sizeto 8 bytes (so thatprog_fdcan be specified) and XDP progs using them succeed in SKB mode now. Adjust the checks.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c | 4 ++-- tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c +0
-0
tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c void test_xdp_with_cpumap_helpers(void) · line 33
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c +2
-2@@ -29,8 +29,8 @@ void test_xdp_with_cpumap_helpers(void)*/ prog_fd =bpf_program__fd (skel->progs.xdp_redir_prog); err =bpf_set_link_xdp_fd (IFINDEX_LO , prog_fd,XDP_FLAGS_SKB_MODE ); -CHECK (err ==0 ,"Generic attach of program with 8-byte CPUMAP" , -"should have failed\n" ); +CHECK (err,"Generic attach of program with 8-byte CPUMAP" , +"shouldn't have failed\n" );
There's a comment right above this that is now wrong... Also, this program is never being detached.
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c +0
-0prog_fd =bpf_program__fd (skel->progs.xdp_dummy_cm); map_fd =bpf_map__fd (skel->maps.cpu_map);tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c +0
-0
tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c void test_xdp_with_devmap_helpers(void) · line 35
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c +2
-2@@ -31,8 +31,8 @@ void test_xdp_with_devmap_helpers(void)*/ dm_fd =bpf_program__fd (skel->progs.xdp_redir_prog); err =bpf_set_link_xdp_fd (IFINDEX_LO , dm_fd,XDP_FLAGS_SKB_MODE ); -CHECK (err ==0 ,"Generic attach of program with 8-byte devmap" , -"should have failed\n" ); +CHECK (err,"Generic attach of program with 8-byte devmap" , +"shouldn't have failed\n" );
... same here
-Toke