A couple of small refactorings of BPF program call sites
Subsystems (8)
v3 cover A couple of small refactorings of BPF program call sites
Stanislav suggested[0] that these small refactorings could be split out from the XDP queueing RFC series and merged separately. The first change is a small repacking of struct softnet_data, the others change the BPF call sites to support full 64-bit values as arguments to bpf_redirect_map() and as the return value of a BPF program, relying on the fact that BPF registers are always 64-bit wide to maintain backwards compatibility.
Please see the individual patches for details.
v3:
- In patch 3, don't change types of return values that are copied to userspace (which should fix selftest errors on big-endian archs like s390)
- Rebase on bpf-next
- Collect Song's ACKs
v2:
- Rebase on bpf-next (CI failure seems to be unrelated to this series)
- Collect Stanislav's Reviewed-by
[0] https://lore.kernel.org/r/CAKH8qBtdnku7StcQ-SamadvAF==DRuLLZO94yOR1WJ9Bg=uX1w@mail.gmail.com
Kumar Kartikeya Dwivedi (1):
bpf: Use 64-bit return value for bpf_prog_run
Toke Høiland-Jørgensen (2):
dev: Move received_rps counter next to RPS members in softnet data
bpf: Expand map key argument of bpf_redirect_map to u64
include/linux/bpf-cgroup.h | 12 +++++------ include/linux/bpf.h | 24 +++++++++++----------- include/linux/filter.h | 42 +++++++++++++++++++------------------- include/linux/netdevice.h | 2 +- include/uapi/linux/bpf.h | 2 +- kernel/bpf/cgroup.c | 12 +++++------ kernel/bpf/core.c | 14 ++++++------- kernel/bpf/cpumap.c | 4 ++-- kernel/bpf/devmap.c | 4 ++-- kernel/bpf/offload.c | 4 ++-- kernel/bpf/verifier.c | 2 +- net/core/filter.c | 4 ++-- net/packet/af_packet.c | 7 +++++-- net/xdp/xskmap.c | 4 ++-- 14 files changed, 70 insertions(+), 67 deletions(-)
Signature · 1 line
2.38.1
PATCH v3 1/3 dev: Move received_rps counter next to RPS members in softnet data
Move the received_rps counter value next to the other RPS-related members in softnet_data. This closes two four-byte holes in the structure, making room for another pointer in the first two cache lines without bumping the xmit struct to its own line.
--- include/linux/netdevice.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/netdevice.h +1 -1
@@ -3114,7 +3114,6 @@ struct softnet_data {/* stats */ unsigned int processed;unsigned int time_squeeze; -unsigned int received_rps;#ifdef CONFIG_RPS struct softnet_data *rps_ipi_list;#endif
@@ -3147,6 +3146,7 @@ struct softnet_data {unsigned int cpu;unsigned int input_queue_tail;#endif +unsigned int received_rps;unsigned int dropped;struct sk_buff_head input_pkt_queue;struct napi_struct backlog;
Signature · 1 line
2.38.1
PATCH v3 2/3 bpf: Expand map key argument of bpf_redirect_map to u64
In reply to Toke Høiland-JørgensenFor queueing packets in XDP we want to add a new redirect map type with support for 64-bit indexes. To prepare fore this, expand the width of the 'key' argument to the bpf_redirect_map() helper. Since BPF registers are always 64-bit, this should be safe to do after the fact.
--- include/linux/bpf.h | 2 +- include/linux/filter.h | 12 ++++++------ include/uapi/linux/bpf.h | 2 +- kernel/bpf/cpumap.c | 4 ++-- kernel/bpf/devmap.c | 4 ++-- kernel/bpf/verifier.c | 2 +- net/core/filter.c | 4 ++-- net/xdp/xskmap.c | 4 ++-- 8 files changed, 17 insertions(+), 17 deletions(-)
include/linux/bpf.h +1 -1
@@ -135,7 +135,7 @@ struct bpf_map_ops {struct bpf_local_storage __rcu ** (*map_owner_storage_ptr)(void *owner);/* Misc helpers.*/ -int (*map_redirect)(struct bpf_map *map,u32 ifindex,u64 flags); +int (*map_redirect)(struct bpf_map *map,u64 key,u64 flags);/* map_meta_equal must be implemented for maps that can be * used as an inner map. It is a runtime check to ensure
include/linux/filter.h +6 -6
@@ -643,13 +643,13 @@ struct bpf_nh_params { };struct bpf_redirect_info { -u32 flags; -u32 tgt_index; +u64 tgt_index;void *tgt_value;struct bpf_map *map; +u32 flags; +u32 kern_flags;u32 map_id;enum bpf_map_type map_type; -u32 kern_flags;struct bpf_nh_params nh; };
@@ -1504,7 +1504,7 @@ static inline bool bpf_sk_lookup_run_v6(struct net *net, int protocol, }#endif /* IS_ENABLED(CONFIG_IPV6) */ -static __always_inline int __bpf_xdp_redirect_map(struct bpf_map *map, u32 ifindex, +static __always_inline int __bpf_xdp_redirect_map(struct bpf_map *map, u64 index,u64 flags,const u64 flag_mask,void *lookup_elem (struct bpf_map *map,u32 key)) {
@@ -1515,7 +1515,7 @@ static __always_inline int __bpf_xdp_redirect_map(struct bpf_map *map, u32 ifindif (unlikely (flags & ~(action_mask | flag_mask)))return XDP_ABORTED ; - ri->tgt_value =lookup_elem (map, ifindex); + ri->tgt_value =lookup_elem (map, index);if (unlikely (!ri->tgt_value) && !(flags &BPF_F_BROADCAST )) {/* If the lookup fails we want to clear out the state in the * redirect_info struct completely, so that if an eBPF program
@@ -1527,7 +1527,7 @@ static __always_inline int __bpf_xdp_redirect_map(struct bpf_map *map, u32 ifindreturn flags & action_mask; } - ri->tgt_index = ifindex; + ri->tgt_index = index; ri->map_id = map->id; ri->map_type = map->map_type;
include/uapi/linux/bpf.h +1 -1
@@ -2647,7 +2647,7 @@ union bpf_attr {* Return * 0 on success, or a negative error in case of failure. * -* long bpf_redirect_map(struct bpf_map *map, u32 key, u64 flags) +* long bpf_redirect_map(struct bpf_map *map, u64 key, u64 flags) * Description * Redirect the packet to the endpoint referenced by *map* at * index *key*. Depending on its type, this *map* can contain
kernel/bpf/cpumap.c +2 -2
@@ -664,9 +664,9 @@ static int cpu_map_get_next_key(struct bpf_map *map, void *key, void *next_key)return 0 ; } -static int cpu_map_redirect (struct bpf_map *map,u32 ifindex,u64 flags) +static int cpu_map_redirect (struct bpf_map *map,u64 index,u64 flags) { -return __bpf_xdp_redirect_map (map, ifindex, flags,0 , +return __bpf_xdp_redirect_map (map, index, flags,0 , __cpu_map_lookup_elem); }
kernel/bpf/devmap.c +2 -2
@@ -992,14 +992,14 @@ static int dev_map_hash_update_elem(struct bpf_map *map, void *key, void *value, map, key, value, map_flags); } -static int dev_map_redirect (struct bpf_map *map,u32 ifindex,u64 flags) +static int dev_map_redirect (struct bpf_map *map,u64 ifindex,u64 flags) {return __bpf_xdp_redirect_map (map, ifindex, flags,BPF_F_BROADCAST |BPF_F_EXCLUDE_INGRESS , __dev_map_lookup_elem); } -static int dev_hash_map_redirect (struct bpf_map *map,u32 ifindex,u64 flags) +static int dev_hash_map_redirect (struct bpf_map *map,u64 ifindex,u64 flags) { return __bpf_xdp_redirect_map(map, ifindex, flags, BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS,
kernel/bpf/verifier.c +1 -1
@@ -14366,7 +14366,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env)BUILD_BUG_ON (!__same_type (ops->map_peek_elem, (int (*)(struct bpf_map *map,void *value))NULL ));BUILD_BUG_ON (!__same_type (ops->map_redirect, - (int (*)(struct bpf_map *map,u32 ifindex,u64 flags))NULL )); + (int (*)(struct bpf_map *map,u64 index,u64 flags))NULL )); BUILD_BUG_ON(!__same_type(ops->map_for_each_callback, (int (*)(struct bpf_map *map, bpf_callback_t callback_fn,
net/core/filter.c +2 -2
@@ -4414,10 +4414,10 @@ static const struct bpf_func_proto bpf_xdp_redirect_proto = { .arg2_type = ARG_ANYTHING, }; -BPF_CALL_3(bpf_xdp_redirect_map, struct bpf_map *, map, u32, ifindex, +BPF_CALL_3(bpf_xdp_redirect_map, struct bpf_map *, map, u64, key, u64, flags) { -return map->ops->map_redirect (map, ifindex, flags); +return map->ops->map_redirect (map, key, flags); }static const struct bpf_func_proto bpf_xdp_redirect_map_proto = {
net/xdp/xskmap.c +2 -2
@@ -231,9 +231,9 @@ static int xsk_map_delete_elem(struct bpf_map *map, void *key)return 0 ; } -static int xsk_map_redirect (struct bpf_map *map,u32 ifindex,u64 flags) +static int xsk_map_redirect (struct bpf_map *map,u64 index,u64 flags) { -return __bpf_xdp_redirect_map (map, ifindex, flags,0 , +return __bpf_xdp_redirect_map (map, index, flags,0 , __xsk_map_lookup_elem); }
Signature · 1 line
2.38.1
PATCH v3 3/3 bpf: Use 64-bit return value for bpf_prog_run
In reply to Toke Høiland-JørgensenFrom: Kumar Kartikeya Dwivedi <[email protected]>
BPF ABI always uses 64-bit return value, but so far __bpf_prog_run and higher level wrappers always truncated the return value to 32-bit. We want to be able to introduce a new BPF program type that returns a PTR_TO_BTF_ID or NULL from the BPF program to the caller context in the kernel. To be able to use this returned pointer value, the bpf_prog_run invocation needs to be able to return a 64-bit value, so update the definitions to allow this.
To avoid code churn in the whole kernel, we let the compiler handle truncation normally, and allow new call sites to utilize the 64-bit return value, by receiving the return value as a u64.
Only the cgroup_bpf_lsm functions have been modified apart from the generic bpf_prog_run wrappers, as their prototype needs to match bpf_func_t to be called generically, but otherwise none of other cases have been modified yet. It will be done when a specific call site requires handling of the 64-bit return value from the BPF program.
--- include/linux/bpf-cgroup.h | 12 ++++++------ include/linux/bpf.h | 22 +++++++++++----------- include/linux/filter.h | 30 +++++++++++++++--------------- kernel/bpf/cgroup.c | 12 ++++++------ kernel/bpf/core.c | 14 +++++++------- kernel/bpf/offload.c | 4 ++-- net/packet/af_packet.c | 7 +++++-- 7 files changed, 52 insertions(+), 49 deletions(-)
include/linux/bpf-cgroup.h +6 -6
@@ -23,12 +23,12 @@ struct ctl_table;struct ctl_table_header ;struct task_struct ; -unsigned int __cgroup_bpf_run_lsm_sock (const void *ctx, -const struct bpf_insn *insn); -unsigned int __cgroup_bpf_run_lsm_socket (const void *ctx, -const struct bpf_insn *insn); -unsigned int __cgroup_bpf_run_lsm_current (const void *ctx, -const struct bpf_insn *insn); +u64 __cgroup_bpf_run_lsm_sock (const void *ctx, +const struct bpf_insn *insn); +u64 __cgroup_bpf_run_lsm_socket (const void *ctx, +const struct bpf_insn *insn); +u64 __cgroup_bpf_run_lsm_current (const void *ctx, +const struct bpf_insn *insn);#ifdef CONFIG_CGROUP_BPF
include/linux/bpf.h +11 -11
@@ -59,8 +59,8 @@ typedef u64 (*bpf_callback_t)(u64, u64, u64, u64, u64);typedef int (*bpf_iter_init_seq_priv_t )(void *private_data,struct bpf_iter_aux_info *aux);typedef void (*bpf_iter_fini_seq_priv_t )(void *private_data); -typedef unsigned int (*bpf_func_t )(const void *, -const struct bpf_insn *); +typedef u64 (*bpf_func_t )(const void *, +const struct bpf_insn *);struct bpf_iter_seq_info {const struct seq_operations *seq_ops;bpf_iter_init_seq_priv_t init_seq_private;
@@ -1004,7 +1004,7 @@ struct bpf_dispatcher {struct bpf_ksym ksym; }; -static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func( +static __always_inline __nocfi u64 bpf_dispatcher_nop_func(const void *ctx,const struct bpf_insn *insnsi,bpf_func_t bpf_func)
@@ -1049,7 +1049,7 @@ int __init bpf_arch_init_dispatcher_early(void *ip);#define DEFINE_BPF_DISPATCHER (name) \ notrace BPF_DISPATCHER_ATTRIBUTES \ - noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \ + noinline __nocfi u64 bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \ bpf_func_t bpf_func) \
@@ -1062,7 +1062,7 @@ int __init bpf_arch_init_dispatcher_early(void *ip); BPF_DISPATCHER_INIT_CALL(bpf_dispatcher_##name);#define DECLARE_BPF_DISPATCHER (name) \ - unsigned int bpf_dispatcher_##name##_func( \ + u64 bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \ bpf_func_t bpf_func); \
@@ -1265,7 +1265,7 @@ struct bpf_prog {u8 tag[BPF_TAG_SIZE ];struct bpf_prog_stats __percpu *stats;int __percpu *active; -unsigned int (*bpf_func)(const void *ctx, +u64 (*bpf_func)(const void *ctx,const struct bpf_insn *insn);struct bpf_prog_aux *aux;/* Auxiliary fields */ struct sock_fprog_kern *orig_prog;/* Original BPF program */
@@ -1619,9 +1619,9 @@ static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)/* BPF program asks to set CN on the packet. */ #define BPF_RET_SET_CN (1 << 0) -typedef u32 (*bpf_prog_run_fn )(const struct bpf_prog *prog,const void *ctx); +typedef u64 (*bpf_prog_run_fn )(const struct bpf_prog *prog,const void *ctx); -static __always_inline u32 +static __always_inline u64bpf_prog_run_array (const struct bpf_prog_array *array,const void *ctx,bpf_prog_run_fn run_prog) {
@@ -1629,7 +1629,7 @@ bpf_prog_run_array(const struct bpf_prog_array *array,const struct bpf_prog *prog;struct bpf_run_ctx *old_run_ctx;struct bpf_trace_run_ctx run_ctx; -u32 ret =1 ; +u64 ret =1 ;RCU_LOCKDEP_WARN (!rcu_read_lock_held (),"no rcu lock held" );
@@ -1659,7 +1659,7 @@ bpf_prog_run_array(const struct bpf_prog_array *array,* section and disable preemption for that program alone, so it can access * rcu-protected dynamically sized maps. */ -static __always_inline u32 +static __always_inline u64bpf_prog_run_array_sleepable (const struct bpf_prog_array __rcu *array_rcu,const void *ctx,bpf_prog_run_fn run_prog) {
@@ -1668,7 +1668,7 @@ bpf_prog_run_array_sleepable(const struct bpf_prog_array __rcu *array_rcu,const struct bpf_prog_array *array;struct bpf_run_ctx *old_run_ctx;struct bpf_trace_run_ctx run_ctx; -u32 ret =1 ; +u64 ret =1 ;might_fault ();
include/linux/filter.h +15 -15
@@ -573,16 +573,16 @@ extern int (*nfct_btf_struct_access)(struct bpf_verifier_log *log, const struct enum bpf_access_type atype, u32 *next_btf_id, enum bpf_type_flag *flag); -typedef unsigned int (*bpf_dispatcher_fn )(const void *ctx, -const struct bpf_insn *insnsi, -unsigned int (*bpf_func)(const void *, -const struct bpf_insn *)); +typedef u64 (*bpf_dispatcher_fn )(const void *ctx, +const struct bpf_insn *insnsi, +u64 (*bpf_func)(const void *, +const struct bpf_insn *)); -static __always_inline u32 __bpf_prog_run(const struct bpf_prog *prog, +static __always_inline u64 __bpf_prog_run(const struct bpf_prog *prog,const void *ctx,bpf_dispatcher_fn dfunc) { -u32 ret; +u64 ret;cant_migrate ();if (static_branch_unlikely (&bpf_stats_enabled_key)) {
@@ -602,7 +602,7 @@ static __always_inline u32 __bpf_prog_run(const struct bpf_prog *prog,return ret; } -static __always_inline u32 bpf_prog_run(const struct bpf_prog *prog, const void *ctx) +static __always_inline u64 bpf_prog_run(const struct bpf_prog *prog, const void *ctx) {return __bpf_prog_run (prog, ctx, bpf_dispatcher_nop_func); }
@@ -615,10 +615,10 @@ static __always_inline u32 bpf_prog_run(const struct bpf_prog *prog, const void* invocation of a BPF program does not require reentrancy protection * against a BPF program which is invoked from a preempting task. */ -static inline u32 bpf_prog_run_pin_on_cpu (const struct bpf_prog *prog, +static inline u64 bpf_prog_run_pin_on_cpu (const struct bpf_prog *prog,const void *ctx) { -u32 ret; +u64 ret;migrate_disable (); ret =bpf_prog_run (prog, ctx);
@@ -714,13 +714,13 @@ static inline u8 *bpf_skb_cb(const struct sk_buff *skb) }/* Must be invoked with migration disabled */ -static inline u32 __bpf_prog_run_save_cb (const struct bpf_prog *prog, +static inline u64 __bpf_prog_run_save_cb (const struct bpf_prog *prog,const void *ctx) {const struct sk_buff *skb = ctx;u8 *cb_data =bpf_skb_cb (skb);u8 cb_saved[BPF_SKB_CB_LEN ]; -u32 res; +u64 res;if (unlikely (prog->cb_access)) {memcpy (cb_saved, cb_data,sizeof (cb_saved));
@@ -735,10 +735,10 @@ static inline u32 __bpf_prog_run_save_cb(const struct bpf_prog *prog,return res; } -static inline u32 bpf_prog_run_save_cb (const struct bpf_prog *prog, +static inline u64 bpf_prog_run_save_cb (const struct bpf_prog *prog,struct sk_buff *skb) { -u32 res; +u64 res;migrate_disable (); res =__bpf_prog_run_save_cb (prog, skb);
@@ -746,11 +746,11 @@ static inline u32 bpf_prog_run_save_cb(const struct bpf_prog *prog,return res; } -static inline u32 bpf_prog_run_clear_cb (const struct bpf_prog *prog, +static inline u64 bpf_prog_run_clear_cb (const struct bpf_prog *prog,struct sk_buff *skb) {u8 *cb_data =bpf_skb_cb (skb); -u32 res; +u64 res;if (unlikely (prog->cb_access))memset (cb_data,0 ,BPF_SKB_CB_LEN );
kernel/bpf/cgroup.c +6 -6
@@ -63,8 +63,8 @@ bpf_prog_run_array_cg(const struct cgroup_bpf *cgrp,return run_ctx.retval; } -unsigned int __cgroup_bpf_run_lsm_sock (const void *ctx, -const struct bpf_insn *insn) +u64 __cgroup_bpf_run_lsm_sock (const void *ctx, +const struct bpf_insn *insn) {const struct bpf_prog *shim_prog;struct sock *sk;
@@ -85,8 +85,8 @@ unsigned int __cgroup_bpf_run_lsm_sock(const void *ctx,return ret; } -unsigned int __cgroup_bpf_run_lsm_socket (const void *ctx, -const struct bpf_insn *insn) +u64 __cgroup_bpf_run_lsm_socket (const void *ctx, +const struct bpf_insn *insn) {const struct bpf_prog *shim_prog;struct socket *sock;
@@ -107,8 +107,8 @@ unsigned int __cgroup_bpf_run_lsm_socket(const void *ctx,return ret; } -unsigned int __cgroup_bpf_run_lsm_current (const void *ctx, -const struct bpf_insn *insn) +u64 __cgroup_bpf_run_lsm_current (const void *ctx, +const struct bpf_insn *insn) {const struct bpf_prog *shim_prog;struct cgroup *cgrp;
kernel/bpf/core.c +7 -7
@@ -2004,7 +2004,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn)#define PROG_NAME (stack_size) __bpf_prog_run##stack_size#define DEFINE_BPF_PROG_RUN (stack_size) \ -static unsigned int PROG_NAME(stack_size)(const void *ctx, const struct bpf_insn *insn) \ +static u64 PROG_NAME(stack_size)(const void *ctx, const struct bpf_insn *insn) \ { \ u64 stack[stack_size / sizeof(u64)]; \ u64 regs[MAX_BPF_EXT_REG] = {}; \
@@ -2048,8 +2048,8 @@ EVAL4(DEFINE_BPF_PROG_RUN_ARGS, 416, 448, 480, 512);#define PROG_NAME_LIST (stack_size) PROG_NAME(stack_size), -static unsigned int (*interpreters[])(const void *ctx, -const struct bpf_insn *insn) = { +static u64 (*interpreters[])(const void *ctx, +const struct bpf_insn *insn) = { EVAL6(PROG_NAME_LIST, 32, 64, 96, 128, 160, 192) EVAL6(PROG_NAME_LIST, 224, 256, 288, 320, 352, 384)EVAL4 (PROG_NAME_LIST ,416 ,448 ,480 ,512 )
@@ -2074,8 +2074,8 @@ void bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth) }#else -static unsigned int __bpf_prog_ret0_warn (const void *ctx, -const struct bpf_insn *insn) +static u64 __bpf_prog_ret0_warn (const void *ctx, +const struct bpf_insn *insn) {/* If this handler ever gets executed, then BPF_JIT_ALWAYS_ON * is not working properly, so warn about it!
@@ -2210,8 +2210,8 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err) }EXPORT_SYMBOL_GPL (bpf_prog_select_runtime); -static unsigned int __bpf_prog_ret1 (const void *ctx, -const struct bpf_insn *insn) +static u64 __bpf_prog_ret1 (const void *ctx, +const struct bpf_insn *insn) {return 1 ; }
kernel/bpf/offload.c +2 -2
@@ -246,8 +246,8 @@ static int bpf_prog_offload_translate(struct bpf_prog *prog)return ret; } -static unsigned int bpf_prog_warn_on_exec (const void *ctx, -const struct bpf_insn *insn) +static u64 bpf_prog_warn_on_exec (const void *ctx, +const struct bpf_insn *insn) {WARN (1 ,"attempt to execute device eBPF program on the host!" );return 0 ;
net/packet/af_packet.c +5 -2
@@ -1444,8 +1444,11 @@ static unsigned int fanout_demux_bpf(struct packet_fanout *f,rcu_read_lock (); prog =rcu_dereference (f->bpf_prog); -if (prog) - ret =bpf_prog_run_clear_cb (prog, skb) % num; +if (prog) { + ret =bpf_prog_run_clear_cb (prog, skb); +/* For some architectures, we need to do modulus in 32-bit width */ + ret %= num; + }rcu_read_unlock ();return ret;
Signature · 1 line
2.38.1
Re: [PATCH bpf-next v3 3/3] bpf: Use 64-bit return value for bpf_prog_run
Quoted 10 lines · Toke Høiland-Jørgensen
From: Kumar Kartikeya Dwivedi <[email protected]>
BPF ABI always uses 64-bit return value, but so far
__bpf_prog_runand higher level wrappers always truncated the return value to 32-bit. We want to be able to introduce a new BPF program type that returns aPTR_TO_BTF_IDor NULL from the BPF program to the caller context in the kernel. To be able to use this returned pointer value, thebpf_prog_runinvocation needs to be able to return a 64-bit value, so update the definitions to allow this.
...
Quoted 11 lines
include/linux/filter.h +2
-2-static __always_inline u32 __bpf_prog_run(const struct bpf_prog *prog, +static __always_inline u64 __bpf_prog_run(const struct bpf_prog *prog,const void *ctx,bpf_dispatcher_fn dfunc) { -u32 ret; +u64 ret;cant_migrate ();if (static_branch_unlikely (&bpf_stats_enabled_key)) {
include/linux/filter.h static __always_inline u32 __bpf_prog_run(const struct bpf_prog *prog, · line 608
Quoted 8 lines
include/linux/filter.h +1
-1@@ -602,7 +602,7 @@ static __always_inline u32 __bpf_prog_run(const struct bpf_prog *prog,return ret; } -static __always_inline u32 bpf_prog_run(const struct bpf_prog *prog, const void *ctx) +static __always_inline u64 bpf_prog_run(const struct bpf_prog *prog, const void *ctx) {return __bpf_prog_run (prog, ctx, bpf_dispatcher_nop_func); }
I suspect 32-bit archs with JITs are partially broken with this change. As long as progs want to return pointers it's ok, but actual 64-bit values will be return garbage in upper 32-bit, since 32-bit JITs won't populate the upper bits.
I don't think changing u32->long retval is a good idea either, since BPF ISA has to be stable regardless of underlying arch. The u32->u64 transition is good long term, but let's add full 64-bit tests to lib/test_bpf and fix JITs.
I've applied the first two patches of the series.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 3/3] bpf: Use 64-bit return value for bpf_prog_run
Quoted 28 lines · Alexei Starovoitov
Quoted 10 lines · Toke Høiland-Jørgensen
From: Kumar Kartikeya Dwivedi <[email protected]>
BPF ABI always uses 64-bit return value, but so far
__bpf_prog_runand higher level wrappers always truncated the return value to 32-bit. We want to be able to introduce a new BPF program type that returns aPTR_TO_BTF_IDor NULL from the BPF program to the caller context in the kernel. To be able to use this returned pointer value, thebpf_prog_runinvocation needs to be able to return a 64-bit value, so update the definitions to allow this....
Quoted 14 lines
include/linux/filter.h +2
-2-static __always_inline u32 __bpf_prog_run(const struct bpf_prog *prog, +static __always_inline u64 __bpf_prog_run(const struct bpf_prog *prog,const void *ctx,bpf_dispatcher_fn dfunc) { -u32 ret; +u64 ret;cant_migrate ();if (static_branch_unlikely (&bpf_stats_enabled_key)) {@@ -602,7 +602,7 @@ static __always_inline u32 __bpf_prog_run(const struct bpf_prog *prog,return ret; }
Quoted 15 lines
include/linux/filter.h static __always_inline u32 __bpf_prog_run(const struct bpf_prog *prog, · line 608
Quoted 5 lines
include/linux/filter.h +1
-1-static __always_inline u32 bpf_prog_run(const struct bpf_prog *prog, const void *ctx) +static __always_inline u64 bpf_prog_run(const struct bpf_prog *prog, const void *ctx) {return __bpf_prog_run (prog, ctx, bpf_dispatcher_nop_func); }I suspect 32-bit archs with JITs are partially broken with this change. As long as progs want to return pointers it's ok, but actual 64-bit values will be return garbage in upper 32-bit, since 32-bit JITs won't populate the upper bits.
I don't think changingu32->longretval is a good idea either, since BPF ISA has to be stable regardless of underlying arch. Theu32->u64transition is good long term, but let's add full 64-bit tests tolib/test_bpfand fix JITs.
I will resubmit with these tests and revisiting the 32-bit JITs (but it will take some time, since my backlog is full).
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 8 lines · On Tue, 8 Nov 2022 15:05:58 +0100 you wrote:
Stanislav suggested[0] that these small refactorings could be split out from the XDP queueing RFC series and merged separately. The first change is a small repacking of struct
softnet_data, the others change the BPF call sites to support full 64-bit values as arguments tobpf_redirect_map()and as the return value of a BPF program, relying on the fact that BPF registers are always 64-bit wide to maintain backwards compatibility.[...]
Here is the summary with links:
- [bpf-next,v3,1/3] dev: Move received_rps counter next to RPS members in softnet data
https://git.kernel.org/bpf/bpf-next/c/14d898f3c1b3 - [bpf-next,v3,2/3] bpf: Expand map key argument of bpf_redirect_map to u64 https://git.kernel.org/bpf/bpf-next/c/32637e33003f
- [bpf-next,v3,3/3] bpf: Use 64-bit return value for bpf_prog_run (no matching commit)
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html