'bpf_fastcall' attribute in vmlinux.h and bpf_helper_defs.h
Eduard Zingerman
maintainercover 'bpf_fastcall' attribute in vmlinux.h and bpf_helper_defs.h
The goal of this patch-set is to reflect attribute bpf_fastcall for supported helpers and kfuncs in generated header files. For helpers this requires a tweak for scripts/bpf_doc.py and an update to uapi/linux/bpf.h doc-comment.
For kfuncs this requires:
- introduction of a new KF_FASTCALL flag;
- modification to pahole to read kfunc flags and generate DECL_TAG "bpf_fastcall" for marked kfuncs;
- modification to bpftool to scan for DECL_TAG "bpf_fastcall" presence.
In both cases the following helper macro is defined in the generated header:
#ifndef __bpf_fastcall
#if __has_attribute(bpf_fastcall)
#define __bpf_fastcall __attribute__((bpf_fastcall))
#else
#define __bpf_fastcall
#endif
#endifAnd is used to mark appropriate function prototypes. More information about bpf_fastcall attribute could be found in [1] and [2].
Modifications to pahole are submitted separately.
[1] LLVM source tree commit:
64e464349bfc ("[BPF] introduce __attribute__((bpf_fastcall))")[2] Linux kernel tree commit (note: feature was renamed from no_caller_saved_registers to bpf_fastcall after this commit):
52839f31cece ("Merge branch 'no_caller_saved_registers-attribute-for-helper-calls'")
Eduard Zingerman (4):
bpf: allow specifying bpf_fastcall attribute for BPF helpers
bpf: __bpf_fastcall for bpf_get_smp_processor_id in uapi
bpf: use KF_FASTCALL to mark kfuncs supporting fastcall contract
bpftool: __bpf_fastcall for kfuncs marked with special decl_tag
include/linux/btf.h | 1 + include/uapi/linux/bpf.h | 2 + kernel/bpf/helpers.c | 4 +- kernel/bpf/verifier.c | 5 +- scripts/bpf_doc.py | 50 ++++++++++++++++- tools/bpf/bpftool/btf.c | 98 ++++++++++++++++++++++++++++++---- tools/include/uapi/linux/bpf.h | 2 + 7 files changed, 144 insertions(+), 18 deletions(-)
Signature · 1 line
2.46.0
Eduard Zingerman
maintainerPATCH 1/4 bpf: allow specifying bpf_fastcall attribute for BPF helpers
Allow a new optional 'Attributes' section to be specified for helper functions description, e.g.:
* u32 bpf_get_smp_processor_id(void)
* ...
* Return
* ...
* Attributes
* __bpf_fastcall
*
Generated header for the example above:
#ifndef __bpf_fastcall #if __has_attribute(__bpf_fastcall) #define __bpf_fastcall __attribute__((bpf_fastcall)) #else #define __bpf_fastcall #endif #endif ... __bpf_fastcall static __u32 (* const bpf_get_smp_processor_id)(void) = (void *) 8;
The following rules apply:
- when present, section must follow 'Return' section;
- attribute names are specified on the line following 'Attribute' keyword;
- attribute names are separated by spaces;
- section ends with an "empty" line (" *\n").
Valid attribute names are recorded in the ATTRS map. ATTRS maps shortcut attribute name to correct C syntax.
--- scripts/bpf_doc.py | 50 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 2 deletions(-)
scripts/bpf_doc.py +48 -2
@@ -37,10 +37,11 @@ class APIElement(object): @desc: textual description of the symbol @ret: (optional) description of any associated return value """ - def __init__(self, proto='', desc='', ret=''): + def __init__(self, proto='', desc='', ret='', attrs=[]): self.proto = proto self.desc = desc self.ret = ret + self.attrs = attrs class Helper(APIElement):
@@ -81,6 +82,11 @@ class Helper(APIElement): return res +ATTRS = { + '__bpf_fastcall': 'bpf_fastcall' +} + + class HeaderParser(object): """ An object used to parse a file in order to extract the documentation of a
@@ -111,7 +117,8 @@ class HeaderParser(object): proto = self.parse_proto() desc = self.parse_desc(proto) ret = self.parse_ret(proto) - return Helper(proto=proto, desc=desc, ret=ret) + attrs = self.parse_attrs(proto) + return Helper(proto=proto, desc=desc, ret=ret, attrs=attrs) def parse_symbol(self): p = re.compile(r' \* ?(BPF\w+)$')
@@ -192,6 +199,28 @@ class HeaderParser(object): raise Exception("No return found for " + proto) return ret + def parse_attrs(self, proto): + p = re.compile(r' \* ?(?:\t| {5,8})Attributes$') + capture = p.match(self.line) + if not capture: + return [] + # Expect a single line with mnemonics for attributes separated by spaces + self.line = self.reader.readline() + p = re.compile(r' \* ?(?:\t| {5,8})(?:\t| {8})(.*)') + capture = p.match(self.line) + if not capture: + raise Exception("Incomplete 'Attributes' section for " + proto) + attrs = capture.group(1).split(' ') + for attr in attrs: + if attr not in ATTRS: + raise Exception("Unexpected attribute '" + attr + "' specified for " + proto) + self.line = self.reader.readline() + if self.line != ' *\n': + raise Exception("Expecting empty line after 'Attributes' section for " + proto) + # Prepare a line for next self.parse_* to consume + self.line = self.reader.readline() + return attrs + def seek_to(self, target, help_message, discard_lines = 1): self.reader.seek(0) offset = self.reader.read().find(target)
@@ -789,6 +818,21 @@ class PrinterHelpers(Printer): print('%s;' % fwd) print('') + used_attrs = set() + for helper in self.elements: + for attr in helper.attrs: + used_attrs.add(attr) + for attr in sorted(used_attrs): + print('#ifndef %s' % attr) + print('#if __has_attribute(%s)' % ATTRS[attr]) + print('#define %s __attribute__((%s))' % (attr, ATTRS[attr])) + print('#else') + print('#define %s' % attr) + print('#endif') + print('#endif') + if used_attrs: + print('') + def print_footer(self): footer = '' print(footer)
@@ -827,6 +871,8 @@ class PrinterHelpers(Printer): print(' *{}{}'.format(' \t' if line else '', line)) print(' */') + if helper.attrs: + print(" ".join(helper.attrs)) print('static %s %s(* const %s)(' % (self.map_type(proto['ret_type']), proto['ret_star'], proto['name']), end='') comma = ''
Signature · 1 line
2.46.0
Re: [PATCH bpf-next v1 1/4] bpf: allow specifying bpf_fastcall attribute for BPF helpers
Quoted 14 lines · Eduard Zingerman
Allow a new optional 'Attributes' section to be specified for helper functions description, e.g.:
* u32
bpf_get_smp_processor_id(void)
* ...
* Return
* ...
* Attributes
*__bpf_fastcall
*Generated header for the example above:
Quoted 9 lines
#ifndef __bpf_fastcall #if __has_attribute(__bpf_fastcall) #define __bpf_fastcall __attribute__((bpf_fastcall)) #else #define __bpf_fastcall #endif #endif ... __bpf_fastcall
I found it a bit annoying that bpf_helper_defs.h uses
__bpf_fastcall
static __u32 ....
format, while in vmlinux we have single-line (and yeah, note that I put extern in front)
extern __bpf_fastcall __u32 ...
So I slightly modified bpf_doc.py with my weak Python-fu:
scripts/bpf_doc.py +3 -2
@@ -871,9 +871,10 @@ class PrinterHelpers(Printer): print(' *{}{}'.format(' \t' if line else '', line)) print(' */') + print('static ', end='') if helper.attrs: - print(" ".join(helper.attrs)) - print('static %s %s(* const %s)(' % (self.map_type(proto['ret_type']), + print('%s ' % (" ".join(helper.attrs)), end='') + print('%s %s(* const %s)(' % (self.map_type(proto['ret_type']), proto['ret_star'],
proto['name']), end='')
comma = ''
for i, a in enumerate(proto['args']):
But now I have:
static __bpf_fastcall __u32 (* const bpf_get_smp_processor_id)(void) = (void *) 8;
and
extern __bpf_fastcall void *bpf_rdonly_cast(const void *obj__ign, u32 btf_id__k) __weak __ksym;
and that makes me a touch happier. I hope you don't mind.
Quoted 9 lines
static __u32 (* const bpf_get_smp_processor_id)(void) = (void *) 8;The following rules apply:
- when present, section must follow 'Return' section;
- attribute names are specified on the line following 'Attribute' keyword;
- attribute names are separated by spaces;
- section ends with an "empty" line (" *\n").
Quoted 8 lines
Valid attribute names are recorded in the ATTRS map. ATTRS maps shortcut attribute name to correct C syntax.
Signed-off-by Eduard Zingerman <[email protected]>--- scripts/bpf_doc.py | 50 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 2 deletions(-)
Looks good to me, and I'm not sure there is anything too controversial here, so I went ahead and applied to bpf-next, thanks.
Eduard Zingerman
maintainerRe: [PATCH bpf-next v1 1/4] bpf: allow specifying bpf_fastcall attribute for BPF helpers
On Fri, 2024-09-27 at 15:31 -0700, Andrii Nakryiko wrote:
[...]
Quoted 20 lines
scripts/bpf_doc.py +3
-2@@ -871,9 +871,10 @@ class PrinterHelpers(Printer): print(' *{}{}'.format(' \t' if line else '', line)) print(' */') + print('static ', end='') if helper.attrs: - print(" ".join(helper.attrs)) - print('static %s %s(* const %s)(' % (self.map_type(proto['ret_type']), + print('%s ' % (" ".join(helper.attrs)), end='') + print('%s %s(* const %s)(' % (self.map_type(proto['ret_type']), proto['ret_star'], proto['name']), end='') comma = ''for i, a in
enumerate(proto['args']):But now I have:
static
__bpf_fastcall__u32(* const bpf_get_smp_processor_id)(void) = (void *) 8;
Quoted 6 lines
and
extern
__bpf_fastcallvoid *bpf_rdonly_cast(const void*obj__ign, u32btf_id__k)__weak__ksym;and that makes me a touch happier. I hope you don't mind.
This change looks fine to me.
[...]
Quoted 2 lines
Looks good to me, and I'm not sure there is anything too controversial here, so I went ahead and applied to bpf-next, thanks.
Great, thank you.
Eduard Zingerman
maintainerPATCH 2/4 bpf: __bpf_fastcall for bpf_get_smp_processor_id in uapi
In reply to Eduard ZingermanSince [1] kernel supports __bpf_fastcall attribute for helper function bpf_get_smp_processor_id(). Update uapi definition for this helper in order to have this attribute in the generated bpf_helper_defs.h
[1] commit 91b7fbf3936f ("bpf, x86, riscv, arm: no_caller_saved_registers for bpf_get_smp_processor_id()")
--- include/uapi/linux/bpf.h | 2 ++ tools/include/uapi/linux/bpf.h | 2 ++ 2 files changed, 4 insertions(+)
include/uapi/linux/bpf.h +2 -0
@@ -1970,6 +1970,8 @@ union bpf_attr {* program. * Return * The SMP id of the processor running the program. +* Attributes +* __bpf_fastcall * * long bpf_skb_store_bytes(struct sk_buff *skb, u32 offset, const void *from, u32 len, u64 flags) * Description
tools/include/uapi/linux/bpf.h +2 -0
@@ -1970,6 +1970,8 @@ union bpf_attr {* program. * Return * The SMP id of the processor running the program. +* Attributes +* __bpf_fastcall * * long bpf_skb_store_bytes(struct sk_buff *skb, u32 offset, const void *from, u32 len, u64 flags) * Description
Signature · 1 line
2.46.0
Eduard Zingerman
maintainerPATCH 3/4 bpf: use KF_FASTCALL to mark kfuncs supporting fastcall contract
In reply to Eduard ZingermanIn order to allow pahole add btf_decl_tag("bpf_fastcall") for kfuncs supporting bpf_fastcall, mark such functions with KF_FASTCALL in id_set8 objects.
--- include/linux/btf.h | 1 + kernel/bpf/helpers.c | 4 ++-- kernel/bpf/verifier.c | 5 +---- 3 files changed, 4 insertions(+), 6 deletions(-)
include/linux/btf.h +1 -0
@@ -75,6 +75,7 @@#define KF_ITER_NEXT (1 << 9)/* kfunc implements BPF iter next method */ #define KF_ITER_DESTROY (1 << 10)/* kfunc implements BPF iter destructor */ #define KF_RCU_PROTECTED (1 << 11)/* kfunc should be protected by rcu cs when they are invoked */ +#define KF_FASTCALL (1 << 12)/* kfunc supports bpf_fastcall protocol */ /* * Tag marking a kernel function as a kfunc. This is meant to minimize the
kernel/bpf/helpers.c +2 -2
@@ -3052,8 +3052,8 @@ BTF_ID(func, bpf_cgroup_release_dtor)#endif BTF_KFUNCS_START(common_btf_ids) -BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx) -BTF_ID_FLAGS(func, bpf_rdonly_cast) +BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_rdonly_cast, KF_FASTCALL) BTF_ID_FLAGS(func, bpf_rcu_read_lock) BTF_ID_FLAGS(func, bpf_rcu_read_unlock) BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL)
kernel/bpf/verifier.c +1 -4
@@ -16199,10 +16199,7 @@ static u32 kfunc_fastcall_clobber_mask(struct bpf_kfunc_call_arg_meta *meta)/* Same as verifier_inlines_helper_call() but for kfuncs, see comment above */ static bool is_fastcall_kfunc_call (struct bpf_kfunc_call_arg_meta *meta) { -if (meta->btf == btf_vmlinux) -return meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || - meta->func_id == special_kfunc_list[KF_bpf_rdonly_cast]; -return false; +return meta->kfunc_flags &KF_FASTCALL ; }/* LLVM define a bpf_fastcall function attribute.
Signature · 1 line
2.46.0
Eduard Zingerman
maintainerPATCH 4/4 bpftool: __bpf_fastcall for kfuncs marked with special decl_tag
In reply to Eduard ZingermanGenerate __attribute__((bpf_fastcall)) for kfuncs marked with "bpf_fastcall" decl tag. E.g. for the following BTF:
$ bpftool btf dump file vmlinux
...
[A] FUNC 'bpf_rdonly_cast' type_id=...
...
[B] DECL_TAG 'bpf_kfunc' type_id=A component_idx=-1
[C] DECL_TAG 'bpf_fastcall' type_id=A component_idx=-1Generate the following vmlinux.h:
#ifndef __VMLINUX_H__
#define __VMLINUX_H__
...
#ifndef __bpf_fastcall
#if __has_attribute(bpf_fastcall)
#define __bpf_fastcall __attribute__((bpf_fastcall))
#else
#define __bpf_fastcall
#endif
#endif
...
__bpf_fastcall extern void *bpf_rdonly_cast(...) ...;The "bpf_fastcall" / "bpf_kfunc" tags pair would generated by pahole when constructing vmlinux BTF.
While at it, sort printed kfuncs by name for better vmlinux.h stability.
--- tools/bpf/bpftool/btf.c | 98 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 88 insertions(+), 10 deletions(-)
tools/bpf/bpftool/btf.c +88 -10
@@ -1,11 +1,15 @@// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) /* Copyright (C) 2019 Facebook */ +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif #include <errno.h> #include <fcntl.h> #include <linux/err.h> #include <stdbool.h> #include <stdio.h> +#include <stdlib.h> #include <string.h> #include <unistd.h> #include <linux/btf.h>
@@ -21,6 +25,7 @@#include "main.h" #define KFUNC_DECL_TAG "bpf_kfunc" +#define FASTCALL_DECL_TAG "bpf_fastcall"static const char *const btf_kind_str[NR_BTF_KINDS ] = { [BTF_KIND_UNKN ] ="UNKNOWN" ,
@@ -464,19 +469,59 @@ static int dump_btf_raw(const struct btf *btf,return 0 ; } +struct ptr_array { +__u32 cnt; +__u32 cap; +const void **elems; +}; + +static int ptr_array_push (const void *ptr,struct ptr_array *arr) +{ +__u32 new_cap; +void *tmp; + +if (arr->cnt == arr->cap) { + new_cap = (arr->cap ?:16 ) *2 ; + tmp =realloc (arr->elems,sizeof (*arr->elems) * new_cap); +if (!tmp) +return -ENOMEM ; + arr->elems = tmp; + arr->cap = new_cap; + } + arr->elems[arr->cnt++] = ptr; +return 0 ; +} + +static void ptr_array_free (struct ptr_array *arr) +{ +free (arr->elems); +} + +static int cmp_kfuncs (const void *pa,const void *pb,void *ctx) +{ +struct btf *btf = ctx; +const struct btf_type *a = *(void **)pa; +const struct btf_type *b = *(void **)pb; + +return strcmp (btf__str_by_offset (btf, a->name_off), +btf__str_by_offset (btf, b->name_off)); +} +static int dump_btf_kfuncs (struct btf_dump *d,const struct btf *btf) {LIBBPF_OPTS (btf_dump_emit_type_decl_opts, opts); -int cnt =btf__type_cnt (btf); -int i; +__u32 cnt =btf__type_cnt (btf), i, j; +struct ptr_array fastcalls = {}; +struct ptr_array kfuncs = {}; +int err =0 ;printf ("\n/* BPF kfuncs */\n" );printf ("#ifndef BPF_NO_KFUNC_PROTOTYPES\n" );for (i =1 ; i < cnt; i++) {const struct btf_type *t =btf__type_by_id (btf, i); +const struct btf_type *ft;const char *name; -int err;if (!btf_is_decl_tag (t))continue ;
@@ -484,27 +529,53 @@ static int dump_btf_kfuncs(struct btf_dump *d, const struct btf *btf)if (btf_decl_tag (t)->component_idx !=-1 )continue ; - name =btf__name_by_offset (btf, t->name_off); -if (strncmp (name,KFUNC_DECL_TAG ,sizeof (KFUNC_DECL_TAG ))) + ft =btf__type_by_id (btf, t->type); +if (!btf_is_func (ft))continue ; - t =btf__type_by_id (btf, t->type); -if (!btf_is_func (t)) -continue ; + name =btf__name_by_offset (btf, t->name_off); +if (strncmp (name,KFUNC_DECL_TAG ,sizeof (KFUNC_DECL_TAG )) ==0 ) { + err =ptr_array_push (ft, &kfuncs); +if (err) + goto out; + } + +if (strncmp (name,FASTCALL_DECL_TAG ,sizeof (FASTCALL_DECL_TAG )) ==0 ) { + err =ptr_array_push (ft, &fastcalls); +if (err) + goto out; + } + } + +/* Sort kfuncs by name for improved vmlinux.h stability */ +qsort_r (kfuncs.elems, kfuncs.cnt,sizeof (*kfuncs.elems), cmp_kfuncs, (void *)btf); +for (i =0 ; i < kfuncs.cnt; i++) { +const struct btf_type *t = kfuncs.elems[i]; + +/* Assume small amount of fastcall kfuncs */ +for (j =0 ; j < fastcalls.cnt; j++) { +if (fastcalls.elems[j] == t) { +printf ("__bpf_fastcall " ); +break ; + } + }printf ("extern " ); opts.field_name =btf__name_by_offset (btf, t->name_off); err =btf_dump__emit_type_decl (d, t->type, &opts);if (err) -return err; + goto out;printf (" __weak __ksym;\n" ); }printf ("#endif\n\n" ); -return 0 ; +out: +ptr_array_free (&fastcalls); +ptr_array_free (&kfuncs); +return err; } static void __printf(2, 0) btf_dump_printf(void *ctx,
@@ -718,6 +789,13 @@ static int dump_btf_c(const struct btf *btf,printf ("#ifndef __weak\n" );printf ("#define __weak __attribute__((weak))\n" );printf ("#endif\n\n" ); +printf ("#ifndef __bpf_fastcall\n" ); +printf ("#if __has_attribute(bpf_fastcall)\n" ); +printf ("#define __bpf_fastcall __attribute__((bpf_fastcall))\n" ); +printf ("#else\n" ); +printf ("#define __bpf_fastcall\n" ); +printf ("#endif\n" ); +printf ("#endif\n\n" );if (root_type_cnt) {for (i =0 ; i < root_type_cnt; i++) {
Signature · 1 line
2.46.0
Eduard Zingerman
maintainerQuoted 1 line · Eduard Zingerman
Modifications to pahole are submitted separately.
Note, corresponding pahole changes are submitted here:
https://lore.kernel.org/bpf/[email protected]/
[...]
This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:
Quoted 11 lines · On Mon, 16 Sep 2024 02:17:08 -0700 you wrote:
The goal of this patch-set is to reflect attribute
bpf_fastcallfor supported helpers and kfuncs in generated header files. For helpers this requires a tweak forscripts/bpf_doc.pyand an update touapi/linux/bpf.hdoc-comment.
For kfuncs this requires:
- introduction of a newKF_FASTCALLflag;
- modification to pahole to read kfunc flags and generateDECL_TAG"bpf_fastcall" for marked kfuncs;
- modification to bpftool to scan forDECL_TAG"bpf_fastcall" presence.
Quoted 1 line
[...]
Here is the summary with links:
- [bpf-next,v1,1/4] bpf: allow specifying bpf_fastcall attribute for BPF helpers
https://git.kernel.org/bpf/bpf-next/c/8143f960d915 - [bpf-next,v1,2/4] bpf: __bpf_fastcall for bpf_get_smp_processor_id in uapi https://git.kernel.org/bpf/bpf-next/c/f399d418c5f2 - [bpf-next,v1,3/4] bpf: use KF_FASTCALL to mark kfuncs supporting fastcall contract https://git.kernel.org/bpf/bpf-next/c/f49520133c3a - [bpf-next,v1,4/4] bpftool: __bpf_fastcall for kfuncs marked with special decl_tag https://git.kernel.org/bpf/bpf-next/c/00df87ce7eac
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html