Add libbpf full support for BPF-to-BPF calls
Subsystems (8)
v2 cover Add libbpf full support for BPF-to-BPF calls
Currently, libbpf supports a limited form of BPF-to-BPF subprogram calls. The restriction is that entry-point BPF program should use *all* of defined sub-programs in BPF .o file. If any of the subprograms is not used, such entry-point BPF program will be rejected by verifier as containing unreachable dead code. This is not a big limitation for cases with single entry-point BPF programs, but is quite a heavy restriction for multi-programs that use only partially overlapping set of subprograms.
This patch set removes all such restrictions and adds complete support for using BPF sub-program calls on BPF side. This is achieved through libbpf tracking subprograms individually and detecting which subprograms are used by any given entry-point BPF program, and subsequently only appending and relocating code for just those used subprograms.
In addition, libbpf now also supports multiple entry-point BPF programs within the same ELF section. This allows to structure code so that there are few variants of BPF programs of the same type and attaching to the same target (e.g., for tracepoints and kprobes) without the need to worry about ELF section name clashes.
This patch set opens way for more wider adoption of BPF subprogram calls, especially for real-world production use-cases with complicated net of subprograms. This will allow to further scale BPF verification process through good use of global functions, which can be verified independently. This is also important prerequisite for static linking which allows static BPF libraries to not worry about naming clashes for section names, as well as use static non-inlined functions (subprograms) without worries of verifier rejecting program due to dead code.
Patch set is structured as follows:
- patched 1-6 contain all the libbpf changes necessary to support multi-prog sections and bpf2bpf subcalls;
- patch 7 adds dedicated selftests validating all combinations of possible sub-calls (within and across sections, static vs global functions);
- patch 8 deprecated bpf_program__title() in favor of bpf_program__section_name(). The intent was to also deprecate bpf_object__find_program_by_title() as it's now non-sensical with multiple programs per section. But there were too many selftests uses of this and I didn't want to delay this patches further and make it even bigger, so left it for a follow up cleanup;
- patches 9-10 remove uses for title-related APIs from bpftool and bpf_program__title() use from selftests;
- patch 11 is converting fexit_bpf2bpf to have explicit subtest (it does contain 4 subtests, which are not handled as sub-tests);
- patches 12-14 convert few complicated BPF selftests to use __noinline functions to further validate correctness of libbpf's bpf2bpf processing logic.
v1->v2:
- rename DEPRECATED to LIBBPF_DEPRECATED to avoid name clashes;
- fix test_subprogs build;
- convert a bunch of complicated selftests to __noinline (Alexei).
Andrii Nakryiko (14):
libbpf: ensure ELF symbols table is found before further ELF
processing
libbpf: parse multi-function sections into multiple BPF programs
libbpf: support CO-RE relocations for multi-prog sections
libbpf: make RELO_CALL work for multi-prog sections and sub-program
calls
libbpf: implement generalized .BTF.ext func/line info adjustment
libbpf: add multi-prog section support for struct_ops selftests/bpf: add selftest for multi-prog sections and bpf-to-bpf
calls
tools/bpftool: replace bpf_program__title() with
bpf_program__section_name()
selftests/bpf: don't use deprecated libbpf APIs
libbpf: deprecate notion of BPF program "title" in favor of "section
name"
selftests/bpf: turn fexit_bpf2bpf into test with subtests selftests/bpf: convert pyperf, strobemeta, and l4lb_noinline to
__noinline
selftests/bpf: modernize xdp_noinline test w/ skeleton and __noinline selftests/bpf: convert cls_redirect selftest to use __noinline
tools/bpf/bpftool/prog.c | 4 +- tools/lib/bpf/btf.h | 18 +- tools/lib/bpf/libbpf.c | 1198 +++++++++++------ tools/lib/bpf/libbpf.h | 5 +- tools/lib/bpf/libbpf.map | 1 + tools/lib/bpf/libbpf_common.h | 2 + .../selftests/bpf/flow_dissector_load.h | 8 +- .../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 21 +- .../selftests/bpf/prog_tests/l4lb_all.c | 9 +- .../bpf/prog_tests/reference_tracking.c | 2 +- .../selftests/bpf/prog_tests/subprogs.c | 31 + .../selftests/bpf/prog_tests/xdp_noinline.c | 49 +- tools/testing/selftests/bpf/progs/pyperf.h | 10 +- .../testing/selftests/bpf/progs/strobemeta.h | 15 +- .../selftests/bpf/progs/test_cls_redirect.c | 99 +- .../selftests/bpf/progs/test_l4lb_noinline.c | 41 +- .../selftests/bpf/progs/test_subprogs.c | 103 ++ .../selftests/bpf/progs/test_xdp_noinline.c | 36 +- .../selftests/bpf/test_socket_cookie.c | 2 +- 19 files changed, 1054 insertions(+), 600 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/subprogs.c create mode 100644 tools/testing/selftests/bpf/progs/test_subprogs.c
Signature · 1 line
2.24.1
PATCH v2 1/14 libbpf: ensure ELF symbols table is found before further ELF processing
libbpf ELF parsing logic might need symbols available before ELF parsing is completed, so we need to make sure that symbols table section is found in a separate pass before all the subsequent sections are processed.
--- tools/lib/bpf/libbpf.c | 40 +++++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 11 deletions(-)
tools/lib/bpf/libbpf.c +29 -11
@@ -2720,14 +2720,38 @@ static int bpf_object__elf_collect(struct bpf_object *obj)Elf *elf = obj->efile.elf;Elf_Data *btf_ext_data =NULL ;Elf_Data *btf_data =NULL ; -Elf_Scn *scn =NULL ;int idx =0 , err =0 ; +const char *name; +Elf_Data *data; +Elf_Scn *scn; +GElf_Shdr sh; +/* a bunch of ELF parsing functionality depends on processing symbols, +* so do the first pass and find the symbol table +*/ + scn =NULL ;while ((scn =elf_nextscn (elf, scn)) !=NULL ) { -const char *name; -GElf_Shdr sh; -Elf_Data *data; +if (elf_sec_hdr (obj, scn, &sh)) +return -LIBBPF_ERRNO__FORMAT ; + +if (sh.sh_type ==SHT_SYMTAB ) { +if (obj->efile.symbols) { +pr_warn ("elf: multiple symbol tables in %s\n" , obj->path); +return -LIBBPF_ERRNO__FORMAT ; + } + data =elf_sec_data (obj, scn); +if (!data) +return -LIBBPF_ERRNO__FORMAT ; + + obj->efile.symbols = data; + obj->efile.symbols_shndx =elf_ndxscn (scn); + obj->efile.strtabidx = sh.sh_link; + } + } + + scn =NULL ; +while ((scn =elf_nextscn (elf, scn)) !=NULL ) { idx++;if (elf_sec_hdr (obj, scn, &sh))
@@ -2766,13 +2790,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj) }else if (strcmp (name,BTF_EXT_ELF_SEC ) ==0 ) { btf_ext_data = data; }else if (sh.sh_type ==SHT_SYMTAB ) { -if (obj->efile.symbols) { -pr_warn ("elf: multiple symbol tables in %s\n" , obj->path); -return -LIBBPF_ERRNO__FORMAT ; - } - obj->efile.symbols = data; - obj->efile.symbols_shndx = idx; - obj->efile.strtabidx = sh.sh_link; +/* already processed during the first pass above */ }else if (sh.sh_type ==SHT_PROGBITS && data->d_size >0 ) {if (sh.sh_flags &SHF_EXECINSTR ) {if (strcmp (name,".text" ) ==0 )
Signature · 1 line
2.24.1
RE: [PATCH v2 bpf-next 01/14] libbpf: ensure ELF symbols table is found before further ELF processing
Quoted 6 lines · Andrii Nakryiko
libbpf ELF parsing logic might need symbols available before ELF parsing is completed, so we need to make sure that symbols table section is found in a separate pass before all the subsequent sections are processed.
Signed-off-by Andrii Nakryiko <[email protected]>---
LGTM
PATCH v2 2/14 libbpf: parse multi-function sections into multiple BPF programs
In reply to Andrii NakryikoTeach libbpf how to parse code sections into potentially multiple bpf_program instances, based on ELF FUNC symbols. Each BPF program will keep track of its position within containing ELF section for translating section instruction offsets into program instruction offsets: regardless of BPF program's location in ELF section, it's first instruction is always at local instruction offset 0, so when libbpf is working with relocations (which use section-based instruction offsets) this is critical to make proper translations.
--- tools/lib/bpf/libbpf.c | 249 +++++++++++++++++++++++------------------ 1 file changed, 142 insertions(+), 107 deletions(-)
tools/lib/bpf/libbpf.c +142 -107
@@ -217,20 +217,45 @@ struct bpf_sec_def {* linux/filter.h. */ struct bpf_program { -/* Index in elf obj file, for relocation use. */ -int idx; -char *name; -int prog_ifindex; -char *section_name;const struct bpf_sec_def *sec_def; +char *section_name; +size_t sec_idx; +/* this program's instruction offset (in number of instructions) +* within its containing ELF section +*/ +size_t sec_insn_off; +/* number of original instructions in ELF section belonging to this +* program, not taking into account subprogram instructions possible +* appended later during relocation +*/ +size_t sec_insn_cnt; +/* Offset (in number of instructions) of the start of instruction +* belonging to this BPF program within its containing main BPF +* program. For the entry-point (main) BPF program, this is always +* zero. For a sub-program, this gets reset before each of main BPF +* programs are processed and relocated and is used to determined +* whether sub-program was already appended to the main program, and +* if yes, at which instruction offset. +*/ +size_t sub_insn_off; + +char *name;/* section_name with / replaced by _; makes recursive pinning * in bpf_object__pin_programs easier */ char *pin_name; + +/* instructions that belong to BPF program; insns[0] is located at +* sec_insn_off instruction within its ELF section in ELF file, so +* when mapping ELF file instruction index to the local instruction, +* one needs to subtract sec_insn_off; and vice versa. +*/ struct bpf_insn *insns; +/* actual number of instruction in this BPF program's image; for +* entry-point BPF programs this includes the size of main program +* itself plus all the used sub-programs, appended at the end +*/ size_t insns_cnt, main_prog_cnt; -enum bpf_prog_type type; -bool load;struct reloc_desc *reloc_desc;int nr_reloc;
@@ -246,7 +271,10 @@ struct bpf_program {void *priv;bpf_program_clear_priv_t clear_priv; +bool load; +enum bpf_prog_type type;enum bpf_attach_type expected_attach_type; +int prog_ifindex;__u32 attach_btf_id;__u32 attach_prog_fd;void *func_info;
@@ -446,6 +474,8 @@ static Elf_Scn *elf_sec_by_name(const struct bpf_object *obj, const char *name);static int elf_sec_hdr (const struct bpf_object *obj,Elf_Scn *scn,GElf_Shdr *hdr);static const char *elf_sec_name (const struct bpf_object *obj,Elf_Scn *scn);static Elf_Data *elf_sec_data (const struct bpf_object *obj,Elf_Scn *scn); +static int elf_sym_by_sec_off (const struct bpf_object *obj,size_t sec_idx, +size_t off,__u32 sym_type,GElf_Sym *sym);void bpf_program__unload (struct bpf_program *prog) {
@@ -493,7 +523,7 @@ static void bpf_program__exit(struct bpf_program *prog) prog->nr_reloc =0 ; prog->insns_cnt =0 ; - prog->idx =-1 ; + prog->sec_idx =-1 ; }static char *__bpf_program__pin_name (struct bpf_program *prog)
@@ -508,130 +538,118 @@ static char *__bpf_program__pin_name(struct bpf_program *prog) }static int -bpf_program__init (void *data,size_t size,const char *section_name,int idx, -struct bpf_program *prog) +bpf_program__init (struct bpf_program *prog,const char *name, +size_t sec_idx,const char *sec_name,size_t sec_off, +void *insn_data,size_t insn_data_sz) { -const size_t bpf_insn_sz =sizeof (struct bpf_insn ); - -if (size ==0 || size % bpf_insn_sz) { -pr_warn ("corrupted section '%s', size: %zu\n" , - section_name, size); +if (insn_data_sz ==0 || insn_data_sz %BPF_INSN_SZ || sec_off %BPF_INSN_SZ ) { +pr_warn ("sec '%s': corrupted program '%s', offset %zu, size %zu\n" , + sec_name, name, sec_off, insn_data_sz);return -EINVAL ; } -memset (prog,0 ,sizeof (*prog)); + prog->sec_idx = sec_idx; + prog->sec_insn_off = sec_off /BPF_INSN_SZ ; + prog->sec_insn_cnt = insn_data_sz /BPF_INSN_SZ ; +/* insns_cnt can later be increased by appending used subprograms */ + prog->insns_cnt = prog->sec_insn_cnt; - prog->section_name =strdup (section_name); -if (!prog->section_name) { -pr_warn ("failed to alloc name for prog under section(%d) %s\n" , - idx, section_name); + prog->type =BPF_PROG_TYPE_UNSPEC ; + prog->load = true; + + prog->instances.fds =NULL ; + prog->instances.nr =-1 ; + + prog->section_name =strdup (sec_name); +if (!prog->section_name) + goto errout; + + prog->name =strdup (name); +if (!prog->name) goto errout; - } prog->pin_name =__bpf_program__pin_name (prog); -if (!prog->pin_name) { -pr_warn ("failed to alloc pin name for prog under section(%d) %s\n" , - idx, section_name); +if (!prog->pin_name) goto errout; - } - prog->insns =malloc (size); -if (!prog->insns) { -pr_warn ("failed to alloc insns for prog under section %s\n" , - section_name); + prog->insns =malloc (insn_data_sz); +if (!prog->insns) goto errout; - } - prog->insns_cnt = size / bpf_insn_sz; -memcpy (prog->insns, data, size); - prog->idx = idx; - prog->instances.fds =NULL ; - prog->instances.nr =-1 ; - prog->type =BPF_PROG_TYPE_UNSPEC ; - prog->load = true; +memcpy (prog->insns, insn_data, insn_data_sz);return 0 ; errout: +pr_warn ("sec '%s': failed to allocate memory for prog '%s'\n" , sec_name, name);bpf_program__exit (prog);return -ENOMEM ; }static int -bpf_object__add_program (struct bpf_object *obj,void *data,size_t size, -const char *section_name,int idx) +bpf_object__add_programs (struct bpf_object *obj,Elf_Data *sec_data, +const char *sec_name,int sec_idx) { -struct bpf_program prog, *progs; +struct bpf_program *prog, *progs; +void *data = sec_data->d_buf; +size_t sec_sz = sec_data->d_size, sec_off, prog_sz;int nr_progs, err; - - err =bpf_program__init (data, size, section_name, idx, &prog); -if (err) -return err; +const char *name; +GElf_Sym sym; progs = obj->programs; nr_progs = obj->nr_programs; + sec_off =0 ; - progs =libbpf_reallocarray (progs, nr_progs +1 ,sizeof (progs[0 ])); -if (!progs) { -/* -* In this case the original obj->programs -* is still valid, so don't need special treat for -* bpf_close_object(). -*/ -pr_warn ("failed to alloc a new program under section '%s'\n" , - section_name); -bpf_program__exit (&prog); -return -ENOMEM ; - } - -pr_debug ("elf: found program '%s'\n" , prog.section_name); - obj->programs = progs; - obj->nr_programs = nr_progs +1 ; - prog.obj = obj; - progs[nr_progs] = prog; -return 0 ; -} - -static int -bpf_object__init_prog_names (struct bpf_object *obj) -{ -Elf_Data *symbols = obj->efile.symbols; -struct bpf_program *prog; -size_t pi, si; +while (sec_off < sec_sz) { +if (elf_sym_by_sec_off (obj, sec_idx, sec_off,STT_FUNC , &sym)) { +pr_warn ("sec '%s': failed to find program symbol at offset %zu\n" , + sec_name, sec_off); +return -LIBBPF_ERRNO__FORMAT ; + } -for (pi =0 ; pi < obj->nr_programs; pi++) { -const char *name =NULL ; + prog_sz = sym.st_size; - prog = &obj->programs[pi]; + name =elf_sym_str (obj, sym.st_name); +if (!name) { +pr_warn ("sec '%s': failed to get symbol name for offset %zu\n" , + sec_name, sec_off); +return -LIBBPF_ERRNO__FORMAT ; + } -for (si =0 ; si < symbols->d_size /sizeof (GElf_Sym) && !name; si++) { -GElf_Sym sym; +if (sec_off + prog_sz > sec_sz) { +pr_warn ("sec '%s': program at offset %zu crosses section boundary\n" , + sec_name, sec_off); +return -LIBBPF_ERRNO__FORMAT ; + } -if (!gelf_getsym (symbols, si, &sym)) -continue ; -if (sym.st_shndx != prog->idx) -continue ; -if (GELF_ST_BIND (sym.st_info) !=STB_GLOBAL ) -continue ; +pr_debug ("sec '%s': found program '%s' at offset %zu, code size %zu bytes\n" , + sec_name, name, sec_off, prog_sz); - name =elf_sym_str (obj, sym.st_name); -if (!name) { -pr_warn ("prog '%s': failed to get symbol name\n" , - prog->section_name); -return -LIBBPF_ERRNO__LIBELF ; - } + progs =reallocarray (progs, nr_progs +1 ,sizeof (*progs)); +if (!progs) { +/* +* In this case the original obj->programs +* is still valid, so don't need special treat for +* bpf_close_object(). +*/ +pr_warn ("sec '%s': failed to alloc memory for new program '%s'\n" , + sec_name, name); +return -ENOMEM ; } + obj->programs = progs; -if (!name && prog->idx == obj->efile.text_shndx) - name =".text" ; + prog = &progs[nr_progs]; +memset (prog,0 ,sizeof (*prog)); + prog->obj = obj; -if (!name) { -pr_warn ("prog '%s': failed to find program symbol\n" , - prog->section_name); -return -EINVAL ; - } + err =bpf_program__init (prog, name, sec_idx, sec_name, sec_off, + data + sec_off, prog_sz); +if (err) +return err; - prog->name =strdup (name); -if (!prog->name) -return -ENOMEM ; + nr_progs++; + obj->nr_programs = nr_progs; + + sec_off += prog_sz; }return 0 ;
@@ -2675,6 +2693,26 @@ static Elf_Data *elf_sec_data(const struct bpf_object *obj, Elf_Scn *scn)return data; } +static int elf_sym_by_sec_off (const struct bpf_object *obj,size_t sec_idx, +size_t off,__u32 sym_type,GElf_Sym *sym) +{ +Elf_Data *symbols = obj->efile.symbols; +size_t n = symbols->d_size /sizeof (GElf_Sym); +int i; + +for (i =0 ; i < n; i++) { +if (!gelf_getsym (symbols, i, sym)) +continue ; +if (sym->st_shndx != sec_idx || sym->st_value != off) +continue ; +if (GELF_ST_TYPE (sym->st_info) != sym_type) +continue ; +return 0 ; + } + +return -ENOENT ; +} +static bool is_sec_name_dwarf (const char *name) {/* approximation, but the actual list is too long */
@@ -2795,9 +2833,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj)if (sh.sh_flags &SHF_EXECINSTR ) {if (strcmp (name,".text" ) ==0 ) obj->efile.text_shndx = idx; - err =bpf_object__add_program (obj, data->d_buf, - data->d_size, - name, idx); + err =bpf_object__add_programs (obj, data, name, idx);if (err)return err; }else if (strcmp (name,DATA_SEC ) ==0 ) {
@@ -3183,7 +3219,7 @@ bpf_object__find_prog_by_idx(struct bpf_object *obj, int idx)for (i =0 ; i < obj->nr_programs; i++) { prog = &obj->programs[i]; -if (prog->idx == idx) +if (prog->sec_idx == idx)return prog; }return NULL ;
@@ -5660,7 +5696,7 @@ bpf_program__reloc_text(struct bpf_program *prog, struct bpf_object *obj,size_t new_cnt;int err; -if (prog->idx != obj->efile.text_shndx && prog->main_prog_cnt ==0 ) { +if (prog->sec_idx != obj->efile.text_shndx && prog->main_prog_cnt ==0 ) { text =bpf_object__find_prog_by_idx (obj, obj->efile.text_shndx);if (!text) {pr_warn ("no .text section found yet relo into text exist\n" );
@@ -5783,7 +5819,7 @@ bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_path)*/ for (i =0 ; i < obj->nr_programs; i++) { prog = &obj->programs[i]; -if (prog->idx != obj->efile.text_shndx) +if (prog->sec_idx != obj->efile.text_shndx)continue ; err =bpf_program__relocate (prog, obj);
@@ -5799,7 +5835,7 @@ bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_path)*/ for (i =0 ; i < obj->nr_programs; i++) { prog = &obj->programs[i]; -if (prog->idx == obj->efile.text_shndx) +if (prog->sec_idx == obj->efile.text_shndx)continue ; err =bpf_program__relocate (prog, obj);
@@ -6215,7 +6251,7 @@ int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver)static bool bpf_program__is_function_storage (const struct bpf_program *prog,const struct bpf_object *obj) { -return prog->idx == obj->efile.text_shndx && obj->has_pseudo_calls; +return prog->sec_idx == obj->efile.text_shndx && obj->has_pseudo_calls; } static int
@@ -6298,7 +6334,6 @@ __bpf_object__open(const char *path, const void *obj_buf, size_t obj_buf_sz, err = err ? :bpf_object__collect_externs (obj); err = err ? :bpf_object__finalize_btf (obj); err = err ? :bpf_object__init_maps (obj, opts); - err = err ? :bpf_object__init_prog_names (obj); err = err ? :bpf_object__collect_reloc (obj);if (err) goto out;
Signature · 1 line
2.24.1
RE: [PATCH v2 bpf-next 02/14] libbpf: parse multi-function sections into multiple BPF programs
Quoted 10 lines · Andrii Nakryiko
Teach libbpf how to parse code sections into potentially multiple
bpf_programinstances, based on ELF FUNC symbols. Each BPF program will keep track of its position within containing ELF section for translating section instruction offsets into program instruction offsets: regardless of BPF program's location in ELF section, it's first instruction is always at local instruction offset 0, so when libbpf is working with relocations (which use section-based instruction offsets) this is critical to make proper translations.Signed-off-by Andrii Nakryiko <[email protected]>---
PATCH v2 3/14 libbpf: support CO-RE relocations for multi-prog sections
In reply to Andrii NakryikoFix up CO-RE relocation code to handle relocations against ELF sections containing multiple BPF programs. This requires lookup of a BPF program by its section name and instruction index it contains. While it could have been done as a simple loop, it could run into performance issues pretty quickly, as number of CO-RE relocations can be quite large in real-world applications, and each CO-RE relocation incurs BPF program look up now. So instead of simple loop, implement a binary search by section name + insn offset.
--- tools/lib/bpf/libbpf.c | 82 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 74 insertions(+), 8 deletions(-)
tools/lib/bpf/libbpf.c +74 -8
@@ -2753,6 +2753,18 @@ static bool ignore_elf_section(GElf_Shdr *hdr, const char *name)return false; } +static int cmp_progs (const void *_a,const void *_b) +{ +const struct bpf_program *a = _a; +const struct bpf_program *b = _b; + +if (a->sec_idx != b->sec_idx) +return a->sec_idx < b->sec_idx ?-1 :1 ; + +/* sec_insn_off can't be the same within the section */ +return a->sec_insn_off < b->sec_insn_off ?-1 :1 ; +} +static int bpf_object__elf_collect (struct bpf_object *obj) {Elf *elf = obj->efile.elf;
@@ -2887,6 +2899,11 @@ static int bpf_object__elf_collect(struct bpf_object *obj)pr_warn ("elf: symbol strings section missing or invalid in %s\n" , obj->path);return -LIBBPF_ERRNO__FORMAT ; } + +/* sort BPF programs by section name and in-section instruction offset +* for faster search */ +qsort (obj->programs, obj->nr_programs,sizeof (*obj->programs), cmp_progs); +return bpf_object__init_btf (obj, btf_data, btf_ext_data); }
@@ -3415,6 +3432,37 @@ static int bpf_program__record_reloc(struct bpf_program *prog,return 0 ; } +static bool prog_contains_insn (const struct bpf_program *prog,size_t insn_idx) +{ +return insn_idx >= prog->sec_insn_off && + insn_idx < prog->sec_insn_off + prog->sec_insn_cnt; +} + +static struct bpf_program *find_prog_by_sec_insn (const struct bpf_object *obj, +size_t sec_idx,size_t insn_idx) +{ +int l =0 , r = obj->nr_programs -1 , m; +struct bpf_program *prog; + +while (l < r) { + m = l + (r - l +1 ) /2 ; + prog = &obj->programs[m]; + +if (prog->sec_idx < sec_idx || + (prog->sec_idx == sec_idx && prog->sec_insn_off <= insn_idx)) + l = m; +else + r = m -1 ; + } +/* matching program could be at index l, but it still might be the +* wrong one, so we need to double check conditions for the last time +*/ + prog = &obj->programs[l]; +if (prog->sec_idx == sec_idx &&prog_contains_insn (prog, insn_idx)) +return prog; +return NULL ; +} +static int bpf_program__collect_reloc (struct bpf_program *prog,GElf_Shdr *shdr,Elf_Data *data,struct bpf_object *obj)
@@ -5229,6 +5277,11 @@ static int bpf_core_patch_insn(struct bpf_program *prog,if (relo->insn_off %BPF_INSN_SZ )return -EINVAL ; insn_idx = relo->insn_off /BPF_INSN_SZ ; +/* adjust insn_idx from section frame of reference to the local +* program's frame of reference; (sub-)program code is not yet +* relocated, so it's enough to just subtract in-section offset +*/ + insn_idx = insn_idx - prog->sec_insn_off; insn = &prog->insns[insn_idx]; class =BPF_CLASS (insn->code);
@@ -5619,7 +5672,7 @@ bpf_object__relocate_core(struct bpf_object *obj, const char *targ_btf_path)struct bpf_program *prog;struct btf *targ_btf;const char *sec_name; -int i, err =0 ; +int i, err =0 , insn_idx, sec_idx;if (obj->btf_ext->core_relo_info.len ==0 )return 0 ;
@@ -5646,24 +5699,37 @@ bpf_object__relocate_core(struct bpf_object *obj, const char *targ_btf_path) err = -EINVAL ; goto out; } +/* bpf_object's ELF is gone by now so it's not easy to find +* section index by section name, but we can find *any* +* bpf_program within desired section name and use it's +* prog->sec_idx to do a proper search by section index and +* instruction offset +*/ prog =NULL ;for (i =0 ; i < obj->nr_programs; i++) { -if (!strcmp (obj->programs[i].section_name, sec_name)) { - prog = &obj->programs[i]; + prog = &obj->programs[i]; +if (strcmp (prog->section_name, sec_name) ==0 )break ; - } }if (!prog) { -pr_warn ("failed to find program '%s' for CO-RE offset relocation\n" , - sec_name); - err = -EINVAL ; - goto out; +pr_warn ("sec '%s': failed to find a BPF program\n" , sec_name); +return -ENOENT ; } + sec_idx = prog->sec_idx;pr_debug ("sec '%s': found %d CO-RE relocations\n" , sec_name, sec->num_info);for_each_btf_ext_rec (seg, sec, i, rec) { + insn_idx = rec->insn_off /BPF_INSN_SZ ; + prog =find_prog_by_sec_insn (obj, sec_idx, insn_idx); +if (!prog) { +pr_warn ("sec '%s': failed to find program at insn #%d for CO-RE offset relocation #%d\n" , + sec_name, insn_idx, i); + err = -EINVAL ; + goto out; + } + err =bpf_core_apply_relo (prog, rec, i, obj->btf, targ_btf, cand_cache);if (err) {
Signature · 1 line
2.24.1
RE: [PATCH v2 bpf-next 03/14] libbpf: support CO-RE relocations for multi-prog sections
Quoted 8 lines · Andrii Nakryiko
Fix up CO-RE relocation code to handle relocations against ELF sections containing multiple BPF programs. This requires lookup of a BPF program by its section name and instruction index it contains. While it could have been done as a simple loop, it could run into performance issues pretty quickly, as number of CO-RE relocations can be quite large in real-world applications, and each CO-RE relocation incurs BPF program look up now. So instead of simple loop, implement a binary search by section name + insn offset.
Quoted 5 lines
Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/libbpf.c | 82 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 74 insertions(+), 8 deletions(-)
PATCH v2 4/14 libbpf: make RELO_CALL work for multi-prog sections and sub-program calls
In reply to Andrii NakryikoThis patch implements general and correct logic for bpf-to-bpf sub-program calls. Only sub-programs used (called into) from entry-point (main) BPF program are going to be appended at the end of main BPF program. This ensures that BPF verifier won't encounter any dead code due to copying unreferenced sub-program. This change means that each entry-point (main) BPF program might have a different set of sub-programs appended to it and potentially in different order. This has implications on how sub-program call relocations need to be handled, described below.
All relocations are now split into two categores: data references (maps and global variables) and code references (sub-program calls). This distinction is important because data references need to be relocated just once per each BPF program and sub-program. These relocation are agnostic to instruction locations, because they are not code-relative and they are relocating against static targets (maps, variables with fixes offsets, etc).
Sub-program RELO_CALL relocations, on the other hand, are highly-dependent on code position, because they are recorded as instruction-relative offset. So BPF sub-programs (those that do calls into other sub-programs) can't be relocated once, they need to be relocated each time such a sub-program is appended at the end of the main entry-point BPF program. As mentioned above, each main BPF program might have different subset and differen order of sub-programs, so call relocations can't be done just once. Splitting data reference and calls relocations as described above allows to do this efficiently and cleanly.
bpf_object__find_program_by_name() will now ignore non-entry BPF programs. Previously one could have looked up '.text' fake BPF program, but the existence of such BPF program was always an implementation detail and you can't do much useful with it. Now, though, all non-entry sub-programs get their own BPF program with name corresponding to a function name, so there is no more '.text' name for BPF program. This means there is no regression, effectively, w.r.t. API behavior. But this is important aspect to highlight, because it's going to be critical once libbpf implements static linking of BPF programs. Non-entry static BPF programs will be allowed to have conflicting names, but global and main-entry BPF program names should be unique. Just like with normal user-space linking process. So it's important to restrict this aspect right now, keep static and non-entry functions as internal implementation details, and not have to deal with regressions in behavior later.
This patch leaves .BTF.ext adjustment as is until next patch.
--- tools/lib/bpf/libbpf.c | 429 ++++++++++++++++++++++++++++------------- 1 file changed, 291 insertions(+), 138 deletions(-)
tools/lib/bpf/libbpf.c +291 -138
@@ -193,6 +193,7 @@ struct reloc_desc {int insn_idx;int map_idx;int sym_off; +bool processed; };struct bpf_sec_def ;
@@ -255,7 +256,7 @@ struct bpf_program {* entry-point BPF programs this includes the size of main program * itself plus all the used sub-programs, appended at the end */ -size_t insns_cnt, main_prog_cnt; +size_t insns_cnt;struct reloc_desc *reloc_desc;int nr_reloc;
@@ -412,7 +413,7 @@ struct bpf_object {int kconfig_map_idx;bool loaded; -bool has_pseudo_calls; +bool has_subcalls;/* * Information when doing elf related work. Only valid if fd
@@ -537,17 +538,32 @@ static char *__bpf_program__pin_name(struct bpf_program *prog)return name; } +static bool insn_is_subprog_call (const struct bpf_insn *insn) +{ +return BPF_CLASS (insn->code) ==BPF_JMP && +BPF_OP (insn->code) ==BPF_CALL && +BPF_SRC (insn->code) ==BPF_K && + insn->src_reg ==BPF_PSEUDO_CALL && + insn->dst_reg ==0 && + insn->off ==0 ; +} +static int -bpf_program__init (struct bpf_program *prog,const char *name, -size_t sec_idx,const char *sec_name,size_t sec_off, -void *insn_data,size_t insn_data_sz) +bpf_object__init_prog (struct bpf_object *obj,struct bpf_program *prog, +const char *name,size_t sec_idx,const char *sec_name, +size_t sec_off,void *insn_data,size_t insn_data_sz) { +int i; +if (insn_data_sz ==0 || insn_data_sz %BPF_INSN_SZ || sec_off %BPF_INSN_SZ ) {pr_warn ("sec '%s': corrupted program '%s', offset %zu, size %zu\n" , sec_name, name, sec_off, insn_data_sz);return -EINVAL ; } +memset (prog,0 ,sizeof (*prog)); + prog->obj = obj; + prog->sec_idx = sec_idx; prog->sec_insn_off = sec_off /BPF_INSN_SZ ; prog->sec_insn_cnt = insn_data_sz /BPF_INSN_SZ ;
@@ -577,6 +593,13 @@ bpf_program__init(struct bpf_program *prog, const char *name, goto errout;memcpy (prog->insns, insn_data, insn_data_sz); +for (i =0 ; i < prog->insns_cnt; i++) { +if (insn_is_subprog_call (&prog->insns[i])) { + obj->has_subcalls = true; +break ; + } + } +return 0 ; errout:pr_warn ("sec '%s': failed to allocate memory for prog '%s'\n" , sec_name, name);
@@ -621,10 +644,10 @@ bpf_object__add_programs(struct bpf_object *obj, Elf_Data *sec_data,return -LIBBPF_ERRNO__FORMAT ; } -pr_debug ("sec '%s': found program '%s' at offset %zu, code size %zu bytes\n" , - sec_name, name, sec_off, prog_sz); +pr_debug ("sec '%s': found program '%s' at insn offset %zu (%zu bytes), code size %zu insns (%zu bytes)\n" , + sec_name, name, sec_off /BPF_INSN_SZ , sec_off, prog_sz /BPF_INSN_SZ , prog_sz); - progs =reallocarray (progs, nr_progs +1 ,sizeof (*progs)); + progs =libbpf_reallocarray (progs, nr_progs +1 ,sizeof (*progs));if (!progs) {/* * In this case the original obj->programs
@@ -638,11 +661,9 @@ bpf_object__add_programs(struct bpf_object *obj, Elf_Data *sec_data, obj->programs = progs; prog = &progs[nr_progs]; -memset (prog,0 ,sizeof (*prog)); - prog->obj = obj; - err =bpf_program__init (prog, name, sec_idx, sec_name, sec_off, - data + sec_off, prog_sz); + err =bpf_object__init_prog (obj, prog, name, sec_idx, sec_name, + sec_off, data + sec_off, prog_sz);if (err)return err;
@@ -3255,6 +3276,12 @@ bpf_object__find_program_by_title(const struct bpf_object *obj,return NULL ; } +static bool prog_is_subprog (const struct bpf_object *obj, +const struct bpf_program *prog) +{ +return prog->sec_idx == obj->efile.text_shndx && obj->has_subcalls; +} +struct bpf_program *bpf_object__find_program_by_name (const struct bpf_object *obj,const char *name)
@@ -3262,6 +3289,8 @@ bpf_object__find_program_by_name(const struct bpf_object *obj,struct bpf_program *prog;bpf_object__for_each_program (prog, obj) { +if (prog_is_subprog (obj, prog)) +continue ;if (!strcmp (prog->name, name))return prog; }
@@ -3311,6 +3340,8 @@ static int bpf_program__record_reloc(struct bpf_program *prog,const char *sym_sec_name;struct bpf_map *map; + reloc_desc->processed = false; +/* sub-program call relocation */ if (insn->code == (BPF_JMP |BPF_CALL )) {if (insn->src_reg !=BPF_PSEUDO_CALL ) {
@@ -3332,7 +3363,6 @@ static int bpf_program__record_reloc(struct bpf_program *prog, reloc_desc->type =RELO_CALL ; reloc_desc->insn_idx = insn_idx; reloc_desc->sym_off = sym->st_value; - obj->has_pseudo_calls = true;return 0 ; }
@@ -3464,13 +3494,18 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, }static int -bpf_program__collect_reloc (struct bpf_program *prog,GElf_Shdr *shdr, -Elf_Data *data,struct bpf_object *obj) +bpf_object__collect_prog_relos (struct bpf_object *obj,GElf_Shdr *shdr,Elf_Data *data) {Elf_Data *symbols = obj->efile.symbols;const char *relo_sec_name, *sec_name;size_t sec_idx = shdr->sh_info; +struct bpf_program *prog; +struct reloc_desc *relos;int err, i, nrels; +const char *sym_name; +__u32 insn_idx; +GElf_Sym sym; +GElf_Rel rel; relo_sec_name =elf_sec_str (obj, shdr->sh_name); sec_name =elf_sec_name (obj,elf_sec_by_idx (obj, sec_idx));
@@ -3481,19 +3516,7 @@ bpf_program__collect_reloc(struct bpf_program *prog, GElf_Shdr *shdr, relo_sec_name, sec_idx, sec_name); nrels = shdr->sh_size / shdr->sh_entsize; - prog->reloc_desc =malloc (sizeof (*prog->reloc_desc) * nrels); -if (!prog->reloc_desc) { -pr_warn ("failed to alloc memory in relocation\n" ); -return -ENOMEM ; - } - prog->nr_reloc = nrels; -for (i =0 ; i < nrels; i++) { -const char *sym_name; -__u32 insn_idx; -GElf_Sym sym; -GElf_Rel rel; -if (!gelf_getrel (data, i, &rel)) {pr_warn ("sec '%s': failed to get relo #%d\n" , relo_sec_name, i);return -LIBBPF_ERRNO__FORMAT ;
@@ -3510,15 +3533,42 @@ bpf_program__collect_reloc(struct bpf_program *prog, GElf_Shdr *shdr, } insn_idx = rel.r_offset /BPF_INSN_SZ ; - sym_name =elf_sym_str (obj, sym.st_name) ?:"<?>" ; +/* relocations against static functions are recorded as +* relocations against the section that contains a function; +* in such case, symbol will be STT_SECTION and sym.st_name +* will point to empty string (0), so fetch section name +* instead +*/ +if (GELF_ST_TYPE (sym.st_info) ==STT_SECTION && sym.st_name ==0 ) + sym_name =elf_sec_name (obj,elf_sec_by_idx (obj, sym.st_shndx)); +else + sym_name =elf_sym_str (obj, sym.st_name); + sym_name = sym_name ?:"<?" ;pr_debug ("sec '%s': relo #%d: insn #%u against '%s'\n" , relo_sec_name, i, insn_idx, sym_name); - err =bpf_program__record_reloc (prog, &prog->reloc_desc[i], + prog =find_prog_by_sec_insn (obj, sec_idx, insn_idx); +if (!prog) { +pr_warn ("sec '%s': relo #%d: program not found in section '%s' for insn #%u\n" , + relo_sec_name, i, sec_name, insn_idx); +return -LIBBPF_ERRNO__RELOC ; + } + + relos =libbpf_reallocarray (prog->reloc_desc, + prog->nr_reloc +1 ,sizeof (*relos)); +if (!relos) +return -ENOMEM ; + prog->reloc_desc = relos; + +/* adjust insn_idx to local BPF program frame of reference */ + insn_idx -= prog->sec_insn_off; + err =bpf_program__record_reloc (prog, &relos[prog->nr_reloc], insn_idx, sym_name, &sym, &rel);if (err)return err; + + prog->nr_reloc++; }return 0 ; }
@@ -5753,89 +5803,32 @@ bpf_object__relocate_core(struct bpf_object *obj, const char *targ_btf_path)return err; } +/* Relocate data references within program code: +* - map references; +* - global variable references; +* - extern references. +*/ static int -bpf_program__reloc_text (struct bpf_program *prog,struct bpf_object *obj, -struct reloc_desc *relo) -{ -struct bpf_insn *insn, *new_insn; -struct bpf_program *text; -size_t new_cnt; -int err; - -if (prog->sec_idx != obj->efile.text_shndx && prog->main_prog_cnt ==0 ) { - text =bpf_object__find_prog_by_idx (obj, obj->efile.text_shndx); -if (!text) { -pr_warn ("no .text section found yet relo into text exist\n" ); -return -LIBBPF_ERRNO__RELOC ; - } - new_cnt = prog->insns_cnt + text->insns_cnt; - new_insn =libbpf_reallocarray (prog->insns, new_cnt,sizeof (*insn)); -if (!new_insn) { -pr_warn ("oom in prog realloc\n" ); -return -ENOMEM ; - } - prog->insns = new_insn; - -if (obj->btf_ext) { - err =bpf_program_reloc_btf_ext (prog, obj, - text->section_name, - prog->insns_cnt); -if (err) -return err; - } - -memcpy (new_insn + prog->insns_cnt, text->insns, - text->insns_cnt *sizeof (*insn)); - prog->main_prog_cnt = prog->insns_cnt; - prog->insns_cnt = new_cnt; -pr_debug ("added %zd insn from %s to prog %s\n" , - text->insns_cnt, text->section_name, - prog->section_name); - } - - insn = &prog->insns[relo->insn_idx]; - insn->imm += relo->sym_off /8 + prog->main_prog_cnt - relo->insn_idx; -return 0 ; -} - -static int -bpf_program__relocate (struct bpf_program *prog,struct bpf_object *obj) +bpf_object__relocate_data (struct bpf_object *obj,struct bpf_program *prog) { -int i, err; - -if (!prog) -return 0 ; - -if (obj->btf_ext) { - err =bpf_program_reloc_btf_ext (prog, obj, - prog->section_name,0 ); -if (err) -return err; - } - -if (!prog->reloc_desc) -return 0 ; +int i;for (i =0 ; i < prog->nr_reloc; i++) {struct reloc_desc *relo = &prog->reloc_desc[i];struct bpf_insn *insn = &prog->insns[relo->insn_idx];struct extern_desc *ext; -if (relo->insn_idx +1 >= (int )prog->insns_cnt) { -pr_warn ("relocation out of range: '%s'\n" , - prog->section_name); -return -LIBBPF_ERRNO__RELOC ; - } -switch (relo->type) {case RELO_LD64 : insn[0 ].src_reg =BPF_PSEUDO_MAP_FD ; insn[0 ].imm = obj->maps[relo->map_idx].fd; + relo->processed = true;break ;case RELO_DATA : insn[0 ].src_reg =BPF_PSEUDO_MAP_VALUE ; insn[1 ].imm = insn[0 ].imm + relo->sym_off; insn[0 ].imm = obj->maps[relo->map_idx].fd; + relo->processed = true;break ;case RELO_EXTERN : ext = &obj->externs[relo->sym_off];
@@ -5847,11 +5840,10 @@ bpf_program__relocate(struct bpf_program *prog, struct bpf_object *obj) insn[0 ].imm = (__u32 )ext->ksym.addr; insn[1 ].imm = ext->ksym.addr >>32 ; } + relo->processed = true;break ;case RELO_CALL : - err =bpf_program__reloc_text (prog, obj, relo); -if (err) -return err; +/* will be handled as a follow up pass */ break ;default : pr_warn("prog '%s': relo #%d: bad relo type %d\n",
@@ -5860,8 +5852,155 @@ bpf_program__relocate(struct bpf_program *prog, struct bpf_object *obj) } } -zfree (&prog->reloc_desc); - prog->nr_reloc =0 ; +return 0 ; +} + +static int cmp_relo_by_insn_idx (const void *key,const void *elem) +{ +size_t insn_idx = *(const size_t *)key; +const struct reloc_desc *relo = elem; + +if (insn_idx == relo->insn_idx) +return 0 ; +return insn_idx < relo->insn_idx ?-1 :1 ; +} + +static struct reloc_desc *find_prog_insn_relo (const struct bpf_program *prog,size_t insn_idx) +{ +return bsearch (&insn_idx, prog->reloc_desc, prog->nr_reloc, +sizeof (*prog->reloc_desc), cmp_relo_by_insn_idx); +} + +static int +bpf_object__reloc_code (struct bpf_object *obj,struct bpf_program *main_prog, +struct bpf_program *prog) +{ +size_t sub_insn_idx, insn_idx, new_cnt; +struct bpf_program *subprog; +struct bpf_insn *insns, *insn; +struct reloc_desc *relo; +int err; + + err =reloc_prog_func_and_line_info (obj, main_prog, prog); +if (err) +return err; + +for (insn_idx =0 ; insn_idx < prog->sec_insn_cnt; insn_idx++) { + insn = &main_prog->insns[prog->sub_insn_off + insn_idx]; +if (!insn_is_subprog_call (insn)) +continue ; + + relo =find_prog_insn_relo (prog, insn_idx); +if (relo && relo->type !=RELO_CALL ) { +pr_warn ("prog '%s': unexpected relo for insn #%zu, type %d\n" , + prog->name, insn_idx, relo->type); +return -LIBBPF_ERRNO__RELOC ; + } +if (relo) { +/* sub-program instruction index is a combination of +* an offset of a symbol pointed to by relocation and +* call instruction's imm field; for global functions, +* call always has imm = -1, but for static functions +* relocation is against STT_SECTION and insn->imm +* points to a start of a static function +*/ + sub_insn_idx = relo->sym_off /BPF_INSN_SZ + insn->imm +1 ; + }else { +/* if subprogram call is to a static function within +* the same ELF section, there won't be any relocation +* emitted, but it also means there is no additional +* offset necessary, insns->imm is relative to +* instruction's original position within the section +*/ + sub_insn_idx = prog->sec_insn_off + insn_idx + insn->imm +1 ; + } + +/* we enforce that sub-programs should be in .text section */ + subprog =find_prog_by_sec_insn (obj, obj->efile.text_shndx, sub_insn_idx); +if (!subprog) { +pr_warn ("prog '%s': no .text section found yet sub-program call exists\n" , + prog->name); +return -LIBBPF_ERRNO__RELOC ; + } + +/* if subprogram hasn't been used in current main program, +* relocate it and append at the end of main program code +*/ +if (subprog->sub_insn_off ==0 ) { + subprog->sub_insn_off = main_prog->insns_cnt; + + new_cnt = main_prog->insns_cnt + subprog->insns_cnt; + insns =libbpf_reallocarray (main_prog->insns, new_cnt,sizeof (*insns)); +if (!insns) { +pr_warn ("prog '%s': failed to realloc prog code\n" , main_prog->name); +return -ENOMEM ; + } + main_prog->insns = insns; + main_prog->insns_cnt = new_cnt; + +memcpy (main_prog->insns + subprog->sub_insn_off, subprog->insns, + subprog->insns_cnt *sizeof (*insns)); + +pr_debug ("prog '%s': added %zu insns from sub-prog '%s'\n" , + main_prog->name, subprog->insns_cnt, subprog->name); + + err =bpf_object__reloc_code (obj, main_prog, subprog); +if (err) +return err; + } + +/* main_prog->insns memory could have been re-allocated, so +* calculate pointer again +*/ + insn = &main_prog->insns[prog->sub_insn_off + insn_idx]; +/* calculate correct instruction position within main prog */ + insn->imm = subprog->sub_insn_off - (prog->sub_insn_off + insn_idx) -1 ; + +if (relo) + relo->processed = true; + +pr_debug ("prog '%s': insn #%zu relocated, imm %d points to subprog '%s' (now at %zu offset)\n" , + prog->name, insn_idx, insn->imm, subprog->name, subprog->sub_insn_off); + } + +return 0 ; +} + +/* +* Relocate sub-program calls +*/ +static int +bpf_object__relocate_calls (struct bpf_object *obj,struct bpf_program *prog) +{ +struct bpf_program *subprog; +int i, j, err; + +if (obj->btf_ext) { + err =bpf_program_reloc_btf_ext (prog, obj, + prog->section_name,0 ); +if (err) +return err; + } + +/* mark all subprogs as not relocated (yet) within the context of +* current main program +*/ +for (i =0 ; i < obj->nr_programs; i++) { + subprog = &obj->programs[i]; +if (!prog_is_subprog (obj, subprog)) +continue ; + + subprog->sub_insn_off =0 ; +for (j =0 ; j < subprog->nr_reloc; j++) +if (subprog->reloc_desc[j].type ==RELO_CALL ) + subprog->reloc_desc[j].processed = false; + } + + err =bpf_object__reloc_code (obj, prog, prog); +if (err) +return err; + +return 0 ; }
@@ -5880,37 +6019,45 @@ bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_path)return err; } } -/* ensure .text is relocated first, as it's going to be copied as-is -* later for sub-program calls +/* relocate data references first for all programs and sub-programs, +* as they don't change relative to code locations, so subsequent +* subprogram processing won't need to re-calculate any of them */ for (i =0 ; i < obj->nr_programs; i++) { prog = &obj->programs[i]; -if (prog->sec_idx != obj->efile.text_shndx) -continue ; - - err =bpf_program__relocate (prog, obj); + err =bpf_object__relocate_data (obj, prog);if (err) {pr_warn ("prog '%s': failed to relocate data references: %d\n" , prog->name, err);return err; } -break ; } -/* now relocate everything but .text, which by now is relocated -* properly, so we can copy raw sub-program instructions as is safely +/* now relocate subprogram calls and append used subprograms to main +* programs; each copy of subprogram code needs to be relocated +* differently for each main program, because its code location might +* have changed */ for (i =0 ; i < obj->nr_programs; i++) { prog = &obj->programs[i]; -if (prog->sec_idx == obj->efile.text_shndx) +/* sub-program's sub-calls are relocated within the context of +* its main program only +*/ +if (prog_is_subprog (obj, prog))continue ; - err =bpf_program__relocate (prog, obj); + err =bpf_object__relocate_calls (obj, prog);if (err) {pr_warn ("prog '%s': failed to relocate calls: %d\n" , prog->name, err);return err; } } +/* free up relocation descriptors */ +for (i =0 ; i < obj->nr_programs; i++) { + prog = &obj->programs[i]; +zfree (&prog->reloc_desc); + prog->nr_reloc =0 ; + }return 0 ; }
@@ -6030,41 +6177,53 @@ static int bpf_object__collect_map_relos(struct bpf_object *obj,return 0 ; } -static int bpf_object__collect_reloc (struct bpf_object *obj) +static int cmp_relocs (const void *_a,const void *_b) { -int i, err; +const struct reloc_desc *a = _a; +const struct reloc_desc *b = _b; -if (!obj_elf_valid (obj)) { -pr_warn ("Internal error: elf object is closed\n" ); -return -LIBBPF_ERRNO__INTERNAL ; - } +if (a->insn_idx != b->insn_idx) +return a->insn_idx < b->insn_idx ?-1 :1 ; + +/* no two relocations should have the same insn_idx, but ... */ +if (a->type != b->type) +return a->type < b->type ?-1 :1 ; + +return 0 ; +} + +static int bpf_object__collect_relos (struct bpf_object *obj) +{ +int i, err;for (i =0 ; i < obj->efile.nr_reloc_sects; i++) {GElf_Shdr *shdr = &obj->efile.reloc_sects[i].shdr;Elf_Data *data = obj->efile.reloc_sects[i].data;int idx = shdr->sh_info; -struct bpf_program *prog;if (shdr->sh_type !=SHT_REL ) {pr_warn ("internal error at %d\n" , __LINE__);return -LIBBPF_ERRNO__INTERNAL ; } -if (idx == obj->efile.st_ops_shndx) { +if (idx == obj->efile.st_ops_shndx) err =bpf_object__collect_st_ops_relos (obj, shdr, data); - }else if (idx == obj->efile.btf_maps_shndx) { +else if (idx == obj->efile.btf_maps_shndx) err =bpf_object__collect_map_relos (obj, shdr, data); - }else { - prog =bpf_object__find_prog_by_idx (obj, idx); -if (!prog) { -pr_warn ("relocation failed: no prog in section(%d)\n" , idx); -return -LIBBPF_ERRNO__RELOC ; - } - err =bpf_program__collect_reloc (prog, shdr, data, obj); - } +else + err =bpf_object__collect_prog_relos (obj, shdr, data);if (err)return err; } + +for (i =0 ; i < obj->nr_programs; i++) { +struct bpf_program *p = &obj->programs[i]; + +if (!p->nr_reloc) +continue ; + +qsort (p->reloc_desc, p->nr_reloc,sizeof (*p->reloc_desc), cmp_relocs); + }return 0 ; }
@@ -6314,12 +6473,6 @@ int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver)return err; } -static bool bpf_program__is_function_storage (const struct bpf_program *prog, -const struct bpf_object *obj) -{ -return prog->sec_idx == obj->efile.text_shndx && obj->has_pseudo_calls; -} -static int bpf_object__load_progs (struct bpf_object *obj,int log_level) {
@@ -6336,7 +6489,7 @@ bpf_object__load_progs(struct bpf_object *obj, int log_level)for (i =0 ; i < obj->nr_programs; i++) { prog = &obj->programs[i]; -if (bpf_program__is_function_storage (prog, obj)) +if (prog_is_subprog (obj, prog))continue ;if (!prog->load) {pr_debug ("prog '%s': skipped loading\n" , prog->name);
@@ -6400,7 +6553,7 @@ __bpf_object__open(const char *path, const void *obj_buf, size_t obj_buf_sz, err = err ? :bpf_object__collect_externs (obj); err = err ? :bpf_object__finalize_btf (obj); err = err ? :bpf_object__init_maps (obj, opts); - err = err ? :bpf_object__collect_reloc (obj); + err = err ? :bpf_object__collect_relos (obj);if (err) goto out;bpf_object__elf_finish (obj);
@@ -7404,7 +7557,7 @@ bpf_program__next(struct bpf_program *prev, const struct bpf_object *obj)do { prog =__bpf_program__iter (prog, obj, true); - }while (prog &&bpf_program__is_function_storage (prog, obj)); + }while (prog &&prog_is_subprog (obj, prog));return prog; }
@@ -7416,7 +7569,7 @@ bpf_program__prev(struct bpf_program *next, const struct bpf_object *obj)do { prog =__bpf_program__iter (prog, obj, false); - }while (prog &&bpf_program__is_function_storage (prog, obj)); + }while (prog &&prog_is_subprog (obj, prog));return prog; }
Signature · 1 line
2.24.1
Re: [PATCH v2 bpf-next 04/14] libbpf: make RELO_CALL work for multi-prog sections and sub-program calls
Quoted 32 lines · Andrii Nakryiko
tools/lib/bpf/libbpf.c +32
-0+ +static int +bpf_object__reloc_code (struct bpf_object *obj,struct bpf_program *main_prog, +struct bpf_program *prog) +{ +size_t sub_insn_idx, insn_idx, new_cnt; +struct bpf_program *subprog; +struct bpf_insn *insns, *insn; +struct reloc_desc *relo; +int err; + + err =reloc_prog_func_and_line_info (obj, main_prog, prog); +if (err) +return err; + +for (insn_idx =0 ; insn_idx < prog->sec_insn_cnt; insn_idx++) { + insn = &main_prog->insns[prog->sub_insn_off + insn_idx]; +if (!insn_is_subprog_call (insn)) +continue ; + + relo =find_prog_insn_relo (prog, insn_idx); +if (relo && relo->type !=RELO_CALL ) { +pr_warn ("prog '%s': unexpected relo for insn #%zu, type %d\n" , + prog->name, insn_idx, relo->type); +return -LIBBPF_ERRNO__RELOC ; + } +if (relo) { +/* sub-program instruction index is a combination of +* an offset of a symbol pointed to by relocation and +* call instruction's imm field; for global functions, +* call always has imm = -1, but for static functions +* relocation is against STT_SECTION and insn->imm
tools/lib/bpf/libbpf.c bpf_program__relocate(struct bpf_program *prog, struct bpf_object *obj) · line 5914
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+* points to a start of a static function +*/ + sub_insn_idx = relo->sym_off /BPF_INSN_SZ + insn->imm +1 ; + }else { +/* if subprogram call is to a static function within +* the same ELF section, there won't be any relocation +* emitted, but it also means there is no additional +* offset necessary, insns->imm is relative to +* instruction's original position within the section +*/
Great two comments. Thanks.
Quoted 4 lines
tools/lib/bpf/libbpf.c +4
-0+ sub_insn_idx = prog->sec_insn_off + insn_idx + insn->imm +1 ; + } + +/* we enforce that sub-programs should be in .text section */
tools/lib/bpf/libbpf.c bpf_program__relocate(struct bpf_program *prog, struct bpf_object *obj) · line 5928
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+ subprog =find_prog_by_sec_insn (obj, obj->efile.text_shndx, sub_insn_idx); +if (!subprog) { +pr_warn ("prog '%s': no .text section found yet sub-program call exists\n" , + prog->name); +return -LIBBPF_ERRNO__RELOC ; + } + +/* if subprogram hasn't been used in current main program, +* relocate it and append at the end of main program code +*/
This one is quite confusing.
"hasn't been used" isn't right.
This subprog was used, but wasn't appeneded yet. That's what sub_insn_off is tracking. Also "relocate and append it" is not right either.
It's "append and start relocating".
Probably shouldn't call it 'main' and 'subprog'.
It equally applies to 'subprog' and 'another subprog'.
Quoted 18 lines
tools/lib/bpf/libbpf.c +18
-0+if (subprog->sub_insn_off ==0 ) { + subprog->sub_insn_off = main_prog->insns_cnt; + + new_cnt = main_prog->insns_cnt + subprog->insns_cnt; + insns =libbpf_reallocarray (main_prog->insns, new_cnt,sizeof (*insns)); +if (!insns) { +pr_warn ("prog '%s': failed to realloc prog code\n" , main_prog->name); +return -ENOMEM ; + } + main_prog->insns = insns; + main_prog->insns_cnt = new_cnt; + +memcpy (main_prog->insns + subprog->sub_insn_off, subprog->insns, + subprog->insns_cnt *sizeof (*insns)); + +pr_debug ("prog '%s': added %zu insns from sub-prog '%s'\n" , + main_prog->name, subprog->insns_cnt, subprog->name); +
tools/lib/bpf/libbpf.c bpf_program__relocate(struct bpf_program *prog, struct bpf_object *obj) · line 5956
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+ err =bpf_object__reloc_code (obj, main_prog, subprog); +if (err) +return err; + } + +/* main_prog->insns memory could have been re-allocated, so +* calculate pointer again +*/ + insn = &main_prog->insns[prog->sub_insn_off + insn_idx]; +/* calculate correct instruction position within main prog */
may be: "calculate position within the prog being relocated?"
tools/lib/bpf/libbpf.c bpf_program__relocate(struct bpf_program *prog, struct bpf_object *obj) · line 5957
Quoted 1 line
tools/lib/bpf/libbpf.c +1
-0+ insn->imm = subprog->sub_insn_off - (prog->sub_insn_off + insn_idx) -1 ;
I think the algorithm is sound.
Could you add a better description of it?
May be some small diagram to illustrate how it recursively relocates?
That it starts with main, walks some number of insn, when it sees pseudo_call to not yet appended subprog, it adds it to the end and recursively starts relocating it. That subprog can have relos too. If they're pointing to not yet appended subprog it will be added again and that 2nd subprog will start relocating while the main and 1st subprog will be pending.
The algorithm didn't have to be recursive, but I guess it's fine to keep this way. It's simple enough. I haven't thought through how it can look without recursion. Probably a bunch of book keeping of things to relocate would have been necessary.
Re: [PATCH v2 bpf-next 04/14] libbpf: make RELO_CALL work for multi-prog sections and sub-program calls
Quoted 57 lines · Alexei Starovoitov
Quoted 32 lines · Andrii Nakryiko
tools/lib/bpf/libbpf.c +32
-0+ +static int +bpf_object__reloc_code (struct bpf_object *obj,struct bpf_program *main_prog, +struct bpf_program *prog) +{ +size_t sub_insn_idx, insn_idx, new_cnt; +struct bpf_program *subprog; +struct bpf_insn *insns, *insn; +struct reloc_desc *relo; +int err; + + err =reloc_prog_func_and_line_info (obj, main_prog, prog); +if (err) +return err; + +for (insn_idx =0 ; insn_idx < prog->sec_insn_cnt; insn_idx++) { + insn = &main_prog->insns[prog->sub_insn_off + insn_idx]; +if (!insn_is_subprog_call (insn)) +continue ; + + relo =find_prog_insn_relo (prog, insn_idx); +if (relo && relo->type !=RELO_CALL ) { +pr_warn ("prog '%s': unexpected relo for insn #%zu, type %d\n" , + prog->name, insn_idx, relo->type); +return -LIBBPF_ERRNO__RELOC ; + } +if (relo) { +/* sub-program instruction index is a combination of +* an offset of a symbol pointed to by relocation and +* call instruction's imm field; for global functions, +* call always has imm = -1, but for static functions +* relocation is against STT_SECTION and insn->imm tools/lib/bpf/libbpf.c bpf_program__relocate(struct bpf_program *prog, struct bpf_object *obj) · line 5914
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+* points to a start of a static function +*/ + sub_insn_idx = relo->sym_off /BPF_INSN_SZ + insn->imm +1 ; + }else { +/* if subprogram call is to a static function within +* the same ELF section, there won't be any relocation +* emitted, but it also means there is no additional +* offset necessary, insns->imm is relative to +* instruction's original position within the section +*/ Great two comments. Thanks.
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+ sub_insn_idx = prog->sec_insn_off + insn_idx + insn->imm +1 ; + } + +/* we enforce that sub-programs should be in .text section */ + subprog =find_prog_by_sec_insn (obj, obj->efile.text_shndx, sub_insn_idx); +if (!subprog) { +pr_warn ("prog '%s': no .text section found yet sub-program call exists\n" , + prog->name); +return -LIBBPF_ERRNO__RELOC ; + }
Quoted 8 lines
tools/lib/bpf/libbpf.c bpf_program__relocate(struct bpf_program *prog, struct bpf_object *obj) · line 5928
Quoted 4 lines
tools/lib/bpf/libbpf.c +4
-0+ +/* if subprogram hasn't been used in current main program, +* relocate it and append at the end of main program code +*/ This one is quite confusing.
"hasn't been used" isn't right.
This subprog was used, but wasn't appeneded yet. That's whatsub_insn_offis tracking.
"hasn't been used *yet*" would be more precise, meaning: up until the current instruction there were no calls to that subprogram.
Quoted 2 lines
Also "relocate and append it" is not right either. It's "append and start relocating".
Right, order of actions is wrong, I'll fix the wording.
Quoted 2 lines
Probably shouldn't call it 'main' and 'subprog'. It equally applies to 'subprog' and 'another subprog'.
Yes, you are right, I'll update comments to be less confusing.
Quoted 23 lines
Quoted 23 lines
tools/lib/bpf/libbpf.c +22
-0+if (subprog->sub_insn_off ==0 ) { + subprog->sub_insn_off = main_prog->insns_cnt; + + new_cnt = main_prog->insns_cnt + subprog->insns_cnt; + insns =libbpf_reallocarray (main_prog->insns, new_cnt,sizeof (*insns)); +if (!insns) { +pr_warn ("prog '%s': failed to realloc prog code\n" , main_prog->name); +return -ENOMEM ; + } + main_prog->insns = insns; + main_prog->insns_cnt = new_cnt; + +memcpy (main_prog->insns + subprog->sub_insn_off, subprog->insns, + subprog->insns_cnt *sizeof (*insns)); + +pr_debug ("prog '%s': added %zu insns from sub-prog '%s'\n" , + main_prog->name, subprog->insns_cnt, subprog->name); + + err =bpf_object__reloc_code (obj, main_prog, subprog); +if (err) +return err; + }
Quoted 8 lines
tools/lib/bpf/libbpf.c bpf_program__relocate(struct bpf_program *prog, struct bpf_object *obj) · line 5956
Quoted 6 lines
tools/lib/bpf/libbpf.c +6
-0+ +/* main_prog->insns memory could have been re-allocated, so +* calculate pointer again +*/ + insn = &main_prog->insns[prog->sub_insn_off + insn_idx]; +/* calculate correct instruction position within main prog */ may be: "calculate position within the prog being relocated?"
no-no, in this case it's an instruction index within the main (entry-point) BPF program with all the used subprograms appended. So even if we have subprog1 calling another subprog2, all the instruction indices are calculated within the main BPF program's "system of coordinates", because each main BPF program can have a different subset of functions appended and subprogs might be in a different order and at different positions.
Quoted 11 lines
Quoted 2 lines
+
insn->imm=subprog->sub_insn_off- (prog->sub_insn_off+insn_idx) - 1;I think the algorithm is sound.
Could you add a better description of it?
May be some small diagram to illustrate how it recursively relocates?
That it starts with main, walks some number of insn, when it seespseudo_callto not yet appended subprog, it adds it to the end and recursively starts relocating it. That subprog can have relos too. If they're pointing to not yet appended subprog it will be added again and that 2nd subprog will start relocating while the main and 1st subprog will be pending.
Ok, I'll try to give some better overview in the comments.
Quoted 3 lines
The algorithm didn't have to be recursive, but I guess it's fine to keep this way. It's simple enough. I haven't thought through how it can look without recursion. Probably a bunch of book keeping of things to relocate would have been necessary.
It's a graph traversing problem to figure out which subprograms need to be appended. I did it DFS style because it's a bit simpler that way (there is no separate pass for detecting used subprogs and then relocating all of them), plus libbpf already relies on recursion (at least for CO-RE), so I didn't feel bad about it. It's possible to do it with BFS, though, by maintaining a queue of to-be-processed subprogs, mark and append them. Then as a second pass relocate calls. But as you said, it's simple enough on a high-level, that I'd stick to what I have. I'll improve the comments, though.
PATCH v2 5/14 libbpf: implement generalized .BTF.ext func/line info adjustment
In reply to Andrii NakryikoComplete multi-prog sections and multi sub-prog support in libbpf by properly adjusting .BTF.ext's line and function information. Mark exposed btf_ext__reloc_func_info() and btf_ext__reloc_func_info() APIs as deprecated. These APIs have simplistic assumption that all sub-programs are going to be appended to all main BPF programs, which doesn't hold in real life. It's unlikely there are any users of this API, as it's very libbpf internals-specific.
--- tools/lib/bpf/btf.h | 18 +-- tools/lib/bpf/libbpf.c | 217 ++++++++++++++++++++++------------ tools/lib/bpf/libbpf_common.h | 2 + 3 files changed, 153 insertions(+), 84 deletions(-)
tools/lib/bpf/btf.h +10 -8
@@ -57,14 +57,16 @@ LIBBPF_API struct btf_ext *btf_ext__new(__u8 *data, __u32 size); LIBBPF_API void btf_ext__free(struct btf_ext *btf_ext); LIBBPF_API const void *btf_ext__get_raw_data(const struct btf_ext *btf_ext,__u32 *size); -LIBBPF_API int btf_ext__reloc_func_info(const struct btf *btf, -const struct btf_ext *btf_ext, -const char *sec_name,__u32 insns_cnt, -void **func_info,__u32 *cnt); -LIBBPF_API int btf_ext__reloc_line_info(const struct btf *btf, -const struct btf_ext *btf_ext, -const char *sec_name,__u32 insns_cnt, -void **line_info,__u32 *cnt); +LIBBPF_API LIBBPF_DEPRECATED("btf_ext__reloc_func_info was never meant as a public API and has wrong assumptions embedded in it; it will be removed in the future libbpf versions") +int btf_ext__reloc_func_info (const struct btf *btf, +const struct btf_ext *btf_ext, +const char *sec_name,__u32 insns_cnt, +void **func_info,__u32 *cnt); +LIBBPF_API LIBBPF_DEPRECATED("btf_ext__reloc_line_info was never meant as a public API and has wrong assumptions embedded in it; it will be removed in the future libbpf versions") +int btf_ext__reloc_line_info (const struct btf *btf, +const struct btf_ext *btf_ext, +const char *sec_name,__u32 insns_cnt, +void **line_info,__u32 *cnt); LIBBPF_API __u32 btf_ext__func_info_rec_size(const struct btf_ext *btf_ext); LIBBPF_API __u32 btf_ext__line_info_rec_size(const struct btf_ext *btf_ext);
tools/lib/bpf/libbpf.c +141 -76
235 lines not on this page · raw message
tools/lib/bpf/libbpf_common.h +2 -0
@@ -15,6 +15,8 @@#define LIBBPF_API __attribute__((visibility("default")))#endif +#define LIBBPF_DEPRECATED (msg) __attribute__((deprecated(msg))) +/* Helper macro to declare and initialize libbpf options struct * * This dance with uninitialized declaration, followed by memset to zero,
Signature · 1 line
2.24.1
PATCH v2 6/14 libbpf: add multi-prog section support for struct_ops
In reply to Andrii NakryikoAdjust struct_ops handling code to work with multi-program ELF sections properly.
--- tools/lib/bpf/libbpf.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-)
tools/lib/bpf/libbpf.c +8 -18
@@ -73,8 +73,6 @@#define __printf (a, b) __attribute__((format(printf, a, b)))static struct bpf_map *bpf_object__add_map (struct bpf_object *obj); -static struct bpf_program *bpf_object__find_prog_by_idx (struct bpf_object *obj, -int idx);static const struct btf_type *skip_mods_and_typedefs (const struct btf *btf,__u32 id,__u32 *res_id);
@@ -3249,20 +3247,6 @@ static int bpf_object__collect_externs(struct bpf_object *obj)return 0 ; } -static struct bpf_program * -bpf_object__find_prog_by_idx (struct bpf_object *obj,int idx) -{ -struct bpf_program *prog; -size_t i; - -for (i =0 ; i < obj->nr_programs; i++) { - prog = &obj->programs[i]; -if (prog->sec_idx == idx) -return prog; - } -return NULL ; -} -struct bpf_program *bpf_object__find_program_by_title (const struct bpf_object *obj,const char *title)
@@ -8109,7 +8093,7 @@ static int bpf_object__collect_st_ops_relos(struct bpf_object *obj,const struct btf *btf;struct bpf_map *map;Elf_Data *symbols; -unsigned int moff; +unsigned int moff, insn_idx;const char *name;__u32 member_idx;GElf_Sym sym;
@@ -8154,6 +8138,12 @@ static int bpf_object__collect_st_ops_relos(struct bpf_object *obj, map->name, (size_t)rel.r_offset, shdr_idx);return -LIBBPF_ERRNO__RELOC ; } +if (sym.st_value %BPF_INSN_SZ ) { +pr_warn ("struct_ops reloc %s: invalid target program offset %llu\n" , + map->name, (__u64 )sym.st_value); +return -LIBBPF_ERRNO__FORMAT ; + } + insn_idx = sym.st_value /BPF_INSN_SZ ; member =find_member_by_offset (st_ops->type, moff *8 );if (!member) {
@@ -8170,7 +8160,7 @@ static int bpf_object__collect_st_ops_relos(struct bpf_object *obj,return -EINVAL ; } - prog =bpf_object__find_prog_by_idx (obj, shdr_idx); + prog =find_prog_by_sec_insn (obj, shdr_idx, insn_idx);if (!prog) {pr_warn ("struct_ops reloc %s: cannot find prog at shdr_idx %u to relocate func ptr %s\n" , map->name, shdr_idx, name);
Signature · 1 line
2.24.1
PATCH v2 7/14 selftests/bpf: add selftest for multi-prog sections and bpf-to-bpf calls
In reply to Andrii NakryikoAdd a selftest excercising bpf-to-bpf subprogram calls, as well as multiple entry-point BPF programs per section. Also make sure that BPF CO-RE works for such set ups both for sub-programs and for multi-entry sections.
--- .../selftests/bpf/prog_tests/subprogs.c | 31 ++++++ .../selftests/bpf/progs/test_subprogs.c | 103 ++++++++++++++++++ 2 files changed, 134 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/subprogs.c create mode 100644 tools/testing/selftests/bpf/progs/test_subprogs.c
tools/testing/selftests/bpf/prog_tests/subprogs.c new file +31 -0
@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ +#include <test_progs.h> +#include <time.h> +#include "test_subprogs.skel.h" + +static int duration; + +void test_subprogs (void ) +{ +struct test_subprogs *skel; +int err; + + skel =test_subprogs__open_and_load (); +if (CHECK (!skel,"skel_open" ,"failed to open skeleton\n" )) +return ; + + err =test_subprogs__attach (skel); +if (CHECK (err,"skel_attach" ,"failed to attach skeleton: %d\n" , err)) + goto cleanup; + +usleep (1 ); + +CHECK (skel->bss->res1 !=12 ,"res1" ,"got %d, exp %d\n" , skel->bss->res1,12 ); +CHECK (skel->bss->res2 !=17 ,"res2" ,"got %d, exp %d\n" , skel->bss->res2,17 ); +CHECK (skel->bss->res3 !=19 ,"res3" ,"got %d, exp %d\n" , skel->bss->res3,19 ); +CHECK (skel->bss->res4 !=36 ,"res4" ,"got %d, exp %d\n" , skel->bss->res4,36 ); + +cleanup: +test_subprogs__destroy (skel); +}
tools/testing/selftests/bpf/progs/test_subprogs.c new file +103 -0
103 lines not on this page · raw message
Signature · 1 line
2.24.1
Re: [PATCH v2 bpf-next 07/14] selftests/bpf: add selftest for multi-prog sections and bpf-to-bpf calls
Quoted 11 lines · Andrii Nakryiko
tools/testing/selftests/bpf/progs/test_subprogs.c new file +11
-0+ +__noinline int sub1(int x) +{ +return x +1 ; +} + +static __noinline int sub5(int v); + +__noinline int sub2(int y) +{ +return sub5 (y +2 );
tools/testing/selftests/bpf/progs/test_subprogs.c line 26
Quoted 10 lines
tools/testing/selftests/bpf/progs/test_subprogs.c new file +10
-0+} + +static __noinline int sub3(int z) +{ +return z +3 +sub1 (4 ); +} + +static __noinline int sub4(int w) +{ +return w +sub3 (5 ) +sub1 (6 );
Did you check that asm has these calls?
Since sub3 is static the compiler doesn't have to do the call. 'static noinline' doesn't mean that compiler have to do the call. It can compute the value and replace a call with a constant. It only has to keep the body of the function if the address of it was taken.
Re: [PATCH v2 bpf-next 07/14] selftests/bpf: add selftest for multi-prog sections and bpf-to-bpf calls
Quoted 17 lines · Alexei Starovoitov
Quoted 15 lines · Andrii Nakryiko
tools/testing/selftests/bpf/progs/test_subprogs.c new file +15
-0+ +__noinline int sub1(int x) +{ +return x +1 ; +} + +static __noinline int sub5(int v); + +__noinline int sub2(int y) +{ +return sub5 (y +2 ); +} + +static __noinline int sub3(int z) +{
Quoted 8 lines
tools/testing/selftests/bpf/progs/test_subprogs.c line 26
Quoted 6 lines
tools/testing/selftests/bpf/progs/test_subprogs.c new file +6
-0+return z +3 +sub1 (4 ); +} + +static __noinline int sub4(int w) +{ +return w +sub3 (5 ) +sub1 (6 );Did you check that asm has these calls?
Yeah, I actually did check. All calls are there.
Quoted 5 lines
Since sub3 is static the compiler doesn't have to do the call. 'static noinline' doesn't mean that compiler have to do the call. It can compute the value and replace a call with a constant. It only has to keep the body of the function if the address of it was taken.
All these subX() functions are either global or call global function (sub1() is global), which seems to keep Clang from optimizing all this. Clang has to assume the worst case for global functions, probably due to LD_PRELOAD, right?
PATCH v2 8/14 tools/bpftool: replace bpf_program__title() with bpf_program__section_name()
In reply to Andrii Nakryikobpf_program__title() is deprecated, switch to bpf_program__section_name() and avoid compilation warnings.
--- tools/bpf/bpftool/prog.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/bpf/bpftool/prog.c +2 -2
@@ -1304,7 +1304,7 @@ static int load_with_options(int argc, char **argv, bool first_prog_only)enum bpf_prog_type prog_type = common_prog_type;if (prog_type ==BPF_PROG_TYPE_UNSPEC ) { -const char *sec_name =bpf_program__title (pos, false); +const char *sec_name =bpf_program__section_name (pos); err =get_prog_type_by_name (sec_name, &prog_type, &expected_attach_type);
@@ -1398,7 +1398,7 @@ static int load_with_options(int argc, char **argv, bool first_prog_only) err =bpf_obj_pin (bpf_program__fd (prog), pinfile);if (err) {p_err ("failed to pin program %s" , -bpf_program__title (prog, false)); +bpf_program__section_name (prog)); goto err_close_obj; } }else {
Signature · 1 line
2.24.1
PATCH v2 9/14 selftests/bpf: don't use deprecated libbpf APIs
In reply to Andrii NakryikoRemove all uses of bpf_program__title() andbpf_program__find_program_by_title().
--- tools/testing/selftests/bpf/flow_dissector_load.h | 8 +++++++- .../testing/selftests/bpf/prog_tests/reference_tracking.c | 2 +- tools/testing/selftests/bpf/test_socket_cookie.c | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-)
tools/testing/selftests/bpf/flow_dissector_load.h +7 -1
@@ -23,7 +23,13 @@ static inline int bpf_flow_load(struct bpf_object **obj,if (ret)return ret; - main_prog =bpf_object__find_program_by_title (*obj, section_name); + main_prog =NULL ; +bpf_object__for_each_program (prog, *obj) { +if (strcmp (section_name,bpf_program__section_name (prog)) ==0 ) { + main_prog = prog; +break ; + } + }if (!main_prog)return -1 ;
tools/testing/selftests/bpf/prog_tests/reference_tracking.c +1 -1
@@ -27,7 +27,7 @@ void test_reference_tracking(void)const char *title;/* Ignore .text sections */ - title =bpf_program__title (prog, false); + title =bpf_program__section_name (prog);if (strstr (title,".text" ) !=NULL )continue ;
tools/testing/selftests/bpf/test_socket_cookie.c +1 -1
@@ -151,7 +151,7 @@ static int run_test(int cgfd) }bpf_object__for_each_program (prog, pobj) { - prog_name =bpf_program__title (prog,/*needs_copy*/ false); + prog_name =bpf_program__section_name (prog);if (libbpf_attach_type_by_name (prog_name, &attach_type)) goto err;
Signature · 1 line
2.24.1
PATCH v2 10/14 libbpf: deprecate notion of BPF program "title" in favor of "section name"
In reply to Andrii NakryikoBPF program title is ambigious and misleading term. It is ELF section name, so let's just call it that and deprecate bpf_program__title() API in favor of bpf_program__section_name().
Additionally, using bpf_object__find_program_by_title() is now inherently dangerous and ambiguous, as multiple BPF program can have the same section name. So deprecate this API as well and recommend to switch to non-ambiguous bpf_object__find_program_by_name().
Internally, clean up usage and mis-usage of BPF program section name for denoting BPF program name. Shorten the field name to prog->sec_name to be consistent with all other prog->sec_* variables.
--- tools/lib/bpf/libbpf.c | 215 ++++++++++++++++++--------------------- tools/lib/bpf/libbpf.h | 5 +- tools/lib/bpf/libbpf.map | 1 + 3 files changed, 101 insertions(+), 120 deletions(-)
tools/lib/bpf/libbpf.c +97 -118
@@ -217,7 +217,7 @@ struct bpf_sec_def {*/ struct bpf_program {const struct bpf_sec_def *sec_def; -char *section_name; +char *sec_name;size_t sec_idx;/* this program's instruction offset (in number of instructions) * within its containing ELF section
@@ -239,7 +239,7 @@ struct bpf_program {size_t sub_insn_off;char *name; -/* section_name with / replaced by _; makes recursive pinning +/* sec_name with / replaced by _; makes recursive pinning * in bpf_object__pin_programs easier */ char *pin_name;
@@ -515,7 +515,7 @@ static void bpf_program__exit(struct bpf_program *prog)bpf_program__unload (prog);zfree (&prog->name); -zfree (&prog->section_name); +zfree (&prog->sec_name);zfree (&prog->pin_name);zfree (&prog->insns);zfree (&prog->reloc_desc);
@@ -529,7 +529,7 @@ static char *__bpf_program__pin_name(struct bpf_program *prog) {char *name, *p; - name = p =strdup (prog->section_name); + name = p =strdup (prog->sec_name);while ((p =strchr (p,'/' ))) *p ='_' ;
@@ -574,8 +574,8 @@ bpf_object__init_prog(struct bpf_object *obj, struct bpf_program *prog, prog->instances.fds =NULL ; prog->instances.nr =-1 ; - prog->section_name =strdup (sec_name); -if (!prog->section_name) + prog->sec_name =strdup (sec_name); +if (!prog->sec_name) goto errout; prog->name =strdup (name);
@@ -3254,7 +3254,7 @@ bpf_object__find_program_by_title(const struct bpf_object *obj,struct bpf_program *pos;bpf_object__for_each_program (pos, obj) { -if (pos->section_name && !strcmp (pos->section_name, title)) +if (pos->sec_name && !strcmp (pos->sec_name, title))return pos; }return NULL ;
@@ -4994,8 +4994,7 @@ static int bpf_core_calc_field_relo(const struct bpf_program *prog, *val = sz; }else {pr_warn ("prog '%s': relo %d at insn #%d can't be applied to array access\n" , -bpf_program__title (prog, false), - relo->kind, relo->insn_off /8 ); + prog->name, relo->kind, relo->insn_off /8 );return -EINVAL ; }if (validate)
@@ -5017,8 +5016,7 @@ static int bpf_core_calc_field_relo(const struct bpf_program *prog,if (byte_sz >=8 ) {/* bitfield can't be read with 64-bit read */ pr_warn ("prog '%s': relo %d at insn #%d can't be satisfied for bitfield\n" , -bpf_program__title (prog, false), - relo->kind, relo->insn_off /8 ); + prog->name, relo->kind, relo->insn_off /8 );return -E2BIG ; } byte_sz *=2 ;
@@ -5183,8 +5181,8 @@ static int bpf_core_calc_relo(const struct bpf_program *prog, }else if (err == -EOPNOTSUPP ) {/* EOPNOTSUPP means unknown/unsupported relocation */ pr_warn ("prog '%s': relo #%d: unrecognized CO-RE relocation %s (%d) at insn #%d\n" , -bpf_program__title (prog, false), relo_idx, -core_relo_kind_str (relo->kind), relo->kind, relo->insn_off /8 ); + prog->name, relo_idx,core_relo_kind_str (relo->kind), + relo->kind, relo->insn_off /8 ); }return err;
@@ -5198,7 +5196,7 @@ static void bpf_core_poison_insn(struct bpf_program *prog, int relo_idx, int insn_idx, struct bpf_insn *insn) { pr_debug("prog '%s': relo #%d: substituting insn #%d w/ invalid insn\n", - bpf_program__title(prog, false), relo_idx, insn_idx); + prog->name, relo_idx, insn_idx); insn->code =BPF_JMP |BPF_CALL ; insn->dst_reg =0 ; insn->src_reg =0 ;
@@ -5270,14 +5268,14 @@ static int bpf_core_patch_insn(struct bpf_program *prog,return -EINVAL ;if (res->validate && insn->imm != orig_val) {pr_warn ("prog '%s': relo #%d: unexpected insn #%d (ALU/ALU64) value: got %u, exp %u -> %u\n" , -bpf_program__title (prog, false), relo_idx, + prog->name, relo_idx, insn_idx, insn->imm, orig_val, new_val);return -EINVAL ; } orig_val = insn->imm; insn->imm = new_val;pr_debug ("prog '%s': relo #%d: patched insn #%d (ALU/ALU64) imm %u -> %u\n" , -bpf_program__title (prog, false), relo_idx, insn_idx, + prog->name, relo_idx, insn_idx, orig_val, new_val);break ;case BPF_LDX :
@@ -5285,21 +5283,18 @@ static int bpf_core_patch_insn(struct bpf_program *prog,case BPF_STX :if (res->validate && insn->off != orig_val) {pr_warn ("prog '%s': relo #%d: unexpected insn #%d (LDX/ST/STX) value: got %u, exp %u -> %u\n" , -bpf_program__title (prog, false), relo_idx, - insn_idx, insn->off, orig_val, new_val); + prog->name, relo_idx, insn_idx, insn->off, orig_val, new_val);return -EINVAL ; }if (new_val >SHRT_MAX ) {pr_warn ("prog '%s': relo #%d: insn #%d (LDX/ST/STX) value too big: %u\n" , -bpf_program__title (prog, false), relo_idx, - insn_idx, new_val); + prog->name, relo_idx, insn_idx, new_val);return -ERANGE ; } orig_val = insn->off; insn->off = new_val;pr_debug ("prog '%s': relo #%d: patched insn #%d (LDX/ST/STX) off %u -> %u\n" , -bpf_program__title (prog, false), relo_idx, insn_idx, - orig_val, new_val); + prog->name, relo_idx, insn_idx, orig_val, new_val);break ;case BPF_LD : {__u64 imm;
@@ -5310,14 +5305,14 @@ static int bpf_core_patch_insn(struct bpf_program *prog, insn[1].code != 0 || insn[1].dst_reg != 0 || insn[1].src_reg != 0 || insn[1].off != 0) {pr_warn ("prog '%s': relo #%d: insn #%d (LDIMM64) has unexpected form\n" , -bpf_program__title (prog, false), relo_idx, insn_idx); + prog->name, relo_idx, insn_idx);return -EINVAL ; } imm = insn[0 ].imm + ((__u64 )insn[1 ].imm <<32 );if (res->validate && imm != orig_val) {pr_warn ("prog '%s': relo #%d: unexpected insn #%d (LDIMM64) value: got %llu, exp %u -> %u\n" , -bpf_program__title (prog, false), relo_idx, + prog->name, relo_idx, insn_idx, (unsigned long long )imm, orig_val, new_val);return -EINVAL ;
@@ -5326,15 +5321,14 @@ static int bpf_core_patch_insn(struct bpf_program *prog, insn[0 ].imm = new_val; insn[1 ].imm =0 ;/* currently only 32-bit values are supported */ pr_debug ("prog '%s': relo #%d: patched insn #%d (LDIMM64) imm64 %llu -> %u\n" , -bpf_program__title (prog, false), relo_idx, insn_idx, + prog->name, relo_idx, insn_idx, (unsigned long long )imm, new_val);break ; }default :pr_warn ("prog '%s': relo #%d: trying to relocate unrecognized insn #%d, code:0x%x, src:0x%x, dst:0x%x, off:0x%x, imm:0x%x\n" , -bpf_program__title (prog, false), relo_idx, - insn_idx, insn->code, insn->src_reg, insn->dst_reg, - insn->off, insn->imm); + prog->name, relo_idx, insn_idx, insn->code, + insn->src_reg, insn->dst_reg, insn->off, insn->imm);return -EINVAL ; }
@@ -5464,7 +5458,6 @@ static int bpf_core_apply_relo(struct bpf_program *prog,const struct btf *targ_btf, struct hashmap *cand_cache) { - const char *prog_name = bpf_program__title(prog, false); struct bpf_core_spec local_spec, cand_spec, targ_spec = {};const void *type_key =u32_as_hash_key (relo->type_id);struct bpf_core_relo_res cand_res, targ_res;
@@ -5491,13 +5484,13 @@ static int bpf_core_apply_relo(struct bpf_program *prog, err =bpf_core_parse_spec (local_btf, local_id, spec_str, relo->kind, &local_spec);if (err) {pr_warn ("prog '%s': relo #%d: parsing [%d] %s %s + %s failed: %d\n" , - prog_name, relo_idx, local_id,btf_kind_str (local_type), + prog->name, relo_idx, local_id,btf_kind_str (local_type),str_is_empty (local_name) ?"<anon>" : local_name, spec_str, err);return -EINVAL ; } -pr_debug ("prog '%s': relo #%d: kind <%s> (%d), spec is " , prog_name, +pr_debug ("prog '%s': relo #%d: kind <%s> (%d), spec is " , prog->name, relo_idx,core_relo_kind_str (relo->kind), relo->kind);bpf_core_dump_spec (LIBBPF_DEBUG , &local_spec);libbpf_print (LIBBPF_DEBUG ,"\n" );
@@ -5514,7 +5507,7 @@ static int bpf_core_apply_relo(struct bpf_program *prog,/* libbpf doesn't support candidate search for anonymous types */ if (str_is_empty (spec_str)) {pr_warn ("prog '%s': relo #%d: <%s> (%d) relocation doesn't support anonymous types\n" , - prog_name, relo_idx,core_relo_kind_str (relo->kind), relo->kind); + prog->name, relo_idx,core_relo_kind_str (relo->kind), relo->kind);return -EOPNOTSUPP ; }
@@ -5522,7 +5515,7 @@ static int bpf_core_apply_relo(struct bpf_program *prog, cand_ids =bpf_core_find_cands (local_btf, local_id, targ_btf);if (IS_ERR (cand_ids)) {pr_warn ("prog '%s': relo #%d: target candidate search failed for [%d] %s %s: %ld" , - prog_name, relo_idx, local_id,btf_kind_str (local_type), + prog->name, relo_idx, local_id,btf_kind_str (local_type), local_name,PTR_ERR (cand_ids));return PTR_ERR (cand_ids); }
@@ -5538,13 +5531,13 @@ static int bpf_core_apply_relo(struct bpf_program *prog, err =bpf_core_spec_match (&local_spec, targ_btf, cand_id, &cand_spec);if (err <0 ) {pr_warn ("prog '%s': relo #%d: error matching candidate #%d " , - prog_name, relo_idx, i); + prog->name, relo_idx, i);bpf_core_dump_spec (LIBBPF_WARN , &cand_spec);libbpf_print (LIBBPF_WARN ,": %d\n" , err);return err; } -pr_debug ("prog '%s': relo #%d: %s candidate #%d " , prog_name, +pr_debug ("prog '%s': relo #%d: %s candidate #%d " , prog->name, relo_idx, err ==0 ?"non-matching" :"matching" , i);bpf_core_dump_spec (LIBBPF_DEBUG , &cand_spec);libbpf_print (LIBBPF_DEBUG ,"\n" );
@@ -5564,7 +5557,7 @@ static int bpf_core_apply_relo(struct bpf_program *prog,* should all resolve to the same bit offset */ pr_warn ("prog '%s': relo #%d: field offset ambiguity: %u != %u\n" , - prog_name, relo_idx, cand_spec.bit_offset, + prog->name, relo_idx, cand_spec.bit_offset, targ_spec.bit_offset);return -EINVAL ; }else if (cand_res.poison != targ_res.poison || cand_res.new_val != targ_res.new_val) {
@@ -5573,7 +5566,7 @@ static int bpf_core_apply_relo(struct bpf_program *prog,* proceed due to ambiguity */ pr_warn ("prog '%s': relo #%d: relocation decision ambiguity: %s %u != %s %u\n" , - prog_name, relo_idx, + prog->name, relo_idx, cand_res.poison ?"failure" :"success" , cand_res.new_val, targ_res.poison ?"failure" :"success" , targ_res.new_val);return -EINVAL ;
@@ -5606,7 +5599,7 @@ static int bpf_core_apply_relo(struct bpf_program *prog,*/ if (j ==0 ) {pr_debug ("prog '%s': relo #%d: no matching targets found\n" , - prog_name, relo_idx); + prog->name, relo_idx);/* calculate single target relo result explicitly */ err =bpf_core_calc_relo (prog, relo, relo_idx, &local_spec,NULL , &targ_res);
@@ -5619,7 +5612,7 @@ static int bpf_core_apply_relo(struct bpf_program *prog, err =bpf_core_patch_insn (prog, relo, relo_idx, &targ_res);if (err) {pr_warn ("prog '%s': relo #%d: failed to patch insn at offset %d: %d\n" , - prog_name, relo_idx, relo->insn_off, err); + prog->name, relo_idx, relo->insn_off, err);return -EINVAL ; }
@@ -5673,7 +5666,7 @@ bpf_object__relocate_core(struct bpf_object *obj, const char *targ_btf_path) prog =NULL ;for (i =0 ; i < obj->nr_programs; i++) { prog = &obj->programs[i]; -if (strcmp (prog->section_name, sec_name) ==0 ) +if (strcmp (prog->sec_name, sec_name) ==0 )break ; }if (!prog) {
@@ -5787,7 +5780,7 @@ static int adjust_prog_btf_ext_info(const struct bpf_object *obj, sec_name =btf__name_by_offset (obj->btf, sec->sec_name_off);if (!sec_name)return -EINVAL ; -if (strcmp (sec_name, prog->section_name) !=0 ) +if (strcmp (sec_name, prog->sec_name) !=0 )continue ;for_each_btf_ext_rec (ext_info, sec, i, rec) {
@@ -6438,8 +6431,7 @@ int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver)int err =0 , fd, i, btf_id;if (prog->obj->loaded) { -pr_warn ("prog '%s'('%s'): can't load after object was loaded\n" , - prog->name, prog->section_name); +pr_warn ("prog '%s': can't load after object was loaded\n" , prog->name);return -EINVAL ; }
@@ -6455,7 +6447,7 @@ int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver)if (prog->instances.nr <0 || !prog->instances.fds) {if (prog->preprocessor) {pr_warn ("Internal error: can't load program '%s'\n" , - prog->section_name); + prog->name);return -LIBBPF_ERRNO__INTERNAL ; }
@@ -6470,8 +6462,8 @@ int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver)if (!prog->preprocessor) {if (prog->instances.nr !=1 ) { -pr_warn ("Program '%s' is inconsistent: nr(%d) != 1\n" , - prog->section_name, prog->instances.nr); +pr_warn ("prog '%s': inconsistent nr(%d) != 1\n" , + prog->name, prog->instances.nr); } err =load_program (prog, prog->insns, prog->insns_cnt, license, kern_ver, &fd);
@@ -6489,13 +6481,13 @@ int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver) prog->insns_cnt, &result);if (err) {pr_warn ("Preprocessing the %dth instance of program '%s' failed\n" , - i, prog->section_name); + i, prog->name); goto out; }if (!result.new_insn_ptr || !result.new_insn_cnt) {pr_debug ("Skip loading the %dth instance of program '%s'\n" , - i, prog->section_name); + i, prog->name); prog->instances.fds[i] =-1 ;if (result.pfd) *result.pfd =-1 ;
@@ -6506,7 +6498,7 @@ int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver) result.new_insn_cnt, license, kern_ver, &fd);if (err) {pr_warn ("Loading the %dth instance of program '%s' failed\n" , - i, prog->section_name); + i, prog->name); goto out; }
@@ -6516,7 +6508,7 @@ int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver) } out:if (err) -pr_warn ("failed to load program '%s'\n" , prog->section_name); +pr_warn ("failed to load program '%s'\n" , prog->name);zfree (&prog->insns); prog->insns_cnt =0 ;return err;
@@ -6608,7 +6600,7 @@ __bpf_object__open(const char *path, const void *obj_buf, size_t obj_buf_sz,bpf_object__elf_finish (obj);bpf_object__for_each_program (prog, obj) { - prog->sec_def =find_sec_def (prog->section_name); + prog->sec_def =find_sec_def (prog->sec_name);if (!prog->sec_def)/* couldn't guess, but user might manually specify */ continue ;
@@ -6989,7 +6981,7 @@ int bpf_program__pin_instance(struct bpf_program *prog, const char *path,if (instance <0 || instance >= prog->instances.nr) {pr_warn ("invalid prog instance %d of prog %s (max %d)\n" , - instance, prog->section_name, prog->instances.nr); + instance, prog->name, prog->instances.nr);return -EINVAL ; }
@@ -7020,7 +7012,7 @@ int bpf_program__unpin_instance(struct bpf_program *prog, const char *path,if (instance <0 || instance >= prog->instances.nr) {pr_warn ("invalid prog instance %d of prog %s (max %d)\n" , - instance, prog->section_name, prog->instances.nr); + instance, prog->name, prog->instances.nr);return -EINVAL ; }
@@ -7050,8 +7042,7 @@ int bpf_program__pin(struct bpf_program *prog, const char *path) }if (prog->instances.nr <=0 ) { -pr_warn ("no instances of prog %s to pin\n" , - prog->section_name); +pr_warn ("no instances of prog %s to pin\n" , prog->name);return -EINVAL ; }
@@ -7113,8 +7104,7 @@ int bpf_program__unpin(struct bpf_program *prog, const char *path) }if (prog->instances.nr <=0 ) { -pr_warn ("no instances of prog %s to pin\n" , - prog->section_name); +pr_warn ("no instances of prog %s to pin\n" , prog->name);return -EINVAL ; }
@@ -7649,11 +7639,16 @@ const char *bpf_program__name(const struct bpf_program *prog)return prog->name; } +const char *bpf_program__section_name (const struct bpf_program *prog) +{ +return prog->sec_name; +} +const char *bpf_program__title (const struct bpf_program *prog,bool needs_copy) {const char *title; - title = prog->section_name; + title = prog->sec_name;if (needs_copy) { title =strdup (title);if (!title) {
@@ -7726,14 +7721,14 @@ int bpf_program__nth_fd(const struct bpf_program *prog, int n)if (n >= prog->instances.nr || n <0 ) {pr_warn ("Can't get the %dth fd from program %s: only %d instances\n" , - n, prog->section_name, prog->instances.nr); + n, prog->name, prog->instances.nr);return -EINVAL ; } fd = prog->instances.fds[n];if (fd <0 ) {pr_warn ("%dth instance of program '%s' is invalid\n" , - n, prog->section_name); + n, prog->name);return -ENOENT ; }
@@ -8170,7 +8165,7 @@ static int bpf_object__collect_st_ops_relos(struct bpf_object *obj,if (prog->type ==BPF_PROG_TYPE_UNSPEC ) {const struct bpf_sec_def *sec_def; - sec_def =find_sec_def (prog->section_name); + sec_def =find_sec_def (prog->sec_name);if (sec_def && sec_def->prog_type !=BPF_PROG_TYPE_STRUCT_OPS ) {/* for pr_warn */
@@ -8193,7 +8188,7 @@ static int bpf_object__collect_st_ops_relos(struct bpf_object *obj, invalid_prog:pr_warn ("struct_ops reloc %s: cannot use prog %s in sec %s with type %u attach_btf_id %u expected_attach_type %u for func ptr %s\n" , - map->name, prog->name, prog->section_name, prog->type, + map->name, prog->name, prog->sec_name, prog->type, prog->attach_btf_id, prog->expected_attach_type, name);return -EINVAL ; }
@@ -8297,7 +8292,7 @@ static int libbpf_find_attach_btf_id(struct bpf_program *prog) {enum bpf_attach_type attach_type = prog->expected_attach_type;__u32 attach_prog_fd = prog->attach_prog_fd; -const char *name = prog->section_name; +const char *name = prog->sec_name;int i, err;if (!name)
@@ -8824,14 +8819,14 @@ struct bpf_link *bpf_program__attach_perf_event(struct bpf_program *prog,int prog_fd, err;if (pfd <0 ) { -pr_warn ("program '%s': invalid perf event FD %d\n" , -bpf_program__title (prog, false), pfd); +pr_warn ("prog '%s': invalid perf event FD %d\n" , + prog->name, pfd);return ERR_PTR (-EINVAL ); } prog_fd =bpf_program__fd (prog);if (prog_fd <0 ) { -pr_warn ("program '%s': can't attach BPF program w/o FD (did you load it?)\n" , -bpf_program__title (prog, false)); +pr_warn ("prog '%s': can't attach BPF program w/o FD (did you load it?)\n" , + prog->name);return ERR_PTR (-EINVAL ); }
@@ -8844,20 +8839,18 @@ struct bpf_link *bpf_program__attach_perf_event(struct bpf_program *prog,if (ioctl (pfd,PERF_EVENT_IOC_SET_BPF , prog_fd) <0 ) { err = -errno;free (link); -pr_warn ("program '%s': failed to attach to pfd %d: %s\n" , -bpf_program__title (prog, false), pfd, -libbpf_strerror_r (err, errmsg,sizeof (errmsg))); +pr_warn ("prog '%s': failed to attach to pfd %d: %s\n" , + prog->name, pfd,libbpf_strerror_r (err, errmsg,sizeof (errmsg)));if (err == -EPROTO ) -pr_warn ("program '%s': try add PERF_SAMPLE_CALLCHAIN to or remove exclude_callchain_[kernel|user] from pfd %d\n" , -bpf_program__title (prog, false), pfd); +pr_warn ("prog '%s': try add PERF_SAMPLE_CALLCHAIN to or remove exclude_callchain_[kernel|user] from pfd %d\n" , + prog->name, pfd);return ERR_PTR (err); }if (ioctl (pfd,PERF_EVENT_IOC_ENABLE ,0 ) <0 ) { err = -errno;free (link); -pr_warn ("program '%s': failed to enable pfd %d: %s\n" , -bpf_program__title (prog, false), pfd, -libbpf_strerror_r (err, errmsg,sizeof (errmsg))); +pr_warn ("prog '%s': failed to enable pfd %d: %s\n" , + prog->name, pfd,libbpf_strerror_r (err, errmsg,sizeof (errmsg)));return ERR_PTR (err); }return link;
@@ -8979,9 +8972,8 @@ struct bpf_link *bpf_program__attach_kprobe(struct bpf_program *prog, pfd =perf_event_open_probe (false/* uprobe */ , retprobe, func_name,0 /* offset */ ,-1 /* pid */ );if (pfd <0 ) { -pr_warn ("program '%s': failed to create %s '%s' perf event: %s\n" , -bpf_program__title (prog, false), - retprobe ?"kretprobe" :"kprobe" , func_name, +pr_warn ("prog '%s': failed to create %s '%s' perf event: %s\n" , + prog->name, retprobe ?"kretprobe" :"kprobe" , func_name,libbpf_strerror_r (pfd, errmsg,sizeof (errmsg)));return ERR_PTR (pfd); }
@@ -8989,9 +8981,8 @@ struct bpf_link *bpf_program__attach_kprobe(struct bpf_program *prog,if (IS_ERR (link)) {close (pfd); err =PTR_ERR (link); -pr_warn ("program '%s': failed to attach to %s '%s': %s\n" , -bpf_program__title (prog, false), - retprobe ?"kretprobe" :"kprobe" , func_name, +pr_warn ("prog '%s': failed to attach to %s '%s': %s\n" , + prog->name, retprobe ?"kretprobe" :"kprobe" , func_name,libbpf_strerror_r (err, errmsg,sizeof (errmsg)));return link; }
@@ -9004,7 +8995,7 @@ static struct bpf_link *attach_kprobe(const struct bpf_sec_def *sec,const char *func_name;bool retprobe; - func_name =bpf_program__title (prog, false) + sec->len; + func_name = prog->sec_name + sec->len; retprobe =strcmp (sec->sec,"kretprobe/" ) ==0 ;return bpf_program__attach_kprobe (prog, retprobe, func_name);
@@ -9022,9 +9013,8 @@ struct bpf_link *bpf_program__attach_uprobe(struct bpf_program *prog, pfd =perf_event_open_probe (true/* uprobe */ , retprobe, binary_path, func_offset, pid);if (pfd <0 ) { -pr_warn ("program '%s': failed to create %s '%s:0x%zx' perf event: %s\n" , -bpf_program__title (prog, false), - retprobe ?"uretprobe" :"uprobe" , +pr_warn ("prog '%s': failed to create %s '%s:0x%zx' perf event: %s\n" , + prog->name, retprobe ?"uretprobe" :"uprobe" , binary_path, func_offset,libbpf_strerror_r (pfd, errmsg,sizeof (errmsg)));return ERR_PTR (pfd);
@@ -9033,9 +9023,8 @@ struct bpf_link *bpf_program__attach_uprobe(struct bpf_program *prog,if (IS_ERR (link)) {close (pfd); err =PTR_ERR (link); -pr_warn ("program '%s': failed to attach to %s '%s:0x%zx': %s\n" , -bpf_program__title (prog, false), - retprobe ?"uretprobe" :"uprobe" , +pr_warn ("prog '%s': failed to attach to %s '%s:0x%zx': %s\n" , + prog->name, retprobe ?"uretprobe" :"uprobe" , binary_path, func_offset,libbpf_strerror_r (err, errmsg,sizeof (errmsg)));return link;
@@ -9103,9 +9092,8 @@ struct bpf_link *bpf_program__attach_tracepoint(struct bpf_program *prog, pfd =perf_event_open_tracepoint (tp_category, tp_name);if (pfd <0 ) { -pr_warn ("program '%s': failed to create tracepoint '%s/%s' perf event: %s\n" , -bpf_program__title (prog, false), - tp_category, tp_name, +pr_warn ("prog '%s': failed to create tracepoint '%s/%s' perf event: %s\n" , + prog->name, tp_category, tp_name,libbpf_strerror_r (pfd, errmsg,sizeof (errmsg)));return ERR_PTR (pfd); }
@@ -9113,9 +9101,8 @@ struct bpf_link *bpf_program__attach_tracepoint(struct bpf_program *prog,if (IS_ERR (link)) {close (pfd); err =PTR_ERR (link); -pr_warn ("program '%s': failed to attach to tracepoint '%s/%s': %s\n" , -bpf_program__title (prog, false), - tp_category, tp_name, +pr_warn ("prog '%s': failed to attach to tracepoint '%s/%s': %s\n" , + prog->name, tp_category, tp_name,libbpf_strerror_r (err, errmsg,sizeof (errmsg)));return link; }
@@ -9128,7 +9115,7 @@ static struct bpf_link *attach_tp(const struct bpf_sec_def *sec,char *sec_name, *tp_cat, *tp_name;struct bpf_link *link; - sec_name =strdup (bpf_program__title (prog, false)); + sec_name =strdup (prog->sec_name);if (!sec_name)return ERR_PTR (-ENOMEM );
@@ -9157,8 +9144,7 @@ struct bpf_link *bpf_program__attach_raw_tracepoint(struct bpf_program *prog, prog_fd =bpf_program__fd (prog);if (prog_fd <0 ) { -pr_warn ("program '%s': can't attach before loaded\n" , -bpf_program__title (prog, false)); +pr_warn ("prog '%s': can't attach before loaded\n" , prog->name);return ERR_PTR (-EINVAL ); }
@@ -9171,9 +9157,8 @@ struct bpf_link *bpf_program__attach_raw_tracepoint(struct bpf_program *prog,if (pfd <0 ) { pfd = -errno;free (link); -pr_warn ("program '%s': failed to attach to raw tracepoint '%s': %s\n" , -bpf_program__title (prog, false), tp_name, -libbpf_strerror_r (pfd, errmsg,sizeof (errmsg))); +pr_warn ("prog '%s': failed to attach to raw tracepoint '%s': %s\n" , + prog->name, tp_name,libbpf_strerror_r (pfd, errmsg,sizeof (errmsg)));return ERR_PTR (pfd); } link->fd = pfd;
@@ -9183,7 +9168,7 @@ struct bpf_link *bpf_program__attach_raw_tracepoint(struct bpf_program *prog,static struct bpf_link *attach_raw_tp (const struct bpf_sec_def *sec,struct bpf_program *prog) { -const char *tp_name =bpf_program__title (prog, false) + sec->len; +const char *tp_name = prog->sec_name + sec->len;return bpf_program__attach_raw_tracepoint (prog, tp_name); }
@@ -9197,8 +9182,7 @@ static struct bpf_link *bpf_program__attach_btf_id(struct bpf_program *prog) prog_fd =bpf_program__fd (prog);if (prog_fd <0 ) { -pr_warn ("program '%s': can't attach before loaded\n" , -bpf_program__title (prog, false)); +pr_warn ("prog '%s': can't attach before loaded\n" , prog->name);return ERR_PTR (-EINVAL ); }
@@ -9211,9 +9195,8 @@ static struct bpf_link *bpf_program__attach_btf_id(struct bpf_program *prog)if (pfd <0 ) { pfd = -errno;free (link); -pr_warn ("program '%s': failed to attach: %s\n" , -bpf_program__title (prog, false), -libbpf_strerror_r (pfd, errmsg,sizeof (errmsg))); +pr_warn ("prog '%s': failed to attach: %s\n" , + prog->name,libbpf_strerror_r (pfd, errmsg,sizeof (errmsg)));return ERR_PTR (pfd); } link->fd = pfd;
@@ -9259,8 +9242,7 @@ bpf_program__attach_fd(struct bpf_program *prog, int target_fd, prog_fd =bpf_program__fd (prog);if (prog_fd <0 ) { -pr_warn ("program '%s': can't attach before loaded\n" , -bpf_program__title (prog, false)); +pr_warn ("prog '%s': can't attach before loaded\n" , prog->name);return ERR_PTR (-EINVAL ); }
@@ -9274,8 +9256,8 @@ bpf_program__attach_fd(struct bpf_program *prog, int target_fd,if (link_fd <0 ) { link_fd = -errno;free (link); -pr_warn ("program '%s': failed to attach to %s: %s\n" , -bpf_program__title (prog, false), target_name, +pr_warn ("prog '%s': failed to attach to %s: %s\n" , + prog->name, target_name,libbpf_strerror_r (link_fd, errmsg,sizeof (errmsg)));return ERR_PTR (link_fd); }
@@ -9319,8 +9301,7 @@ bpf_program__attach_iter(struct bpf_program *prog, prog_fd =bpf_program__fd (prog);if (prog_fd <0 ) { -pr_warn ("program '%s': can't attach before loaded\n" , -bpf_program__title (prog, false)); +pr_warn ("prog '%s': can't attach before loaded\n" , prog->name);return ERR_PTR (-EINVAL ); }
@@ -9334,9 +9315,8 @@ bpf_program__attach_iter(struct bpf_program *prog,if (link_fd <0 ) { link_fd = -errno;free (link); -pr_warn ("program '%s': failed to attach to iterator: %s\n" , -bpf_program__title (prog, false), -libbpf_strerror_r (link_fd, errmsg,sizeof (errmsg))); +pr_warn ("prog '%s': failed to attach to iterator: %s\n" , + prog->name,libbpf_strerror_r (link_fd, errmsg,sizeof (errmsg)));return ERR_PTR (link_fd); } link->fd = link_fd;
@@ -9347,7 +9327,7 @@ struct bpf_link *bpf_program__attach(struct bpf_program *prog) {const struct bpf_sec_def *sec_def; - sec_def =find_sec_def (bpf_program__title (prog, false)); + sec_def =find_sec_def (prog->sec_name);if (!sec_def || !sec_def->attach_fn)return ERR_PTR (-ESRCH );
@@ -10417,12 +10397,11 @@ int bpf_object__attach_skeleton(struct bpf_object_skeleton *s)struct bpf_program *prog = *s->progs[i].prog;struct bpf_link **link = s->progs[i].link;const struct bpf_sec_def *sec_def; -const char *sec_name =bpf_program__title (prog, false);if (!prog->load)continue ; - sec_def =find_sec_def (sec_name); + sec_def =find_sec_def (prog->sec_name);if (!sec_def || !sec_def->attach_fn)continue ;
tools/lib/bpf/libbpf.h +3 -2
@@ -198,8 +198,9 @@ LIBBPF_API void bpf_program__set_ifindex(struct bpf_program *prog, __u32 ifindex); LIBBPF_API const char *bpf_program__name(const struct bpf_program *prog); -LIBBPF_API const char *bpf_program__title(const struct bpf_program *prog, -bool needs_copy); +LIBBPF_API const char *bpf_program__section_name(const struct bpf_program *prog); +LIBBPF_API LIBBPF_DEPRECATED("BPF program title is confusing term; please use bpf_program__section_name() instead") +const char *bpf_program__title (const struct bpf_program *prog,bool needs_copy); LIBBPF_API bool bpf_program__autoload(const struct bpf_program *prog); LIBBPF_API int bpf_program__set_autoload(struct bpf_program *prog, bool autoload);
tools/lib/bpf/libbpf.map +1 -0
@@ -302,6 +302,7 @@ LIBBPF_0.1.0 { LIBBPF_0.2.0 { global: + bpf_program__section_name; perf_buffer__buffer_cnt; perf_buffer__buffer_fd; perf_buffer__epoll_fd;
Signature · 1 line
2.24.1
PATCH v2 11/14 selftests/bpf: turn fexit_bpf2bpf into test with subtests
In reply to Andrii NakryikoThere are clearly 4 subtests, so make it official.
--- .../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 21 ++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-)
tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c +14 -7
@@ -208,11 +208,18 @@ static void test_func_map_prog_compatibility(void)void test_fexit_bpf2bpf (void ) { -test_target_no_callees (); -test_target_yes_callees (); -test_func_replace (); -test_func_replace_verify (); -test_func_sockmap_update (); -test_func_replace_return_code (); -test_func_map_prog_compatibility (); +if (test__start_subtest ("target_no_callees" )) +test_target_no_callees (); +if (test__start_subtest ("target_yes_callees" )) +test_target_yes_callees (); +if (test__start_subtest ("func_replace" )) +test_func_replace (); +if (test__start_subtest ("func_replace_verify" )) +test_func_replace_verify (); +if (test__start_subtest ("func_sockmap_update" )) +test_func_sockmap_update (); +if (test__start_subtest ("func_replace_return_code" )) +test_func_replace_return_code (); +if (test__start_subtest ("func_map_prog_compatibility" )) +test_func_map_prog_compatibility (); }
Signature · 1 line
2.24.1
PATCH v2 12/14 selftests/bpf: convert pyperf, strobemeta, and l4lb_noinline to __noinline
In reply to Andrii NakryikoConvert few bigger selftests utilizing helper functions from __always_inline to __noinline to excercise libbpf's bpf2bpf handling logic. Also split l4lb_all selftest into two sub-tests.
--- .../selftests/bpf/prog_tests/l4lb_all.c | 9 ++-- tools/testing/selftests/bpf/progs/pyperf.h | 10 ++--- .../testing/selftests/bpf/progs/strobemeta.h | 15 ++++--- .../selftests/bpf/progs/test_l4lb_noinline.c | 41 +++++++++---------- 4 files changed, 34 insertions(+), 41 deletions(-)
tools/testing/selftests/bpf/prog_tests/l4lb_all.c +4 -5
@@ -80,9 +80,8 @@ static void test_l4lb(const char *file)void test_l4lb_all (void ) { -const char *file1 ="./test_l4lb.o" ; -const char *file2 ="./test_l4lb_noinline.o" ; - -test_l4lb (file1); -test_l4lb (file2); +if (test__start_subtest ("l4lb_inline" )) +test_l4lb ("test_l4lb.o" ); +if (test__start_subtest ("l4lb_noinline" )) +test_l4lb ("test_l4lb_noinline.o" ); }
tools/testing/selftests/bpf/progs/pyperf.h +4 -6
@@ -67,7 +67,7 @@ typedef struct {void * co_name;// PyCodeObject.co_name } FrameData; -static __always_inline void *get_thread_state(void *tls_base, PidData *pidData) +static __noinline void *get_thread_state(void *tls_base, PidData *pidData) {void * thread_state;int key;
@@ -154,12 +154,10 @@ struct {__uint (value_size,sizeof (long long ) *127 ); } stackmap SEC(".maps"); -#ifdef GLOBAL_FUNC -__attribute__((noinline)) -#else -static __always_inline +#ifndef GLOBAL_FUNC +static#endif -int __on_event(struct bpf_raw_tracepoint_args *ctx) +__noinline int __on_event(struct bpf_raw_tracepoint_args *ctx) {uint64_t pid_tgid =bpf_get_current_pid_tgid ();pid_t pid = (pid_t)(pid_tgid >>32 );
tools/testing/selftests/bpf/progs/strobemeta.h +7 -8
@@ -266,8 +266,7 @@ struct tls_index {uint64_t offset; }; -static __always_inline void *calc_location(struct strobe_value_loc *loc, -void *tls_base) +static __noinline void *calc_location(struct strobe_value_loc *loc, void *tls_base) {/* * tls_mode value is:
@@ -327,10 +326,10 @@ static __always_inline void *calc_location(struct strobe_value_loc *loc, : NULL; } -static __always_inline void read_int_var(struct strobemeta_cfg *cfg, -size_t idx,void *tls_base, -struct strobe_value_generic *value, -struct strobemeta_payload *data) +static __noinline void read_int_var(struct strobemeta_cfg *cfg, +size_t idx,void *tls_base, +struct strobe_value_generic *value, +struct strobemeta_payload *data) {void *location =calc_location (&cfg->int_locs[idx], tls_base);if (!location)
@@ -440,8 +439,8 @@ static __always_inline void *read_map_var(struct strobemeta_cfg *cfg,* read_strobe_meta returns NULL, if no metadata was read; otherwise returns * pointer to *right after* payload ends */ -static __always_inline void *read_strobe_meta(struct task_struct *task, -struct strobemeta_payload *data) +static __noinline void *read_strobe_meta(struct task_struct *task, +struct strobemeta_payload *data) {pid_t pid =bpf_get_current_pid_tgid () >>32 ;struct strobe_value_generic value = {0 };
tools/testing/selftests/bpf/progs/test_l4lb_noinline.c +19 -22
107 lines not on this page · raw message
Signature · 1 line
2.24.1
Re: [PATCH v2 bpf-next 12/14] selftests/bpf: convert pyperf, strobemeta, and l4lb_noinline to __noinline
Quoted 34 lines · Andrii Nakryiko
tools/testing/selftests/bpf/progs/pyperf.h +4
-6@@ -67,7 +67,7 @@ typedef struct {void * co_name;// PyCodeObject.co_name } FrameData; -static __always_inline void *get_thread_state(void *tls_base, PidData *pidData) +static __noinline void *get_thread_state(void *tls_base, PidData *pidData) {void * thread_state;int key;@@ -154,12 +154,10 @@ struct {__uint (value_size,sizeof (long long ) *127 ); } stackmap SEC(".maps"); -#ifdef GLOBAL_FUNC -__attribute__((noinline)) -#else -static __always_inline +#ifndef GLOBAL_FUNC +static#endif -int __on_event(struct bpf_raw_tracepoint_args *ctx) +__noinline int __on_event(struct bpf_raw_tracepoint_args *ctx) {uint64_t pid_tgid =bpf_get_current_pid_tgid ();pid_t pid = (pid_t)(pid_tgid >>32 );tools/testing/selftests/bpf/progs/strobemeta.h +0
-0
tools/testing/selftests/bpf/progs/strobemeta.h struct tls_index { · line 269
Quoted 6 lines
tools/testing/selftests/bpf/progs/strobemeta.h +1
-2@@ -266,8 +266,7 @@ struct tls_index {uint64_t offset; }; -static __always_inline void *calc_location(struct strobe_value_loc *loc, -void *tls_base) +static __noinline void *calc_location(struct strobe_value_loc *loc, void *tls_base)
hmm. this reduces the existing test coverage. Unless I'm misreading it. Could you keep existing strobemta tests and add new one? With new ifdefs. Like this GLOBAL_FUNC.
Re: [PATCH v2 bpf-next 12/14] selftests/bpf: convert pyperf, strobemeta, and l4lb_noinline to __noinline
Quoted 39 lines · Alexei Starovoitov
Quoted 37 lines · Andrii Nakryiko
tools/testing/selftests/bpf/progs/pyperf.h +4
-6@@ -67,7 +67,7 @@ typedef struct {void * co_name;// PyCodeObject.co_name } FrameData; -static __always_inline void *get_thread_state(void *tls_base, PidData *pidData) +static __noinline void *get_thread_state(void *tls_base, PidData *pidData) {void * thread_state;int key;@@ -154,12 +154,10 @@ struct {__uint (value_size,sizeof (long long ) *127 ); } stackmap SEC(".maps"); -#ifdef GLOBAL_FUNC -__attribute__((noinline)) -#else -static __always_inline +#ifndef GLOBAL_FUNC +static#endif -int __on_event(struct bpf_raw_tracepoint_args *ctx) +__noinline int __on_event(struct bpf_raw_tracepoint_args *ctx) {uint64_t pid_tgid =bpf_get_current_pid_tgid ();pid_t pid = (pid_t)(pid_tgid >>32 );tools/testing/selftests/bpf/progs/strobemeta.h +0
-0@@ -266,8 +266,7 @@ struct tls_index {uint64_t offset; };
Quoted 7 lines
tools/testing/selftests/bpf/progs/strobemeta.h struct tls_index { · line 269
Quoted 3 lines
tools/testing/selftests/bpf/progs/strobemeta.h +1
-2-static __always_inline void *calc_location(struct strobe_value_loc *loc, -void *tls_base) +static __noinline void *calc_location(struct strobe_value_loc *loc, void *tls_base)hmm. this reduces the existing test coverage. Unless I'm misreading it. Could you keep existing strobemta tests and add new one? With new ifdefs. Like this
GLOBAL_FUNC.
Oh, you mean testing single BPF program complexity when everything is inlined? Yeah, haven't thought about that. Ok, I'll add new variants with or without subprogram calls.
PATCH v2 13/14 selftests/bpf: modernize xdp_noinline test w/ skeleton and __noinline
In reply to Andrii NakryikoUpdate xdp_noinline to use BPF skeleton and force __noinline on helper sub-programs. Also, split existing logic into v4- and v6-only to complicate sub-program calling patterns (partially overlapped sets of functions for entry-level BPF programs).
--- .../selftests/bpf/prog_tests/xdp_noinline.c | 49 +++++++------------ .../selftests/bpf/progs/test_xdp_noinline.c | 36 ++++++++++---- 2 files changed, 43 insertions(+), 42 deletions(-)
tools/testing/selftests/bpf/prog_tests/xdp_noinline.c +17 -32
@@ -1,11 +1,12 @@// SPDX-License-Identifier: GPL-2.0 #include <test_progs.h> #include <network_helpers.h> +#include "test_xdp_noinline.skel.h" void test_xdp_noinline (void ) { -const char *file ="./test_xdp_noinline.o" ;unsigned int nr_cpus =bpf_num_possible_cpus (); +struct test_xdp_noinline *skel;struct vip key = {.protocol =6 };struct vip_meta {__u32 flags;
@@ -25,58 +26,42 @@ void test_xdp_noinline(void) } real_def = {.dst = MAGIC_VAL};__u32 ch_key =11 , real_num =3 ;__u32 duration, retval, size; -int err, i, prog_fd, map_fd; +int err, i;__u64 bytes =0 , pkts =0 ; -struct bpf_object *obj;char buf[128 ];u32 *magic = (u32 *)buf; - err =bpf_prog_load (file,BPF_PROG_TYPE_XDP , &obj, &prog_fd); -if (CHECK_FAIL (err)) + skel =test_xdp_noinline__open_and_load (); +if (CHECK (!skel,"skel_open_and_load" ,"failed\n" ))return ; - map_fd =bpf_find_map (__func__, obj,"vip_map" ); -if (map_fd <0 ) - goto out; -bpf_map_update_elem (map_fd, &key, &value,0 ); +bpf_map_update_elem (bpf_map__fd (skel->maps.vip_map), &key, &value,0 ); +bpf_map_update_elem (bpf_map__fd (skel->maps.ch_rings), &ch_key, &real_num,0 ); +bpf_map_update_elem (bpf_map__fd (skel->maps.reals), &real_num, &real_def,0 ); - map_fd =bpf_find_map (__func__, obj,"ch_rings" ); -if (map_fd <0 ) - goto out; -bpf_map_update_elem (map_fd, &ch_key, &real_num,0 ); - - map_fd =bpf_find_map (__func__, obj,"reals" ); -if (map_fd <0 ) - goto out; -bpf_map_update_elem (map_fd, &real_num, &real_def,0 ); - - err =bpf_prog_test_run (prog_fd,NUM_ITER , &pkt_v4,sizeof (pkt_v4), + err =bpf_prog_test_run (bpf_program__fd (skel->progs.balancer_ingress_v4), +NUM_ITER , &pkt_v4,sizeof (pkt_v4), buf, &size, &retval, &duration);CHECK (err || retval !=1 || size !=54 || *magic !=MAGIC_VAL ,"ipv4" ,"err %d errno %d retval %d size %d magic %x\n" , err, errno, retval, size, *magic); - err =bpf_prog_test_run (prog_fd,NUM_ITER , &pkt_v6,sizeof (pkt_v6), + err =bpf_prog_test_run (bpf_program__fd (skel->progs.balancer_ingress_v6), +NUM_ITER , &pkt_v6,sizeof (pkt_v6), buf, &size, &retval, &duration);CHECK (err || retval !=1 || size !=74 || *magic !=MAGIC_VAL ,"ipv6" ,"err %d errno %d retval %d size %d magic %x\n" , err, errno, retval, size, *magic); - map_fd =bpf_find_map (__func__, obj,"stats" ); -if (map_fd <0 ) - goto out; -bpf_map_lookup_elem (map_fd, &stats_key, stats); +bpf_map_lookup_elem (bpf_map__fd (skel->maps.stats), &stats_key, stats);for (i =0 ; i < nr_cpus; i++) { bytes += stats[i].bytes; pkts += stats[i].pkts; } -if (CHECK_FAIL (bytes !=MAGIC_BYTES *NUM_ITER *2 || - pkts !=NUM_ITER *2 )) { -printf ("test_xdp_noinline:FAIL:stats %lld %lld\n" , - bytes, pkts); - } -out: -bpf_object__close (obj); +CHECK (bytes !=MAGIC_BYTES *NUM_ITER *2 || pkts !=NUM_ITER *2 , +"stats" ,"bytes %lld pkts %lld\n" , + (unsigned long long )bytes, (unsigned long long )pkts); +test_xdp_noinline__destroy (skel); }
tools/testing/selftests/bpf/progs/test_xdp_noinline.c +26 -10
@@ -16,7 +16,7 @@#include <bpf/bpf_helpers.h> #include <bpf/bpf_endian.h> -static __u32 rol32 (__u32 word,unsigned int shift) +static __always_inline __u32 rol32(__u32 word, unsigned int shift) {return (word << shift) | (word >> ((-shift) &31 )); }
@@ -49,7 +49,7 @@ static __u32 rol32(__u32 word, unsigned int shift)typedef unsigned int u32 ; -static __attribute__ ((noinline)) +static __noinline u32 jhash(const void *key, u32 length, u32 initval) {u32 a, b, c;
@@ -86,7 +86,7 @@ u32 jhash(const void *key, u32 length, u32 initval)return c; } -__attribute__ ((noinline)) +__noinline u32 __jhash_nwords(u32 a, u32 b, u32 c, u32 initval) { a += initval;
@@ -96,7 +96,7 @@ u32 __jhash_nwords(u32 a, u32 b, u32 c, u32 initval)return c; } -__attribute__ ((noinline)) +__noinline u32 jhash_2words(u32 a, u32 b, u32 initval) {return __jhash_nwords (a, b,0 , initval +JHASH_INITVAL + (2 <<2 ));
@@ -213,7 +213,7 @@ struct eth_hdr {unsigned short eth_proto; }; -static inline __u64 calc_offset (bool is_ipv6,bool is_icmp) +static __noinline __u64 calc_offset(bool is_ipv6, bool is_icmp) {__u64 off =sizeof (struct eth_hdr );if (is_ipv6) {
@@ -797,8 +797,8 @@ static int process_packet(void *data, __u64 off, void *data_end,return XDP_DROP ; } -__attribute__ ((section ("xdp-test" ), used)) -int balancer_ingress (struct xdp_md *ctx) +SEC("xdp-test-v4") +int balancer_ingress_v4 (struct xdp_md *ctx) {void *data = (void *)(long )ctx->data;void *data_end = (void *)(long )ctx->data_end;
@@ -812,11 +812,27 @@ int balancer_ingress(struct xdp_md *ctx) eth_proto =bpf_ntohs (eth->eth_proto);if (eth_proto ==ETH_P_IP )return process_packet (data, nh_off, data_end,0 , ctx); -else if (eth_proto ==ETH_P_IPV6 ) +else +return XDP_DROP ; +} + +SEC("xdp-test-v6") +int balancer_ingress_v6 (struct xdp_md *ctx) +{ +void *data = (void *)(long )ctx->data; +void *data_end = (void *)(long )ctx->data_end; +struct eth_hdr *eth = data; +__u32 eth_proto; +__u32 nh_off; + + nh_off =sizeof (struct eth_hdr ); +if (data + nh_off > data_end) +return XDP_DROP ; + eth_proto =bpf_ntohs (eth->eth_proto); +if (eth_proto ==ETH_P_IPV6 )return process_packet (data, nh_off, data_end,1 , ctx);else return XDP_DROP ; } -char _license[] __attribute__ ((section("license"), used)) = "GPL"; -int _version __attribute__ ((section("version"), used)) = 1; +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.24.1
PATCH v2 14/14 selftests/bpf: convert cls_redirect selftest to use __noinline
In reply to Andrii NakryikoAs one of the most complicated and close-to-real-world programs, cls_redirect is a good candidate to excercise libbpf's logic of handling bpf2bpf calls. So convert it to explicit __noinline for majority of functions except few most basic ones. If those few functions are inlined, verifier starts to complain about program instruction limit of 1mln instructions being exceeded, most probably due to instruction overhead of doing a sub-program call.
--- .../selftests/bpf/progs/test_cls_redirect.c | 99 ++++++++++--------- 1 file changed, 50 insertions(+), 49 deletions(-)
tools/testing/selftests/bpf/progs/test_cls_redirect.c +50 -49
@@ -125,7 +125,7 @@ typedef struct buf {uint8_t *const tail; } buf_t; -static size_t buf_off (const buf_t *buf) +static __always_inline size_t buf_off(const buf_t *buf) {/* Clang seems to optimize constructs like * a - b + c
@@ -145,7 +145,7 @@ static size_t buf_off(const buf_t *buf)return off; } -static bool buf_copy (buf_t *buf,void *dst,size_t len) +static __always_inline bool buf_copy(buf_t *buf, void *dst, size_t len) {if (bpf_skb_load_bytes (buf->skb,buf_off (buf), dst, len)) {return false;
@@ -155,7 +155,7 @@ static bool buf_copy(buf_t *buf, void *dst, size_t len)return true; } -static bool buf_skip (buf_t *buf,const size_t len) +static __always_inline bool buf_skip(buf_t *buf, const size_t len) {/* Check whether off + len is valid in the non-linear part. */ if (buf_off (buf) + len > buf->skb->len) {
@@ -173,7 +173,7 @@ static bool buf_skip(buf_t *buf, const size_t len)* If scratch is not NULL, the function will attempt to load non-linear * data via bpf_skb_load_bytes. On success, scratch is returned. */ -static void *buf_assign (buf_t *buf,const size_t len,void *scratch) +static __always_inline void *buf_assign(buf_t *buf, const size_t len, void *scratch) {if (buf->head + len > buf->tail) {if (scratch ==NULL ) {
@@ -188,7 +188,7 @@ static void *buf_assign(buf_t *buf, const size_t len, void *scratch)return ptr; } -static bool pkt_skip_ipv4_options (buf_t *buf,const struct iphdr *ipv4) +static __noinline bool pkt_skip_ipv4_options(buf_t *buf, const struct iphdr *ipv4) {if (ipv4->ihl <=5 ) {return true;
@@ -197,13 +197,13 @@ static bool pkt_skip_ipv4_options(buf_t *buf, const struct iphdr *ipv4)return buf_skip (buf, (ipv4->ihl -5 ) *4 ); } -static bool ipv4_is_fragment (const struct iphdr *ip) +static __noinline bool ipv4_is_fragment(const struct iphdr *ip) {uint16_t frag_off = ip->frag_off &bpf_htons (IP_OFFSET_MASK );return (ip->frag_off &bpf_htons (IP_MF )) !=0 || frag_off >0 ; } -static struct iphdr *pkt_parse_ipv4 (buf_t *pkt,struct iphdr *scratch) +static __always_inline struct iphdr *pkt_parse_ipv4(buf_t *pkt, struct iphdr *scratch) {struct iphdr *ipv4 =buf_assign (pkt,sizeof (*ipv4), scratch);if (ipv4 ==NULL ) {
@@ -222,7 +222,7 @@ static struct iphdr *pkt_parse_ipv4(buf_t *pkt, struct iphdr *scratch) }/* Parse the L4 ports from a packet, assuming a layout like TCP or UDP. */ -static bool pkt_parse_icmp_l4_ports (buf_t *pkt,flow_ports_t *ports) +static __noinline bool pkt_parse_icmp_l4_ports(buf_t *pkt, flow_ports_t *ports) {if (!buf_copy (pkt, ports,sizeof (*ports))) {return false;
@@ -237,7 +237,7 @@ static bool pkt_parse_icmp_l4_ports(buf_t *pkt, flow_ports_t *ports)return true; } -static uint16_t pkt_checksum_fold (uint32_t csum) +static __noinline uint16_t pkt_checksum_fold(uint32_t csum) {/* The highest reasonable value for an IPv4 header * checksum requires two folds, so we just do that always.
@@ -247,7 +247,7 @@ static uint16_t pkt_checksum_fold(uint32_t csum)return (uint16_t )~csum; } -static void pkt_ipv4_checksum (struct iphdr *iph) +static __noinline void pkt_ipv4_checksum(struct iphdr *iph) { iph->check =0 ;
@@ -268,10 +268,11 @@ static void pkt_ipv4_checksum(struct iphdr *iph) iph->check =pkt_checksum_fold (acc); } -static bool pkt_skip_ipv6_extension_headers (buf_t *pkt, -const struct ipv6hdr *ipv6, -uint8_t *upper_proto, -bool *is_fragment) +static __noinline +bool pkt_skip_ipv6_extension_headers(buf_t *pkt, +const struct ipv6hdr *ipv6, +uint8_t *upper_proto, +bool *is_fragment) {/* We understand five extension headers. * https://tools.ietf.org/html/rfc8200#section-4.1 states that all
@@ -336,7 +337,7 @@ static bool pkt_skip_ipv6_extension_headers(buf_t *pkt,* scratch is allocated on the stack. However, this usage should be safe since * it's the callers stack after all. */ -static inline __attribute__((__always_inline__))struct ipv6hdr * +static __always_inline struct ipv6hdr *pkt_parse_ipv6 (buf_t *pkt,struct ipv6hdr *scratch,uint8_t *proto,bool *is_fragment) {
@@ -354,20 +355,20 @@ pkt_parse_ipv6(buf_t *pkt, struct ipv6hdr *scratch, uint8_t *proto,/* Global metrics, per CPU */ -struct bpf_map_def metrics_map SEC("maps") = { - .type = BPF_MAP_TYPE_PERCPU_ARRAY, - .key_size = sizeof(unsigned int), - .value_size = sizeof(metrics_t), - .max_entries = 1, -}; - -static metrics_t *get_global_metrics (void ) +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __uint(max_entries, 1); + __type(key, unsigned int); + __type(value, metrics_t); +} metrics_map SEC(".maps"); + +static __noinline metrics_t *get_global_metrics(void) {uint64_t key =0 ;return bpf_map_lookup_elem (&metrics_map, &key); } -static ret_t accept_locally (struct __sk_buff *skb,encap_headers_t *encap) +static __noinline ret_t accept_locally(struct __sk_buff *skb, encap_headers_t *encap) {const int payload_off = sizeof(*encap) +
@@ -388,8 +389,8 @@ static ret_t accept_locally(struct __sk_buff *skb, encap_headers_t *encap)return bpf_redirect (skb->ifindex,BPF_F_INGRESS ); } -static ret_t forward_with_gre (struct __sk_buff *skb,encap_headers_t *encap, -struct in_addr *next_hop,metrics_t *metrics) +static __noinline ret_t forward_with_gre(struct __sk_buff *skb, encap_headers_t *encap, +struct in_addr *next_hop,metrics_t *metrics) { metrics->forwarded_packets_total_gre++;
@@ -509,8 +510,8 @@ static ret_t forward_with_gre(struct __sk_buff *skb, encap_headers_t *encap,return bpf_redirect (skb->ifindex,0 ); } -static ret_t forward_to_next_hop (struct __sk_buff *skb,encap_headers_t *encap, -struct in_addr *next_hop,metrics_t *metrics) +static __noinline ret_t forward_to_next_hop(struct __sk_buff *skb, encap_headers_t *encap, +struct in_addr *next_hop,metrics_t *metrics) {/* swap L2 addresses */ /* This assumes that packets are received from a router.
@@ -546,7 +547,7 @@ static ret_t forward_to_next_hop(struct __sk_buff *skb, encap_headers_t *encap,return bpf_redirect (skb->ifindex,0 ); } -static ret_t skip_next_hops (buf_t *pkt,int n) +static __noinline ret_t skip_next_hops(buf_t *pkt, int n) {switch (n) {case 1 :
@@ -566,8 +567,8 @@ static ret_t skip_next_hops(buf_t *pkt, int n)* pkt is positioned just after the variable length GLB header * iff the call is successful. */ -static ret_t get_next_hop (buf_t *pkt,encap_headers_t *encap, -struct in_addr *next_hop) +static __noinline ret_t get_next_hop(buf_t *pkt, encap_headers_t *encap, +struct in_addr *next_hop) {if (encap->unigue.next_hop > encap->unigue.hop_count) {return TC_ACT_SHOT ;
@@ -601,8 +602,8 @@ static ret_t get_next_hop(buf_t *pkt, encap_headers_t *encap,* return value, and calling code works while still being "generic" to * IPv4 and IPv6. */ -static uint64_t fill_tuple (struct bpf_sock_tuple *tuple,void *iph, -uint64_t iphlen,uint16_t sport,uint16_t dport) +static __noinline uint64_t fill_tuple(struct bpf_sock_tuple *tuple, void *iph, +uint64_t iphlen,uint16_t sport,uint16_t dport) {switch (iphlen) {case sizeof (struct iphdr ): {
@@ -630,9 +631,9 @@ static uint64_t fill_tuple(struct bpf_sock_tuple *tuple, void *iph, } } -static verdict_t classify_tcp (struct __sk_buff *skb, -struct bpf_sock_tuple *tuple,uint64_t tuplen, -void *iph,struct tcphdr *tcp) +static __noinline verdict_t classify_tcp(struct __sk_buff *skb, +struct bpf_sock_tuple *tuple,uint64_t tuplen, +void *iph,struct tcphdr *tcp) {struct bpf_sock *sk =bpf_skc_lookup_tcp (skb, tuple, tuplen,BPF_F_CURRENT_NETNS ,0 );
@@ -663,8 +664,8 @@ static verdict_t classify_tcp(struct __sk_buff *skb,return UNKNOWN ; } -static verdict_t classify_udp (struct __sk_buff *skb, -struct bpf_sock_tuple *tuple,uint64_t tuplen) +static __noinline verdict_t classify_udp(struct __sk_buff *skb, +struct bpf_sock_tuple *tuple,uint64_t tuplen) {struct bpf_sock *sk =bpf_sk_lookup_udp (skb, tuple, tuplen,BPF_F_CURRENT_NETNS ,0 );
@@ -681,9 +682,9 @@ static verdict_t classify_udp(struct __sk_buff *skb,return UNKNOWN ; } -static verdict_t classify_icmp (struct __sk_buff *skb,uint8_t proto, -struct bpf_sock_tuple *tuple,uint64_t tuplen, -metrics_t *metrics) +static __noinline verdict_t classify_icmp(struct __sk_buff *skb, uint8_t proto, +struct bpf_sock_tuple *tuple,uint64_t tuplen, +metrics_t *metrics) {switch (proto) {case IPPROTO_TCP :
@@ -698,7 +699,7 @@ static verdict_t classify_icmp(struct __sk_buff *skb, uint8_t proto, } } -static verdict_t process_icmpv4 (buf_t *pkt,metrics_t *metrics) +static __noinline verdict_t process_icmpv4(buf_t *pkt, metrics_t *metrics) {struct icmphdr icmp;if (!buf_copy (pkt, &icmp,sizeof (icmp))) {
@@ -745,7 +746,7 @@ static verdict_t process_icmpv4(buf_t *pkt, metrics_t *metrics) sizeof(tuple.ipv4), metrics); } -static verdict_t process_icmpv6 (buf_t *pkt,metrics_t *metrics) +static __noinline verdict_t process_icmpv6(buf_t *pkt, metrics_t *metrics) {struct icmp6hdr icmp6;if (!buf_copy (pkt, &icmp6,sizeof (icmp6))) {
@@ -797,8 +798,8 @@ static verdict_t process_icmpv6(buf_t *pkt, metrics_t *metrics) metrics); } -static verdict_t process_tcp (buf_t *pkt,void *iph,uint64_t iphlen, -metrics_t *metrics) +static __noinline verdict_t process_tcp(buf_t *pkt, void *iph, uint64_t iphlen, +metrics_t *metrics) { metrics->l4_protocol_packets_total_tcp++;
@@ -819,8 +820,8 @@ static verdict_t process_tcp(buf_t *pkt, void *iph, uint64_t iphlen,return classify_tcp (pkt->skb, &tuple, tuplen, iph, tcp); } -static verdict_t process_udp (buf_t *pkt,void *iph,uint64_t iphlen, -metrics_t *metrics) +static __noinline verdict_t process_udp(buf_t *pkt, void *iph, uint64_t iphlen, +metrics_t *metrics) { metrics->l4_protocol_packets_total_udp++;
@@ -837,7 +838,7 @@ static verdict_t process_udp(buf_t *pkt, void *iph, uint64_t iphlen,return classify_udp (pkt->skb, &tuple, tuplen); } -static verdict_t process_ipv4 (buf_t *pkt,metrics_t *metrics) +static __noinline verdict_t process_ipv4(buf_t *pkt, metrics_t *metrics) { metrics->l3_protocol_packets_total_ipv4++;
@@ -874,7 +875,7 @@ static verdict_t process_ipv4(buf_t *pkt, metrics_t *metrics) } } -static verdict_t process_ipv6 (buf_t *pkt,metrics_t *metrics) +static __noinline verdict_t process_ipv6(buf_t *pkt, metrics_t *metrics) { metrics->l3_protocol_packets_total_ipv6++;
Signature · 1 line
2.24.1
Re: [PATCH v2 bpf-next 14/14] selftests/bpf: convert cls_redirect selftest to use __noinline
tools/testing/selftests/bpf/progs/test_cls_redirect.c static bool pkt_skip_ipv4_options(buf_t *buf, const struct iphdr *ipv4) · line 206
Quoted 9 lines · Andrii Nakryiko
tools/testing/selftests/bpf/progs/test_cls_redirect.c +2
-2-static bool ipv4_is_fragment (const struct iphdr *ip) +static __noinline bool ipv4_is_fragment(const struct iphdr *ip) {uint16_t frag_off = ip->frag_off &bpf_htons (IP_OFFSET_MASK );return (ip->frag_off &bpf_htons (IP_MF )) !=0 || frag_off >0 ; } -static struct iphdr *pkt_parse_ipv4 (buf_t *pkt,struct iphdr *scratch) +static __always_inline struct iphdr *pkt_parse_ipv4(buf_t *pkt, struct iphdr *scratch)
similar concern. Could you keep old and add new one with macro magic?
Re: [PATCH v2 bpf-next 14/14] selftests/bpf: convert cls_redirect selftest to use __noinline
Quoted 9 lines · Alexei Starovoitov
Quoted 7 lines · Andrii Nakryiko
tools/testing/selftests/bpf/progs/test_cls_redirect.c +1
-1-static bool ipv4_is_fragment (const struct iphdr *ip) +static __noinline bool ipv4_is_fragment(const struct iphdr *ip) {uint16_t frag_off = ip->frag_off &bpf_htons (IP_OFFSET_MASK );return (ip->frag_off &bpf_htons (IP_MF )) !=0 || frag_off >0 ; }
Quoted 4 lines
tools/testing/selftests/bpf/progs/test_cls_redirect.c static bool pkt_skip_ipv4_options(buf_t *buf, const struct iphdr *ipv4) · line 206
Quoted 2 lines
tools/testing/selftests/bpf/progs/test_cls_redirect.c +1
-1-static struct iphdr *pkt_parse_ipv4 (buf_t *pkt,struct iphdr *scratch) +static __always_inline struct iphdr *pkt_parse_ipv4(buf_t *pkt, struct iphdr *scratch)similar concern. Could you keep old and add new one with macro magic?
Ok.