Introduce BPF trampoline
Subsystems (13)
Alexei Starovoitov
maintainerv3 cover Introduce BPF trampoline
Introduce BPF trampoline that works as a bridge between kernel functions, BPF programs and other BPF programs.
The first use case is fentry/fexit BPF programs that are roughly equivalent to kprobe/kretprobe. Unlike k[ret]probe there is practically zero overhead to call a set of BPF programs before or after kernel function.
The second use case is heavily influenced by pain points in XDP development. BPF trampoline allows attaching similar fentry/fexit BPF program to any networking BPF program. It's now possible to see packets on input and output of any XDP, TC, lwt, cgroup programs without disturbing them. This greatly helps BPF-based network troubleshooting.
The third use case of BPF trampoline will be explored in the follow up patches. The BPF trampoline will be used to dynamicly link BPF programs. It's more generic mechanism than array and link list of programs used in tracing, networking, cgroups. In many cases it can be used as a replacement for bpf_tail_call-based program chaining.
v2->v3:
- Addressed Song's and Andrii's comments
- Fixed few minor bugs discovered while testing
- Added one more libbpf patch
v1->v2:
- Addressed Andrii's comments
- Added more test for fentry/fexit to kernel functions. Including stress test for maximum number of progs per trampoline.
- Fixed a race btf_resolve_helper_id()
- Added a patch to compare BTF types of functions arguments with actual types.
- Added support for attaching BPF program to another BPF program via trampoline
- Converted to use text_poke() API. That's the only viable mechanism to implement BPF-to-BPF attach. BPF-to-kernel attach can be refactored to use register_ftrace_direct() whenever it's available.
Most of the interesting details are in patches 2,3,14,15.
Alexei Starovoitov (18):
bpf: refactor x86 JIT into helpers
bpf: Add bpf_arch_text_poke() helper
bpf: Introduce BPF trampoline
libbpf: Introduce btf__find_by_name_kind()
libbpf: Add support to attach to fentry/fexit tracing progs selftest/bpf: Simple test for fentry/fexit
bpf: Add kernel test functions for fentry testing selftests/bpf: Add test for BPF trampoline
selftests/bpf: Add fexit tests for BPF trampoline selftests/bpf: Add combined fentry/fexit test selftests/bpf: Add stress test for maximum number of progs
bpf: Reserve space for BPF trampoline in BPF programs
bpf: Fix race in btf_resolve_helper_id()
bpf: Compare BTF types of functions arguments with actual types
bpf: Support attaching tracing BPF program to other BPF programs
libbpf: Add support for attaching BPF programs to other BPF programs selftests/bpf: Extend test_pkt_access test
selftests/bpf: Add a test for attaching BPF prog to another BPF prog
and subprog
arch/x86/net/bpf_jit_comp.c | 425 +++++++++++++++--- include/linux/bpf.h | 121 ++++- include/linux/bpf_verifier.h | 1 + include/linux/btf.h | 1 + include/uapi/linux/bpf.h | 3 + kernel/bpf/Makefile | 1 + kernel/bpf/btf.c | 274 ++++++++++- kernel/bpf/core.c | 9 + kernel/bpf/syscall.c | 73 ++- kernel/bpf/trampoline.c | 250 +++++++++++ kernel/bpf/verifier.c | 131 +++++- kernel/trace/bpf_trace.c | 2 - net/bpf/test_run.c | 41 ++ net/core/filter.c | 2 +- tools/include/uapi/linux/bpf.h | 3 + tools/lib/bpf/bpf.c | 9 +- tools/lib/bpf/bpf.h | 5 +- tools/lib/bpf/bpf_helpers.h | 13 + tools/lib/bpf/btf.c | 22 + tools/lib/bpf/btf.h | 2 + tools/lib/bpf/libbpf.c | 154 +++++-- tools/lib/bpf/libbpf.h | 7 +- tools/lib/bpf/libbpf.map | 3 + .../selftests/bpf/prog_tests/fentry_fexit.c | 90 ++++ .../selftests/bpf/prog_tests/fentry_test.c | 64 +++ .../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 76 ++++ .../selftests/bpf/prog_tests/fexit_stress.c | 77 ++++ .../selftests/bpf/prog_tests/fexit_test.c | 64 +++ .../selftests/bpf/prog_tests/kfree_skb.c | 39 +- .../testing/selftests/bpf/progs/fentry_test.c | 90 ++++ .../selftests/bpf/progs/fexit_bpf2bpf.c | 91 ++++ .../testing/selftests/bpf/progs/fexit_test.c | 98 ++++ tools/testing/selftests/bpf/progs/kfree_skb.c | 52 +++ .../selftests/bpf/progs/test_pkt_access.c | 38 +- 34 files changed, 2198 insertions(+), 133 deletions(-) create mode 100644 kernel/bpf/trampoline.c create mode 100644 tools/testing/selftests/bpf/prog_tests/fentry_fexit.c create mode 100644 tools/testing/selftests/bpf/prog_tests/fentry_test.c create mode 100644 tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c create mode 100644 tools/testing/selftests/bpf/prog_tests/fexit_stress.c create mode 100644 tools/testing/selftests/bpf/prog_tests/fexit_test.c create mode 100644 tools/testing/selftests/bpf/progs/fentry_test.c create mode 100644 tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c create mode 100644 tools/testing/selftests/bpf/progs/fexit_test.c
Signature · 1 line
2.23.0
Alexei Starovoitov
maintainerPATCH v3 1/18 bpf: refactor x86 JIT into helpers
Refactor x86 JITing of LDX, STX, CALL instructions into separate helper functions. No functional changes in LDX and STX helpers. There is a minor change in CALL helper. It will populate target address correctly on the first pass of JIT instead of second pass. That won't reduce total number of JIT passes though.
--- arch/x86/net/bpf_jit_comp.c | 153 +++++++++++++++++++++++------------- 1 file changed, 99 insertions(+), 54 deletions(-)
arch/x86/net/bpf_jit_comp.c +99 -54
@@ -198,6 +198,8 @@ struct jit_context {/* Maximum number of bytes emitted while JITing one eBPF insn */ #define BPF_MAX_INSN_SIZE 128#define BPF_INSN_SAFETY 64 +/* number of bytes emit_call() needs to generate call instruction */ +#define X86_CALL_SIZE 5#define PROLOGUE_SIZE 20
@@ -390,6 +392,100 @@ static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) *pprog = prog; } +/* LDX: dst_reg = *(u8*)(src_reg + off) */ +static void emit_ldx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) +{ +u8 *prog = *pprog; +int cnt =0 ; + +switch (size) { +case BPF_B : +/* Emit 'movzx rax, byte ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xB6 ); +break ; +case BPF_H : +/* Emit 'movzx rax, word ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xB7 ); +break ; +case BPF_W : +/* Emit 'mov eax, dword ptr [rax+0x14]' */ +if (is_ereg (dst_reg) ||is_ereg (src_reg)) +EMIT2 (add_2mod (0x40 , src_reg, dst_reg),0x8B ); +else +EMIT1 (0x8B ); +break ; +case BPF_DW : +/* Emit 'mov rax, qword ptr [rax+0x14]' */ +EMIT2 (add_2mod (0x48 , src_reg, dst_reg),0x8B ); +break ; + } +/* +* If insn->off == 0 we can save one extra byte, but +* special case of x86 R13 which always needs an offset +* is not worth the hassle +*/ +if (is_imm8 (off)) +EMIT2 (add_2reg (0x40 , src_reg, dst_reg), off); +else +EMIT1_off32 (add_2reg (0x80 , src_reg, dst_reg), off); + *pprog = prog; +} + +/* STX: *(u8*)(dst_reg + off) = src_reg */ +static void emit_stx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) +{ +u8 *prog = *pprog; +int cnt =0 ; + +switch (size) { +case BPF_B : +/* Emit 'mov byte ptr [rax + off], al' */ +if (is_ereg (dst_reg) ||is_ereg (src_reg) || +/* We have to add extra byte for x86 SIL, DIL regs */ + src_reg ==BPF_REG_1 || src_reg ==BPF_REG_2 ) +EMIT2 (add_2mod (0x40 , dst_reg, src_reg),0x88 ); +else +EMIT1 (0x88 ); +break ; +case BPF_H : +if (is_ereg (dst_reg) ||is_ereg (src_reg)) +EMIT3 (0x66 ,add_2mod (0x40 , dst_reg, src_reg),0x89 ); +else +EMIT2 (0x66 ,0x89 ); +break ; +case BPF_W : +if (is_ereg (dst_reg) ||is_ereg (src_reg)) +EMIT2 (add_2mod (0x40 , dst_reg, src_reg),0x89 ); +else +EMIT1 (0x89 ); +break ; +case BPF_DW : +EMIT2 (add_2mod (0x48 , dst_reg, src_reg),0x89 ); +break ; + } +if (is_imm8 (off)) +EMIT2 (add_2reg (0x40 , dst_reg, src_reg), off); +else +EMIT1_off32 (add_2reg (0x80 , dst_reg, src_reg), + off); + *pprog = prog; +} + +static int emit_call (u8 **pprog,void *func,void *ip) +{ +u8 *prog = *pprog; +int cnt =0 ; +s64 offset; + + offset = func - (ip +X86_CALL_SIZE ); +if (!is_simm32 (offset)) { +pr_err ("Target call %p is out of range\n" , func); +return -EINVAL ; + } +EMIT1_off32 (0xE8 , offset); + *pprog = prog; +return 0 ; +} static bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs, int trapnr,
@@ -773,68 +869,22 @@ st: if (is_imm8(insn->off))/* STX: *(u8*)(dst_reg + off) = src_reg */ case BPF_STX |BPF_MEM |BPF_B : -/* Emit 'mov byte ptr [rax + off], al' */ -if (is_ereg (dst_reg) ||is_ereg (src_reg) || -/* We have to add extra byte for x86 SIL, DIL regs */ - src_reg ==BPF_REG_1 || src_reg ==BPF_REG_2 ) -EMIT2 (add_2mod (0x40 , dst_reg, src_reg),0x88 ); -else -EMIT1 (0x88 ); - goto stx;case BPF_STX |BPF_MEM |BPF_H : -if (is_ereg (dst_reg) ||is_ereg (src_reg)) -EMIT3 (0x66 ,add_2mod (0x40 , dst_reg, src_reg),0x89 ); -else -EMIT2 (0x66 ,0x89 ); - goto stx;case BPF_STX |BPF_MEM |BPF_W : -if (is_ereg (dst_reg) ||is_ereg (src_reg)) -EMIT2 (add_2mod (0x40 , dst_reg, src_reg),0x89 ); -else -EMIT1 (0x89 ); - goto stx;case BPF_STX |BPF_MEM |BPF_DW : -EMIT2 (add_2mod (0x48 , dst_reg, src_reg),0x89 ); -stx:if (is_imm8 (insn->off)) -EMIT2 (add_2reg (0x40 , dst_reg, src_reg), insn->off); -else -EMIT1_off32 (add_2reg (0x80 , dst_reg, src_reg), - insn->off); +emit_stx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn->off);break ;/* LDX: dst_reg = *(u8*)(src_reg + off) */ case BPF_LDX |BPF_MEM |BPF_B :case BPF_LDX |BPF_PROBE_MEM |BPF_B : -/* Emit 'movzx rax, byte ptr [rax + off]' */ -EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xB6 ); - goto ldx;case BPF_LDX |BPF_MEM |BPF_H :case BPF_LDX |BPF_PROBE_MEM |BPF_H : -/* Emit 'movzx rax, word ptr [rax + off]' */ -EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xB7 ); - goto ldx;case BPF_LDX |BPF_MEM |BPF_W :case BPF_LDX |BPF_PROBE_MEM |BPF_W : -/* Emit 'mov eax, dword ptr [rax+0x14]' */ -if (is_ereg (dst_reg) ||is_ereg (src_reg)) -EMIT2 (add_2mod (0x40 , src_reg, dst_reg),0x8B ); -else -EMIT1 (0x8B ); - goto ldx;case BPF_LDX |BPF_MEM |BPF_DW :case BPF_LDX |BPF_PROBE_MEM |BPF_DW : -/* Emit 'mov rax, qword ptr [rax+0x14]' */ -EMIT2 (add_2mod (0x48 , src_reg, dst_reg),0x8B ); -ldx:/* -* If insn->off == 0 we can save one extra byte, but -* special case of x86 R13 which always needs an offset -* is not worth the hassle -*/ -if (is_imm8 (insn->off)) -EMIT2 (add_2reg (0x40 , src_reg, dst_reg), insn->off); -else -EMIT1_off32 (add_2reg (0x80 , src_reg, dst_reg), - insn->off); +emit_ldx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn->off);if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ) {struct exception_table_entry *ex;u8 *_insn = image + proglen;
@@ -899,13 +949,8 @@ xadd: if (is_imm8(insn->off))/* call */ case BPF_JMP |BPF_CALL : func = (u8 *) __bpf_call_base + imm32; - jmp_offset = func - (image + addrs[i]); -if (!imm32 || !is_simm32 (jmp_offset)) { -pr_err ("unsupported BPF func %d addr %p image %p\n" , - imm32, func, image); +if (!imm32 ||emit_call (&prog, func, image + addrs[i -1 ]))return -EINVAL ; - } -EMIT1_off32 (0xE8 , jmp_offset);break ;case BPF_JMP |BPF_TAIL_CALL :
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 01/18] bpf: refactor x86 JIT into helpers
Quoted 10 lines · Alexei Starovoitov
Refactor x86 JITing of LDX, STX, CALL instructions into separate helper functions. No functional changes in LDX and STX helpers. There is a minor change in CALL helper. It will populate target address correctly on the first pass of JIT instead of second pass. That won't reduce total number of JIT passes though.
Signed-off-by Alexei Starovoitov <[email protected]>Acked-by Song Liu <[email protected]>---
Nice, logic is cleaner without extra gotos now, thanks!
[...]
Alexei Starovoitov
maintainerPATCH v3 2/18 bpf: Add bpf_arch_text_poke() helper
In reply to Alexei StarovoitovAdd bpf_arch_text_poke() helper that is used by BPF trampoline logic to patch nops/calls in kernel text into calls into BPF trampoline and to patch calls/nops inside BPF programs too.
--- arch/x86/net/bpf_jit_comp.c | 51 +++++++++++++++++++++++++++++++++++++ include/linux/bpf.h | 8 ++++++ kernel/bpf/core.c | 6 +++++ 3 files changed, 65 insertions(+)
arch/x86/net/bpf_jit_comp.c +51 -0
@@ -9,9 +9,11 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bpf.h> +#include <linux/memory.h> #include <asm/extable.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> +#include <asm/text-patching.h> static u8 *emit_code (u8 *ptr,u32 bytes,unsigned int len) {
@@ -487,6 +489,55 @@ static int emit_call(u8 **pprog, void *func, void *ip)return 0 ; } +int bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t, +void *old_addr,void *new_addr) +{ +u8 old_insn[X86_CALL_SIZE ] = {}; +u8 new_insn[X86_CALL_SIZE ] = {}; +u8 *prog; +int ret; + +if (!is_kernel_text ((long )ip)) +/* BPF trampoline in modules is not supported */ +return -EINVAL ; + +if (old_addr) { + prog = old_insn; + ret =emit_call (&prog, old_addr, (void *)ip); +if (ret) +return ret; + } +if (new_addr) { + prog = new_insn; + ret =emit_call (&prog, new_addr, (void *)ip); +if (ret) +return ret; + } + ret = -EBUSY ; +mutex_lock (&text_mutex); +switch (t) { +case BPF_MOD_NOP_TO_CALL : +if (memcmp (ip, ideal_nops[NOP_ATOMIC5 ],X86_CALL_SIZE )) + goto out; +text_poke (ip, new_insn,X86_CALL_SIZE ); +break ; +case BPF_MOD_CALL_TO_CALL : +if (memcmp (ip, old_insn,X86_CALL_SIZE )) + goto out; +text_poke (ip, new_insn,X86_CALL_SIZE ); +break ; +case BPF_MOD_CALL_TO_NOP : +if (memcmp (ip, old_insn,X86_CALL_SIZE )) + goto out; +text_poke (ip, ideal_nops[NOP_ATOMIC5 ],X86_CALL_SIZE ); +break ; + } + ret =0 ; +out: +mutex_unlock (&text_mutex); +return ret; +} +static bool ex_handler_bpf (const struct exception_table_entry *x,struct pt_regs *regs,int trapnr,unsigned long error_code,unsigned long fault_addr)
include/linux/bpf.h +8 -0
@@ -1157,4 +1157,12 @@ static inline u32 bpf_xdp_sock_convert_ctx_access(enum bpf_access_type type, }#endif /* CONFIG_INET */ +enum bpf_text_poke_type { +BPF_MOD_NOP_TO_CALL , +BPF_MOD_CALL_TO_CALL , +BPF_MOD_CALL_TO_NOP , +}; +int bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t, +void *addr1,void *addr2); +#endif /* _LINUX_BPF_H */
kernel/bpf/core.c +6 -0
@@ -2140,6 +2140,12 @@ int __weak skb_copy_bits(const struct sk_buff *skb, int offset, void *to,return -EFAULT ; } +int __weak bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t, +void *addr1,void *addr2) +{ +return -ENOTSUPP ; +} +DEFINE_STATIC_KEY_FALSE (bpf_stats_enabled_key);EXPORT_SYMBOL (bpf_stats_enabled_key);
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
Quoted 9 lines · Alexei Starovoitov
Add
bpf_arch_text_poke()helper that is used by BPF trampoline logic to patch nops/calls in kernel text into calls into BPF trampoline and to patch calls/nops inside BPF programs too.Signed-off-by Alexei Starovoitov <[email protected]>
Quoted 115 lines
---
arch/x86/net/bpf_jit_comp.c | 51 +++++++++++++++++++++++++++++++++++++ include/linux/bpf.h | 8 ++++++ kernel/bpf/core.c | 6 +++++ 3 files changed, 65 insertions(+)arch/x86/net/bpf_jit_comp.c +51
-0@@ -9,9 +9,11 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bpf.h> +#include <linux/memory.h> #include <asm/extable.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> +#include <asm/text-patching.h> static u8 *emit_code (u8 *ptr,u32 bytes,unsigned int len) {@@ -487,6 +489,55 @@ static int emit_call(u8 **pprog, void *func, void *ip)return 0 ; } +int bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t, +void *old_addr,void *new_addr) +{ +u8 old_insn[X86_CALL_SIZE ] = {}; +u8 new_insn[X86_CALL_SIZE ] = {}; +u8 *prog; +int ret; + +if (!is_kernel_text ((long )ip)) +/* BPF trampoline in modules is not supported */ +return -EINVAL ; + +if (old_addr) { + prog = old_insn; + ret =emit_call (&prog, old_addr, (void *)ip); +if (ret) +return ret; + } +if (new_addr) { + prog = new_insn; + ret =emit_call (&prog, new_addr, (void *)ip); +if (ret) +return ret; + } + ret = -EBUSY ; +mutex_lock (&text_mutex); +switch (t) { +case BPF_MOD_NOP_TO_CALL : +if (memcmp (ip, ideal_nops[NOP_ATOMIC5 ],X86_CALL_SIZE )) + goto out; +text_poke (ip, new_insn,X86_CALL_SIZE ); +break ; +case BPF_MOD_CALL_TO_CALL : +if (memcmp (ip, old_insn,X86_CALL_SIZE )) + goto out; +text_poke (ip, new_insn,X86_CALL_SIZE ); +break ; +case BPF_MOD_CALL_TO_NOP : +if (memcmp (ip, old_insn,X86_CALL_SIZE )) + goto out; +text_poke (ip, ideal_nops[NOP_ATOMIC5 ],X86_CALL_SIZE ); +break ; + } + ret =0 ; +out: +mutex_unlock (&text_mutex); +return ret; +} +static bool ex_handler_bpf (const struct exception_table_entry *x,struct pt_regs *regs,int trapnr,unsigned long error_code,unsigned long fault_addr)include/linux/bpf.h +8
-0@@ -1157,4 +1157,12 @@ static inline u32 bpf_xdp_sock_convert_ctx_access(enum bpf_access_type type, }#endif /* CONFIG_INET */ +enum bpf_text_poke_type { +BPF_MOD_NOP_TO_CALL , +BPF_MOD_CALL_TO_CALL , +BPF_MOD_CALL_TO_NOP , +}; +int bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t, +void *addr1,void *addr2); +#endif /* _LINUX_BPF_H */ kernel/bpf/core.c +6
-0@@ -2140,6 +2140,12 @@ int __weak skb_copy_bits(const struct sk_buff *skb, int offset, void *to,return -EFAULT ; } +int __weak bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t, +void *addr1,void *addr2) +{ +return -ENOTSUPP ; +} +DEFINE_STATIC_KEY_FALSE (bpf_stats_enabled_key);EXPORT_SYMBOL (bpf_stats_enabled_key);Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
In reply to Alexei StarovoitovQuoted 53 lines · Alexei Starovoitov
Add
bpf_arch_text_poke()helper that is used by BPF trampoline logic to patch nops/calls in kernel text into calls into BPF trampoline and to patch calls/nops inside BPF programs too.Signed-off-by Alexei Starovoitov <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 51 +++++++++++++++++++++++++++++++++++++ include/linux/bpf.h | 8 ++++++ kernel/bpf/core.c | 6 +++++ 3 files changed, 65 insertions(+)arch/x86/net/bpf_jit_comp.c +23
-0@@ -9,9 +9,11 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bpf.h> +#include <linux/memory.h> #include <asm/extable.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> +#include <asm/text-patching.h> static u8 *emit_code (u8 *ptr,u32 bytes,unsigned int len) {@@ -487,6 +489,55 @@ static int emit_call(u8 **pprog, void *func, void *ip)return 0 ; } +int bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t, +void *old_addr,void *new_addr) +{ +u8 old_insn[X86_CALL_SIZE ] = {}; +u8 new_insn[X86_CALL_SIZE ] = {}; +u8 *prog; +int ret; + +if (!is_kernel_text ((long )ip)) +/* BPF trampoline in modules is not supported */ +return -EINVAL ; + +if (old_addr) { + prog = old_insn; + ret =emit_call (&prog, old_addr, (void *)ip); +if (ret) +return ret; + } +if (new_addr) { + prog = new_insn; + ret =emit_call (&prog, new_addr, (void *)ip);
arch/x86/net/bpf_jit_comp.c static int emit_call(u8 **pprog, void *func, void *ip) · line 522
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+if (ret) +return ret; + } + ret = -EBUSY ; +mutex_lock (&text_mutex); +switch (t) { +case BPF_MOD_NOP_TO_CALL : +if (memcmp (ip, ideal_nops[NOP_ATOMIC5 ],X86_CALL_SIZE )) + goto out; +text_poke (ip, new_insn,X86_CALL_SIZE );
I'm probably missing something, but why isn't text_poke_bp() needed here?
Quoted 58 lines
arch/x86/net/bpf_jit_comp.c +18
-0+break ; +case BPF_MOD_CALL_TO_CALL : +if (memcmp (ip, old_insn,X86_CALL_SIZE )) + goto out; +text_poke (ip, new_insn,X86_CALL_SIZE ); +break ; +case BPF_MOD_CALL_TO_NOP : +if (memcmp (ip, old_insn,X86_CALL_SIZE )) + goto out; +text_poke (ip, ideal_nops[NOP_ATOMIC5 ],X86_CALL_SIZE ); +break ; + } + ret =0 ; +out: +mutex_unlock (&text_mutex); +return ret; +} +static bool ex_handler_bpf (const struct exception_table_entry *x,struct pt_regs *regs,int trapnr,unsigned long error_code,unsigned long fault_addr)include/linux/bpf.h +8
-0@@ -1157,4 +1157,12 @@ static inline u32 bpf_xdp_sock_convert_ctx_access(enum bpf_access_type type, }#endif /* CONFIG_INET */ +enum bpf_text_poke_type { +BPF_MOD_NOP_TO_CALL , +BPF_MOD_CALL_TO_CALL , +BPF_MOD_CALL_TO_NOP , +}; +int bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t, +void *addr1,void *addr2); +#endif /* _LINUX_BPF_H */ kernel/bpf/core.c +6
-1@@ -2140,6 +2140,12 @@ int __weak skb_copy_bits(const struct sk_buff *skb, int offset, void *to,return -EFAULT ; } +int __weak bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t, +void *addr1,void *addr2) +{ +return -ENOTSUPP ; +} +DEFINE_STATIC_KEY_FALSE (bpf_stats_enabled_key);EXPORT_SYMBOL (bpf_stats_enabled_key); --2.23.0
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
Quoted 58 lines · Björn Töpel
Quoted 57 lines · Alexei Starovoitov
Add
bpf_arch_text_poke()helper that is used by BPF trampoline logic to patch nops/calls in kernel text into calls into BPF trampoline and to patch calls/nops inside BPF programs too.Signed-off-by Alexei Starovoitov <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 51 +++++++++++++++++++++++++++++++++++++ include/linux/bpf.h | 8 ++++++ kernel/bpf/core.c | 6 +++++ 3 files changed, 65 insertions(+)arch/x86/net/bpf_jit_comp.c +27
-0@@ -9,9 +9,11 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bpf.h> +#include <linux/memory.h> #include <asm/extable.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> +#include <asm/text-patching.h> static u8 *emit_code (u8 *ptr,u32 bytes,unsigned int len) {@@ -487,6 +489,55 @@ static int emit_call(u8 **pprog, void *func, void *ip)return 0 ; } +int bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t, +void *old_addr,void *new_addr) +{ +u8 old_insn[X86_CALL_SIZE ] = {}; +u8 new_insn[X86_CALL_SIZE ] = {}; +u8 *prog; +int ret; + +if (!is_kernel_text ((long )ip)) +/* BPF trampoline in modules is not supported */ +return -EINVAL ; + +if (old_addr) { + prog = old_insn; + ret =emit_call (&prog, old_addr, (void *)ip); +if (ret) +return ret; + } +if (new_addr) { + prog = new_insn; + ret =emit_call (&prog, new_addr, (void *)ip); +if (ret) +return ret; + } + ret = -EBUSY ;
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static int emit_call(u8 **pprog, void *func, void *ip) · line 522
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +6
-0+mutex_lock (&text_mutex); +switch (t) { +case BPF_MOD_NOP_TO_CALL : +if (memcmp (ip, ideal_nops[NOP_ATOMIC5 ],X86_CALL_SIZE )) + goto out; +text_poke (ip, new_insn,X86_CALL_SIZE );I'm probably missing something, but why isn't
text_poke_bp()needed here?
I should have documented the intent better.text_poke_bp() is being changed by Peter to emulate instructions properly in his ftrace->text_poke conversion set.
So I cannot use it just yet.
To you point that text_poke() is technically incorrect here. Yep. Well aware. This is temporarily. As I said in the cover letter this needs to change to register_ftrace_direct() for kernel text poking to play nice with ftrace. Thinking about it more... I guess I can use text_poke_bp(). Just need to setup handler properly. I may need to do it for bpf prog poking anyway. Wanted to avoid extra churn that is going to be removed during merge window when trees converge.
Since we're on this subject.
Peter,
why you don't do 8 byte atomic rewrite when start addr of insn is properly aligned? This trap dance would be unnecessary. That will make everything so much simpler.
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
In reply to Alexei StarovoitovQuoted 3 lines · Alexei Starovoitov
Add
bpf_arch_text_poke()helper that is used by BPF trampoline logic to patch nops/calls in kernel text into calls into BPF trampoline and to patch calls/nops inside BPF programs too.
This thing assumes the text is unused, right? That isn't spelled out anywhere. The implementation is very much unsafe vs concurrent execution of the text.
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Alexei Starovoitov
Add
bpf_arch_text_poke()helper that is used by BPF trampoline logic to patch nops/calls in kernel text into calls into BPF trampoline and to patch calls/nops inside BPF programs too.This thing assumes the text is unused, right? That isn't spelled out anywhere. The implementation is very much unsafe vs concurrent execution of the text.
Also, what NOP/CALL instructions will you be hijacking? If you're planning on using the fentry nops, then what ensures this and ftrace don't trample on one another? Similar for kprobes.
In general, what ensures every instruction only has a single modifier?
I'm very uncomfortable letting random bpf proglets poke around in the kernel text.
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Alexei Starovoitov
Add
bpf_arch_text_poke()helper that is used by BPF trampoline logic to patch nops/calls in kernel text into calls into BPF trampoline and to patch calls/nops inside BPF programs too.This thing assumes the text is unused, right? That isn't spelled out anywhere. The implementation is very much unsafe vs concurrent execution of the text.
Quoted 5 lines
Also, what NOP/CALL instructions will you be hijacking? If you're planning on using the fentry nops, then what ensures this and ftrace don't trample on one another? Similar for kprobes.
In general, what ensures every instruction only has a single modifier?
Looks like you didn't bother reading cover letter and missed a month of discussions between my and Steven regarding exactly this topic though you were directly cc-ed in all threads :(
tldr for kernel fentry nops it will be converted to use register_ftrace_direct() whenever it's available.
For all other nops, calls, jumps that are inside BPF programs BPF infra will continue modifying them through this helper.
Daniel's upcoming bpf_tail_call() optimization will use text_poke as well.
Quoted 1 line
I'm very uncomfortable letting random bpf proglets poke around in the
kernel text.
1. There is no such thing as 'proglet'. Please don't invent meaningless names.
2. BPF programs have no ability to modify kernel text.
3. BPF infra taking all necessary measures to make sure that poking kernel's and BPF generated text is safe.
If you see specific issue please say so. We'll be happy to address all issues. Being 'uncomfortable' is not constructive.
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
Quoted 30 lines · Alexei Starovoitov
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Alexei Starovoitov
Add
bpf_arch_text_poke()helper that is used by BPF trampoline logic to patch nops/calls in kernel text into calls into BPF trampoline and to patch calls/nops inside BPF programs too.This thing assumes the text is unused, right? That isn't spelled out anywhere. The implementation is very much unsafe vs concurrent execution of the text.
Quoted 5 lines
Also, what NOP/CALL instructions will you be hijacking? If you're planning on using the fentry nops, then what ensures this and ftrace don't trample on one another? Similar for kprobes.
In general, what ensures every instruction only has a single modifier?
Looks like you didn't bother reading cover letter and missed a month of discussions between my and Steven regarding exactly this topic though you were directly cc-ed in all threads :(
tldr for kernel fentry nops it will be converted to useregister_ftrace_direct()whenever it's available.
For all other nops, calls, jumps that are inside BPF programs BPF infra will continue modifying them through this helper.
Daniel's upcomingbpf_tail_call()optimization will usetext_pokeas well.Quoted 1 line
I'm very uncomfortable letting random bpf proglets poke around in the
kernel text.
Quoted 8 lines
1. There is no such thing as 'proglet'. Please don't invent meaningless names.
2. BPF programs have no ability to modify kernel text.
3. BPF infra taking all necessary measures to make sure that poking kernel's and BPF generated text is safe.
If you see specific issue please say so. We'll be happy to address all issues. Being 'uncomfortable' is not constructive.
I was thinking more about this.
Peter,
do you mind we apply your first patch:
https://lore.kernel.org/lkml/[email protected]/
to both tip and bpf-next trees?
Then I can use text_poke_bp() as-is without any additional ugliness on my side that would need to be removed in few weeks.
Do you have it in tip already?
I can cherry-pick from there to make sure it's exactly the same commit log then there will be no merge issues during merge window.
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
Quoted 13 lines · Alexei Starovoitov
Quoted 12 lines · Alexei Starovoitov
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Alexei Starovoitov
Add
bpf_arch_text_poke()helper that is used by BPF trampoline logic to patch nops/calls in kernel text into calls into BPF trampoline and to patch calls/nops inside BPF programs too.This thing assumes the text is unused, right? That isn't spelled out anywhere. The implementation is very much unsafe vs concurrent execution of the text.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Also, what NOP/CALL instructions will you be hijacking? If you're planning on using the fentry nops, then what ensures this and ftrace don't trample on one another? Similar for kprobes.
In general, what ensures every instruction only has a single modifier?
Looks like you didn't bother reading cover letter and missed a month
I did indeed not. A Changelog should be self sufficient and this one is sorely lacking. The cover leter is not preserved and should therefore not contain anything of value that is not also covered in the Changelogs.
Quoted 2 lines
Quoted 2 lines
of discussions between my and Steven regarding exactly this topic though you were directly cc-ed in all threads :(
I read some of it; it is a sad fact that I cannot read all email in my inbox, esp. not if, like in the last week or so, I'm busy hunting a regression.
And what I did remember of the emails I saw left me with the questions that were not answered by the changelog.
Quoted 2 lines
Quoted 2 lines
tldr for kernel fentry nops it will be converted to use
register_ftrace_direct()whenever it's available.
So why the rush and not wait for that work to complete? It appears to me that without due coordination between bpf and ftrace badness could happen.
Quoted 3 lines
Quoted 3 lines
For all other nops, calls, jumps that are inside BPF programs BPF infra will continue modifying them through this helper.
Daniel's upcomingbpf_tail_call()optimization will usetext_pokeas well.
This is probably off topic, but isn't tail-call optimization something done at JIT time and therefore not in need ot text_poke()?
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
I'm very uncomfortable letting random bpf proglets poke around in the
kernel text.
1. There is no such thing as 'proglet'. Please don't invent meaningless names.
Again offtopic, but I'm not inventing stuff here:
/prog'let/ [UK] A short extempore program written to meet an immediate, transient need.
which, methinks, succinctly captures the spirit of BPF.
Quoted 1 line
Quoted 1 line
2. BPF programs have no ability to modify kernel text.
OK, that is good.
Quoted 2 lines
Quoted 2 lines
3. BPF infra taking all necessary measures to make sure that poking kernel's and BPF generated text is safe.
From the other subthread and your response below, it seems you are aware that you in fact need text_poke_bp(). Again, it would've been excellent Changelog/comment material to call out that the patch as presented is in fact broken.
Quoted 5 lines
I was thinking more about this.
Peter,
do you mind we apply your first patch:
https://lore.kernel.org/lkml/[email protected]/
to both tip and bpf-next trees?
That would indeed be a much better solution. I'll repost much of that on Monday, and then we'll work on getting at the very least that one patch in a tip/branch we can share.
Quoted 2 lines
Then I can use
text_poke_bp()as-is without any additional ugliness on my side that would need to be removed in few weeks.
This I do _NOT_ understand. Why are you willing to merge a known broken patch? What is the rush, why can't you wait for all the prerequisites to land?
David Miller
maintainerRe: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
From: Peter Zijlstra <[email protected]>
Date: Fri, 8 Nov 2019 22:36:24 +0100
Quoted 1 line
The cover leter is not preserved and should therefore
...
The cover letter is +ALWAYS+ preserved, we put them in the merge commit.
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
Quoted 8 lines · David Miller
From: Peter Zijlstra <[email protected]>
Date: Fri, 8 Nov 2019 22:36:24 +0100Quoted 1 line
The cover leter is not preserved and should therefore
...
The cover letter is +ALWAYS+ preserved, we put them in the merge commit.
Good to know; is this a netdev special? I've not seen this before.
Is there a convenient git command to find the merge commit for a given regular commit? Say I've used git-blame to find a troublesome commit, then how do I find the merge commit for it?
Also, I still think this does not excuse weak individual Changelogs.
Daniel Borkmann
maintainerRe: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
Quoted 11 lines · Peter Zijlstra
Quoted 8 lines · David Miller
From: Peter Zijlstra <[email protected]>
Date: Fri, 8 Nov 2019 22:36:24 +0100Quoted 1 line
The cover leter is not preserved and should therefore
...
The cover letter is +ALWAYS+ preserved, we put them in the merge commit.
Good to know; is this a netdev special? I've not seen this before.
I think it might be netdev special, and given developers are often used to this practice on netdev, we've adopted the same for both bpf trees as well if the cover letter contains a useful high level summary of the whole set.
We've recently changed our workflow a bit after last maintainers summit and reuse Thomas' mb2q [0] in our small collection of scripts [1], so aside from other useful features, for every commit under bpf/bpf-next there is now also a 'Link:' tag pointing to https://lore.kernel.org/bpf/ archive, thus cover letter or discussions could alternatively be found this way.
One downside of merge commit as cover letter is that they are usually lost (*) once commits get cherry-picked into stable trees or other downstream, backport heavy kernels, so with a 'Link:' tag it's a convenient way to quickly get more context or discussions for those cases.
Quoted 3 lines
Is there a convenient git command to find the merge commit for a given regular commit? Say I've used git-blame to find a troublesome commit, then how do I find the merge commit for it?
Once you have the sha, you could for example retrieve the corresponding merge commit from the upstream tree (as top commit) via:
$ git log <sha>..master --ancestry-path --merges --reverse
Quoted 1 line
Also, I still think this does not excuse weak individual Changelogs.
Ideally commit messages should be as self-contained as possible to have all the necessary context w/o having to look up other resources also given issue (*) above.
Thanks,
Daniel
[0] https://git.kernel.org/pub/scm/linux/kernel/git/tglx/quilttools.git/
[1] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/pw.git/
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
In reply to Peter ZijlstraQuoted 11 lines · On Mon, 11 Nov 2019 09:14:03 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 8 lines · David Miller
From: Peter Zijlstra <[email protected]>
Date: Fri, 8 Nov 2019 22:36:24 +0100Quoted 1 line
The cover leter is not preserved and should therefore
...
The cover letter is +ALWAYS+ preserved, we put them in the merge commit.
Good to know; is this a netdev special? I've not seen this before.
I read a *lot* of merge commit changelogs; it's not just netdev that does this. I try to do it with significant documentation sets as well. I agree that changelogs should contain all relevant information, but there is value in an overview as well. But then, I make my living in the overview business...:)
jon
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
In reply to Peter ZijlstraQuoted 44 lines · Peter Zijlstra
Quoted 13 lines · Alexei Starovoitov
Quoted 12 lines · Alexei Starovoitov
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Alexei Starovoitov
Add
bpf_arch_text_poke()helper that is used by BPF trampoline logic to patch nops/calls in kernel text into calls into BPF trampoline and to patch calls/nops inside BPF programs too.This thing assumes the text is unused, right? That isn't spelled out anywhere. The implementation is very much unsafe vs concurrent execution of the text.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Also, what NOP/CALL instructions will you be hijacking? If you're planning on using the fentry nops, then what ensures this and ftrace don't trample on one another? Similar for kprobes.
In general, what ensures every instruction only has a single modifier?
Looks like you didn't bother reading cover letter and missed a month
I did indeed not. A Changelog should be self sufficient and this one is sorely lacking. The cover leter is not preserved and should therefore not contain anything of value that is not also covered in the Changelogs.
Quoted 2 lines
Quoted 2 lines
of discussions between my and Steven regarding exactly this topic though you were directly cc-ed in all threads :(
I read some of it; it is a sad fact that I cannot read all email in my inbox, esp. not if, like in the last week or so, I'm busy hunting a regression.
And what I did remember of the emails I saw left me with the questions that were not answered by the changelog.
Quoted 2 lines
Quoted 2 lines
tldr for kernel fentry nops it will be converted to use
register_ftrace_direct()whenever it's available.So why the rush and not wait for that work to complete? It appears to me that without due coordination between bpf and ftrace badness could happen.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
For all other nops, calls, jumps that are inside BPF programs BPF infra will continue modifying them through this helper.
Daniel's upcomingbpf_tail_call()optimization will usetext_pokeas well.This is probably off topic, but isn't tail-call optimization something done at JIT time and therefore not in need ot
text_poke()?
Not quite. bpf_tail_call() are done via prog_array which is indirect jmp and it suffers from retpoline. The verifier can see that in a lot of cases the prog_array is used with constant index into array instead of a variable. In such case indirect jmps can be optimized with direct jmps. That is essentially what Daniel's patches are doing that are building on top of bpf_arch_text_poke() and trampoline that I'm introducing in this set.
Another set is being prepared by Bjorn that also builds on top of bpf_arch_text_poke() and trampoline. It's serving the purpose of getting rid of indirect call when driver calls into BPF program for the first time. We've looked at your static_call and concluded that it doesn't quite cut for this use case.
The third framework is worked on by Martin. Who is using BPF trampoline for BPF-based TCP extensions. This bit is not related to indirect call/jmp optimization, but needs trampoline.
Quoted 9 lines
Quoted 5 lines
I was thinking more about this.
Peter,
do you mind we apply your first patch:
https://lore.kernel.org/lkml/[email protected]/
to both tip and bpf-next trees?That would indeed be a much better solution. I'll repost much of that on Monday, and then we'll work on getting at the very least that one patch in a tip/branch we can share.
Awesome! I can certainly wait till next week. I just don't want to miss the merge window for the work that it is ready. More below.
Quoted 6 lines
Quoted 2 lines
Then I can use
text_poke_bp()as-is without any additional ugliness on my side that would need to be removed in few weeks.This I do _NOT_ understand. Why are you willing to merge a known broken patch? What is the rush, why can't you wait for all the prerequisites to land?
People have deadlines and here I'm not talking about fb deadlines. If it was only up to me I could have waited until yours and Steven's patches land in Linus's tree. Then Dave would pick them up after the merge window into net-next and bpf things would be ready for the next release. Which is in 1.5 + 2 + 8 weeks (assuming 1.5 weeks until merge window, 2 weeks merge window, and 8 weeks next release cycle).
But most of bpf things are ready. I have one more follow up to do for another feature. The first 4-5 patches of my set will enable Bjorn, Daniel, and Martin's work. So I'm mainly looking for a way to converge three trees during the merge window with no conflicts.
Just saw that Steven posted his set. That is great. If you land your first part of text_poke_pb() next week into tip it will enable us to cherry-pick the first few patches from tip and continue with bpf trampoline in net-next. Then during the merge window tip, Steven's and net-next land into Linus's tree. Then I'll send small follow up to switch to Steven's register_ftrace_direct() in places that can use it and the other bits of bpf will keep using yours text_poke_bp() because it's for the code inside generated bpf progs, various generated trampolines and such. The conversion of some of bpf bits to register_ftrace_direct() can be delayed by a release if really necessary. Since text_poke_bp() approach will work fine, just not as nice if there is a full integration via ftrace.
imo it's the best path for 3 trees to converge without delaying things for bpf folks by a full release. At the end the deadlines are met and a bunch of people are unblocked and happy. I hope that explains the rush.
Re: [PATCH v3 bpf-next 02/18] bpf: Add bpf_arch_text_poke() helper
Quoted 5 lines · Alexei Starovoitov
Quoted 3 lines · Peter Zijlstra
This I do _NOT_ understand. Why are you willing to merge a known broken patch? What is the rush, why can't you wait for all the prerequisites to land?
Quoted 10 lines
People have deadlines and here I'm not talking about fb deadlines. If it was only up to me I could have waited until yours and Steven's patches land in Linus's tree. Then Dave would pick them up after the merge window into net-next and bpf things would be ready for the next release. Which is in 1.5 + 2 + 8 weeks (assuming 1.5 weeks until merge window, 2 weeks merge window, and 8 weeks next release cycle).
But most of bpf things are ready. I have one more follow up to do for another feature. The first 4-5 patches of my set will enable Bjorn, Daniel, and Martin's work. So I'm mainly looking for a way to converge three trees during the merge window with no conflicts.
No. Nobodys deadlines are justifying anything.
You recently gave me a lecture how to do proper kernel development just because I did the right thing, i.e. preventing a bad interaction by making a Kconfig dependency which does not affect you in any way.
Now for your own interests you try to justify something which is fundamentaly worse: Merging known to be broken code.
BPF is not special and has to wait for the next merge window if the prerequisites are not ready in time as any other patch set does.
Thanks,
tglx
Alexei Starovoitov
maintainerPATCH v3 3/18 bpf: Introduce BPF trampoline
In reply to Alexei StarovoitovIntroduce BPF trampoline concept to allow kernel code to call into BPF programs with practically zero overhead. The trampoline generation logic is architecture dependent. It's converting native calling convention into BPF calling convention. BPF ISA is 64-bit (even on 32-bit architectures). The registers R1 to R5 are used to pass arguments into BPF functions. The main BPF program accepts only single argument "ctx" in R1. Whereas CPU native calling convention is different. x86-64 is passing first 6 arguments in registers and the rest on the stack. x86-32 is passing first 3 arguments in registers. sparc64 is passing first 6 in registers. And so on.
The trampolines between BPF and kernel already exist. BPF_CALL_x macros in include/linux/filter.h statically compile trampolines from BPF into kernel helpers. They convert up to five u64 arguments into kernel C pointers and integers. On 64-bit architectures this BPF_to_kernel trampolines are nops. On 32-bit architecture they're meaningful.
The opposite job kernel_to_BPF trampolines is done by CAST_TO_U64 macros and __bpf_trace_##call() shim functions in include/trace/bpf_probe.h. They convert kernel function arguments into array of u64s that BPF program consumes via R1=ctx pointer.
This patch set is doing the same job as __bpf_trace_##call() static trampolines, but dynamically for any kernel function. There are ~22k global kernel functions that are attachable via nop at function entry. The function arguments and types are described in BTF. The job of btf_distill_func_proto() function is to extract useful information from BTF into "function model" that architecture dependent trampoline generators will use to generate assembly code to cast kernel function arguments into array of u64s. For example the kernel function eth_type_trans has two pointers. They will be casted to u64 and stored into stack of generated trampoline. The pointer to that stack space will be passed into BPF program in R1. On x86-64 such generated trampoline will consume 16 bytes of stack and two stores of %rdi and %rsi into stack. The verifier will make sure that only two u64 are accessed read-only by BPF program. The verifier will also recognize the precise type of the pointers being accessed and will not allow typecasting of the pointer to a different type within BPF program.
The tracing use case in the datacenter demonstrated that certain key kernel functions have (like tcp_retransmit_skb) have 2 or more kprobes that are always active. Other functions have both kprobe and kretprobe. So it is essential to keep both kernel code and BPF programs executing at maximum speed. Hence generated BPF trampoline is re-generated every time new program is attached or detached to maintain maximum performance.
To avoid the high cost of retpoline the attached BPF programs are called directly. __bpf_prog_enter/exit() are used to support per-program execution stats. In the future this logic will be optimized further by adding support for bpf_stats_enabled_key inside generated assembly code. Introduction of preemptible and sleepable BPF programs will completely remove the need to call to __bpf_prog_enter/exit().
Detach of a BPF program from the trampoline should not fail. To avoid memory allocation in detach path the half of the page is used as a reserve and flipped after each attach/detach. 2k bytes is enough to call 40+ BPF programs directly which is enough for BPF tracing use cases. This limit can be increased in the future.
BPF_TRACE_FENTRY programs have access to raw kernel function arguments while BPF_TRACE_FEXIT programs have access to kernel return value as well. Often kprobe BPF program remembers function arguments in a map while kretprobe fetches arguments from a map and analyzes them together with return value. BPF_TRACE_FEXIT accelerates this typical use case.
Recursion prevention for kprobe BPF programs is done via per-cpu bpf_prog_active counter. In practice that turned out to be a mistake. It caused programs to randomly skip execution. The tracing tools missed results they were looking for. Hence BPF trampoline doesn't provide builtin recursion prevention. It's a job of BPF program itself and will be addressed in the follow up patches.
BPF trampoline is intended to be used beyond tracing and fentry/fexit use cases in the future. For example to remove retpoline cost from XDP programs.
--- arch/x86/net/bpf_jit_comp.c | 211 +++++++++++++++++++++++++++++- include/linux/bpf.h | 98 ++++++++++++++ include/uapi/linux/bpf.h | 2 + kernel/bpf/Makefile | 1 + kernel/bpf/btf.c | 77 ++++++++++- kernel/bpf/core.c | 1 + kernel/bpf/syscall.c | 53 +++++++- kernel/bpf/trampoline.c | 250 ++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 39 ++++++ 9 files changed, 722 insertions(+), 10 deletions(-) create mode 100644 kernel/bpf/trampoline.c
arch/x86/net/bpf_jit_comp.c +209 -2
@@ -98,6 +98,7 @@ static int bpf_size_to_x86_bytes(int bpf_size)/* Pick a register outside of BPF range for JIT internal work */ #define AUX_REG (MAX_BPF_JIT_REG + 1) +#define X86_REG_R9 (MAX_BPF_JIT_REG + 2)/* * The following table maps BPF registers to x86-64 registers.
@@ -106,8 +107,8 @@ static int bpf_size_to_x86_bytes(int bpf_size)* register in load/store instructions, it always needs an * extra byte of encoding and is callee saved. * -* Also x86-64 register R9 is unused. x86-64 register R10 is -* used for blinding (if enabled). +* x86-64 register R9 is not used by BPF programs, but can be used by BPF +* trampoline. x86-64 register R10 is used for blinding (if enabled). */ static const int reg2hex[] = { [BPF_REG_0 ] =0 ,/* RAX */
@@ -123,6 +124,7 @@ static const int reg2hex[] = { [BPF_REG_FP] = 5, /* RBP readonly */ [BPF_REG_AX] = 2, /* R10 temp register */ [AUX_REG] = 3, /* R11 temp register */ + [X86_REG_R9] = 1, /* R9 register, 6th function argument */ };static const int reg2pt_regs[] = {
@@ -150,6 +152,7 @@ static bool is_ereg(u32 reg)BIT (BPF_REG_7 ) |BIT (BPF_REG_8 ) |BIT (BPF_REG_9 ) | +BIT (X86_REG_R9 ) | BIT(BPF_REG_AX)); }
@@ -1234,6 +1237,210 @@ xadd: if (is_imm8(insn->off))return proglen; } +static void save_regs (struct btf_func_model *m,u8 **prog,int nr_args, +int stack_size) +{ +int i; +/* Store function arguments to stack. +* For a function that accepts two pointers the sequence will be: +* mov QWORD PTR [rbp-0x10],rdi +* mov QWORD PTR [rbp-0x8],rsi +*/ +for (i =0 ; i <min (nr_args,6 ); i++) +emit_stx (prog,bytes_to_bpf_size (m->arg_size[i]), +BPF_REG_FP , + i ==5 ?X86_REG_R9 :BPF_REG_1 + i, + -(stack_size - i *8 )); +} + +static void restore_regs (struct btf_func_model *m,u8 **prog,int nr_args, +int stack_size) +{ +int i; + +/* Restore function arguments from stack. +* For a function that accepts two pointers the sequence will be: +* EMIT4(0x48, 0x8B, 0x7D, 0xF0); mov rdi,QWORD PTR [rbp-0x10] +* EMIT4(0x48, 0x8B, 0x75, 0xF8); mov rsi,QWORD PTR [rbp-0x8] +*/ +for (i =0 ; i <min (nr_args,6 ); i++) +emit_ldx (prog,bytes_to_bpf_size (m->arg_size[i]), + i ==5 ?X86_REG_R9 :BPF_REG_1 + i, +BPF_REG_FP , + -(stack_size - i *8 )); +} + +static int invoke_bpf (struct btf_func_model *m,u8 **pprog, +struct bpf_prog **progs,int prog_cnt,int stack_size) +{ +u8 *prog = *pprog; +int cnt =0 , i; + +for (i =0 ; i < prog_cnt; i++) { +if (emit_call (&prog, __bpf_prog_enter, prog)) +return -EINVAL ; +/* remember prog start time returned by __bpf_prog_enter */ +emit_mov_reg (&prog, true,BPF_REG_6 ,BPF_REG_0 ); + +/* arg1: lea rdi, [rbp - stack_size] */ +EMIT4 (0x48 ,0x8D ,0x7D , -stack_size); +/* arg2: progs[i]->insnsi for interpreter */ +if (!progs[i]->jited) +emit_mov_imm64 (&prog,BPF_REG_2 , + (long ) progs[i]->insnsi >>32 , + (u32 ) (long ) progs[i]->insnsi); +/* call JITed bpf program or interpreter */ +if (emit_call (&prog, progs[i]->bpf_func, prog)) +return -EINVAL ; + +/* arg1: mov rdi, progs[i] */ +emit_mov_imm64 (&prog,BPF_REG_1 , (long ) progs[i] >>32 , + (u32 ) (long ) progs[i]); +/* arg2: mov rsi, rbx <- start time in nsec */ +emit_mov_reg (&prog, true,BPF_REG_2 ,BPF_REG_6 ); +if (emit_call (&prog, __bpf_prog_exit, prog)) +return -EINVAL ; + } + *pprog = prog; +return 0 ; +} + +/* Example: +* __be16 eth_type_trans(struct sk_buff *skb, struct net_device *dev); +* its 'struct btf_func_model' will be nr_args=2 +* The assembly code when eth_type_trans is executing after trampoline: +* +* push rbp +* mov rbp, rsp +* sub rsp, 16 // space for skb and dev +* push rbx // temp regs to pass start time +* mov qword ptr [rbp - 16], rdi // save skb pointer to stack +* mov qword ptr [rbp - 8], rsi // save dev pointer to stack +* call __bpf_prog_enter // rcu_read_lock and preempt_disable +* mov rbx, rax // remember start time in bpf stats are enabled +* lea rdi, [rbp - 16] // R1==ctx of bpf prog +* call addr_of_jited_FENTRY_prog +* movabsq rdi, 64bit_addr_of_struct_bpf_prog // unused if bpf stats are off +* mov rsi, rbx // prog start time +* call __bpf_prog_exit // rcu_read_unlock, preempt_enable and stats math +* mov rdi, qword ptr [rbp - 16] // restore skb pointer from stack +* mov rsi, qword ptr [rbp - 8] // restore dev pointer from stack +* pop rbx +* leave +* ret +* +* eth_type_trans has 5 byte nop at the beginning. These 5 bytes will be +* replaced with 'call generated_bpf_trampoline'. When it returns +* eth_type_trans will continue executing with original skb and dev pointers. +* +* The assembly code when eth_type_trans is called from trampoline: +* +* push rbp +* mov rbp, rsp +* sub rsp, 24 // space for skb, dev, return value +* push rbx // temp regs to pass start time +* mov qword ptr [rbp - 24], rdi // save skb pointer to stack +* mov qword ptr [rbp - 16], rsi // save dev pointer to stack +* call __bpf_prog_enter // rcu_read_lock and preempt_disable +* mov rbx, rax // remember start time if bpf stats are enabled +* lea rdi, [rbp - 24] // R1==ctx of bpf prog +* call addr_of_jited_FENTRY_prog // bpf prog can access skb and dev +* movabsq rdi, 64bit_addr_of_struct_bpf_prog // unused if bpf stats are off +* mov rsi, rbx // prog start time +* call __bpf_prog_exit // rcu_read_unlock, preempt_enable and stats math +* mov rdi, qword ptr [rbp - 24] // restore skb pointer from stack +* mov rsi, qword ptr [rbp - 16] // restore dev pointer from stack +* call eth_type_trans+5 // execute body of eth_type_trans +* mov qword ptr [rbp - 8], rax // save return value +* call __bpf_prog_enter // rcu_read_lock and preempt_disable +* mov rbx, rax // remember start time in bpf stats are enabled +* lea rdi, [rbp - 24] // R1==ctx of bpf prog +* call addr_of_jited_FEXIT_prog // bpf prog can access skb, dev, return value +* movabsq rdi, 64bit_addr_of_struct_bpf_prog // unused if bpf stats are off +* mov rsi, rbx // prog start time +* call __bpf_prog_exit // rcu_read_unlock, preempt_enable and stats math +* mov rax, qword ptr [rbp - 8] // restore eth_type_trans's return value +* pop rbx +* leave +* add rsp, 8 // skip eth_type_trans's frame +* ret // return to its caller +*/ +int arch_prepare_bpf_trampoline (void *image,struct btf_func_model *m,u32 flags, +struct bpf_prog **fentry_progs,int fentry_cnt, +struct bpf_prog **fexit_progs,int fexit_cnt, +void *orig_call) +{ +int cnt =0 , nr_args = m->nr_args; +int stack_size = nr_args *8 ; +u8 *prog; + +/* x86-64 supports up to 6 arguments. 7+ can be added in the future */ +if (nr_args >6 ) +return -ENOTSUPP ; + +if ((flags &BPF_TRAMP_F_RESTORE_REGS ) && + (flags &BPF_TRAMP_F_SKIP_FRAME )) +return -EINVAL ; + +if (flags &BPF_TRAMP_F_CALL_ORIG ) + stack_size +=8 ;/* room for return value of orig_call */ + +if (flags &BPF_TRAMP_F_SKIP_FRAME ) +/* skip patched call instruction and point orig_call to actual +* body of the kernel function. +*/ + orig_call +=X86_CALL_SIZE ; + + prog = image; + +EMIT1 (0x55 );/* push rbp */ +EMIT3 (0x48 ,0x89 ,0xE5 );/* mov rbp, rsp */ +EMIT4 (0x48 ,0x83 ,0xEC , stack_size);/* sub rsp, stack_size */ +EMIT1 (0x53 );/* push rbx */ + +save_regs (m, &prog, nr_args, stack_size); + +if (fentry_cnt) +if (invoke_bpf (m, &prog, fentry_progs, fentry_cnt, stack_size)) +return -EINVAL ; + +if (flags &BPF_TRAMP_F_CALL_ORIG ) { +if (fentry_cnt) +restore_regs (m, &prog, nr_args, stack_size); + +/* call original function */ +if (emit_call (&prog, orig_call, prog)) +return -EINVAL ; +/* remember return value in a stack for bpf prog to access */ +emit_stx (&prog,BPF_DW ,BPF_REG_FP ,BPF_REG_0 ,-8 ); + } + +if (fexit_cnt) +if (invoke_bpf (m, &prog, fexit_progs, fexit_cnt, stack_size)) +return -EINVAL ; + +if (flags &BPF_TRAMP_F_RESTORE_REGS ) +restore_regs (m, &prog, nr_args, stack_size); + +if (flags &BPF_TRAMP_F_CALL_ORIG ) +/* restore original return value back into RAX */ +emit_ldx (&prog,BPF_DW ,BPF_REG_0 ,BPF_REG_FP ,-8 ); + +EMIT1 (0x5B );/* pop rbx */ +EMIT1 (0xC9 );/* leave */ +if (flags &BPF_TRAMP_F_SKIP_FRAME ) +/* skip our return address and return to parent */ +EMIT4 (0x48 ,0x83 ,0xC4 ,8 );/* add rsp, 8 */ +EMIT1 (0xC3 );/* ret */ +/* One half of the page has active running trampoline. +* Another half is an area for next trampoline. +* Make sure the trampoline generation logic doesn't overflow. +*/ +if (WARN_ON_ONCE (prog - (u8 *)image >PAGE_SIZE /2 -BPF_INSN_SAFETY )) +return -EFAULT ; +return 0 ; +} +struct x64_jit_data {struct bpf_binary_header *header;int *addrs;
include/linux/bpf.h +98 -0
122 lines not on this page · raw message
include/uapi/linux/bpf.h +2 -0
@@ -201,6 +201,8 @@ enum bpf_attach_type {BPF_CGROUP_GETSOCKOPT ,BPF_CGROUP_SETSOCKOPT ,BPF_TRACE_RAW_TP , +BPF_TRACE_FENTRY , +BPF_TRACE_FEXIT , __MAX_BPF_ATTACH_TYPE };
kernel/bpf/Makefile +1 -0
@@ -6,6 +6,7 @@ obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o +obj-$(CONFIG_BPF_JIT) += trampoline.o obj-$(CONFIG_BPF_SYSCALL) += btf.o ifeq ($(CONFIG_NET),y) obj-$(CONFIG_BPF_SYSCALL) += devmap.o
kernel/bpf/btf.c +73 -4
92 lines not on this page · raw message
kernel/bpf/core.c +1 -0
@@ -2011,6 +2011,7 @@ static void bpf_prog_free_deferred(struct work_struct *work)if (aux->prog->has_callchain_buf)put_callchain_buffers ();#endif +bpf_trampoline_put (aux->trampoline);for (i =0 ; i < aux->func_cnt; i++)bpf_jit_free (aux->func[i]);if (aux->func_cnt) {
kernel/bpf/syscall.c +49 -4
69 lines not on this page · raw message
kernel/bpf/trampoline.c new file +250 -0
250 lines not on this page · raw message
kernel/bpf/verifier.c +39 -0
@@ -9382,8 +9382,11 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)struct bpf_prog *prog = env->prog;u32 btf_id = prog->aux->attach_btf_id;const char prefix[] ="btf_trace_" ; +struct bpf_trampoline *tr;const struct btf_type *t;const char *tname; +long addr; +int ret;if (prog->type !=BPF_PROG_TYPE_TRACING )return 0 ;
@@ -9432,6 +9435,42 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) prog->aux->attach_func_proto = t; prog->aux->attach_btf_trace = true;return 0 ; +case BPF_TRACE_FENTRY : +case BPF_TRACE_FEXIT : +if (!btf_type_is_func (t)) { +verbose (env,"attach_btf_id %u is not a function\n" , + btf_id); +return -EINVAL ; + } + t =btf_type_by_id (btf_vmlinux, t->type); +if (!btf_type_is_func_proto (t)) +return -EINVAL ; + tr =bpf_trampoline_lookup (btf_id); +if (!tr) +return -ENOMEM ; + prog->aux->attach_func_name = tname; + prog->aux->attach_func_proto = t; +if (tr->func.addr) { + prog->aux->trampoline = tr; +return 0 ; + } + ret =btf_distill_func_proto (&env->log, btf_vmlinux, t, + tname, &tr->func.model); +if (ret <0 ) { +bpf_trampoline_put (tr); +return ret; + } + addr =kallsyms_lookup_name (tname); +if (!addr) { +verbose (env, +"The address of function %s cannot be found\n" , + tname); +bpf_trampoline_put (tr); +return -ENOENT ; + } + tr->func.addr = (void *)addr; + prog->aux->trampoline = tr; +return 0 ;default :return -EINVAL ; }
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 03/18] bpf: Introduce BPF trampoline
Quoted 3 lines · Alexei Starovoitov
[...]
Quoted 5 lines
BPF trampoline is intended to be used beyond tracing and fentry/fexit use cases in the future. For example to remove retpoline cost from XDP programs.
Signed-off-by Alexei Starovoitov <[email protected]>Acked-by Andrii Nakryiko <[email protected]>
Alexei Starovoitov
maintainerPATCH v3 4/18 libbpf: Introduce btf__find_by_name_kind()
In reply to Alexei StarovoitovIntroduce btf__find_by_name_kind() helper to search BTF by name and kind, since name alone can be ambiguous.
--- tools/lib/bpf/btf.c | 22 ++++++++++++++++++++++ tools/lib/bpf/btf.h | 2 ++ tools/lib/bpf/libbpf.map | 1 + 3 files changed, 25 insertions(+)
tools/lib/bpf/btf.c +22 -0
@@ -316,6 +316,28 @@ __s32 btf__find_by_name(const struct btf *btf, const char *type_name)return -ENOENT ; } +__s32 btf__find_by_name_kind (const struct btf *btf,const char *type_name, +__u32 kind) +{ +__u32 i; + +if (kind ==BTF_KIND_UNKN || !strcmp (type_name,"void" )) +return 0 ; + +for (i =1 ; i <= btf->nr_types; i++) { +const struct btf_type *t = btf->types[i]; +const char *name; + +if (btf_kind (t) != kind) +continue ; + name =btf__name_by_offset (btf, t->name_off); +if (name && !strcmp (type_name, name)) +return i; + } + +return -ENOENT ; +} +void btf__free (struct btf *btf) {if (!btf)
tools/lib/bpf/btf.h +2 -0
@@ -72,6 +72,8 @@ LIBBPF_API int btf__finalize_data(struct bpf_object *obj, struct btf *btf); LIBBPF_API int btf__load(struct btf *btf); LIBBPF_API __s32 btf__find_by_name(const struct btf *btf,const char *type_name); +LIBBPF_API __s32 btf__find_by_name_kind(const struct btf *btf, +const char *type_name,__u32 kind); LIBBPF_API __u32 btf__get_nr_types(const struct btf *btf); LIBBPF_API const struct btf_type *btf__type_by_id(const struct btf *btf,__u32 id);
tools/lib/bpf/libbpf.map +1 -0
@@ -202,4 +202,5 @@ LIBBPF_0.0.6 { bpf_program__get_type; bpf_program__is_tracing; bpf_program__set_tracing; + btf__find_by_name_kind; } LIBBPF_0.0.5;
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 04/18] libbpf: Introduce btf__find_by_name_kind()
Quoted 8 lines · Alexei Starovoitov
Introduce
btf__find_by_name_kind()helper to search BTF by name and kind, since name alone can be ambiguous.Signed-off-by Alexei Starovoitov <[email protected]>
Re: [PATCH v3 bpf-next 04/18] libbpf: Introduce btf__find_by_name_kind()
In reply to Alexei StarovoitovQuoted 6 lines · Alexei Starovoitov
Introduce
btf__find_by_name_kind()helper to search BTF by name and kind, since name alone can be ambiguous.Signed-off-by Alexei Starovoitov <[email protected]>---
Ok, makes sense.
Quoted 4 lines
tools/lib/bpf/btf.c | 22 ++++++++++++++++++++++ tools/lib/bpf/btf.h | 2 ++ tools/lib/bpf/libbpf.map | 1 + 3 files changed, 25 insertions(+)
[...]
Alexei Starovoitov
maintainerPATCH v3 5/18 libbpf: Add support to attach to fentry/fexit tracing progs
In reply to Alexei StarovoitovTeach libbpf to recognize tracing programs types and attach them to fentry/fexit.
--- tools/include/uapi/linux/bpf.h | 2 + tools/lib/bpf/libbpf.c | 99 +++++++++++++++++++++++++--------- tools/lib/bpf/libbpf.h | 4 ++ tools/lib/bpf/libbpf.map | 2 + 4 files changed, 82 insertions(+), 25 deletions(-)
tools/include/uapi/linux/bpf.h +2 -0
@@ -201,6 +201,8 @@ enum bpf_attach_type {BPF_CGROUP_GETSOCKOPT ,BPF_CGROUP_SETSOCKOPT ,BPF_TRACE_RAW_TP , +BPF_TRACE_FENTRY , +BPF_TRACE_FEXIT , __MAX_BPF_ATTACH_TYPE };
tools/lib/bpf/libbpf.c +74 -25
153 lines not on this page · raw message
tools/lib/bpf/libbpf.h +4 -0
@@ -188,6 +188,8 @@ libbpf_prog_type_by_name(const char *name, enum bpf_prog_type *prog_type, enum bpf_attach_type *expected_attach_type); LIBBPF_API int libbpf_attach_type_by_name(const char *name,enum bpf_attach_type *attach_type); +LIBBPF_API int libbpf_find_vmlinux_btf_id(const char *name, +enum bpf_attach_type attach_type);/* Accessors of bpf_program */ struct bpf_program ;
@@ -248,6 +250,8 @@ LIBBPF_API struct bpf_link * bpf_program__attach_raw_tracepoint(struct bpf_program *prog, const char *tp_name); +LIBBPF_API struct bpf_link * +bpf_program__attach_trace (struct bpf_program *prog);struct bpf_insn ;/*
tools/lib/bpf/libbpf.map +2 -0
@@ -198,9 +198,11 @@ LIBBPF_0.0.6 { bpf_map__set_pin_path; bpf_object__open_file; bpf_object__open_mem; + bpf_program__attach_trace; bpf_program__get_expected_attach_type; bpf_program__get_type; bpf_program__is_tracing; bpf_program__set_tracing; btf__find_by_name_kind; + libbpf_find_vmlinux_btf_id; } LIBBPF_0.0.5;
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 05/18] libbpf: Add support to attach to fentry/fexit tracing progs
Quoted 8 lines · Alexei Starovoitov
Teach libbpf to recognize tracing programs types and attach them to fentry/fexit.
Signed-off-by Alexei Starovoitov <[email protected]>
Re: [PATCH v3 bpf-next 05/18] libbpf: Add support to attach to fentry/fexit tracing progs
In reply to Alexei StarovoitovQuoted 6 lines · Alexei Starovoitov
Teach libbpf to recognize tracing programs types and attach them to fentry/fexit.
Signed-off-by Alexei Starovoitov <[email protected]>---
Quoted 6 lines
tools/include/uapi/linux/bpf.h | 2 + tools/lib/bpf/libbpf.c | 99 +++++++++++++++++++++++++--------- tools/lib/bpf/libbpf.h | 4 ++ tools/lib/bpf/libbpf.map | 2 + 4 files changed, 82 insertions(+), 25 deletions(-)
[...]
Alexei Starovoitov
maintainerPATCH v3 6/18 selftest/bpf: Simple test for fentry/fexit
In reply to Alexei StarovoitovAdd simple test for fentry and fexit programs around eth_type_trans.
--- .../selftests/bpf/prog_tests/kfree_skb.c | 39 ++++++++++++-- tools/testing/selftests/bpf/progs/kfree_skb.c | 52 +++++++++++++++++++ 2 files changed, 88 insertions(+), 3 deletions(-)
tools/testing/selftests/bpf/prog_tests/kfree_skb.c +36 -3
@@ -60,15 +60,17 @@ void test_kfree_skb(void) .file = "./kfree_skb.o", }; +struct bpf_link *link =NULL , *link_fentry =NULL , *link_fexit =NULL ; +struct bpf_map *perf_buf_map, *global_data; +struct bpf_program *prog, *fentry, *fexit;struct bpf_object *obj, *obj2 =NULL ;struct perf_buffer_opts pb_opts = {};struct perf_buffer *pb =NULL ; -struct bpf_link *link =NULL ; -struct bpf_map *perf_buf_map; -struct bpf_program *prog;int err, kfree_skb_fd;bool passed = false;__u32 duration =0 ; +const int zero =0 ; +bool test_ok[2 ]; err =bpf_prog_load ("./test_pkt_access.o" ,BPF_PROG_TYPE_SCHED_CLS , &obj, &tattr.prog_fd);
@@ -82,9 +84,28 @@ void test_kfree_skb(void) prog =bpf_object__find_program_by_title (obj2,"tp_btf/kfree_skb" );if (CHECK (!prog,"find_prog" ,"prog kfree_skb not found\n" )) goto close_prog; + fentry =bpf_object__find_program_by_title (obj2,"fentry/eth_type_trans" ); +if (CHECK (!fentry,"find_prog" ,"prog eth_type_trans not found\n" )) + goto close_prog; + fexit =bpf_object__find_program_by_title (obj2,"fexit/eth_type_trans" ); +if (CHECK (!fexit,"find_prog" ,"prog eth_type_trans not found\n" )) + goto close_prog; + + global_data =bpf_object__find_map_by_name (obj2,"kfree_sk.bss" ); +if (CHECK (!global_data,"find global data" ,"not found\n" )) + goto close_prog; + link =bpf_program__attach_raw_tracepoint (prog,NULL );if (CHECK (IS_ERR (link),"attach_raw_tp" ,"err %ld\n" ,PTR_ERR (link))) goto close_prog; + link_fentry =bpf_program__attach_trace (fentry); +if (CHECK (IS_ERR (link_fentry),"attach fentry" ,"err %ld\n" , +PTR_ERR (link_fentry))) + goto close_prog; + link_fexit =bpf_program__attach_trace (fexit); +if (CHECK (IS_ERR (link_fexit),"attach fexit" ,"err %ld\n" , +PTR_ERR (link_fexit))) + goto close_prog; perf_buf_map =bpf_object__find_map_by_name (obj2,"perf_buf_map" );if (CHECK (!perf_buf_map,"find_perf_buf_map" ,"not found\n" ))
@@ -108,14 +129,26 @@ void test_kfree_skb(void) err =perf_buffer__poll (pb,100 );if (CHECK (err <0 ,"perf_buffer__poll" ,"err %d\n" , err)) goto close_prog; +/* make sure kfree_skb program was triggered * and it sent expected skb into ring buffer */ CHECK_FAIL (!passed); + + err =bpf_map_lookup_elem (bpf_map__fd (global_data), &zero, test_ok); +if (CHECK (err,"get_result" , +"failed to get output data: %d\n" , err)) + goto close_prog; + +CHECK_FAIL (!test_ok[0 ] || !test_ok[1 ]); close_prog:perf_buffer__free (pb);if (!IS_ERR_OR_NULL (link))bpf_link__destroy (link); +if (!IS_ERR_OR_NULL (link_fentry)) +bpf_link__destroy (link_fentry); +if (!IS_ERR_OR_NULL (link_fexit)) +bpf_link__destroy (link_fexit);bpf_object__close (obj);bpf_object__close (obj2); }
tools/testing/selftests/bpf/progs/kfree_skb.c +52 -0
61 lines not on this page · raw message
Signature · 1 line
2.23.0
Alexei Starovoitov
maintainerPATCH v3 7/18 bpf: Add kernel test functions for fentry testing
In reply to Alexei StarovoitovAdd few kernel functions with various number of arguments, their types and sizes for BPF trampoline testing to cover different calling conventions.
--- net/bpf/test_run.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
net/bpf/test_run.c +41 -0
@@ -105,6 +105,40 @@ static int bpf_test_finish(const union bpf_attr *kattr,return err; } +/* Integer types of various sizes and pointer combinations cover variety of +* architecture dependent calling conventions. 7+ can be supported in the +* future. +*/ +int noinline bpf_fentry_test1(int a) +{ +return a +1 ; +} + +int noinline bpf_fentry_test2(int a, u64 b) +{ +return a + b; +} + +int noinline bpf_fentry_test3(char a, int b, u64 c) +{ +return a + b + c; +} + +int noinline bpf_fentry_test4(void *a, char b, int c, u64 d) +{ +return (long )a + b + c + d; +} + +int noinline bpf_fentry_test5(u64 a, void *b, short c, int d, u64 e) +{ +return a + (long )b + c + d + e; +} + +int noinline bpf_fentry_test6(u64 a, void *b, short c, int d, void *e, u64 f) +{ +return a + (long )b + c + d + (long )e + f; +} +static void *bpf_test_init (const union bpf_attr *kattr,u32 size,u32 headroom,u32 tailroom) {
@@ -122,6 +156,13 @@ static void *bpf_test_init(const union bpf_attr *kattr, u32 size,kfree (data);return ERR_PTR (-EFAULT ); } +if (bpf_fentry_test1 (1 ) !=2 || +bpf_fentry_test2 (2 ,3 ) !=5 || +bpf_fentry_test3 (4 ,5 ,6 ) !=15 || +bpf_fentry_test4 ((void *)7 ,8 ,9 ,10 ) !=34 || +bpf_fentry_test5 (11 , (void *)12 ,13 ,14 ,15 ) !=65 || +bpf_fentry_test6 (16 , (void *)17 ,18 ,19 , (void *)20 ,21 ) !=111 ) +return ERR_PTR (-EFAULT );return data; }
Signature · 1 line
2.23.0
Alexei Starovoitov
maintainerPATCH v3 8/18 selftests/bpf: Add test for BPF trampoline
In reply to Alexei StarovoitovAdd sanity test for BPF trampoline that checks kernel functions with up to 6 arguments of different sizes.
--- tools/lib/bpf/bpf_helpers.h | 13 +++ .../selftests/bpf/prog_tests/fentry_test.c | 64 +++++++++++++ .../testing/selftests/bpf/progs/fentry_test.c | 90 +++++++++++++++++++ 3 files changed, 167 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/fentry_test.c create mode 100644 tools/testing/selftests/bpf/progs/fentry_test.c
tools/lib/bpf/bpf_helpers.h +13 -0
@@ -44,4 +44,17 @@ enum libbpf_pin_type { LIBBPF_PIN_BY_NAME, }; +/* The following types should be used by BPF_PROG_TYPE_TRACING program to +* access kernel function arguments. BPF trampoline and raw tracepoints +* typecast arguments to 'unsigned long long'. +*/ +typedef int __attribute__((aligned(8))) ks32; +typedef char __attribute__((aligned(8))) ks8; +typedef short __attribute__((aligned(8))) ks16; +typedef long long __attribute__((aligned(8))) ks64; +typedef unsigned int __attribute__((aligned(8))) ku32; +typedef unsigned char __attribute__((aligned(8))) ku8; +typedef unsigned short __attribute__((aligned(8))) ku16; +typedef unsigned long long __attribute__((aligned(8))) ku64; +#endif
tools/testing/selftests/bpf/prog_tests/fentry_test.c new file +64 -0
64 lines not on this page · raw message
tools/testing/selftests/bpf/progs/fentry_test.c new file +90 -0
90 lines not on this page · raw message
Signature · 1 line
2.23.0
Alexei Starovoitov
maintainerPATCH v3 9/18 selftests/bpf: Add fexit tests for BPF trampoline
In reply to Alexei StarovoitovAdd fexit tests for BPF trampoline that checks kernel functions with up to 6 arguments of different sizes and their return values.
--- .../selftests/bpf/prog_tests/fexit_test.c | 64 ++++++++++++ .../testing/selftests/bpf/progs/fexit_test.c | 98 +++++++++++++++++++ 2 files changed, 162 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/fexit_test.c create mode 100644 tools/testing/selftests/bpf/progs/fexit_test.c
tools/testing/selftests/bpf/prog_tests/fexit_test.c new file +64 -0
@@ -0,0 +1,64 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019 Facebook */ +#include <test_progs.h> + +void test_fexit_test (void ) +{ +struct bpf_prog_load_attr attr = { + .file ="./fexit_test.o" , + }; + +char prog_name[] ="fexit/bpf_fentry_testX" ; +struct bpf_object *obj =NULL , *pkt_obj; +int err, pkt_fd, kfree_skb_fd, i; +struct bpf_link *link[6 ] = {}; +struct bpf_program *prog[6 ]; +__u32 duration, retval; +struct bpf_map *data_map; +const int zero =0 ; +u64 result[6 ]; + + err =bpf_prog_load ("./test_pkt_access.o" ,BPF_PROG_TYPE_SCHED_CLS , + &pkt_obj, &pkt_fd); +if (CHECK (err,"prog_load sched cls" ,"err %d errno %d\n" , err, errno)) +return ; + err =bpf_prog_load_xattr (&attr, &obj, &kfree_skb_fd); +if (CHECK (err,"prog_load fail" ,"err %d errno %d\n" , err, errno)) + goto close_prog; + +for (i =0 ; i <6 ; i++) { + prog_name[sizeof (prog_name) -2 ] ='1' + i; + prog[i] =bpf_object__find_program_by_title (obj, prog_name); +if (CHECK (!prog[i],"find_prog" ,"prog %s not found\n" , prog_name)) + goto close_prog; + link[i] =bpf_program__attach_trace (prog[i]); +if (CHECK (IS_ERR (link[i]),"attach_trace" ,"failed to link\n" )) + goto close_prog; + } + data_map =bpf_object__find_map_by_name (obj,"fexit_te.bss" ); +if (CHECK (!data_map,"find_data_map" ,"data map not found\n" )) + goto close_prog; + + err =bpf_prog_test_run (pkt_fd,1 , &pkt_v6,sizeof (pkt_v6), +NULL ,NULL , &retval, &duration); +CHECK (err || retval,"ipv6" , +"err %d errno %d retval %d duration %d\n" , + err, errno, retval, duration); + + err =bpf_map_lookup_elem (bpf_map__fd (data_map), &zero, &result); +if (CHECK (err,"get_result" , +"failed to get output data: %d\n" , err)) + goto close_prog; + +for (i =0 ; i <6 ; i++) +if (CHECK (result[i] !=1 ,"result" ,"bpf_fentry_test%d failed err %ld\n" , + i +1 , result[i])) + goto close_prog; + +close_prog: +for (i =0 ; i <6 ; i++) +if (!IS_ERR_OR_NULL (link[i])) +bpf_link__destroy (link[i]); +bpf_object__close (obj); +bpf_object__close (pkt_obj); +}
tools/testing/selftests/bpf/progs/fexit_test.c new file +98 -0
98 lines not on this page · raw message
Signature · 1 line
2.23.0
Alexei Starovoitov
maintainerPATCH v3 10/18 selftests/bpf: Add combined fentry/fexit test
In reply to Alexei StarovoitovAdd a combined fentry/fexit test.
--- .../selftests/bpf/prog_tests/fentry_fexit.c | 90 +++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/fentry_fexit.c
tools/testing/selftests/bpf/prog_tests/fentry_fexit.c new file +90 -0
@@ -0,0 +1,90 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019 Facebook */ +#include <test_progs.h> + +void test_fentry_fexit (void ) +{ +struct bpf_prog_load_attr attr_fentry = { + .file ="./fentry_test.o" , + }; +struct bpf_prog_load_attr attr_fexit = { + .file ="./fexit_test.o" , + }; + +struct bpf_object *obj_fentry =NULL , *obj_fexit =NULL , *pkt_obj; +struct bpf_map *data_map_fentry, *data_map_fexit; +char fentry_name[] ="fentry/bpf_fentry_testX" ; +char fexit_name[] ="fexit/bpf_fentry_testX" ; +int err, pkt_fd, kfree_skb_fd, i; +struct bpf_link *link[12 ] = {}; +struct bpf_program *prog[12 ]; +__u32 duration, retval; +const int zero =0 ; +u64 result[12 ]; + + err =bpf_prog_load ("./test_pkt_access.o" ,BPF_PROG_TYPE_SCHED_CLS , + &pkt_obj, &pkt_fd); +if (CHECK (err,"prog_load sched cls" ,"err %d errno %d\n" , err, errno)) +return ; + err =bpf_prog_load_xattr (&attr_fentry, &obj_fentry, &kfree_skb_fd); +if (CHECK (err,"prog_load fail" ,"err %d errno %d\n" , err, errno)) + goto close_prog; + err =bpf_prog_load_xattr (&attr_fexit, &obj_fexit, &kfree_skb_fd); +if (CHECK (err,"prog_load fail" ,"err %d errno %d\n" , err, errno)) + goto close_prog; + +for (i =0 ; i <6 ; i++) { + fentry_name[sizeof (fentry_name) -2 ] ='1' + i; + prog[i] =bpf_object__find_program_by_title (obj_fentry, fentry_name); +if (CHECK (!prog[i],"find_prog" ,"prog %s not found\n" , fentry_name)) + goto close_prog; + link[i] =bpf_program__attach_trace (prog[i]); +if (CHECK (IS_ERR (link[i]),"attach_trace" ,"failed to link\n" )) + goto close_prog; + } + data_map_fentry =bpf_object__find_map_by_name (obj_fentry,"fentry_t.bss" ); +if (CHECK (!data_map_fentry,"find_data_map" ,"data map not found\n" )) + goto close_prog; + +for (i =6 ; i <12 ; i++) { + fexit_name[sizeof (fexit_name) -2 ] ='1' + i -6 ; + prog[i] =bpf_object__find_program_by_title (obj_fexit, fexit_name); +if (CHECK (!prog[i],"find_prog" ,"prog %s not found\n" , fexit_name)) + goto close_prog; + link[i] =bpf_program__attach_trace (prog[i]); +if (CHECK (IS_ERR (link[i]),"attach_trace" ,"failed to link\n" )) + goto close_prog; + } + data_map_fexit =bpf_object__find_map_by_name (obj_fexit,"fexit_te.bss" ); +if (CHECK (!data_map_fexit,"find_data_map" ,"data map not found\n" )) + goto close_prog; + + err =bpf_prog_test_run (pkt_fd,1 , &pkt_v6,sizeof (pkt_v6), +NULL ,NULL , &retval, &duration); +CHECK (err || retval,"ipv6" , +"err %d errno %d retval %d duration %d\n" , + err, errno, retval, duration); + + err =bpf_map_lookup_elem (bpf_map__fd (data_map_fentry), &zero, &result); +if (CHECK (err,"get_result" , +"failed to get output data: %d\n" , err)) + goto close_prog; + + err =bpf_map_lookup_elem (bpf_map__fd (data_map_fexit), &zero, result +6 ); +if (CHECK (err,"get_result" , +"failed to get output data: %d\n" , err)) + goto close_prog; + +for (i =0 ; i <12 ; i++) +if (CHECK (result[i] !=1 ,"result" ,"bpf_fentry_test%d failed err %ld\n" , + i %6 +1 , result[i])) + goto close_prog; + +close_prog: +for (i =0 ; i <12 ; i++) +if (!IS_ERR_OR_NULL (link[i])) +bpf_link__destroy (link[i]); +bpf_object__close (obj_fentry); +bpf_object__close (obj_fexit); +bpf_object__close (pkt_obj); +}
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 10/18] selftests/bpf: Add combined fentry/fexit test
Quoted 7 lines · Alexei Starovoitov
Add a combined fentry/fexit test.
Signed-off-by Alexei Starovoitov <[email protected]>
Alexei Starovoitov
maintainerPATCH v3 11/18 selftests/bpf: Add stress test for maximum number of progs
In reply to Alexei StarovoitovAdd stress test for maximum number of attached BPF programs per BPF trampoline.
--- .../selftests/bpf/prog_tests/fexit_stress.c | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/fexit_stress.c
tools/testing/selftests/bpf/prog_tests/fexit_stress.c new file +77 -0
@@ -0,0 +1,77 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019 Facebook */ +#include <test_progs.h> +#include <linux/nbd.h> + +/* x86-64 fits 55 JITed and 43 interpreted progs into half page */ +#define CNT 40 + +void test_fexit_stress (void ) +{ +char test_skb[128 ] = {}; +int fexit_fd[CNT ] = {}; +int link_fd[CNT ] = {}; +__u32 duration =0 ; +char error[4096 ]; +__u32 prog_ret; +int err, i, filter_fd; + +const struct bpf_insn trace_program[] = { +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + }; + +struct bpf_load_program_attr load_attr = { + .prog_type =BPF_PROG_TYPE_TRACING , + .license ="GPL" , + .insns = trace_program, + .insns_cnt =sizeof (trace_program) /sizeof (struct bpf_insn ), + .expected_attach_type =BPF_TRACE_FEXIT , + }; + +const struct bpf_insn skb_program[] = { +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_EXIT_INSN (), + }; + +struct bpf_load_program_attr skb_load_attr = { + .prog_type =BPF_PROG_TYPE_SOCKET_FILTER , + .license ="GPL" , + .insns = skb_program, + .insns_cnt =sizeof (skb_program) /sizeof (struct bpf_insn ), + }; + + err =libbpf_find_vmlinux_btf_id ("bpf_fentry_test1" , + load_attr.expected_attach_type); +if (CHECK (err <=0 ,"find_vmlinux_btf_id" ,"failed: %d\n" , err)) + goto out; + load_attr.attach_btf_id = err; + +for (i =0 ; i <CNT ; i++) { + fexit_fd[i] =bpf_load_program_xattr (&load_attr, error,sizeof (error)); +if (CHECK (fexit_fd[i] <0 ,"fexit loaded" , +"failed: %d errno %d\n" , fexit_fd[i], errno)) + goto out; + link_fd[i] =bpf_raw_tracepoint_open (NULL , fexit_fd[i]); +if (CHECK (link_fd[i] <0 ,"fexit attach failed" , +"prog %d failed: %d err %d\n" , i, link_fd[i], errno)) + goto out; + } + + filter_fd =bpf_load_program_xattr (&skb_load_attr, error,sizeof (error)); +if (CHECK (filter_fd <0 ,"test_program_loaded" ,"failed: %d errno %d\n" , + filter_fd, errno)) + goto out; + + err =bpf_prog_test_run (filter_fd,1 , test_skb,sizeof (test_skb),0 , +0 , &prog_ret,0 ); +close (filter_fd); +CHECK_FAIL (err); +out: +for (i =0 ; i <CNT ; i++) { +if (link_fd[i]) +close (link_fd[i]); +if (fexit_fd[i]) +close (fexit_fd[i]); + } +}
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 11/18] selftests/bpf: Add stress test for maximum number of progs
Quoted 7 lines · Alexei Starovoitov
Add stress test for maximum number of attached BPF programs per BPF trampoline.
Signed-off-by Alexei Starovoitov <[email protected]>
Alexei Starovoitov
maintainerPATCH v3 12/18 bpf: Reserve space for BPF trampoline in BPF programs
In reply to Alexei StarovoitovBPF trampoline can be made to work with existing 5 bytes of BPF program prologue, but let's add 5 bytes of NOPs to the beginning of every JITed BPF program to make BPF trampoline job easier. They can be removed in the future.
--- arch/x86/net/bpf_jit_comp.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
arch/x86/net/bpf_jit_comp.c +7 -2
@@ -206,7 +206,7 @@ struct jit_context {/* number of bytes emit_call() needs to generate call instruction */ #define X86_CALL_SIZE 5 -#define PROLOGUE_SIZE 20 +#define PROLOGUE_SIZE 25/* * Emit x86-64 prologue code for BPF program and check its size.
@@ -215,8 +215,13 @@ struct jit_context {static void emit_prologue (u8 **pprog,u32 stack_depth,bool ebpf_from_cbpf) {u8 *prog = *pprog; -int cnt =0 ; +int cnt =X86_CALL_SIZE ; +/* BPF trampoline can be made to work without these nops, +* but let's waste 5 bytes for now and optimize later +*/ +memcpy (prog, ideal_nops[NOP_ATOMIC5 ], cnt); + prog += cnt;EMIT1 (0x55 );/* push rbp */ EMIT3 (0x48 ,0x89 ,0xE5 );/* mov rbp, rsp */ /* sub rsp, rounded_stack_depth */
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 12/18] bpf: Reserve space for BPF trampoline in BPF programs
Quoted 10 lines · Alexei Starovoitov
BPF trampoline can be made to work with existing 5 bytes of BPF program prologue, but let's add 5 bytes of NOPs to the beginning of every JITed BPF program to make BPF trampoline job easier. They can be removed in the future.
Signed-off-by Alexei Starovoitov <[email protected]>Acked-by Andrii Nakryiko <[email protected]>
Alexei Starovoitov
maintainerPATCH v3 13/18 bpf: Fix race in btf_resolve_helper_id()
In reply to Alexei Starovoitovbtf_resolve_helper_id() caching logic is a bit racy, since under root the verifier can verify several programs in parallel. Fix it with READ/WRITE_ONCE. Fix the type as well, since error is also recorded.
--- include/linux/bpf.h | 5 +++-- kernel/bpf/btf.c | 26 +++++++++++++++++++++++++- kernel/bpf/verifier.c | 8 +++----- net/core/filter.c | 2 +- 4 files changed, 32 insertions(+), 9 deletions(-)
include/linux/bpf.h +3 -2
@@ -247,7 +247,7 @@ struct bpf_func_proto { };enum bpf_arg_type arg_type[5 ]; }; -u32 *btf_id;/* BTF ids of arguments */ +int *btf_id;/* BTF ids of arguments */ };/* bpf_context is intentionally undefined structure. Pointer to bpf_context is
@@ -874,7 +874,8 @@ int btf_struct_access(struct bpf_verifier_log *log, const struct btf_type *t, int off, int size, enum bpf_access_type atype, u32 *next_btf_id); -u32 btf_resolve_helper_id (struct bpf_verifier_log *log,void *,int ); +int btf_resolve_helper_id (struct bpf_verifier_log *log, +const struct bpf_func_proto *fn,int ); int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf,
kernel/bpf/btf.c +25 -1
@@ -3721,7 +3721,8 @@ int btf_struct_access(struct bpf_verifier_log *log,return -EINVAL ; } -u32 btf_resolve_helper_id (struct bpf_verifier_log *log,void *fn,int arg) +static int __btf_resolve_helper_id (struct bpf_verifier_log *log,void *fn, +int arg) {char fnname[KSYM_SYMBOL_LEN +4 ] ="btf_" ;const struct btf_param *args;
@@ -3789,6 +3790,29 @@ u32 btf_resolve_helper_id(struct bpf_verifier_log *log, void *fn, int arg)return btf_id; } +int btf_resolve_helper_id (struct bpf_verifier_log *log, +const struct bpf_func_proto *fn,int arg) +{ +int *btf_id = &fn->btf_id[arg]; +int ret; + +if (fn->arg_type[arg] !=ARG_PTR_TO_BTF_ID ) +return -EINVAL ; + + ret =READ_ONCE (*btf_id); +if (ret) +return ret; +/* ok to race the search. The result is the same */ + ret =__btf_resolve_helper_id (log, fn->func, arg); +if (!ret) { +/* Function argument cannot be type 'void' */ +bpf_log (log,"BTF resolution bug\n" ); +return -EFAULT ; + } +WRITE_ONCE (*btf_id, ret); +return ret; +} +static int __get_type_size (struct btf *btf,u32 btf_id,const struct btf_type **bad_type) {
kernel/bpf/verifier.c +3 -5
@@ -4147,11 +4147,9 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn meta.func_id = func_id;/* check args */ for (i =0 ; i <5 ; i++) { -if (fn->arg_type[i] ==ARG_PTR_TO_BTF_ID ) { -if (!fn->btf_id[i]) - fn->btf_id[i] =btf_resolve_helper_id (&env->log, fn->func, i); - meta.btf_id = fn->btf_id[i]; - } + err =btf_resolve_helper_id (&env->log, fn, i); +if (err >0 ) + meta.btf_id = err; err =check_func_arg (env,BPF_REG_1 + i, fn->arg_type[i], &meta);if (err)return err;
net/core/filter.c +1 -1
@@ -3816,7 +3816,7 @@ static const struct bpf_func_proto bpf_skb_event_output_proto = { .arg5_type = ARG_CONST_SIZE_OR_ZERO, }; -static u32 bpf_skb_output_btf_ids[5 ]; +static int bpf_skb_output_btf_ids[5 ];const struct bpf_func_proto bpf_skb_output_proto = { .func = bpf_skb_event_output, .gpl_only = true,
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 13/18] bpf: Fix race in btf_resolve_helper_id()
Quoted 10 lines · Alexei Starovoitov
btf_resolve_helper_id()caching logic is a bit racy, since under root the verifier can verify several programs in parallel. Fix it with READ/WRITE_ONCE. Fix the type as well, since error is also recorded.Fixes a7658e1a4164 ("bpf: Check types of arguments passed into helpers")Signed-off-by Alexei Starovoitov <[email protected]>
Re: [PATCH v3 bpf-next 13/18] bpf: Fix race in btf_resolve_helper_id()
In reply to Alexei StarovoitovQuoted 8 lines · Alexei Starovoitov
btf_resolve_helper_id()caching logic is a bit racy, since under root the verifier can verify several programs in parallel. Fix it with READ/WRITE_ONCE. Fix the type as well, since error is also recorded.Fixes a7658e1a4164 ("bpf: Check types of arguments passed into helpers")Signed-off-by Alexei Starovoitov <[email protected]>---
looks good
Quoted 6 lines
include/linux/bpf.h | 5 +++-- kernel/bpf/btf.c | 26 +++++++++++++++++++++++++- kernel/bpf/verifier.c | 8 +++----- net/core/filter.c | 2 +- 4 files changed, 32 insertions(+), 9 deletions(-)
[...]
Alexei Starovoitov
maintainerPATCH v3 14/18 bpf: Compare BTF types of functions arguments with actual types
In reply to Alexei StarovoitovMake the verifier check that BTF types of function arguments match actual types passed into top-level BPF program and into BPF-to-BPF calls. If types match such BPF programs and sub-programs will have full support of BPF trampoline. If types mismatch the trampoline has to be conservative. It has to save/restore all 5 program arguments and assume 64-bit scalars. If FENTRY/FEXIT program is attached to this program in the future such FENTRY/FEXIT program will be able to follow pointers only via bpf_probe_read_kernel().
--- include/linux/bpf.h | 8 +++ include/linux/bpf_verifier.h | 1 + kernel/bpf/btf.c | 117 +++++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 1 + kernel/bpf/verifier.c | 18 +++++- 5 files changed, 142 insertions(+), 3 deletions(-)
include/linux/bpf.h +8 -0
@@ -473,6 +473,10 @@ static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog)static inline void bpf_trampoline_put (struct bpf_trampoline *tr) {}#endif +struct bpf_func_info_aux { +bool unreliable; +}; +struct bpf_prog_aux {atomic_t refcnt;u32 used_map_cnt;
@@ -487,6 +491,7 @@ struct bpf_prog_aux {bool verifier_zext;/* Zero extensions has been inserted by verifier. */ bool offload_requested;bool attach_btf_trace;/* true if attaching to BTF-enabled raw tp */ +bool func_proto_unreliable;enum bpf_tramp_prog_type trampoline_prog_type;struct bpf_trampoline *trampoline;struct hlist_node tramp_hlist;
@@ -511,6 +516,7 @@ struct bpf_prog_aux {struct bpf_prog_offload *offload;struct btf *btf;struct bpf_func_info *func_info; +struct bpf_func_info_aux *func_info_aux;/* bpf_line_info loaded from userspace. linfo->insn_off * has the xlated insn offset. * Both the main and sub prog share the same linfo.
@@ -883,6 +889,8 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,const char *func_name, struct btf_func_model *m); +int btf_check_func_arg_match (struct bpf_verifier_env *env,int subprog); +#else /* !CONFIG_BPF_SYSCALL */ static inline struct bpf_prog *bpf_prog_get (u32 ufd) {
include/linux/bpf_verifier.h +1 -0
@@ -343,6 +343,7 @@ static inline bool bpf_verifier_log_needed(const struct bpf_verifier_log *log)#define BPF_MAX_SUBPROGS 256struct bpf_subprog_info { +/* 'start' has to be the first field otherwise find_subprog() won't work */ u32 start;/* insn idx of function entry point */ u32 linfo_idx;/* The idx to the main_prog->aux->linfo */ u16 stack_depth;/* max. stack depth used by this function */
kernel/bpf/btf.c +117 -0
123 lines not on this page · raw message
kernel/bpf/syscall.c +1 -0
@@ -1328,6 +1328,7 @@ static void __bpf_prog_put_rcu(struct rcu_head *rcu) struct bpf_prog_aux *aux = container_of(rcu, struct bpf_prog_aux, rcu);kvfree (aux->func_info); +kfree (aux->func_info_aux);free_used_maps (aux);bpf_prog_uncharge_memlock (aux->prog);security_bpf_prog_free (aux);
kernel/bpf/verifier.c +15 -3
@@ -3970,6 +3970,9 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,/* only increment it after check_reg_arg() finished */ state->curframe++; +if (btf_check_func_arg_match (env, subprog)) +return -EINVAL ; +/* and go analyze first insn of the callee */ *insn_idx = target_insn;
@@ -6564,6 +6567,7 @@ static int check_btf_func(struct bpf_verifier_env *env,u32 i, nfuncs, urec_size, min_size;u32 krec_size =sizeof (struct bpf_func_info );struct bpf_func_info *krecord; +struct bpf_func_info_aux *info_aux =NULL ;const struct btf_type *type;struct bpf_prog *prog;const struct btf *btf;
@@ -6597,6 +6601,9 @@ static int check_btf_func(struct bpf_verifier_env *env, krecord =kvcalloc (nfuncs, krec_size,GFP_KERNEL | __GFP_NOWARN);if (!krecord)return -ENOMEM ; + info_aux =kcalloc (nfuncs,sizeof (*info_aux),GFP_KERNEL | __GFP_NOWARN); +if (!info_aux) + goto err_free;for (i =0 ; i < nfuncs; i++) { ret =bpf_check_uarg_tail_zero (urecord, krec_size, urec_size);
@@ -6648,29 +6655,31 @@ static int check_btf_func(struct bpf_verifier_env *env, ret = -EINVAL ; goto err_free; } - prev_offset = krecord[i].insn_off; urecord += urec_size; } prog->aux->func_info = krecord; prog->aux->func_info_cnt = nfuncs; + prog->aux->func_info_aux = info_aux;return 0 ; err_free:kvfree (krecord); +kfree (info_aux);return ret; }static void adjust_btf_func (struct bpf_verifier_env *env) { +struct bpf_prog_aux *aux = env->prog->aux;int i; -if (!env->prog->aux->func_info) +if (!aux->func_info)return ;for (i =0 ; i < env->subprog_cnt; i++) - env->prog->aux->func_info[i].insn_off = env->subprog_info[i].start; + aux->func_info[i].insn_off = env->subprog_info[i].start; }#define MIN_BPF_LINEINFO_SIZE (offsetof(struct bpf_line_info, line_col) + \
@@ -7651,6 +7660,9 @@ static int do_check(struct bpf_verifier_env *env)0 /* frameno */ , 0 /* subprogno, zero == main subprog */); +if (btf_check_func_arg_match (env,0 )) +return -EINVAL ; +for (;;) {struct bpf_insn *insn;u8 class;
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 14/18] bpf: Compare BTF types of functions arguments with actual types
Quoted 12 lines · Alexei Starovoitov
Make the verifier check that BTF types of function arguments match actual types passed into top-level BPF program and into BPF-to-BPF calls. If types match such BPF programs and sub-programs will have full support of BPF trampoline. If types mismatch the trampoline has to be conservative. It has to save/restore all 5 program arguments and assume 64-bit scalars. If FENTRY/FEXIT program is attached to this program in the future such FENTRY/FEXIT program will be able to follow pointers only via
bpf_probe_read_kernel().
Quoted 1 line
Signed-off-by Alexei Starovoitov <[email protected]>
Re: [PATCH v3 bpf-next 14/18] bpf: Compare BTF types of functions arguments with actual types
Quoted 16 lines · Song Liu
Quoted 13 lines · Alexei Starovoitov
Make the verifier check that BTF types of function arguments match actual types passed into top-level BPF program and into BPF-to-BPF calls. If types match such BPF programs and sub-programs will have full support of BPF trampoline. If types mismatch the trampoline has to be conservative. It has to save/restore all 5 program arguments and assume 64-bit scalars. If FENTRY/FEXIT program is attached to this program in the future such FENTRY/FEXIT program will be able to follow pointers only via
bpf_probe_read_kernel().
Quoted 3 lines
Quoted 1 line
Signed-off-by Alexei Starovoitov <[email protected]>Acked-by Song Liu <[email protected]>
One nit though: maybe use "reliable" instead of "unreliable"
+struct bpf_func_info_aux {
+ bool reliable;
+};
+
+ bool func_proto_reliable;
So the default value 0, is not reliable.
Thanks,
Song
Re: [PATCH v3 bpf-next 14/18] bpf: Compare BTF types of functions arguments with actual types
Quoted 29 lines · Song Liu
Quoted 16 lines · Song Liu
Quoted 13 lines · Alexei Starovoitov
Make the verifier check that BTF types of function arguments match actual types passed into top-level BPF program and into BPF-to-BPF calls. If types match such BPF programs and sub-programs will have full support of BPF trampoline. If types mismatch the trampoline has to be conservative. It has to save/restore all 5 program arguments and assume 64-bit scalars. If FENTRY/FEXIT program is attached to this program in the future such FENTRY/FEXIT program will be able to follow pointers only via
bpf_probe_read_kernel().Quoted 3 lines
Quoted 1 line
Signed-off-by Alexei Starovoitov <[email protected]>Acked-by Song Liu <[email protected]>One nit though: maybe use "reliable" instead of "unreliable"
include/linux/bpf.h +5
-0+struct bpf_func_info_aux { +bool reliable; +}; + +bool func_proto_reliable;
Quoted 1 line
So the default value 0, is not reliable.
I don't see how this can work.
Once particular func proto was found unreliable the verifier won't keep checking. If we start with 'bool reliable = false'
how do you see the whole mechanism working ?
Say the first time the verifier analyzed the subroutine and everything matches. Can it do reliable = true ? No. It has to check all other callsites. Then it would need another variable and extra pass ?
Re: [PATCH v3 bpf-next 14/18] bpf: Compare BTF types of functions arguments with actual types
Quoted 34 lines · Alexei Starovoitov
Quoted 29 lines · Song Liu
Quoted 16 lines · Song Liu
Quoted 13 lines · Alexei Starovoitov
Make the verifier check that BTF types of function arguments match actual types passed into top-level BPF program and into BPF-to-BPF calls. If types match such BPF programs and sub-programs will have full support of BPF trampoline. If types mismatch the trampoline has to be conservative. It has to save/restore all 5 program arguments and assume 64-bit scalars. If FENTRY/FEXIT program is attached to this program in the future such FENTRY/FEXIT program will be able to follow pointers only via
bpf_probe_read_kernel().Quoted 3 lines
Quoted 1 line
Signed-off-by Alexei Starovoitov <[email protected]>Acked-by Song Liu <[email protected]>One nit though: maybe use "reliable" instead of "unreliable"
include/linux/bpf.h +5
-0+struct bpf_func_info_aux { +bool reliable; +}; + +bool func_proto_reliable;
Quoted 9 lines
Quoted 1 line
So the default value 0, is not reliable.
I don't see how this can work.
Once particular func proto was found unreliable the verifier won't keep checking. If we start with 'bool reliable = false'
how do you see the whole mechanism working ?
Say the first time the verifier analyzed the subroutine and everything matches. Can it do reliable = true ? No. It has to check all other callsites. Then it would need another variable and extra pass ?
I see. I missed the multiple call sites part.
Thanks for the explanation.
Song
Re: [PATCH v3 bpf-next 14/18] bpf: Compare BTF types of functions arguments with actual types
In reply to Alexei StarovoitovQuoted 11 lines · Alexei Starovoitov
Make the verifier check that BTF types of function arguments match actual types passed into top-level BPF program and into BPF-to-BPF calls. If types match such BPF programs and sub-programs will have full support of BPF trampoline. If types mismatch the trampoline has to be conservative. It has to save/restore all 5 program arguments and assume 64-bit scalars. If FENTRY/FEXIT program is attached to this program in the future such FENTRY/FEXIT program will be able to follow pointers only via
bpf_probe_read_kernel().Signed-off-by Alexei Starovoitov <[email protected]>---
Quoted 7 lines
include/linux/bpf.h | 8 +++ include/linux/bpf_verifier.h | 1 + kernel/bpf/btf.c | 117 +++++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 1 + kernel/bpf/verifier.c | 18 +++++- 5 files changed, 142 insertions(+), 3 deletions(-)
[...]
Alexei Starovoitov
maintainerPATCH v3 15/18 bpf: Support attaching tracing BPF program to other BPF programs
In reply to Alexei StarovoitovAllow FENTRY/FEXIT BPF programs to attach to other BPF programs of any type including their subprograms. This feature allows snooping on input and output packets in XDP, TC programs including their return values. In order to do that the verifier needs to track types not only of vmlinux, but types of other BPF programs as well. The verifier also needs to translate uapi/linux/bpf.h types used by networking programs into kernel internal BTF types used by FENTRY/FEXIT BPF programs. In some cases LLVM optimizations can remove arguments from BPF subprograms without adjusting BTF info that LLVM backend knows. When BTF info disagrees with actual types that the verifiers sees the BPF trampoline has to fallback to conservative and treat all arguments as u64. The FENTRY/FEXIT program can still attach to such subprograms, but won't be able to recognize pointer types like 'struct sk_buff *' into won't be able to pass them to bpf_skb_output() for dumping to user space.
The BPF_PROG_LOAD command is extended with attach_prog_fd field. When it's set to zero the attach_btf_id is one vmlinux BTF type ids. When attach_prog_fd points to previously loaded BPF program the attach_btf_id is BTF type id of main function or one of its subprograms.
--- arch/x86/net/bpf_jit_comp.c | 3 +- include/linux/bpf.h | 2 + include/linux/btf.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/btf.c | 58 +++++++++++++++++++--- kernel/bpf/core.c | 2 + kernel/bpf/syscall.c | 19 +++++-- kernel/bpf/verifier.c | 98 +++++++++++++++++++++++++++++-------- kernel/trace/bpf_trace.c | 2 - 9 files changed, 151 insertions(+), 35 deletions(-)
arch/x86/net/bpf_jit_comp.c +2 -1
@@ -505,7 +505,8 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,u8 *prog;int ret; -if (!is_kernel_text ((long )ip)) +if (!is_kernel_text ((long )ip) && + !is_bpf_text_address ((long )ip))/* BPF trampoline in modules is not supported */ return -EINVAL ;
include/linux/bpf.h +2 -0
@@ -488,6 +488,7 @@ struct bpf_prog_aux {u32 func_cnt;/* used by non-func prog as the number of func progs */ u32 func_idx;/* 0 for non-func prog, the index in func array for func prog */ u32 attach_btf_id;/* in-kernel BTF type id to attach to */ +struct bpf_prog *linked_prog;bool verifier_zext;/* Zero extensions has been inserted by verifier. */ bool offload_requested;bool attach_btf_trace;/* true if attaching to BTF-enabled raw tp */
@@ -1174,6 +1175,7 @@ extern const struct bpf_func_proto bpf_get_local_storage_proto;extern const struct bpf_func_proto bpf_strtol_proto;extern const struct bpf_func_proto bpf_strtoul_proto;extern const struct bpf_func_proto bpf_tcp_sock_proto; +extern const struct bpf_func_proto bpf_skb_output_proto;/* Shared helpers among cBPF and eBPF. */ void bpf_user_rnd_init_once (void );
include/linux/btf.h +1 -0
@@ -88,6 +88,7 @@ static inline bool btf_type_is_func_proto(const struct btf_type *t)const struct btf_type *btf_type_by_id (const struct btf *btf,u32 type_id);const char *btf_name_by_offset (const struct btf *btf,u32 offset);struct btf *btf_parse_vmlinux (void ); +struct btf *bpf_prog_get_target_btf (const struct bpf_prog *prog);#else static inline const struct btf_type *btf_type_by_id (const struct btf *btf,u32 type_id)
include/uapi/linux/bpf.h +1 -0
@@ -425,6 +425,7 @@ union bpf_attr {__aligned_u64 line_info;/* line info */ __u32 line_info_cnt;/* number of bpf_line_info records */ __u32 attach_btf_id;/* in-kernel BTF type id to attach to */ +__u32 attach_prog_fd;/* 0 to attach to vmlinux */ };struct {/* anonymous struct used by BPF_OBJ_* commands */
kernel/bpf/btf.c +50 -8
111 lines not on this page · raw message
kernel/bpf/core.c +2 -0
@@ -2027,6 +2027,8 @@ void bpf_prog_free(struct bpf_prog *fp) {struct bpf_prog_aux *aux = fp->aux; +if (aux->linked_prog) +bpf_prog_put (aux->linked_prog);INIT_WORK (&aux->work, bpf_prog_free_deferred);schedule_work (&aux->work); }
kernel/bpf/syscall.c +15 -4
@@ -1577,7 +1577,7 @@ static void bpf_prog_load_fixup_attach_type(union bpf_attr *attr)static int bpf_prog_load_check_attach (enum bpf_prog_type prog_type,enum bpf_attach_type expected_attach_type, -u32 btf_id) +u32 btf_id,u32 prog_fd) {switch (prog_type) {case BPF_PROG_TYPE_TRACING :
@@ -1585,7 +1585,7 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type,return -EINVAL ;break ;default : -if (btf_id) +if (btf_id || prog_fd)return -EINVAL ;break ; }
@@ -1636,7 +1636,7 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, }/* last field in 'union bpf_attr' used by this command */ -#define BPF_PROG_LOAD_LAST_FIELD attach_btf_id +#define BPF_PROG_LOAD_LAST_FIELD attach_prog_fdstatic int bpf_prog_load (union bpf_attr *attr,union bpf_attr __user *uattr) {
@@ -1679,7 +1679,8 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr)bpf_prog_load_fixup_attach_type (attr);if (bpf_prog_load_check_attach (type, attr->expected_attach_type, - attr->attach_btf_id)) + attr->attach_btf_id, + attr->attach_prog_fd))return -EINVAL ;/* plain bpf_prog allocation */
@@ -1689,6 +1690,16 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) prog->expected_attach_type = attr->expected_attach_type; prog->aux->attach_btf_id = attr->attach_btf_id; +if (attr->attach_prog_fd) { +struct bpf_prog *tgt_prog; + + tgt_prog =bpf_prog_get (attr->attach_prog_fd); +if (IS_ERR (tgt_prog)) { + err =PTR_ERR (tgt_prog); + goto free_prog_nouncharge; + } + prog->aux->linked_prog = tgt_prog; + } prog->aux->offload_requested = !!attr->prog_ifindex;
kernel/bpf/verifier.c +78 -20
156 lines not on this page · raw message
kernel/trace/bpf_trace.c +0 -2
@@ -1082,8 +1082,6 @@ static const struct bpf_func_proto bpf_perf_event_output_proto_raw_tp = { .arg5_type = ARG_CONST_SIZE_OR_ZERO, }; -extern const struct bpf_func_proto bpf_skb_output_proto; - BPF_CALL_3(bpf_get_stackid_raw_tp, struct bpf_raw_tracepoint_args *, args, struct bpf_map *, map, u64, flags) {
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 15/18] bpf: Support attaching tracing BPF program to other BPF programs
Quoted 16 lines · Alexei Starovoitov
Allow FENTRY/FEXIT BPF programs to attach to other BPF programs of any type including their subprograms. This feature allows snooping on input and output packets in XDP, TC programs including their return values. In order to do that the verifier needs to track types not only of vmlinux, but types of other BPF programs as well. The verifier also needs to translate
uapi/linux/bpf.htypes used by networking programs into kernel internal BTF types used by FENTRY/FEXIT BPF programs. In some cases LLVM optimizations can remove arguments from BPF subprograms without adjusting BTF info that LLVM backend knows. When BTF info disagrees with actual types that the verifiers sees the BPF trampoline has to fallback to conservative and treat all arguments as u64. The FENTRY/FEXIT program can still attach to such subprograms, but won't be able to recognize pointer types like 'structsk_buff*' into won't be able to pass them to
^^^^^ these few words are confusing
Quoted 9 lines
bpf_skb_output()for dumping to user space.The
BPF_PROG_LOADcommand is extended withattach_prog_fdfield. When it's set to zero theattach_btf_idis one vmlinux BTF type ids. Whenattach_prog_fdpoints to previously loaded BPF program theattach_btf_idis BTF type id of main function or one of its subprograms.Signed-off-by Alexei Starovoitov <[email protected]>
[...]
Quoted 24 lines
kernel/bpf/verifier.c +5
-1@@ -9390,13 +9390,17 @@ static void print_verification_stats(struct bpf_verifier_env *env)static int check_attach_btf_id (struct bpf_verifier_env *env) {struct bpf_prog *prog = env->prog; +struct bpf_prog *tgt_prog = prog->aux->linked_prog;u32 btf_id = prog->aux->attach_btf_id;const char prefix[] ="btf_trace_" ;struct bpf_trampoline *tr;const struct btf_type *t; +int ret, subprog =-1 , i; +bool conservative = true;const char *tname; +struct btf *btf;long addr; -int ret; +u64 key;if (prog->type !=BPF_PROG_TYPE_TRACING )return 0 ;
kernel/bpf/verifier.c static int check_attach_btf_id(struct bpf_verifier_env *env) · line 9412
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -9405,19 +9409,42 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)verbose (env,"Tracing programs must provide btf_id\n" );return -EINVAL ; } - t =btf_type_by_id (btf_vmlinux, btf_id); + btf =bpf_prog_get_target_btf (prog);
btf could be NULL here, so we need to check it?
[...]
Re: [PATCH v3 bpf-next 15/18] bpf: Support attaching tracing BPF program to other BPF programs
Quoted 16 lines · Song Liu
Quoted 16 lines · Alexei Starovoitov
Allow FENTRY/FEXIT BPF programs to attach to other BPF programs of any type including their subprograms. This feature allows snooping on input and output packets in XDP, TC programs including their return values. In order to do that the verifier needs to track types not only of vmlinux, but types of other BPF programs as well. The verifier also needs to translate
uapi/linux/bpf.htypes used by networking programs into kernel internal BTF types used by FENTRY/FEXIT BPF programs. In some cases LLVM optimizations can remove arguments from BPF subprograms without adjusting BTF info that LLVM backend knows. When BTF info disagrees with actual types that the verifiers sees the BPF trampoline has to fallback to conservative and treat all arguments as u64. The FENTRY/FEXIT program can still attach to such subprograms, but won't be able to recognize pointer types like 'structsk_buff*' into won't be able to pass them to
Quoted 1 line
^^^^^ these few words are confusing
yep. will fix.
Quoted 33 lines
Quoted 9 lines
bpf_skb_output()for dumping to user space.The
BPF_PROG_LOADcommand is extended withattach_prog_fdfield. When it's set to zero theattach_btf_idis one vmlinux BTF type ids. Whenattach_prog_fdpoints to previously loaded BPF program theattach_btf_idis BTF type id of main function or one of its subprograms.Signed-off-by Alexei Starovoitov <[email protected]>[...]
Quoted 21 lines
kernel/bpf/verifier.c +5
-1@@ -9390,13 +9390,17 @@ static void print_verification_stats(struct bpf_verifier_env *env)static int check_attach_btf_id (struct bpf_verifier_env *env) {struct bpf_prog *prog = env->prog; +struct bpf_prog *tgt_prog = prog->aux->linked_prog;u32 btf_id = prog->aux->attach_btf_id;const char prefix[] ="btf_trace_" ;struct bpf_trampoline *tr;const struct btf_type *t; +int ret, subprog =-1 , i; +bool conservative = true;const char *tname; +struct btf *btf;long addr; -int ret; +u64 key;
Quoted 10 lines
kernel/bpf/verifier.c static int check_attach_btf_id(struct bpf_verifier_env *env) · line 9412
Quoted 8 lines
kernel/bpf/verifier.c +1
-1if (prog->type !=BPF_PROG_TYPE_TRACING )return 0 ;@@ -9405,19 +9409,42 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)verbose (env,"Tracing programs must provide btf_id\n" );return -EINVAL ; } - t =btf_type_by_id (btf_vmlinux, btf_id); + btf =bpf_prog_get_target_btf (prog);btf could be NULL here, so we need to check it?
yep. will fix.
Re: [PATCH v3 bpf-next 15/18] bpf: Support attaching tracing BPF program to other BPF programs
In reply to Alexei StarovoitovQuoted 14 lines · Alexei Starovoitov
Allow FENTRY/FEXIT BPF programs to attach to other BPF programs of any type including their subprograms. This feature allows snooping on input and output packets in XDP, TC programs including their return values. In order to do that the verifier needs to track types not only of vmlinux, but types of other BPF programs as well. The verifier also needs to translate
uapi/linux/bpf.htypes used by networking programs into kernel internal BTF types used by FENTRY/FEXIT BPF programs. In some cases LLVM optimizations can remove arguments from BPF subprograms without adjusting BTF info that LLVM backend knows. When BTF info disagrees with actual types that the verifiers sees the BPF trampoline has to fallback to conservative and treat all arguments as u64. The FENTRY/FEXIT program can still attach to such subprograms, but won't be able to recognize pointer types like 'structsk_buff*' into won't be able to pass them tobpf_skb_output()for dumping to user space.
Quoted 6 lines
The
BPF_PROG_LOADcommand is extended withattach_prog_fdfield. When it's set to zero theattach_btf_idis one vmlinux BTF type ids. Whenattach_prog_fdpoints to previously loaded BPF program theattach_btf_idis BTF type id of main function or one of its subprograms.Signed-off-by Alexei Starovoitov <[email protected]>
This is cool! Certainly solves the xdpdump use case; thanks!
I do have a few questions (thinking about whether it can also be used for running multiple XDP programs):
- Can a FEXIT function loaded this way only *observe* the return code of the BPF program it attaches to, or can it also change it?
- Is it possible to attach multiple FENTRY/FEXIT programs to the same XDP program and/or to recursively attach FENTRY/FEXIT programs to each other?
- Could it be possible for an FENTRY/FEXIT program to call into another XDP program (i.e., one that has the regular XDP program type)?
-Toke
Re: [PATCH v3 bpf-next 15/18] bpf: Support attaching tracing BPF program to other BPF programs
Quoted 21 lines · Toke Høiland-Jørgensen
Quoted 19 lines · Alexei Starovoitov
Allow FENTRY/FEXIT BPF programs to attach to other BPF programs of any type including their subprograms. This feature allows snooping on input and output packets in XDP, TC programs including their return values. In order to do that the verifier needs to track types not only of vmlinux, but types of other BPF programs as well. The verifier also needs to translate
uapi/linux/bpf.htypes used by networking programs into kernel internal BTF types used by FENTRY/FEXIT BPF programs. In some cases LLVM optimizations can remove arguments from BPF subprograms without adjusting BTF info that LLVM backend knows. When BTF info disagrees with actual types that the verifiers sees the BPF trampoline has to fallback to conservative and treat all arguments as u64. The FENTRY/FEXIT program can still attach to such subprograms, but won't be able to recognize pointer types like 'structsk_buff*' into won't be able to pass them tobpf_skb_output()for dumping to user space.The
BPF_PROG_LOADcommand is extended withattach_prog_fdfield. When it's set to zero theattach_btf_idis one vmlinux BTF type ids. Whenattach_prog_fdpoints to previously loaded BPF program theattach_btf_idis BTF type id of main function or one of its subprograms.
Quoted 6 lines
Quoted 1 line
Signed-off-by Alexei Starovoitov <[email protected]>This is cool! Certainly solves the xdpdump use case; thanks!
I do have a few questions (thinking about whether it can also be used for running multiple XDP programs):
excellent questions.
Quoted 2 lines
- Can a FEXIT function loaded this way only *observe* the return code of the BPF program it attaches to, or can it also change it?
yes. the verifier can be taught to support that for certain class of programs. That needs careful thinking to make sure it's safe.
Quoted 2 lines
- Is it possible to attach multiple FENTRY/FEXIT programs to the same XDP program
Yes. Already possible. See fexit_stress.c that attaches 40 progs to the same kernel function. Same thing when attaching fexit BPF to any XDP program. Since all of them are read only tracing prog all progs have access to skb on input and ouput along with unmodified return value.
Quoted 2 lines
and/or to recursively attach FENTRY/FEXIT programs to each
other?
Not right now to avoid complex logic of detecting cycles. See simple bit:
if (tgt_prog->type == BPF_PROG_TYPE_TRACING) {
/* prevent cycles */
verbose(env, "Cannot recursively attach\n");Quoted 2 lines
- Could it be possible for an FENTRY/FEXIT program to call into another XDP program (i.e., one that has the regular XDP program type)?
It's possible to teach verifier to do that, but we probably shouldn't take that route. Instead I've started exploring the idea of dynamic linking. The trampoline logic will be used to replace existing BPF program or subprogram instead of attaching read-only to it. If types match the new program can replace existing one. The concept allows to build any kind of callchain programmatically. Pretty much what Ed proposed with static linking, but doing it dynamically. I'll start a separate email thread explaining details.
Re: [PATCH v3 bpf-next 15/18] bpf: Support attaching tracing BPF program to other BPF programs
Quoted 29 lines · Alexei Starovoitov
Quoted 21 lines · Toke Høiland-Jørgensen
Quoted 19 lines · Alexei Starovoitov
Allow FENTRY/FEXIT BPF programs to attach to other BPF programs of any type including their subprograms. This feature allows snooping on input and output packets in XDP, TC programs including their return values. In order to do that the verifier needs to track types not only of vmlinux, but types of other BPF programs as well. The verifier also needs to translate
uapi/linux/bpf.htypes used by networking programs into kernel internal BTF types used by FENTRY/FEXIT BPF programs. In some cases LLVM optimizations can remove arguments from BPF subprograms without adjusting BTF info that LLVM backend knows. When BTF info disagrees with actual types that the verifiers sees the BPF trampoline has to fallback to conservative and treat all arguments as u64. The FENTRY/FEXIT program can still attach to such subprograms, but won't be able to recognize pointer types like 'structsk_buff*' into won't be able to pass them tobpf_skb_output()for dumping to user space.The
BPF_PROG_LOADcommand is extended withattach_prog_fdfield. When it's set to zero theattach_btf_idis one vmlinux BTF type ids. Whenattach_prog_fdpoints to previously loaded BPF program theattach_btf_idis BTF type id of main function or one of its subprograms.Quoted 6 lines
Quoted 1 line
Signed-off-by Alexei Starovoitov <[email protected]>This is cool! Certainly solves the xdpdump use case; thanks!
I do have a few questions (thinking about whether it can also be used for running multiple XDP programs):
Quoted 7 lines
excellent questions.
Quoted 2 lines
- Can a FEXIT function loaded this way only *observe* the return code of the BPF program it attaches to, or can it also change it?
yes. the verifier can be taught to support that for certain class of programs. That needs careful thinking to make sure it's safe.
OK. I think this could potentially be useful to have for XDP (for instance, to have xdpdump "steal" any packets it is observing by changing the return code to XDP_DROP).
Quoted 7 lines
Quoted 2 lines
- Is it possible to attach multiple FENTRY/FEXIT programs to the same XDP program
Yes. Already possible. See
fexit_stress.cthat attaches 40 progs to the same kernel function. Same thing when attaching fexit BPF to any XDP program. Since all of them are read only tracing prog all progs have access to skb on input and ouput along with unmodified return value.
Right, cool.
Quoted 7 lines
Quoted 2 lines
and/or to recursively attach FENTRY/FEXIT programs to each
other?Not right now to avoid complex logic of detecting cycles. See simple bit:
if (tgt_prog->type == BPF_PROG_TYPE_TRACING) { /* prevent cycles */ verbose(env, "Cannot recursively attach\n");
OK, that is probably a reasonable tradeoff.
Quoted 10 lines
Quoted 2 lines
- Could it be possible for an FENTRY/FEXIT program to call into another XDP program (i.e., one that has the regular XDP program type)?
It's possible to teach verifier to do that, but we probably shouldn't take that route. Instead I've started exploring the idea of dynamic linking. The trampoline logic will be used to replace existing BPF program or subprogram instead of attaching read-only to it. If types match the new program can replace existing one. The concept allows to build any kind of callchain programmatically. Pretty much what Ed proposed with static linking, but doing it dynamically. I'll start a separate email thread explaining details.
SGTM; will wait for the sequel, then :)
-Toke
Re: [PATCH v3 bpf-next 15/18] bpf: Support attaching tracing BPF program to other BPF programs
In reply to Alexei StarovoitovQuoted 33 lines · Alexei Starovoitov
Allow FENTRY/FEXIT BPF programs to attach to other BPF programs of any type including their subprograms. This feature allows snooping on input and output packets in XDP, TC programs including their return values. In order to do that the verifier needs to track types not only of vmlinux, but types of other BPF programs as well. The verifier also needs to translate
uapi/linux/bpf.htypes used by networking programs into kernel internal BTF types used by FENTRY/FEXIT BPF programs. In some cases LLVM optimizations can remove arguments from BPF subprograms without adjusting BTF info that LLVM backend knows. When BTF info disagrees with actual types that the verifiers sees the BPF trampoline has to fallback to conservative and treat all arguments as u64. The FENTRY/FEXIT program can still attach to such subprograms, but won't be able to recognize pointer types like 'structsk_buff*' into won't be able to pass them tobpf_skb_output()for dumping to user space.The
BPF_PROG_LOADcommand is extended withattach_prog_fdfield. When it's set to zero theattach_btf_idis one vmlinux BTF type ids. Whenattach_prog_fdpoints to previously loaded BPF program theattach_btf_idis BTF type id of main function or one of its subprograms.Signed-off-by Alexei Starovoitov <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 3 +- include/linux/bpf.h | 2 + include/linux/btf.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/btf.c | 58 +++++++++++++++++++--- kernel/bpf/core.c | 2 + kernel/bpf/syscall.c | 19 +++++-- kernel/bpf/verifier.c | 98 +++++++++++++++++++++++++++++-------- kernel/trace/bpf_trace.c | 2 - 9 files changed, 151 insertions(+), 35 deletions(-)
[...]
Quoted 4 lines
kernel/bpf/btf.c +4
-0+ +static bool btf_translate_to_vmlinux(struct bpf_verifier_log *log, + struct btf *btf, + const struct btf_type *t,
kernel/bpf/btf.c struct btf *btf_parse_vmlinux(void) · line 3519
Quoted 10 lines
kernel/bpf/btf.c +10
-0+ struct bpf_insn_access_aux *info) +{ +const char *tname =__btf_name_by_offset (btf, t->name_off); +int btf_id; + +if (!tname) { +bpf_log (log,"Program's type doesn't have a name\n" ); +return false; + } +if (strcmp (tname,"__sk_buff" ) ==0 ) {
might be a good idea to ensure that t's type is also a struct?
Quoted 1 line
+
btf_id=btf_resolve_helper_id(log, &bpf_skb_output_proto, 0);
This is kind of ugly and high-maintenance. Have you considered having something like this, to do this mapping:
struct bpf_ctx_mapping {
struct sk_buff *__sk_buff;
struct xdp_buff *xdp_md;
};So field name is a name you are trying to match, while field type is actual type you are mapping to? You won't need to find special function protos (like bpf_skb_output_proto), it will be easy to extend, you'll have real vmlinux types automatically captured for you (you'll just have to initially find bpf_ctx_mapping's btf_id).
Quoted 8 lines
kernel/bpf/btf.c +7
-0+if (btf_id <0 ) +return false; + info->btf_id = btf_id; +return true; + } +return false; +}
[...]
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+if (tgt_prog && conservative) { +struct btf_func_model *m = &tr->func.model; + +/* BTF function prototype doesn't match the verifier types.
kernel/bpf/verifier.c static int check_attach_btf_id(struct bpf_verifier_env *env) · line 9508
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+* Fall back to 5 u64 args. +*/ +for (i =0 ; i <5 ; i++) + m->arg_size[i] =8 ; + m->ret_size =8 ; + m->nr_args =5 ; + prog->aux->attach_func_proto =NULL ; + }else { + ret =btf_distill_func_proto (&env->log, btf, t, + tname, &tr->func.model);
there is nothing preventing some parallel thread to modify tr->func.model in parallel, right? Should these modifications be either locked or at least WRITE_ONCE, similar to btf_resolve_helper_id?
Quoted 3 lines
kernel/bpf/verifier.c +3
-0+if (ret <0 ) + goto out; + }
[...]
Re: [PATCH v3 bpf-next 15/18] bpf: Support attaching tracing BPF program to other BPF programs
Quoted 64 lines · Andrii Nakryiko
Quoted 33 lines · Alexei Starovoitov
Allow FENTRY/FEXIT BPF programs to attach to other BPF programs of any type including their subprograms. This feature allows snooping on input and output packets in XDP, TC programs including their return values. In order to do that the verifier needs to track types not only of vmlinux, but types of other BPF programs as well. The verifier also needs to translate
uapi/linux/bpf.htypes used by networking programs into kernel internal BTF types used by FENTRY/FEXIT BPF programs. In some cases LLVM optimizations can remove arguments from BPF subprograms without adjusting BTF info that LLVM backend knows. When BTF info disagrees with actual types that the verifiers sees the BPF trampoline has to fallback to conservative and treat all arguments as u64. The FENTRY/FEXIT program can still attach to such subprograms, but won't be able to recognize pointer types like 'structsk_buff*' into won't be able to pass them tobpf_skb_output()for dumping to user space.The
BPF_PROG_LOADcommand is extended withattach_prog_fdfield. When it's set to zero theattach_btf_idis one vmlinux BTF type ids. Whenattach_prog_fdpoints to previously loaded BPF program theattach_btf_idis BTF type id of main function or one of its subprograms.Signed-off-by Alexei Starovoitov <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 3 +- include/linux/bpf.h | 2 + include/linux/btf.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/btf.c | 58 +++++++++++++++++++--- kernel/bpf/core.c | 2 + kernel/bpf/syscall.c | 19 +++++-- kernel/bpf/verifier.c | 98 +++++++++++++++++++++++++++++-------- kernel/trace/bpf_trace.c | 2 - 9 files changed, 151 insertions(+), 35 deletions(-)[...]
Quoted 4 lines
kernel/bpf/btf.c +4
-0+ +static bool btf_translate_to_vmlinux(struct bpf_verifier_log *log, + struct btf *btf, + const struct btf_type *t,kernel/bpf/btf.c struct btf *btf_parse_vmlinux(void) · line 3519
Quoted 10 lines
kernel/bpf/btf.c +10
-0+ struct bpf_insn_access_aux *info) +{ +const char *tname =__btf_name_by_offset (btf, t->name_off); +int btf_id; + +if (!tname) { +bpf_log (log,"Program's type doesn't have a name\n" ); +return false; + } +if (strcmp (tname,"__sk_buff" ) ==0 ) {might be a good idea to ensure that t's type is also a struct?
Quoted 1 line
+
btf_id=btf_resolve_helper_id(log, &bpf_skb_output_proto, 0);This is kind of ugly and high-maintenance. Have you considered having something like this, to do this mapping:
struct bpf_ctx_mapping { struct sk_buff *__sk_buff; struct xdp_buff *xdp_md; };
Quoted 5 lines
So field name is a name you are trying to match, while field type is actual type you are mapping to? You won't need to find special function protos (like
bpf_skb_output_proto), it will be easy to extend, you'll have real vmlinux types automatically captured for you (you'll just have to initially find bpf_ctx_mapping'sbtf_id).
I was thinking something along these lines.
The problem with single struct like above is that it's centralized. convert_ctx_access callbacks are all over the place.
So I'm thinking to add macro like this to bpf.h
+#define BPF_RECORD_CTX_CONVERSION(user_type, kernel_type) \
+ ({typedef kernel_type (*bpf_ctx_convert)(user_type); \
+ (void) (bpf_ctx_convert) (void *) 0;})
and then doBPF_RECORD_CTX_CONVERSION(struct bpf_xdp_sock, struct xdp_sock); inside convert_ctx_access functions (like bpf_xdp_sock_convert_ctx_access). There will be several typedefs with 'bpf_ctx_convert' name. The btf_translate_to_vmlinux() will iterate over them. Speed is not criticial here, but long term we probably need to merge prog's BTF with vmlinux's BTF, so most of the type comparison is done during prog load. It probably should reduce the size of prog's BTF too. Renumbering of prog's BTF will be annoying though. Something to consider long term.
Quoted 27 lines
Quoted 9 lines
kernel/bpf/btf.c +7
-0+if (btf_id <0 ) +return false; + info->btf_id = btf_id; +return true; + } +return false; +}[...]
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+if (tgt_prog && conservative) { +struct btf_func_model *m = &tr->func.model; + +/* BTF function prototype doesn't match the verifier types. kernel/bpf/verifier.c static int check_attach_btf_id(struct bpf_verifier_env *env) · line 9508
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+* Fall back to 5 u64 args. +*/ +for (i =0 ; i <5 ; i++) + m->arg_size[i] =8 ; + m->ret_size =8 ; + m->nr_args =5 ; + prog->aux->attach_func_proto =NULL ; + }else { + ret =btf_distill_func_proto (&env->log, btf, t, + tname, &tr->func.model);
Quoted 4 lines
there is nothing preventing some parallel thread to modify
tr->func.modelin parallel, right? Should these modifications be either locked or at leastWRITE_ONCE, similar tobtf_resolve_helper_id?
hmm. Right. There is a race with bpf_trampoline_lookup. One thread could have just created the trampoline and still doing distill, while another thread is trying to use it after getting it from bpf_trampoline_lookup. The fix choices are not pretty. Either to add a mutex to check_attach_btf_id() or do bpf_trampoline_lookup_or_create() with extra callback that does btf_distill_func_proto while bpf_trampoline_lookup_or_create is holding trampoline_mutex or move most of the check_attach_btf_id() logic into bpf_trampoline_lookup_or_create().
I tried to keep trampoline as abstract concept, but with callback or move the verifer and btf logic will bleed into trampoline. Hmm.
Re: [PATCH v3 bpf-next 15/18] bpf: Support attaching tracing BPF program to other BPF programs
Quoted 80 lines · Alexei Starovoitov
Quoted 64 lines · Andrii Nakryiko
Quoted 33 lines · Alexei Starovoitov
Allow FENTRY/FEXIT BPF programs to attach to other BPF programs of any type including their subprograms. This feature allows snooping on input and output packets in XDP, TC programs including their return values. In order to do that the verifier needs to track types not only of vmlinux, but types of other BPF programs as well. The verifier also needs to translate
uapi/linux/bpf.htypes used by networking programs into kernel internal BTF types used by FENTRY/FEXIT BPF programs. In some cases LLVM optimizations can remove arguments from BPF subprograms without adjusting BTF info that LLVM backend knows. When BTF info disagrees with actual types that the verifiers sees the BPF trampoline has to fallback to conservative and treat all arguments as u64. The FENTRY/FEXIT program can still attach to such subprograms, but won't be able to recognize pointer types like 'structsk_buff*' into won't be able to pass them tobpf_skb_output()for dumping to user space.The
BPF_PROG_LOADcommand is extended withattach_prog_fdfield. When it's set to zero theattach_btf_idis one vmlinux BTF type ids. Whenattach_prog_fdpoints to previously loaded BPF program theattach_btf_idis BTF type id of main function or one of its subprograms.Signed-off-by Alexei Starovoitov <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 3 +- include/linux/bpf.h | 2 + include/linux/btf.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/btf.c | 58 +++++++++++++++++++--- kernel/bpf/core.c | 2 + kernel/bpf/syscall.c | 19 +++++-- kernel/bpf/verifier.c | 98 +++++++++++++++++++++++++++++-------- kernel/trace/bpf_trace.c | 2 - 9 files changed, 151 insertions(+), 35 deletions(-)[...]
Quoted 4 lines
kernel/bpf/btf.c +4
-0+ +static bool btf_translate_to_vmlinux(struct bpf_verifier_log *log, + struct btf *btf, + const struct btf_type *t,kernel/bpf/btf.c struct btf *btf_parse_vmlinux(void) · line 3519
Quoted 10 lines
kernel/bpf/btf.c +10
-0+ struct bpf_insn_access_aux *info) +{ +const char *tname =__btf_name_by_offset (btf, t->name_off); +int btf_id; + +if (!tname) { +bpf_log (log,"Program's type doesn't have a name\n" ); +return false; + } +if (strcmp (tname,"__sk_buff" ) ==0 ) {might be a good idea to ensure that t's type is also a struct?
Quoted 1 line
+
btf_id=btf_resolve_helper_id(log, &bpf_skb_output_proto, 0);This is kind of ugly and high-maintenance. Have you considered having something like this, to do this mapping:
struct bpf_ctx_mapping { struct sk_buff *__sk_buff; struct xdp_buff *xdp_md; };Quoted 5 lines
So field name is a name you are trying to match, while field type is actual type you are mapping to? You won't need to find special function protos (like
bpf_skb_output_proto), it will be easy to extend, you'll have real vmlinux types automatically captured for you (you'll just have to initially find bpf_ctx_mapping'sbtf_id).I was thinking something along these lines.
The problem with single struct like above is that it's centralized.convert_ctx_accesscallbacks are all over the place.
So I'm thinking to add macro like this tobpf.h(quoted diff) +3
-0+#define BPF_RECORD_CTX_CONVERSION(user_type, kernel_type) \ + ({typedef kernel_type (*bpf_ctx_convert)(user_type); \ + (void) (bpf_ctx_convert) (void *) 0;})
Quoted 5 lines
and then do
BPF_RECORD_CTX_CONVERSION(struct bpf_xdp_sock, struct xdp_sock); insideconvert_ctx_accessfunctions (likebpf_xdp_sock_convert_ctx_access). There will be several typedefs with 'bpf_ctx_convert' name. Thebtf_translate_to_vmlinux()will iterate over them. Speed is not criticial here,
I guess that works as well. Please leave a comment explaining the idea behind this distributed mapping :)
Quoted 37 lines
but long term we probably need to merge prog's BTF with vmlinux's BTF, so most of the type comparison is done during prog load. It probably should reduce the size of prog's BTF too. Renumbering of prog's BTF will be annoying though. Something to consider long term.
Quoted 27 lines
Quoted 9 lines
kernel/bpf/btf.c +7
-0+if (btf_id <0 ) +return false; + info->btf_id = btf_id; +return true; + } +return false; +}[...]
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+if (tgt_prog && conservative) { +struct btf_func_model *m = &tr->func.model; + +/* BTF function prototype doesn't match the verifier types. kernel/bpf/verifier.c static int check_attach_btf_id(struct bpf_verifier_env *env) · line 9508
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+* Fall back to 5 u64 args. +*/ +for (i =0 ; i <5 ; i++) + m->arg_size[i] =8 ; + m->ret_size =8 ; + m->nr_args =5 ; + prog->aux->attach_func_proto =NULL ; + }else { + ret =btf_distill_func_proto (&env->log, btf, t, + tname, &tr->func.model);Quoted 4 lines
there is nothing preventing some parallel thread to modify
tr->func.modelin parallel, right? Should these modifications be either locked or at leastWRITE_ONCE, similar tobtf_resolve_helper_id?
Quoted 10 lines
hmm. Right. There is a race with
bpf_trampoline_lookup. One thread could have just created the trampoline and still doing distill, while another thread is trying to use it after getting it frombpf_trampoline_lookup. The fix choices are not pretty. Either to add a mutex tocheck_attach_btf_id()or dobpf_trampoline_lookup_or_create()with extra callback that doesbtf_distill_func_protowhilebpf_trampoline_lookup_or_createis holdingtrampoline_mutexor move most of thecheck_attach_btf_id()logic intobpf_trampoline_lookup_or_create().
I tried to keep trampoline as abstract concept, but with callback or move the verifer and btf logic will bleed into trampoline. Hmm.
yeah, that sounds too intrusive. I'd change btf_distill_func_proto to accept struct btf_func_model **m, allocate model dynamically, and then compare_exchange the final constructed model pointer.
Similarly for "fallback to conservative" case.
Quoted 1 line
Re: [PATCH v3 bpf-next 15/18] bpf: Support attaching tracing BPF program to other BPF programs
Quoted 139 lines · Andrii Nakryiko
Quoted 80 lines · Alexei Starovoitov
Quoted 64 lines · Andrii Nakryiko
Quoted 33 lines · Alexei Starovoitov
Allow FENTRY/FEXIT BPF programs to attach to other BPF programs of any type including their subprograms. This feature allows snooping on input and output packets in XDP, TC programs including their return values. In order to do that the verifier needs to track types not only of vmlinux, but types of other BPF programs as well. The verifier also needs to translate
uapi/linux/bpf.htypes used by networking programs into kernel internal BTF types used by FENTRY/FEXIT BPF programs. In some cases LLVM optimizations can remove arguments from BPF subprograms without adjusting BTF info that LLVM backend knows. When BTF info disagrees with actual types that the verifiers sees the BPF trampoline has to fallback to conservative and treat all arguments as u64. The FENTRY/FEXIT program can still attach to such subprograms, but won't be able to recognize pointer types like 'structsk_buff*' into won't be able to pass them tobpf_skb_output()for dumping to user space.The
BPF_PROG_LOADcommand is extended withattach_prog_fdfield. When it's set to zero theattach_btf_idis one vmlinux BTF type ids. Whenattach_prog_fdpoints to previously loaded BPF program theattach_btf_idis BTF type id of main function or one of its subprograms.Signed-off-by Alexei Starovoitov <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 3 +- include/linux/bpf.h | 2 + include/linux/btf.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/btf.c | 58 +++++++++++++++++++--- kernel/bpf/core.c | 2 + kernel/bpf/syscall.c | 19 +++++-- kernel/bpf/verifier.c | 98 +++++++++++++++++++++++++++++-------- kernel/trace/bpf_trace.c | 2 - 9 files changed, 151 insertions(+), 35 deletions(-)[...]
Quoted 4 lines
kernel/bpf/btf.c +4
-0+ +static bool btf_translate_to_vmlinux(struct bpf_verifier_log *log, + struct btf *btf, + const struct btf_type *t,kernel/bpf/btf.c struct btf *btf_parse_vmlinux(void) · line 3519
Quoted 10 lines
kernel/bpf/btf.c +10
-0+ struct bpf_insn_access_aux *info) +{ +const char *tname =__btf_name_by_offset (btf, t->name_off); +int btf_id; + +if (!tname) { +bpf_log (log,"Program's type doesn't have a name\n" ); +return false; + } +if (strcmp (tname,"__sk_buff" ) ==0 ) {might be a good idea to ensure that t's type is also a struct?
Quoted 1 line
+
btf_id=btf_resolve_helper_id(log, &bpf_skb_output_proto, 0);This is kind of ugly and high-maintenance. Have you considered having something like this, to do this mapping:
struct bpf_ctx_mapping { struct sk_buff *__sk_buff; struct xdp_buff *xdp_md; };Quoted 5 lines
So field name is a name you are trying to match, while field type is actual type you are mapping to? You won't need to find special function protos (like
bpf_skb_output_proto), it will be easy to extend, you'll have real vmlinux types automatically captured for you (you'll just have to initially find bpf_ctx_mapping'sbtf_id).I was thinking something along these lines.
The problem with single struct like above is that it's centralized.convert_ctx_accesscallbacks are all over the place.
So I'm thinking to add macro like this tobpf.h(quoted diff) +3
-0+#define BPF_RECORD_CTX_CONVERSION(user_type, kernel_type) \ + ({typedef kernel_type (*bpf_ctx_convert)(user_type); \ + (void) (bpf_ctx_convert) (void *) 0;})Quoted 5 lines
and then do
BPF_RECORD_CTX_CONVERSION(struct bpf_xdp_sock, struct xdp_sock); insideconvert_ctx_accessfunctions (likebpf_xdp_sock_convert_ctx_access). There will be several typedefs with 'bpf_ctx_convert' name. Thebtf_translate_to_vmlinux()will iterate over them. Speed is not criticial here,I guess that works as well. Please leave a comment explaining the idea behind this distributed mapping :)
Quoted 37 lines
but long term we probably need to merge prog's BTF with vmlinux's BTF, so most of the type comparison is done during prog load. It probably should reduce the size of prog's BTF too. Renumbering of prog's BTF will be annoying though. Something to consider long term.
Quoted 27 lines
Quoted 9 lines
kernel/bpf/btf.c +7
-0+if (btf_id <0 ) +return false; + info->btf_id = btf_id; +return true; + } +return false; +}[...]
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+if (tgt_prog && conservative) { +struct btf_func_model *m = &tr->func.model; + +/* BTF function prototype doesn't match the verifier types. kernel/bpf/verifier.c static int check_attach_btf_id(struct bpf_verifier_env *env) · line 9508
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+* Fall back to 5 u64 args. +*/ +for (i =0 ; i <5 ; i++) + m->arg_size[i] =8 ; + m->ret_size =8 ; + m->nr_args =5 ; + prog->aux->attach_func_proto =NULL ; + }else { + ret =btf_distill_func_proto (&env->log, btf, t, + tname, &tr->func.model);Quoted 4 lines
there is nothing preventing some parallel thread to modify
tr->func.modelin parallel, right? Should these modifications be either locked or at leastWRITE_ONCE, similar tobtf_resolve_helper_id?Quoted 10 lines
hmm. Right. There is a race with
bpf_trampoline_lookup. One thread could have just created the trampoline and still doing distill, while another thread is trying to use it after getting it frombpf_trampoline_lookup. The fix choices are not pretty. Either to add a mutex tocheck_attach_btf_id()or dobpf_trampoline_lookup_or_create()with extra callback that doesbtf_distill_func_protowhilebpf_trampoline_lookup_or_createis holdingtrampoline_mutexor move most of thecheck_attach_btf_id()logic intobpf_trampoline_lookup_or_create().
I tried to keep trampoline as abstract concept, but with callback or move the verifer and btf logic will bleed into trampoline. Hmm.
Quoted 3 lines
yeah, that sounds too intrusive. I'd change
btf_distill_func_prototo accept structbtf_func_model**m, allocate model dynamically, and thencompare_exchangethe final constructed model pointer.
cmpxchg is too ugly and also not covering all other fields that may need to have serialized access in the future. I went with simpler model of additional mutex per trampoline. It also helped to avoid global mutex for link/unlink.
Alexei Starovoitov
maintainerPATCH v3 16/18 libbpf: Add support for attaching BPF programs to other BPF programs
In reply to Alexei StarovoitovExtend libbpf api to pass attach_prog_fd into bpf_object__open.
--- tools/include/uapi/linux/bpf.h | 1 + tools/lib/bpf/bpf.c | 9 +++-- tools/lib/bpf/bpf.h | 5 ++- tools/lib/bpf/libbpf.c | 65 +++++++++++++++++++++++++++++----- tools/lib/bpf/libbpf.h | 3 +- 5 files changed, 69 insertions(+), 14 deletions(-)
tools/include/uapi/linux/bpf.h +1 -0
@@ -425,6 +425,7 @@ union bpf_attr {__aligned_u64 line_info;/* line info */ __u32 line_info_cnt;/* number of bpf_line_info records */ __u32 attach_btf_id;/* in-kernel BTF type id to attach to */ +__u32 attach_prog_fd;/* 0 to attach to vmlinux */ };struct {/* anonymous struct used by BPF_OBJ_* commands */
tools/lib/bpf/bpf.c +6 -3
@@ -228,10 +228,14 @@ int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr,memset (&attr,0 ,sizeof (attr)); attr.prog_type = load_attr->prog_type; attr.expected_attach_type = load_attr->expected_attach_type; -if (attr.prog_type ==BPF_PROG_TYPE_TRACING ) +if (attr.prog_type ==BPF_PROG_TYPE_TRACING ) { attr.attach_btf_id = load_attr->attach_btf_id; -else +if (load_attr->attach_prog_fd) + attr.attach_prog_fd = load_attr->attach_prog_fd; + }else { attr.prog_ifindex = load_attr->prog_ifindex; + attr.kern_version = load_attr->kern_version; + } attr.insn_cnt = (__u32 )load_attr->insns_cnt; attr.insns =ptr_to_u64 (load_attr->insns); attr.license =ptr_to_u64 (load_attr->license);
@@ -245,7 +249,6 @@ int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr, attr.log_size =0 ; } - attr.kern_version = load_attr->kern_version; attr.prog_btf_fd = load_attr->prog_btf_fd; attr.func_info_rec_size = load_attr->func_info_rec_size; attr.func_info_cnt = load_attr->func_info_cnt;
tools/lib/bpf/bpf.h +4 -1
@@ -77,7 +77,10 @@ struct bpf_load_program_attr {const struct bpf_insn *insns;size_t insns_cnt;const char *license; -__u32 kern_version; +union { +__u32 kern_version; +__u32 attach_prog_fd; + };union {__u32 prog_ifindex;__u32 attach_btf_id;
tools/lib/bpf/libbpf.c +56 -9
123 lines not on this page · raw message
tools/lib/bpf/libbpf.h +2 -1
@@ -108,8 +108,9 @@ struct bpf_object_open_opts {* auto-pinned to that path on load; defaults to "/sys/fs/bpf". */ const char *pin_root_path; +__u32 attach_prog_fd; }; -#define bpf_object_open_opts__last_field pin_root_path +#define bpf_object_open_opts__last_field attach_prog_fd LIBBPF_API struct bpf_object *bpf_object__open(const char *path); LIBBPF_API struct bpf_object *
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 16/18] libbpf: Add support for attaching BPF programs to other BPF programs
Quoted 8 lines · Alexei Starovoitov
Extend libbpf api to pass
attach_prog_fdintobpf_object__open.Signed-off-by Alexei Starovoitov <[email protected]>---
[...]
Quoted 17 lines
tools/lib/bpf/libbpf.c +17
-0+static int libbpf_find_prog_btf_id (const char *name,__u32 attach_prog_fd) +{ +struct bpf_prog_info_linear *info_linear; +struct bpf_prog_info *info; +struct btf *btf =NULL ; +int err = -EINVAL ; + + info_linear =bpf_program__get_prog_info_linear (attach_prog_fd,0 ); +if (IS_ERR_OR_NULL (info_linear)) { +pr_warn ("failed get_prog_info_linear for FD %d\n" , + attach_prog_fd); +return -EINVAL ; + } + info = &info_linear->info; +if (!info->btf_id) { +pr_warn ("The target program doesn't have BTF\n" ); + goto out;
tools/lib/bpf/libbpf.c int libbpf_find_vmlinux_btf_id(const char *name, · line 5117
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+ } +if (btf__get_from_id (info->btf_id, &btf)) { +pr_warn ("Failed to get BTF of the program\n" ); + goto out; + } + err =btf__find_by_name_kind (btf, name,BTF_KIND_FUNC ); +btf__free (btf); +if (err <=0 ) { +pr_warn ("%s is not found in prog's BTF\n" , name); + goto out;
^^^ This goto doesn't really do much.
tools/lib/bpf/libbpf.c int libbpf_find_vmlinux_btf_id(const char *name, · line 5122
Quoted 5 lines
tools/lib/bpf/libbpf.c +5
-0+ } +out: +free (info_linear); +return err; +}
Otherwise
Re: [PATCH v3 bpf-next 16/18] libbpf: Add support for attaching BPF programs to other BPF programs
Quoted 31 lines · Song Liu
Quoted 8 lines · Alexei Starovoitov
Extend libbpf api to pass
attach_prog_fdintobpf_object__open.Signed-off-by Alexei Starovoitov <[email protected]>---
[...]
Quoted 20 lines
tools/lib/bpf/libbpf.c +20
-0+static int libbpf_find_prog_btf_id (const char *name,__u32 attach_prog_fd) +{ +struct bpf_prog_info_linear *info_linear; +struct bpf_prog_info *info; +struct btf *btf =NULL ; +int err = -EINVAL ; + + info_linear =bpf_program__get_prog_info_linear (attach_prog_fd,0 ); +if (IS_ERR_OR_NULL (info_linear)) { +pr_warn ("failed get_prog_info_linear for FD %d\n" , + attach_prog_fd); +return -EINVAL ; + } + info = &info_linear->info; +if (!info->btf_id) { +pr_warn ("The target program doesn't have BTF\n" ); + goto out; + } +if (btf__get_from_id (info->btf_id, &btf)) { +pr_warn ("Failed to get BTF of the program\n" );
Quoted 8 lines
tools/lib/bpf/libbpf.c int libbpf_find_vmlinux_btf_id(const char *name, · line 5117
Quoted 7 lines
tools/lib/bpf/libbpf.c +7
-0+ goto out; + } + err =btf__find_by_name_kind (btf, name,BTF_KIND_FUNC ); +btf__free (btf); +if (err <=0 ) { +pr_warn ("%s is not found in prog's BTF\n" , name); + goto out;^^^ This goto doesn't really do much.
yeah. it does look a bit weird.
I wanted to keep uniform error handling, but can remove it if you insist.
Quoted 10 lines
tools/lib/bpf/libbpf.c int libbpf_find_vmlinux_btf_id(const char *name, · line 5122
Quoted 5 lines
tools/lib/bpf/libbpf.c +5
-0+ } +out: +free (info_linear); +return err; +}Otherwise
Acked-by Song Liu <[email protected]>
Re: [PATCH v3 bpf-next 16/18] libbpf: Add support for attaching BPF programs to other BPF programs
Quoted 45 lines · Alexei Starovoitov
Quoted 31 lines · Song Liu
Quoted 8 lines · Alexei Starovoitov
Extend libbpf api to pass
attach_prog_fdintobpf_object__open.Signed-off-by Alexei Starovoitov <[email protected]>---
[...]
Quoted 20 lines
tools/lib/bpf/libbpf.c +20
-0+static int libbpf_find_prog_btf_id (const char *name,__u32 attach_prog_fd) +{ +struct bpf_prog_info_linear *info_linear; +struct bpf_prog_info *info; +struct btf *btf =NULL ; +int err = -EINVAL ; + + info_linear =bpf_program__get_prog_info_linear (attach_prog_fd,0 ); +if (IS_ERR_OR_NULL (info_linear)) { +pr_warn ("failed get_prog_info_linear for FD %d\n" , + attach_prog_fd); +return -EINVAL ; + } + info = &info_linear->info; +if (!info->btf_id) { +pr_warn ("The target program doesn't have BTF\n" ); + goto out; + } +if (btf__get_from_id (info->btf_id, &btf)) { +pr_warn ("Failed to get BTF of the program\n" );Quoted 8 lines
tools/lib/bpf/libbpf.c int libbpf_find_vmlinux_btf_id(const char *name, · line 5117
Quoted 7 lines
tools/lib/bpf/libbpf.c +7
-0+ goto out; + } + err =btf__find_by_name_kind (btf, name,BTF_KIND_FUNC ); +btf__free (btf); +if (err <=0 ) { +pr_warn ("%s is not found in prog's BTF\n" , name); + goto out;^^^ This goto doesn't really do much.
Quoted 3 lines
yeah. it does look a bit weird.
I wanted to keep uniform error handling, but can remove it if you insist.
I think it is good as-is.
Thanks,
Song
Re: [PATCH v3 bpf-next 16/18] libbpf: Add support for attaching BPF programs to other BPF programs
In reply to Alexei StarovoitovQuoted 29 lines · Alexei Starovoitov
Extend libbpf api to pass
attach_prog_fdintobpf_object__open.Signed-off-by Alexei Starovoitov <[email protected]>--- tools/include/uapi/linux/bpf.h | 1 + tools/lib/bpf/bpf.c | 9 +++-- tools/lib/bpf/bpf.h | 5 ++- tools/lib/bpf/libbpf.c | 65 +++++++++++++++++++++++++++++----- tools/lib/bpf/libbpf.h | 3 +- 5 files changed, 69 insertions(+), 14 deletions(-)tools/include/uapi/linux/bpf.h +1
-0@@ -425,6 +425,7 @@ union bpf_attr {__aligned_u64 line_info;/* line info */ __u32 line_info_cnt;/* number of bpf_line_info records */ __u32 attach_btf_id;/* in-kernel BTF type id to attach to */ +__u32 attach_prog_fd;/* 0 to attach to vmlinux */ };struct {/* anonymous struct used by BPF_OBJ_* commands */ tools/lib/bpf/bpf.c +0
-0
tools/lib/bpf/bpf.c int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr, · line 234
Quoted 9 lines
tools/lib/bpf/bpf.c +3
-2@@ -228,10 +228,14 @@ int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr,memset (&attr,0 ,sizeof (attr)); attr.prog_type = load_attr->prog_type; attr.expected_attach_type = load_attr->expected_attach_type; -if (attr.prog_type ==BPF_PROG_TYPE_TRACING ) +if (attr.prog_type ==BPF_PROG_TYPE_TRACING ) { attr.attach_btf_id = load_attr->attach_btf_id; -else +if (load_attr->attach_prog_fd) + attr.attach_prog_fd = load_attr->attach_prog_fd;
why the if? if it's zero, attr.attach_prog_fd will stay zero.
Quoted 7 lines
tools/lib/bpf/bpf.c +3
-0+ }else { attr.prog_ifindex = load_attr->prog_ifindex; + attr.kern_version = load_attr->kern_version; + } attr.insn_cnt = (__u32 )load_attr->insns_cnt; attr.insns =ptr_to_u64 (load_attr->insns); attr.license =ptr_to_u64 (load_attr->license);
[...]
tools/lib/bpf/bpf.c __bpf_object__open(const char *path, const void *obj_buf, size_t obj_buf_sz, · line 3938
Quoted 9 lines
CHECK_ERR(bpf_object__elf_init(obj), err, out); CHECK_ERR(bpf_object__check_endianness(obj), err, out);tools/lib/bpf/bpf.c +2
-1@@ -3927,10 +3934,12 @@ __bpf_object__open(const char *path, const void *obj_buf, size_t obj_buf_sz,bpf_program__set_expected_attach_type (prog, attach_type);if (prog_type ==BPF_PROG_TYPE_TRACING ) { err =libbpf_attach_btf_id_by_name (prog->section_name, - attach_type); + attach_type, + attach_prog_fd);
libbpf_find_attach_btf_id seems like a better name at this point
Quoted 7 lines
tools/lib/bpf/bpf.c +1
-0if (err <=0 ) goto out; prog->attach_btf_id = err; + prog->attach_prog_fd = attach_prog_fd; } }
[...]
Alexei Starovoitov
maintainerPATCH v3 17/18 selftests/bpf: Extend test_pkt_access test
In reply to Alexei StarovoitovThe test_pkt_access.o is used by multiple tests. Fix its section name so that program type can be automatically detected by libbpf and make it call other subprograms with skb argument.
--- .../selftests/bpf/progs/test_pkt_access.c | 38 ++++++++++++++++++- 1 file changed, 36 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/progs/test_pkt_access.c +36 -2
@@ -17,8 +17,38 @@#define barrier () __asm__ __volatile__("": : :"memory") int _version SEC("version") = 1; -SEC("test1") -int process (struct __sk_buff *skb) +/* llvm will optimize both subprograms into exactly the same BPF assembly +* +* Disassembly of section .text: +* +* 0000000000000000 test_pkt_access_subprog1: +* ; return skb->len * 2; +* 0: 61 10 00 00 00 00 00 00 r0 = *(u32 *)(r1 + 0) +* 1: 64 00 00 00 01 00 00 00 w0 <<= 1 +* 2: 95 00 00 00 00 00 00 00 exit +* +* 0000000000000018 test_pkt_access_subprog2: +* ; return skb->len * val; +* 3: 61 10 00 00 00 00 00 00 r0 = *(u32 *)(r1 + 0) +* 4: 64 00 00 00 01 00 00 00 w0 <<= 1 +* 5: 95 00 00 00 00 00 00 00 exit +* +* Which makes it an interesting test for BTF-enabled verifier. +*/ +static __attribute__ ((noinline)) +int test_pkt_access_subprog1 (volatile struct __sk_buff *skb) +{ +return skb->len *2 ; +} + +static __attribute__ ((noinline)) +int test_pkt_access_subprog2 (int val,volatile struct __sk_buff *skb) +{ +return skb->len * val; +} + +SEC("classifier/test_pkt_access") +int test_pkt_access (struct __sk_buff *skb) {void *data_end = (void *)(long )skb->data_end;void *data = (void *)(long )skb->data;
@@ -48,6 +78,10 @@ int process(struct __sk_buff *skb) tcp = (struct tcphdr *)((void *)(ip6h) + ihl_len); } +if (test_pkt_access_subprog1 (skb) != skb->len *2 ) +return TC_ACT_SHOT ; +if (test_pkt_access_subprog2 (2 , skb) != skb->len *2 ) +return TC_ACT_SHOT ;if (tcp) {if (((void *)(tcp) +20 ) > data_end || proto !=6 )return TC_ACT_SHOT ;
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 17/18] selftests/bpf: Extend test_pkt_access test
Quoted 9 lines · Alexei Starovoitov
The test_pkt_access.o is used by multiple tests. Fix its section name so that program type can be automatically detected by libbpf and make it call other subprograms with skb argument.
Signed-off-by Alexei Starovoitov <[email protected]>
Quoted 21 lines
---
.../selftests/bpf/progs/test_pkt_access.c| 38 ++++++++++++++++++-1 file changed, 36 insertions(+), 2 deletions(-)tools/testing/selftests/bpf/progs/test_pkt_access.c +7
-2@@ -17,8 +17,38 @@#define barrier () __asm__ __volatile__("": : :"memory") int _version SEC("version") = 1; -SEC("test1") -int process (struct __sk_buff *skb) +/* llvm will optimize both subprograms into exactly the same BPF assembly +* +* Disassembly of section .text: +* +* 0000000000000000 test_pkt_access_subprog1: +* ; return skb->len * 2; +* 0: 61 10 00 00 00 00 00 00 r0 = *(u32 *)(r1 + 0)
tools/testing/selftests/bpf/progs/test_pkt_access.c line 36
Quoted 10 lines
tools/testing/selftests/bpf/progs/test_pkt_access.c +10
-0+* 1: 64 00 00 00 01 00 00 00 w0 <<= 1 +* 2: 95 00 00 00 00 00 00 00 exit +* +* 0000000000000018 test_pkt_access_subprog2: +* ; return skb->len * val; +* 3: 61 10 00 00 00 00 00 00 r0 = *(u32 *)(r1 + 0) +* 4: 64 00 00 00 01 00 00 00 w0 <<= 1 +* 5: 95 00 00 00 00 00 00 00 exit +* +* Which makes it an interesting test for BTF-enabled verifier.
This is interesting!
Re: [PATCH v3 bpf-next 17/18] selftests/bpf: Extend test_pkt_access test
In reply to Alexei StarovoitovQuoted 7 lines · Alexei Starovoitov
The test_pkt_access.o is used by multiple tests. Fix its section name so that program type can be automatically detected by libbpf and make it call other subprograms with skb argument.
Signed-off-by Alexei Starovoitov <[email protected]>---
Quoted 3 lines
.../selftests/bpf/progs/test_pkt_access.c | 38 ++++++++++++++++++- 1 file changed, 36 insertions(+), 2 deletions(-)
[...]
Alexei Starovoitov
maintainerPATCH v3 18/18 selftests/bpf: Add a test for attaching BPF prog to another BPF prog and subprog
In reply to Alexei StarovoitovAdd a test that attaches one FEXIT program to main sched_cls networking program and two other FEXIT programs to subprograms. All three tracing programs access return values and skb->len of networking program and subprograms.
--- .../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 76 ++++++++++++++++ .../selftests/bpf/progs/fexit_bpf2bpf.c | 91 +++++++++++++++++++ 2 files changed, 167 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c create mode 100644 tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c
tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c new file +76 -0
@@ -0,0 +1,76 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019 Facebook */ +#include <test_progs.h> + +#define PROG_CNT 3 + +void test_fexit_bpf2bpf (void ) +{ +const char *prog_name[PROG_CNT ] = { +"fexit/test_pkt_access" , +"fexit/test_pkt_access_subprog1" , +"fexit/test_pkt_access_subprog2" , + }; +struct bpf_object *obj =NULL , *pkt_obj; +int err, pkt_fd, i; +struct bpf_link *link[PROG_CNT ] = {}; +struct bpf_program *prog[PROG_CNT ]; +__u32 duration, retval; +struct bpf_map *data_map; +const int zero =0 ; +u64 result[PROG_CNT ]; + + err =bpf_prog_load ("./test_pkt_access.o" ,BPF_PROG_TYPE_UNSPEC , + &pkt_obj, &pkt_fd); +if (CHECK (err,"prog_load sched cls" ,"err %d errno %d\n" , err, errno)) +return ; +DECLARE_LIBBPF_OPTS (bpf_object_open_opts, opts, + .attach_prog_fd = pkt_fd, + ); + + obj =bpf_object__open_file ("./fexit_bpf2bpf.o" , &opts); +if (CHECK (IS_ERR_OR_NULL (obj),"obj_open" , +"failed to open fexit_bpf2bpf: %ld\n" , +PTR_ERR (obj))) + goto close_prog; + + err =bpf_object__load (obj); +if (CHECK (err,"obj_load" ,"err %d\n" , err)) + goto close_prog; + +for (i =0 ; i <PROG_CNT ; i++) { + prog[i] =bpf_object__find_program_by_title (obj, prog_name[i]); +if (CHECK (!prog[i],"find_prog" ,"prog %s not found\n" , prog_name[i])) + goto close_prog; + link[i] =bpf_program__attach_trace (prog[i]); +if (CHECK (IS_ERR (link[i]),"attach_trace" ,"failed to link\n" )) + goto close_prog; + } + data_map =bpf_object__find_map_by_name (obj,"fexit_bp.bss" ); +if (CHECK (!data_map,"find_data_map" ,"data map not found\n" )) + goto close_prog; + + err =bpf_prog_test_run (pkt_fd,1 , &pkt_v6,sizeof (pkt_v6), +NULL ,NULL , &retval, &duration); +CHECK (err || retval,"ipv6" , +"err %d errno %d retval %d duration %d\n" , + err, errno, retval, duration); + + err =bpf_map_lookup_elem (bpf_map__fd (data_map), &zero, &result); +if (CHECK (err,"get_result" , +"failed to get output data: %d\n" , err)) + goto close_prog; + +for (i =0 ; i <PROG_CNT ; i++) +if (CHECK (result[i] !=1 ,"result" ,"fexit_bpf2bpf failed err %ld\n" , + result[i])) + goto close_prog; + +close_prog: +for (i =0 ; i <PROG_CNT ; i++) +if (!IS_ERR_OR_NULL (link[i])) +bpf_link__destroy (link[i]); +if (!IS_ERR_OR_NULL (obj)) +bpf_object__close (obj); +bpf_object__close (pkt_obj); +}
tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c new file +91 -0
91 lines not on this page · raw message
Signature · 1 line
2.23.0
Re: [PATCH v3 bpf-next 18/18] selftests/bpf: Add a test for attaching BPF prog to another BPF prog and subprog
Quoted 9 lines · Alexei Starovoitov
Add a test that attaches one FEXIT program to main
sched_clsnetworking program and two other FEXIT programs to subprograms. All three tracing programs access return values andskb->lenof networking program and subprograms.Signed-off-by Alexei Starovoitov <[email protected]>
Re: [PATCH v3 bpf-next 18/18] selftests/bpf: Add a test for attaching BPF prog to another BPF prog and subprog
In reply to Alexei StarovoitovQuoted 7 lines · Alexei Starovoitov
Add a test that attaches one FEXIT program to main
sched_clsnetworking program and two other FEXIT programs to subprograms. All three tracing programs access return values andskb->lenof networking program and subprograms.Signed-off-by Alexei Starovoitov <[email protected]>---
Quoted 6 lines
.../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 76 ++++++++++++++++ .../selftests/bpf/progs/fexit_bpf2bpf.c | 91 +++++++++++++++++++ 2 files changed, 167 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c create mode 100644 tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c
[...]
tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c line 77
Quoted 9 lines
tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c new file +9
-0+SEC("fexit/test_pkt_access_subprog2") +int test_subprog2 (struct args_subprog2 *ctx) +{ +struct sk_buff *skb = (void *)ctx->args[0 ]; +__u64 ret; +int len; + +bpf_probe_read (&len,sizeof (len), +__builtin_preserve_access_index (&skb->len));
nit: we have bpf_core_read() for this, but I suspect you may have wanted __builtin spelled out explicitly
Quoted 17 lines
tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c new file +14
-1+ + ret = ctx->ret; +/* bpf_prog_load() loads "test_pkt_access.o" with BPF_F_TEST_RND_HI32 +* which randomizes upper 32 bits after BPF_ALU32 insns. +* Hence after 'w0 <<= 1' upper bits of $rax are random. +* That is expected and correct. Trim them. +*/ + ret = (__u32 ) ret; +if (len !=74 || ret !=148 ) +return 0 ; + test_result_subprog2 =1 ; +return 0 ; +} +char _license[] SEC("license") = "GPL"; --2.23.0
Re: [PATCH v3 bpf-next 18/18] selftests/bpf: Add a test for attaching BPF prog to another BPF prog and subprog
Quoted 20 lines · Andrii Nakryiko
Quoted 7 lines · Alexei Starovoitov
Add a test that attaches one FEXIT program to main
sched_clsnetworking program and two other FEXIT programs to subprograms. All three tracing programs access return values andskb->lenof networking program and subprograms.Signed-off-by Alexei Starovoitov <[email protected]>---
Acked-by Andrii Nakryiko <[email protected]>Quoted 6 lines
.../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 76 ++++++++++++++++ .../selftests/bpf/progs/fexit_bpf2bpf.c | 91 +++++++++++++++++++ 2 files changed, 167 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c create mode 100644 tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c[...]
Quoted 12 lines
tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c line 77
Quoted 9 lines
tools/testing/selftests/bpf/progs/fexit_bpf2bpf.c new file +9
-0+SEC("fexit/test_pkt_access_subprog2") +int test_subprog2 (struct args_subprog2 *ctx) +{ +struct sk_buff *skb = (void *)ctx->args[0 ]; +__u64 ret; +int len; + +bpf_probe_read (&len,sizeof (len), +__builtin_preserve_access_index (&skb->len));nit: we have
bpf_core_read()for this, but I suspect you may have wanted__builtinspelled out explicitly
exactly. it should have been bpf_probe_read_kernel though. will fix.