bpf: Support private stack for bpf progs
Subsystems (6)
Yonghong Song
reviewerPATCH 1/2 bpf: Support private stack for bpf progs
The main motivation for private stack comes from nested scheduler in sched-ext from Tejun. The basic idea is that
- each cgroup will its own associated bpf program,
- bpf program with parent cgroup will call bpf programs in immediate child cgroups.
Let us say we have the following cgroup hierarchy:
root_cg (prog0):
cg1 (prog1):
cg11 (prog11):
cg111 (prog111)
cg112 (prog112)
cg12 (prog12):
cg121 (prog121)
cg122 (prog122)
cg2 (prog2):
cg21 (prog21)
cg22 (prog22)
cg23 (prog23)In the above example, prog0 will call a kfunc which will call prog1 and prog2 to get sched info for cg1 and cg2 and then the information is summarized and sent back to prog0. Similarly, prog11 and prog12 will be invoked in the kfunc and the result will be summarized and sent back to prog1, etc.
Currently, for each thread, the x86 kernel allocate 8KB stack. The each bpf program (including its subprograms) has maximum 512B stack size to avoid potential stack overflow. And nested bpf programs increase the risk of stack overflow. To avoid potential stack overflow caused by bpf programs, this patch implemented a private stack so bpf program stack space is allocated dynamically when the program is jited. Such private stack is applied to tracing programs like kprobe/uprobe, perf_event, tracepoint, raw tracepoint and tracing.
But more than one instance of the same bpf program may run in the system. To make things simple, percpu private stack is allocated for each program, so if the same program is running on different cpus concurrently, we won't have any issue. Note that the kernel already have logic to prevent the recursion for the same bpf program on the same cpu (kprobe, fentry, etc.).
The patch implemented a percpu private stack based approach for x86 arch.
- The stack size will be 0 and any stack access is from jit-time allocated percpu storage.
- In the beginning of jit, r9 is used to save percpu private stack pointer.
- Each rbp in the bpf asm insn is replaced by r9.
- For each call, push r9 before the call and pop r9 after the call to preserve r9 value.
Compared to previous RFC patch [1], this patch added some conditions to enable private stack, e.g., verifier calculated stack size, prog type, etc. The new patch also added a performance test to compare private stack vs. no private stack.
The following are some code example to illustrate the idea for selftest cgroup_skb_sk_lookup:
the existing code the private-stack approach code
endbr64 endbr64
nop DWORD PTR [rax+rax*1+0x0] nop DWORD PTR [rax+rax*1+0x0]
xchg ax,ax xchg ax,ax
push rbp push rbp
mov rbp,rsp mov rbp,rsp
endbr64 endbr64
sub rsp,0x68
push rbx push rbx
... ...
... mov r9d,0x8c1c860
... add r9,QWORD PTR gs:0x21a00
... ...
mov rdx,rbp mov rdx, r9
add rdx,0xffffffffffffffb4 rdx,0xffffffffffffffb4
... ...
mov ecx,0x28 mov ecx,0x28
push r9
call 0xffffffffe305e474 call 0xffffffffe305e524
pop r9
mov rdi,rax mov rdi,rax
... ...
movzx rdi,BYTE PTR [rbp-0x46] movzx rdi,BYTE PTR [r9-0x46]
... ...So the number of insns is increased by 1 + num_of_calls * 2. Here the number of calls are those calls in the final jited binary. Comparing function call itself, the push/pop overhead should be minimum in most common cases.
Our original use case is for sched-ext nested scheduler. This will be done in the future.
[1] https://lore.kernel.org/bpf/[email protected]/T/
--- arch/x86/net/bpf_jit_comp.c | 63 ++++++++++++++++++++++++++++++++++--- include/linux/bpf.h | 2 ++ kernel/bpf/core.c | 20 ++++++++++++ kernel/bpf/syscall.c | 1 + 4 files changed, 82 insertions(+), 4 deletions(-)
arch/x86/net/bpf_jit_comp.c +59 -4
@@ -1309,6 +1309,22 @@ static void emit_shiftx(u8 **pprog, u32 dst_reg, u8 src_reg, bool is64, u8 op) *pprog = prog; } +static void emit_private_frame_ptr (u8 **pprog,void *private_frame_ptr) +{ +u8 *prog = *pprog; + +/* movabs r9, private_frame_ptr */ +emit_mov_imm64 (&prog,X86_REG_R9 , (long ) private_frame_ptr >>32 , + (u32 ) (long ) private_frame_ptr); + +/* add <r9>, gs:[<off>] */ +EMIT2 (0x65 ,0x4c ); +EMIT3 (0x03 ,0x0c ,0x25 ); +EMIT ((u32 )(unsigned long )&this_cpu_off,4 ); + + *pprog = prog; +} +#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (prog - temp)))/* mov rax, qword ptr [rbp - rounded_stack_depth - 8] */
@@ -1324,18 +1340,25 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageint insn_cnt = bpf_prog->len;bool seen_exit = false;u8 temp[BPF_MAX_INSN_SIZE +BPF_INSN_SAFETY ]; +u32 stack_depth = bpf_prog->aux->stack_depth; +void __percpu *private_frame_ptr =NULL ;u64 arena_vm_start, user_vm_start;int i, excnt =0 ;int ilen, proglen =0 ;u8 *prog = temp;int err; +if (bpf_prog->private_stack_ptr) { + private_frame_ptr = bpf_prog->private_stack_ptr +round_up (stack_depth,8 ); + stack_depth =0 ; + } + arena_vm_start =bpf_arena_get_kern_vm_start (bpf_prog->aux->arena); user_vm_start =bpf_arena_get_user_vm_start (bpf_prog->aux->arena);detect_reg_usage (insn, insn_cnt, callee_regs_used); -emit_prologue (&prog, bpf_prog->aux->stack_depth, +emit_prologue (&prog, stack_depth,bpf_prog_was_classic (bpf_prog), tail_call_reachable,bpf_is_subprog (bpf_prog), bpf_prog->aux->exception_cb);/* Exception callback will clobber callee regs for its own use, and
@@ -1357,6 +1380,9 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageemit_mov_imm64 (&prog,X86_REG_R12 , arena_vm_start >>32 , (u32 ) arena_vm_start); +if (private_frame_ptr) +emit_private_frame_ptr (&prog, private_frame_ptr); + ilen = prog - temp;if (rw_image)memcpy (rw_image + proglen, temp, ilen);
@@ -1376,6 +1402,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageu8 *func;int nops; +if (private_frame_ptr) { +if (src_reg ==BPF_REG_FP ) + src_reg =X86_REG_R9 ; + +if (dst_reg ==BPF_REG_FP ) + dst_reg =X86_REG_R9 ; + } +switch (insn->code) {/* ALU */ case BPF_ALU |BPF_ADD |BPF_X :
@@ -2007,6 +2041,7 @@ st: if (is_imm8(insn->off))emit_mov_reg (&prog, is64, real_src_reg,BPF_REG_0 );/* Restore R0 after clobbering RAX */ emit_mov_reg (&prog, true,BPF_REG_0 ,BPF_REG_AX ); +break ; }
@@ -2031,14 +2066,20 @@ st: if (is_imm8(insn->off)) func = (u8 *) __bpf_call_base + imm32;if (tail_call_reachable) { -RESTORE_TAIL_CALL_CNT (bpf_prog->aux->stack_depth); +RESTORE_TAIL_CALL_CNT (stack_depth); ip +=7 ; }if (!imm32)return -EINVAL ; +if (private_frame_ptr) { +EMIT2 (0x41 ,0x51 );/* push r9 */ + ip +=2 ; + } ip +=x86_call_depth_emit_accounting (&prog, func, ip);if (emit_call (&prog, func, ip))return -EINVAL ; +if (private_frame_ptr) +EMIT2 (0x41 ,0x59 );/* pop r9 */ break ; }
@@ -2048,13 +2089,13 @@ st: if (is_imm8(insn->off)) &bpf_prog->aux->poke_tab[imm32 -1 ], &prog, image + addrs[i -1 ], callee_regs_used, - bpf_prog->aux->stack_depth, + stack_depth, ctx); else emit_bpf_tail_call_indirect(bpf_prog, &prog, callee_regs_used, - bpf_prog->aux->stack_depth, + stack_depth, image + addrs[i - 1], ctx);break ;
@@ -3218,6 +3259,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) {struct bpf_binary_header *rw_header =NULL ;struct bpf_binary_header *header =NULL ; +void __percpu *private_stack_ptr =NULL ;struct bpf_prog *tmp, *orig_prog = prog;struct x64_jit_data *jit_data;int proglen, oldproglen =0 ;
@@ -3284,6 +3326,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) ctx.cleanup_addr = proglen; skip_init_addrs: +if (bpf_enable_private_stack (prog) && !prog->private_stack_ptr) { + private_stack_ptr =__alloc_percpu_gfp (prog->aux->stack_depth,8 ,GFP_KERNEL ); +if (!private_stack_ptr) { + prog = orig_prog; + goto out_addrs; + } + prog->private_stack_ptr = private_stack_ptr; + } +/* * JITed image shrinks with every pass and the loop iterates * until the image stops shrinking. Very large BPF programs
@@ -3309,6 +3360,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog->jited =0 ; prog->jited_len =0 ; } +if (private_stack_ptr) { +free_percpu (private_stack_ptr); + prog->private_stack_ptr =NULL ; + } goto out_addrs; }if (image) {
include/linux/bpf.h +2 -0
@@ -1563,6 +1563,7 @@ struct bpf_prog { const struct bpf_insn *insn);struct bpf_prog_aux *aux;/* Auxiliary fields */ struct sock_fprog_kern *orig_prog;/* Original BPF program */ +void __percpu *private_stack_ptr;/* Instructions for interpreter */ union { DECLARE_FLEX_ARRAY(struct sock_filter, insns);
@@ -1819,6 +1820,7 @@ static inline void bpf_module_put(const void *data, struct module *owner)module_put (owner); }int bpf_struct_ops_link_create (union bpf_attr *attr); +bool bpf_enable_private_stack (struct bpf_prog *prog);#ifdef CONFIG_NET /* Define it here to avoid the use of forward declaration */
kernel/bpf/core.c +20 -0
@@ -2813,6 +2813,26 @@ void bpf_prog_free(struct bpf_prog *fp) }EXPORT_SYMBOL_GPL (bpf_prog_free); +bool bpf_enable_private_stack (struct bpf_prog *prog) +{ +if (prog->aux->stack_depth <=64 ) +return false; + +switch (prog->aux->prog->type) { +case BPF_PROG_TYPE_KPROBE : +case BPF_PROG_TYPE_TRACEPOINT : +case BPF_PROG_TYPE_PERF_EVENT : +case BPF_PROG_TYPE_RAW_TRACEPOINT : +return true; +case BPF_PROG_TYPE_TRACING : +if (prog->expected_attach_type !=BPF_TRACE_ITER ) +return true; + fallthrough; +default : +return false; + } +} +/* RNG for unprivileged user space with separated state from prandom_u32(). */ static DEFINE_PER_CPU(struct rnd_state, bpf_user_rnd_state);
kernel/bpf/syscall.c +1 -0
@@ -2244,6 +2244,7 @@ static void __bpf_prog_put_rcu(struct rcu_head *rcu)kvfree (aux->func_info);kfree (aux->func_info_aux); +free_percpu (aux->prog->private_stack_ptr);free_uid (aux->user);security_bpf_prog_free (aux->prog);bpf_prog_free (aux->prog);
Signature · 1 line
2.43.0
Yonghong Song
reviewerPATCH 2/2 selftests/bpf: Benchmark runtime performance with private stack
This patch intends to show some benchmark results comparing a bpf program with vs. without private stack. The patch is not intended to land since it hacks existing kernel interface in order to do proper comparison.
The following is the jited code for bpf prog in progs/private_stack.c without private stack. The number of batch iterations is 4096.
0: f3 0f 1e fa endbr64 4: 0f 1f 44 00 00 nop DWORD PTR [rax+rax*1+0x0] 9: 66 90 xchg ax,ax b: 55 push rbp c: 48 89 e5 mov rbp,rsp f: f3 0f 1e fa endbr64 13: 48 81 ec 60 00 00 00 sub rsp,0x60 1a: 53 push rbx 1b: 41 55 push r13 1d: 48 bf 00 50 5d 00 00 movabs rdi,0xffffc900005d5000 24: c9 ff ff 27: 48 8b 77 00 mov rsi,QWORD PTR [rdi+0x0] 2b: 48 83 c6 01 add rsi,0x1 2f: 48 89 77 00 mov QWORD PTR [rdi+0x0],rsi 33: 31 ff xor edi,edi 35: 48 89 7d f8 mov QWORD PTR [rbp-0x8],rdi 39: be 05 00 00 00 mov esi,0x5 3e: 89 75 f8 mov DWORD PTR [rbp-0x8],esi 41: 48 89 7d f0 mov QWORD PTR [rbp-0x10],rdi 45: 48 89 7d e8 mov QWORD PTR [rbp-0x18],rdi 49: 48 89 7d e0 mov QWORD PTR [rbp-0x20],rdi 4d: 48 89 7d d8 mov QWORD PTR [rbp-0x28],rdi 51: 48 89 7d d0 mov QWORD PTR [rbp-0x30],rdi 55: 48 89 7d c0 mov QWORD PTR [rbp-0x40],rdi 59: 48 89 7d c8 mov QWORD PTR [rbp-0x38],rdi 5d: 48 89 7d b8 mov QWORD PTR [rbp-0x48],rdi 61: 48 89 7d b0 mov QWORD PTR [rbp-0x50],rdi 65: 48 89 7d a8 mov QWORD PTR [rbp-0x58],rdi 69: 48 89 7d a0 mov QWORD PTR [rbp-0x60],rdi 6d: bf 0a 00 00 00 mov edi,0xa 72: 89 7d c0 mov DWORD PTR [rbp-0x40],edi 75: 48 bb 00 80 5d 00 00 movabs rbx,0xffffc900005d8000 7c: c9 ff ff 7f: 8b 7b 00 mov edi,DWORD PTR [rbx+0x0] 82: 83 ff 01 cmp edi,0x1 85: 7c 27 jl 0xae 87: 45 31 ed xor r13d,r13d 8a: eb 29 jmp 0xb5 8c: 48 89 ee mov rsi,rbp 8f: 48 83 c6 d0 add rsi,0xffffffffffffffd0 93: 48 bf 00 38 f9 09 81 movabs rdi,0xffff888109f93800 9a: 88 ff ff 9d: e8 e2 e1 5e e1 call 0xffffffffe15ee284 a2: 41 83 c5 01 add r13d,0x1 a6: 8b 7b 00 mov edi,DWORD PTR [rbx+0x0] a9: 41 39 fd cmp r13d,edi ac: 7c 07 jl 0xb5 ae: 31 c0 xor eax,eax b0: 41 5d pop r13 b2: 5b pop rbx b3: c9 leave b4: c3 ret b5: 48 89 ee mov rsi,rbp b8: 48 83 c6 d0 add rsi,0xffffffffffffffd0 bc: 48 bf 00 38 f9 09 81 movabs rdi,0xffff888109f93800
c3: 88 ff ff
c6: e8 49 1f 5f e1 call 0xffffffffe15f2014
cb: 48 85 c0 test rax,rax
ce: 74 04 je 0xd4
d0: 48 83 c0 60 add rax,0x60
d4: 48 85 c0 test rax,rax
d7: 75 b3 jne 0x8c
d9: 48 89 ee mov rsi,rbp
dc: 48 83 c6 d0 add rsi,0xffffffffffffffd0
e0: 48 89 ea mov rdx,rbp
e3: 48 83 c2 a0 add rdx,0xffffffffffffffa0
e7: 48 bf 00 38 f9 09 81 movabs rdi,0xffff888109f93800
ee: 88 ff ff
f1: 31 c9 xor ecx,ecx
f3: e8 dc d8 5e e1 call 0xffffffffe15ed9d4
f8: eb 92 jmp 0x8c
The following is the corresponding jited code with private stack:
0: f3 0f 1e fa endbr64 4: 0f 1f 44 00 00 nop DWORD PTR [rax+rax*1+0x0] 9: 66 90 xchg ax,ax b: 55 push rbp c: 48 89 e5 mov rbp,rsp f: f3 0f 1e fa endbr64 13: 53 push rbx 14: 41 55 push r13 16: 49 b9 c0 a8 c1 08 7e movabs r9,0x607e08c1a8c0
1d: 60 00 00
20: 65 4c 03 0c 25 00 1a add r9,QWORD PTR gs:0x21a00
27: 02 00
29: 48 bf 00 80 61 00 00 movabs rdi,0xffffc90000618000
30: c9 ff ff
33: 48 8b 77 00 mov rsi,QWORD PTR [rdi+0x0] 37: 48 83 c6 01 add rsi,0x1 3b: 48 89 77 00 mov QWORD PTR [rdi+0x0],rsi 3f: 31 ff xor edi,edi 41: 49 89 79 f8 mov QWORD PTR [r9-0x8],rdi 45: be 05 00 00 00 mov esi,0x5 4a: 41 89 71 f8 mov DWORD PTR [r9-0x8],esi 4e: 49 89 79 f0 mov QWORD PTR [r9-0x10],rdi 52: 49 89 79 e8 mov QWORD PTR [r9-0x18],rdi 56: 49 89 79 e0 mov QWORD PTR [r9-0x20],rdi 5a: 49 89 79 d8 mov QWORD PTR [r9-0x28],rdi 5e: 49 89 79 d0 mov QWORD PTR [r9-0x30],rdi 62: 49 89 79 c0 mov QWORD PTR [r9-0x40],rdi 66: 49 89 79 c8 mov QWORD PTR [r9-0x38],rdi 6a: 49 89 79 b8 mov QWORD PTR [r9-0x48],rdi 6e: 49 89 79 b0 mov QWORD PTR [r9-0x50],rdi 72: 49 89 79 a8 mov QWORD PTR [r9-0x58],rdi 76: 49 89 79 a0 mov QWORD PTR [r9-0x60],rdi 7a: bf 0a 00 00 00 mov edi,0xa 7f: 41 89 79 c0 mov DWORD PTR [r9-0x40],edi 83: 48 bb 00 b0 61 00 00 movabs rbx,0xffffc9000061b000 8a: c9 ff ff 8d: 8b 7b 00 mov edi,DWORD PTR [rbx+0x0] 90: 83 ff 01 cmp edi,0x1 93: 7c 2b jl 0xc0 95: 45 31 ed xor r13d,r13d 98: eb 2d jmp 0xc7 9a: 4c 89 ce mov rsi,r9 9d: 48 83 c6 d0 add rsi,0xffffffffffffffd0 a1: 48 bf 00 40 f9 09 81 movabs rdi,0xffff888109f94000
a8: 88 ff ff
ab: 41 51 push r9
ad: e8 fa e1 5e e1 call 0xffffffffe15ee2ac
b2: 41 59 pop r9
b4: 41 83 c5 01 add r13d,0x1
b8: 8b 7b 00 mov edi,DWORD PTR [rbx+0x0]
bb: 41 39 fd cmp r13d,edi
be: 7c 07 jl 0xc7
c0: 31 c0 xor eax,eax
c2: 41 5d pop r13
c4: 5b pop rbx
c5: c9 leave
c6: c3 ret
c7: 4c 89 ce mov rsi,r9
ca: 48 83 c6 d0 add rsi,0xffffffffffffffd0
ce: 48 bf 00 40 f9 09 81 movabs rdi,0xffff888109f94000
d5: 88 ff ff
d8: 41 51 push r9
da: e8 5d 1f 5f e1 call 0xffffffffe15f203c
df: 41 59 pop r9
e1: 48 85 c0 test rax,rax
e4: 74 04 je 0xea
e6: 48 83 c0 60 add rax,0x60
ea: 48 85 c0 test rax,rax
ed: 75 ab jne 0x9a
ef: 4c 89 ce mov rsi,r9
f2: 48 83 c6 d0 add rsi,0xffffffffffffffd0
f6: 4c 89 ca mov rdx,r9
f9: 48 83 c2 a0 add rdx,0xffffffffffffffa0
fd: 48 bf 00 40 f9 09 81 movabs rdi,0xffff888109f94000
104: 88 ff ff 107: 31 c9 xor ecx,ecx 109: 41 51 push r9 10b: e8 ec d8 5e e1 call 0xffffffffe15ed9fc 110: 41 59 pop r9 112: eb 86 jmp 0x9a
It is clear that the main overhead is the push/pop r9 for three calls since those calls are in a loop. The initial r9 assignment
16: 49 b9 c0 a8 c1 08 7e movabs r9,0x607e08c1a8c0
1d: 60 00 00
20: 65 4c 03 0c 25 00 1a add r9,QWORD PTR gs:0x21a00
27: 02 00
is the overhead per prog run.
I did some benchmarking on an intel box (Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz) which has 20 cores and 80 cpus. Note that the number of hits are in the unit of loop iterations. More loop iterations per prog means more time will be spent with bpf programs.
I did four runs of tests. [no-]private-stack-[num-of-loop-iterations-per-prog] shows whether the run is with/without private stack and the number of loop iterations per program run. The number of hits equals to the total number of loop iterations in bpf prog.
The following are two of benchmark results:
$ ./benchs/run_bench_private_stack.sh no-private-stack-1: 2.771 ± 0.001M/s (drops 0.000 ± 0.000M/s) private-stack-1: 2.734 ± 0.031M/s (drops 0.000 ± 0.000M/s) no-private-stack-8: 4.613 ± 0.003M/s (drops 0.000 ± 0.000M/s) private-stack-8: 4.611 ± 0.013M/s (drops 0.000 ± 0.000M/s) no-private-stack-64: 5.062 ± 0.006M/s (drops 0.000 ± 0.000M/s) private-stack-64: 5.024 ± 0.004M/s (drops 0.000 ± 0.000M/s) no-private-stack-512: 5.127 ± 0.005M/s (drops 0.000 ± 0.000M/s) private-stack-512: 5.120 ± 0.009M/s (drops 0.000 ± 0.000M/s) no-private-stack-2048: 5.132 ± 0.011M/s (drops 0.000 ± 0.000M/s) private-stack-2048: 5.131 ± 0.008M/s (drops 0.000 ± 0.000M/s) no-private-stack-4096: 5.116 ± 0.023M/s (drops 0.000 ± 0.000M/s) private-stack-4096: 5.123 ± 0.012M/s (drops 0.000 ± 0.000M/s) $ ./benchs/run_bench_private_stack.sh no-private-stack-1: 2.769 ± 0.005M/s (drops 0.000 ± 0.000M/s) private-stack-1: 2.740 ± 0.004M/s (drops 0.000 ± 0.000M/s) no-private-stack-8: 4.617 ± 0.005M/s (drops 0.000 ± 0.000M/s) private-stack-8: 4.578 ± 0.018M/s (drops 0.000 ± 0.000M/s) no-private-stack-64: 5.059 ± 0.009M/s (drops 0.000 ± 0.000M/s) private-stack-64: 5.051 ± 0.007M/s (drops 0.000 ± 0.000M/s) no-private-stack-512: 5.125 ± 0.007M/s (drops 0.000 ± 0.000M/s) private-stack-512: 5.116 ± 0.016M/s (drops 0.000 ± 0.000M/s) no-private-stack-2048: 5.132 ± 0.008M/s (drops 0.000 ± 0.000M/s) private-stack-2048: 5.135 ± 0.013M/s (drops 0.000 ± 0.000M/s) no-private-stack-4096: 5.142 ± 0.013M/s (drops 0.000 ± 0.000M/s) private-stack-4096: 5.109 ± 0.023M/s (drops 0.000 ± 0.000M/s)
The other two are simialr such that for batch size 2048/4096, private-stack might show better results than non-private-stack due to noisek But in general, the no-private-stack is slighter better than private-stack, esp. the number of loop iterations in the bpf prog is large.
I also collected some perf results. With one loop iteration per program run, I got
$ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=1 no-private-stack
18.48% bench [kernel.vmlinux] [k] htab_map_hash
13.04% bench [kernel.vmlinux] [k] _raw_spin_lock
7.00% bench libc.so.6 [.] syscall
5.91% bench [kernel.vmlinux] [k] htab_map_update_elem
5.68% bench [kernel.vmlinux] [k] entry_SYSRETQ_unsafe_stack
4.55% bench [kernel.vmlinux] [k] perf_syscall_enter
4.37% bench [kernel.vmlinux] [k] htab_map_delete_elem
2.89% bench bpf_prog_a8e2493fe867b453_stack0 [k] bpf_prog_a8e2493fe867b453_stack0
2.83% bench [kernel.vmlinux] [k] memcpy_orig
2.60% bench [kernel.vmlinux] [k] __htab_map_lookup_elem
2.53% bench [kernel.vmlinux] [k] alloc_htab_elem
2.52% bench [kernel.vmlinux] [k] trace_call_bpf
2.37% bench [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe
2.29% bench [kernel.vmlinux] [k] do_syscall_64I only showed functions with cpu consumption >= 2%. You can see 'syscall' overhead itself is 7% and bpf progrm run didn't take majority of time. The stack trace for private stack is very similar to the above.
With 4096 loop ierations per program run, I got
$ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 no-private-stack
27.89% bench [kernel.vmlinux] [k] htab_map_hash
21.55% bench [kernel.vmlinux] [k] _raw_spin_lock
11.51% bench [kernel.vmlinux] [k] htab_map_delete_elem
10.26% bench [kernel.vmlinux] [k] htab_map_update_elem
4.85% bench [kernel.vmlinux] [k] __pcpu_freelist_push
4.34% bench [kernel.vmlinux] [k] alloc_htab_elem
3.50% bench [kernel.vmlinux] [k] memcpy_orig
3.22% bench [kernel.vmlinux] [k] __pcpu_freelist_pop
2.68% bench [kernel.vmlinux] [k] bcmp
2.52% bench [kernel.vmlinux] [k] __htab_map_lookup_elem
...
0.01% bench libc.so.6 [.] syscallThe 'syscall' overhead is 0.01% now and majority cpu time is on bpf programs. Again, the stack trace for private stack is very similar to the above.
--- include/linux/bpf.h | 3 +- include/uapi/linux/bpf.h | 3 + kernel/bpf/core.c | 2 +- kernel/bpf/syscall.c | 4 +- tools/include/uapi/linux/bpf.h | 3 + tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 6 + .../bpf/benchs/bench_private_stack.c | 144 ++++++++++++++++++ .../bpf/benchs/run_bench_private_stack.sh | 11 ++ .../selftests/bpf/progs/private_stack.c | 44 ++++++ 10 files changed, 219 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/bpf/benchs/bench_private_stack.c create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_private_stack.sh create mode 100644 tools/testing/selftests/bpf/progs/private_stack.c
include/linux/bpf.h +2 -1
@@ -1551,7 +1551,8 @@ struct bpf_prog { call_get_stack:1, /* Do we call bpf_get_stack() or bpf_get_stackid() */ call_get_func_ip:1, /* Do we call get_func_ip() */ tstamp_type_access:1, /* Accessed __sk_buff->tstamp_type */ - sleepable:1; /* BPF program is sleepable */ + sleepable:1, /* BPF program is sleepable */ + disable_private_stack:1; /* Disable private stack */enum bpf_prog_type type;/* Type of BPF program */ enum bpf_attach_type expected_attach_type;/* For some prog types */ u32 len;/* Number of filter blocks */
include/uapi/linux/bpf.h +3 -0
@@ -1409,6 +1409,9 @@ enum {/* Do not translate kernel bpf_arena pointers to user pointers */ BPF_F_NO_USER_CONV = (1U <<18 ), + +/* Disable private stack */ + BPF_F_DISABLE_PRIVATE_STACK = (1U << 19), };/* Flags for BPF_PROG_QUERY. */
kernel/bpf/core.c +1 -1
@@ -2815,7 +2815,7 @@ EXPORT_SYMBOL_GPL(bpf_prog_free);bool bpf_enable_private_stack (struct bpf_prog *prog) { -if (prog->aux->stack_depth <=64 ) +if (prog->disable_private_stack || prog->aux->stack_depth <=64 )return false;switch (prog->aux->prog->type) {
kernel/bpf/syscall.c +3 -1
@@ -2715,7 +2715,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)BPF_F_XDP_HAS_FRAGS |BPF_F_XDP_DEV_BOUND_ONLY |BPF_F_TEST_REG_INVARIANTS | - BPF_F_TOKEN_FD)) +BPF_F_TOKEN_FD | + BPF_F_DISABLE_PRIVATE_STACK)) return -EINVAL;bpf_prog_load_fixup_attach_type (attr);
@@ -2828,6 +2829,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) prog->expected_attach_type = attr->expected_attach_type; prog->sleepable = !!(attr->prog_flags &BPF_F_SLEEPABLE ); + prog->disable_private_stack = !!(attr->prog_flags &BPF_F_DISABLE_PRIVATE_STACK ); prog->aux->attach_btf = attach_btf; prog->aux->attach_btf_id = attr->attach_btf_id; prog->aux->dst_prog = dst_prog;
tools/include/uapi/linux/bpf.h +3 -0
@@ -1409,6 +1409,9 @@ enum {/* Do not translate kernel bpf_arena pointers to user pointers */ BPF_F_NO_USER_CONV = (1U <<18 ), + +/* Disable private stack */ + BPF_F_DISABLE_PRIVATE_STACK = (1U << 19), };/* Flags for BPF_PROG_QUERY. */
tools/testing/selftests/bpf/Makefile +2 -0
@@ -733,6 +733,7 @@ $(OUTPUT)/bench_local_storage_create.o: $(OUTPUT)/bench_local_storage_create.ske $(OUTPUT)/bench_bpf_hashmap_lookup.o: $(OUTPUT)/bpf_hashmap_lookup.skel.h $(OUTPUT)/bench_htab_mem.o: $(OUTPUT)/htab_mem_bench.skel.h $(OUTPUT)/bench_bpf_crypto.o: $(OUTPUT)/crypto_bench.skel.h +$(OUTPUT)/bench_private_stack.o: $(OUTPUT)/private_stack.skel.h $(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ) $(OUTPUT)/bench: LDLIBS += -lm $(OUTPUT)/bench: $(OUTPUT)/bench.o \
@@ -753,6 +754,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_local_storage_create.o \ $(OUTPUT)/bench_htab_mem.o \ $(OUTPUT)/bench_bpf_crypto.o \ + $(OUTPUT)/bench_private_stack.o \ # $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@
tools/testing/selftests/bpf/bench.c +6 -0
@@ -282,6 +282,7 @@ extern struct argp bench_local_storage_create_argp;extern struct argp bench_htab_mem_argp;extern struct argp bench_trigger_batch_argp;extern struct argp bench_crypto_argp; +extern struct argp bench_private_stack_argp;static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp,0 ,"Ring buffers benchmark" ,0 },
@@ -296,6 +297,7 @@ static const struct argp_child bench_parsers[] = { { &bench_htab_mem_argp, 0, "hash map memory benchmark", 0 }, { &bench_trigger_batch_argp, 0, "BPF triggering benchmark", 0 }, { &bench_crypto_argp, 0, "bpf crypto benchmark", 0 }, + { &bench_private_stack_argp, 0, "bpf private stack benchmark", 0 }, {}, };
@@ -542,6 +544,8 @@ extern const struct bench bench_local_storage_create;extern const struct bench bench_htab_mem;extern const struct bench bench_crypto_encrypt;extern const struct bench bench_crypto_decrypt; +extern const struct bench bench_no_private_stack; +extern const struct bench bench_private_stack;static const struct bench *benchs[] = { &bench_count_global,
@@ -596,6 +600,8 @@ static const struct bench *benchs[] = { &bench_htab_mem, &bench_crypto_encrypt, &bench_crypto_decrypt, + &bench_no_private_stack, + &bench_private_stack, };static void find_benchmark (void )
tools/testing/selftests/bpf/benchs/bench_private_stack.c new file +144 -0
144 lines not on this page · raw message
tools/testing/selftests/bpf/benchs/run_bench_private_stack.sh new file +11 -0
@@ -0,0 +1,11 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +source ./benchs/run_common.sh + +set -eufo pipefail + +for b in 1 8 64 512 2048 4096; do + summarize "no-private-stack-${b}: " "$($RUN_BENCH --nr-batch-iters=${b} no-private-stack)" + summarize "private-stack-${b}: " "$($RUN_BENCH --nr-batch-iters=${b} private-stack)" +done
tools/testing/selftests/bpf/progs/private_stack.c new file +44 -0
44 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH bpf-next v1 2/2] [no_merge] selftests/bpf: Benchmark runtime performance with private stack
Quoted 13 lines · Yonghong Song
With 4096 loop ierations per program run, I got
$ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 no-private-stack 27.89% bench [kernel.vmlinux] [k] htab_map_hash 21.55% bench [kernel.vmlinux] [k] _raw_spin_lock 11.51% bench [kernel.vmlinux] [k] htab_map_delete_elem 10.26% bench [kernel.vmlinux] [k] htab_map_update_elem 4.85% bench [kernel.vmlinux] [k] __pcpu_freelist_push 4.34% bench [kernel.vmlinux] [k] alloc_htab_elem 3.50% bench [kernel.vmlinux] [k] memcpy_orig 3.22% bench [kernel.vmlinux] [k] __pcpu_freelist_pop 2.68% bench [kernel.vmlinux] [k] bcmp 2.52% bench [kernel.vmlinux] [k] __htab_map_lookup_elem
so the prog itself is not even in the top 10 which means that the test doesn't measure anything meaningful about the private stack itself.
It just benchmarks hash map and overhead of extra push/pop is invisible.
Quoted 7 lines
tools/testing/selftests/bpf/progs/private_stack.c new file +7
-0+SEC("tp/syscalls/sys_enter_getpgid") +int stack0 (void *ctx) +{ +struct data_t key = {}, value = {}; +struct data_t *pvalue; +int i; +
tools/testing/selftests/bpf/progs/private_stack.c line 40
Quoted 10 lines
tools/testing/selftests/bpf/progs/private_stack.c new file +10
-0+ hits++; + key.d[10 ] =5 ; + value.d[8 ] =10 ; + +for (i =0 ; i < batch_iters; i++) { + pvalue =bpf_map_lookup_elem (&htab, &key); +if (!pvalue) +bpf_map_update_elem (&htab, &key, &value,0 ); +bpf_map_delete_elem (&htab, &key); + }
Instead of calling helpers that do a lot of work the test should call global subprograms or noinline static functions that are nops. Only then we might see the overhead of push/pop r9.
Once you do that you'll see that +SEC("tp/syscalls/sys_enter_getpgid")
approach has too much overhead.
(you don't see right now since hashmap dominates). Pls use an approach I mentioned earlier by fentry-ing into a helper and another prog calling that helper in for() loop.
pw-bot: cr
Yonghong Song
reviewerRe: [PATCH bpf-next v1 2/2] [no_merge] selftests/bpf: Benchmark runtime performance with private stack
Quoted 40 lines · Alexei Starovoitov
Quoted 12 lines · Yonghong Song
With 4096 loop ierations per program run, I got
$ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 no-private-stack 27.89% bench [kernel.vmlinux] [k] htab_map_hash 21.55% bench [kernel.vmlinux] [k] _raw_spin_lock 11.51% bench [kernel.vmlinux] [k] htab_map_delete_elem 10.26% bench [kernel.vmlinux] [k] htab_map_update_elem 4.85% bench [kernel.vmlinux] [k] __pcpu_freelist_push 4.34% bench [kernel.vmlinux] [k] alloc_htab_elem 3.50% bench [kernel.vmlinux] [k] memcpy_orig 3.22% bench [kernel.vmlinux] [k] __pcpu_freelist_pop 2.68% bench [kernel.vmlinux] [k] bcmp 2.52% bench [kernel.vmlinux] [k] __htab_map_lookup_elemso the prog itself is not even in the top 10 which means that the test doesn't measure anything meaningful about the private stack itself.
It just benchmarks hash map and overhead of extra push/pop is invisible.Quoted 7 lines
tools/testing/selftests/bpf/progs/private_stack.c new file +7
-0+SEC("tp/syscalls/sys_enter_getpgid") +int stack0 (void *ctx) +{ +struct data_t key = {}, value = {}; +struct data_t *pvalue; +int i; +tools/testing/selftests/bpf/progs/private_stack.c line 40
Quoted 10 lines
tools/testing/selftests/bpf/progs/private_stack.c new file +10
-0+ hits++; + key.d[10 ] =5 ; + value.d[8 ] =10 ; + +for (i =0 ; i < batch_iters; i++) { + pvalue =bpf_map_lookup_elem (&htab, &key); +if (!pvalue) +bpf_map_update_elem (&htab, &key, &value,0 ); +bpf_map_delete_elem (&htab, &key); + }Instead of calling helpers that do a lot of work the test should call global subprograms or noinline static functions that are nops. Only then we might see the overhead of push/pop r9.
Quoted 6 lines
Once you do that you'll see that +SEC("tp/syscalls/sys_enter_getpgid")
approach has too much overhead.
(you don't see right now since hashmap dominates). Pls use an approach I mentioned earlier by fentry-ing into a helper and another prog calling that helper infor()loop.
Thanks for suggestion. Will use fentry program with empty functions to test maximum worst performance.
Quoted 2 lines
pw-bot: cr