bpf: avoid retpoline for bpf_for_each_map_elem
PATCH 1/1 bpf: avoid retpoline for bpf_for_each_map_elem
Similarly to 09772d92cd5a ("bpf: avoid retpoline for lookup/update/delete calls on maps") and 84430d4232c3 ("bpf, verifier:
avoid retpoline for map push/pop/peek operation") avoid indirect call while calling bpf_for_each_map_elem.
Before (a program fragment):
; if (rules_map) {
142: (15) if r4 == 0x0 goto pc+8
143: (bf) r3 = r10
; bpf_for_each_map_elem(rules_map, process_each_rule, &ctx, 0);
144: (07) r3 += -24
145: (bf) r1 = r4
146: (18) r2 = subprog[+5]
148: (b7) r4 = 0
149: (85) call bpf_for_each_map_elem#143680 <-- indirect call via
helperAfter (same program fragment):
; if (rules_map) {
142: (15) if r4 == 0x0 goto pc+8
143: (bf) r3 = r10
; bpf_for_each_map_elem(rules_map, process_each_rule, &ctx, 0);
144: (07) r3 += -24
145: (bf) r1 = r4
146: (18) r2 = subprog[+5]
148: (b7) r4 = 0
149: (85) call bpf_for_each_array_elem#170336 <-- direct callOn a benchmark that calls bpf_for_each_map_elem() once and does many other things (mostly checking fields in skb) with CONFIG_RETPOLINE=y it makes program faster.
Before:
============================================================================
Benchmark.cpp time/iter iters/s
============================================================================
IngressMatchByRemoteEndpoint 80.78ns 12.38M IngressMatchByRemoteIP 80.66ns 12.40M IngressMatchByRemotePort 80.87ns 12.37M
After:
============================================================================
Benchmark.cpp time/iter iters/s
============================================================================
IngressMatchByRemoteEndpoint 73.49ns 13.61M IngressMatchByRemoteIP 71.48ns 13.99M IngressMatchByRemotePort 70.39ns 14.21M
--- kernel/bpf/verifier.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
kernel/bpf/verifier.c +10 -1
@@ -12946,7 +12946,8 @@ static int do_misc_fixups(struct bpf_verifier_env *env) insn->imm == BPF_FUNC_map_push_elem || insn->imm == BPF_FUNC_map_pop_elem || insn->imm == BPF_FUNC_map_peek_elem || - insn->imm == BPF_FUNC_redirect_map)) { + insn->imm == BPF_FUNC_redirect_map || + insn->imm == BPF_FUNC_for_each_map_elem)) { aux = &env->insn_aux_data[i + delta];if (bpf_map_ptr_poisoned (aux)) goto patch_call_imm;
@@ -12990,6 +12991,11 @@ static int do_misc_fixups(struct bpf_verifier_env *env) (int (*)(struct bpf_map *map, void *value))NULL));BUILD_BUG_ON (!__same_type (ops->map_redirect, (int (*)(struct bpf_map *map,u32 ifindex,u64 flags))NULL )); +BUILD_BUG_ON (!__same_type (ops->map_for_each_callback, + (int (*)(struct bpf_map *map, +bpf_callback_t callback_fn, +void *callback_ctx, +u64 flags))NULL )); patch_map_ops_generic:switch (insn->imm) {
@@ -13014,6 +13020,9 @@ static int do_misc_fixups(struct bpf_verifier_env *env)case BPF_FUNC_redirect_map: insn->imm =BPF_CALL_IMM (ops->map_redirect);continue ; +case BPF_FUNC_for_each_map_elem: + insn->imm =BPF_CALL_IMM (ops->map_for_each_callback); +continue ; } goto patch_call_imm;
Signature · 1 line
2.30.2
Quoted 7 lines · Andrey Ignatov
Before:
IngressMatchByRemoteEndpoint 80.78ns 12.38M IngressMatchByRemoteIP 80.66ns 12.40M IngressMatchByRemotePort 80.87ns 12.37M
Quoted 5 lines
After:
IngressMatchByRemoteEndpoint 73.49ns 13.61M IngressMatchByRemoteIP 71.48ns 13.99M IngressMatchByRemotePort 70.39ns 14.21M
Nice gains :)
Applied.