bpf: Apply the rewrites of bpf_do_misc_fixups() at once
Subsystems (4)
cover bpf: Apply the rewrites of bpf_do_misc_fixups() at once
Eduard pointed out that bpf_patch_insn_data() takes a large share of the time to load pyperf180 [1]. Every patch moves the tail of the instruction and aux data arrays and walks the whole program to fix up branches, so a pass that patches M instructions of an N instruction program does O(N * M) work. Kumar's commit 261b61d3735b ("bpf: Make post-verification instruction rewrites killable") made that work preemptible, but its cost is still quadratic.
On current bpf-next nearly all of it comes from one place in pyperf:bpf_do_misc_fixups() inlines each bpf_map_lookup_elem() on a hash map into 3 instructions, 930 times in a ~24k instruction program for pyperf180 and 1530 times in ~42k instructions for pyperf600.
Patch 1 adds a list of patches that a pass queues and then applies in one O(N + inserted instructions) step. It keeps the semantics of bpf_patch_insn_data(): a patched instruction is named by the first instruction of its patch, and jumps from a patch out of it are relative to the patch in place. Branches, aux data, subprog starts, line info, instruction arrays, function pointers and poke descriptors are remapped together. Patch 2 moves the main loop of bpf_do_misc_fixups() to it, patch 3 drops the delta that is now always 0, and patch 4 adds xlated tests for jumps around patches.
The other passes still use bpf_patch_insn_data() and can move over to the list one at a time in follow-up series.
perf stat -B --all-kernel -r30 -- veristat -q <obj>, mean of two rounds, x86_64 VM with 32 vCPUs:
base patched pyperf180 0.575 s 0.458 s -20.4% pyperf600 1.491 s 1.041 s -30.2% strobemeta 0.531 s 0.532 s test_verif_scale2 0.561 s 0.562 s
For the 1042 objects of the selftests, the 2863 programs that load have the same xlated code on both kernels, with the immediates that hold kernel addresses or BTF ids masked. test_verifier passes on both. Of test_progs, only missed/tp_recursion failed once on the patched kernel in a parallel run, it passes when run alone.
On a side note, the largest part of the time to load pyperf180 is the arg tracking analysis: analyze_subprog() takes about half of it on base, __arg_track_join() alone about 30%.
[1] https://lore.kernel.org/bpf/[email protected]/
---
Qiliang Yuan (4):
bpf: Add a list of deferred instruction patches
bpf: Queue the rewrites of bpf_do_misc_fixups() and apply them at once
bpf: Drop the constant delta from bpf_do_misc_fixups() selftests/bpf: Test jumps around patches of bpf_do_misc_fixups()
include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 26 + kernel/bpf/bpf_insn_array.c | 18 + kernel/bpf/fixups.c | 545 ++++++++++++++------- kernel/bpf/verifier.c | 1 + tools/testing/selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_patch_list.c | 120 +++++ 7 files changed, 546 insertions(+), 167 deletions(-)
---
base-commit: 6a75c73eebd4d497ded7d08b47894f9ddbebb5a9
change-id: 20261001-bpf-verifier-patch-batch-2442080e837d
Best regards,
Signature · 1 line
Qiliang Yuan <[email protected]>
PATCH 1/4 bpf: Add a list of deferred instruction patches
bpf_patch_insn_data() applies a patch right away. It moves the tail of the instruction and aux data arrays, and it walks the whole program to fix up branches and every table that names instructions. A pass that patches M instructions of an N instruction program does O(N * M) work.
pyperf180 from the selftests has 930 bpf_map_lookup_elem() calls on hash maps. bpf_do_misc_fixups() inlines each of them into 3 instructions of a 24k instruction program, which takes about a fifth of the time to load it.
Add a list of patches that a pass queues with bpf_patch_list_add() and applies with bpf_patch_list_commit() in O(N + inserted instructions). Until the commit nothing moves, the pass keeps addressing instructions and aux data by their index in the unpatched program, so patches are queued in increasing order.
The commit maps each old index to a new one and names a patched instruction by the first instruction of its patch, as bpf_patch_insn_data() does. Branches, aux data, subprog starts, line info, instruction arrays, function pointers and poke descriptors all follow the map. The new image is built aside first, so a branch that gets out of range leaves the program untouched.
A patch of a single instruction moves nothing and is applied right away.
--- include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 26 ++++ kernel/bpf/bpf_insn_array.c | 18 +++ kernel/bpf/fixups.c | 295 +++++++++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 1 + 5 files changed, 341 insertions(+)
include/linux/bpf.h +1 -0
@@ -4299,6 +4299,7 @@ int bpf_insn_array_ready(struct bpf_map *map);void bpf_insn_array_release (struct bpf_map *map);void bpf_insn_array_adjust (struct bpf_map *map,u32 first,u32 len);void bpf_insn_array_adjust_after_remove (struct bpf_map *map,u32 off,u32 len); +void bpf_insn_array_remap (struct bpf_map *map,const u32 *new_off,u32 cnt,u32 grow);#ifdef CONFIG_BPF_SYSCALL void bpf_prog_update_insn_ptrs (struct bpf_prog *prog,u32 *offsets,void *image);
include/linux/bpf_verifier.h +26 -0
@@ -929,6 +929,27 @@ struct bpf_fd_array { }; }; +/* one deferred patch: insn 'off' is replaced by 'len' insns */ +struct bpf_insn_patch { +u32 off; +u32 len; +u32 start;/* first insn of the patch in bpf_patch_list.insns */ +struct bpf_insn orig;/* the replaced insn when the patch was queued */ +}; + +/* +* Patches queued by a rewrite pass, applied together by +* bpf_patch_list_commit() in O(prog->len + inserted insns). +*/ +struct bpf_patch_list { +struct bpf_insn_patch *patches; +struct bpf_insn *insns; +u32 cnt; +u32 cap; +u32 insn_cnt; +u32 insn_cap; +}; +/* single container for all structs * one verifier_env per bpf_check() call */
@@ -973,6 +994,7 @@ struct bpf_verifier_env {bool signature;u32 insn_aux_data_len;struct bpf_insn_aux_data *insn_aux_data;/* array of per-insn state */ +struct bpf_patch_list patch_list;const struct bpf_line_info *prev_linfo;struct bpf_verifier_log log;struct bpf_diag *diag;
@@ -1834,5 +1856,9 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env);int bpf_fixup_call_args (struct bpf_verifier_env *env);int bpf_do_misc_fixups (struct bpf_verifier_env *env);int bpf_insn_def32 (struct bpf_prog *prog,struct bpf_insn *insn); +struct bpf_insn *bpf_patch_list_add (struct bpf_verifier_env *env,u32 off, +const struct bpf_insn *patch,u32 len); +int bpf_patch_list_commit (struct bpf_verifier_env *env); +void bpf_patch_list_free (struct bpf_verifier_env *env);#endif /* _LINUX_BPF_VERIFIER_H */
kernel/bpf/bpf_insn_array.c +18 -0
@@ -253,6 +253,24 @@ void bpf_insn_array_adjust(struct bpf_map *map, u32 first, u32 len) } } +/* +* Move each offset to new_off[offset], new_off[] covers [0, cnt]. Offsets +* past the end of the old program only move by 'grow'. +*/ +void bpf_insn_array_remap (struct bpf_map *map,const u32 *new_off,u32 cnt,u32 grow) +{ +struct bpf_insn_array *insn_array =cast_insn_array (map); +u32 off; +int i; + +for (i =0 ; i < map->max_entries; i++) { + off = insn_array->values[i].xlated_off; +if (off ==INSN_DELETED ) +continue ; + insn_array->values[i].xlated_off = off <= cnt ? new_off[off] : off + grow; + } +} +void bpf_insn_array_adjust_after_remove (struct bpf_map *map,u32 off,u32 len) {struct bpf_insn_array *insn_array =cast_insn_array (map);
kernel/bpf/fixups.c +295 -0
301 lines not on this page · raw message
kernel/bpf/verifier.c +1 -0
@@ -22753,6 +22753,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, err_free_env:bpf_free_subprog_jts (env);vfree (env->insn_aux_data); +bpf_patch_list_free (env);kvfree (env->fd_array);bpf_stack_liveness_free (env);kvfree (env->cfg.insn_postorder);
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH bpf-next 1/4] bpf: Add a list of deferred instruction patches
TL;DR: Nit only: patch_list_adj_aux() duplicates the aux-data propagation logic of adjust_insn_aux_data() in fixups.c; the review asks whether a shared helper could keep those rules in one place.…
Quoted 1 line
bpf: Add a list of deferred instruction patches
TL;DR: Nit only: patch_list_adj_aux() duplicates the aux-data propagation logic of adjust_insn_aux_data() in fixups.c; the review asks whether a shared helper could keep those rules in one place. Not a bug.
Quoted 5 lines
kernel/bpf/fixups.c +0
-0@@ -404,6 +404,301 @@ struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
[ ... ]
Quoted 28 lines
kernel/bpf/fixups.c +28
-0+/* +* Fill the aux data of the 'p->len' insns at 'slot' from the aux data 'old' +* of the insn they replace, the same way as adjust_insn_aux_data() does. +*/ +static void patch_list_adj_aux (struct bpf_prog *prog,struct bpf_insn_patch *p, +struct bpf_insn *patch, +const struct bpf_insn_aux_data *old, +struct bpf_insn_aux_data *slot) +{ +struct bpf_insn_aux_data *last = slot + p->len -1 ; +struct bpf_insn *orig = &p->orig; +u32 i; + + *last = *old; + last->zext_dst =bpf_insn_def32 (prog, &patch[p->len -1 ]) >=0 ; +memset (slot,0 ,sizeof (*slot) * (p->len -1 )); + +for (i =0 ; i < p->len -1 ; i++) { + slot[i].seen = last->seen; + slot[i].zext_dst =bpf_insn_def32 (prog, &patch[i]) >=0 ; +if (!memcmp (&patch[i], orig,sizeof (*orig))) { + slot[i].non_stack_access = last->non_stack_access; + last->non_stack_access = false; + }else if (bpf_is_mem_insn (&patch[i])) { + slot[i].non_stack_access = true; + } + } +
kernel/bpf/fixups.c struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, · line 574
Quoted 10 lines
kernel/bpf/fixups.c +10
-0+if (bpf_is_mem_insn (&patch[p->len -1 ]) && +memcmp (&patch[p->len -1 ], orig,sizeof (*orig))) + last->non_stack_access = true; + +/* indirect jumps to the replaced insn land on the first new one */ +if (last->indirect_target) { + slot[0 ].indirect_target =1 ; + last->indirect_target =0 ; + } +}
this isn't a bug, but patch_list_adj_aux() repeats the per-slot aux fill that adjust_insn_aux_data() already does for the BPF_PATCH_KEEP_TARGET case, as its own comment says. The seen propagation, the zext_dst computation via bpf_insn_def32(), the non_stack_access transfer keyed on memcmp() against the original insn, the last-slot bpf_is_mem_insn() marking and the move of indirect_target to the first slot are all duplicated.
Could adjust_insn_aux_data() and patch_list_adj_aux() share a small helper that fills 'len' aux slots from the insns that replaced 'orig' (taking the slot base, the patch insns and the mode), so the aux propagation rules live in one place?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36845462631
PATCH 2/4 bpf: Queue the rewrites of bpf_do_misc_fixups() and apply them at once
In reply to Qiliang YuanEach rewrite in the main loop of bpf_do_misc_fixups() patches the program right away, which costs O(prog->len) per rewrite. Queue them with bpf_patch_list_add() instead and commit them once the loop is done, before the stack of subprogs is initialized for may_goto.
The loop now walks the unpatched program, delta stays 0. A rewrite that fixes up the helper call after queueing it goes on to patch_call_imm with insn pointing to the queued copy of the call, so step insn from the program by index instead of incrementing it.
--- kernel/bpf/fixups.c | 228 +++++++++++++++++----------------------------------- 1 file changed, 72 insertions(+), 156 deletions(-)
kernel/bpf/fixups.c +72 -156
@@ -2099,13 +2099,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt = patch - insn_buf; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2190,13 +2186,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt = patch - insn_buf; } - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2220,13 +2212,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) *patch++ =BPF_MOV64_IMM (insn->dst_reg,0 ); cnt = patch - insn_buf; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2240,13 +2228,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)return -EFAULT ; } - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2293,13 +2277,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) *patch++ =BPF_ALU64_IMM (BPF_MUL , off_reg,-1 ); cnt = patch - insn_buf; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2336,13 +2316,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =may_goto_expand (insn_buf, insn->off, stack_off_cnt, tail,ARRAY_SIZE (tail)); - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }else if (bpf_is_may_goto_insn (insn)) {int stack_off = -stack_depth -8 ;
@@ -2355,13 +2331,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =may_goto_expand (insn_buf, insn->off, stack_off, tail,ARRAY_SIZE (tail)); - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2392,13 +2364,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =2 ; i++; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2413,13 +2381,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)if (cnt ==0 ) goto next_insn; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2500,13 +2464,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) map)->index_mask); insn_buf[2 ] = *insn; cnt =3 ; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2534,13 +2494,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[2 ] = *insn; cnt =3 ; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto patch_call_imm; }
@@ -2553,13 +2509,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[1 ] = *insn; cnt =2 ; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto patch_call_imm; }
@@ -2596,14 +2548,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)if (bpf_map_is_percpu_map (map_ptr->map_type)) prog->jit_required = true; - new_prog =bpf_patch_insn_data (env, i + delta, - insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2679,14 +2626,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) BPF_REG_0, 0); cnt =3 ; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, - cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2709,13 +2651,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[0 ] =BPF_ALU32_REG (BPF_XOR ,BPF_REG_0 ,BPF_REG_0 ); cnt =1 ;#endif - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2728,13 +2666,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[2 ] =BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_0 ,0 ); cnt =3 ; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }#endif
@@ -2762,13 +2696,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[cnt++] =BPF_JMP_A (1 ); insn_buf[cnt++] =BPF_MOV64_IMM (BPF_REG_0 , -EINVAL ); - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2794,13 +2724,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =1 ; } - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2820,13 +2746,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =2 ; } - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2836,12 +2758,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)/* Load IP address from ctx - 16 */ insn_buf[0 ] =BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_1 ,-16 ); - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf,1 ); -if (!new_prog) -return -ENOMEM ; - - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf,1 ); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2891,13 +2810,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[10 ] =BPF_MOV64_IMM (BPF_REG_0 , -ENOENT ); cnt =11 ; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
@@ -2909,13 +2824,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[1 ] =BPF_ATOMIC_OP (BPF_DW ,BPF_XCHG ,BPF_REG_1 ,BPF_REG_0 ,0 ); cnt =2 ; - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; } patch_call_imm:
@@ -2946,9 +2857,14 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) stack_depth_extra =0 ; } i++; - insn++; + insn = &prog->insnsi[i + delta]; } + ret =bpf_patch_list_commit (env); +if (ret) +return ret; + prog = env->prog; + env->prog->aux->stack_depth = subprogs[0 ].stack_depth;for (i =0 ; i < env->subprog_cnt; i++) {int delta =bpf_jit_supports_timed_may_goto () ?2 :1 ;
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH bpf-next 2/4] bpf: Queue the rewrites of bpf_do_misc_fixups() and apply them at once
TL;DR: Possible regression in bpf_do_misc_fixups(): may_goto_expand() now sees unpatched offsets, so a backward may_goto at exactly -32768 may get a 16-bit JEQ instead of gotol, and a previously…
Quoted 1 line
bpf: Queue the rewrites of
bpf_do_misc_fixups()and apply them at once
TL;DR: Possible regression in bpf_do_misc_fixups(): may_goto_expand() now sees unpatched offsets, so a backward may_goto at exactly -32768 may get a 16-bit JEQ instead of gotol, and a previously loading program may fail with -ERANGE.
Quoted 4 lines
kernel/bpf/fixups.c +0
-0
[ ... ]
Quoted 24 lines
kernel/bpf/fixups.c +3
-10@@ -2336,13 +2316,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =may_goto_expand (insn_buf, insn->off, stack_off_cnt, tail,ARRAY_SIZE (tail)); - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ; - - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }else if (bpf_is_may_goto_insn (insn)) {int stack_off = -stack_depth -8 ;@@ -2355,13 +2331,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =may_goto_expand (insn_buf, insn->off, stack_off, tail,ARRAY_SIZE (tail)); - new_prog =bpf_patch_insn_data (env, i + delta, insn_buf, cnt); -if (!new_prog) -return -ENOMEM ;
kernel/bpf/fixups.c int bpf_do_misc_fixups(struct bpf_verifier_env *env) · line 2339
Quoted 10 lines
kernel/bpf/fixups.c +3
-4- - delta += cnt -1 ; - env->prog = prog = new_prog; - insn = new_prog->insnsi + i + delta; + insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); +if (IS_ERR (insn)) +return PTR_ERR (insn); goto next_insn; }
Can this change make a program that loaded before get rejected at commit time?
may_goto_expand() picks between a 16-bit JEQ and JNE + gotol based on the offset it is given:
off = off >= 0 ? off + tail_cnt : off - 1;
...
if (off == (s16)off) {
insn_buf[cnt++] = BPF_JMP_IMM(BPF_JEQ, BPF_REG_AX, 0, off);
} else {
insn_buf[cnt++] = BPF_JMP_IMM(BPF_JNE, BPF_REG_AX, 0, 1);
insn_buf[cnt++] = BPF_JMP32_A(off >= 0 ? off : off - 1);
}Before this change every earlier rewrite was applied right away through bpf_patch_insn_data(), and bpf_adj_branches() updated the jumps after each one. So when the loop reached a backward may_goto, insn->off already included all the growth between the jump target and the may_goto. That is the final backward distance, and the JEQ-or-gotol choice was made on it.
Now the loop walks the unpatched program, so insn->off is the original, unpatched offset. The real distance is only computed later, in patch_list_adj_insn() called from bpf_patch_list_commit():
rel = (s64)new_off[tgt] - pos - 1;
...
if (rel < S16_MIN || rel > S16_MAX)
return -ERANGE;Take a backward may_goto whose grown offset is exactly -32768 (S16_MIN). The old code saw off = -32768, computed off - 1 = -32769, which does not fit in s16, and emitted JNE + gotol, so the program loaded. The new code sees the smaller original offset, so off - 1 fits in s16 and it emits JEQ. At commit the JEQ's displacement is -32769, patch_list_adj_insn() returns -ERANGE, and the program is rejected with "insn %d cannot be patched due to 16-bit range".
The non-timed path under bpf_is_may_goto_insn(), which also calls may_goto_expand(), has the same problem. Forward may_goto behaves as before, since the old code also chose JEQ from the unpatched forward offset. The trigger is narrow, a single exact displacement.
The selftest added later in the series (f8ef7933910d) does not appear to cover this gotol boundary, and no later commit in the series changes it. Should the JEQ-or-gotol decision be deferred until the final offsets are known?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/36845462631
Re: [PATCH bpf-next 2/4] bpf: Queue the rewrites of bpf_do_misc_fixups() and apply them at once
Quoted 1 line
Take a backward
may_gotowhose grown offset is exactly -32768 (S16_MIN).
[...]
Quoted 2 lines
Should the JEQ-or-gotol decision be deferred until the final offsets are known?
This is a real regression, and it is wider than one displacement: any backward may_goto whose offset fits in s16 before the patches between its target and itself are applied, but not after, now gets a JEQ and fails at commit. Those patches are already queued when the loop reaches the may_goto, so v2 will add their growth to the offset before may_goto_expand() picks the encoding, which gives the old choice, and add a selftest around the boundary.
PATCH 3/4 bpf: Drop the constant delta from bpf_do_misc_fixups()
In reply to Qiliang YuanNothing changes the program while bpf_do_misc_fixups() walks it anymore, so delta is always 0 in the loop. Index instructions and aux data by i alone.
--- kernel/bpf/fixups.c | 68 ++++++++++++++++++++++++++--------------------------- 1 file changed, 34 insertions(+), 34 deletions(-)
kernel/bpf/fixups.c +34 -34
@@ -2044,7 +2044,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)struct bpf_insn *insn_buf = env->insn_buf;struct bpf_prog *new_prog;struct bpf_map *map_ptr; -int i, ret, cnt, delta =0 , cur_subprog =0 ; +int i, ret, cnt, cur_subprog =0 ;struct bpf_subprog_info *subprogs = env->subprog_info;u16 stack_depth = subprogs[cur_subprog].stack_depth;u16 stack_depth_extra =0 ;
@@ -2076,7 +2076,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) goto next_insn; } -if (env->insn_aux_data[i + delta].needs_zext) +if (env->insn_aux_data[i].needs_zext)/* Convert BPF_CLASS(insn->code) == BPF_ALU64 to 32-bit ALU */ insn->code =BPF_ALU |BPF_OP (insn->code) |BPF_SRC (insn->code);
@@ -2099,7 +2099,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt = patch - insn_buf; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2186,7 +2186,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt = patch - insn_buf; } - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2212,7 +2212,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) *patch++ =BPF_MOV64_IMM (insn->dst_reg,0 ); cnt = patch - insn_buf; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2228,7 +2228,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)return -EFAULT ; } - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2243,7 +2243,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)bool issrc, isneg, isimm;u32 off_reg; - aux = &env->insn_aux_data[i + delta]; + aux = &env->insn_aux_data[i];if (!aux->alu_state || aux->alu_state ==BPF_ALU_NON_POINTER ) goto next_insn;
@@ -2277,7 +2277,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) *patch++ =BPF_ALU64_IMM (BPF_MUL , off_reg,-1 ); cnt = patch - insn_buf; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2316,7 +2316,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =may_goto_expand (insn_buf, insn->off, stack_off_cnt, tail,ARRAY_SIZE (tail)); - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2331,7 +2331,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =may_goto_expand (insn_buf, insn->off, stack_off, tail,ARRAY_SIZE (tail)); - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2343,7 +2343,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) {struct bpf_map *map; - aux = &env->insn_aux_data[i + delta]; + aux = &env->insn_aux_data[i]; map = env->used_maps[aux->map_index];if (map->map_type !=BPF_MAP_TYPE_PERCPU_ARRAY ) goto next_insn;
@@ -2364,7 +2364,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =2 ; i++; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2375,13 +2375,13 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)if (insn->src_reg ==BPF_PSEUDO_CALL ) goto next_insn;if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL ) { - ret =bpf_fixup_kfunc_call (env, insn, insn_buf, i + delta, &cnt); + ret =bpf_fixup_kfunc_call (env, insn, insn_buf, i, &cnt);if (ret)return ret;if (cnt ==0 ) goto next_insn; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2418,7 +2418,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn->imm =0 ; insn->code =BPF_JMP |BPF_TAIL_CALL ; - aux = &env->insn_aux_data[i + delta]; + aux = &env->insn_aux_data[i]; if (env->bpf_capable && !prog->blinding_requested && prog->jit_requested && !bpf_map_key_poisoned(aux) &&
@@ -2428,7 +2428,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) .reason = BPF_POKE_REASON_TAIL_CALL, .tail_call.map = aux->map_ptr_state.map_ptr, .tail_call.key = bpf_map_key_immediate(aux), - .insn_idx = i + delta, + .insn_idx = i, }; ret =bpf_jit_add_poke_descriptor (prog, &desc);
@@ -2464,13 +2464,13 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) map)->index_mask); insn_buf[2 ] = *insn; cnt =3 ; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn; } - aux = &env->insn_aux_data[i + delta]; + aux = &env->insn_aux_data[i];if (aux->arg_prog) {/* The verifier will process callback_fn as many times as necessary * with different maps and the register states prepared by
@@ -2494,14 +2494,14 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[2 ] = *insn; cnt =3 ; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto patch_call_imm; }/* bpf_per_cpu_ptr() and bpf_this_cpu_ptr() */ -if (env->insn_aux_data[i + delta].call_with_percpu_alloc_ptr) { +if (env->insn_aux_data[i].call_with_percpu_alloc_ptr) {/* patch with 'r1 = *(u64 *)(r1 + 0)' since for percpu data, * bpf_mem_alloc() returns a ptr to the percpu data ptr. */
@@ -2509,7 +2509,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[1 ] = *insn; cnt =2 ; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto patch_call_imm;
@@ -2529,7 +2529,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn->imm == BPF_FUNC_redirect_map || insn->imm == BPF_FUNC_for_each_map_elem || insn->imm == BPF_FUNC_map_lookup_percpu_elem)) { - aux = &env->insn_aux_data[i + delta]; + aux = &env->insn_aux_data[i];if (bpf_map_ptr_poisoned (aux)) goto patch_call_imm;
@@ -2548,7 +2548,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)if (bpf_map_is_percpu_map (map_ptr->map_type)) prog->jit_required = true; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2626,7 +2626,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) BPF_REG_0, 0); cnt =3 ; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2651,7 +2651,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[0 ] =BPF_ALU32_REG (BPF_XOR ,BPF_REG_0 ,BPF_REG_0 ); cnt =1 ;#endif - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2666,7 +2666,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[2 ] =BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_0 ,0 ); cnt =3 ; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2696,7 +2696,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[cnt++] =BPF_JMP_A (1 ); insn_buf[cnt++] =BPF_MOV64_IMM (BPF_REG_0 , -EINVAL ); - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2724,7 +2724,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =1 ; } - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2746,7 +2746,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) cnt =2 ; } - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2758,7 +2758,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)/* Load IP address from ctx - 16 */ insn_buf[0 ] =BPF_LDX_MEM (BPF_DW ,BPF_REG_0 ,BPF_REG_1 ,-16 ); - insn =bpf_patch_list_add (env, i + delta, insn_buf,1 ); + insn =bpf_patch_list_add (env, i, insn_buf,1 );if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2810,7 +2810,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[10 ] =BPF_MOV64_IMM (BPF_REG_0 , -ENOENT ); cnt =11 ; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2824,7 +2824,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) insn_buf[1 ] =BPF_ATOMIC_OP (BPF_DW ,BPF_XCHG ,BPF_REG_1 ,BPF_REG_0 ,0 ); cnt =2 ; - insn =bpf_patch_list_add (env, i + delta, insn_buf, cnt); + insn =bpf_patch_list_add (env, i, insn_buf, cnt);if (IS_ERR (insn))return PTR_ERR (insn); goto next_insn;
@@ -2842,7 +2842,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) } insn->imm =BPF_CALL_IMM (fn->func); next_insn: -if (subprogs[cur_subprog +1 ].start == i + delta +1 ) { +if (subprogs[cur_subprog +1 ].start == i +1 ) { subprogs[cur_subprog].stack_depth += stack_depth_extra; subprogs[cur_subprog].stack_extra = stack_depth_extra;
@@ -2857,7 +2857,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) stack_depth_extra =0 ; } i++; - insn = &prog->insnsi[i + delta]; + insn = &prog->insnsi[i]; } ret =bpf_patch_list_commit (env);
Signature · 1 line
2.43.0
PATCH 4/4 selftests/bpf: Test jumps around patches of bpf_do_misc_fixups()
In reply to Qiliang Yuanbpf_do_misc_fixups() now queues its patches and applies them at once. Check the xlated code for forward and backward jumps to a patched insn, jumps over patches, and a may_goto whose jump leaves its own patch and crosses another one.
--- tools/testing/selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_patch_list.c | 120 +++++++++++++++++++++ 2 files changed, 122 insertions(+)
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -90,6 +90,7 @@#include "verifier_netfilter_retcode.skel.h" #include "verifier_bpf_fastcall.skel.h" #include "verifier_or_jmp32_k.skel.h" +#include "verifier_patch_list.skel.h" #include "verifier_percpu_addr.skel.h" #include "verifier_precision.skel.h" #include "verifier_prevent_map_lookup.skel.h"
@@ -273,6 +274,7 @@ void test_verifier_netfilter_ctx(void) { RUN(verifier_netfilter_ctx); }void test_verifier_netfilter_retcode (void ) {RUN (verifier_netfilter_retcode); }void test_verifier_bpf_fastcall (void ) {RUN (verifier_bpf_fastcall); }void test_verifier_or_jmp32_k (void ) {RUN (verifier_or_jmp32_k); } +void test_verifier_patch_list (void ) {RUN (verifier_patch_list); }void test_verifier_percpu_addr (void ) {RUN (verifier_percpu_addr); }void test_verifier_precision (void ) {RUN (verifier_precision); }void test_verifier_prevent_map_lookup (void ) {RUN (verifier_prevent_map_lookup); }
tools/testing/selftests/bpf/progs/verifier_patch_list.c new file +120 -0
120 lines not on this page · raw message
Signature · 1 line
2.43.0
Quoted 2 lines · Qiliang Yuan
The other passes still use
bpf_patch_insn_data()and can move over to the list one at a time in follow-up series.
That's not going to fly.
After this series there are two implementations of the same thing. patch_list_adj_aux() is a copy of adjust_insn_aux_data(). patch_list_adj_insn() redoes bpf_adj_branches().
The tail of bpf_patch_list_commit() redoes adjust_subprog_starts(), bpf_adj_linfo(), adjust_insn_arrays(), bpf_adjust_func_ptrs() and adjust_poke_descs().__bpf_patch_insn_data() and these helpers were changed four times in September. Every such change would have to be done twice.
Jiong's RFC converted all passes and deleted the old code in the last patch:
https://lore.kernel.org/bpf/[email protected]/
Pls do the same in one series. One-off patches can be add + commit.
Quoted 3 lines
base patched pyperf180 0.575 s 0.458 s -20.4% pyperf600 1.491 s 1.041 s -30.2%
That's veristat on x86.test_progs loads pyperf with BPF_F_TEST_RND_HI32, and s390, riscv, powerpc need zext. bpf_opt_subreg_zext_lo32_rnd_hi32() does more patches than misc_fixups there, I think, and it has one call site.
Quoted 3 lines
On a side note, the largest part of the time to load pyperf180 is the arg tracking analysis:
analyze_subprog()takes about half of it on base,__arg_track_join()alone about 30%.
That's unexpected. Pls share the perf report.
pw-bot: cr
Quoted 2 lines · Alexei Starovoitov
That's not going to fly.
After this series there are two implementations of the same thing.
[...]
Quoted 4 lines
Jiong's RFC converted all passes and deleted the old code in the last patch:
https://lore.kernel.org/bpf/[email protected]/
Pls do the same in one series. One-off patches can be add + commit.
Agreed. v2 will move all passes to the list, turn bpf_patch_insn_data() into add + commit, and remove adjust_insn_aux_data(), bpf_adj_branches() and the per-patch adjust helpers in the last patch. That also covers the bot's comment on patch 1.
Quoted 4 lines
That's veristat on x86.
test_progsloads pyperf withBPF_F_TEST_RND_HI32, and s390, riscv, powerpc need zext.bpf_opt_subreg_zext_lo32_rnd_hi32()does more patches thanmisc_fixupsthere, I think, and it has one call site.
Right. On x86 with BPF_F_TEST_RND_HI32:
patches load time
zext misc flag off flag on
pyperf180 4610 930 0.587 s 1.390 s
pyperf600 7620 1530 1.496 s 4.105 sPatching is 67% of the profile of pyperf180 with the flag, 57% in bpf_opt_subreg_zext_lo32_rnd_hi32() alone. v2 will report both cases.
Quoted 1 line
That's unexpected. Pls share the perf report.
perf record -F 4999 --all-kernel -g -- veristat -q pyperf180.bpf.o, bpf-next 6a75c73eebd4, x86_64 VM:
Children Self
50.66% 0.12% bpf_compute_live_registers
50.43% 0.00% bpf_compute_subprog_arg_access
50.29% 0.67% analyze_subprog
49.47% 8.99% compute_subprog_args.constprop.0
39.08% 8.43% arg_track_join
25.71% 25.67% __arg_track_join
24.97% 0.62% do_check_common
21.49% 0.04% __bpf_patch_insn_data
4.07% 3.99% arg_track_eqpyperf180 has 2 subprogs, on_event and the global __get_frame_data, but analyze_subprog() ran 458 times during that veristat run, which loaded 6 programs counting libbpf's probes.