verifier: add prune points to live registers print
Subsystems (4)
PATCH 1/1 verifier: add prune points to live registers print
Explicitly printing where prune points are placed within the instructions allows for better debugging of state pruning issues.
Live regs before insn, prune points (p), and force checkpoints (P): 0: .......... (b7) r1 = 0 1: .1........ (63) *(u32 *)(r10 -4) = r1 2: .......... (bf) r2 = r10 3: ..2....... (07) r2 += -4 4: ..2....... (18) r1 = 0xffff8cb747b16000 6: .12....... (85) call bpf_map_lookup_elem#1 7: 0..345.... p (bf) r6 = r0 8: ...3456... p (15) if r6 == 0x0 goto pc+6 9: ...3456... (b7) r1 = 5 10: .1.3456... (b7) r2 = 3 11: .123456... p (85) call pc+5 12: 0.....6... p (67) r0 <<= 32 13: 0.....6... (c7) r0 s>>= 32 14: 0.....6... (7b) *(u64 *)(r6 +0) = r0 15: .......... p (b7) r0 = 0 16: 0......... (95) exit 17: .12....... p (bf) r0 = r2 18: 01........ (0f) r0 += r1 19: 0......... (95) exit
Also uses uppercase P for force checkpoints, which are a subset of prune points (a force checkpoint should already be a prune point).
Live regs before insn, prune points (p), and force checkpoints (P): 0: .......... p (b7) r1 = 1 1: .1........ P (e5) may_goto pc+1 2: .......... (05) goto pc-3 3: .1........ p (bf) r0 = r1 4: 0......... (95) exit
Existing tests on liveness tracking had to be updated with the new output format including the prune points.
This proposal patch was presented at Linux Plumbers 2025 in Tokyo along the "Making Sense of State Pruning" presentation[^1] with the intent of making state pruning more transparent to the user.
[^1]: https://lpc.events/event/19/contributions/2162/
--- kernel/bpf/verifier.c | 9 +- .../bpf/progs/compute_live_registers.c | 172 +++++++++--------- .../selftests/bpf/progs/verifier_live_stack.c | 18 +- 3 files changed, 102 insertions(+), 97 deletions(-)
kernel/bpf/verifier.c +7 -2
@@ -24892,7 +24892,7 @@ static int compute_live_registers(struct bpf_verifier_env *env) insn_aux[i].live_regs_before = state[i].in;if (env->log.level &BPF_LOG_LEVEL2 ) { -verbose (env,"Live regs before insn:\n" ); +verbose (env,"Live regs before insn, pruning points (p), and force checkpoints (P):\n" );for (i =0 ; i < insn_cnt; ++i) {if (env->insn_aux_data[i].scc)verbose (env,"%3d " , env->insn_aux_data[i].scc);
@@ -24904,7 +24904,12 @@ static int compute_live_registers(struct bpf_verifier_env *env)verbose (env,"%d" , j); else verbose(env, "."); -verbose (env," " ); +if (is_force_checkpoint (env, i)) +verbose (env," P " ); +else if (is_prune_point (env, i)) +verbose (env," p " ); +else +verbose (env," " );verbose_insn (env, &insns[i]);if (bpf_is_ldimm64 (&insns[i])) i++;
tools/testing/selftests/bpf/progs/compute_live_registers.c +86 -86
282 lines not on this page · raw message
tools/testing/selftests/bpf/progs/verifier_live_stack.c +9 -9
@@ -136,12 +136,12 @@ static __used __naked void write_first_param(void) SEC("socket")__log_level (2 ) /* caller_stack_read() function */ -__msg("2: .12345.... (85) call pc+4") -__msg("5: .12345.... (85) call pc+1") -__msg ("6: 0......... (95) exit" ) +__msg("2: .12345.... p (85) call pc+4") +__msg("5: .12345.... p (85) call pc+1") +__msg ("6: 0......... p (95) exit" ) /* read_first_param() function */ -__msg("7: .1........ (79) r0 = *(u64 *)(r1 +0)") -__msg ("8: 0......... (95) exit" ) +__msg("7: .1........ p (79) r0 = *(u64 *)(r1 +0)") +__msg ("8: 0......... (95) exit" ) /* update for callsite at (2) */ __msg("(2,7) frame 0 insn 7 +live -8")__msg ("(2,7) live stack update done in 2 iterations" )
@@ -177,10 +177,10 @@ SEC("socket") __flag(BPF_F_TEST_STATE_FREQ) __log_level(2) /* read_first_param2() function */ -__msg(" 9: .1........ (79) r0 = *(u64 *)(r1 +0)") -__msg("10: .......... (b7) r0 = 0") -__msg("11: 0......... (05) goto pc+0") -__msg ("12: 0......... (95) exit" ) +__msg(" 9: .1........ p (79) r0 = *(u64 *)(r1 +0)") +__msg("10: .......... (b7) r0 = 0") +__msg("11: 0......... (05) goto pc+0") +__msg ("12: 0......... p (95) exit" )/* * The purpose of the test is to check that checkpoint in * read_first_param2() stops path traversal. This will only happen if
--
2.34.1
Quoted 41 lines · Mahe Tardy
Explicitly printing where prune points are placed within the instructions allows for better debugging of state pruning issues.
Live regs before insn, prune points (p), and force checkpoints (P): 0: .......... (b7) r1 = 0 1: .1........ (63) *(u32 *)(r10 -4) = r1 2: .......... (bf) r2 = r10 3: ..2....... (07) r2 += -4 4: ..2....... (18) r1 = 0xffff8cb747b16000 6: .12....... (85) call bpf_map_lookup_elem#1 7: 0..345.... p (bf) r6 = r0 8: ...3456... p (15) if r6 == 0x0 goto pc+6 9: ...3456... (b7) r1 = 5 10: .1.3456... (b7) r2 = 3 11: .123456... p (85) call pc+5 12: 0.....6... p (67) r0 <<= 32 13: 0.....6... (c7) r0 s>>= 32 14: 0.....6... (7b) *(u64 *)(r6 +0) = r0 15: .......... p (b7) r0 = 0 16: 0......... (95) exit 17: .12....... p (bf) r0 = r2 18: 01........ (0f) r0 += r1 19: 0......... (95) exitAlso uses uppercase P for force checkpoints, which are a subset of prune points (a force checkpoint should already be a prune point).
Live regs before insn, prune points (p), and force checkpoints (P): 0: .......... p (b7) r1 = 1 1: .1........ P (e5) may_goto pc+1 2: .......... (05) goto pc-3 3: .1........ p (bf) r0 = r1 4: 0......... (95) exitExisting tests on liveness tracking had to be updated with the new output format including the prune points.
This proposal patch was presented at Linux Plumbers 2025 in Tokyo along the "Making Sense of State Pruning" presentation[^1] with the intent of making state pruning more transparent to the user.
Quoted 5 lines
[^1]: https://lpc.events/event/19/contributions/2162/
Co-developed-by Paul Chaignon <[email protected]>Signed-off-by Paul Chaignon <[email protected]>Signed-off-by Mahe Tardy <[email protected]>
Quoted 67 lines · Mahe Tardy
Explicitly printing where prune points are placed within the instructions allows for better debugging of state pruning issues.
Live regs before insn, prune points (p), and force checkpoints (P): 0: .......... (b7) r1 = 0 1: .1........ (63) *(u32 *)(r10 -4) = r1 2: .......... (bf) r2 = r10 3: ..2....... (07) r2 += -4 4: ..2....... (18) r1 = 0xffff8cb747b16000 6: .12....... (85) call bpf_map_lookup_elem#1 7: 0..345.... p (bf) r6 = r0 8: ...3456... p (15) if r6 == 0x0 goto pc+6 9: ...3456... (b7) r1 = 5 10: .1.3456... (b7) r2 = 3 11: .123456... p (85) call pc+5 12: 0.....6... p (67) r0 <<= 32 13: 0.....6... (c7) r0 s>>= 32 14: 0.....6... (7b) *(u64 *)(r6 +0) = r0 15: .......... p (b7) r0 = 0 16: 0......... (95) exit 17: .12....... p (bf) r0 = r2 18: 01........ (0f) r0 += r1 19: 0......... (95) exitAlso uses uppercase P for force checkpoints, which are a subset of prune points (a force checkpoint should already be a prune point).
Live regs before insn, prune points (p), and force checkpoints (P): 0: .......... p (b7) r1 = 1 1: .1........ P (e5) may_goto pc+1 2: .......... (05) goto pc-3 3: .1........ p (bf) r0 = r1 4: 0......... (95) exitExisting tests on liveness tracking had to be updated with the new output format including the prune points.
This proposal patch was presented at Linux Plumbers 2025 in Tokyo along the "Making Sense of State Pruning" presentation[^1] with the intent of making state pruning more transparent to the user.
[^1]: https://lpc.events/event/19/contributions/2162/
Co-developed-by Paul Chaignon <[email protected]>Signed-off-by Paul Chaignon <[email protected]>Signed-off-by Mahe Tardy <[email protected]>--- kernel/bpf/verifier.c | 9 +- .../bpf/progs/compute_live_registers.c | 172 +++++++++--------- .../selftests/bpf/progs/verifier_live_stack.c | 18 +- 3 files changed, 102 insertions(+), 97 deletions(-)kernel/bpf/verifier.c +1
-1@@ -24892,7 +24892,7 @@ static int compute_live_registers(struct bpf_verifier_env *env) insn_aux[i].live_regs_before = state[i].in;if (env->log.level &BPF_LOG_LEVEL2 ) { -verbose (env,"Live regs before insn:\n" ); +verbose (env,"Live regs before insn, pruning points (p), and force checkpoints (P):\n" );for (i =0 ; i < insn_cnt; ++i) {if (env->insn_aux_data[i].scc)verbose (env,"%3d " , env->insn_aux_data[i].scc);
kernel/bpf/verifier.c static int compute_live_registers(struct bpf_verifier_env *env) · line 24912
Quoted 10 lines
kernel/bpf/verifier.c +6
-1@@ -24904,7 +24904,12 @@ static int compute_live_registers(struct bpf_verifier_env *env)verbose (env,"%d" , j); else verbose(env, "."); -verbose (env," " ); +if (is_force_checkpoint (env, i)) +verbose (env," P " ); +else if (is_prune_point (env, i)) +verbose (env," p " ); +else +verbose (env," " );
tbh I don't quite see the value. I never needed to know the exact pruning points while working on the verifier. It has to work with existing pruning heuristics and with BPF_F_TEST_STATE_FREQ. So pruning points shouldn't matter to the verifier algorithms. If they are we have a bigger problem to solve than show them in the verifier log to users who won't be able to make much sense of them.
It's my .02. If other folks feel that it's definitely useful we can introduce this extra verbosity, but all the churn in the selftests is another indication of a feature that "nice, but..."
pw-bot: cr
Quoted 26 lines · Alexei Starovoitov
Quoted 14 lines · Mahe Tardy
kernel/bpf/verifier.c +1
-1@@ -24892,7 +24892,7 @@ static int compute_live_registers(struct bpf_verifier_env *env) insn_aux[i].live_regs_before = state[i].in;if (env->log.level &BPF_LOG_LEVEL2 ) { -verbose (env,"Live regs before insn:\n" ); +verbose (env,"Live regs before insn, pruning points (p), and force checkpoints (P):\n" );for (i =0 ; i < insn_cnt; ++i) {if (env->insn_aux_data[i].scc)verbose (env,"%3d " , env->insn_aux_data[i].scc);kernel/bpf/verifier.c static int compute_live_registers(struct bpf_verifier_env *env) · line 24912
Quoted 10 lines
kernel/bpf/verifier.c +6
-1@@ -24904,7 +24904,12 @@ static int compute_live_registers(struct bpf_verifier_env *env)verbose (env,"%d" , j); else verbose(env, "."); -verbose (env," " ); +if (is_force_checkpoint (env, i)) +verbose (env," P " ); +else if (is_prune_point (env, i)) +verbose (env," p " ); +else +verbose (env," " );
Quoted 7 lines
tbh I don't quite see the value. I never needed to know the exact pruning points while working on the verifier. It has to work with existing pruning heuristics and with
BPF_F_TEST_STATE_FREQ. So pruning points shouldn't matter to the verifier algorithms. If they are we have a bigger problem to solve than show them in the verifier log to users who won't be able to make much sense of them.
Yeah I think we would agree with Paul on that. And as you mention, with the addition of the heuristics on top of prune points, it would maybe be more useful to know when the verifier actually saves a new state (but that would increase log verbosity).
Quoted 4 lines
It's my .02. If other folks feel that it's definitely useful we can introduce this extra verbosity, but all the churn in the selftests is another indication of a feature that "nice, but..."
Tbh that's also when I realized that indeed it was "nice, but..." since because of those changes, all those liveness tests would depend on the position of prune points.
At the same time, the new print would allow us to write a series of tests to check for all the possible cases of prune points as presented in the talk, not sure it's actually useful as well...
Eduard Zingerman
maintainerQuoted 45 lines · Mahe Tardy
Quoted 26 lines · Alexei Starovoitov
Quoted 14 lines · Mahe Tardy
kernel/bpf/verifier.c +1
-1@@ -24892,7 +24892,7 @@ static int compute_live_registers(struct bpf_verifier_env *env) insn_aux[i].live_regs_before = state[i].in;if (env->log.level &BPF_LOG_LEVEL2 ) { -verbose (env,"Live regs before insn:\n" ); +verbose (env,"Live regs before insn, pruning points (p), and force checkpoints (P):\n" );for (i =0 ; i < insn_cnt; ++i) {if (env->insn_aux_data[i].scc)verbose (env,"%3d " , env->insn_aux_data[i].scc);kernel/bpf/verifier.c static int compute_live_registers(struct bpf_verifier_env *env) · line 24912
Quoted 10 lines
kernel/bpf/verifier.c +6
-1@@ -24904,7 +24904,12 @@ static int compute_live_registers(struct bpf_verifier_env *env)verbose (env,"%d" , j); else verbose(env, "."); -verbose (env," " ); +if (is_force_checkpoint (env, i)) +verbose (env," P " ); +else if (is_prune_point (env, i)) +verbose (env," p " ); +else +verbose (env," " );Quoted 7 lines
tbh I don't quite see the value. I never needed to know the exact pruning points while working on the verifier. It has to work with existing pruning heuristics and with
BPF_F_TEST_STATE_FREQ. So pruning points shouldn't matter to the verifier algorithms. If they are we have a bigger problem to solve than show them in the verifier log to users who won't be able to make much sense of them.Yeah I think we would agree with Paul on that. And as you mention, with the addition of the heuristics on top of prune points, it would maybe be more useful to know when the verifier actually saves a new state (but that would increase log verbosity).
Quoted 4 lines
It's my .02. If other folks feel that it's definitely useful we can introduce this extra verbosity, but all the churn in the selftests is another indication of a feature that "nice, but..."
Quoted 7 lines
Tbh that's also when I realized that indeed it was "nice, but..." since because of those changes, all those liveness tests would depend on the position of prune points.
At the same time, the new print would allow us to write a series of tests to check for all the possible cases of prune points as presented in the talk, not sure it's actually useful as well...
Hi Everyone,
Sorry, a bit late to the discussion, here are another .02 cents. Tbh, I'm neither for nor against printing these marks. Knowing where exactly the checkpoints are is helpful to me sometimes when I'd like to construct a specific test. On the other hand, I can always add debug prints locally + you do learn the rules for checkpoints after some time. If we go for it, we should probably distinguish between prune points and "force checkpoints" ('f'?).
Imo, it would be indeed more interesting to print where checkpoint match had been attempted and why it failed, e.g. as I do in [1]. Here is a sample:
cache miss at (140, 5389): frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0) from 5387 to 5389: frame1: R0=1 R1=0xffffffff ...
However, in the current form it slows down log level 2 output significantly (~5 times). Okay for my debugging purposes but is not good for upstream submission.
Thanks,
Eduard.
[1] https://github.com/kernel-patches/bpf/commit/65fcd66d03ad9d6979df79628e569b90563d5368
Quoted 57 lines · Eduard Zingerman
Quoted 45 lines · Mahe Tardy
Quoted 26 lines · Alexei Starovoitov
Quoted 14 lines · Mahe Tardy
kernel/bpf/verifier.c +1
-1@@ -24892,7 +24892,7 @@ static int compute_live_registers(struct bpf_verifier_env *env) insn_aux[i].live_regs_before = state[i].in;if (env->log.level &BPF_LOG_LEVEL2 ) { -verbose (env,"Live regs before insn:\n" ); +verbose (env,"Live regs before insn, pruning points (p), and force checkpoints (P):\n" );for (i =0 ; i < insn_cnt; ++i) {if (env->insn_aux_data[i].scc)verbose (env,"%3d " , env->insn_aux_data[i].scc);kernel/bpf/verifier.c static int compute_live_registers(struct bpf_verifier_env *env) · line 24912
Quoted 10 lines
kernel/bpf/verifier.c +6
-1@@ -24904,7 +24904,12 @@ static int compute_live_registers(struct bpf_verifier_env *env)verbose (env,"%d" , j); else verbose(env, "."); -verbose (env," " ); +if (is_force_checkpoint (env, i)) +verbose (env," P " ); +else if (is_prune_point (env, i)) +verbose (env," p " ); +else +verbose (env," " );Quoted 7 lines
tbh I don't quite see the value. I never needed to know the exact pruning points while working on the verifier. It has to work with existing pruning heuristics and with
BPF_F_TEST_STATE_FREQ. So pruning points shouldn't matter to the verifier algorithms. If they are we have a bigger problem to solve than show them in the verifier log to users who won't be able to make much sense of them.Yeah I think we would agree with Paul on that. And as you mention, with the addition of the heuristics on top of prune points, it would maybe be more useful to know when the verifier actually saves a new state (but that would increase log verbosity).
Quoted 4 lines
It's my .02. If other folks feel that it's definitely useful we can introduce this extra verbosity, but all the churn in the selftests is another indication of a feature that "nice, but..."
Quoted 7 lines
Tbh that's also when I realized that indeed it was "nice, but..." since because of those changes, all those liveness tests would depend on the position of prune points.
At the same time, the new print would allow us to write a series of tests to check for all the possible cases of prune points as presented in the talk, not sure it's actually useful as well...
Hi Everyone,
Quoted 7 lines
Sorry, a bit late to the discussion, here are another .02 cents. Tbh, I'm neither for nor against printing these marks. Knowing where exactly the checkpoints are is helpful to me sometimes when I'd like to construct a specific test. On the other hand, I can always add debug prints locally + you do learn the rules for checkpoints after some time. If we go for it, we should probably distinguish between prune points and "force checkpoints" ('f'?).
and this kind of bikeshedding is why I think we shouldn't print either:
prune point, force checkpoint, jmp point.
They are useful in some debugging, but rarely universally needed. Also the change may or may not screw up bpfvv parsing.
Not saying that we shouldn't improve verifier logging, but it really needs to have a strong signal.
Quoted 7 lines
Imo, it would be indeed more interesting to print where checkpoint match had been attempted and why it failed, e.g. as I do in [1]. Here is a sample:
cache miss at (140, 5389): frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0) from 5387 to 5389: frame1: R0=1 R1=0xffffffff ...
Quoted 8 lines
However, in the current form it slows down log level 2 output significantly (~5 times). Okay for my debugging purposes but is not good for upstream submission.
Thanks,
Eduard.[1] https://github.com/kernel-patches/bpf/commit/65fcd66d03ad9d6979df79628e569b90563d5368
bpf_print_stack_state() refactor can land.
While the rest potentially bpfvv can do.
With log_level==2 all the previous paths through particular instruction will be in the log earlier, so I can imagine clicking on an insn and it will show current and all previous seen states.
The verifier heuristic will drop some of them, so it will show more than actually known during the verification, but that's probably ok for debugging to see why states don't converge.
bpfvv can make it easier to see the difference too instead of "frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0)"
which is not easy to understand.
Only after reading the diff I realized that reg R0 is the one that caused a mismatch.
Eduard Zingerman
maintainerOn Mon, 2025-12-29 at 16:42 -0800, Alexei Starovoitov wrote:
[...]
Quoted 16 lines
Quoted 7 lines
Imo, it would be indeed more interesting to print where checkpoint match had been attempted and why it failed, e.g. as I do in [1]. Here is a sample:
cache miss at (140, 5389): frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0) from 5387 to 5389: frame1: R0=1 R1=0xffffffff ...
Quoted 8 lines
However, in the current form it slows down log level 2 output significantly (~5 times). Okay for my debugging purposes but is not good for upstream submission.
Thanks,
Eduard.[1] https://github.com/kernel-patches/bpf/commit/65fcd66d03ad9d6979df79628e569b90563d5368
Quoted 13 lines
bpf_print_stack_state()refactor can land.
While the rest potentially bpfvv can do.
With log_level==2 all the previous paths through particular instruction will be in the log earlier, so I can imagine clicking on an insn and it will show current and all previous seen states.
The verifier heuristic will drop some of them, so it will show more than actually known during the verification, but that's probably ok for debugging to see why states don't converge.
bpfvv can make it easier to see the difference too instead of "frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0)"
which is not easy to understand.
Only after reading the diff I realized that reg R0 is the one that caused a mismatch.
In theory this can be handled in post-processing completely, however I'd expect mirroring states-equal logic in bpfvv (or any other tool) to be error prone. Which is very undesirable when you are debugging. To make post-processing simpler I'd print:
- state id upon state creation
- state ids upon cache miss + register or spi number.
This way post-processing tool would only need to collect register values for state ids in question.
Idk, not sure if any of this should be upstreamed, I'm fine with it living in my debug branch. On the other hand, this might be useful for people debugging 1M instruction issues, as I discussed with Paul Chaignon and Shung-Hsi Yu at LPC.
Quoted 42 lines · Eduard Zingerman
On Mon, 2025-12-29 at 16:42 -0800, Alexei Starovoitov wrote:
[...]
Quoted 16 lines
Quoted 7 lines
Imo, it would be indeed more interesting to print where checkpoint match had been attempted and why it failed, e.g. as I do in [1]. Here is a sample:
cache miss at (140, 5389): frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0) from 5387 to 5389: frame1: R0=1 R1=0xffffffff ...
Quoted 8 lines
However, in the current form it slows down log level 2 output significantly (~5 times). Okay for my debugging purposes but is not good for upstream submission.
Thanks,
Eduard.[1] https://github.com/kernel-patches/bpf/commit/65fcd66d03ad9d6979df79628e569b90563d5368
Quoted 13 lines
bpf_print_stack_state()refactor can land.
While the rest potentially bpfvv can do.
With log_level==2 all the previous paths through particular instruction will be in the log earlier, so I can imagine clicking on an insn and it will show current and all previous seen states.
The verifier heuristic will drop some of them, so it will show more than actually known during the verification, but that's probably ok for debugging to see why states don't converge.
bpfvv can make it easier to see the difference too instead of "frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0)"
which is not easy to understand.
Only after reading the diff I realized that reg R0 is the one that caused a mismatch.In theory this can be handled in post-processing completely, however I'd expect mirroring states-equal logic in bpfvv (or any other tool) to be error prone. Which is very undesirable when you are debugging. To make post-processing simpler I'd print:
- state id upon state creation
- state ids upon cache miss + register or spi number.
Quoted 2 lines
This way post-processing tool would only need to collect register values for state ids in question.
that will make post processing easier, but print on every miss will greatly increase log_level=2 size, right ?
and whole new concept of state ids just to make a post processing better. I'm not convinced it's worth doing.
Eduard Zingerman
maintainerQuoted 43 lines · Alexei Starovoitov
Quoted 42 lines · Eduard Zingerman
On Mon, 2025-12-29 at 16:42 -0800, Alexei Starovoitov wrote:
[...]
Quoted 16 lines
Quoted 7 lines
Imo, it would be indeed more interesting to print where checkpoint match had been attempted and why it failed, e.g. as I do in [1]. Here is a sample:
cache miss at (140, 5389): frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0) from 5387 to 5389: frame1: R0=1 R1=0xffffffff ...
Quoted 8 lines
However, in the current form it slows down log level 2 output significantly (~5 times). Okay for my debugging purposes but is not good for upstream submission.
Thanks,
Eduard.[1] https://github.com/kernel-patches/bpf/commit/65fcd66d03ad9d6979df79628e569b90563d5368
Quoted 13 lines
bpf_print_stack_state()refactor can land.
While the rest potentially bpfvv can do.
With log_level==2 all the previous paths through particular instruction will be in the log earlier, so I can imagine clicking on an insn and it will show current and all previous seen states.
The verifier heuristic will drop some of them, so it will show more than actually known during the verification, but that's probably ok for debugging to see why states don't converge.
bpfvv can make it easier to see the difference too instead of "frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0)"
which is not easy to understand.
Only after reading the diff I realized that reg R0 is the one that caused a mismatch.In theory this can be handled in post-processing completely, however I'd expect mirroring states-equal logic in bpfvv (or any other tool) to be error prone. Which is very undesirable when you are debugging. To make post-processing simpler I'd print:
- state id upon state creation
- state ids upon cache miss + register or spi number.
Quoted 5 lines
Quoted 2 lines
This way post-processing tool would only need to collect register values for state ids in question.
that will make post processing easier, but print on every miss will greatly increase log_level=2 size, right ?
Here are some stats for pyperf180:
| Experiment | Log | Log | | Kind | Lines | Size | |---------------------------------------------+-------+------| | Print cache hits, misses and diffing values | 626K | 88M | | Print cache misses and diffing values | 618K | 88M | | Print cache misses | 618K | 87M | | Default level 2 log | 577K | 85M |
By "cache hit", "cache miss", "cache miss and diffing values" I mean printing lines like:
cache hit at (44677): loop=0
cache miss at (36030): frame=0, reg=7, spi=-1, loop=0
cache miss at (36030): frame=0, reg=7, spi=-1, loop=0 (cur: scalar(id=3618)) vs (old: Pscalar(id=3258,umin=1))
Didn't try printing message at each new state creation.
Quoted 2 lines
and whole new concept of state ids just to make a post processing better. I'm not convinced it's worth doing.
That might be true.
But we do need some instrument to help debugging 1M instructions situation.
Quoted 7 lines · Eduard Zingerman
Quoted 4 lines
that will make post processing easier, but print on every miss will greatly increase log_level=2 size, right ?
Here are some stats for pyperf180:
Quoted 7 lines
| Experiment | Log | Log | | Kind | Lines | Size | |---------------------------------------------+-------+------| | Print cache hits, misses and diffing values | 626K | 88M | | Print cache misses and diffing values | 618K | 88M | | Print cache misses | 618K | 87M | | Default level 2 log | 577K | 85M |
hmm. That's not that much.
Then I don't understand why you said:
"slows down log level 2 output significantly (~5 times)"
If the total output is roughly the same, how come it's 5 times slower??
Quoted 14 lines
By "cache hit", "cache miss", "cache miss and diffing values" I mean printing lines like:
cache hit at (44677): loop=0
cache miss at (36030): frame=0, reg=7, spi=-1, loop=0
cache miss at (36030): frame=0, reg=7, spi=-1, loop=0 (cur: scalar(id=3618)) vs (old:Pscalar(id=3258,umin=1))Didn't try printing message at each new state creation.
Quoted 2 lines
and whole new concept of state ids just to make a post processing better. I'm not convinced it's worth doing.
That might be true.
But we do need some instrument to help debugging 1M instructions situation.
Eduard Zingerman
maintainerQuoted 16 lines · Alexei Starovoitov
Quoted 14 lines · Eduard Zingerman
Quoted 4 lines
that will make post processing easier, but print on every miss will greatly increase log_level=2 size, right ?
Here are some stats for pyperf180:
Quoted 7 lines
Experiment | Log | Log | Kind | Lines | Size | ---------------------------------------------+-------+------| Print cache hits, misses and diffing values | 626K | 88M | Print cache misses and diffing values | 618K | 88M | Print cache misses | 618K | 87M | Default level 2 log | 577K | 85M |
Quoted 5 lines
hmm. That's not that much.
Then I don't understand why you said:
"slows down log level 2 output significantly (~5 times)"If the total output is roughly the same, how come it's 5 times slower??
That's an interesting question.
Double checked the log for the failing program I was debugging (internal, hits 1M limit): 6M lines total, 1.4M lines are "cache ...", so 20% instead of 8% for pyperf180.
Still does not explain the performance difference.
I'll check what happens using profiler.