Skip to contentSkip to conversation

verifier: add prune points to live registers print

PATCH In review Mahe Tardy 4 participants 11 messages Dec 22, 2025 – Dec 31, 2025

Subsystems (4)
lore
Share & tools
lore

Mahe Tardy

author
#0 lore raw

PATCH 1/1 verifier: add prune points to live registers print

Explicitly printing where prune points are placed within the instructions allows for better debugging of state pruning issues.

	Live regs before insn, prune points (p), and force checkpoints (P):
	      0: ..........   (b7) r1 = 0
	      1: .1........   (63) *(u32 *)(r10 -4) = r1
	      2: ..........   (bf) r2 = r10
	      3: ..2.......   (07) r2 += -4
	      4: ..2.......   (18) r1 = 0xffff8cb747b16000
	      6: .12.......   (85) call bpf_map_lookup_elem#1
	      7: 0..345.... p (bf) r6 = r0
	      8: ...3456... p (15) if r6 == 0x0 goto pc+6
	      9: ...3456...   (b7) r1 = 5
	     10: .1.3456...   (b7) r2 = 3
	     11: .123456... p (85) call pc+5
	     12: 0.....6... p (67) r0 <<= 32
	     13: 0.....6...   (c7) r0 s>>= 32
	     14: 0.....6...   (7b) *(u64 *)(r6 +0) = r0
	     15: .......... p (b7) r0 = 0
	     16: 0.........   (95) exit
	     17: .12....... p (bf) r0 = r2
	     18: 01........   (0f) r0 += r1
	     19: 0.........   (95) exit

Also uses uppercase P for force checkpoints, which are a subset of prune points (a force checkpoint should already be a prune point).

	Live regs before insn, prune points (p), and force checkpoints (P):
	      0: .......... p (b7) r1 = 1
	      1: .1........ P (e5) may_goto pc+1
	      2: ..........   (05) goto pc-3
	      3: .1........ p (bf) r0 = r1
	      4: 0.........   (95) exit

Existing tests on liveness tracking had to be updated with the new output format including the prune points.

This proposal patch was presented at Linux Plumbers 2025 in Tokyo along the "Making Sense of State Pruning" presentation[^1] with the intent of making state pruning more transparent to the user.

[^1]: https://lpc.events/event/19/contributions/2162/

---
 kernel/bpf/verifier.c                         |   9 +-
 .../bpf/progs/compute_live_registers.c        | 172 +++++++++---------
 .../selftests/bpf/progs/verifier_live_stack.c |  18 +-
 3 files changed, 102 insertions(+), 97 deletions(-)
kernel/bpf/verifier.c +7 -2
index d6b8a77fbe3b..a82702405c12 100644
@@ -24892,7 +24892,7 @@ static int compute_live_registers(struct bpf_verifier_env *env)
 		insn_aux[i].live_regs_before = state[i].in;
 
 	if (env->log.level & BPF_LOG_LEVEL2) {
-		verbose(env, "Live regs before insn:\n");
+		verbose(env, "Live regs before insn, pruning points (p), and force checkpoints (P):\n");
 		for (i = 0; i < insn_cnt; ++i) {
 			if (env->insn_aux_data[i].scc)
 				verbose(env, "%3d ", env->insn_aux_data[i].scc);
@@ -24904,7 +24904,12 @@ static int compute_live_registers(struct bpf_verifier_env *env)
 					verbose(env, "%d", j);
 				else
 					verbose(env, ".");
-			verbose(env, " ");
+			if (is_force_checkpoint(env, i))
+				verbose(env, " P ");
+			else if (is_prune_point(env, i))
+				verbose(env, " p ");
+			else
+				verbose(env, "   ");
 			verbose_insn(env, &insns[i]);
 			if (bpf_is_ldimm64(&insns[i]))
 				i++;
tools/testing/selftests/bpf/progs/compute_live_registers.c +86 -86

282 lines not on this page · raw message

tools/testing/selftests/bpf/progs/verifier_live_stack.c +9 -9
index 2de105057bbc..105b453be6d3 100644
@@ -136,12 +136,12 @@ static __used __naked void write_first_param(void)
 SEC("socket")
 __log_level(2)
 /* caller_stack_read() function */
-__msg("2: .12345.... (85) call pc+4")
-__msg("5: .12345.... (85) call pc+1")
-__msg("6: 0......... (95) exit")
+__msg("2: .12345.... p (85) call pc+4")
+__msg("5: .12345.... p (85) call pc+1")
+__msg("6: 0......... p (95) exit")
 /* read_first_param() function */
-__msg("7: .1........ (79) r0 = *(u64 *)(r1 +0)")
-__msg("8: 0......... (95) exit")
+__msg("7: .1........ p (79) r0 = *(u64 *)(r1 +0)")
+__msg("8: 0.........   (95) exit")
 /* update for callsite at (2) */
 __msg("(2,7) frame 0 insn 7 +live -8")
 __msg("(2,7) live stack update done in 2 iterations")
@@ -177,10 +177,10 @@ SEC("socket")
 __flag(BPF_F_TEST_STATE_FREQ)
 __log_level(2)
 /* read_first_param2() function */
-__msg(" 9: .1........ (79) r0 = *(u64 *)(r1 +0)")
-__msg("10: .......... (b7) r0 = 0")
-__msg("11: 0......... (05) goto pc+0")
-__msg("12: 0......... (95) exit")
+__msg(" 9: .1........ p (79) r0 = *(u64 *)(r1 +0)")
+__msg("10: ..........   (b7) r0 = 0")
+__msg("11: 0.........   (05) goto pc+0")
+__msg("12: 0......... p (95) exit")
 /*
  * The purpose of the test is to check that checkpoint in
  * read_first_param2() stops path traversal. This will only happen if

--
2.34.1

reviewer Acked-by on v1 1/1
#1 lore raw
Quoted 41 lines · Mahe Tardy

Explicitly printing where prune points are placed within the instructions allows for better debugging of state pruning issues.

	Live regs before insn, prune points (p), and force checkpoints (P):
	      0: ..........   (b7) r1 = 0
	      1: .1........   (63) *(u32 *)(r10 -4) = r1
	      2: ..........   (bf) r2 = r10
	      3: ..2.......   (07) r2 += -4
	      4: ..2.......   (18) r1 = 0xffff8cb747b16000
	      6: .12.......   (85) call bpf_map_lookup_elem#1
	      7: 0..345.... p (bf) r6 = r0
	      8: ...3456... p (15) if r6 == 0x0 goto pc+6
	      9: ...3456...   (b7) r1 = 5
	     10: .1.3456...   (b7) r2 = 3
	     11: .123456... p (85) call pc+5
	     12: 0.....6... p (67) r0 <<= 32
	     13: 0.....6...   (c7) r0 s>>= 32
	     14: 0.....6...   (7b) *(u64 *)(r6 +0) = r0
	     15: .......... p (b7) r0 = 0
	     16: 0.........   (95) exit
	     17: .12....... p (bf) r0 = r2
	     18: 01........   (0f) r0 += r1
	     19: 0.........   (95) exit

Also uses uppercase P for force checkpoints, which are a subset of prune points (a force checkpoint should already be a prune point).

	Live regs before insn, prune points (p), and force checkpoints (P):
	      0: .......... p (b7) r1 = 1
	      1: .1........ P (e5) may_goto pc+1
	      2: ..........   (05) goto pc-3
	      3: .1........ p (bf) r0 = r1
	      4: 0.........   (95) exit

Existing tests on liveness tracking had to be updated with the new output format including the prune points.

This proposal patch was presented at Linux Plumbers 2025 in Tokyo along the "Making Sense of State Pruning" presentation[^1] with the intent of making state pruning more transparent to the user.

Quoted 5 lines

[^1]: https://lpc.events/event/19/contributions/2162/

#2 lore raw
In reply to Mahe Tardy
Quoted 67 lines · Mahe Tardy

Explicitly printing where prune points are placed within the instructions allows for better debugging of state pruning issues.

        Live regs before insn, prune points (p), and force checkpoints (P):
              0: ..........   (b7) r1 = 0
              1: .1........   (63) *(u32 *)(r10 -4) = r1
              2: ..........   (bf) r2 = r10
              3: ..2.......   (07) r2 += -4
              4: ..2.......   (18) r1 = 0xffff8cb747b16000
              6: .12.......   (85) call bpf_map_lookup_elem#1
              7: 0..345.... p (bf) r6 = r0
              8: ...3456... p (15) if r6 == 0x0 goto pc+6
              9: ...3456...   (b7) r1 = 5
             10: .1.3456...   (b7) r2 = 3
             11: .123456... p (85) call pc+5
             12: 0.....6... p (67) r0 <<= 32
             13: 0.....6...   (c7) r0 s>>= 32
             14: 0.....6...   (7b) *(u64 *)(r6 +0) = r0
             15: .......... p (b7) r0 = 0
             16: 0.........   (95) exit
             17: .12....... p (bf) r0 = r2
             18: 01........   (0f) r0 += r1
             19: 0.........   (95) exit

Also uses uppercase P for force checkpoints, which are a subset of prune points (a force checkpoint should already be a prune point).

        Live regs before insn, prune points (p), and force checkpoints (P):
              0: .......... p (b7) r1 = 1
              1: .1........ P (e5) may_goto pc+1
              2: ..........   (05) goto pc-3
              3: .1........ p (bf) r0 = r1
              4: 0.........   (95) exit

Existing tests on liveness tracking had to be updated with the new output format including the prune points.

This proposal patch was presented at Linux Plumbers 2025 in Tokyo along the "Making Sense of State Pruning" presentation[^1] with the intent of making state pruning more transparent to the user.

[^1]: https://lpc.events/event/19/contributions/2162/

---
 kernel/bpf/verifier.c                         |   9 +-
 .../bpf/progs/compute_live_registers.c        | 172 +++++++++---------
 .../selftests/bpf/progs/verifier_live_stack.c |  18 +-
 3 files changed, 102 insertions(+), 97 deletions(-)
kernel/bpf/verifier.c +1 -1
index d6b8a77fbe3b..a82702405c12 100644
@@ -24892,7 +24892,7 @@ static int compute_live_registers(struct bpf_verifier_env *env)
                insn_aux[i].live_regs_before = state[i].in;
 
        if (env->log.level & BPF_LOG_LEVEL2) {
-               verbose(env, "Live regs before insn:\n");
+               verbose(env, "Live regs before insn, pruning points (p), and force checkpoints (P):\n");
                for (i = 0; i < insn_cnt; ++i) {
                        if (env->insn_aux_data[i].scc)
                                verbose(env, "%3d ", env->insn_aux_data[i].scc);

kernel/bpf/verifier.c static int compute_live_registers(struct bpf_verifier_env *env) · line 24912

Quoted 10 lines
kernel/bpf/verifier.c +6 -1
@@ -24904,7 +24904,12 @@ static int compute_live_registers(struct bpf_verifier_env *env)
                                        verbose(env, "%d", j);
                                else
                                        verbose(env, ".");
-                       verbose(env, " ");
+                       if (is_force_checkpoint(env, i))
+                               verbose(env, " P ");
+                       else if (is_prune_point(env, i))
+                               verbose(env, " p ");
+                       else
+                               verbose(env, "   ");

tbh I don't quite see the value. I never needed to know the exact pruning points while working on the verifier. It has to work with existing pruning heuristics and with BPF_F_TEST_STATE_FREQ. So pruning points shouldn't matter to the verifier algorithms. If they are we have a bigger problem to solve than show them in the verifier log to users who won't be able to make much sense of them.

It's my .02. If other folks feel that it's definitely useful we can introduce this extra verbosity, but all the churn in the selftests is another indication of a feature that "nice, but..."

pw-bot: cr

author
#3 lore raw
Quoted 26 lines · Alexei Starovoitov
Quoted 14 lines · Mahe Tardy
kernel/bpf/verifier.c +1 -1
index d6b8a77fbe3b..a82702405c12 100644
@@ -24892,7 +24892,7 @@ static int compute_live_registers(struct bpf_verifier_env *env)
                insn_aux[i].live_regs_before = state[i].in;
 
        if (env->log.level & BPF_LOG_LEVEL2) {
-               verbose(env, "Live regs before insn:\n");
+               verbose(env, "Live regs before insn, pruning points (p), and force checkpoints (P):\n");
                for (i = 0; i < insn_cnt; ++i) {
                        if (env->insn_aux_data[i].scc)
                                verbose(env, "%3d ", env->insn_aux_data[i].scc);

kernel/bpf/verifier.c static int compute_live_registers(struct bpf_verifier_env *env) · line 24912

Quoted 10 lines
kernel/bpf/verifier.c +6 -1
@@ -24904,7 +24904,12 @@ static int compute_live_registers(struct bpf_verifier_env *env)
                                        verbose(env, "%d", j);
                                else
                                        verbose(env, ".");
-                       verbose(env, " ");
+                       if (is_force_checkpoint(env, i))
+                               verbose(env, " P ");
+                       else if (is_prune_point(env, i))
+                               verbose(env, " p ");
+                       else
+                               verbose(env, "   ");
Quoted 7 lines

tbh I don't quite see the value. I never needed to know the exact pruning points while working on the verifier. It has to work with existing pruning heuristics and with BPF_F_TEST_STATE_FREQ. So pruning points shouldn't matter to the verifier algorithms. If they are we have a bigger problem to solve than show them in the verifier log to users who won't be able to make much sense of them.

Yeah I think we would agree with Paul on that. And as you mention, with the addition of the heuristics on top of prune points, it would maybe be more useful to know when the verifier actually saves a new state (but that would increase log verbosity).

Quoted 4 lines

It's my .02. If other folks feel that it's definitely useful we can introduce this extra verbosity, but all the churn in the selftests is another indication of a feature that "nice, but..."

Tbh that's also when I realized that indeed it was "nice, but..." since because of those changes, all those liveness tests would depend on the position of prune points.

At the same time, the new print would allow us to write a series of tests to check for all the possible cases of prune points as presented in the talk, not sure it's actually useful as well...

maintainer
#4 lore raw
Quoted 45 lines · Mahe Tardy
Quoted 26 lines · Alexei Starovoitov
Quoted 14 lines · Mahe Tardy
kernel/bpf/verifier.c +1 -1
index d6b8a77fbe3b..a82702405c12 100644
@@ -24892,7 +24892,7 @@ static int compute_live_registers(struct bpf_verifier_env *env)
                insn_aux[i].live_regs_before = state[i].in;
 
        if (env->log.level & BPF_LOG_LEVEL2) {
-               verbose(env, "Live regs before insn:\n");
+               verbose(env, "Live regs before insn, pruning points (p), and force checkpoints (P):\n");
                for (i = 0; i < insn_cnt; ++i) {
                        if (env->insn_aux_data[i].scc)
                                verbose(env, "%3d ", env->insn_aux_data[i].scc);

kernel/bpf/verifier.c static int compute_live_registers(struct bpf_verifier_env *env) · line 24912

Quoted 10 lines
kernel/bpf/verifier.c +6 -1
@@ -24904,7 +24904,12 @@ static int compute_live_registers(struct bpf_verifier_env *env)
                                        verbose(env, "%d", j);
                                else
                                        verbose(env, ".");
-                       verbose(env, " ");
+                       if (is_force_checkpoint(env, i))
+                               verbose(env, " P ");
+                       else if (is_prune_point(env, i))
+                               verbose(env, " p ");
+                       else
+                               verbose(env, "   ");
Quoted 7 lines

tbh I don't quite see the value. I never needed to know the exact pruning points while working on the verifier. It has to work with existing pruning heuristics and with BPF_F_TEST_STATE_FREQ. So pruning points shouldn't matter to the verifier algorithms. If they are we have a bigger problem to solve than show them in the verifier log to users who won't be able to make much sense of them.

Yeah I think we would agree with Paul on that. And as you mention, with the addition of the heuristics on top of prune points, it would maybe be more useful to know when the verifier actually saves a new state (but that would increase log verbosity).

Quoted 4 lines

It's my .02. If other folks feel that it's definitely useful we can introduce this extra verbosity, but all the churn in the selftests is another indication of a feature that "nice, but..."

Quoted 7 lines

Tbh that's also when I realized that indeed it was "nice, but..." since because of those changes, all those liveness tests would depend on the position of prune points.

At the same time, the new print would allow us to write a series of tests to check for all the possible cases of prune points as presented in the talk, not sure it's actually useful as well...

Hi Everyone,

Sorry, a bit late to the discussion, here are another .02 cents. Tbh, I'm neither for nor against printing these marks. Knowing where exactly the checkpoints are is helpful to me sometimes when I'd like to construct a specific test. On the other hand, I can always add debug prints locally + you do learn the rules for checkpoints after some time. If we go for it, we should probably distinguish between prune points and "force checkpoints" ('f'?).

Imo, it would be indeed more interesting to print where checkpoint match had been attempted and why it failed, e.g. as I do in [1]. Here is a sample:

cache miss at (140, 5389): frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0) from 5387 to 5389: frame1: R0=1 R1=0xffffffff ...

However, in the current form it slows down log level 2 output significantly (~5 times). Okay for my debugging purposes but is not good for upstream submission.

Thanks,
Eduard.

[1] https://github.com/kernel-patches/bpf/commit/65fcd66d03ad9d6979df79628e569b90563d5368

#5 lore raw
Quoted 57 lines · Eduard Zingerman
Quoted 45 lines · Mahe Tardy
Quoted 26 lines · Alexei Starovoitov
Quoted 14 lines · Mahe Tardy
kernel/bpf/verifier.c +1 -1
index d6b8a77fbe3b..a82702405c12 100644
@@ -24892,7 +24892,7 @@ static int compute_live_registers(struct bpf_verifier_env *env)
                insn_aux[i].live_regs_before = state[i].in;
 
        if (env->log.level & BPF_LOG_LEVEL2) {
-               verbose(env, "Live regs before insn:\n");
+               verbose(env, "Live regs before insn, pruning points (p), and force checkpoints (P):\n");
                for (i = 0; i < insn_cnt; ++i) {
                        if (env->insn_aux_data[i].scc)
                                verbose(env, "%3d ", env->insn_aux_data[i].scc);

kernel/bpf/verifier.c static int compute_live_registers(struct bpf_verifier_env *env) · line 24912

Quoted 10 lines
kernel/bpf/verifier.c +6 -1
@@ -24904,7 +24904,12 @@ static int compute_live_registers(struct bpf_verifier_env *env)
                                        verbose(env, "%d", j);
                                else
                                        verbose(env, ".");
-                       verbose(env, " ");
+                       if (is_force_checkpoint(env, i))
+                               verbose(env, " P ");
+                       else if (is_prune_point(env, i))
+                               verbose(env, " p ");
+                       else
+                               verbose(env, "   ");
Quoted 7 lines

tbh I don't quite see the value. I never needed to know the exact pruning points while working on the verifier. It has to work with existing pruning heuristics and with BPF_F_TEST_STATE_FREQ. So pruning points shouldn't matter to the verifier algorithms. If they are we have a bigger problem to solve than show them in the verifier log to users who won't be able to make much sense of them.

Yeah I think we would agree with Paul on that. And as you mention, with the addition of the heuristics on top of prune points, it would maybe be more useful to know when the verifier actually saves a new state (but that would increase log verbosity).

Quoted 4 lines

It's my .02. If other folks feel that it's definitely useful we can introduce this extra verbosity, but all the churn in the selftests is another indication of a feature that "nice, but..."

Quoted 7 lines

Tbh that's also when I realized that indeed it was "nice, but..." since because of those changes, all those liveness tests would depend on the position of prune points.

At the same time, the new print would allow us to write a series of tests to check for all the possible cases of prune points as presented in the talk, not sure it's actually useful as well...

Hi Everyone,

Quoted 7 lines

Sorry, a bit late to the discussion, here are another .02 cents. Tbh, I'm neither for nor against printing these marks. Knowing where exactly the checkpoints are is helpful to me sometimes when I'd like to construct a specific test. On the other hand, I can always add debug prints locally + you do learn the rules for checkpoints after some time. If we go for it, we should probably distinguish between prune points and "force checkpoints" ('f'?).

and this kind of bikeshedding is why I think we shouldn't print either:
prune point, force checkpoint, jmp point.
They are useful in some debugging, but rarely universally needed. Also the change may or may not screw up bpfvv parsing.
Not saying that we shouldn't improve verifier logging, but it really needs to have a strong signal.

Quoted 7 lines

Imo, it would be indeed more interesting to print where checkpoint match had been attempted and why it failed, e.g. as I do in [1]. Here is a sample:

cache miss at (140, 5389): frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0) from 5387 to 5389: frame1: R0=1 R1=0xffffffff ...

Quoted 8 lines

However, in the current form it slows down log level 2 output significantly (~5 times). Okay for my debugging purposes but is not good for upstream submission.

Thanks,
Eduard.

[1] https://github.com/kernel-patches/bpf/commit/65fcd66d03ad9d6979df79628e569b90563d5368

bpf_print_stack_state() refactor can land.
While the rest potentially bpfvv can do.
With log_level==2 all the previous paths through particular instruction will be in the log earlier, so I can imagine clicking on an insn and it will show current and all previous seen states.
The verifier heuristic will drop some of them, so it will show more than actually known during the verification, but that's probably ok for debugging to see why states don't converge.
bpfvv can make it easier to see the difference too instead of "frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0)"
which is not easy to understand.
Only after reading the diff I realized that reg R0 is the one that caused a mismatch.

maintainer
#6 lore raw

On Mon, 2025-12-29 at 16:42 -0800, Alexei Starovoitov wrote:

[...]

Quoted 16 lines
Quoted 7 lines

Imo, it would be indeed more interesting to print where checkpoint match had been attempted and why it failed, e.g. as I do in [1]. Here is a sample:

cache miss at (140, 5389): frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0) from 5387 to 5389: frame1: R0=1 R1=0xffffffff ...

Quoted 8 lines

However, in the current form it slows down log level 2 output significantly (~5 times). Okay for my debugging purposes but is not good for upstream submission.

Thanks,
Eduard.

[1] https://github.com/kernel-patches/bpf/commit/65fcd66d03ad9d6979df79628e569b90563d5368

Quoted 13 lines

bpf_print_stack_state() refactor can land.
While the rest potentially bpfvv can do.
With log_level==2 all the previous paths through particular instruction will be in the log earlier, so I can imagine clicking on an insn and it will show current and all previous seen states.
The verifier heuristic will drop some of them, so it will show more than actually known during the verification, but that's probably ok for debugging to see why states don't converge.
bpfvv can make it easier to see the difference too instead of "frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0)"
which is not easy to understand.
Only after reading the diff I realized that reg R0 is the one that caused a mismatch.

In theory this can be handled in post-processing completely, however I'd expect mirroring states-equal logic in bpfvv (or any other tool) to be error prone. Which is very undesirable when you are debugging. To make post-processing simpler I'd print:
- state id upon state creation
- state ids upon cache miss + register or spi number.

This way post-processing tool would only need to collect register values for state ids in question.

Idk, not sure if any of this should be upstreamed, I'm fine with it living in my debug branch. On the other hand, this might be useful for people debugging 1M instruction issues, as I discussed with Paul Chaignon and Shung-Hsi Yu at LPC.

#7 lore raw
Quoted 42 lines · Eduard Zingerman

On Mon, 2025-12-29 at 16:42 -0800, Alexei Starovoitov wrote:

[...]

Quoted 16 lines
Quoted 7 lines

Imo, it would be indeed more interesting to print where checkpoint match had been attempted and why it failed, e.g. as I do in [1]. Here is a sample:

cache miss at (140, 5389): frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0) from 5387 to 5389: frame1: R0=1 R1=0xffffffff ...

Quoted 8 lines

However, in the current form it slows down log level 2 output significantly (~5 times). Okay for my debugging purposes but is not good for upstream submission.

Thanks,
Eduard.

[1] https://github.com/kernel-patches/bpf/commit/65fcd66d03ad9d6979df79628e569b90563d5368

Quoted 13 lines

bpf_print_stack_state() refactor can land.
While the rest potentially bpfvv can do.
With log_level==2 all the previous paths through particular instruction will be in the log earlier, so I can imagine clicking on an insn and it will show current and all previous seen states.
The verifier heuristic will drop some of them, so it will show more than actually known during the verification, but that's probably ok for debugging to see why states don't converge.
bpfvv can make it easier to see the difference too instead of "frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0)"
which is not easy to understand.
Only after reading the diff I realized that reg R0 is the one that caused a mismatch.

In theory this can be handled in post-processing completely, however I'd expect mirroring states-equal logic in bpfvv (or any other tool) to be error prone. Which is very undesirable when you are debugging. To make post-processing simpler I'd print:
- state id upon state creation
- state ids upon cache miss + register or spi number.

Quoted 2 lines

This way post-processing tool would only need to collect register values for state ids in question.

that will make post processing easier, but print on every miss will greatly increase log_level=2 size, right ?
and whole new concept of state ids just to make a post processing better. I'm not convinced it's worth doing.

maintainer
#8 lore raw
Quoted 43 lines · Alexei Starovoitov
Quoted 42 lines · Eduard Zingerman

On Mon, 2025-12-29 at 16:42 -0800, Alexei Starovoitov wrote:

[...]

Quoted 16 lines
Quoted 7 lines

Imo, it would be indeed more interesting to print where checkpoint match had been attempted and why it failed, e.g. as I do in [1]. Here is a sample:

cache miss at (140, 5389): frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0) from 5387 to 5389: frame1: R0=1 R1=0xffffffff ...

Quoted 8 lines

However, in the current form it slows down log level 2 output significantly (~5 times). Okay for my debugging purposes but is not good for upstream submission.

Thanks,
Eduard.

[1] https://github.com/kernel-patches/bpf/commit/65fcd66d03ad9d6979df79628e569b90563d5368

Quoted 13 lines

bpf_print_stack_state() refactor can land.
While the rest potentially bpfvv can do.
With log_level==2 all the previous paths through particular instruction will be in the log earlier, so I can imagine clicking on an insn and it will show current and all previous seen states.
The verifier heuristic will drop some of them, so it will show more than actually known during the verification, but that's probably ok for debugging to see why states don't converge.
bpfvv can make it easier to see the difference too instead of "frame=1, reg=0, spi=-1, loop=0 (cur: 1) vs (old: P0)"
which is not easy to understand.
Only after reading the diff I realized that reg R0 is the one that caused a mismatch.

In theory this can be handled in post-processing completely, however I'd expect mirroring states-equal logic in bpfvv (or any other tool) to be error prone. Which is very undesirable when you are debugging. To make post-processing simpler I'd print:
- state id upon state creation
- state ids upon cache miss + register or spi number.

Quoted 5 lines
Quoted 2 lines

This way post-processing tool would only need to collect register values for state ids in question.

that will make post processing easier, but print on every miss will greatly increase log_level=2 size, right ?

Here are some stats for pyperf180:

| Experiment                                  | Log   | Log  |
| Kind                                        | Lines | Size |
|---------------------------------------------+-------+------|
| Print cache hits, misses and diffing values | 626K  | 88M  |
| Print cache misses and diffing values       | 618K  | 88M  |
| Print cache misses                          | 618K  | 87M  |
| Default level 2 log                         | 577K  | 85M  |

By "cache hit", "cache miss", "cache miss and diffing values" I mean printing lines like:

cache hit at (44677): loop=0
cache miss at (36030): frame=0, reg=7, spi=-1, loop=0
cache miss at (36030): frame=0, reg=7, spi=-1, loop=0 (cur: scalar(id=3618)) vs (old: Pscalar(id=3258,umin=1))

Didn't try printing message at each new state creation.

Quoted 2 lines

and whole new concept of state ids just to make a post processing better. I'm not convinced it's worth doing.

That might be true.
But we do need some instrument to help debugging 1M instructions situation.

#9 lore raw
Quoted 7 lines · Eduard Zingerman
Quoted 4 lines

that will make post processing easier, but print on every miss will greatly increase log_level=2 size, right ?

Here are some stats for pyperf180:

Quoted 7 lines
| Experiment                                  | Log   | Log  |
| Kind                                        | Lines | Size |
|---------------------------------------------+-------+------|
| Print cache hits, misses and diffing values | 626K  | 88M  |
| Print cache misses and diffing values       | 618K  | 88M  |
| Print cache misses                          | 618K  | 87M  |
| Default level 2 log                         | 577K  | 85M  |

hmm. That's not that much.
Then I don't understand why you said:
"slows down log level 2 output significantly (~5 times)"

If the total output is roughly the same, how come it's 5 times slower??

Quoted 14 lines

By "cache hit", "cache miss", "cache miss and diffing values" I mean printing lines like:

cache hit at (44677): loop=0
cache miss at (36030): frame=0, reg=7, spi=-1, loop=0
cache miss at (36030): frame=0, reg=7, spi=-1, loop=0 (cur: scalar(id=3618)) vs (old: Pscalar(id=3258,umin=1))

Didn't try printing message at each new state creation.

Quoted 2 lines

and whole new concept of state ids just to make a post processing better. I'm not convinced it's worth doing.

That might be true.
But we do need some instrument to help debugging 1M instructions situation.

maintainer
#10 lore raw
Quoted 16 lines · Alexei Starovoitov
Quoted 14 lines · Eduard Zingerman
Quoted 4 lines

that will make post processing easier, but print on every miss will greatly increase log_level=2 size, right ?

Here are some stats for pyperf180:

Quoted 7 lines
Experiment                                  | Log   | Log  |
Kind                                        | Lines | Size |
---------------------------------------------+-------+------|
Print cache hits, misses and diffing values | 626K  | 88M  |
Print cache misses and diffing values       | 618K  | 88M  |
Print cache misses                          | 618K  | 87M  |
Default level 2 log                         | 577K  | 85M  |
Quoted 5 lines

hmm. That's not that much.
Then I don't understand why you said:
"slows down log level 2 output significantly (~5 times)"

If the total output is roughly the same, how come it's 5 times slower??

That's an interesting question.
Double checked the log for the failing program I was debugging (internal, hits 1M limit): 6M lines total, 1.4M lines are "cache ...", so 20% instead of 8% for pyperf180.
Still does not explain the performance difference.
I'll check what happens using profiler.