Skip to contentSkip to conversation

Add precision propagation for subprogs and callbacks

PATCH Superseded Andrii Nakryiko 3 participants 37 messages Apr 25, 2023 – May 4, 2023

Subsystems (4)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Andrii Nakryiko

author maintainer
#0 lore raw

cover Add precision propagation for subprogs and callbacks

As more and more real-world BPF programs become more complex
and increasingly use subprograms (both static and global), scalar precision tracking and its (previously weak) support for BPF subprograms (and callbacks as a special case of that) is becoming more and more of an issue and limitation. Couple that with increasing reliance on state equivalence (BPF open-coded iterators have a hard requirement for state equivalence to converge and successfully validate loops), and it becomes pretty critical to address this limitation and make precision tracking universally supported for BPF programs of any complexity and composition.

This patch set teaches BPF verifier to support SCALAR precision backpropagation across multiple frames (for subprogram calls and callback simulations) and addresses most practical situations (SCALAR stack loads/stores using registers other than r10 being the last remaining limitation, though thankfully rarely used in practice).

Main logic is explained in details in patch #8. The rest are preliminary preparations, refactorings, clean ups, and fixes. See respective patches for details.

Patch #8 has also veristat comparison of results for selftests, Cilium, and some of Meta production BPF programs before and after these changes.

Andrii Nakryiko (10):
veristat: add -t flag for adding BPF_F_TEST_STATE_FREQ program flag
bpf: mark relevant stack slots scratched for register read instructions
bpf: encapsulate precision backtracking bookkeeping
bpf: improve precision backtrack logging
bpf: maintain bitmasks across all active frames in __mark_chain_precision
bpf: fix propagate_precision() logic for inner frames
bpf: fix mark_all_scalars_precise use in mark_chain_precision
bpf: support precision propagation in the presence of subprogs selftests/bpf: add precision propagation tests in the presence of subprogs selftests/bpf: revert iter test subprog precision workaround

 include/linux/bpf_verifier.h                  |  28 +-
 kernel/bpf/verifier.c                         | 608 ++++++++++++++----
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 tools/testing/selftests/bpf/progs/bpf_misc.h  |   4 +
 tools/testing/selftests/bpf/progs/iters.c     |  26 +-
 .../bpf/progs/verifier_subprog_precision.c    | 536 +++++++++++++++
 .../testing/selftests/bpf/verifier/precise.c  | 107 +--
 tools/testing/selftests/bpf/veristat.c        |   9 +
 8 files changed, 1107 insertions(+), 213 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH 1/10 veristat: add -t flag for adding BPF_F_TEST_STATE_FREQ program flag

Sometimes during debugging it's important that BPF program is loaded with BPF_F_TEST_STATE_FREQ flag set to force verifier to do frequent state checkpointing. Teach veristat to do this when -t ("test state") flag is specified.

---
 tools/testing/selftests/bpf/veristat.c | 9 +++++++++
 1 file changed, 9 insertions(+)
tools/testing/selftests/bpf/veristat.c +9 -0
index 1db7185181da..655095810d4a 100644
@@ -141,6 +141,7 @@ static struct env {
 	bool verbose;
 	bool debug;
 	bool quiet;
+	bool force_checkpoints;
 	enum resfmt out_fmt;
 	bool show_version;
 	bool comparison_mode;
@@ -209,6 +210,8 @@ static const struct argp_option opts[] = {
 	{ "log-level", 'l', "LEVEL", 0, "Verifier log level (default 0 for normal mode, 1 for verbose mode)" },
 	{ "log-fixed", OPT_LOG_FIXED, NULL, 0, "Disable verifier log rotation" },
 	{ "log-size", OPT_LOG_SIZE, "BYTES", 0, "Customize verifier log size (default to 16MB)" },
+	{ "test-states", 't', NULL, 0,
+	  "Force frequent BPF verifier state checkpointing (set BPF_F_TEST_STATE_FREQ program flag)" },
 	{ "quiet", 'q', NULL, 0, "Quiet mode" },
 	{ "emit", 'e', "SPEC", 0, "Specify stats to be emitted" },
 	{ "sort", 's', "SPEC", 0, "Specify sort order" },
@@ -284,6 +287,9 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state)
 			argp_usage(state);
 		}
 		break;
+	case 't':
+		env.force_checkpoints = true;
+		break;
 	case 'C':
 		env.comparison_mode = true;
 		break;
@@ -989,6 +995,9 @@ static int process_prog(const char *filename, struct bpf_object *obj, struct bpf
 	/* increase chances of successful BPF object loading */
 	fixup_obj(obj, prog, base_filename);
 
+	if (env.force_checkpoints)
+		bpf_program__set_flags(prog, bpf_program__flags(prog) | BPF_F_TEST_STATE_FREQ);
+
 	err = bpf_object__load(obj);
 	env.progs_processed++;
 
Signature · 1 line
2.34.1
author maintainer
#2 lore raw

PATCH 2/10 bpf: mark relevant stack slots scratched for register read instructions

In reply to Andrii Nakryiko

When handling instructions that read register slots, mark relevant stack slots as scratched so that verifier log would contain those slots' states, in addition to currently emitted registers with stack slot offsets.

---
 kernel/bpf/verifier.c | 3 +++
 1 file changed, 3 insertions(+)
kernel/bpf/verifier.c +3 -0
index fbcf5a4e2fcd..fea6fe4acba2 100644
@@ -4067,6 +4067,7 @@ static void mark_reg_stack_read(struct bpf_verifier_env *env,
 	for (i = min_off; i < max_off; i++) {
 		slot = -i - 1;
 		spi = slot / BPF_REG_SIZE;
+		mark_stack_slot_scratched(env, spi);
 		stype = ptr_state->stack[spi].slot_type;
 		if (stype[slot % BPF_REG_SIZE] != STACK_ZERO)
 			break;
@@ -4118,6 +4119,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 	stype = reg_state->stack[spi].slot_type;
 	reg = &reg_state->stack[spi].spilled_ptr;
 
+	mark_stack_slot_scratched(env, spi);
+
 	if (is_spilled_reg(&reg_state->stack[spi])) {
 		u8 spill_size = 1;
 
Signature · 1 line
2.34.1
author maintainer
#3 lore raw

PATCH 3/10 bpf: encapsulate precision backtracking bookkeeping

In reply to Andrii Nakryiko

Add struct backtrack_state and straightforward API around it to keep track of register and stack masks used and maintained during precision backtracking process. Having this logic separately allow to keep high-level backtracking algorithm cleaner, but also it sets us up to cleanly keep track of register and stack masks per frame, allowing (with some further logic adjustments) to perform precision backpropagation across multiple frames (i.e., subprog calls).

---
 include/linux/bpf_verifier.h |  15 ++
 kernel/bpf/verifier.c        | 258 ++++++++++++++++++++++++++---------
 2 files changed, 206 insertions(+), 67 deletions(-)
include/linux/bpf_verifier.h +15 -0
index 3dd29a53b711..185bfaf0ec6b 100644
@@ -238,6 +238,10 @@ enum bpf_stack_slot_type {
 
 #define BPF_REG_SIZE 8	/* size of eBPF register in bytes */
 
+#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \
+			  (1 << BPF_REG_3) | (1 << BPF_REG_4) | \
+			  (1 << BPF_REG_5))
+
 #define BPF_DYNPTR_SIZE		sizeof(struct bpf_dynptr_kern)
 #define BPF_DYNPTR_NR_SLOTS		(BPF_DYNPTR_SIZE / BPF_REG_SIZE)
 
@@ -541,6 +545,16 @@ struct bpf_subprog_info {
 	bool is_async_cb;
 };
 
+struct bpf_verifier_env;
+
+struct backtrack_state {
+	struct bpf_verifier_env *env;
+	u32 frame;
+	u32 bitcnt;
+	u32 reg_masks[MAX_CALL_FRAMES];
+	u64 stack_masks[MAX_CALL_FRAMES];
+};
+
 /* single container for all structs
  * one verifier_env per bpf_check() call
  */
@@ -578,6 +592,7 @@ struct bpf_verifier_env {
 		int *insn_stack;
 		int cur_stack;
 	} cfg;
+	struct backtrack_state bt;
 	u32 pass_cnt; /* number of times do_check() was called */
 	u32 subprog_cnt;
 	/* number of instructions analyzed by the verifier */
kernel/bpf/verifier.c +191 -67

454 lines not on this page · raw message

Signature · 1 line
2.34.1
#4 lore raw

Re: [PATCH bpf-next 03/10] bpf: encapsulate precision backtracking bookkeeping

Few early comments so far...

Quoted 81 lines · Andrii Nakryiko

Add struct backtrack_state and straightforward API around it to keep track of register and stack masks used and maintained during precision backtracking process. Having this logic separately allow to keep high-level backtracking algorithm cleaner, but also it sets us up to cleanly keep track of register and stack masks per frame, allowing (with some further logic adjustments) to perform precision backpropagation across multiple frames (i.e., subprog calls).

---
 include/linux/bpf_verifier.h |  15 ++
 kernel/bpf/verifier.c        | 258 ++++++++++++++++++++++++++---------
 2 files changed, 206 insertions(+), 67 deletions(-)
include/linux/bpf_verifier.h +15 -0
index 3dd29a53b711..185bfaf0ec6b 100644
@@ -238,6 +238,10 @@ enum bpf_stack_slot_type {
 
 #define BPF_REG_SIZE 8	/* size of eBPF register in bytes */
 
+#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \
+			  (1 << BPF_REG_3) | (1 << BPF_REG_4) | \
+			  (1 << BPF_REG_5))
+
 #define BPF_DYNPTR_SIZE		sizeof(struct bpf_dynptr_kern)
 #define BPF_DYNPTR_NR_SLOTS		(BPF_DYNPTR_SIZE / BPF_REG_SIZE)
 
@@ -541,6 +545,16 @@ struct bpf_subprog_info {
 	bool is_async_cb;
 };
 
+struct bpf_verifier_env;
+
+struct backtrack_state {
+	struct bpf_verifier_env *env;
+	u32 frame;
+	u32 bitcnt;
+	u32 reg_masks[MAX_CALL_FRAMES];
+	u64 stack_masks[MAX_CALL_FRAMES];
+};
+
 /* single container for all structs
  * one verifier_env per bpf_check() call
  */
@@ -578,6 +592,7 @@ struct bpf_verifier_env {
 		int *insn_stack;
 		int cur_stack;
 	} cfg;
+	struct backtrack_state bt;
 	u32 pass_cnt; /* number of times do_check() was called */
 	u32 subprog_cnt;
 	/* number of instructions analyzed by the verifier */
kernel/bpf/verifier.c +12 -0
index fea6fe4acba2..1cb89fe00507 100644
@@ -1254,6 +1254,12 @@ static bool is_spilled_reg(const struct bpf_stack_state *stack)
 	return stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL;
 }
 
+static bool is_spilled_scalar_reg(const struct bpf_stack_state *stack)
+{
+	return stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL &&
+	       stack->spilled_ptr.type == SCALAR_VALUE;
+}
+
 static void scrub_spilled_slot(u8 *stype)
 {
 	if (*stype != STACK_INVALID)
@@ -3144,12 +3150,137 @@ static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn)
 	return btf_name_by_offset(desc_btf, func->name_off);
 }
 
+static inline void bt_init(struct backtrack_state *bt, u32 frame)
+{
+	bt->frame = frame;
+}
+
+static inline void bt_reset(struct backtrack_state *bt)

kernel/bpf/verifier.c static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn) · line 3168

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+{
+	struct bpf_verifier_env *env = bt->env;
+	memset(bt, 0, sizeof(*bt));
+	bt->env = env;
+}
+
+static inline u32 bt_bitcnt(struct backtrack_state *bt)
+{
+	return bt->bitcnt;
+}

I could have missed it, but it doesn't look that any further patch uses the actual number of bits set.
All uses are: if (bt_bitcnt(bt) != 0)

Hence keeping bitcnt as extra 4 bytes and doing ++, -- on it seems wasteful.
Maybe rename bt_bitcnt into bt_empty or bt_non_empty that will do !!bt->reg_masks[bt->frame] | !!bt->stack_masks[bt->frame]

Quoted 20 lines
kernel/bpf/verifier.c +20 -0
+static inline int bt_subprog_enter(struct backtrack_state *bt)
+{
+	if (bt->frame == MAX_CALL_FRAMES - 1) {
+		verbose(bt->env, "BUG subprog enter from frame %d\n", bt->frame);
+		WARN_ONCE(1, "verifier backtracking bug");
+		return -EFAULT;
+	}
+	bt->frame++;
+	return 0;
+}
+
+static inline int bt_subprog_exit(struct backtrack_state *bt)
+{
+	if (bt->frame == 0) {
+		verbose(bt->env, "BUG subprog exit from frame 0\n");
+		WARN_ONCE(1, "verifier backtracking bug");
+		return -EFAULT;
+	}
+	bt->frame--;
+	return 0;

kernel/bpf/verifier.c static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn) · line 3199

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+}
+
+static inline void bt_set_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg)
+{
+	if (bt->reg_masks[frame] & (1 << reg))
+		return;
+
+	bt->reg_masks[frame] |= 1 << reg;
+	bt->bitcnt++;
+}

It doesnt' look that any further patch is using bt_set_frame_reg with explicit frame. If not, collapse bt_set_frame_reg and bt_set_reg ?

kernel/bpf/verifier.c static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn) · line 3208

Quoted 9 lines
kernel/bpf/verifier.c +9 -0
+
+static inline void bt_clear_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg)
+{
+	if (!(bt->reg_masks[frame] & (1 << reg)))
+		return;
+
+	bt->reg_masks[frame] &= ~(1 << reg);
+	bt->bitcnt--;
+}

If we remove ++,-- of bitcnt this function will be much shorter and faster:
+static inline void bt_clear_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg) +{
+ bt->reg_masks[frame] &= ~(1 << reg);
+}

Removing runtime conditional has a nice perf benefit. Obviously tiny in a grand scheme, but still.

Overall it's a nice cleanup.

#5 lore raw

Re: [PATCH bpf-next 03/10] bpf: encapsulate precision backtracking bookkeeping

Quoted 100 lines · Alexei Starovoitov

Few early comments so far...

Quoted 81 lines · Andrii Nakryiko

Add struct backtrack_state and straightforward API around it to keep track of register and stack masks used and maintained during precision backtracking process. Having this logic separately allow to keep high-level backtracking algorithm cleaner, but also it sets us up to cleanly keep track of register and stack masks per frame, allowing (with some further logic adjustments) to perform precision backpropagation across multiple frames (i.e., subprog calls).

---
 include/linux/bpf_verifier.h |  15 ++
 kernel/bpf/verifier.c        | 258 ++++++++++++++++++++++++++---------
 2 files changed, 206 insertions(+), 67 deletions(-)
include/linux/bpf_verifier.h +15 -0
index 3dd29a53b711..185bfaf0ec6b 100644
@@ -238,6 +238,10 @@ enum bpf_stack_slot_type {
 
 #define BPF_REG_SIZE 8       /* size of eBPF register in bytes */
 
+#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \
+                       (1 << BPF_REG_3) | (1 << BPF_REG_4) | \
+                       (1 << BPF_REG_5))
+
 #define BPF_DYNPTR_SIZE              sizeof(struct bpf_dynptr_kern)
 #define BPF_DYNPTR_NR_SLOTS          (BPF_DYNPTR_SIZE / BPF_REG_SIZE)
 
@@ -541,6 +545,16 @@ struct bpf_subprog_info {
      bool is_async_cb;
 };
 
+struct bpf_verifier_env;
+
+struct backtrack_state {
+     struct bpf_verifier_env *env;
+     u32 frame;
+     u32 bitcnt;
+     u32 reg_masks[MAX_CALL_FRAMES];
+     u64 stack_masks[MAX_CALL_FRAMES];
+};
+
 /* single container for all structs
  * one verifier_env per bpf_check() call
  */
@@ -578,6 +592,7 @@ struct bpf_verifier_env {
              int *insn_stack;
              int cur_stack;
      } cfg;
+     struct backtrack_state bt;
      u32 pass_cnt; /* number of times do_check() was called */
      u32 subprog_cnt;
      /* number of instructions analyzed by the verifier */
kernel/bpf/verifier.c +12 -0
index fea6fe4acba2..1cb89fe00507 100644
@@ -1254,6 +1254,12 @@ static bool is_spilled_reg(const struct bpf_stack_state *stack)
      return stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL;
 }
 
+static bool is_spilled_scalar_reg(const struct bpf_stack_state *stack)
+{
+     return stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL &&
+            stack->spilled_ptr.type == SCALAR_VALUE;
+}
+
 static void scrub_spilled_slot(u8 *stype)
 {
      if (*stype != STACK_INVALID)
@@ -3144,12 +3150,137 @@ static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn)
      return btf_name_by_offset(desc_btf, func->name_off);
 }
 
+static inline void bt_init(struct backtrack_state *bt, u32 frame)
+{
+     bt->frame = frame;
+}
+
+static inline void bt_reset(struct backtrack_state *bt)

kernel/bpf/verifier.c static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn) · line 3168

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+{
+     struct bpf_verifier_env *env = bt->env;
+     memset(bt, 0, sizeof(*bt));
+     bt->env = env;
+}
+
+static inline u32 bt_bitcnt(struct backtrack_state *bt)
+{
+     return bt->bitcnt;
+}

I could have missed it, but it doesn't look that any further patch uses the actual number of bits set.
All uses are: if (bt_bitcnt(bt) != 0)

Quoted 4 lines

Hence keeping bitcnt as extra 4 bytes and doing ++, -- on it seems wasteful.
Maybe rename bt_bitcnt into bt_empty or bt_non_empty that will do !!bt->reg_masks[bt->frame] | !!bt->stack_masks[bt->frame]

Yes, number of bits doesn't matter, it's whether there are any or not. So I'll rename.

As for the counter. I did it to avoid going over all MAX_CALL_FRAMES frames to calculate the final mask. So it was a choice of maintaining count proactively, or doing a loop each time we need to know if there are any bits set:

u64 mask = 0;

for (i = 0; i <= bt->frame; i++)

    mask |= bt->reg_masks[i] | bt->stack_masks[i];

return mask != 0;

I don't think this one is very expensive either, so I can just switch to this, if you prefer that. That will eliminate all the ifs, of course.

Quoted 27 lines
Quoted 27 lines
kernel/bpf/verifier.c +25 -0
+static inline int bt_subprog_enter(struct backtrack_state *bt)
+{
+     if (bt->frame == MAX_CALL_FRAMES - 1) {
+             verbose(bt->env, "BUG subprog enter from frame %d\n", bt->frame);
+             WARN_ONCE(1, "verifier backtracking bug");
+             return -EFAULT;
+     }
+     bt->frame++;
+     return 0;
+}
+
+static inline int bt_subprog_exit(struct backtrack_state *bt)
+{
+     if (bt->frame == 0) {
+             verbose(bt->env, "BUG subprog exit from frame 0\n");
+             WARN_ONCE(1, "verifier backtracking bug");
+             return -EFAULT;
+     }
+     bt->frame--;
+     return 0;
+}
+
+static inline void bt_set_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg)
+{
+     if (bt->reg_masks[frame] & (1 << reg))
Quoted 8 lines

kernel/bpf/verifier.c static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn) · line 3199

Quoted 5 lines
kernel/bpf/verifier.c +5 -0
+             return;
+
+     bt->reg_masks[frame] |= 1 << reg;
+     bt->bitcnt++;
+}

It doesnt' look that any further patch is using bt_set_frame_reg with explicit frame. If not, collapse bt_set_frame_reg and bt_set_reg ?

We do later on, in patch #8, e.g., to propagate r1-r5 bits from subprog back to caller.

Quoted 17 lines

kernel/bpf/verifier.c static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn) · line 3208

Quoted 10 lines
kernel/bpf/verifier.c +9 -0
+
+static inline void bt_clear_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg)
+{
+     if (!(bt->reg_masks[frame] & (1 << reg)))
+             return;
+
+     bt->reg_masks[frame] &= ~(1 << reg);
+     bt->bitcnt--;
+}

If we remove ++,-- of bitcnt this function will be much shorter and faster:

kernel/bpf/verifier.c +4 -0
+static inline void bt_clear_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg)
+{
+       bt->reg_masks[frame] &= ~(1 << reg);
+}
Quoted 2 lines

Removing runtime conditional has a nice perf benefit. Obviously tiny in a grand scheme, but still.

Yep, see above. I did it to avoid doing a loop over up to 8 frame levels to OR all reg_masks and stack_masks. Maybe a wrong tradeoff and it's best to do one small loop when we need to check if there are any bits left to clear?

Quoted 1 line

Overall it's a nice cleanup.

Thanks!

will fix missing empty line you mentioned in another reply

#6 lore raw

Re: [PATCH bpf-next 03/10] bpf: encapsulate precision backtracking bookkeeping

Quoted 118 lines · Andrii Nakryiko
Quoted 100 lines · Alexei Starovoitov

Few early comments so far...

Quoted 81 lines · Andrii Nakryiko

Add struct backtrack_state and straightforward API around it to keep track of register and stack masks used and maintained during precision backtracking process. Having this logic separately allow to keep high-level backtracking algorithm cleaner, but also it sets us up to cleanly keep track of register and stack masks per frame, allowing (with some further logic adjustments) to perform precision backpropagation across multiple frames (i.e., subprog calls).

---
 include/linux/bpf_verifier.h |  15 ++
 kernel/bpf/verifier.c        | 258 ++++++++++++++++++++++++++---------
 2 files changed, 206 insertions(+), 67 deletions(-)
include/linux/bpf_verifier.h +15 -0
index 3dd29a53b711..185bfaf0ec6b 100644
@@ -238,6 +238,10 @@ enum bpf_stack_slot_type {
 
 #define BPF_REG_SIZE 8       /* size of eBPF register in bytes */
 
+#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \
+                       (1 << BPF_REG_3) | (1 << BPF_REG_4) | \
+                       (1 << BPF_REG_5))
+
 #define BPF_DYNPTR_SIZE              sizeof(struct bpf_dynptr_kern)
 #define BPF_DYNPTR_NR_SLOTS          (BPF_DYNPTR_SIZE / BPF_REG_SIZE)
 
@@ -541,6 +545,16 @@ struct bpf_subprog_info {
      bool is_async_cb;
 };
 
+struct bpf_verifier_env;
+
+struct backtrack_state {
+     struct bpf_verifier_env *env;
+     u32 frame;
+     u32 bitcnt;
+     u32 reg_masks[MAX_CALL_FRAMES];
+     u64 stack_masks[MAX_CALL_FRAMES];
+};
+
 /* single container for all structs
  * one verifier_env per bpf_check() call
  */
@@ -578,6 +592,7 @@ struct bpf_verifier_env {
              int *insn_stack;
              int cur_stack;
      } cfg;
+     struct backtrack_state bt;
      u32 pass_cnt; /* number of times do_check() was called */
      u32 subprog_cnt;
      /* number of instructions analyzed by the verifier */
kernel/bpf/verifier.c +12 -0
index fea6fe4acba2..1cb89fe00507 100644
@@ -1254,6 +1254,12 @@ static bool is_spilled_reg(const struct bpf_stack_state *stack)
      return stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL;
 }
 
+static bool is_spilled_scalar_reg(const struct bpf_stack_state *stack)
+{
+     return stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL &&
+            stack->spilled_ptr.type == SCALAR_VALUE;
+}
+
 static void scrub_spilled_slot(u8 *stype)
 {
      if (*stype != STACK_INVALID)
@@ -3144,12 +3150,137 @@ static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn)
      return btf_name_by_offset(desc_btf, func->name_off);
 }
 
+static inline void bt_init(struct backtrack_state *bt, u32 frame)
+{
+     bt->frame = frame;
+}
+
+static inline void bt_reset(struct backtrack_state *bt)

kernel/bpf/verifier.c static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn) · line 3168

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+{
+     struct bpf_verifier_env *env = bt->env;
+     memset(bt, 0, sizeof(*bt));
+     bt->env = env;
+}
+
+static inline u32 bt_bitcnt(struct backtrack_state *bt)
+{
+     return bt->bitcnt;
+}

I could have missed it, but it doesn't look that any further patch uses the actual number of bits set.
All uses are: if (bt_bitcnt(bt) != 0)

Quoted 4 lines

Hence keeping bitcnt as extra 4 bytes and doing ++, -- on it seems wasteful.
Maybe rename bt_bitcnt into bt_empty or bt_non_empty that will do !!bt->reg_masks[bt->frame] | !!bt->stack_masks[bt->frame]

Yes, number of bits doesn't matter, it's whether there are any or not. So I'll rename.

As for the counter. I did it to avoid going over all MAX_CALL_FRAMES frames to calculate the final mask. So it was a choice of maintaining count proactively, or doing a loop each time we need to know if there are any bits set:

u64 mask = 0;

for (i = 0; i <= bt->frame; i++)

Quoted 8 lines
    mask |= bt->reg_masks[i] | bt->stack_masks[i];

return mask != 0;

I don't think this one is very expensive either, so I can just switch to this, if you prefer that. That will eliminate all the ifs, of course.

I see. Missed the loop over frames part.
I guess I'm fine whichever way. Interesting trade off.

#7 lore raw

Re: [PATCH bpf-next 03/10] bpf: encapsulate precision backtracking bookkeeping

In reply to Andrii Nakryiko

kernel/bpf/verifier.c static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn) · line 3160

Quoted 4 lines · Andrii Nakryiko
kernel/bpf/verifier.c +4 -0
+
+static inline void bt_reset(struct backtrack_state *bt)
+{
+	struct bpf_verifier_env *env = bt->env;

empty line

Quoted 3 lines
kernel/bpf/verifier.c +3 -0
+	memset(bt, 0, sizeof(*bt));
+	bt->env = env;
+}
author maintainer
#8 lore raw

PATCH 4/10 bpf: improve precision backtrack logging

In reply to Andrii Nakryiko

Add helper to format register and stack masks in more human-readable format. Adjust logging a bit during backtrack propagation and especially during forcing precision fallback logic to make it clearer what's going on (with log_level=2, of course), and also start reporting affected frame depth. This is in preparation for having more than one active frame later when precision propagation between subprog calls is added.

---
 include/linux/bpf_verifier.h                  |  13 ++-
 kernel/bpf/verifier.c                         |  72 ++++++++++--
 .../testing/selftests/bpf/verifier/precise.c  | 106 +++++++++---------
 3 files changed, 128 insertions(+), 63 deletions(-)
include/linux/bpf_verifier.h +9 -4
index 185bfaf0ec6b..0ca367e13dd8 100644
@@ -18,8 +18,11 @@
  * that converting umax_value to int cannot overflow.
  */
 #define BPF_MAX_VAR_SIZ	(1 << 29)
-/* size of type_str_buf in bpf_verifier. */
-#define TYPE_STR_BUF_LEN 128
+/* size of tmp_str_buf in bpf_verifier.
+ * we need at least 306 bytes to fit full stack mask representation
+ * (in the "-8,-16,...,-512" form)
+ */
+#define TMP_STR_BUF_LEN 320
 
 /* Liveness marks, used for registers and spilled-regs (in stack slots).
  * Read marks propagate upwards until they find a write mark; they record that
@@ -621,8 +624,10 @@ struct bpf_verifier_env {
 	/* Same as scratched_regs but for stack slots */
 	u64 scratched_stack_slots;
 	u64 prev_log_pos, prev_insn_print_pos;
-	/* buffer used in reg_type_str() to generate reg_type string */
-	char type_str_buf[TYPE_STR_BUF_LEN];
+	/* buffer used to generate temporary string representations,
+	 * e.g., in reg_type_str() to generate reg_type string
+	 */
+	char tmp_str_buf[TMP_STR_BUF_LEN];
 };
 
 __printf(2, 0) void bpf_verifier_vlog(struct bpf_verifier_log *log,
kernel/bpf/verifier.c +65 -7

118 lines not on this page · raw message

tools/testing/selftests/bpf/verifier/precise.c +54 -52

130 lines not on this page · raw message

Signature · 1 line
2.34.1
#9 lore raw

Re: [PATCH bpf-next 04/10] bpf: improve precision backtrack logging

Quoted 108 lines · Andrii Nakryiko

Add helper to format register and stack masks in more human-readable format. Adjust logging a bit during backtrack propagation and especially during forcing precision fallback logic to make it clearer what's going on (with log_level=2, of course), and also start reporting affected frame depth. This is in preparation for having more than one active frame later when precision propagation between subprog calls is added.

---
 include/linux/bpf_verifier.h                  |  13 ++-
 kernel/bpf/verifier.c                         |  72 ++++++++++--
 .../testing/selftests/bpf/verifier/precise.c  | 106 +++++++++---------
 3 files changed, 128 insertions(+), 63 deletions(-)
include/linux/bpf_verifier.h +9 -4
index 185bfaf0ec6b..0ca367e13dd8 100644
@@ -18,8 +18,11 @@
  * that converting umax_value to int cannot overflow.
  */
 #define BPF_MAX_VAR_SIZ	(1 << 29)
-/* size of type_str_buf in bpf_verifier. */
-#define TYPE_STR_BUF_LEN 128
+/* size of tmp_str_buf in bpf_verifier.
+ * we need at least 306 bytes to fit full stack mask representation
+ * (in the "-8,-16,...,-512" form)
+ */
+#define TMP_STR_BUF_LEN 320
 
 /* Liveness marks, used for registers and spilled-regs (in stack slots).
  * Read marks propagate upwards until they find a write mark; they record that
@@ -621,8 +624,10 @@ struct bpf_verifier_env {
 	/* Same as scratched_regs but for stack slots */
 	u64 scratched_stack_slots;
 	u64 prev_log_pos, prev_insn_print_pos;
-	/* buffer used in reg_type_str() to generate reg_type string */
-	char type_str_buf[TYPE_STR_BUF_LEN];
+	/* buffer used to generate temporary string representations,
+	 * e.g., in reg_type_str() to generate reg_type string
+	 */
+	char tmp_str_buf[TMP_STR_BUF_LEN];
 };
 
 __printf(2, 0) void bpf_verifier_vlog(struct bpf_verifier_log *log,
kernel/bpf/verifier.c +41 -2
index 1cb89fe00507..8faf9170acf0 100644
@@ -604,9 +604,9 @@ static const char *reg_type_str(struct bpf_verifier_env *env,
 		 type & PTR_TRUSTED ? "trusted_" : ""
 	);
 
-	snprintf(env->type_str_buf, TYPE_STR_BUF_LEN, "%s%s%s",
+	snprintf(env->tmp_str_buf, TMP_STR_BUF_LEN, "%s%s%s",
 		 prefix, str[base_type(type)], postfix);
-	return env->type_str_buf;
+	return env->tmp_str_buf;
 }
 
 static char slot_type_char[] = {
@@ -3275,6 +3275,45 @@ static inline bool bt_is_slot_set(struct backtrack_state *bt, u32 slot)
 	return bt->stack_masks[bt->frame] & (1ull << slot);
 }
 
+/* format registers bitmask, e.g., "r0,r2,r4" for 0x15 mask */
+static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
+{
+	DECLARE_BITMAP(mask, 64);
+	bool first = true;
+	int i, n;
+
+	buf[0] = '\0';
+
+	bitmap_from_u64(mask, reg_mask);
+	for_each_set_bit(i, mask, 32) {
+		n = snprintf(buf, buf_sz, "%sr%d", first ? "" : ",", i);
+		first = false;
+		buf += n;
+		buf_sz -= n;
+		if (buf_sz < 0)
+			break;
+	}
+}
+/* format stack slots bitmask, e.g., "-8,-24,-40" for 0x15 mask */
+static void fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)
+{
+	DECLARE_BITMAP(mask, 64);
+	bool first = true;
+	int i, n;
+
+	buf[0] = '\0';
+
+	bitmap_from_u64(mask, stack_mask);
+	for_each_set_bit(i, mask, 64) {
+		n = snprintf(buf, buf_sz, "%s%d", first ? "" : ",", -(i + 1) * 8);
+		first = false;
+		buf += n;
+		buf_sz -= n;
+		if (buf_sz < 0)
+			break;
+	}
+}
+
 /* For given verifier state backtrack_insn() is called from the last insn to
  * the first insn. Its purpose is to compute a bitmask of registers and
  * stack slots that needs precision in the parent verifier state.

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3344

Quoted 9 lines
kernel/bpf/verifier.c +5 -1
@@ -3298,7 +3337,11 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
 	if (insn->code == 0)
 		return 0;
 	if (env->log.level & BPF_LOG_LEVEL2) {
-		verbose(env, "regs=%x stack=%llx before ", bt_reg_mask(bt), bt_stack_mask(bt));
+		fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_reg_mask(bt));
+		verbose(env, "mark_precise: frame%d: regs(0x%x)=%s ",
+			bt->frame, bt_reg_mask(bt), env->tmp_str_buf);
+		fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
+		verbose(env, "stack(0x%llx)=%s before ", bt_stack_mask(bt), env->tmp_str_buf);

Let's drop (0x%llx) part from regs and stack.
With nice human readable addition no one will be reading the hex anymore. It's just wasting screen real estate.

Quoted 8 lines
kernel/bpf/verifier.c +8 -0
+	"mark_precise: frame0: last_idx 26 first_idx 20\
+	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 25\
+	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 24\
+	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 23\
+	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 22\
+	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 20\
+	parent didn't have regs=4 stack=0 marks:\
+	mark_precise: frame0: last_idx 19 first_idx 10\

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx,

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 19\
+	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 18\
+	mark_precise: frame0: regs(0x300)=r8,r9 stack(0x0)= before 17\
+	mark_precise: frame0: regs(0x201)=r0,r9 stack(0x0)= before 15\
+	mark_precise: frame0: regs(0x201)=r0,r9 stack(0x0)= before 14\
+	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 13\
+	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 12\
+	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 11\
+	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 10\
+	parent already had regs=0 stack=0 marks:",

This part would be much cleaner without (0x...)

#10 lore raw

Re: [PATCH bpf-next 04/10] bpf: improve precision backtrack logging

Quoted 115 lines · Alexei Starovoitov
Quoted 113 lines · Andrii Nakryiko

Add helper to format register and stack masks in more human-readable format. Adjust logging a bit during backtrack propagation and especially during forcing precision fallback logic to make it clearer what's going on (with log_level=2, of course), and also start reporting affected frame depth. This is in preparation for having more than one active frame later when precision propagation between subprog calls is added.

---
 include/linux/bpf_verifier.h                  |  13 ++-
 kernel/bpf/verifier.c                         |  72 ++++++++++--
 .../testing/selftests/bpf/verifier/precise.c  | 106 +++++++++---------
 3 files changed, 128 insertions(+), 63 deletions(-)
include/linux/bpf_verifier.h +9 -4
index 185bfaf0ec6b..0ca367e13dd8 100644
@@ -18,8 +18,11 @@
  * that converting umax_value to int cannot overflow.
  */
 #define BPF_MAX_VAR_SIZ      (1 << 29)
-/* size of type_str_buf in bpf_verifier. */
-#define TYPE_STR_BUF_LEN 128
+/* size of tmp_str_buf in bpf_verifier.
+ * we need at least 306 bytes to fit full stack mask representation
+ * (in the "-8,-16,...,-512" form)
+ */
+#define TMP_STR_BUF_LEN 320
 
 /* Liveness marks, used for registers and spilled-regs (in stack slots).
  * Read marks propagate upwards until they find a write mark; they record that
@@ -621,8 +624,10 @@ struct bpf_verifier_env {
      /* Same as scratched_regs but for stack slots */
      u64 scratched_stack_slots;
      u64 prev_log_pos, prev_insn_print_pos;
-     /* buffer used in reg_type_str() to generate reg_type string */
-     char type_str_buf[TYPE_STR_BUF_LEN];
+     /* buffer used to generate temporary string representations,
+      * e.g., in reg_type_str() to generate reg_type string
+      */
+     char tmp_str_buf[TMP_STR_BUF_LEN];
 };
 
 __printf(2, 0) void bpf_verifier_vlog(struct bpf_verifier_log *log,
kernel/bpf/verifier.c +42 -3
index 1cb89fe00507..8faf9170acf0 100644
@@ -604,9 +604,9 @@ static const char *reg_type_str(struct bpf_verifier_env *env,
               type & PTR_TRUSTED ? "trusted_" : ""
      );
 
-     snprintf(env->type_str_buf, TYPE_STR_BUF_LEN, "%s%s%s",
+     snprintf(env->tmp_str_buf, TMP_STR_BUF_LEN, "%s%s%s",
               prefix, str[base_type(type)], postfix);
-     return env->type_str_buf;
+     return env->tmp_str_buf;
 }
 
 static char slot_type_char[] = {
@@ -3275,6 +3275,45 @@ static inline bool bt_is_slot_set(struct backtrack_state *bt, u32 slot)
      return bt->stack_masks[bt->frame] & (1ull << slot);
 }
 
+/* format registers bitmask, e.g., "r0,r2,r4" for 0x15 mask */
+static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
+{
+     DECLARE_BITMAP(mask, 64);
+     bool first = true;
+     int i, n;
+
+     buf[0] = '\0';
+
+     bitmap_from_u64(mask, reg_mask);
+     for_each_set_bit(i, mask, 32) {
+             n = snprintf(buf, buf_sz, "%sr%d", first ? "" : ",", i);
+             first = false;
+             buf += n;
+             buf_sz -= n;
+             if (buf_sz < 0)
+                     break;
+     }
+}
+/* format stack slots bitmask, e.g., "-8,-24,-40" for 0x15 mask */
+static void fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)
+{
+     DECLARE_BITMAP(mask, 64);
+     bool first = true;
+     int i, n;
+
+     buf[0] = '\0';
+
+     bitmap_from_u64(mask, stack_mask);
+     for_each_set_bit(i, mask, 64) {
+             n = snprintf(buf, buf_sz, "%s%d", first ? "" : ",", -(i + 1) * 8);
+             first = false;
+             buf += n;
+             buf_sz -= n;
+             if (buf_sz < 0)
+                     break;
+     }
+}
+
 /* For given verifier state backtrack_insn() is called from the last insn to
  * the first insn. Its purpose is to compute a bitmask of registers and
  * stack slots that needs precision in the parent verifier state.
@@ -3298,7 +3337,11 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
      if (insn->code == 0)
              return 0;
      if (env->log.level & BPF_LOG_LEVEL2) {
-             verbose(env, "regs=%x stack=%llx before ", bt_reg_mask(bt), bt_stack_mask(bt));
+             fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_reg_mask(bt));
Quoted 8 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3344

Quoted 4 lines
kernel/bpf/verifier.c +4 -0
+             verbose(env, "mark_precise: frame%d: regs(0x%x)=%s ",
+                     bt->frame, bt_reg_mask(bt), env->tmp_str_buf);
+             fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
+             verbose(env, "stack(0x%llx)=%s before ", bt_stack_mask(bt), env->tmp_str_buf);

Let's drop (0x%llx) part from regs and stack.
With nice human readable addition no one will be reading the hex anymore. It's just wasting screen real estate.

ack, will drop

Quoted 21 lines
Quoted 9 lines
kernel/bpf/verifier.c +8 -0
+     "mark_precise: frame0: last_idx 26 first_idx 20\
+     mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 25\
+     mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 24\
+     mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 23\
+     mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 22\
+     mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 20\
+     parent didn't have regs=4 stack=0 marks:\
+     mark_precise: frame0: last_idx 19 first_idx 10\

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx,

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+     mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 19\
+     mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 18\
+     mark_precise: frame0: regs(0x300)=r8,r9 stack(0x0)= before 17\
+     mark_precise: frame0: regs(0x201)=r0,r9 stack(0x0)= before 15\
+     mark_precise: frame0: regs(0x201)=r0,r9 stack(0x0)= before 14\
+     mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 13\
+     mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 12\
+     mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 11\
+     mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 10\
+     parent already had regs=0 stack=0 marks:",

This part would be much cleaner without (0x...)

author maintainer
#11 lore raw

PATCH 5/10 bpf: maintain bitmasks across all active frames in __mark_chain_precision

In reply to Andrii Nakryiko

Teach __mark_chain_precision logic to maintain register/stack masks across all active frames when going from child state to parent state. Currently this should be mostly no-op, as precision backtracking usually bails out when encountering subprog entry/exit.

It's not very apparent from the diff due to increased indentation, but the logic remains the same, except everything is done on specific fr frame index. Calls to bt_clear_reg() and bt_clear_slot() are replaced with frame-specific bt_clear_frame_reg() and bt_clear_frame_slot(), where frame index is passed explicitly, instead of using current frame number.

We also adjust logging to emit affected frame number. And we also add better logging of human-readable register and stack slot masks, similar to previous patch.

---
 kernel/bpf/verifier.c                         | 101 ++++++++++--------
 .../testing/selftests/bpf/verifier/precise.c  |  18 ++--
 2 files changed, 63 insertions(+), 56 deletions(-)
kernel/bpf/verifier.c +54 -47
index 8faf9170acf0..0b19b3d9af65 100644
@@ -3703,7 +3703,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 	struct bpf_func_state *func;
 	struct bpf_reg_state *reg;
 	bool skip_first = true;
-	int i, err;
+	int i, fr, err;
 
 	if (!env->bpf_capable)
 		return 0;
@@ -3812,56 +3812,63 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 		if (!st)
 			break;
 
-		func = st->frame[frame];
-		bitmap_from_u64(mask, bt_reg_mask(bt));
-		for_each_set_bit(i, mask, 32) {
-			reg = &func->regs[i];
-			if (reg->type != SCALAR_VALUE) {
-				bt_clear_reg(bt, i);
-				continue;
+		for (fr = bt->frame; fr >= 0; fr--) {
+			func = st->frame[fr];
+			bitmap_from_u64(mask, bt_frame_reg_mask(bt, fr));
+			for_each_set_bit(i, mask, 32) {
+				reg = &func->regs[i];
+				if (reg->type != SCALAR_VALUE) {
+					bt_clear_frame_reg(bt, fr, i);
+					continue;
+				}
+				if (reg->precise)
+					bt_clear_frame_reg(bt, fr, i);
+				else
+					reg->precise = true;
 			}
-			if (reg->precise)
-				bt_clear_reg(bt, i);
-			else
-				reg->precise = true;
-		}
 
-		bitmap_from_u64(mask, bt_stack_mask(bt));
-		for_each_set_bit(i, mask, 64) {
-			if (i >= func->allocated_stack / BPF_REG_SIZE) {
-				/* the sequence of instructions:
-				 * 2: (bf) r3 = r10
-				 * 3: (7b) *(u64 *)(r3 -8) = r0
-				 * 4: (79) r4 = *(u64 *)(r10 -8)
-				 * doesn't contain jmps. It's backtracked
-				 * as a single block.
-				 * During backtracking insn 3 is not recognized as
-				 * stack access, so at the end of backtracking
-				 * stack slot fp-8 is still marked in stack_mask.
-				 * However the parent state may not have accessed
-				 * fp-8 and it's "unallocated" stack space.
-				 * In such case fallback to conservative.
-				 */
-				mark_all_scalars_precise(env, st);
-				bt_reset(bt);
-				return 0;
-			}
+			bitmap_from_u64(mask, bt_frame_stack_mask(bt, fr));
+			for_each_set_bit(i, mask, 64) {
+				if (i >= func->allocated_stack / BPF_REG_SIZE) {
+					/* the sequence of instructions:
+					 * 2: (bf) r3 = r10
+					 * 3: (7b) *(u64 *)(r3 -8) = r0
+					 * 4: (79) r4 = *(u64 *)(r10 -8)
+					 * doesn't contain jmps. It's backtracked
+					 * as a single block.
+					 * During backtracking insn 3 is not recognized as
+					 * stack access, so at the end of backtracking
+					 * stack slot fp-8 is still marked in stack_mask.
+					 * However the parent state may not have accessed
+					 * fp-8 and it's "unallocated" stack space.
+					 * In such case fallback to conservative.
+					 */
+					mark_all_scalars_precise(env, st);
+					bt_reset(bt);
+					return 0;
+				}
 
-			if (!is_spilled_scalar_reg(&func->stack[i])) {
-				bt_clear_slot(bt, i);
-				continue;
+				if (!is_spilled_scalar_reg(&func->stack[i])) {
+					bt_clear_frame_slot(bt, fr, i);
+					continue;
+				}
+				reg = &func->stack[i].spilled_ptr;
+				if (reg->precise)
+					bt_clear_frame_slot(bt, fr, i);
+				else
+					reg->precise = true;
+			}
+			if (env->log.level & BPF_LOG_LEVEL2) {
+				fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
+					     bt_frame_reg_mask(bt, fr));
+				verbose(env, "mark_precise: frame%d: parent state regs(0x%x)=%s ",
+					fr, bt_frame_reg_mask(bt, fr), env->tmp_str_buf);
+				fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
+					       bt_frame_stack_mask(bt, fr));
+				verbose(env, "stack(0x%llx)=%s: ",
+					bt_frame_stack_mask(bt, fr), env->tmp_str_buf);
+				print_verifier_state(env, func, true);
 			}
-			reg = &func->stack[i].spilled_ptr;
-			if (reg->precise)
-				bt_clear_slot(bt, i);
-			else
-				reg->precise = true;
-		}
-		if (env->log.level & BPF_LOG_LEVEL2) {
-			verbose(env, "parent %s regs=%x stack=%llx marks:",
-				bt_bitcnt(bt) ? "didn't have" : "already had",
-				bt_reg_mask(bt), bt_stack_mask(bt));
-			print_verifier_state(env, func, true);
 		}
 
 		if (bt_bitcnt(bt) == 0)
tools/testing/selftests/bpf/verifier/precise.c +9 -9
index fce831667b06..ac9be4c576d6 100644
@@ -44,7 +44,7 @@
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 23\
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 22\
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 20\
-	parent didn't have regs=4 stack=0 marks:\
+	mark_precise: frame0: parent state regs(0x4)=r2 stack(0x0)=:\
 	mark_precise: frame0: last_idx 19 first_idx 10\
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 19\
 	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 18\
@@ -55,7 +55,7 @@
 	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 12\
 	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 11\
 	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 10\
-	parent already had regs=0 stack=0 marks:",
+	mark_precise: frame0: parent state regs(0x0)= stack(0x0)=:",
 },
 {
 	"precise: test 2",
@@ -104,15 +104,15 @@
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 24\
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 23\
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 22\
-	parent didn't have regs=4 stack=0 marks:\
+	mark_precise: frame0: parent state regs(0x4)=r2 stack(0x0)=:\
 	mark_precise: frame0: last_idx 20 first_idx 20\
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 20\
-	parent didn't have regs=4 stack=0 marks:\
+	mark_precise: frame0: parent state regs(0x4)=r2 stack(0x0)=:\
 	mark_precise: frame0: last_idx 19 first_idx 17\
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 19\
 	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 18\
 	mark_precise: frame0: regs(0x300)=r8,r9 stack(0x0)= before 17\
-	parent already had regs=0 stack=0 marks:",
+	mark_precise: frame0: parent state regs(0x0)= stack(0x0)=:",
 },
 {
 	"precise: cross frame pruning",
@@ -153,14 +153,14 @@
 	.prog_type = BPF_PROG_TYPE_XDP,
 	.flags = BPF_F_TEST_STATE_FREQ,
 	.errstr = "mark_precise: frame0: last_idx 5 first_idx 5\
-	parent didn't have regs=10 stack=0 marks:\
+	mark_precise: frame0: parent state regs(0x10)=r4 stack(0x0)=:\
 	mark_precise: frame0: last_idx 4 first_idx 2\
 	mark_precise: frame0: regs(0x10)=r4 stack(0x0)= before 4\
 	mark_precise: frame0: regs(0x10)=r4 stack(0x0)= before 3\
 	mark_precise: frame0: regs(0x0)= stack(0x1)=-8 before 2\
 	mark_precise: frame0: falling back to forcing all scalars precise\
 	mark_precise: frame0: last_idx 5 first_idx 5\
-	parent didn't have regs=1 stack=0 marks:",
+	mark_precise: frame0: parent state regs(0x1)=r0 stack(0x0)=:",
 	.result = VERBOSE_ACCEPT,
 	.retval = -1,
 },
@@ -179,7 +179,7 @@
 	.prog_type = BPF_PROG_TYPE_XDP,
 	.flags = BPF_F_TEST_STATE_FREQ,
 	.errstr = "mark_precise: frame0: last_idx 6 first_idx 6\
-	parent didn't have regs=10 stack=0 marks:\
+	mark_precise: frame0: parent state regs(0x10)=r4 stack(0x0)=:\
 	mark_precise: frame0: last_idx 5 first_idx 3\
 	mark_precise: frame0: regs(0x10)=r4 stack(0x0)= before 5\
 	mark_precise: frame0: regs(0x10)=r4 stack(0x0)= before 4\
@@ -188,7 +188,7 @@
 	force_precise: frame0: forcing r0 to be precise\
 	force_precise: frame0: forcing r0 to be precise\
 	mark_precise: frame0: last_idx 6 first_idx 6\
-	parent didn't have regs=1 stack=0 marks:\
+	mark_precise: frame0: parent state regs(0x1)=r0 stack(0x0)=:\
 	mark_precise: frame0: last_idx 5 first_idx 3\
 	mark_precise: frame0: regs(0x1)=r0 stack(0x0)= before 5",
 	.result = VERBOSE_ACCEPT,
Signature · 1 line
2.34.1
#12 lore raw

Re: [PATCH bpf-next 05/10] bpf: maintain bitmasks across all active frames in __mark_chain_precision

Quoted 37 lines · Andrii Nakryiko

Teach __mark_chain_precision logic to maintain register/stack masks across all active frames when going from child state to parent state. Currently this should be mostly no-op, as precision backtracking usually bails out when encountering subprog entry/exit.

It's not very apparent from the diff due to increased indentation, but the logic remains the same, except everything is done on specific fr frame index. Calls to bt_clear_reg() and bt_clear_slot() are replaced with frame-specific bt_clear_frame_reg() and bt_clear_frame_slot(), where frame index is passed explicitly, instead of using current frame number.

We also adjust logging to emit affected frame number. And we also add better logging of human-readable register and stack slot masks, similar to previous patch.

---
 kernel/bpf/verifier.c                         | 101 ++++++++++--------
 .../testing/selftests/bpf/verifier/precise.c  |  18 ++--
 2 files changed, 63 insertions(+), 56 deletions(-)
kernel/bpf/verifier.c +1 -1
index 8faf9170acf0..0b19b3d9af65 100644
@@ -3703,7 +3703,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 	struct bpf_func_state *func;
 	struct bpf_reg_state *reg;
 	bool skip_first = true;
-	int i, err;
+	int i, fr, err;
 
 	if (!env->bpf_capable)
 		return 0;
@@ -3812,56 +3812,63 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 		if (!st)

kernel/bpf/verifier.c static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r · line 3815

Quoted 10 lines
kernel/bpf/verifier.c +1 -7
 			break;
 
-		func = st->frame[frame];
-		bitmap_from_u64(mask, bt_reg_mask(bt));
-		for_each_set_bit(i, mask, 32) {
-			reg = &func->regs[i];
-			if (reg->type != SCALAR_VALUE) {
-				bt_clear_reg(bt, i);
-				continue;
+		for (fr = bt->frame; fr >= 0; fr--) {

I'm lost.
'frame' arg is now unused and the next patch passes -1 into it anyway? Probably this patch alone will break something and not bi-sectable?

Quoted 2 lines
kernel/bpf/verifier.c +2 -0
+			func = st->frame[fr];
+			bitmap_from_u64(mask, bt_frame_reg_mask(bt, fr));

..

Quoted 14 lines
tools/testing/selftests/bpf/verifier/precise.c +1 -1
index fce831667b06..ac9be4c576d6 100644
@@ -44,7 +44,7 @@
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 23\
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 22\
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 20\
-	parent didn't have regs=4 stack=0 marks:\
+	mark_precise: frame0: parent state regs(0x4)=r2 stack(0x0)=:\
 	mark_precise: frame0: last_idx 19 first_idx 10\
 	mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 19\
 	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 18\

tools/testing/selftests/bpf/verifier/precise.c line 58

Quoted 5 lines
tools/testing/selftests/bpf/verifier/precise.c +1 -1
@@ -55,7 +55,7 @@
 	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 12\
 	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 11\
 	mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 10\
-	parent already had regs=0 stack=0 marks:",
+	mark_precise: frame0: parent state regs(0x0)= stack(0x0)=:",

The effect of the patch looks minor, so it might be correct, just super confusing.

#13 lore raw

Re: [PATCH bpf-next 05/10] bpf: maintain bitmasks across all active frames in __mark_chain_precision

Quoted 41 lines · Alexei Starovoitov
Quoted 39 lines · Andrii Nakryiko

Teach __mark_chain_precision logic to maintain register/stack masks across all active frames when going from child state to parent state. Currently this should be mostly no-op, as precision backtracking usually bails out when encountering subprog entry/exit.

It's not very apparent from the diff due to increased indentation, but the logic remains the same, except everything is done on specific fr frame index. Calls to bt_clear_reg() and bt_clear_slot() are replaced with frame-specific bt_clear_frame_reg() and bt_clear_frame_slot(), where frame index is passed explicitly, instead of using current frame number.

We also adjust logging to emit affected frame number. And we also add better logging of human-readable register and stack slot masks, similar to previous patch.

---
 kernel/bpf/verifier.c                         | 101 ++++++++++--------
 .../testing/selftests/bpf/verifier/precise.c  |  18 ++--
 2 files changed, 63 insertions(+), 56 deletions(-)
kernel/bpf/verifier.c +1 -1
index 8faf9170acf0..0b19b3d9af65 100644
@@ -3703,7 +3703,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
      struct bpf_func_state *func;
      struct bpf_reg_state *reg;
      bool skip_first = true;
-     int i, err;
+     int i, fr, err;
 
      if (!env->bpf_capable)
              return 0;
@@ -3812,56 +3812,63 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
              if (!st)
                      break;
 
Quoted 11 lines

kernel/bpf/verifier.c static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r · line 3815

Quoted 8 lines
kernel/bpf/verifier.c +1 -7
-             func = st->frame[frame];
-             bitmap_from_u64(mask, bt_reg_mask(bt));
-             for_each_set_bit(i, mask, 32) {
-                     reg = &func->regs[i];
-                     if (reg->type != SCALAR_VALUE) {
-                             bt_clear_reg(bt, i);
-                             continue;
+             for (fr = bt->frame; fr >= 0; fr--) {

I'm lost.
'frame' arg is now unused and the next patch passes -1 into it anyway?

Patch #3 has bt_init(bt, frame); which sets bt->frame = frame, so since patch #3 we maintain a real frame number, it's just that the rest of backtrack_insn() logic makes sure we never change frame (which is why there should be no detectable change in behavior). Only in patch #8 I add inter-frame bits propagation and generally changing frame number with going into/out of subprog.

So it should be all good here.

Quoted 2 lines

Probably this patch alone will break something and not bi-sectable?

I actually painstakingly compiled kernel and selftests after each patch to make sure I'm not regressing anything (quite time consuming effort, but necessary), so bisectability should be preserved.

Quoted 16 lines
Quoted 2 lines
kernel/bpf/verifier.c +2 -0
+                     func = st->frame[fr];
+                     bitmap_from_u64(mask, bt_frame_reg_mask(bt, fr));

..

Quoted 13 lines
tools/testing/selftests/bpf/verifier/precise.c +1 -1
index fce831667b06..ac9be4c576d6 100644
@@ -44,7 +44,7 @@
      mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 23\
      mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 22\
      mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 20\
-     parent didn't have regs=4 stack=0 marks:\
+     mark_precise: frame0: parent state regs(0x4)=r2 stack(0x0)=:\
      mark_precise: frame0: last_idx 19 first_idx 10\
      mark_precise: frame0: regs(0x4)=r2 stack(0x0)= before 19\
      mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 18\
Quoted 8 lines

tools/testing/selftests/bpf/verifier/precise.c line 58

Quoted 6 lines
tools/testing/selftests/bpf/verifier/precise.c +1 -1
@@ -55,7 +55,7 @@
      mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 12\
      mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 11\
      mark_precise: frame0: regs(0x200)=r9 stack(0x0)= before 10\
-     parent already had regs=0 stack=0 marks:",
+     mark_precise: frame0: parent state regs(0x0)= stack(0x0)=:",

The effect of the patch looks minor, so it might be correct, just super confusing.

It's intentional that there is no effect. It's a preparation for the next patch where we do take advantage of having precision bits across multiple frames. I didn't want to bundle these indentation shifting changes with the logic changes.

#14 lore raw

Re: [PATCH bpf-next 05/10] bpf: maintain bitmasks across all active frames in __mark_chain_precision

Quoted 55 lines · Andrii Nakryiko
Quoted 41 lines · Alexei Starovoitov
Quoted 39 lines · Andrii Nakryiko

Teach __mark_chain_precision logic to maintain register/stack masks across all active frames when going from child state to parent state. Currently this should be mostly no-op, as precision backtracking usually bails out when encountering subprog entry/exit.

It's not very apparent from the diff due to increased indentation, but the logic remains the same, except everything is done on specific fr frame index. Calls to bt_clear_reg() and bt_clear_slot() are replaced with frame-specific bt_clear_frame_reg() and bt_clear_frame_slot(), where frame index is passed explicitly, instead of using current frame number.

We also adjust logging to emit affected frame number. And we also add better logging of human-readable register and stack slot masks, similar to previous patch.

---
 kernel/bpf/verifier.c                         | 101 ++++++++++--------
 .../testing/selftests/bpf/verifier/precise.c  |  18 ++--
 2 files changed, 63 insertions(+), 56 deletions(-)
kernel/bpf/verifier.c +1 -1
index 8faf9170acf0..0b19b3d9af65 100644
@@ -3703,7 +3703,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
      struct bpf_func_state *func;
      struct bpf_reg_state *reg;
      bool skip_first = true;
-     int i, err;
+     int i, fr, err;
 
      if (!env->bpf_capable)
              return 0;
@@ -3812,56 +3812,63 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
              if (!st)
                      break;
 
Quoted 11 lines

kernel/bpf/verifier.c static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r · line 3815

Quoted 8 lines
kernel/bpf/verifier.c +1 -7
-             func = st->frame[frame];
-             bitmap_from_u64(mask, bt_reg_mask(bt));
-             for_each_set_bit(i, mask, 32) {
-                     reg = &func->regs[i];
-                     if (reg->type != SCALAR_VALUE) {
-                             bt_clear_reg(bt, i);
-                             continue;
+             for (fr = bt->frame; fr >= 0; fr--) {

I'm lost.
'frame' arg is now unused and the next patch passes -1 into it anyway?

Quoted 6 lines

Patch #3 has bt_init(bt, frame); which sets bt->frame = frame, so since patch #3 we maintain a real frame number, it's just that the rest of backtrack_insn() logic makes sure we never change frame (which is why there should be no detectable change in behavior). Only in patch #8 I add inter-frame bits propagation and generally changing frame number with going into/out of subprog.

Got it, but then remove 'frame' arg in this patch too, since it's unused here and anything passed into it the later patches is also ignored.

author maintainer
#15 lore raw

PATCH 6/10 bpf: fix propagate_precision() logic for inner frames

In reply to Andrii Nakryiko

Fix propagate_precision() logic to perform propagation of all necessary registers and stack slots across all active frames *in one batch step*.

Doing this for each register/slot in each individual frame is wasteful, but the main problem is that backtracking of instruction in any frame except the deepest one just doesn't work. This is due to backtracking logic relying on jump history, and available jump history always starts (or ends, depending how you view it) in current frame. So, if prog A (frame #0) called subprog B (frame #1) and we need to propagate precision of, say, register R6 (callee-saved) within frame #0, we actually don't even know where jump history that corresponds to prog A even starts. We'd need to skip subprog part of jump history first to be able to do this.

Luckily, with struct backtrack_state and __mark_chain_precision() handling bitmasks tracking/propagation across all active frames at the same time (added in previous patch), propagate_precision() can be both fixed and sped up by setting all the necessary bits across all frames and then performing one __mark_chain_precision() pass. This makes it unnecessary to skip subprog parts of jump history.

We also improve logging along the way, to clearly specify which registers' and slots' precision markings are propagated within which frame.

Fixes 529409ea92d5 ("bpf: propagate precision across all frames, not just the last one")
---
 kernel/bpf/verifier.c | 49 +++++++++++++++++++++++++++----------------
 1 file changed, 31 insertions(+), 18 deletions(-)
kernel/bpf/verifier.c +31 -18
index 0b19b3d9af65..66d64ac10fb1 100644
@@ -3885,14 +3885,12 @@ int mark_chain_precision(struct bpf_verifier_env *env, int regno)
 	return __mark_chain_precision(env, env->cur_state->curframe, regno, -1);
 }
 
-static int mark_chain_precision_frame(struct bpf_verifier_env *env, int frame, int regno)
+static int mark_chain_precision_batch(struct bpf_verifier_env *env, int frame)
 {
-	return __mark_chain_precision(env, frame, regno, -1);
-}
-
-static int mark_chain_precision_stack_frame(struct bpf_verifier_env *env, int frame, int spi)
-{
-	return __mark_chain_precision(env, frame, -1, spi);
+	/* we assume env->bt was set outside with desired reg and stack masks
+	 * for all frames
+	 */
+	return __mark_chain_precision(env, frame, -1, -1);
 }
 
 static bool is_spillable_regtype(enum bpf_reg_type type)
@@ -15308,20 +15306,25 @@ static int propagate_precision(struct bpf_verifier_env *env,
 	struct bpf_reg_state *state_reg;
 	struct bpf_func_state *state;
 	int i, err = 0, fr;
+	bool first;
 
 	for (fr = old->curframe; fr >= 0; fr--) {
 		state = old->frame[fr];
 		state_reg = state->regs;
+		first = true;
 		for (i = 0; i < BPF_REG_FP; i++, state_reg++) {
 			if (state_reg->type != SCALAR_VALUE ||
 			    !state_reg->precise ||
 			    !(state_reg->live & REG_LIVE_READ))
 				continue;
-			if (env->log.level & BPF_LOG_LEVEL2)
-				verbose(env, "frame %d: propagating r%d\n", fr, i);
-			err = mark_chain_precision_frame(env, fr, i);
-			if (err < 0)
-				return err;
+			if (env->log.level & BPF_LOG_LEVEL2) {
+				if (first)
+					verbose(env, "frame %d: propagating r%d", fr, i);
+				else
+					verbose(env, ",r%d", i);
+			}
+			bt_set_frame_reg(&env->bt, fr, i);
+			first = false;
 		}
 
 		for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
@@ -15332,14 +15335,24 @@ static int propagate_precision(struct bpf_verifier_env *env,
 			    !state_reg->precise ||
 			    !(state_reg->live & REG_LIVE_READ))
 				continue;
-			if (env->log.level & BPF_LOG_LEVEL2)
-				verbose(env, "frame %d: propagating fp%d\n",
-					fr, (-i - 1) * BPF_REG_SIZE);
-			err = mark_chain_precision_stack_frame(env, fr, i);
-			if (err < 0)
-				return err;
+			if (env->log.level & BPF_LOG_LEVEL2) {
+				if (first)
+					verbose(env, "frame %d: propagating fp%d",
+						fr, (-i - 1) * BPF_REG_SIZE);
+				else
+					verbose(env, ",fp%d", (-i - 1) * BPF_REG_SIZE);
+			}
+			bt_set_frame_slot(&env->bt, fr, i);
+			first = false;
 		}
+		if (!first)
+			verbose(env, "\n");
 	}
+
+	err = mark_chain_precision_batch(env, old->curframe);
+	if (err < 0)
+		return err;
+
 	return 0;
 }
 
Signature · 1 line
2.34.1
#16 lore raw

Re: [PATCH bpf-next 06/10] bpf: fix propagate_precision() logic for inner frames

Quoted 101 lines · Andrii Nakryiko

Fix propagate_precision() logic to perform propagation of all necessary registers and stack slots across all active frames *in one batch step*.

Doing this for each register/slot in each individual frame is wasteful, but the main problem is that backtracking of instruction in any frame except the deepest one just doesn't work. This is due to backtracking logic relying on jump history, and available jump history always starts (or ends, depending how you view it) in current frame. So, if prog A (frame #0) called subprog B (frame #1) and we need to propagate precision of, say, register R6 (callee-saved) within frame #0, we actually don't even know where jump history that corresponds to prog A even starts. We'd need to skip subprog part of jump history first to be able to do this.

Luckily, with struct backtrack_state and __mark_chain_precision() handling bitmasks tracking/propagation across all active frames at the same time (added in previous patch), propagate_precision() can be both fixed and sped up by setting all the necessary bits across all frames and then performing one __mark_chain_precision() pass. This makes it unnecessary to skip subprog parts of jump history.

We also improve logging along the way, to clearly specify which registers' and slots' precision markings are propagated within which frame.

Fixes 529409ea92d5 ("bpf: propagate precision across all frames, not just the last one")
---
 kernel/bpf/verifier.c | 49 +++++++++++++++++++++++++++----------------
 1 file changed, 31 insertions(+), 18 deletions(-)
kernel/bpf/verifier.c +20 -18
index 0b19b3d9af65..66d64ac10fb1 100644
@@ -3885,14 +3885,12 @@ int mark_chain_precision(struct bpf_verifier_env *env, int regno)
 	return __mark_chain_precision(env, env->cur_state->curframe, regno, -1);
 }
 
-static int mark_chain_precision_frame(struct bpf_verifier_env *env, int frame, int regno)
+static int mark_chain_precision_batch(struct bpf_verifier_env *env, int frame)
 {
-	return __mark_chain_precision(env, frame, regno, -1);
-}
-
-static int mark_chain_precision_stack_frame(struct bpf_verifier_env *env, int frame, int spi)
-{
-	return __mark_chain_precision(env, frame, -1, spi);
+	/* we assume env->bt was set outside with desired reg and stack masks
+	 * for all frames
+	 */
+	return __mark_chain_precision(env, frame, -1, -1);
 }
 
 static bool is_spillable_regtype(enum bpf_reg_type type)
@@ -15308,20 +15306,25 @@ static int propagate_precision(struct bpf_verifier_env *env,
 	struct bpf_reg_state *state_reg;
 	struct bpf_func_state *state;
 	int i, err = 0, fr;
+	bool first;
 
 	for (fr = old->curframe; fr >= 0; fr--) {
 		state = old->frame[fr];
 		state_reg = state->regs;
+		first = true;
 		for (i = 0; i < BPF_REG_FP; i++, state_reg++) {
 			if (state_reg->type != SCALAR_VALUE ||
 			    !state_reg->precise ||
 			    !(state_reg->live & REG_LIVE_READ))
 				continue;
-			if (env->log.level & BPF_LOG_LEVEL2)
-				verbose(env, "frame %d: propagating r%d\n", fr, i);
-			err = mark_chain_precision_frame(env, fr, i);
-			if (err < 0)
-				return err;
+			if (env->log.level & BPF_LOG_LEVEL2) {
+				if (first)
+					verbose(env, "frame %d: propagating r%d", fr, i);
+				else
+					verbose(env, ",r%d", i);
+			}
+			bt_set_frame_reg(&env->bt, fr, i);
+			first = false;
 		}
 
 		for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
@@ -15332,14 +15335,24 @@ static int propagate_precision(struct bpf_verifier_env *env,
 			    !state_reg->precise ||
 			    !(state_reg->live & REG_LIVE_READ))
 				continue;
-			if (env->log.level & BPF_LOG_LEVEL2)
-				verbose(env, "frame %d: propagating fp%d\n",
-					fr, (-i - 1) * BPF_REG_SIZE);
-			err = mark_chain_precision_stack_frame(env, fr, i);
-			if (err < 0)
-				return err;
+			if (env->log.level & BPF_LOG_LEVEL2) {
+				if (first)
+					verbose(env, "frame %d: propagating fp%d",
+						fr, (-i - 1) * BPF_REG_SIZE);
+				else

kernel/bpf/verifier.c static int propagate_precision(struct bpf_verifier_env *env, · line 15352

Quoted 10 lines
kernel/bpf/verifier.c +8 -0
+					verbose(env, ",fp%d", (-i - 1) * BPF_REG_SIZE);
+			}
+			bt_set_frame_slot(&env->bt, fr, i);
+			first = false;
 		}
+		if (!first)
+			verbose(env, "\n");
 	}
+
+	err = mark_chain_precision_batch(env, old->curframe);

The optimization makes sense, sort-of, but 'first' flag is to separate frames on different lines ?
The code in propagate_precision() before mark_chain_precision_batch() will only print '... propagating...' few times.
regs and stack will be on one line anyway.
Is it that ugly to print all frames on one line in practice?
The 2nd and 3rd frames will be on one line too. Only 1st frame is on its own line? I'm not getting the idea of 'first'.

#17 lore raw

Re: [PATCH bpf-next 06/10] bpf: fix propagate_precision() logic for inner frames

Quoted 108 lines · Alexei Starovoitov
Quoted 106 lines · Andrii Nakryiko

Fix propagate_precision() logic to perform propagation of all necessary registers and stack slots across all active frames *in one batch step*.

Doing this for each register/slot in each individual frame is wasteful, but the main problem is that backtracking of instruction in any frame except the deepest one just doesn't work. This is due to backtracking logic relying on jump history, and available jump history always starts (or ends, depending how you view it) in current frame. So, if prog A (frame #0) called subprog B (frame #1) and we need to propagate precision of, say, register R6 (callee-saved) within frame #0, we actually don't even know where jump history that corresponds to prog A even starts. We'd need to skip subprog part of jump history first to be able to do this.

Luckily, with struct backtrack_state and __mark_chain_precision() handling bitmasks tracking/propagation across all active frames at the same time (added in previous patch), propagate_precision() can be both fixed and sped up by setting all the necessary bits across all frames and then performing one __mark_chain_precision() pass. This makes it unnecessary to skip subprog parts of jump history.

We also improve logging along the way, to clearly specify which registers' and slots' precision markings are propagated within which frame.

Fixes 529409ea92d5 ("bpf: propagate precision across all frames, not just the last one")
---
 kernel/bpf/verifier.c | 49 +++++++++++++++++++++++++++----------------
 1 file changed, 31 insertions(+), 18 deletions(-)
kernel/bpf/verifier.c +24 -18
index 0b19b3d9af65..66d64ac10fb1 100644
@@ -3885,14 +3885,12 @@ int mark_chain_precision(struct bpf_verifier_env *env, int regno)
      return __mark_chain_precision(env, env->cur_state->curframe, regno, -1);
 }
 
-static int mark_chain_precision_frame(struct bpf_verifier_env *env, int frame, int regno)
+static int mark_chain_precision_batch(struct bpf_verifier_env *env, int frame)
 {
-     return __mark_chain_precision(env, frame, regno, -1);
-}
-
-static int mark_chain_precision_stack_frame(struct bpf_verifier_env *env, int frame, int spi)
-{
-     return __mark_chain_precision(env, frame, -1, spi);
+     /* we assume env->bt was set outside with desired reg and stack masks
+      * for all frames
+      */
+     return __mark_chain_precision(env, frame, -1, -1);
 }
 
 static bool is_spillable_regtype(enum bpf_reg_type type)
@@ -15308,20 +15306,25 @@ static int propagate_precision(struct bpf_verifier_env *env,
      struct bpf_reg_state *state_reg;
      struct bpf_func_state *state;
      int i, err = 0, fr;
+     bool first;
 
      for (fr = old->curframe; fr >= 0; fr--) {
              state = old->frame[fr];
              state_reg = state->regs;
+             first = true;
              for (i = 0; i < BPF_REG_FP; i++, state_reg++) {
                      if (state_reg->type != SCALAR_VALUE ||
                          !state_reg->precise ||
                          !(state_reg->live & REG_LIVE_READ))
                              continue;
-                     if (env->log.level & BPF_LOG_LEVEL2)
-                             verbose(env, "frame %d: propagating r%d\n", fr, i);
-                     err = mark_chain_precision_frame(env, fr, i);
-                     if (err < 0)
-                             return err;
+                     if (env->log.level & BPF_LOG_LEVEL2) {
+                             if (first)
+                                     verbose(env, "frame %d: propagating r%d", fr, i);
+                             else
+                                     verbose(env, ",r%d", i);
+                     }
+                     bt_set_frame_reg(&env->bt, fr, i);
+                     first = false;
              }
 
              for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
@@ -15332,14 +15335,24 @@ static int propagate_precision(struct bpf_verifier_env *env,
                          !state_reg->precise ||
                          !(state_reg->live & REG_LIVE_READ))
                              continue;
-                     if (env->log.level & BPF_LOG_LEVEL2)
-                             verbose(env, "frame %d: propagating fp%d\n",
-                                     fr, (-i - 1) * BPF_REG_SIZE);
-                     err = mark_chain_precision_stack_frame(env, fr, i);
-                     if (err < 0)
-                             return err;
+                     if (env->log.level & BPF_LOG_LEVEL2) {
+                             if (first)
+                                     verbose(env, "frame %d: propagating fp%d",
+                                             fr, (-i - 1) * BPF_REG_SIZE);
+                             else
+                                     verbose(env, ",fp%d", (-i - 1) * BPF_REG_SIZE);
+                     }
+                     bt_set_frame_slot(&env->bt, fr, i);
+                     first = false;
              }
Quoted 8 lines

kernel/bpf/verifier.c static int propagate_precision(struct bpf_verifier_env *env, · line 15352

Quoted 5 lines
kernel/bpf/verifier.c +4 -0
+             if (!first)
+                     verbose(env, "\n");
      }
+
+     err = mark_chain_precision_batch(env, old->curframe);

The optimization makes sense, sort-of, but 'first' flag is to separate frames on different lines ?

Yes, first is purely for formatting. Each frame will have a separate line, but all registers and stack frames for that frame will be on that single line, like so:

frame 1: propagating r1,r2,r3,fp-8,fp-16
frame 0: propagating r3,r9,fp-120

Quoted 6 lines

The code in propagate_precision() before mark_chain_precision_batch() will only print '... propagating...' few times.
regs and stack will be on one line anyway.
Is it that ugly to print all frames on one line in practice?
The 2nd and 3rd frames will be on one line too. Only 1st frame is on its own line? I'm not getting the idea of 'first'.

So the above example shows how it looks now. Same situation without this formatting would look like:

frame 1: propagating r1
frame 1: propagating r2
frame 1: propagating r3
frame 1: propagating fp-8
frame 1: propagating fp-16
frame 0: propagating r3
frame 0: propagating r9
frame 0: propagating fp-120

I wanted to have the same formatted set of registers and stack slots, so make it more succinct and printing entire sets of registers/stack slots, consistent with the backtrack log format.

#18 lore raw

Re: [PATCH bpf-next 06/10] bpf: fix propagate_precision() logic for inner frames

Quoted 119 lines · Andrii Nakryiko
Quoted 108 lines · Alexei Starovoitov
Quoted 106 lines · Andrii Nakryiko

Fix propagate_precision() logic to perform propagation of all necessary registers and stack slots across all active frames *in one batch step*.

Doing this for each register/slot in each individual frame is wasteful, but the main problem is that backtracking of instruction in any frame except the deepest one just doesn't work. This is due to backtracking logic relying on jump history, and available jump history always starts (or ends, depending how you view it) in current frame. So, if prog A (frame #0) called subprog B (frame #1) and we need to propagate precision of, say, register R6 (callee-saved) within frame #0, we actually don't even know where jump history that corresponds to prog A even starts. We'd need to skip subprog part of jump history first to be able to do this.

Luckily, with struct backtrack_state and __mark_chain_precision() handling bitmasks tracking/propagation across all active frames at the same time (added in previous patch), propagate_precision() can be both fixed and sped up by setting all the necessary bits across all frames and then performing one __mark_chain_precision() pass. This makes it unnecessary to skip subprog parts of jump history.

We also improve logging along the way, to clearly specify which registers' and slots' precision markings are propagated within which frame.

Fixes 529409ea92d5 ("bpf: propagate precision across all frames, not just the last one")
---
 kernel/bpf/verifier.c | 49 +++++++++++++++++++++++++++----------------
 1 file changed, 31 insertions(+), 18 deletions(-)
kernel/bpf/verifier.c +24 -18
index 0b19b3d9af65..66d64ac10fb1 100644
@@ -3885,14 +3885,12 @@ int mark_chain_precision(struct bpf_verifier_env *env, int regno)
      return __mark_chain_precision(env, env->cur_state->curframe, regno, -1);
 }
 
-static int mark_chain_precision_frame(struct bpf_verifier_env *env, int frame, int regno)
+static int mark_chain_precision_batch(struct bpf_verifier_env *env, int frame)
 {
-     return __mark_chain_precision(env, frame, regno, -1);
-}
-
-static int mark_chain_precision_stack_frame(struct bpf_verifier_env *env, int frame, int spi)
-{
-     return __mark_chain_precision(env, frame, -1, spi);
+     /* we assume env->bt was set outside with desired reg and stack masks
+      * for all frames
+      */
+     return __mark_chain_precision(env, frame, -1, -1);
 }
 
 static bool is_spillable_regtype(enum bpf_reg_type type)
@@ -15308,20 +15306,25 @@ static int propagate_precision(struct bpf_verifier_env *env,
      struct bpf_reg_state *state_reg;
      struct bpf_func_state *state;
      int i, err = 0, fr;
+     bool first;
 
      for (fr = old->curframe; fr >= 0; fr--) {
              state = old->frame[fr];
              state_reg = state->regs;
+             first = true;
              for (i = 0; i < BPF_REG_FP; i++, state_reg++) {
                      if (state_reg->type != SCALAR_VALUE ||
                          !state_reg->precise ||
                          !(state_reg->live & REG_LIVE_READ))
                              continue;
-                     if (env->log.level & BPF_LOG_LEVEL2)
-                             verbose(env, "frame %d: propagating r%d\n", fr, i);
-                     err = mark_chain_precision_frame(env, fr, i);
-                     if (err < 0)
-                             return err;
+                     if (env->log.level & BPF_LOG_LEVEL2) {
+                             if (first)
+                                     verbose(env, "frame %d: propagating r%d", fr, i);
+                             else
+                                     verbose(env, ",r%d", i);
+                     }
+                     bt_set_frame_reg(&env->bt, fr, i);
+                     first = false;
              }
 
              for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
@@ -15332,14 +15335,24 @@ static int propagate_precision(struct bpf_verifier_env *env,
                          !state_reg->precise ||
                          !(state_reg->live & REG_LIVE_READ))
                              continue;
-                     if (env->log.level & BPF_LOG_LEVEL2)
-                             verbose(env, "frame %d: propagating fp%d\n",
-                                     fr, (-i - 1) * BPF_REG_SIZE);
-                     err = mark_chain_precision_stack_frame(env, fr, i);
-                     if (err < 0)
-                             return err;
+                     if (env->log.level & BPF_LOG_LEVEL2) {
+                             if (first)
+                                     verbose(env, "frame %d: propagating fp%d",
+                                             fr, (-i - 1) * BPF_REG_SIZE);
+                             else
+                                     verbose(env, ",fp%d", (-i - 1) * BPF_REG_SIZE);
+                     }
+                     bt_set_frame_slot(&env->bt, fr, i);
+                     first = false;
              }
Quoted 8 lines

kernel/bpf/verifier.c static int propagate_precision(struct bpf_verifier_env *env, · line 15352

Quoted 5 lines
kernel/bpf/verifier.c +4 -0
+             if (!first)
+                     verbose(env, "\n");
      }
+
+     err = mark_chain_precision_batch(env, old->curframe);

The optimization makes sense, sort-of, but 'first' flag is to separate frames on different lines ?

Quoted 6 lines

Yes, first is purely for formatting. Each frame will have a separate line, but all registers and stack frames for that frame will be on that single line, like so:

frame 1: propagating r1,r2,r3,fp-8,fp-16
frame 0: propagating r3,r9,fp-120

I see. The example of the output helped a lot.
Now the code makes sense. Pls include it in a commit or a comment.

#19 lore raw

Re: [PATCH bpf-next 06/10] bpf: fix propagate_precision() logic for inner frames

In reply to Andrii Nakryiko

kernel/bpf/verifier.c static int propagate_precision(struct bpf_verifier_env *env, · line 15352

Quoted 2 lines · Andrii Nakryiko
kernel/bpf/verifier.c +2 -0
+
+	err = mark_chain_precision_batch(env, old->curframe);

I think I'm sort-of starting to get it, but
above should be env->cur_state->curframe instead of old->curframe, no? mark_chain_precision_batch will be using branch history of current frame. __mark_chain_precision() always operates on
struct bpf_verifier_state *st = env->cur_state;

#20 lore raw

Re: [PATCH bpf-next 06/10] bpf: fix propagate_precision() logic for inner frames

Quoted 10 lines · Alexei Starovoitov

kernel/bpf/verifier.c static int propagate_precision(struct bpf_verifier_env *env, · line 15352

Quoted 2 lines · Andrii Nakryiko
kernel/bpf/verifier.c +2 -0
+
+     err = mark_chain_precision_batch(env, old->curframe);

I think I'm sort-of starting to get it, but
above should be env->cur_state->curframe instead of old->curframe, no? mark_chain_precision_batch will be using branch history of current frame. __mark_chain_precision() always operates on
struct bpf_verifier_state *st = env->cur_state;

wait. patch 5 made __mark_chain_precision() to ignore 'frame' argument. Why did you keep it in mark_chain_precision_batch() and pass it here?

#21 lore raw

Re: [PATCH bpf-next 06/10] bpf: fix propagate_precision() logic for inner frames

Quoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Alexei Starovoitov
Quoted 2 lines · Andrii Nakryiko
kernel/bpf/verifier.c +2 -0
+
+     err = mark_chain_precision_batch(env, old->curframe);
Quoted 8 lines
Quoted 5 lines

I think I'm sort-of starting to get it, but
above should be env->cur_state->curframe instead of old->curframe, no? mark_chain_precision_batch will be using branch history of current frame. __mark_chain_precision() always operates on
struct bpf_verifier_state *st = env->cur_state;

wait. patch 5 made __mark_chain_precision() to ignore 'frame' argument. Why did you keep it in mark_chain_precision_batch() and pass it here?

Ok, few things here.

1. We don't ignore frame, bt_init(bt, frame) is what sets that for backtrack_state. Currently we get it as input argument, but see below, we can get it from the current state now.

2. old->curframe and cur_state->curframe should be the same because two states are equivalent. I chose to pass old->curframe because that's what existing for loop was using for iteration.

But you are right, it's a missed opportunity to simplify. __mark_chain_precision() always works with env->cur_state and should always start from its deepest frame, so I'll drop the frame argument from __mark_chain_precision() completely and will be getting it from cur_state. Good observation!

author maintainer
#22 lore raw

PATCH 7/10 bpf: fix mark_all_scalars_precise use in mark_chain_precision

In reply to Andrii Nakryiko

When precision backtracking bails out due to some unsupported sequence of instructions (e.g., stack access through register other than r10), we need to mark all SCALAR registers as precise to be safe. Currently, though, we mark SCALARs precise only starting from the state we detected unsupported condition, which could be one of the parent states of the actual current state. This will leave some registers potentially not marked as precise, even though they should. So make sure we start marking scalars as precise from current state (env->cur_state).

Further, we don't currently detect a situation when we end up with some stack slots marked as needing precision, but we ran out of available states to find the instructions that populate those stack slots. This is akin the i >= func->allocated_stack / BPF_REG_SIZE check and should be handled similarly by falling back to marking all SCALARs precise. Add this check when we run out of states.

---
 kernel/bpf/verifier.c                          | 16 +++++++++++++---
 tools/testing/selftests/bpf/verifier/precise.c |  9 +++++----
 2 files changed, 18 insertions(+), 7 deletions(-)
kernel/bpf/verifier.c +13 -3
index 66d64ac10fb1..35f34c977819 100644
@@ -3781,7 +3781,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 				err = backtrack_insn(env, i, bt);
 			}
 			if (err == -ENOTSUPP) {
-				mark_all_scalars_precise(env, st);
+				mark_all_scalars_precise(env, env->cur_state);
 				bt_reset(bt);
 				return 0;
 			} else if (err) {
@@ -3843,7 +3843,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 					 * fp-8 and it's "unallocated" stack space.
 					 * In such case fallback to conservative.
 					 */
-					mark_all_scalars_precise(env, st);
+					mark_all_scalars_precise(env, env->cur_state);
 					bt_reset(bt);
 					return 0;
 				}
@@ -3872,11 +3872,21 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 		}
 
 		if (bt_bitcnt(bt) == 0)
-			break;
+			return 0;
 
 		last_idx = st->last_insn_idx;
 		first_idx = st->first_insn_idx;
 	}
+
+	/* if we still have requested precise regs or slots, we missed
+	 * something (e.g., stack access through non-r10 register), so
+	 * fallback to marking all precise
+	 */
+	if (bt_bitcnt(bt) != 0) {
+		mark_all_scalars_precise(env, env->cur_state);
+		bt_reset(bt);
+	}
+
 	return 0;
 }
 
tools/testing/selftests/bpf/verifier/precise.c +5 -4
index ac9be4c576d6..f4f65cb9f9b1 100644
@@ -159,8 +159,9 @@
 	mark_precise: frame0: regs(0x10)=r4 stack(0x0)= before 3\
 	mark_precise: frame0: regs(0x0)= stack(0x1)=-8 before 2\
 	mark_precise: frame0: falling back to forcing all scalars precise\
+	force_precise: frame0: forcing r0 to be precise\
 	mark_precise: frame0: last_idx 5 first_idx 5\
-	mark_precise: frame0: parent state regs(0x1)=r0 stack(0x0)=:",
+	mark_precise: frame0: parent state regs(0x0)= stack(0x0)=:",
 	.result = VERBOSE_ACCEPT,
 	.retval = -1,
 },
@@ -187,10 +188,10 @@
 	mark_precise: frame0: falling back to forcing all scalars precise\
 	force_precise: frame0: forcing r0 to be precise\
 	force_precise: frame0: forcing r0 to be precise\
+	force_precise: frame0: forcing r0 to be precise\
+	force_precise: frame0: forcing r0 to be precise\
 	mark_precise: frame0: last_idx 6 first_idx 6\
-	mark_precise: frame0: parent state regs(0x1)=r0 stack(0x0)=:\
-	mark_precise: frame0: last_idx 5 first_idx 3\
-	mark_precise: frame0: regs(0x1)=r0 stack(0x0)= before 5",
+	mark_precise: frame0: parent state regs(0x0)= stack(0x0)=:",
 	.result = VERBOSE_ACCEPT,
 	.retval = -1,
 },
Signature · 1 line
2.34.1
#23 lore raw

Re: [PATCH bpf-next 07/10] bpf: fix mark_all_scalars_precise use in mark_chain_precision

Quoted 53 lines · Andrii Nakryiko

When precision backtracking bails out due to some unsupported sequence of instructions (e.g., stack access through register other than r10), we need to mark all SCALAR registers as precise to be safe. Currently, though, we mark SCALARs precise only starting from the state we detected unsupported condition, which could be one of the parent states of the actual current state. This will leave some registers potentially not marked as precise, even though they should. So make sure we start marking scalars as precise from current state (env->cur_state).

Further, we don't currently detect a situation when we end up with some stack slots marked as needing precision, but we ran out of available states to find the instructions that populate those stack slots. This is akin the i >= func->allocated_stack / BPF_REG_SIZE check and should be handled similarly by falling back to marking all SCALARs precise. Add this check when we run out of states.

---
 kernel/bpf/verifier.c                          | 16 +++++++++++++---
 tools/testing/selftests/bpf/verifier/precise.c |  9 +++++----
 2 files changed, 18 insertions(+), 7 deletions(-)
kernel/bpf/verifier.c +3 -3
index 66d64ac10fb1..35f34c977819 100644
@@ -3781,7 +3781,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 				err = backtrack_insn(env, i, bt);
 			}
 			if (err == -ENOTSUPP) {
-				mark_all_scalars_precise(env, st);
+				mark_all_scalars_precise(env, env->cur_state);
 				bt_reset(bt);
 				return 0;
 			} else if (err) {
@@ -3843,7 +3843,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 					 * fp-8 and it's "unallocated" stack space.
 					 * In such case fallback to conservative.
 					 */
-					mark_all_scalars_precise(env, st);
+					mark_all_scalars_precise(env, env->cur_state);
 					bt_reset(bt);
 					return 0;
 				}
@@ -3872,11 +3872,21 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 		}
 
 		if (bt_bitcnt(bt) == 0)
-			break;
+			return 0;
 
 		last_idx = st->last_insn_idx;
 		first_idx = st->first_insn_idx;

kernel/bpf/verifier.c static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r · line 3888

Quoted 10 lines
kernel/bpf/verifier.c +9 -0
 	}
+
+	/* if we still have requested precise regs or slots, we missed
+	 * something (e.g., stack access through non-r10 register), so
+	 * fallback to marking all precise
+	 */
+	if (bt_bitcnt(bt) != 0) {
+		mark_all_scalars_precise(env, env->cur_state);
+		bt_reset(bt);
+	}

We get here only after:
st = st->parent;
if (!st)

          break;

which is the case when we reach the very beginning of the program (parent == NULL) and there are still regs or stack with marks.
That's a situation when backtracking encountered something we didn't foresee. Some new condition. Currently we don't have selftest that trigger this.
So as a defensive mechanism it makes sense to do mark_all_scalars_precise(env, env->cur_state); Probably needs verbose("verifier backtracking bug") as well.

But for the other two cases mark_all_scalars_precise(env, st); is safe.
What's the reason to mark everything precise from the very beginning of backtracking (last seen state == current state). Since unsupported condition was in the middle it's safe to mark from that condition till start of prog.

#24 lore raw

Re: [PATCH bpf-next 07/10] bpf: fix mark_all_scalars_precise use in mark_chain_precision

Quoted 69 lines · Alexei Starovoitov
Quoted 53 lines · Andrii Nakryiko

When precision backtracking bails out due to some unsupported sequence of instructions (e.g., stack access through register other than r10), we need to mark all SCALAR registers as precise to be safe. Currently, though, we mark SCALARs precise only starting from the state we detected unsupported condition, which could be one of the parent states of the actual current state. This will leave some registers potentially not marked as precise, even though they should. So make sure we start marking scalars as precise from current state (env->cur_state).

Further, we don't currently detect a situation when we end up with some stack slots marked as needing precision, but we ran out of available states to find the instructions that populate those stack slots. This is akin the i >= func->allocated_stack / BPF_REG_SIZE check and should be handled similarly by falling back to marking all SCALARs precise. Add this check when we run out of states.

---
 kernel/bpf/verifier.c                          | 16 +++++++++++++---
 tools/testing/selftests/bpf/verifier/precise.c |  9 +++++----
 2 files changed, 18 insertions(+), 7 deletions(-)
kernel/bpf/verifier.c +3 -3
index 66d64ac10fb1..35f34c977819 100644
@@ -3781,7 +3781,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
                              err = backtrack_insn(env, i, bt);
                      }
                      if (err == -ENOTSUPP) {
-                             mark_all_scalars_precise(env, st);
+                             mark_all_scalars_precise(env, env->cur_state);
                              bt_reset(bt);
                              return 0;
                      } else if (err) {
@@ -3843,7 +3843,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
                                       * fp-8 and it's "unallocated" stack space.
                                       * In such case fallback to conservative.
                                       */
-                                     mark_all_scalars_precise(env, st);
+                                     mark_all_scalars_precise(env, env->cur_state);
                                      bt_reset(bt);
                                      return 0;
                              }
@@ -3872,11 +3872,21 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
              }
 
              if (bt_bitcnt(bt) == 0)
-                     break;
+                     return 0;
 
              last_idx = st->last_insn_idx;
              first_idx = st->first_insn_idx;

kernel/bpf/verifier.c static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r · line 3888

Quoted 10 lines
kernel/bpf/verifier.c +9 -0
      }
+
+     /* if we still have requested precise regs or slots, we missed
+      * something (e.g., stack access through non-r10 register), so
+      * fallback to marking all precise
+      */
+     if (bt_bitcnt(bt) != 0) {
+             mark_all_scalars_precise(env, env->cur_state);
+             bt_reset(bt);
+     }

We get here only after:
st = st->parent;
if (!st)

Quoted 8 lines
          break;

which is the case when we reach the very beginning of the program (parent == NULL) and there are still regs or stack with marks.
That's a situation when backtracking encountered something we didn't foresee. Some new condition. Currently we don't have selftest that trigger this.
So as a defensive mechanism it makes sense to do mark_all_scalars_precise(env, env->cur_state); Probably needs verbose("verifier backtracking bug") as well.

I hesitated to add a bug message because of a known case where this could happen: stack access through non-r10 register. So it's not a bug, it's similar to -ENOTSUPP cases above, kind of expected, if rare.

Quoted 4 lines

But for the other two cases mark_all_scalars_precise(env, st); is safe.
What's the reason to mark everything precise from the very beginning of backtracking (last seen state == current state). Since unsupported condition was in the middle it's safe to mark from that condition till start of prog.

So I don't have a constructed example and it's more of a hunch. But let's use the same stack slot precision when some instructions use non-r10 registers. We can't always reliably and immediately detect this situation, so the point at which we realize that something is off could be in parent state, but that same slot might be used in children state and if we don't mark everything precise from the current state, then we are running a risk of leaving some of the slots as imprecise.

Before patch #8 similar situation could be also with r0 returned from static subprog. Something like this:

P2: call subprog
<<- checkpoint ->>
P1: r1 = r0;
<<- checkpoint ->>
CUR: r2 = <map value pointer>

     r2 += r1   <--- need to mark r0 precise and propagate all the way

to P2 and beyond

Precision propagation will be triggered in CUR state. Will go through P1, come back to P2 and (before patch #8 changes) will realize this is unsupported case. Currently we'll mark R0 precise only in P2 and its parents (could be also r6-r9 registers situation). But really we should make sure that P1 also has r1 marked as precise.

So as a general rule, I think the only safe default is to mark everything precise from current state.

Is the above convincing or should I revert back to old behavior?

#25 lore raw

Re: [PATCH bpf-next 07/10] bpf: fix mark_all_scalars_precise use in mark_chain_precision

Quoted 80 lines · Andrii Nakryiko
Quoted 69 lines · Alexei Starovoitov
Quoted 53 lines · Andrii Nakryiko

When precision backtracking bails out due to some unsupported sequence of instructions (e.g., stack access through register other than r10), we need to mark all SCALAR registers as precise to be safe. Currently, though, we mark SCALARs precise only starting from the state we detected unsupported condition, which could be one of the parent states of the actual current state. This will leave some registers potentially not marked as precise, even though they should. So make sure we start marking scalars as precise from current state (env->cur_state).

Further, we don't currently detect a situation when we end up with some stack slots marked as needing precision, but we ran out of available states to find the instructions that populate those stack slots. This is akin the i >= func->allocated_stack / BPF_REG_SIZE check and should be handled similarly by falling back to marking all SCALARs precise. Add this check when we run out of states.

---
 kernel/bpf/verifier.c                          | 16 +++++++++++++---
 tools/testing/selftests/bpf/verifier/precise.c |  9 +++++----
 2 files changed, 18 insertions(+), 7 deletions(-)
kernel/bpf/verifier.c +3 -3
index 66d64ac10fb1..35f34c977819 100644
@@ -3781,7 +3781,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
                              err = backtrack_insn(env, i, bt);
                      }
                      if (err == -ENOTSUPP) {
-                             mark_all_scalars_precise(env, st);
+                             mark_all_scalars_precise(env, env->cur_state);
                              bt_reset(bt);
                              return 0;
                      } else if (err) {
@@ -3843,7 +3843,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
                                       * fp-8 and it's "unallocated" stack space.
                                       * In such case fallback to conservative.
                                       */
-                                     mark_all_scalars_precise(env, st);
+                                     mark_all_scalars_precise(env, env->cur_state);
                                      bt_reset(bt);
                                      return 0;
                              }
@@ -3872,11 +3872,21 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
              }
 
              if (bt_bitcnt(bt) == 0)
-                     break;
+                     return 0;
 
              last_idx = st->last_insn_idx;
              first_idx = st->first_insn_idx;

kernel/bpf/verifier.c static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r · line 3888

Quoted 10 lines
kernel/bpf/verifier.c +9 -0
      }
+
+     /* if we still have requested precise regs or slots, we missed
+      * something (e.g., stack access through non-r10 register), so
+      * fallback to marking all precise
+      */
+     if (bt_bitcnt(bt) != 0) {
+             mark_all_scalars_precise(env, env->cur_state);
+             bt_reset(bt);
+     }

We get here only after:
st = st->parent;
if (!st)

Quoted 8 lines
          break;

which is the case when we reach the very beginning of the program (parent == NULL) and there are still regs or stack with marks.
That's a situation when backtracking encountered something we didn't foresee. Some new condition. Currently we don't have selftest that trigger this.
So as a defensive mechanism it makes sense to do mark_all_scalars_precise(env, env->cur_state); Probably needs verbose("verifier backtracking bug") as well.

Quoted 3 lines

I hesitated to add a bug message because of a known case where this could happen: stack access through non-r10 register. So it's not a bug, it's similar to -ENOTSUPP cases above, kind of expected, if rare.

fair enough. I'm ok to skip verbose().

Quoted 22 lines
Quoted 4 lines

But for the other two cases mark_all_scalars_precise(env, st); is safe.
What's the reason to mark everything precise from the very beginning of backtracking (last seen state == current state). Since unsupported condition was in the middle it's safe to mark from that condition till start of prog.

So I don't have a constructed example and it's more of a hunch. But let's use the same stack slot precision when some instructions use non-r10 registers. We can't always reliably and immediately detect this situation, so the point at which we realize that something is off could be in parent state, but that same slot might be used in children state and if we don't mark everything precise from the current state, then we are running a risk of leaving some of the slots as imprecise.

Before patch #8 similar situation could be also with r0 returned from static subprog. Something like this:

P2: call subprog
<<- checkpoint ->>
P1: r1 = r0;
<<- checkpoint ->>
CUR: r2 = <map value pointer>

Quoted 8 lines
     r2 += r1   <--- need to mark r0 precise and propagate all the way

to P2 and beyond

Precision propagation will be triggered in CUR state. Will go through P1, come back to P2 and (before patch #8 changes) will realize this is unsupported case. Currently we'll mark R0 precise only in P2 and its parents (could be also r6-r9 registers situation).

Correct. I'm with you so far...

Quoted 2 lines

But really we
should make sure that P1 also has r1 marked as precise.

and here I'm a bit lost.
Precision marking will do r1->precise=true while walking P1 in that state.

The change:
mark_all_scalars_precise(env, env->cur_state); will mark both r0 and r1 in P1, but that's unnecessary.

Quoted 4 lines

So as a general rule, I think the only safe default is to mark everything precise from current state.

Is the above convincing or should I revert back to old behavior?

I'm not seeing the issue yet.

#26 lore raw

Re: [PATCH bpf-next 07/10] bpf: fix mark_all_scalars_precise use in mark_chain_precision

Quoted 131 lines · Alexei Starovoitov
Quoted 80 lines · Andrii Nakryiko
Quoted 69 lines · Alexei Starovoitov
Quoted 53 lines · Andrii Nakryiko

When precision backtracking bails out due to some unsupported sequence of instructions (e.g., stack access through register other than r10), we need to mark all SCALAR registers as precise to be safe. Currently, though, we mark SCALARs precise only starting from the state we detected unsupported condition, which could be one of the parent states of the actual current state. This will leave some registers potentially not marked as precise, even though they should. So make sure we start marking scalars as precise from current state (env->cur_state).

Further, we don't currently detect a situation when we end up with some stack slots marked as needing precision, but we ran out of available states to find the instructions that populate those stack slots. This is akin the i >= func->allocated_stack / BPF_REG_SIZE check and should be handled similarly by falling back to marking all SCALARs precise. Add this check when we run out of states.

---
 kernel/bpf/verifier.c                          | 16 +++++++++++++---
 tools/testing/selftests/bpf/verifier/precise.c |  9 +++++----
 2 files changed, 18 insertions(+), 7 deletions(-)
kernel/bpf/verifier.c +3 -3
index 66d64ac10fb1..35f34c977819 100644
@@ -3781,7 +3781,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
                              err = backtrack_insn(env, i, bt);
                      }
                      if (err == -ENOTSUPP) {
-                             mark_all_scalars_precise(env, st);
+                             mark_all_scalars_precise(env, env->cur_state);
                              bt_reset(bt);
                              return 0;
                      } else if (err) {
@@ -3843,7 +3843,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
                                       * fp-8 and it's "unallocated" stack space.
                                       * In such case fallback to conservative.
                                       */
-                                     mark_all_scalars_precise(env, st);
+                                     mark_all_scalars_precise(env, env->cur_state);
                                      bt_reset(bt);
                                      return 0;
                              }
@@ -3872,11 +3872,21 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
              }
 
              if (bt_bitcnt(bt) == 0)
-                     break;
+                     return 0;
 
              last_idx = st->last_insn_idx;
              first_idx = st->first_insn_idx;

kernel/bpf/verifier.c static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r · line 3888

Quoted 10 lines
kernel/bpf/verifier.c +9 -0
      }
+
+     /* if we still have requested precise regs or slots, we missed
+      * something (e.g., stack access through non-r10 register), so
+      * fallback to marking all precise
+      */
+     if (bt_bitcnt(bt) != 0) {
+             mark_all_scalars_precise(env, env->cur_state);
+             bt_reset(bt);
+     }

We get here only after:
st = st->parent;
if (!st)

Quoted 8 lines
          break;

which is the case when we reach the very beginning of the program (parent == NULL) and there are still regs or stack with marks.
That's a situation when backtracking encountered something we didn't foresee. Some new condition. Currently we don't have selftest that trigger this.
So as a defensive mechanism it makes sense to do mark_all_scalars_precise(env, env->cur_state); Probably needs verbose("verifier backtracking bug") as well.

Quoted 3 lines

I hesitated to add a bug message because of a known case where this could happen: stack access through non-r10 register. So it's not a bug, it's similar to -ENOTSUPP cases above, kind of expected, if rare.

fair enough. I'm ok to skip verbose().

Quoted 22 lines
Quoted 4 lines

But for the other two cases mark_all_scalars_precise(env, st); is safe.
What's the reason to mark everything precise from the very beginning of backtracking (last seen state == current state). Since unsupported condition was in the middle it's safe to mark from that condition till start of prog.

So I don't have a constructed example and it's more of a hunch. But let's use the same stack slot precision when some instructions use non-r10 registers. We can't always reliably and immediately detect this situation, so the point at which we realize that something is off could be in parent state, but that same slot might be used in children state and if we don't mark everything precise from the current state, then we are running a risk of leaving some of the slots as imprecise.

Before patch #8 similar situation could be also with r0 returned from static subprog. Something like this:

P2: call subprog
<<- checkpoint ->>
P1: r1 = r0;
<<- checkpoint ->>
CUR: r2 = <map value pointer>

Quoted 8 lines
     r2 += r1   <--- need to mark r0 precise and propagate all the way

to P2 and beyond

Precision propagation will be triggered in CUR state. Will go through P1, come back to P2 and (before patch #8 changes) will realize this is unsupported case. Currently we'll mark R0 precise only in P2 and its parents (could be also r6-r9 registers situation).

Correct. I'm with you so far...

Quoted 2 lines

But really we
should make sure that P1 also has r1 marked as precise.

and here I'm a bit lost.
Precision marking will do r1->precise=true while walking P1 in that state.

The change:
mark_all_scalars_precise(env, env->cur_state); will mark both r0 and r1 in P1, but that's unnecessary.

Quoted 6 lines
Quoted 4 lines

So as a general rule, I think the only safe default is to mark everything precise from current state.

Is the above convincing or should I revert back to old behavior?

I'm not seeing the issue yet.

you are right, second example is not relevant because we'll be properly propagating register precision along the way. So I think stack access is the only one that might be problematic.

I assume you are worried about regressing due to marking more stuff precise, right? My thinking (besides the subtle correctness issues with stack access through non-r10 registers) was that all the other patterns should be now handled properly, so we shouldn't be even seeing mark_all_precise() use in practice.

But I'm fine reverting those two existing mark_all_precise() calls, if you insist.

#27 lore raw

Re: [PATCH bpf-next 07/10] bpf: fix mark_all_scalars_precise use in mark_chain_precision

Quoted 144 lines · Andrii Nakryiko
Quoted 131 lines · Alexei Starovoitov
Quoted 80 lines · Andrii Nakryiko
Quoted 69 lines · Alexei Starovoitov
Quoted 53 lines · Andrii Nakryiko

When precision backtracking bails out due to some unsupported sequence of instructions (e.g., stack access through register other than r10), we need to mark all SCALAR registers as precise to be safe. Currently, though, we mark SCALARs precise only starting from the state we detected unsupported condition, which could be one of the parent states of the actual current state. This will leave some registers potentially not marked as precise, even though they should. So make sure we start marking scalars as precise from current state (env->cur_state).

Further, we don't currently detect a situation when we end up with some stack slots marked as needing precision, but we ran out of available states to find the instructions that populate those stack slots. This is akin the i >= func->allocated_stack / BPF_REG_SIZE check and should be handled similarly by falling back to marking all SCALARs precise. Add this check when we run out of states.

---
 kernel/bpf/verifier.c                          | 16 +++++++++++++---
 tools/testing/selftests/bpf/verifier/precise.c |  9 +++++----
 2 files changed, 18 insertions(+), 7 deletions(-)
kernel/bpf/verifier.c +3 -3
index 66d64ac10fb1..35f34c977819 100644
@@ -3781,7 +3781,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
                              err = backtrack_insn(env, i, bt);
                      }
                      if (err == -ENOTSUPP) {
-                             mark_all_scalars_precise(env, st);
+                             mark_all_scalars_precise(env, env->cur_state);
                              bt_reset(bt);
                              return 0;
                      } else if (err) {
@@ -3843,7 +3843,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
                                       * fp-8 and it's "unallocated" stack space.
                                       * In such case fallback to conservative.
                                       */
-                                     mark_all_scalars_precise(env, st);
+                                     mark_all_scalars_precise(env, env->cur_state);
                                      bt_reset(bt);
                                      return 0;
                              }
@@ -3872,11 +3872,21 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
              }
 
              if (bt_bitcnt(bt) == 0)
-                     break;
+                     return 0;
 
              last_idx = st->last_insn_idx;
              first_idx = st->first_insn_idx;

kernel/bpf/verifier.c static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r · line 3888

Quoted 10 lines
kernel/bpf/verifier.c +9 -0
      }
+
+     /* if we still have requested precise regs or slots, we missed
+      * something (e.g., stack access through non-r10 register), so
+      * fallback to marking all precise
+      */
+     if (bt_bitcnt(bt) != 0) {
+             mark_all_scalars_precise(env, env->cur_state);
+             bt_reset(bt);
+     }

We get here only after:
st = st->parent;
if (!st)

Quoted 8 lines
          break;

which is the case when we reach the very beginning of the program (parent == NULL) and there are still regs or stack with marks.
That's a situation when backtracking encountered something we didn't foresee. Some new condition. Currently we don't have selftest that trigger this.
So as a defensive mechanism it makes sense to do mark_all_scalars_precise(env, env->cur_state); Probably needs verbose("verifier backtracking bug") as well.

Quoted 3 lines

I hesitated to add a bug message because of a known case where this could happen: stack access through non-r10 register. So it's not a bug, it's similar to -ENOTSUPP cases above, kind of expected, if rare.

fair enough. I'm ok to skip verbose().

Quoted 22 lines
Quoted 4 lines

But for the other two cases mark_all_scalars_precise(env, st); is safe.
What's the reason to mark everything precise from the very beginning of backtracking (last seen state == current state). Since unsupported condition was in the middle it's safe to mark from that condition till start of prog.

So I don't have a constructed example and it's more of a hunch. But let's use the same stack slot precision when some instructions use non-r10 registers. We can't always reliably and immediately detect this situation, so the point at which we realize that something is off could be in parent state, but that same slot might be used in children state and if we don't mark everything precise from the current state, then we are running a risk of leaving some of the slots as imprecise.

Before patch #8 similar situation could be also with r0 returned from static subprog. Something like this:

P2: call subprog
<<- checkpoint ->>
P1: r1 = r0;
<<- checkpoint ->>
CUR: r2 = <map value pointer>

Quoted 8 lines
     r2 += r1   <--- need to mark r0 precise and propagate all the way

to P2 and beyond

Precision propagation will be triggered in CUR state. Will go through P1, come back to P2 and (before patch #8 changes) will realize this is unsupported case. Currently we'll mark R0 precise only in P2 and its parents (could be also r6-r9 registers situation).

Correct. I'm with you so far...

Quoted 2 lines

But really we
should make sure that P1 also has r1 marked as precise.

and here I'm a bit lost.
Precision marking will do r1->precise=true while walking P1 in that state.

The change:
mark_all_scalars_precise(env, env->cur_state); will mark both r0 and r1 in P1, but that's unnecessary.

Quoted 6 lines
Quoted 4 lines

So as a general rule, I think the only safe default is to mark everything precise from current state.

Is the above convincing or should I revert back to old behavior?

I'm not seeing the issue yet.

you are right, second example is not relevant because we'll be properly propagating register precision along the way. So I think stack access is the only one that might be problematic.

Quoted 8 lines

I assume you are worried about regressing due to marking more stuff precise, right? My thinking (besides the subtle correctness issues with stack access through non-r10 registers) was that all the other patterns should be now handled properly, so we shouldn't be even seeing mark_all_precise() use in practice.

But I'm fine reverting those two existing mark_all_precise() calls, if you insist.

Regressing due to this was the main concern, but since you've run all possible things with veristat I guess it's fine then.
It doesn't look like extra pessimism hurts too much.
I'm guessing this change was the reason for small increase in cilium progs.

author maintainer
#28 lore raw

PATCH 8/10 bpf: support precision propagation in the presence of subprogs

In reply to Andrii Nakryiko

Add support precision backtracking in the presence of subprogram frames in jump history.

This means supporting a few different kinds of subprogram invocation situations, all requiring a slightly different handling in precision backtracking handling logic:
- static subprogram calls;
- global subprogram calls;
- callback-calling helpers/kfuncs.

For each of those we need to handle a few precision propagation cases:
- what to do with precision of subprog returns (r0);
- what to do with precision of input arguments;
- for all of them callee-saved registers in caller function should be propagated ignoring subprog/callback part of jump history.

N.B. Async callback-calling helpers (currently only bpf_timer_set_callback()) are transparent to all this because they set a separate async callback environment and thus callback's history is not shared with main program's history. So as far as all the changes in this commit goes, such helper is just a regular helper.

Let's look at all these situation in more details. Let's start with static subprogram being called, using an exxerpt of a simple main program and its static subprog, indenting subprog's frame slightly to make everything clear.

frame 0 frame 1 precision set
======= ======= =============

9: r6 = 456;
10: r1 = 123; r6
11: call pc+10; r1, r6

				22: r0 = r1;		r1
				23: exit		r0

12: r1 = <map_pointer> r0, r6
13: r1 += r0; r0, r6
14: r1 += r6; r6;
15: exit

As can be seen above main function is passing 123 as single argument to an identity (return x;) subprog. Returned value is used to adjust map pointer offset, which forces r0 to be marked as precise. Then instruction #14 does the same for callee-saved r6, which will have to be backtracked all the way to instruction #9. For brevity, precision sets for instruction #13 and #14 are combined in the diagram above.

First, for subprog calls, r0 returned from subprog (in frame 0) has to go into subprog's frame 1, and should be cleared from frame 0. So we go back into subprog's frame knowing we need to mark r0 precise. We then see that insn #22 sets r0 from r1, so now we care about marking r1 precise. When we pop up from subprog's frame back into caller at insn #11 we keep r1, as it's an argument-passing register, so we eventually find 10: r1 = 123; and satify precision propagation chain for insn #13.

This example demonstrates two sets of rules:
- r0 returned after subprog call has to be moved into subprog's r0 set;
- *static* subprog arguments (r1-r5) are moved back to caller precision set.

Let's look at what happens with callee-saved precision propagation. Insn #14 mark r6 as precise. When we get into subprog's frame, we keep r6 in frame 0's precision set *only*. Subprog itself has its own set of independent r6-r10 registers and is not affected. When we eventually made our way out of subprog frame we keep r6 in precision set until we reach 9: r6 = 456;, satisfying propagation. r6-r10 propagation is perhaps the simplest aspect, it always stays in its original frame.

That's pretty much all we have to do to support precision propagation across *static subprog* invocation.

Let's look at what happens when we have global subprog invocation.

frame 0 frame 1 precision set
======= ======= =============

9: r6 = 456;
10: r1 = 123; r6
11: call pc+10; # global subprog r6
12: r1 = <map_pointer> r0, r6
13: r1 += r0; r0, r6
14: r1 += r6; r6;
15: exit

Starting from insn #13, r0 has to be precise. We backtrack all the way to insn #11 (call pc+10) and see that subprog is global, so was already validated in isolation. As opposed to static subprog, global subprog always returns unknown scalar r0, so that satisfies precision propagation and we drop r0 from precision set. We are done for insns #13.

Now for insn #14. r6 is in precision set, we backtrack to call pc+10;. Here we need to recognize that this is effectively both exit and entry to global subprog, which means we stay in caller's frame. So we carry on with r6 still in precision set, until we satisfy it at insn #9. The only hard part with global subprogs is just knowing when it's a global func.

Lastly, callback-calling helpers and kfuncs do simulate subprog calls, so jump history will have subprog instructions in between caller program's instructions, but the rules of propagating r0 and r1-r5 differ, because we don't actually directly call callback. We actually call helper/kfunc, which at runtime will call subprog, so the only difference between normal helper/kfunc handling is that we need to make sure to skip callback simulatinog part of jump history. Let's look at an example to make this clearer.

frame 0 frame 1 precision set
======= ======= =============

8: r6 = 456;
9: r1 = 123; r6
10: r2 = &callback; r6
11: call bpf_loop; r6

				22: r0 = r1;
				23: exit

12: r1 = <map_pointer> r0, r6
13: r1 += r0; r0, r6
14: r1 += r6; r6;
15: exit

Again, insn #13 forces r0 to be precise. As soon as we get to 23: exit we see that this isn't actually a static subprog call (it's call bpf_loop; helper call instead). So we clear r0 from precision set.

For callee-saved register, there is no difference: it stays in frame 0's precision set, we go through insn #22 and #23, ignoring them until we get back to caller frame 0, eventually satisfying precision backtrack logic at insn #8 (r6 = 456;).

Assuming callback needed to set r0 as precise at insn #23, we'd backtrack to insn #22, switching from r0 to r1, and then at the point when we pop back to frame 0 at insn #11, we'll clear r1-r5 from precision set, as we don't really do a subprog call directly, so there is no input argument precision propagation.

That's pretty much it. With these changes, it seems like the only still unsupported situation for precision backpropagation is the case when program is accessing stack through registers other than r10. This is still left as unsupported (though rare) case for now.

As for results. For selftests, few positive changes for bigger programs, cls_redirect in dynptr variant benefitting the most:

[vmuser@archvm bpf]$ ./veristat -C ~/subprog-precise-before-results.csv ~/subprog-precise-after-results.csv -f @veristat.cfg -e file,prog,insns -f 'insns_diff!=0'

File                                      Program        Insns (A)  Insns (B)  Insns     (DIFF)
----------------------------------------  -------------  ---------  ---------  ----------------
pyperf600_bpf_loop.bpf.linked1.o          on_event            2060       2002      -58 (-2.82%)
test_cls_redirect_dynptr.bpf.linked1.o    cls_redirect       15660       2914  -12746 (-81.39%)
test_cls_redirect_subprogs.bpf.linked1.o  cls_redirect       61620      59088    -2532 (-4.11%)
xdp_synproxy_kern.bpf.linked1.o           syncookie_tc      109980      86278  -23702 (-21.55%)
xdp_synproxy_kern.bpf.linked1.o           syncookie_xdp      97716      85147  -12569 (-12.86%)

Cilium progress don't really regress. They don't use subprogs and are mostly unaffected, but some other fixes and improvements could have changed something. This doesn't appear to be the case:

[vmuser@archvm bpf]$ ./veristat -C ~/subprog-precise-before-results-cilium.csv ~/subprog-precise-after-results-cilium.csv -e file,prog,insns -f 'insns_diff!=0'

File           Program                         Insns (A)  Insns (B)  Insns (DIFF)
-------------  ------------------------------  ---------  ---------  ------------
bpf_host.o     tail_nodeport_nat_ingress_ipv6       4983       5003  +20 (+0.40%)
bpf_lxc.o      tail_nodeport_nat_ingress_ipv6       4983       5003  +20 (+0.40%)
bpf_overlay.o  tail_nodeport_nat_ingress_ipv6       4983       5003  +20 (+0.40%)
bpf_xdp.o      tail_handle_nat_fwd_ipv6            12475      12504  +29 (+0.23%)
bpf_xdp.o      tail_nodeport_nat_ingress_ipv6       6363       6371   +8 (+0.13%)

Looking at (somewhat anonymized) Meta production programs, we see mostly insignificant variation in number of instructions, with one program (syar_bind6_protect6) benefitting the most at -17%.

[vmuser@archvm bpf]$ ./veristat -C ~/subprog-precise-before-results-fbcode.csv ~/subprog-precise-after-results-fbcode.csv -e prog,insns -f 'insns_diff!=0'

Program                   Insns (A)  Insns (B)  Insns     (DIFF)
------------------------  ---------  ---------  ----------------
on_request_context_event        597        585      -12 (-2.01%)
read_async_py_stack           43789      43657     -132 (-0.30%)
read_sync_py_stack            35041      37599    +2558 (+7.30%)
rrm_usdt                        946        940       -6 (-0.63%)
sysarmor_inet6_bind           28863      28249     -614 (-2.13%)
sysarmor_inet_bind            28845      28240     -605 (-2.10%)
syar_bind4_protect4          154145     147640    -6505 (-4.22%)
syar_bind6_protect6          165242     137088  -28154 (-17.04%)
syar_task_exit_setgid         21289      19720    -1569 (-7.37%)
syar_task_exit_setuid         21290      19721    -1569 (-7.37%)
do_uprobe                     19967      19413     -554 (-2.77%)
tw_twfw_ingress              215877     204833   -11044 (-5.12%)
tw_twfw_tc_in                215877     204833   -11044 (-5.12%)

But checking duration (wall clock) differences, that is the actual time taken by verifier to validate programs, we see a sometimes dramatic improvements, all the way to about 16x improvements:

[vmuser@archvm bpf]$ ./veristat -C ~/subprog-precise-before-results-meta.csv ~/subprog-precise-after-results-meta.csv -e prog,duration -s duration_diff^ | head -n20

Program                                   Duration (us) (A)  Duration (us) (B)  Duration (us) (DIFF)
----------------------------------------  -----------------  -----------------  --------------------
tw_twfw_ingress                                     4488374             272836    -4215538 (-93.92%)
tw_twfw_tc_in                                       4339111             268175    -4070936 (-93.82%)
tw_twfw_egress                                      3521816             270751    -3251065 (-92.31%)
tw_twfw_tc_eg                                       3472878             284294    -3188584 (-91.81%)
balancer_ingress                                     343119             291391      -51728 (-15.08%)
syar_bind6_protect6                                   78992              64782      -14210 (-17.99%)
ttls_tc_ingress                                       11739               8176       -3563 (-30.35%)
kprobe__security_inode_link                           13864              11341       -2523 (-18.20%)
read_sync_py_stack                                    21927              19442       -2485 (-11.33%)
read_async_py_stack                                   30444              28136        -2308 (-7.58%)
syar_task_exit_setuid                                 10256               8440       -1816 (-17.71%)
---
 kernel/bpf/verifier.c | 149 ++++++++++++++++++++++++++++++++++++------
 1 file changed, 129 insertions(+), 20 deletions(-)
kernel/bpf/verifier.c +129 -20
index 35f34c977819..dd436bf96e81 100644
@@ -240,6 +240,12 @@ static void bpf_map_key_store(struct bpf_insn_aux_data *aux, u64 state)
 			     (poisoned ? BPF_MAP_KEY_POISON : 0ULL);
 }
 
+static bool bpf_helper_call(const struct bpf_insn *insn)
+{
+	return insn->code == (BPF_JMP | BPF_CALL) &&
+	       insn->src_reg == 0;
+}
+
 static bool bpf_pseudo_call(const struct bpf_insn *insn)
 {
 	return insn->code == (BPF_JMP | BPF_CALL) &&
@@ -468,6 +474,13 @@ static struct btf_record *reg_btf_record(const struct bpf_reg_state *reg)
 	return rec;
 }
 
+static bool subprog_is_global(const struct bpf_verifier_env *env, int subprog)
+{
+	struct bpf_func_info_aux *aux = env->prog->aux->func_info_aux;
+
+	return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
+}
+
 static bool reg_may_point_to_spin_lock(const struct bpf_reg_state *reg)
 {
 	return btf_record_has_field(reg_btf_record(reg), BPF_SPIN_LOCK);
@@ -515,6 +528,8 @@ static bool is_dynptr_ref_function(enum bpf_func_id func_id)
 	return func_id == BPF_FUNC_dynptr_data;
 }
 
+static bool is_callback_calling_kfunc(u32 btf_id);
+
 static bool is_callback_calling_function(enum bpf_func_id func_id)
 {
 	return func_id == BPF_FUNC_for_each_map_elem ||
@@ -524,6 +539,11 @@ static bool is_callback_calling_function(enum bpf_func_id func_id)
 	       func_id == BPF_FUNC_user_ringbuf_drain;
 }
 
+static bool is_async_callback_calling_function(enum bpf_func_id func_id)
+{
+	return func_id == BPF_FUNC_timer_set_callback;
+}
+
 static bool is_storage_get_function(enum bpf_func_id func_id)
 {
 	return func_id == BPF_FUNC_sk_storage_get ||
@@ -3317,8 +3337,13 @@ static void fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)
 /* For given verifier state backtrack_insn() is called from the last insn to
  * the first insn. Its purpose is to compute a bitmask of registers and
  * stack slots that needs precision in the parent verifier state.
+ *
+ * @idx is an index of the instruction we are currently processing;
+ * @subseq_idx is an index of the subsequent instruction that:
+ *   - *would be* executed next, if jump history is viewed in forward order;
+ *   - *was* processed previously during backtracking.
  */
-static int backtrack_insn(struct bpf_verifier_env *env, int idx,
+static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 			  struct backtrack_state *bt)
 {
 	const struct bpf_insn_cbs cbs = {
@@ -3332,7 +3357,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
 	u8 mode = BPF_MODE(insn->code);
 	u32 dreg = insn->dst_reg;
 	u32 sreg = insn->src_reg;
-	u32 spi;
+	u32 spi, i;
 
 	if (insn->code == 0)
 		return 0;
@@ -3424,14 +3449,72 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
 		if (class == BPF_STX)
 			bt_set_reg(bt, sreg);
 	} else if (class == BPF_JMP || class == BPF_JMP32) {
-		if (opcode == BPF_CALL) {
-			if (insn->src_reg == BPF_PSEUDO_CALL)
-				return -ENOTSUPP;
-			/* BPF helpers that invoke callback subprogs are
-			 * equivalent to BPF_PSEUDO_CALL above
+		if (bpf_pseudo_call(insn)) {
+			int subprog_insn_idx, subprog;
+			bool is_global;
+
+			subprog_insn_idx = idx + insn->imm + 1;
+			subprog = find_subprog(env, subprog_insn_idx);
+			if (subprog < 0)
+				return -EFAULT;
+			is_global = subprog_is_global(env, subprog);
+
+			if (is_global) {
+				/* r1-r5 are invalidated after subprog call,
+				 * so for global func call it shouldn't be set
+				 * anymore
+				 */
+				if (bt_reg_mask(bt) & BPF_REGMASK_ARGS)
+					return -EFAULT;
+				/* global subprog always sets R0 */
+				bt_clear_reg(bt, BPF_REG_0);
+				return 0;
+			} else {
+				/* static subprog call instruction, which
+				 * means that we are exiting current subprog,
+				 * so only r1-r5 could be still requested as
+				 * precise, r0 and r6-r10 or any stack slot in
+				 * the current frame should be zero by now
+				 */
+				if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS)
+					return -EFAULT;
+				/* we don't track register spills perfectly,
+				 * so fallback to force-precise instead of failing */
+				if (bt_stack_mask(bt) != 0)
+					return -ENOTSUPP;
+				/* propagate r1-r5 to the caller */
+				for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+					if (bt_is_reg_set(bt, i)) {
+						bt_clear_reg(bt, i);
+						bt_set_frame_reg(bt, bt->frame - 1, i);
+					}
+				}
+				if (bt_subprog_exit(bt))
+					return -EFAULT;
+				return 0;
+			}
+		} else if ((bpf_helper_call(insn) &&
+			    is_callback_calling_function(insn->imm) &&
+			    !is_async_callback_calling_function(insn->imm)) ||
+			   (bpf_pseudo_kfunc_call(insn) && is_callback_calling_kfunc(insn->imm))) {
+			/* callback-calling helper or kfunc call, which means
+			 * we are exiting from subprog, but unlike the subprog
+			 * call handling above, we shouldn't propagate
+			 * precision of r1-r5 (if any requested), as they are
+			 * not actually arguments passed directly to callback
+			 * subprogs
 			 */
-			if (insn->src_reg == 0 && is_callback_calling_function(insn->imm))
+			if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS)
+				return -EFAULT;
+			if (bt_stack_mask(bt) != 0)
 				return -ENOTSUPP;
+			/* clear r1-r5 in callback subprog's mask */
+			for (i = BPF_REG_1; i <= BPF_REG_5; i++)
+				bt_clear_reg(bt, i);
+			if (bt_subprog_exit(bt))
+				return -EFAULT;
+			return 0;
+		} else if (opcode == BPF_CALL) {
 			/* kfunc with imm==0 is invalid and fixup_kfunc_call will
 			 * catch this error later. Make backtracking conservative
 			 * with ENOTSUPP.
@@ -3449,7 +3532,39 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
 				return -EFAULT;
 			}
 		} else if (opcode == BPF_EXIT) {
-			return -ENOTSUPP;
+			bool r0_precise;
+
+			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
+				/* if backtracing was looking for registers R1-R5
+				 * they should have been found already.
+				 */
+				verbose(env, "BUG regs %x\n", bt_reg_mask(bt));
+				WARN_ONCE(1, "verifier backtracking bug");
+				return -EFAULT;
+			}
+
+			/* BPF_EXIT in subprog or callback always jump right
+			 * after the call instruction, so by check whether the
+			 * instruction at subseq_idx-1 is subprog call or not we
+			 * can distinguish actual exit from *subprog* from
+			 * exit from *callback*. In the former case, we need
+			 * to propagate r0 precision, if necessary. In the
+			 * former we never do that.
+			 */
+			r0_precise = subseq_idx - 1 >= 0 &&
+				     bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
+				     bt_is_reg_set(bt, BPF_REG_0);
+
+			bt_clear_reg(bt, BPF_REG_0);
+			if (bt_subprog_enter(bt))
+				return -EFAULT;
+
+			if (r0_precise)
+				bt_set_reg(bt, BPF_REG_0);
+			/* r6-r9 and stack slots will stay set in caller frame
+			 * bitmasks until we return back from callee(s)
+			 */
+			return 0;
 		} else if (BPF_SRC(insn->code) == BPF_X) {
 			if (!bt_is_reg_set(bt, dreg) && !bt_is_reg_set(bt, sreg))
 				return 0;
@@ -3703,7 +3818,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 	struct bpf_func_state *func;
 	struct bpf_reg_state *reg;
 	bool skip_first = true;
-	int i, fr, err;
+	int i, prev_i, fr, err;
 
 	if (!env->bpf_capable)
 		return 0;
@@ -3773,12 +3888,12 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 			return -EFAULT;
 		}
 
-		for (i = last_idx;;) {
+		for (i = last_idx, prev_i = -1;;) {
 			if (skip_first) {
 				err = 0;
 				skip_first = false;
 			} else {
-				err = backtrack_insn(env, i, bt);
+				err = backtrack_insn(env, i, prev_i, bt);
 			}
 			if (err == -ENOTSUPP) {
 				mark_all_scalars_precise(env, env->cur_state);
@@ -3795,6 +3910,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r
 				return 0;
 			if (i == first_idx)
 				break;
+			prev_i = i;
 			i = get_prev_insn_idx(st, i, &history);
 			if (i >= env->prog->len) {
 				/* This can happen if backtracking reached insn 0
@@ -8376,17 +8492,13 @@ static int set_callee_state(struct bpf_verifier_env *env,
 			    struct bpf_func_state *caller,
 			    struct bpf_func_state *callee, int insn_idx);
 
-static bool is_callback_calling_kfunc(u32 btf_id);
-
 static int __check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			     int *insn_idx, int subprog,
 			     set_callee_state_fn set_callee_state_cb)
 {
 	struct bpf_verifier_state *state = env->cur_state;
-	struct bpf_func_info_aux *func_info_aux;
 	struct bpf_func_state *caller, *callee;
 	int err;
-	bool is_global = false;
 
 	if (state->curframe + 1 >= MAX_CALL_FRAMES) {
 		verbose(env, "the call stack of %d frames is too deep\n",
@@ -8401,13 +8513,10 @@ static int __check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 		return -EFAULT;
 	}
 
-	func_info_aux = env->prog->aux->func_info_aux;
-	if (func_info_aux)
-		is_global = func_info_aux[subprog].linkage == BTF_FUNC_GLOBAL;
 	err = btf_check_subprog_call(env, subprog, caller->regs);
 	if (err == -EFAULT)
 		return err;
-	if (is_global) {
+	if (subprog_is_global(env, subprog)) {
 		if (err) {
 			verbose(env, "Caller passes invalid args into func#%d\n",
 				subprog);
Signature · 1 line
2.34.1
#29 lore raw

Re: [PATCH bpf-next 08/10] bpf: support precision propagation in the presence of subprogs

Quoted 55 lines · Andrii Nakryiko

Add support precision backtracking in the presence of subprogram frames in jump history.

This means supporting a few different kinds of subprogram invocation situations, all requiring a slightly different handling in precision backtracking handling logic:
- static subprogram calls;
- global subprogram calls;
- callback-calling helpers/kfuncs.

For each of those we need to handle a few precision propagation cases:
- what to do with precision of subprog returns (r0);
- what to do with precision of input arguments;
- for all of them callee-saved registers in caller function should be propagated ignoring subprog/callback part of jump history.

N.B. Async callback-calling helpers (currently only bpf_timer_set_callback()) are transparent to all this because they set a separate async callback environment and thus callback's history is not shared with main program's history. So as far as all the changes in this commit goes, such helper is just a regular helper.

Let's look at all these situation in more details. Let's start with static subprogram being called, using an exxerpt of a simple main program and its static subprog, indenting subprog's frame slightly to make everything clear.

frame 0 frame 1 precision set
======= ======= =============

9: r6 = 456;
10: r1 = 123; r6
11: call pc+10; r1, r6

				22: r0 = r1;		r1
				23: exit		r0

12: r1 = <map_pointer> r0, r6
13: r1 += r0; r0, r6
14: r1 += r6; r6;
15: exit

As can be seen above main function is passing 123 as single argument to an identity (return x;) subprog. Returned value is used to adjust map pointer offset, which forces r0 to be marked as precise. Then instruction #14 does the same for callee-saved r6, which will have to be backtracked all the way to instruction #9. For brevity, precision sets for instruction #13 and #14 are combined in the diagram above.

First, for subprog calls, r0 returned from subprog (in frame 0) has to go into subprog's frame 1, and should be cleared from frame 0. So we go back into subprog's frame knowing we need to mark r0 precise. We then see that insn #22 sets r0 from r1, so now we care about marking r1 precise. When we pop up from subprog's frame back into caller at insn #11 we keep r1, as it's an argument-passing register, so we eventually find 10: r1 = 123; and satify precision propagation chain for insn #13.

Quoted 3 lines

This example demonstrates two sets of rules:
- r0 returned after subprog call has to be moved into subprog's r0 set;
- *static* subprog arguments (r1-r5) are moved back to caller precision set.

Haven't read the rest. Only commenting on the above...

The description of "precision set" combines multiple frames and skips the lower which makes it hard to reason.
I think it should be:

10: r1 = 123; fr0: r6
11: call pc+10; fr0: r1, r6

				22: r0 = r1;		fr0: r6; fr1: r1
				23: exit		fr0: r6; fr1: r0

12: r1 = <map_pointer> fr0: r0, r6
13: r1 += r0; fr0: r0, r6
14: r1 += r6; fr0: r6

Right?

#30 lore raw

Re: [PATCH bpf-next 08/10] bpf: support precision propagation in the presence of subprogs

Quoted 57 lines · Alexei Starovoitov
Quoted 55 lines · Andrii Nakryiko

Add support precision backtracking in the presence of subprogram frames in jump history.

This means supporting a few different kinds of subprogram invocation situations, all requiring a slightly different handling in precision backtracking handling logic:
- static subprogram calls;
- global subprogram calls;
- callback-calling helpers/kfuncs.

For each of those we need to handle a few precision propagation cases:
- what to do with precision of subprog returns (r0);
- what to do with precision of input arguments;
- for all of them callee-saved registers in caller function should be propagated ignoring subprog/callback part of jump history.

N.B. Async callback-calling helpers (currently only bpf_timer_set_callback()) are transparent to all this because they set a separate async callback environment and thus callback's history is not shared with main program's history. So as far as all the changes in this commit goes, such helper is just a regular helper.

Let's look at all these situation in more details. Let's start with static subprogram being called, using an exxerpt of a simple main program and its static subprog, indenting subprog's frame slightly to make everything clear.

frame 0                               frame 1                 precision set
=======                               =======                 =============

9: r6 = 456;
10: r1 = 123; r6
11: call pc+10; r1, r6
22: r0 = r1; r1
23: exit r0
12: r1 = <map_pointer> r0, r6
13: r1 += r0; r0, r6
14: r1 += r6; r6;
15: exit

As can be seen above main function is passing 123 as single argument to an identity (return x;) subprog. Returned value is used to adjust map pointer offset, which forces r0 to be marked as precise. Then instruction #14 does the same for callee-saved r6, which will have to be backtracked all the way to instruction #9. For brevity, precision sets for instruction #13 and #14 are combined in the diagram above.

First, for subprog calls, r0 returned from subprog (in frame 0) has to go into subprog's frame 1, and should be cleared from frame 0. So we go back into subprog's frame knowing we need to mark r0 precise. We then see that insn #22 sets r0 from r1, so now we care about marking r1 precise. When we pop up from subprog's frame back into caller at insn #11 we keep r1, as it's an argument-passing register, so we eventually find 10: r1 = 123; and satify precision propagation chain for insn #13.

Quoted 8 lines
Quoted 3 lines

This example demonstrates two sets of rules:
- r0 returned after subprog call has to be moved into subprog's r0 set;
- *static* subprog arguments (r1-r5) are moved back to caller precision set.

Haven't read the rest. Only commenting on the above...

The description of "precision set" combines multiple frames and skips the lower which makes it hard to reason.

I currently print "current frame" mask only, depending on whether instruction is in parent or subprog. But yeah, I can make it more explicit and print both frames' masks, to make it easier to follow

Quoted 11 lines

I think it should be:

10: r1 = 123;                                           fr0: r6
11: call pc+10;                                         fr0: r1, r6
                                22: r0 = r1;            fr0: r6; fr1: r1
                                23: exit                fr0: r6; fr1: r0
12: r1 = <map_pointer>                                  fr0: r0, r6
13: r1 += r0;                                           fr0: r0, r6
14: r1 += r6;                                           fr0: r6

Right?

right, exactly, I'll use this format

#31 lore raw

Re: [PATCH bpf-next 08/10] bpf: support precision propagation in the presence of subprogs

In reply to Andrii Nakryiko
Quoted 12 lines · Andrii Nakryiko
kernel/bpf/verifier.c +1 -5
 	if (insn->code == 0)
 		return 0;
@@ -3424,14 +3449,72 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
 		if (class == BPF_STX)
 			bt_set_reg(bt, sreg);
 	} else if (class == BPF_JMP || class == BPF_JMP32) {
-		if (opcode == BPF_CALL) {
-			if (insn->src_reg == BPF_PSEUDO_CALL)
-				return -ENOTSUPP;
-			/* BPF helpers that invoke callback subprogs are
-			 * equivalent to BPF_PSEUDO_CALL above
+		if (bpf_pseudo_call(insn)) {

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3462

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+			int subprog_insn_idx, subprog;
+			bool is_global;
+
+			subprog_insn_idx = idx + insn->imm + 1;
+			subprog = find_subprog(env, subprog_insn_idx);
+			if (subprog < 0)
+				return -EFAULT;
+			is_global = subprog_is_global(env, subprog);
+
+			if (is_global) {

could you add a warn_on here that checks that jmp history doesn't have insns from subprog.

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3468

Quoted 6 lines
kernel/bpf/verifier.c +6 -0
+				/* r1-r5 are invalidated after subprog call,
+				 * so for global func call it shouldn't be set
+				 * anymore
+				 */
+				if (bt_reg_mask(bt) & BPF_REGMASK_ARGS)
+					return -EFAULT;

This shouldn't be happening, but backtracking is delicate.
Could you add verbose("backtracking bug") here, so we know why the prog got rejected. I'd probably do -ENOTSUPP, but EFAULT is ok too.

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3480

Quoted 12 lines
kernel/bpf/verifier.c +12 -0
+				/* global subprog always sets R0 */
+				bt_clear_reg(bt, BPF_REG_0);
+				return 0;
+			} else {
+				/* static subprog call instruction, which
+				 * means that we are exiting current subprog,
+				 * so only r1-r5 could be still requested as
+				 * precise, r0 and r6-r10 or any stack slot in
+				 * the current frame should be zero by now
+				 */
+				if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS)
+					return -EFAULT;

same here.

Quoted 27 lines
kernel/bpf/verifier.c +25 -1
+				/* we don't track register spills perfectly,
+				 * so fallback to force-precise instead of failing */
+				if (bt_stack_mask(bt) != 0)
+					return -ENOTSUPP;
+				/* propagate r1-r5 to the caller */
+				for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+					if (bt_is_reg_set(bt, i)) {
+						bt_clear_reg(bt, i);
+						bt_set_frame_reg(bt, bt->frame - 1, i);
+					}
+				}
+				if (bt_subprog_exit(bt))
+					return -EFAULT;
+				return 0;
+			}
+		} else if ((bpf_helper_call(insn) &&
+			    is_callback_calling_function(insn->imm) &&
+			    !is_async_callback_calling_function(insn->imm)) ||
+			   (bpf_pseudo_kfunc_call(insn) && is_callback_calling_kfunc(insn->imm))) {
+			/* callback-calling helper or kfunc call, which means
+			 * we are exiting from subprog, but unlike the subprog
+			 * call handling above, we shouldn't propagate
+			 * precision of r1-r5 (if any requested), as they are
+			 * not actually arguments passed directly to callback
+			 * subprogs
 			 */
-			if (insn->src_reg == 0 && is_callback_calling_function(insn->imm))

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3516

Quoted 10 lines
kernel/bpf/verifier.c +9 -0
+			if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS)
+				return -EFAULT;
+			if (bt_stack_mask(bt) != 0)
 				return -ENOTSUPP;
+			/* clear r1-r5 in callback subprog's mask */
+			for (i = BPF_REG_1; i <= BPF_REG_5; i++)
+				bt_clear_reg(bt, i);
+			if (bt_subprog_exit(bt))
+				return -EFAULT;
+			return 0;

jmp history will include callback insn, right?
So skip of jmp history is missing here ?

Quoted 11 lines
kernel/bpf/verifier.c +3 -1
+		} else if (opcode == BPF_CALL) {
 			/* kfunc with imm==0 is invalid and fixup_kfunc_call will
 			 * catch this error later. Make backtracking conservative
 			 * with ENOTSUPP.
@@ -3449,7 +3532,39 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
 				return -EFAULT;
 			}
 		} else if (opcode == BPF_EXIT) {
-			return -ENOTSUPP;
+			bool r0_precise;
+

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3546

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
+				/* if backtracing was looking for registers R1-R5
+				 * they should have been found already.
+				 */
+				verbose(env, "BUG regs %x\n", bt_reg_mask(bt));
+				WARN_ONCE(1, "verifier backtracking bug");
+				return -EFAULT;
+			}
+
+			/* BPF_EXIT in subprog or callback always jump right

I'd say 'subprog always returns right after the call'. 'jump' is a bit confusing here, since it doesn't normally used to describe function return address.

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3557

Quoted 11 lines
kernel/bpf/verifier.c +11 -0
+			 * after the call instruction, so by check whether the
+			 * instruction at subseq_idx-1 is subprog call or not we
+			 * can distinguish actual exit from *subprog* from
+			 * exit from *callback*. In the former case, we need
+			 * to propagate r0 precision, if necessary. In the
+			 * former we never do that.
+			 */
+			r0_precise = subseq_idx - 1 >= 0 &&
+				     bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
+				     bt_is_reg_set(bt, BPF_REG_0);
+

The rest all makes sense.

#32 lore raw

Re: [PATCH bpf-next 08/10] bpf: support precision propagation in the presence of subprogs

Quoted 18 lines · Alexei Starovoitov
Quoted 16 lines · Andrii Nakryiko
      if (insn->code == 0)
              return 0;
kernel/bpf/verifier.c +5 -5
@@ -3424,14 +3449,72 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
              if (class == BPF_STX)
                      bt_set_reg(bt, sreg);
      } else if (class == BPF_JMP || class == BPF_JMP32) {
-             if (opcode == BPF_CALL) {
-                     if (insn->src_reg == BPF_PSEUDO_CALL)
-                             return -ENOTSUPP;
-                     /* BPF helpers that invoke callback subprogs are
-                      * equivalent to BPF_PSEUDO_CALL above
+             if (bpf_pseudo_call(insn)) {
+                     int subprog_insn_idx, subprog;
+                     bool is_global;
+
+                     subprog_insn_idx = idx + insn->imm + 1;
Quoted 8 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3462

Quoted 6 lines
kernel/bpf/verifier.c +6 -0
+                     subprog = find_subprog(env, subprog_insn_idx);
+                     if (subprog < 0)
+                             return -EFAULT;
+                     is_global = subprog_is_global(env, subprog);
+
+                     if (is_global) {

could you add a warn_on here that checks that jmp history doesn't have insns from subprog.

wouldn't this be very expensive to go over the entire jmp history to check that no jump point there overlaps with the global function? Or what do you have in mind specifically for this check?

Quoted 11 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3468

Quoted 7 lines
kernel/bpf/verifier.c +6 -0
+                             /* r1-r5 are invalidated after subprog call,
+                              * so for global func call it shouldn't be set
+                              * anymore
+                              */
+                             if (bt_reg_mask(bt) & BPF_REGMASK_ARGS)
+                                     return -EFAULT;

This shouldn't be happening, but backtracking is delicate.
Could you add verbose("backtracking bug") here, so we know why the prog got rejected. I'd probably do -ENOTSUPP, but EFAULT is ok too.

Will add verbose(). EFAULT because valid code should never use r1-r5 after call. Invalid code should be rejected before that, and if not, then it is really a bug and best to bail out.

Quoted 14 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3480

Quoted 13 lines
kernel/bpf/verifier.c +12 -0
+                             /* global subprog always sets R0 */
+                             bt_clear_reg(bt, BPF_REG_0);
+                             return 0;
+                     } else {
+                             /* static subprog call instruction, which
+                              * means that we are exiting current subprog,
+                              * so only r1-r5 could be still requested as
+                              * precise, r0 and r6-r10 or any stack slot in
+                              * the current frame should be zero by now
+                              */
+                             if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS)
+                                     return -EFAULT;
Quoted 1 line

same here.

ack

Quoted 39 lines
Quoted 28 lines
kernel/bpf/verifier.c +25 -1
+                             /* we don't track register spills perfectly,
+                              * so fallback to force-precise instead of failing */
+                             if (bt_stack_mask(bt) != 0)
+                                     return -ENOTSUPP;
+                             /* propagate r1-r5 to the caller */
+                             for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+                                     if (bt_is_reg_set(bt, i)) {
+                                             bt_clear_reg(bt, i);
+                                             bt_set_frame_reg(bt, bt->frame - 1, i);
+                                     }
+                             }
+                             if (bt_subprog_exit(bt))
+                                     return -EFAULT;
+                             return 0;
+                     }
+             } else if ((bpf_helper_call(insn) &&
+                         is_callback_calling_function(insn->imm) &&
+                         !is_async_callback_calling_function(insn->imm)) ||
+                        (bpf_pseudo_kfunc_call(insn) && is_callback_calling_kfunc(insn->imm))) {
+                     /* callback-calling helper or kfunc call, which means
+                      * we are exiting from subprog, but unlike the subprog
+                      * call handling above, we shouldn't propagate
+                      * precision of r1-r5 (if any requested), as they are
+                      * not actually arguments passed directly to callback
+                      * subprogs
                       */
-                     if (insn->src_reg == 0 && is_callback_calling_function(insn->imm))

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3516

Quoted 10 lines
kernel/bpf/verifier.c +9 -0
+                     if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS)
+                             return -EFAULT;
+                     if (bt_stack_mask(bt) != 0)
                              return -ENOTSUPP;
+                     /* clear r1-r5 in callback subprog's mask */
+                     for (i = BPF_REG_1; i <= BPF_REG_5; i++)
+                             bt_clear_reg(bt, i);
+                     if (bt_subprog_exit(bt))
+                             return -EFAULT;
+                     return 0;
Quoted 2 lines

jmp history will include callback insn, right?
So skip of jmp history is missing here ?

This is, say, call bpf_loop; instruction. Which means we just got out of bpf_loop's callback's jump history (so already "skipped" them, except we didn't skip, we properly processed them). So there is nothing to skip anymore. We are in the parent program already.

This is exactly the situation described in the third example in the commit description. We are at instruction #11 here.

Note how we don't bail out on r1-r5 being set, because it's expected that callback might require some of its input arguments to be precise and that's not an error condition. This is similar to handling global function, where r1-r5 might be still required to be precise when we already processed the entire jump history.

It's a bit mind bending to reason about all this, because everything is backwards.

Quoted 17 lines
Quoted 17 lines
kernel/bpf/verifier.c +8 -1
+             } else if (opcode == BPF_CALL) {
                      /* kfunc with imm==0 is invalid and fixup_kfunc_call will
                       * catch this error later. Make backtracking conservative
                       * with ENOTSUPP.
@@ -3449,7 +3532,39 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
                              return -EFAULT;
                      }
              } else if (opcode == BPF_EXIT) {
-                     return -ENOTSUPP;
+                     bool r0_precise;
+
+                     if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
+                             /* if backtracing was looking for registers R1-R5
+                              * they should have been found already.
+                              */
+                             verbose(env, "BUG regs %x\n", bt_reg_mask(bt));
Quoted 8 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3546

Quoted 5 lines
kernel/bpf/verifier.c +5 -0
+                             WARN_ONCE(1, "verifier backtracking bug");
+                             return -EFAULT;
+                     }
+
+                     /* BPF_EXIT in subprog or callback always jump right

I'd say 'subprog always returns right after the call'. 'jump' is a bit confusing here, since it doesn't normally used to describe function return address.

this was described in the context of "jump history", but I'll adjust the wording to "return".

Quoted 13 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3557

Quoted 12 lines
kernel/bpf/verifier.c +11 -0
+                      * after the call instruction, so by check whether the
+                      * instruction at subseq_idx-1 is subprog call or not we
+                      * can distinguish actual exit from *subprog* from
+                      * exit from *callback*. In the former case, we need
+                      * to propagate r0 precision, if necessary. In the
+                      * former we never do that.
+                      */
+                     r0_precise = subseq_idx - 1 >= 0 &&
+                                  bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
+                                  bt_is_reg_set(bt, BPF_REG_0);
+
Quoted 1 line

The rest all makes sense.

Cool. Thanks for taking a thorough look. I'm a bit confused on what exactly you'd like me to check on global subprog call, so please elaborate. I'll address all the other feedback meanwhile.

#33 lore raw

Re: [PATCH bpf-next 08/10] bpf: support precision propagation in the presence of subprogs

Quoted 29 lines · Andrii Nakryiko
Quoted 18 lines · Alexei Starovoitov
Quoted 16 lines · Andrii Nakryiko
      if (insn->code == 0)
              return 0;
kernel/bpf/verifier.c +5 -5
@@ -3424,14 +3449,72 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
              if (class == BPF_STX)
                      bt_set_reg(bt, sreg);
      } else if (class == BPF_JMP || class == BPF_JMP32) {
-             if (opcode == BPF_CALL) {
-                     if (insn->src_reg == BPF_PSEUDO_CALL)
-                             return -ENOTSUPP;
-                     /* BPF helpers that invoke callback subprogs are
-                      * equivalent to BPF_PSEUDO_CALL above
+             if (bpf_pseudo_call(insn)) {
+                     int subprog_insn_idx, subprog;
+                     bool is_global;
+
+                     subprog_insn_idx = idx + insn->imm + 1;
Quoted 8 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3462

Quoted 6 lines
kernel/bpf/verifier.c +6 -0
+                     subprog = find_subprog(env, subprog_insn_idx);
+                     if (subprog < 0)
+                             return -EFAULT;
+                     is_global = subprog_is_global(env, subprog);
+
+                     if (is_global) {

could you add a warn_on here that checks that jmp history doesn't have insns from subprog.

Quoted 3 lines

wouldn't this be very expensive to go over the entire jmp history to check that no jump point there overlaps with the global function? Or what do you have in mind specifically for this check?

recalling how jmp_history works and reading this comment when we process any call:

        /* when we exit from subprog, we need to record non-linear history */
        mark_jmp_point(env, t + 1);

so for static subprog the history will be:
call
jmps inside subprog
insn after call.

for global it will be:
call
insn after call.

I was thinking that we can do simple check that call + 1 == subseq_idx for globals. For statics that should never be the case.

We don't have to do it. Mostly checking my understanding of patches and jmp history.

Quoted 75 lines
Quoted 9 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3468

Quoted 8 lines
kernel/bpf/verifier.c +6 -0
+                             /* r1-r5 are invalidated after subprog call,
+                              * so for global func call it shouldn't be set
+                              * anymore
+                              */
+                             if (bt_reg_mask(bt) & BPF_REGMASK_ARGS)
+                                     return -EFAULT;
Quoted 3 lines

This shouldn't be happening, but backtracking is delicate.
Could you add verbose("backtracking bug") here, so we know why the prog got rejected. I'd probably do -ENOTSUPP, but EFAULT is ok too.

Will add verbose(). EFAULT because valid code should never use r1-r5 after call. Invalid code should be rejected before that, and if not, then it is really a bug and best to bail out.

Quoted 14 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3480

Quoted 13 lines
kernel/bpf/verifier.c +12 -0
+                             /* global subprog always sets R0 */
+                             bt_clear_reg(bt, BPF_REG_0);
+                             return 0;
+                     } else {
+                             /* static subprog call instruction, which
+                              * means that we are exiting current subprog,
+                              * so only r1-r5 could be still requested as
+                              * precise, r0 and r6-r10 or any stack slot in
+                              * the current frame should be zero by now
+                              */
+                             if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS)
+                                     return -EFAULT;
Quoted 1 line

same here.

ack

Quoted 39 lines
Quoted 38 lines
kernel/bpf/verifier.c +34 -1
+                             /* we don't track register spills perfectly,
+                              * so fallback to force-precise instead of failing */
+                             if (bt_stack_mask(bt) != 0)
+                                     return -ENOTSUPP;
+                             /* propagate r1-r5 to the caller */
+                             for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+                                     if (bt_is_reg_set(bt, i)) {
+                                             bt_clear_reg(bt, i);
+                                             bt_set_frame_reg(bt, bt->frame - 1, i);
+                                     }
+                             }
+                             if (bt_subprog_exit(bt))
+                                     return -EFAULT;
+                             return 0;
+                     }
+             } else if ((bpf_helper_call(insn) &&
+                         is_callback_calling_function(insn->imm) &&
+                         !is_async_callback_calling_function(insn->imm)) ||
+                        (bpf_pseudo_kfunc_call(insn) && is_callback_calling_kfunc(insn->imm))) {
+                     /* callback-calling helper or kfunc call, which means
+                      * we are exiting from subprog, but unlike the subprog
+                      * call handling above, we shouldn't propagate
+                      * precision of r1-r5 (if any requested), as they are
+                      * not actually arguments passed directly to callback
+                      * subprogs
                       */
-                     if (insn->src_reg == 0 && is_callback_calling_function(insn->imm))
+                     if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS)
+                             return -EFAULT;
+                     if (bt_stack_mask(bt) != 0)
                              return -ENOTSUPP;
+                     /* clear r1-r5 in callback subprog's mask */
+                     for (i = BPF_REG_1; i <= BPF_REG_5; i++)
+                             bt_clear_reg(bt, i);
+                     if (bt_subprog_exit(bt))
+                             return -EFAULT;
+                     return 0;
Quoted 7 lines
Quoted 2 lines

jmp history will include callback insn, right?
So skip of jmp history is missing here ?

This is, say, call bpf_loop; instruction. Which means we just got out of bpf_loop's callback's jump history (so already "skipped" them, except we didn't skip, we properly processed them). So there is nothing to skip anymore. We are in the parent program already.

Got it. Makes sense now.

#34 lore raw

Re: [PATCH bpf-next 08/10] bpf: support precision propagation in the presence of subprogs

Quoted 44 lines · Alexei Starovoitov
Quoted 29 lines · Andrii Nakryiko
Quoted 18 lines · Alexei Starovoitov
Quoted 16 lines · Andrii Nakryiko
      if (insn->code == 0)
              return 0;
kernel/bpf/verifier.c +5 -5
@@ -3424,14 +3449,72 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
              if (class == BPF_STX)
                      bt_set_reg(bt, sreg);
      } else if (class == BPF_JMP || class == BPF_JMP32) {
-             if (opcode == BPF_CALL) {
-                     if (insn->src_reg == BPF_PSEUDO_CALL)
-                             return -ENOTSUPP;
-                     /* BPF helpers that invoke callback subprogs are
-                      * equivalent to BPF_PSEUDO_CALL above
+             if (bpf_pseudo_call(insn)) {
+                     int subprog_insn_idx, subprog;
+                     bool is_global;
+
+                     subprog_insn_idx = idx + insn->imm + 1;
Quoted 8 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3462

Quoted 6 lines
kernel/bpf/verifier.c +6 -0
+                     subprog = find_subprog(env, subprog_insn_idx);
+                     if (subprog < 0)
+                             return -EFAULT;
+                     is_global = subprog_is_global(env, subprog);
+
+                     if (is_global) {

could you add a warn_on here that checks that jmp history doesn't have insns from subprog.

Quoted 3 lines

wouldn't this be very expensive to go over the entire jmp history to check that no jump point there overlaps with the global function? Or what do you have in mind specifically for this check?

recalling how jmp_history works and reading this comment when we process any call:

        /* when we exit from subprog, we need to record non-linear history */
        mark_jmp_point(env, t + 1);

so for static subprog the history will be:
call
jmps inside subprog
insn after call.

Quoted 6 lines

for global it will be:
call
insn after call.

I was thinking that we can do simple check that call + 1 == subseq_idx for globals. For statics that should never be the case.

Got it. Yes, I think you are right, and it's simple to check and enforce. Will add.

Quoted 87 lines

We don't have to do it. Mostly checking my understanding of patches and jmp history.

Quoted 75 lines
Quoted 9 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3468

Quoted 8 lines
kernel/bpf/verifier.c +6 -0
+                             /* r1-r5 are invalidated after subprog call,
+                              * so for global func call it shouldn't be set
+                              * anymore
+                              */
+                             if (bt_reg_mask(bt) & BPF_REGMASK_ARGS)
+                                     return -EFAULT;
Quoted 3 lines

This shouldn't be happening, but backtracking is delicate.
Could you add verbose("backtracking bug") here, so we know why the prog got rejected. I'd probably do -ENOTSUPP, but EFAULT is ok too.

Will add verbose(). EFAULT because valid code should never use r1-r5 after call. Invalid code should be rejected before that, and if not, then it is really a bug and best to bail out.

Quoted 14 lines

kernel/bpf/verifier.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, · line 3480

Quoted 13 lines
kernel/bpf/verifier.c +12 -0
+                             /* global subprog always sets R0 */
+                             bt_clear_reg(bt, BPF_REG_0);
+                             return 0;
+                     } else {
+                             /* static subprog call instruction, which
+                              * means that we are exiting current subprog,
+                              * so only r1-r5 could be still requested as
+                              * precise, r0 and r6-r10 or any stack slot in
+                              * the current frame should be zero by now
+                              */
+                             if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS)
+                                     return -EFAULT;
Quoted 1 line

same here.

ack

Quoted 39 lines
Quoted 38 lines
kernel/bpf/verifier.c +34 -1
+                             /* we don't track register spills perfectly,
+                              * so fallback to force-precise instead of failing */
+                             if (bt_stack_mask(bt) != 0)
+                                     return -ENOTSUPP;
+                             /* propagate r1-r5 to the caller */
+                             for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+                                     if (bt_is_reg_set(bt, i)) {
+                                             bt_clear_reg(bt, i);
+                                             bt_set_frame_reg(bt, bt->frame - 1, i);
+                                     }
+                             }
+                             if (bt_subprog_exit(bt))
+                                     return -EFAULT;
+                             return 0;
+                     }
+             } else if ((bpf_helper_call(insn) &&
+                         is_callback_calling_function(insn->imm) &&
+                         !is_async_callback_calling_function(insn->imm)) ||
+                        (bpf_pseudo_kfunc_call(insn) && is_callback_calling_kfunc(insn->imm))) {
+                     /* callback-calling helper or kfunc call, which means
+                      * we are exiting from subprog, but unlike the subprog
+                      * call handling above, we shouldn't propagate
+                      * precision of r1-r5 (if any requested), as they are
+                      * not actually arguments passed directly to callback
+                      * subprogs
                       */
-                     if (insn->src_reg == 0 && is_callback_calling_function(insn->imm))
+                     if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS)
+                             return -EFAULT;
+                     if (bt_stack_mask(bt) != 0)
                              return -ENOTSUPP;
+                     /* clear r1-r5 in callback subprog's mask */
+                     for (i = BPF_REG_1; i <= BPF_REG_5; i++)
+                             bt_clear_reg(bt, i);
+                     if (bt_subprog_exit(bt))
+                             return -EFAULT;
+                     return 0;
Quoted 7 lines
Quoted 2 lines

jmp history will include callback insn, right?
So skip of jmp history is missing here ?

This is, say, call bpf_loop; instruction. Which means we just got out of bpf_loop's callback's jump history (so already "skipped" them, except we didn't skip, we properly processed them). So there is nothing to skip anymore. We are in the parent program already.

Got it. Makes sense now.

author maintainer
#35 lore raw

PATCH 9/10 selftests/bpf: add precision propagation tests in the presence of subprogs

In reply to Andrii Nakryiko

Add a bunch of tests validating verifier's precision backpropagation logic in the presence of subprog calls and/or callback-calling helpers/kfuncs.

We validate the following conditions:
- subprog_result_precise: static subprog r0 result precision handling;
- global_subprog_result_precise: global subprog r0 precision

    shortcutting, similar to BPF helper handling;
  - callback_result_precise: similarly r0 marking precise for
    callback-calling helpers;
  - parent_callee_saved_reg_precise, parent_callee_saved_reg_precise_global:
    propagation of precision for callee-saved registers bypassing
    static/global subprogs;
  - parent_callee_saved_reg_precise_with_callback: same as above, but in
    the presence of callback-calling helper;
  - parent_stack_slot_precise, parent_stack_slot_precise_global:
    similar to above, but instead propagating precision of stack slot
    (spilled SCALAR reg);
  - parent_stack_slot_precise_with_callback: same as above, but in the
    presence of callback-calling helper;
  - subprog_arg_precise: propagation of precision of static subprog's
    input argument back to caller;

- subprog_spill_into_parent_stack_slot_precise: negative test validating that verifier currently can't support backtracking of stack access with non-r10 register, we validate that we fallback to forcing precision for all SCALARs.

---
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 tools/testing/selftests/bpf/progs/bpf_misc.h  |   4 +
 .../bpf/progs/verifier_subprog_precision.c    | 536 ++++++++++++++++++
 3 files changed, 542 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
index 2497716ee379..531621adef42 100644
@@ -55,6 +55,7 @@
 #include "verifier_spill_fill.skel.h"
 #include "verifier_spin_lock.skel.h"
 #include "verifier_stack_ptr.skel.h"
+#include "verifier_subprog_precision.skel.h"
 #include "verifier_subreg.skel.h"
 #include "verifier_uninit.skel.h"
 #include "verifier_unpriv.skel.h"
@@ -154,6 +155,7 @@ void test_verifier_sock(void)                 { RUN(verifier_sock); }
 void test_verifier_spill_fill(void)           { RUN(verifier_spill_fill); }
 void test_verifier_spin_lock(void)            { RUN(verifier_spin_lock); }
 void test_verifier_stack_ptr(void)            { RUN(verifier_stack_ptr); }
+void test_verifier_subprog_precision(void)    { RUN(verifier_subprog_precision); }
 void test_verifier_subreg(void)               { RUN(verifier_subreg); }
 void test_verifier_uninit(void)               { RUN(verifier_uninit); }
 void test_verifier_unpriv(void)               { RUN(verifier_unpriv); }
tools/testing/selftests/bpf/progs/bpf_misc.h +4 -0
index d3c1217ba79a..38a57a2e70db 100644
@@ -86,6 +86,10 @@
 #define POINTER_VALUE	0xcafe4all
 #define TEST_DATA_LEN	64
 
+#ifndef __used
+#define __used __attribute__((used))
+#endif
+
 #if defined(__TARGET_ARCH_x86)
 #define SYSCALL_WRAPPER 1
 #define SYS_PREFIX "__x64_"
tools/testing/selftests/bpf/progs/verifier_subprog_precision.c new file +536 -0

536 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#36 lore raw

PATCH 10/10 selftests/bpf: revert iter test subprog precision workaround

In reply to Andrii Nakryiko

Now that precision propagation is supported fully in the presence of subprogs, there is no need to work around iter test. Revert original workaround.

This reverts be7dbd275dc6 ("selftests/bpf: avoid mark_all_scalars_precise() trigger in one of iter tests").

---
 tools/testing/selftests/bpf/progs/iters.c | 26 ++++++++++-------------
 1 file changed, 11 insertions(+), 15 deletions(-)
tools/testing/selftests/bpf/progs/iters.c +11 -15
index be16143ae292..6b9b3c56f009 100644
@@ -651,29 +651,25 @@ int iter_stack_array_loop(const void *ctx)
 	return sum;
 }
 
-#define ARR_SZ 16
-
-static __noinline void fill(struct bpf_iter_num *it, int *arr, int mul)
+static __noinline void fill(struct bpf_iter_num *it, int *arr, __u32 n, int mul)
 {
-	int *t;
-	__u64 i;
+	int *t, i;
 
 	while ((t = bpf_iter_num_next(it))) {
 		i = *t;
-		if (i >= ARR_SZ)
+		if (i >= n)
 			break;
 		arr[i] =  i * mul;
 	}
 }
 
-static __noinline int sum(struct bpf_iter_num *it, int *arr)
+static __noinline int sum(struct bpf_iter_num *it, int *arr, __u32 n)
 {
-	int *t, sum = 0;;
-	__u64 i;
+	int *t, i, sum = 0;;
 
 	while ((t = bpf_iter_num_next(it))) {
 		i = *t;
-		if (i >= ARR_SZ)
+		if (i >= n)
 			break;
 		sum += arr[i];
 	}
@@ -685,7 +681,7 @@ SEC("raw_tp")
 __success
 int iter_pass_iter_ptr_to_subprog(const void *ctx)
 {
-	int arr1[ARR_SZ], arr2[ARR_SZ];
+	int arr1[16], arr2[32];
 	struct bpf_iter_num it;
 	int n, sum1, sum2;
 
@@ -694,25 +690,25 @@ int iter_pass_iter_ptr_to_subprog(const void *ctx)
 	/* fill arr1 */
 	n = ARRAY_SIZE(arr1);
 	bpf_iter_num_new(&it, 0, n);
-	fill(&it, arr1, 2);
+	fill(&it, arr1, n, 2);
 	bpf_iter_num_destroy(&it);
 
 	/* fill arr2 */
 	n = ARRAY_SIZE(arr2);
 	bpf_iter_num_new(&it, 0, n);
-	fill(&it, arr2, 10);
+	fill(&it, arr2, n, 10);
 	bpf_iter_num_destroy(&it);
 
 	/* sum arr1 */
 	n = ARRAY_SIZE(arr1);
 	bpf_iter_num_new(&it, 0, n);
-	sum1 = sum(&it, arr1);
+	sum1 = sum(&it, arr1, n);
 	bpf_iter_num_destroy(&it);
 
 	/* sum arr2 */
 	n = ARRAY_SIZE(arr2);
 	bpf_iter_num_new(&it, 0, n);
-	sum2 = sum(&it, arr2);
+	sum2 = sum(&it, arr2, n);
 	bpf_iter_num_destroy(&it);
 
 	bpf_printk("sum1=%d, sum2=%d", sum1, sum2);
Signature · 1 line
2.34.1