Skip to contentSkip to conversation

bpf: allow ctx writes using BPF_ST_MEM instruction

PATCH Applied Eduard Zingerman 2 participants 5 messages Mar 4, 2023

Subsystems (6)
lore
Share & tools
lore

Eduard Zingerman

author maintainer
#0 lore raw

v2 cover bpf: allow ctx writes using BPF_ST_MEM instruction

Changes v1 -> v2, suggested by Alexei:
- Resolved conflict with recent commit:
6fcd486b3a0a ("bpf: Refactor RCU enforcement in the verifier");
- Variable ctx_access removed in function convert_ctx_accesses();
- Macro BPF_COPY_STORE renamed to BPF_EMIT_STORE and fixed to correctly extract original store instruction class from code.

Original message follows:

The function verifier.c:convert_ctx_access() applies some rewrites to BPF instructions that read from or write to the BPF program context. For example, the write instruction for the struct bpf_sockopt::retval field:

    *(u32 *)(r1 + offsetof(struct bpf_sockopt, retval)) = r2

Is transformed to:

    *(u64 *)(r1 + offsetof(struct bpf_sockopt_kern, tmp_reg)) = r9
    r9 = *(u64 *)(r1 + offsetof(struct bpf_sockopt_kern, current_task))
    r9 = *(u64 *)(r9 + offsetof(struct task_struct, bpf_ctx))
    *(u32 *)(r9 + offsetof(struct bpf_cg_run_ctx, retval)) = r2
    r9 = *(u64 *)(r1 + offsetof(struct bpf_sockopt_kern, tmp_reg))

Currently, the verifier only supports such transformations for LDX (memory-to-register read) and STX (register-to-memory write) instructions. Error is reported for ST instructions (immediate-to-memory write). This is fine because clang does not currently emit ST instructions.

However, new -mcpu=v4 clang flag is planned, which would allow to emit ST instructions (discussed in [1]).

This patch-set adjusts the verifier to support ST instructions in verifier.c:convert_ctx_access().

The patches #1 and #2 were previously shared as part of RFC [2]. The changes compared to that RFC are:
- In patch #1, a bug in the handling of the
struct __sk_buff::queue_mapping field was fixed.
- Patch #3 is added, which is a set of disassembler-based test cases for context access rewrites. The test cases cover all fields for which the handling code is modified in patch #1.

[1] Propose some new instructions for -mcpu=v4

    https://lore.kernel.org/bpf/[email protected]/
[2] RFC Support for BPF_ST instruction in LLVM C compiler
    https://lore.kernel.org/bpf/[email protected]/
[3] v1
    https://lore.kernel.org/bpf/[email protected]/

Eduard Zingerman (3):
bpf: allow ctx writes using BPF_ST_MEM instruction selftests/bpf: test if pointer type is tracked for BPF_ST_MEM selftests/bpf: Disassembler tests for verifier.c:convert_ctx_access()

 kernel/bpf/cgroup.c                           |  49 +-
 kernel/bpf/verifier.c                         | 110 +--
 net/core/filter.c                             |  79 +-
 tools/testing/selftests/bpf/Makefile          |   2 +-
 tools/testing/selftests/bpf/disasm.c          |   1 +
 tools/testing/selftests/bpf/disasm.h          |   1 +
 .../selftests/bpf/prog_tests/ctx_rewrite.c    | 917 ++++++++++++++++++
 tools/testing/selftests/bpf/verifier/ctx.c    |  11 -
 tools/testing/selftests/bpf/verifier/unpriv.c |  23 +
 9 files changed, 1069 insertions(+), 124 deletions(-)
 create mode 120000 tools/testing/selftests/bpf/disasm.c
 create mode 120000 tools/testing/selftests/bpf/disasm.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c
Signature · 1 line
2.39.1
author maintainer
#1 lore raw

PATCH v2 1/3 bpf: allow ctx writes using BPF_ST_MEM instruction

Lift verifier restriction to use BPF_ST_MEM instructions to write to context data structures. This requires the following changes:
- verifier.c:do_check() for BPF_ST updated to:
- no longer forbid writes to registers of type PTR_TO_CTX;
- track dst_reg type in the env->insn_aux_data[...].ptr_type field (same way it is done for BPF_STX and BPF_LDX instructions).
- verifier.c:convert_ctx_access() and various callbacks invoked by it are updated to handled BPF_ST instruction alongside BPF_STX.

---
 kernel/bpf/cgroup.c                        |  49 +++++----
 kernel/bpf/verifier.c                      | 110 ++++++++++-----------
 net/core/filter.c                          |  79 ++++++++-------
 tools/testing/selftests/bpf/verifier/ctx.c |  11 ---
 4 files changed, 126 insertions(+), 123 deletions(-)
kernel/bpf/cgroup.c +30 -19
index a4ae422b8f12..53edb8ad2471 100644
@@ -2223,10 +2223,12 @@ static u32 sysctl_convert_ctx_access(enum bpf_access_type type,
 				BPF_FIELD_SIZEOF(struct bpf_sysctl_kern, ppos),
 				treg, si->dst_reg,
 				offsetof(struct bpf_sysctl_kern, ppos));
-			*insn++ = BPF_STX_MEM(
-				BPF_SIZEOF(u32), treg, si->src_reg,
+			*insn++ = BPF_RAW_INSN(
+				BPF_CLASS(si->code) | BPF_MEM | BPF_SIZEOF(u32),
+				treg, si->src_reg,
 				bpf_ctx_narrow_access_offset(
-					0, sizeof(u32), sizeof(loff_t)));
+					0, sizeof(u32), sizeof(loff_t)),
+				si->imm);
 			*insn++ = BPF_LDX_MEM(
 				BPF_DW, treg, si->dst_reg,
 				offsetof(struct bpf_sysctl_kern, tmp_reg));
@@ -2376,10 +2378,17 @@ static bool cg_sockopt_is_valid_access(int off, int size,
 	return true;
 }
 
-#define CG_SOCKOPT_ACCESS_FIELD(T, F)					\
-	T(BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F),			\
-	  si->dst_reg, si->src_reg,					\
-	  offsetof(struct bpf_sockopt_kern, F))
+#define CG_SOCKOPT_READ_FIELD(F)					\
+	BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F),	\
+		    si->dst_reg, si->src_reg,				\
+		    offsetof(struct bpf_sockopt_kern, F))
+
+#define CG_SOCKOPT_WRITE_FIELD(F)					\
+	BPF_RAW_INSN((BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F) |	\
+		      BPF_MEM | BPF_CLASS(si->code)),			\
+		     si->dst_reg, si->src_reg,				\
+		     offsetof(struct bpf_sockopt_kern, F),		\
+		     si->imm)
 
 static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type,
 					 const struct bpf_insn *si,
@@ -2391,25 +2400,25 @@ static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type,
 
 	switch (si->off) {
 	case offsetof(struct bpf_sockopt, sk):
-		*insn++ = CG_SOCKOPT_ACCESS_FIELD(BPF_LDX_MEM, sk);
+		*insn++ = CG_SOCKOPT_READ_FIELD(sk);
 		break;
 	case offsetof(struct bpf_sockopt, level):
 		if (type == BPF_WRITE)
-			*insn++ = CG_SOCKOPT_ACCESS_FIELD(BPF_STX_MEM, level);
+			*insn++ = CG_SOCKOPT_WRITE_FIELD(level);
 		else
-			*insn++ = CG_SOCKOPT_ACCESS_FIELD(BPF_LDX_MEM, level);
+			*insn++ = CG_SOCKOPT_READ_FIELD(level);
 		break;
 	case offsetof(struct bpf_sockopt, optname):
 		if (type == BPF_WRITE)
-			*insn++ = CG_SOCKOPT_ACCESS_FIELD(BPF_STX_MEM, optname);
+			*insn++ = CG_SOCKOPT_WRITE_FIELD(optname);
 		else
-			*insn++ = CG_SOCKOPT_ACCESS_FIELD(BPF_LDX_MEM, optname);
+			*insn++ = CG_SOCKOPT_READ_FIELD(optname);
 		break;
 	case offsetof(struct bpf_sockopt, optlen):
 		if (type == BPF_WRITE)
-			*insn++ = CG_SOCKOPT_ACCESS_FIELD(BPF_STX_MEM, optlen);
+			*insn++ = CG_SOCKOPT_WRITE_FIELD(optlen);
 		else
-			*insn++ = CG_SOCKOPT_ACCESS_FIELD(BPF_LDX_MEM, optlen);
+			*insn++ = CG_SOCKOPT_READ_FIELD(optlen);
 		break;
 	case offsetof(struct bpf_sockopt, retval):
 		BUILD_BUG_ON(offsetof(struct bpf_cg_run_ctx, run_ctx) != 0);
@@ -2429,9 +2438,11 @@ static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type,
 			*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct task_struct, bpf_ctx),
 					      treg, treg,
 					      offsetof(struct task_struct, bpf_ctx));
-			*insn++ = BPF_STX_MEM(BPF_FIELD_SIZEOF(struct bpf_cg_run_ctx, retval),
-					      treg, si->src_reg,
-					      offsetof(struct bpf_cg_run_ctx, retval));
+			*insn++ = BPF_RAW_INSN(BPF_CLASS(si->code) | BPF_MEM |
+					       BPF_FIELD_SIZEOF(struct bpf_cg_run_ctx, retval),
+					       treg, si->src_reg,
+					       offsetof(struct bpf_cg_run_ctx, retval),
+					       si->imm);
 			*insn++ = BPF_LDX_MEM(BPF_DW, treg, si->dst_reg,
 					      offsetof(struct bpf_sockopt_kern, tmp_reg));
 		} else {
@@ -2447,10 +2458,10 @@ static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type,
 		}
 		break;
 	case offsetof(struct bpf_sockopt, optval):
-		*insn++ = CG_SOCKOPT_ACCESS_FIELD(BPF_LDX_MEM, optval);
+		*insn++ = CG_SOCKOPT_READ_FIELD(optval);
 		break;
 	case offsetof(struct bpf_sockopt, optval_end):
-		*insn++ = CG_SOCKOPT_ACCESS_FIELD(BPF_LDX_MEM, optval_end);
+		*insn++ = CG_SOCKOPT_READ_FIELD(optval_end);
 		break;
 	}
 
kernel/bpf/verifier.c +52 -58

172 lines not on this page · raw message

net/core/filter.c +44 -35

168 lines not on this page · raw message

tools/testing/selftests/bpf/verifier/ctx.c +0 -11
index c8eaf0536c24..2fd31612c0b8 100644
@@ -1,14 +1,3 @@
-{
-	"context stores via ST",
-	.insns = {
-	BPF_MOV64_IMM(BPF_REG_0, 0),
-	BPF_ST_MEM(BPF_DW, BPF_REG_1, offsetof(struct __sk_buff, mark), 0),
-	BPF_EXIT_INSN(),
-	},
-	.errstr = "BPF_ST stores into R1 ctx is not allowed",
-	.result = REJECT,
-	.prog_type = BPF_PROG_TYPE_SCHED_CLS,
-},
 {
 	"context stores via BPF_ATOMIC",
 	.insns = {
Signature · 1 line
2.39.1
author maintainer
#2 lore raw

PATCH v2 2/3 selftests/bpf: test if pointer type is tracked for BPF_ST_MEM

In reply to Eduard Zingerman

Check that verifier tracks pointer types for BPF_ST_MEM instructions and reports error if pointer types do not match for different execution branches.

---
 tools/testing/selftests/bpf/verifier/unpriv.c | 23 +++++++++++++++++++
 1 file changed, 23 insertions(+)
tools/testing/selftests/bpf/verifier/unpriv.c +23 -0
index 878ca26c3f0a..af0c0f336625 100644
@@ -239,6 +239,29 @@
 	.errstr = "same insn cannot be used with different pointers",
 	.prog_type = BPF_PROG_TYPE_SCHED_CLS,
 },
+{
+	/* Same as above, but use BPF_ST_MEM to save 42
+	 * instead of BPF_STX_MEM.
+	 */
+	"unpriv: spill/fill of different pointers st",
+	.insns = {
+	BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10),
+	BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8),
+	BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 3),
+	BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+	BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -16),
+	BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_2, 0),
+	BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1),
+	BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0),
+	BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0),
+	BPF_ST_MEM(BPF_W, BPF_REG_1, offsetof(struct __sk_buff, mark), 42),
+	BPF_MOV64_IMM(BPF_REG_0, 0),
+	BPF_EXIT_INSN(),
+	},
+	.result = REJECT,
+	.errstr = "same insn cannot be used with different pointers",
+	.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+},
 {
 	"unpriv: spill/fill of different pointers stx - ctx and sock",
 	.insns = {
Signature · 1 line
2.39.1
author maintainer
#3 lore raw

PATCH v2 3/3 selftests/bpf: Disassembler tests for verifier.c:convert_ctx_access()

In reply to Eduard Zingerman

Function verifier.c:convert_ctx_access() applies some rewrites to BPF instructions that read or write BPF program context. This commit adds machinery to allow test cases that inspect BPF program after these rewrites are applied.

An example of a test case:

  {
        // Shorthand for field offset and size specification
	N(CGROUP_SOCKOPT, struct bpf_sockopt, retval),

        // Pattern generated for field read
	.read  = "$dst = *(u64 *)($ctx + bpf_sockopt_kern::current_task);"
		 "$dst = *(u64 *)($dst + task_struct::bpf_ctx);"
		 "$dst = *(u32 *)($dst + bpf_cg_run_ctx::retval);",

        // Pattern generated for field write
	.write = "*(u64 *)($ctx + bpf_sockopt_kern::tmp_reg) = r9;"
		 "r9 = *(u64 *)($ctx + bpf_sockopt_kern::current_task);"
		 "r9 = *(u64 *)(r9 + task_struct::bpf_ctx);"
		 "*(u32 *)(r9 + bpf_cg_run_ctx::retval) = $src;"
		 "r9 = *(u64 *)($ctx + bpf_sockopt_kern::tmp_reg);" ,

},

For each test case, up to three programs are created:
- One that uses BPF_LDX_MEM to read the context field.
- One that uses BPF_STX_MEM to write to the context field.
- One that uses BPF_ST_MEM to write to the context field.

The disassembly of each program is compared with the pattern specified in the test case.

Kernel code for disassembly is reused (as is in the bpftool). To keep Makefile changes to the minimum, symbolic links to kernel/bpf/disasm.c and kernel/bpf/disasm.h are added.

---
 tools/testing/selftests/bpf/Makefile          |   2 +-
 tools/testing/selftests/bpf/disasm.c          |   1 +
 tools/testing/selftests/bpf/disasm.h          |   1 +
 .../selftests/bpf/prog_tests/ctx_rewrite.c    | 917 ++++++++++++++++++
 4 files changed, 920 insertions(+), 1 deletion(-)
 create mode 120000 tools/testing/selftests/bpf/disasm.c
 create mode 120000 tools/testing/selftests/bpf/disasm.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c
tools/testing/selftests/bpf/Makefile +1 -1
index eab3cf5399f5..16f404aa1b23 100644
@@ -559,7 +559,7 @@ TRUNNER_BPF_PROGS_DIR := progs
 TRUNNER_EXTRA_SOURCES := test_progs.c cgroup_helpers.c trace_helpers.c	\
 			 network_helpers.c testing_helpers.c		\
 			 btf_helpers.c flow_dissector_load.h		\
-			 cap_helpers.c test_loader.c xsk.c
+			 cap_helpers.c test_loader.c xsk.c disasm.c
 TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read $(OUTPUT)/bpf_testmod.ko	\
 		       $(OUTPUT)/liburandom_read.so			\
 		       $(OUTPUT)/xdp_synproxy				\
tools/testing/selftests/bpf/disasm.c new file +1 -0
new file mode 120000
index 000000000000..b1571927bd54
@@ -0,0 +1 @@
+../../../../kernel/bpf/disasm.c
\ No newline at end of file
tools/testing/selftests/bpf/disasm.h new file +1 -0
new file mode 120000
index 000000000000..8054fd497340
@@ -0,0 +1 @@
+../../../../kernel/bpf/disasm.h
\ No newline at end of file
tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c new file +917 -0

917 lines not on this page · raw message

Signature · 1 line
2.39.1
#4 lore raw
In reply to Eduard Zingerman
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 8 lines · On Sat, 4 Mar 2023 03:12:44 +0200 you wrote:

Changes v1 -> v2, suggested by Alexei:
- Resolved conflict with recent commit:
6fcd486b3a0a ("bpf: Refactor RCU enforcement in the verifier");
- Variable ctx_access removed in function convert_ctx_accesses();
- Macro BPF_COPY_STORE renamed to BPF_EMIT_STORE and fixed to correctly extract original store instruction class from code.

[...]

Here is the summary with links:
- [bpf-next,v2,1/3] bpf: allow ctx writes using BPF_ST_MEM instruction

    https://git.kernel.org/bpf/bpf-next/c/0d80a619c113
  - [bpf-next,v2,2/3] selftests/bpf: test if pointer type is tracked for BPF_ST_MEM
    https://git.kernel.org/bpf/bpf-next/c/806f81cd1ee3
  - [bpf-next,v2,3/3] selftests/bpf: Disassembler tests for verifier.c:convert_ctx_access()
    https://git.kernel.org/bpf/bpf-next/c/71cf4d027ad5

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html