Skip to contentSkip to conversation

Sleepable BPF programs on cgroup {get,set}sockopt

RFC New [email protected] 1 participant 7 messages Aug 19, 2023

Subsystems (5)
Kselftest BPF BPF selftests BPF core +1 more subsystems
lore
Share & tools
lore
#0 lore raw

v4 cover Sleepable BPF programs on cgroup {get,set}sockopt

From: Kui-Feng Lee <[email protected]>

Major Changes from v3:

- Remove bpf_sockopt_dynptr_copy_to() and bpf_sockopt_dynptr_install().

- Rename bpf_sockopt_dynptr_from() to bpf_dynptr_from_sockopt().

- Add PTR_TO_AUX[_END] and use it for optval and optval_end.

- bpf_dynptr_write() and bpf_dynptr_read() will write to/read from the user space buffer directly if necessary.

Major Changes from v2:

- Add test cases mixing sleepable and non-sleepable BPF programs.

- Don't expose bpf_sockopt_kern.flags to BPF programs.

- Rename kfuncs to *_sockopt_dynptr_*()

Major Changes from v1:

- Add bpf_so_optval_copy_to() and bpf_so_optval_copy_to_r() to copy data from a dynptr or raw buffer to the optval of a context, either in the kernel or user space, to simplify BPF programs.

- Restrict to having atmost one instance of dynptr initialized by bpf_so_optval_from() at any moment. It simplifies the memory management of the optval buffer in kernel.

- Fix the issue of bpf_prog_array_free() by replacing it with bpf_prog_array_free_sleepable().

Make BPF programs attached on cgroup/{get,set}sockopt hooks sleepable and able to call bpf_copy_from_user() and bpf_copy_to_user(), a new kfunc.

The Issue with CGroup {get,set}sockopt Hooks
============================================

Calling {get,set}sockopt from user space, optval is a pointer to a buffer. The format of the buffer depends on the level and optname, and its size is specified by optlen. The buffer is used by user space programs to pass values to setsockopt and retrieve values from getsockopt.

The problem is that BPF programs protected by RCU read lock cannot access the buffers located in user space. This is because these programs are non-sleepable and using copy_from_user() or copy_to_user() to access user space memory can result in paging.

The kernel makes a copy of the buffer specified by optval and optlen in kernel space before passing it to the cgroup {get,set}sockopt hooks. After the hooks are executed, the content of the buffer in kernel space is copied to user space if necessary.

Programs may send a significant amount of data, stored in buffer indicated by optval, to the kernel. One example is iptables, which can send several megabytes to the kernel. However, BPF programs on the hooks can only see up to the first PAGE_SIZE bytes of the buffer. The optlen value that BPF programs observe may appear to be PAGE_SIZE, but in reality, it is larger than that. On the other hand, the value of optlen represents the amount of data retrieved by getsockopt(). Additionally, both the buffer content and optlen can be modified by BPF programs.

Kernel may wrongly modify the value of optlen returned to user space to PAGE_SIZE. This can happen because the kernel cannot distinguish if the value was set by BPF programs or by the kernel itself.

To fix it, we perform various hacks; for example, the commit d8fe449a9c51 ("bpf: Don't return EINVAL from {get,set}sockopt when optlen > PAGE_SIZE") and the commit 29ebbba7d461 ("bpf: Don't EFAULT for {g,s}setsockopt with
wrong optlen").

Make CGroup {get,set}sockopt Hooks Sleepable
============================================

The long term solution is to make these hooks sleepable to enable BPF programs call copy_from_user() and copy_to_user(),
a.k.a. bpf_copy_from_user() and bpf_copy_to_user(). It prevents manipulation of optval and optlen values, and allows BPF programs to access the complete contents of the buffer referenced by optval.

Mix Sleepable and Non-Sleepable Programs
========================================

Installing both sleepable and non-sleepable programs simultaneously on the same hook leads to the mixing of sleepable and non-sleepable programs. For programs that cannot sleep, the kernel first copies data from the user buffer to a kernel buffer before invoking BPF programs. This process introduces intricate interactions between sleepable and non-sleepable programs.

For instance, due to kernel copies for non-sleepable programs, a sleepable program may receive optval in either the user space or the kernel space. These two scenarios require different handling approaches to update the buffer pointed to by optval. Consequently, sleepable programs can become significantly complex.

To simplify the programs, we introduce a set of kfuncs that enable data copying to optval without requiring knowledge of the underlying details. (bpf_sockopt_dynptr_copy_to() and bpf_sockopt_dynptr_copy_to_r())

---
v1: https://lore.kernel.org/bpf/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/

Kui-Feng Lee (6):
bpf: enable sleepable BPF programs attached to
cgroup/{get,set}sockopt.
libbpf: add sleepable sections for {get,set}sockopt() Add PTR_TO_AUX
bpf: Prevent BPF programs from access the buffer pointed by
user_optval.
bpf: Add a new dynptr type for CGRUP_SOCKOPT. selftests/bpf: Add test cases for sleepable BPF programs of the
CGROUP_SOCKOPT type

 include/linux/bpf.h                           |  15 +-
 include/linux/bpf_verifier.h                  |   6 +-
 include/linux/filter.h                        |  10 +
 kernel/bpf/btf.c                              |   3 +
 kernel/bpf/cgroup.c                           | 227 +++++++++++---
 kernel/bpf/helpers.c                          | 140 +++++++++
 kernel/bpf/verifier.c                         | 288 +++++++++++-------
 tools/lib/bpf/libbpf.c                        |   2 +
 .../testing/selftests/bpf/bpf_experimental.h  |  22 ++
 tools/testing/selftests/bpf/bpf_kfuncs.h      |  22 ++
 .../selftests/bpf/prog_tests/sockopt_sk.c     | 112 ++++++-
 .../testing/selftests/bpf/progs/sockopt_sk.c  | 254 +++++++++++++++
 .../selftests/bpf/verifier/sleepable.c        |   2 +-
 13 files changed, 947 insertions(+), 156 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

RFC v4 1/6 bpf: enable sleepable BPF programs attached to cgroup/{get,set}sockopt.

From: Kui-Feng Lee <[email protected]>

Enable sleepable cgroup/{get,set}sockopt hooks.

The sleepable BPF programs attached to cgroup/{get,set}sockopt hooks may received a pointer to the optval in user space instead of a kernel copy. ctx->optval and ctx->optval_end are the pointers to the begin and end of the user space buffer if receiving a user space buffer. No matter where the buffer is, sleepable programs can not access the content from the pointers directly. They are supposed to access the buffer through dynptr functions.

---
 include/linux/bpf.h    |   6 ++
 include/linux/filter.h |   6 ++
 kernel/bpf/cgroup.c    | 208 ++++++++++++++++++++++++++++++++---------
 kernel/bpf/verifier.c  |   5 +-
 4 files changed, 178 insertions(+), 47 deletions(-)
include/linux/bpf.h +6 -0
index cfabbcf47bdb..edb35bcfa548 100644
@@ -1769,9 +1769,15 @@ struct bpf_prog_array_item {
 
 struct bpf_prog_array {
 	struct rcu_head rcu;
+	u32 flags;
 	struct bpf_prog_array_item items[];
 };
 
+enum bpf_prog_array_flags {
+	BPF_PROG_ARRAY_F_SLEEPABLE = 1 << 0,
+	BPF_PROG_ARRAY_F_NON_SLEEPABLE = 1 << 1,
+};
+
 struct bpf_empty_prog_array {
 	struct bpf_prog_array hdr;
 	struct bpf_prog *null_prog;
include/linux/filter.h +6 -0
index 761af6b3cf2b..2aa2a96526de 100644
@@ -1337,12 +1337,18 @@ struct bpf_sockopt_kern {
 	s32		level;
 	s32		optname;
 	s32		optlen;
+	u32		flags;
 	/* for retval in struct bpf_cg_run_ctx */
 	struct task_struct *current_task;
 	/* Temporary "register" for indirect stores to ppos. */
 	u64		tmp_reg;
 };
 
+enum bpf_sockopt_kern_flags {
+	/* optval is a pointer to user space memory */
+	BPF_SOCKOPT_FLAG_OPTVAL_USER    = (1U << 0),
+};
+
 int copy_bpf_fprog_from_user(struct sock_fprog *dst, sockptr_t src, int len);
 
 struct bpf_sk_lookup_kern {
kernel/bpf/cgroup.c +163 -45

366 lines not on this page · raw message

kernel/bpf/verifier.c +3 -2
index 4ccca1f6c998..61be432b9420 100644
@@ -19549,7 +19549,8 @@ static bool can_be_sleepable(struct bpf_prog *prog)
 	}
 	return prog->type == BPF_PROG_TYPE_LSM ||
 	       prog->type == BPF_PROG_TYPE_KPROBE /* only for uprobes */ ||
-	       prog->type == BPF_PROG_TYPE_STRUCT_OPS;
+	       prog->type == BPF_PROG_TYPE_STRUCT_OPS ||
+	       prog->type == BPF_PROG_TYPE_CGROUP_SOCKOPT;
 }
 
 static int check_attach_btf_id(struct bpf_verifier_env *env)
@@ -19571,7 +19572,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
 	}
 
 	if (prog->aux->sleepable && !can_be_sleepable(prog)) {
-		verbose(env, "Only fentry/fexit/fmod_ret, lsm, iter, uprobe, and struct_ops programs can be sleepable\n");
+		verbose(env, "Only fentry/fexit/fmod_ret, lsm, iter, uprobe, cgroup, and struct_ops programs can be sleepable\n");
 		return -EINVAL;
 	}
 
Signature · 1 line
2.34.1
#2 lore raw

RFC v4 2/6 libbpf: add sleepable sections for {get,set}sockopt()

In reply to [email protected]

From: Kui-Feng Lee <[email protected]>

Enable libbpf users to define sleepable programs attached on {get,set}sockopt(). The sleepable programs should be defined with SEC("cgroup/getsockopt.s") and SEC("cgroup/setsockopt.s") respectively.

---
 tools/lib/bpf/libbpf.c | 2 ++
 1 file changed, 2 insertions(+)
tools/lib/bpf/libbpf.c +2 -0
index b14a4376a86e..ddd6dc166e3e 100644
@@ -8766,7 +8766,9 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("cgroup/getsockname6",	CGROUP_SOCK_ADDR, BPF_CGROUP_INET6_GETSOCKNAME, SEC_ATTACHABLE),
 	SEC_DEF("cgroup/sysctl",	CGROUP_SYSCTL, BPF_CGROUP_SYSCTL, SEC_ATTACHABLE),
 	SEC_DEF("cgroup/getsockopt",	CGROUP_SOCKOPT, BPF_CGROUP_GETSOCKOPT, SEC_ATTACHABLE),
+	SEC_DEF("cgroup/getsockopt.s",	CGROUP_SOCKOPT, BPF_CGROUP_GETSOCKOPT, SEC_ATTACHABLE | SEC_SLEEPABLE),
 	SEC_DEF("cgroup/setsockopt",	CGROUP_SOCKOPT, BPF_CGROUP_SETSOCKOPT, SEC_ATTACHABLE),
+	SEC_DEF("cgroup/setsockopt.s",	CGROUP_SOCKOPT, BPF_CGROUP_SETSOCKOPT, SEC_ATTACHABLE | SEC_SLEEPABLE),
 	SEC_DEF("cgroup/dev",		CGROUP_DEVICE, BPF_CGROUP_DEVICE, SEC_ATTACHABLE_OPT),
 	SEC_DEF("struct_ops+",		STRUCT_OPS, 0, SEC_NONE),
 	SEC_DEF("struct_ops.s+",	STRUCT_OPS, 0, SEC_SLEEPABLE),
Signature · 1 line
2.34.1
#4 lore raw

RFC v4 4/6 bpf: Prevent BPF programs from access the buffer pointed by user_optval.

In reply to [email protected]

From: Kui-Feng Lee <[email protected]>

Since the buffer pointed by ctx->optval can be in user space, BPF programs in kernel space should not access it directly. They should use kfuncs provided later to access data.

---
 kernel/bpf/cgroup.c   | 16 ++++++-
 kernel/bpf/verifier.c | 98 +++++++++++++++++++++----------------------
 2 files changed, 63 insertions(+), 51 deletions(-)
kernel/bpf/cgroup.c +14 -2
index b4f37960274d..1b2006dac4d5 100644
@@ -2495,12 +2495,24 @@ static bool cg_sockopt_is_valid_access(int off, int size,
 	case offsetof(struct bpf_sockopt, optval):
 		if (size != sizeof(__u64))
 			return false;
-		info->reg_type = PTR_TO_PACKET;
+		if (prog->aux->sleepable)
+			/* Prohibit access to the memory pointed by optval
+			 * in sleepable programs.
+			 */
+			info->reg_type = PTR_TO_AUX | MEM_USER;
+		else
+			info->reg_type = PTR_TO_AUX;
 		break;
 	case offsetof(struct bpf_sockopt, optval_end):
 		if (size != sizeof(__u64))
 			return false;
-		info->reg_type = PTR_TO_PACKET_END;
+		if (prog->aux->sleepable)
+			/* Prohibit access to the memory pointed by
+			 * optval_end in sleepable programs.
+			 */
+			info->reg_type = PTR_TO_AUX_END | MEM_USER;
+		else
+			info->reg_type = PTR_TO_AUX_END;
 		break;
 	case offsetof(struct bpf_sockopt, retval):
 		if (size != size_default)
kernel/bpf/verifier.c +49 -49

156 lines not on this page · raw message

Signature · 1 line
2.34.1
#3 lore raw

RFC v4 3/6 Add PTR_TO_AUX

In reply to [email protected]

From: Kui-Feng Lee <[email protected]>

---
 include/linux/bpf.h          |   2 +
 include/linux/bpf_verifier.h |   6 +-
 kernel/bpf/verifier.c        | 195 ++++++++++++++++++++++-------------
 3 files changed, 127 insertions(+), 76 deletions(-)
include/linux/bpf.h +2 -0
index edb35bcfa548..40a3d392b7f1 100644
@@ -865,6 +865,8 @@ enum bpf_reg_type {
 	PTR_TO_BUF,		 /* reg points to a read/write buffer */
 	PTR_TO_FUNC,		 /* reg points to a bpf program function */
 	CONST_PTR_TO_DYNPTR,	 /* reg points to a const struct bpf_dynptr */
+	PTR_TO_AUX,		 /* reg points to context aux memory */
+	PTR_TO_AUX_END,		 /* aux + len */
 	__BPF_REG_TYPE_MAX,
 
 	/* Extended reg_types. */
include/linux/bpf_verifier.h +3 -3
index f70f9ac884d2..eb1f9e18bc8d 100644
@@ -76,7 +76,7 @@ struct bpf_reg_state {
 	/* Fixed part of pointer offset, pointer types only */
 	s32 off;
 	union {
-		/* valid when type == PTR_TO_PACKET */
+		/* valid when type == PTR_TO_PACKET or PTR_TO_AUX */
 		int range;
 
 		/* valid when type == CONST_PTR_TO_MAP | PTR_TO_MAP_VALUE |
@@ -154,8 +154,8 @@ struct bpf_reg_state {
 	s32 s32_max_value; /* maximum possible (s32)value */
 	u32 u32_min_value; /* minimum possible (u32)value */
 	u32 u32_max_value; /* maximum possible (u32)value */
-	/* For PTR_TO_PACKET, used to find other pointers with the same variable
-	 * offset, so they can share range knowledge.
+	/* For PTR_TO_PACKET and PTR_TO_AUX, used to find other pointers
+	 * with the same variable offset, so they can share range knowledge.
 	 * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we
 	 * came from, when one is tested for != NULL.
 	 * For PTR_TO_MEM_OR_NULL this is used to identify memory allocation
kernel/bpf/verifier.c +122 -73

482 lines not on this page · raw message

Signature · 1 line
2.34.1
#5 lore raw

RFC v4 5/6 bpf: Add a new dynptr type for CGRUP_SOCKOPT.

In reply to [email protected]

From: Kui-Feng Lee <[email protected]>

The new dynptr type (BPF_DYNPTR_TYPE_CGROUP_SOCKOPT) will be used by BPF programs to create a buffer that can be installed on ctx to replace exisiting optval or user_optval.

Installation is only allowed if ctx->flags & BPF_SOCKOPT_FLAG_OPTVAL_REPLACE is true. It is enabled only for sleepable programs on the cgroup/setsockopt hook. BPF programs can install a new buffer holding by a dynptr to increase the size of optval passed to setsockopt().

Installation is not enabled for cgroup/getsockopt since you can not increased a buffer created, by user program, to return data from getsockopt().

---
 include/linux/bpf.h    |   7 ++-
 include/linux/filter.h |   4 ++
 kernel/bpf/btf.c       |   3 +
 kernel/bpf/cgroup.c    |   5 +-
 kernel/bpf/helpers.c   | 140 +++++++++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c  |  38 ++++++++++-
 6 files changed, 193 insertions(+), 4 deletions(-)
include/linux/bpf.h +6 -1
index 40a3d392b7f1..aad34298bfd3 100644
@@ -663,12 +663,15 @@ enum bpf_type_flag {
 	/* DYNPTR points to xdp_buff */
 	DYNPTR_TYPE_XDP		= BIT(16 + BPF_BASE_TYPE_BITS),
 
+	/* DYNPTR points to optval buffer of bpf_sockopt */
+	DYNPTR_TYPE_CGROUP_SOCKOPT = BIT(17 + BPF_BASE_TYPE_BITS),
+
 	__BPF_TYPE_FLAG_MAX,
 	__BPF_TYPE_LAST_FLAG	= __BPF_TYPE_FLAG_MAX - 1,
 };
 
 #define DYNPTR_TYPE_FLAG_MASK	(DYNPTR_TYPE_LOCAL | DYNPTR_TYPE_RINGBUF | DYNPTR_TYPE_SKB \
-				 | DYNPTR_TYPE_XDP)
+				 | DYNPTR_TYPE_XDP | DYNPTR_TYPE_CGROUP_SOCKOPT)
 
 /* Max number of base types. */
 #define BPF_BASE_TYPE_LIMIT	(1UL << BPF_BASE_TYPE_BITS)
@@ -1208,6 +1211,8 @@ enum bpf_dynptr_type {
 	BPF_DYNPTR_TYPE_SKB,
 	/* Underlying data is a xdp_buff */
 	BPF_DYNPTR_TYPE_XDP,
+	/* Underlying data is for the optval of a cgroup sock */
+	BPF_DYNPTR_TYPE_CGROUP_SOCKOPT,
 };
 
 int bpf_dynptr_check_size(u32 size);
include/linux/filter.h +4 -0
index 2aa2a96526de..df12fddd2f21 100644
@@ -1347,6 +1347,10 @@ struct bpf_sockopt_kern {
 enum bpf_sockopt_kern_flags {
 	/* optval is a pointer to user space memory */
 	BPF_SOCKOPT_FLAG_OPTVAL_USER    = (1U << 0),
+	/* able to install new optval */
+	BPF_SOCKOPT_FLAG_OPTVAL_REPLACE	= (1U << 1),
+	/* optval is referenced by a dynptr */
+	BPF_SOCKOPT_FLAG_OPTVAL_DYNPTR	= (1U << 2),
 };
 
 int copy_bpf_fprog_from_user(struct sock_fprog *dst, sockptr_t src, int len);
kernel/bpf/btf.c +3 -0
index 249657c466dd..6d6a040688be 100644
@@ -217,6 +217,7 @@ enum btf_kfunc_hook {
 	BTF_KFUNC_HOOK_SOCKET_FILTER,
 	BTF_KFUNC_HOOK_LWT,
 	BTF_KFUNC_HOOK_NETFILTER,
+	BTF_KFUNC_HOOK_CGROUP_SOCKOPT,
 	BTF_KFUNC_HOOK_MAX,
 };
 
@@ -7846,6 +7847,8 @@ static int bpf_prog_type_to_kfunc_hook(enum bpf_prog_type prog_type)
 		return BTF_KFUNC_HOOK_LWT;
 	case BPF_PROG_TYPE_NETFILTER:
 		return BTF_KFUNC_HOOK_NETFILTER;
+	case BPF_PROG_TYPE_CGROUP_SOCKOPT:
+		return BTF_KFUNC_HOOK_CGROUP_SOCKOPT;
 	default:
 		return BTF_KFUNC_HOOK_MAX;
 	}
kernel/bpf/cgroup.c +4 -1
index 1b2006dac4d5..b27a4fbc6ffe 100644
@@ -1866,6 +1866,8 @@ static int filter_setsockopt_progs_cb(void *arg,
 	if (max_optlen < 0)
 		return max_optlen;
 
+	ctx->flags = BPF_SOCKOPT_FLAG_OPTVAL_REPLACE;
+
 	if (copy_from_user(ctx->optval, optval,
 			   min(ctx->optlen, max_optlen)) != 0)
 		return -EFAULT;
@@ -1894,7 +1896,8 @@ int __cgroup_bpf_run_filter_setsockopt(struct sock *sk, int *level,
 	ctx.optlen = *optlen;
 	ctx.optval = optval;
 	ctx.optval_end = optval + *optlen;
-	ctx.flags = BPF_SOCKOPT_FLAG_OPTVAL_USER;
+	ctx.flags = BPF_SOCKOPT_FLAG_OPTVAL_USER |
+		BPF_SOCKOPT_FLAG_OPTVAL_REPLACE;
 
 	lock_sock(sk);
 	ret = bpf_prog_run_array_cg_cb(&cgrp->bpf, CGROUP_SETSOCKOPT,
kernel/bpf/helpers.c +140 -0

188 lines not on this page · raw message

kernel/bpf/verifier.c +36 -2
index 83731e998b09..15119ff90bff 100644
@@ -755,6 +755,8 @@ static const char *dynptr_type_str(enum bpf_dynptr_type type)
 		return "skb";
 	case BPF_DYNPTR_TYPE_XDP:
 		return "xdp";
+	case BPF_DYNPTR_TYPE_CGROUP_SOCKOPT:
+		return "cgroup_sockopt";
 	case BPF_DYNPTR_TYPE_INVALID:
 		return "<invalid>";
 	default:
@@ -836,6 +838,8 @@ static enum bpf_dynptr_type arg_to_dynptr_type(enum bpf_arg_type arg_type)
 		return BPF_DYNPTR_TYPE_SKB;
 	case DYNPTR_TYPE_XDP:
 		return BPF_DYNPTR_TYPE_XDP;
+	case DYNPTR_TYPE_CGROUP_SOCKOPT:
+		return BPF_DYNPTR_TYPE_CGROUP_SOCKOPT;
 	default:
 		return BPF_DYNPTR_TYPE_INVALID;
 	}
@@ -852,6 +856,8 @@ static enum bpf_type_flag get_dynptr_type_flag(enum bpf_dynptr_type type)
 		return DYNPTR_TYPE_SKB;
 	case BPF_DYNPTR_TYPE_XDP:
 		return DYNPTR_TYPE_XDP;
+	case BPF_DYNPTR_TYPE_CGROUP_SOCKOPT:
+		return DYNPTR_TYPE_CGROUP_SOCKOPT;
 	default:
 		return 0;
 	}
@@ -859,7 +865,8 @@ static enum bpf_type_flag get_dynptr_type_flag(enum bpf_dynptr_type type)
 
 static bool dynptr_type_refcounted(enum bpf_dynptr_type type)
 {
-	return type == BPF_DYNPTR_TYPE_RINGBUF;
+	return type == BPF_DYNPTR_TYPE_RINGBUF ||
+		type == BPF_DYNPTR_TYPE_CGROUP_SOCKOPT;
 }
 
 static void __mark_dynptr_reg(struct bpf_reg_state *reg,
@@ -10300,6 +10307,8 @@ enum special_kfunc_type {
 	KF_bpf_dynptr_slice,
 	KF_bpf_dynptr_slice_rdwr,
 	KF_bpf_dynptr_clone,
+	KF_bpf_sockopt_dynptr_release,
+	KF_bpf_dynptr_from_sockopt,
 };
 
 BTF_SET_START(special_kfunc_set)
@@ -10320,6 +10329,8 @@ BTF_ID(func, bpf_dynptr_from_xdp)
 BTF_ID(func, bpf_dynptr_slice)
 BTF_ID(func, bpf_dynptr_slice_rdwr)
 BTF_ID(func, bpf_dynptr_clone)
+BTF_ID(func, bpf_sockopt_dynptr_release)
+BTF_ID(func, bpf_dynptr_from_sockopt)
 BTF_SET_END(special_kfunc_set)
 
 BTF_ID_LIST(special_kfunc_list)
@@ -10342,6 +10353,8 @@ BTF_ID(func, bpf_dynptr_from_xdp)
 BTF_ID(func, bpf_dynptr_slice)
 BTF_ID(func, bpf_dynptr_slice_rdwr)
 BTF_ID(func, bpf_dynptr_clone)
+BTF_ID(func, bpf_sockopt_dynptr_release)
+BTF_ID(func, bpf_dynptr_from_sockopt)
 
 static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta)
 {
@@ -10995,6 +11008,19 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 			arg_type |= OBJ_RELEASE;
 			break;
 		case KF_ARG_PTR_TO_DYNPTR:
+			if (meta->func_id == special_kfunc_list[KF_bpf_sockopt_dynptr_release]) {
+				int ref_obj_id = dynptr_ref_obj_id(env, reg);
+
+				if (ref_obj_id < 0) {
+					verbose(env, "R%d is not a valid dynptr\n", regno);
+					return -EINVAL;
+				}
+
+				/* Required by check_func_arg_reg_off() */
+				arg_type |= ARG_PTR_TO_DYNPTR | OBJ_RELEASE;
+				meta->release_regno = regno;
+			}
+			break;
 		case KF_ARG_PTR_TO_ITER:
 		case KF_ARG_PTR_TO_LIST_HEAD:
 		case KF_ARG_PTR_TO_LIST_NODE:
@@ -11082,6 +11108,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 					verbose(env, "verifier internal error: missing ref obj id for parent of clone\n");
 					return -EFAULT;
 				}
+			} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_sockopt] &&
+				   (dynptr_arg_type & MEM_UNINIT)) {
+				dynptr_arg_type |= DYNPTR_TYPE_CGROUP_SOCKOPT;
 			}
 
 			ret = process_dynptr_func(env, regno, insn_idx, dynptr_arg_type, clone_ref_obj_id);
@@ -11390,7 +11419,12 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	 * PTR_TO_BTF_ID in bpf_kfunc_arg_meta, do the release now.
 	 */
 	if (meta.release_regno) {
-		err = release_reference(env, regs[meta.release_regno].ref_obj_id);
+		verbose(env, "release refcounted PTR_TO_BTF_ID %s\n",
+			meta.func_name);
+		if (meta.func_id == special_kfunc_list[KF_bpf_sockopt_dynptr_release])
+			err = unmark_stack_slots_dynptr(env, &regs[meta.release_regno]);
+		else
+			err = release_reference(env, regs[meta.release_regno].ref_obj_id);
 		if (err) {
 			verbose(env, "kfunc %s#%d reference has not been acquired before\n",
 				func_name, meta.func_id);
Signature · 1 line
2.34.1
#6 lore raw

RFC v4 6/6 selftests/bpf: Add test cases for sleepable BPF programs of the CGROUP_SOCKOPT type

In reply to [email protected]

From: Kui-Feng Lee <[email protected]>

Do the same test as non-sleepable ones.

---
 .../testing/selftests/bpf/bpf_experimental.h  |  22 ++
 tools/testing/selftests/bpf/bpf_kfuncs.h      |  22 ++
 .../selftests/bpf/prog_tests/sockopt_sk.c     | 112 +++++++-
 .../testing/selftests/bpf/progs/sockopt_sk.c  | 254 ++++++++++++++++++
 .../selftests/bpf/verifier/sleepable.c        |   2 +-
 5 files changed, 409 insertions(+), 3 deletions(-)
tools/testing/selftests/bpf/bpf_experimental.h +22 -0
index 209811b1993a..20821a5960f0 100644
@@ -131,4 +131,26 @@ extern int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *nod
  */
 extern struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) __ksym;
 
+/* Description
+ *	Release the buffer allocated by bpf_dynptr_from_sockopt.
+ * Returns
+ *	0 on success
+ *	-EINVAL if the buffer was not allocated by bpf_dynptr_from_sockopt
+ */
+extern int bpf_sockopt_dynptr_release(struct bpf_sockopt *sopt,
+				      struct bpf_dynptr *ptr) __ksym;
+
+/* Description
+ *	Initialize a dynptr to access the content of optval passing
+ *      to {get,set}sockopt()s.
+ * Returns
+ *	> 0 on success, the size of the allocated buffer
+ *	-ENOMEM or -EINVAL on failure
+ */
+extern int bpf_dynptr_from_sockopt(struct bpf_sockopt *sopt,
+				   struct bpf_dynptr *ptr__uninit) __ksym;
+
+extern int bpf_sockopt_grow_to(struct bpf_sockopt *sopt,
+			       __u32 newsize) __ksym;
+
 #endif
tools/testing/selftests/bpf/bpf_kfuncs.h +22 -0
index 642dda0e758a..f50a976a315d 100644
@@ -41,4 +41,26 @@ extern bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *ptr) __ksym;
 extern __u32 bpf_dynptr_size(const struct bpf_dynptr *ptr) __ksym;
 extern int bpf_dynptr_clone(const struct bpf_dynptr *ptr, struct bpf_dynptr *clone__init) __ksym;
 
+/* Description
+ *	Release the buffer allocated by bpf_dynptr_from_sockopt.
+ * Returns
+ *	0 on success
+ *	-EINVAL if the buffer was not allocated by bpf_dynptr_from_sockopt
+ */
+extern int bpf_sockopt_dynptr_release(struct bpf_sockopt *sopt,
+				      struct bpf_dynptr *ptr) __ksym;
+
+/* Description
+ *	Initialize a dynptr to access the content of optval passing
+ *      to {get,set}sockopt()s.
+ * Returns
+ *	> 0 on success, the size of the allocated buffer
+ *	-ENOMEM or -EINVAL on failure
+ */
+extern int bpf_dynptr_from_sockopt(struct bpf_sockopt *sopt,
+				   struct bpf_dynptr *ptr__uninit) __ksym;
+
+extern int bpf_sockopt_grow_to(struct bpf_sockopt *sopt,
+			       __u32 newsize) __ksym;
+
 #endif
tools/testing/selftests/bpf/prog_tests/sockopt_sk.c +110 -2

135 lines not on this page · raw message

tools/testing/selftests/bpf/progs/sockopt_sk.c +254 -0

285 lines not on this page · raw message

tools/testing/selftests/bpf/verifier/sleepable.c +1 -1
index 1f0d2bdc673f..4b6c1117ec9f 100644
@@ -85,7 +85,7 @@
 	.expected_attach_type = BPF_TRACE_RAW_TP,
 	.kfunc = "sched_switch",
 	.result = REJECT,
-	.errstr = "Only fentry/fexit/fmod_ret, lsm, iter, uprobe, and struct_ops programs can be sleepable",
+	.errstr = "Only fentry/fexit/fmod_ret, lsm, iter, uprobe, cgroup, and struct_ops programs can be sleepable",
 	.flags = BPF_F_SLEEPABLE,
 	.runs = -1,
 },
Signature · 1 line
2.34.1