Fix bpf_tcp_sock and bpf_sk_fullsock issue related to bpf_sk_release
Subsystems (7)
v4 cover Fix bpf_tcp_sock and bpf_sk_fullsock issue related to bpf_sk_release
This set addresses issue about accessing invalid ptr returned from bpf_tcp_sock() and bpf_sk_fullsock() after bpf_sk_release().
v4:
- Tried the one "id" approach. It does not work well and the reason is in the Patch 1 commit message.
- Rename refcount_id to ref_obj_id.
- With ref_obj_id, resetting reg->id to 0 is fine in mark_ptr_or_null_reg() because ref_obj_id is passed to release_reference() instead of reg->id.
- Also reset reg->ref_obj_id in mark_ptr_or_null_reg() when is_null == true
- sk_to_full_sk() is removed from bpf_sk_fullsock() and bpf_tcp_sock().
- bpf_get_listener_sock() is added to do sk_to_full_sk() in Patch 2.
- If tp is from bpf_tcp_sock(sk) and sk is a refcounted ptr, bpf_sk_release(tp) is also allowed.
v3:
- reset reg->refcount_id for the is_null case in mark_ptr_or_null_reg()
v2:
- Remove refcount_id arg from release_reference() because id == refcount_id
- Add a WARN_ON_ONCE to mark_ptr_or_null_regs() to catch an internal verifier bug.
Martin KaFai Lau (5):
bpf: Fix bpf_tcp_sock and bpf_sk_fullsock issue related to
bpf_sk_release
bpf: Add bpf_get_listener_sock(struct bpf_sock *sk) helper
bpf: Sync bpf.h to tools/
bpf: Test ref release issue in bpf_tcp_sock and bpf_sk_fullsock
bpf: Add an example for bpf_get_listener_sock
include/linux/bpf.h | 1 - include/linux/bpf_verifier.h | 40 +++++ include/uapi/linux/bpf.h | 11 +- kernel/bpf/verifier.c | 131 ++++++++------ net/core/filter.c | 27 ++- tools/include/uapi/linux/bpf.h | 11 +- tools/testing/selftests/bpf/bpf_helpers.h | 2 + .../bpf/progs/test_sock_fields_kern.c | 88 +++++++-- .../testing/selftests/bpf/test_sock_fields.c | 134 +++++++++++--- .../selftests/bpf/verifier/ref_tracking.c | 168 ++++++++++++++++++ tools/testing/selftests/bpf/verifier/sock.c | 4 +- 11 files changed, 506 insertions(+), 111 deletions(-)
Signature · 1 line
2.17.1
PATCH v4 1/5 bpf: Fix bpf_tcp_sock and bpf_sk_fullsock issue related to bpf_sk_release
Lorenz Bauer [thanks!] reported that a ptr returned by bpf_tcp_sock(sk) can still be accessed after bpf_sk_release(sk).
Both bpf_tcp_sock() and bpf_sk_fullsock() have the same issue. This patch addresses them together.
A simple reproducer looks like this:
sk = bpf_sk_lookup_tcp(); /* if (!sk) ... */ tp = bpf_tcp_sock(sk); /* if (!tp) ... */ bpf_sk_release(sk); snd_cwnd = tp->snd_cwnd; /* oops! The verifier does not complain. */
The problem is the verifier did not scrub the register's states of the tcp_sock ptr (tp) after bpf_sk_release(sk).
[ Note that when calling bpf_tcp_sock(sk), the sk is not always
refcount-acquired. e.g. bpf_tcp_sock(skb->sk). The verifier works fine for this case. ]
Currently, the verifier does not track if a helper's return ptr (in REG_0) is "carry"-ing one of its argument's refcount status. To carry this info, the reg1->id needs to be stored in reg0.
One approach was tried, like "reg0->id = reg1->id", when calling "bpf_tcp_sock()". The main idea was to avoid adding another "ref_obj_id" for the same reg. However, overlapping the NULL marking and ref tracking purpose in one "id" does not work well:
ref_sk = bpf_sk_lookup_tcp();
fullsock = bpf_sk_fullsock(ref_sk);
tp = bpf_tcp_sock(ref_sk);
if (!fullsock) {
bpf_sk_release(ref_sk);
return 0;
}
/* fullsock_reg->id is marked for NOT-NULL.
* Same for tp_reg->id because they have the same id.
*/
/* oops. verifier did not complain about the missing !tp check */
snd_cwnd = tp->snd_cwnd;Hence, a new "ref_obj_id" is needed in "struct bpf_reg_state". With a new ref_obj_id, when bpf_sk_release(sk) is called, the verifier can scrub all reg states which has a ref_obj_id match. It is done with the changes in release_reg_references() in this patch.
While fixing it, sk_to_full_sk() is removed from bpf_tcp_sock() and bpf_sk_fullsock() to avoid these helpers from returning another ptr. It will make bpf_sk_release(tp) possible:
sk = bpf_sk_lookup_tcp(); /* if (!sk) ... */ tp = bpf_tcp_sock(sk); /* if (!tp) ... */ bpf_sk_release(tp);
A separate helper "bpf_get_listener_sock()" will be added in a later patch to do sk_to_full_sk().
Misc change notes:
- To allow bpf_sk_release(tp), the arg of bpf_sk_release() is changed from ARG_PTR_TO_SOCKET to ARG_PTR_TO_SOCK_COMMON. ARG_PTR_TO_SOCKET is removed from bpf.h since no helper is using it.
- arg_type_is_refcounted() is renamed to arg_type_may_be_refcounted() because ARG_PTR_TO_SOCK_COMMON is the only one and skb->sk is not refcounted. All bpf_sk_release(), bpf_sk_fullsock() and bpf_tcp_sock() take ARG_PTR_TO_SOCK_COMMON.
- check_refcount_ok() ensures is_acquire_function() cannot take arg_type_may_be_refcounted() as its argument.
- The check_func_arg() can only allow one refcount-ed arg. It is guaranteed by check_refcount_ok() which ensures at most one arg can be refcounted. Hence, it is a verifier internal error if >1 refcount arg found in check_func_arg().
- In release_reference(), release_reference_state() is called first to ensure a match on "reg->ref_obj_id" can be found before scrubbing the reg states with release_reg_references().
- reg_is_refcounted() is no longer needed.
1. In mark_ptr_or_null_regs(), its usage is replaced by
"ref_obj_id && ref_obj_id == id" because,
when is_null == true, release_reference_state() should only be
called on the ref_obj_id obtained by a acquire helper (i.e.
is_acquire_function() == true). Otherwise, the following
would happen:
sk = bpf_sk_lookup_tcp();
/* if (!sk) { ... } */
fullsock = bpf_sk_fullsock(sk);
if (!fullsock) {
/*
* release_reference_state(fullsock_reg->ref_obj_id)
* where fullsock_reg->ref_obj_id == sk_reg->ref_obj_id.
*
* Hence, the following bpf_sk_release(sk) will fail
* because the ref state has already been released in the
* earlier release_reference_state(fullsock_reg->ref_obj_id).
*/
bpf_sk_release(sk);
} 2. In release_reg_references(), the current reg_is_refcounted() call is unnecessary because the id check is enough.
- The type_is_refcounted() and type_is_refcounted_or_null() are no longer needed also because reg_is_refcounted() is removed.
--- include/linux/bpf.h | 1 - include/linux/bpf_verifier.h | 40 +++++++++++ kernel/bpf/verifier.c | 131 ++++++++++++++++++++--------------- net/core/filter.c | 6 +- 4 files changed, 115 insertions(+), 63 deletions(-)
include/linux/bpf.h +0 -1
@@ -193,7 +193,6 @@ enum bpf_arg_type {ARG_PTR_TO_CTX ,/* pointer to context */ ARG_ANYTHING ,/* any (initialized) argument is ok */ -ARG_PTR_TO_SOCKET ,/* pointer to bpf_sock */ ARG_PTR_TO_SPIN_LOCK ,/* pointer to bpf_spin_lock */ ARG_PTR_TO_SOCK_COMMON, /* pointer to sock_common */ };
include/linux/bpf_verifier.h +40 -0
@@ -66,6 +66,46 @@ struct bpf_reg_state {* same reference to the socket, to determine proper reference freeing. */ u32 id; +/* PTR_TO_SOCKET and PTR_TO_TCP_SOCK could be a ptr returned +* from a pointer-cast helper, bpf_sk_fullsock() and +* bpf_tcp_sock(). +* +* Consider the following where "sk" is a reference counted +* pointer returned from "sk = bpf_sk_lookup_tcp();": +* +* 1: sk = bpf_sk_lookup_tcp(); +* 2: if (!sk) { return 0; } +* 3: fullsock = bpf_sk_fullsock(sk); +* 4: if (!fullsock) { bpf_sk_release(sk); return 0; } +* 5: tp = bpf_tcp_sock(fullsock); +* 6: if (!tp) { bpf_sk_release(sk); return 0; } +* 7: bpf_sk_release(sk); +* 8: snd_cwnd = tp->snd_cwnd; // verifier will complain +* +* After bpf_sk_release(sk) at line 7, both "fullsock" ptr and +* "tp" ptr should be invalidated also. In order to do that, +* the reg holding "fullsock" and "sk" need to remember +* the original refcounted ptr id (i.e. sk_reg->id) in ref_obj_id +* such that the verifier can reset all regs which have +* ref_obj_id matching the sk_reg->id. +* +* sk_reg->ref_obj_id is set to sk_reg->id at line 1. +* sk_reg->id will stay as NULL-marking purpose only. +* After NULL-marking is done, sk_reg->id can be reset to 0. +* +* After "fullsock = bpf_sk_fullsock(sk);" at line 3, +* fullsock_reg->ref_obj_id is set to sk_reg->ref_obj_id. +* +* After "tp = bpf_tcp_sock(fullsock);" at line 5, +* tp_reg->ref_obj_id is set to fullsock_reg->ref_obj_id +* which is the same as sk_reg->ref_obj_id. +* +* From the verifier perspective, if sk, fullsock and tp +* are not NULL, they are the same ptr with different +* reg->type. In particular, bpf_sk_release(tp) is also +* allowed and has the same effect as bpf_sk_release(sk). +*/ +u32 ref_obj_id;/* For scalar types (SCALAR_VALUE), this represents our knowledge of * the actual value. * For pointer types, this represents the variable part of the offset
kernel/bpf/verifier.c +74 -57
261 lines not on this page · raw message
net/core/filter.c +1 -5
@@ -1796,8 +1796,6 @@ static const struct bpf_func_proto bpf_skb_pull_data_proto = { BPF_CALL_1(bpf_sk_fullsock, struct sock *, sk) { - sk =sk_to_full_sk (sk); -return sk_fullsock (sk) ? (unsigned long )sk : (unsigned long )NULL ; }
@@ -5266,7 +5264,7 @@ static const struct bpf_func_proto bpf_sk_release_proto = { .func = bpf_sk_release, .gpl_only = false, .ret_type = RET_INTEGER, - .arg1_type = ARG_PTR_TO_SOCKET, + .arg1_type = ARG_PTR_TO_SOCK_COMMON, }; BPF_CALL_5(bpf_xdp_sk_lookup_udp, struct xdp_buff *, ctx,
@@ -5407,8 +5405,6 @@ u32 bpf_tcp_sock_convert_ctx_access(enum bpf_access_type type, BPF_CALL_1(bpf_tcp_sock, struct sock *, sk) { - sk =sk_to_full_sk (sk); -if (sk_fullsock (sk) && sk->sk_protocol ==IPPROTO_TCP )return (unsigned long )sk;
Signature · 1 line
2.17.1
PATCH v4 2/5 bpf: Add bpf_get_listener_sock(struct bpf_sock *sk) helper
In reply to Martin KaFai LauAdd a new helper "struct bpf_sock *bpf_get_listener_sock(struct bpf_sock *sk)" which returns a bpf_sock in TCP_LISTEN state. It will trace back to the listener sk from a request_sock if possible. It returns NULL for all other cases.
No reference is taken because the helper ensures the sk is in SOCK_RCU_FREE (where the TCP_LISTEN sock should be in). Hence, bpf_sk_release() is unnecessary and the verifier does not allow bpf_sk_release(listen_sk) to be called either.
The following is also allowed because the bpf_prog is run under rcu_read_lock():
sk = bpf_sk_lookup_tcp();
/* if (!sk) { ... } */
listen_sk = bpf_get_listener_sock(sk);
/* if (!listen_sk) { ... } */
bpf_sk_release(sk);
src_port = listen_sk->src_port; /* Allowed */--- include/uapi/linux/bpf.h | 11 ++++++++++- net/core/filter.c | 21 +++++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-)
include/uapi/linux/bpf.h +10 -1
@@ -2366,6 +2366,14 @@ union bpf_attr {* current value is ect (ECN capable). Works with IPv6 and IPv4. * Return * 1 if set, 0 if not set. +* +* struct bpf_sock *bpf_get_listener_sock(struct bpf_sock *sk) +* Description +* Return a **struct bpf_sock** pointer in TCP_LISTEN state. +* bpf_sk_release() is unnecessary and not allowed. +* Return +* A **struct bpf_sock** pointer on success, or NULL in +* case of failure. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -2465,7 +2473,8 @@ union bpf_attr {FN (spin_unlock), \FN (sk_fullsock), \FN (tcp_sock), \ -FN (skb_ecn_set_ce), +FN (skb_ecn_set_ce), \ +FN (get_listener_sock),/* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call
net/core/filter.c +21 -0
@@ -5418,6 +5418,23 @@ static const struct bpf_func_proto bpf_tcp_sock_proto = { .arg1_type = ARG_PTR_TO_SOCK_COMMON, }; +BPF_CALL_1(bpf_get_listener_sock, struct sock *, sk) +{ + sk =sk_to_full_sk (sk); + +if (sk->sk_state ==TCP_LISTEN &&sock_flag (sk,SOCK_RCU_FREE )) +return (unsigned long )sk; + +return (unsigned long )NULL ; +} + +static const struct bpf_func_proto bpf_get_listener_sock_proto = { + .func = bpf_get_listener_sock, + .gpl_only = false, + .ret_type =RET_PTR_TO_SOCKET_OR_NULL , + .arg1_type =ARG_PTR_TO_SOCK_COMMON , +}; + BPF_CALL_1(bpf_skb_ecn_set_ce, struct sk_buff *, skb) {unsigned int iphdr_len;
@@ -5603,6 +5620,8 @@ cg_skb_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)#ifdef CONFIG_INET case BPF_FUNC_tcp_sock:return &bpf_tcp_sock_proto; +case BPF_FUNC_get_listener_sock: +return &bpf_get_listener_sock_proto;case BPF_FUNC_skb_ecn_set_ce:return &bpf_skb_ecn_set_ce_proto;#endif
@@ -5698,6 +5717,8 @@ tc_cls_act_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return &bpf_sk_release_proto;case BPF_FUNC_tcp_sock:return &bpf_tcp_sock_proto; +case BPF_FUNC_get_listener_sock: +return &bpf_get_listener_sock_proto;#endif default :return bpf_base_func_proto (func_id);
Signature · 1 line
2.17.1
PATCH v4 3/5 bpf: Sync bpf.h to tools/
In reply to Martin KaFai LauThis patch sync the uapi bpf.h to tools/.
--- tools/include/uapi/linux/bpf.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
tools/include/uapi/linux/bpf.h +10 -1
@@ -2366,6 +2366,14 @@ union bpf_attr {* current value is ect (ECN capable). Works with IPv6 and IPv4. * Return * 1 if set, 0 if not set. +* +* struct bpf_sock *bpf_get_listener_sock(struct bpf_sock *sk) +* Description +* Return a **struct bpf_sock** pointer in TCP_LISTEN state. +* bpf_sk_release() is unnecessary and not allowed. +* Return +* A **struct bpf_sock** pointer on success, or NULL in +* case of failure. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -2465,7 +2473,8 @@ union bpf_attr {FN (spin_unlock), \FN (sk_fullsock), \FN (tcp_sock), \ -FN (skb_ecn_set_ce), +FN (skb_ecn_set_ce), \ +FN (get_listener_sock),/* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call
Signature · 1 line
2.17.1
PATCH v4 4/5 bpf: Test ref release issue in bpf_tcp_sock and bpf_sk_fullsock
In reply to Martin KaFai LauAdding verifier tests to ensure the ptr returned from bpf_tcp_sock() and bpf_sk_fullsock() cannot be accessed after bpf_sk_release() is called. A few of the tests are derived from a reproducer test by Lorenz Bauer.
--- .../selftests/bpf/verifier/ref_tracking.c | 168 ++++++++++++++++++ tools/testing/selftests/bpf/verifier/sock.c | 4 +- 2 files changed, 170 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/verifier/ref_tracking.c +168 -0
@@ -605,3 +605,171 @@ .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = ACCEPT, }, +{ + "reference tracking: use ptr from bpf_tcp_sock() after release", + .insns = { +BPF_SK_LOOKUP , +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_tcp_sock), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,3 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_7 , offsetof(struct bpf_tcp_sock , snd_cwnd)), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = REJECT, + .errstr = "invalid mem access", +}, +{ +"reference tracking: use ptr from bpf_sk_fullsock() after release" , + .insns = { +BPF_SK_LOOKUP , +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_sk_fullsock), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,3 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_7 , offsetof(struct bpf_sock , type)), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = REJECT, + .errstr = "invalid mem access", +}, +{ +"reference tracking: use ptr from bpf_sk_fullsock(tp) after release" , + .insns = { +BPF_SK_LOOKUP , +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_tcp_sock), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,3 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_sk_fullsock), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_6 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_6 , offsetof(struct bpf_sock , type)), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = REJECT, + .errstr = "invalid mem access", +}, +{ +"reference tracking: use sk after bpf_sk_release(tp)" , + .insns = { +BPF_SK_LOOKUP , +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_tcp_sock), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,3 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_6 , offsetof(struct bpf_sock , type)), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = REJECT, + .errstr = "invalid mem access", +}, +{ +"reference tracking: use ptr from bpf_get_listener_sock() after bpf_sk_release(sk)" , + .insns = { +BPF_SK_LOOKUP , +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_get_listener_sock), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,3 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_6 , offsetof(struct bpf_sock , src_port)), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = ACCEPT, +}, +{ +"reference tracking: bpf_sk_release(listen_sk)" , + .insns = { +BPF_SK_LOOKUP , +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_get_listener_sock), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,3 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_6 , offsetof(struct bpf_sock , type)), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = REJECT, + .errstr = "reference has not been acquired before", +}, +{ +/* !bpf_sk_fullsock(sk) is checked but !bpf_tcp_sock(sk) is not checked */ +"reference tracking: tp->snd_cwnd after bpf_sk_fullsock(sk) and bpf_tcp_sock(sk)" , + .insns = { +BPF_SK_LOOKUP , +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_6 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_EMIT_CALL (BPF_FUNC_sk_fullsock), +BPF_MOV64_REG (BPF_REG_7 ,BPF_REG_0 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_tcp_sock), +BPF_MOV64_REG (BPF_REG_8 ,BPF_REG_0 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_7 ,0 ,3 ), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), +BPF_EXIT_INSN (), +BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_8 , offsetof(struct bpf_tcp_sock , snd_cwnd)), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_6 ), +BPF_EMIT_CALL (BPF_FUNC_sk_release), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = REJECT, + .errstr = "invalid mem access", +},
tools/testing/selftests/bpf/verifier/sock.c +2 -2
@@ -342,7 +342,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "type=sock_common expected=sock", + .errstr = "reference has not been acquired before", }, { "bpf_sk_release(bpf_sk_fullsock(skb->sk))",
@@ -380,5 +380,5 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "type=tcp_sock expected=sock", + .errstr = "reference has not been acquired before", },
Signature · 1 line
2.17.1
PATCH v4 5/5 bpf: Add an example for bpf_get_listener_sock
In reply to Martin KaFai LauThis patch adds an example in using the new helper bpf_get_listener_sock().
--- tools/testing/selftests/bpf/bpf_helpers.h | 2 + .../bpf/progs/test_sock_fields_kern.c | 88 +++++++++--- .../testing/selftests/bpf/test_sock_fields.c | 134 ++++++++++++++---- 3 files changed, 180 insertions(+), 44 deletions(-)
tools/testing/selftests/bpf/bpf_helpers.h +2 -0
@@ -180,6 +180,8 @@ static struct bpf_sock *(*bpf_sk_fullsock)(struct bpf_sock *sk) = (void *) BPF_FUNC_sk_fullsock;static struct bpf_tcp_sock *(*bpf_tcp_sock)(struct bpf_sock *sk) = (void *) BPF_FUNC_tcp_sock; +static struct bpf_sock *(*bpf_get_listener_sock)(struct bpf_sock *sk) = + (void *) BPF_FUNC_get_listener_sock;static int (*bpf_skb_ecn_set_ce)(void *ctx) = (void *) BPF_FUNC_skb_ecn_set_ce;
tools/testing/selftests/bpf/progs/test_sock_fields_kern.c +72 -16
142 lines not on this page · raw message
tools/testing/selftests/bpf/test_sock_fields.c +106 -28
224 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 4 lines · Martin KaFai Lau
This set addresses issue about accessing invalid ptr returned from
bpf_tcp_sock()andbpf_sk_fullsock()afterbpf_sk_release().
Quoted 11 lines
v4:
- Tried the one "id" approach. It does not work well and the reason is in the Patch 1 commit message.
- Renamerefcount_idtoref_obj_id.
- Withref_obj_id, resettingreg->idto 0 is fine inmark_ptr_or_null_reg()becauseref_obj_idis passed torelease_reference()instead ofreg->id.
- Also resetreg->ref_obj_idinmark_ptr_or_null_reg()whenis_null== true
-sk_to_full_sk()is removed frombpf_sk_fullsock()andbpf_tcp_sock().
-bpf_get_listener_sock()is added to dosk_to_full_sk()in Patch 2.
- If tp is frombpf_tcp_sock(sk)and sk is a refcounted ptr,bpf_sk_release(tp)is also allowed.
Thanks for the hard work on this fix.
I applied the set to bpf tree.
I have a small question regarding patch 2:
+BPF_CALL_1(bpf_get_listener_sock, struct sock *, sk) +{
+ sk = sk_to_full_sk(sk);
+
+ if (sk->sk_state == TCP_LISTEN && sock_flag(sk, SOCK_RCU_FREE))
+ return (unsigned long)sk;
+
+ return (unsigned long)NULL;
+}
My understanding that sk->sk_state == TCP_LISTEN is enough for correctness and additional sock_flag(sk, SOCK_RCU_FREE) check is to prevent silent breakage in case listener socks will be re-implemented without rcu_free? In such case should we add warn_on_once(!sock_flag(sk, SOCK_RCU_FREE)) too? I mean in the very unlikely scenario that kernel will have such drastic implementation change the bpf progs will not work correctly because NULL is returned, but it will be harder for humans to debug?
I think it's also ok without warn, since such huge re-implemenation will require all sorts of efforts, so highly unlikely we will miss this spot. warn_on_once feels like too much paranoia. May be just a comment ? Or I'm overthinking?
I guess I'm fine with the code as-is.
Quoted 17 lines · Alexei Starovoitov
Quoted 4 lines · Martin KaFai Lau
This set addresses issue about accessing invalid ptr returned from
bpf_tcp_sock()andbpf_sk_fullsock()afterbpf_sk_release().Quoted 11 lines
v4:
- Tried the one "id" approach. It does not work well and the reason is in the Patch 1 commit message.
- Renamerefcount_idtoref_obj_id.
- Withref_obj_id, resettingreg->idto 0 is fine inmark_ptr_or_null_reg()becauseref_obj_idis passed torelease_reference()instead ofreg->id.
- Also resetreg->ref_obj_idinmark_ptr_or_null_reg()whenis_null== true
-sk_to_full_sk()is removed frombpf_sk_fullsock()andbpf_tcp_sock().
-bpf_get_listener_sock()is added to dosk_to_full_sk()in Patch 2.
- If tp is frombpf_tcp_sock(sk)and sk is a refcounted ptr,bpf_sk_release(tp)is also allowed.
Quoted 2 lines
Thanks for the hard work on this fix.
I applied the set to bpf tree.
Thanks for reviewing.
Quoted 12 lines
I have a small question regarding patch 2:
(quoted diff) +9
-0+BPF_CALL_1(bpf_get_listener_sock, struct sock *, sk) +{ + sk = sk_to_full_sk(sk); + + if (sk->sk_state == TCP_LISTEN && sock_flag(sk, SOCK_RCU_FREE)) + return (unsigned long)sk; + + return (unsigned long)NULL; +}
Quoted 11 lines
My understanding that
sk->sk_state==TCP_LISTENis enough for correctness and additionalsock_flag(sk, SOCK_RCU_FREE)check is to prevent silent breakage in case listener socks will be re-implemented withoutrcu_free? In such case should we addwarn_on_once(!sock_flag(sk, SOCK_RCU_FREE))too? I mean in the very unlikely scenario that kernel will have such drastic implementation change the bpf progs will not work correctly because NULL is returned, but it will be harder for humans to debug?
I think it's also ok without warn, since such huge re-implemenation will require all sorts of efforts, so highly unlikely we will miss this spot.warn_on_oncefeels like too much paranoia. May be just a comment ? Or I'm overthinking?
No. It is a valid question. Sorry that I missed details in this patch.
Here is what I had thought about in the patch:
1. inet_reqsk(sk)->rsk_listener is only set for TCP (and dccp). Hence, sk_to_full_sk(sk) is only useful for TCP (and dccp).
2. The TCP_LISTEN sock has SOCK_RCU_FREE set when it is added to the listening_hash in __inet_hash(). It is done for TCP (and dccp).
3. However, not all TCP_LISTEN socket is in SOCK_RCU_FREE. e.g. af_unix.c. It may be accessible through skb->sk. If WARN_ON_ONCE() was used, it would be a false alarm.
For those non TCP sk, sk_to_full_sk(sk) is a no-op. If its passed-in sk is not already in TCP_LISTEN, bpf_get_listener_sock() has to return NULL anyway.
It is not ideal for those sk because this helper still returns NULL if its passed-in sk is already TCP_LISTEN because of the sock_flag(sk, SOCK_RCU_FREE) check.
On the other hand, there is sk->sk_state for the bpf_prog to check first before calling this helper.
I have re-thought about (3). How about this:
1. Limit this helper to "sk->sk_protocol == IPPROTO_TCP". Return NULL for others because this helper cannot do anything extra for those sk anyway.
2. Rename this helper to bpf_get_tcp_listener_sock(). It still returns "struct bpf_sock *" because I think the send_cwnd/srtt_us/rtt_min/... of the listener's "struct bpf_tcp_sock *" is not very useful.
If needed, bpf_tcp_sock() can be used to get "struct bpf_tcp_sock *".
3. Add WARN_ON_ONCE() on the SOCK_RCU_FREE flag.
Quoted 63 lines · Martin Lau
Quoted 17 lines · Alexei Starovoitov
Quoted 4 lines · Martin KaFai Lau
This set addresses issue about accessing invalid ptr returned from
bpf_tcp_sock()andbpf_sk_fullsock()afterbpf_sk_release().Quoted 11 lines
v4:
- Tried the one "id" approach. It does not work well and the reason is in the Patch 1 commit message.
- Renamerefcount_idtoref_obj_id.
- Withref_obj_id, resettingreg->idto 0 is fine inmark_ptr_or_null_reg()becauseref_obj_idis passed torelease_reference()instead ofreg->id.
- Also resetreg->ref_obj_idinmark_ptr_or_null_reg()whenis_null== true
-sk_to_full_sk()is removed frombpf_sk_fullsock()andbpf_tcp_sock().
-bpf_get_listener_sock()is added to dosk_to_full_sk()in Patch 2.
- If tp is frombpf_tcp_sock(sk)and sk is a refcounted ptr,bpf_sk_release(tp)is also allowed.Quoted 2 lines
Thanks for the hard work on this fix.
I applied the set to bpf tree.Thanks for reviewing.
Quoted 12 lines
I have a small question regarding patch 2:
(quoted diff) +9
-0+BPF_CALL_1(bpf_get_listener_sock, struct sock *, sk) +{ + sk = sk_to_full_sk(sk); + + if (sk->sk_state == TCP_LISTEN && sock_flag(sk, SOCK_RCU_FREE)) + return (unsigned long)sk; + + return (unsigned long)NULL; +}Quoted 11 lines
My understanding that
sk->sk_state==TCP_LISTENis enough for correctness and additionalsock_flag(sk, SOCK_RCU_FREE)check is to prevent silent breakage in case listener socks will be re-implemented withoutrcu_free? In such case should we addwarn_on_once(!sock_flag(sk, SOCK_RCU_FREE))too? I mean in the very unlikely scenario that kernel will have such drastic implementation change the bpf progs will not work correctly because NULL is returned, but it will be harder for humans to debug?
I think it's also ok without warn, since such huge re-implemenation will require all sorts of efforts, so highly unlikely we will miss this spot.warn_on_oncefeels like too much paranoia. May be just a comment ? Or I'm overthinking?No. It is a valid question. Sorry that I missed details in this patch.
Here is what I had thought about in the patch:
1. inet_reqsk(sk)->rsk_listener is only set for TCP (and dccp). Hence,sk_to_full_sk(sk)is only useful for TCP (and dccp).2. The
TCP_LISTENsock hasSOCK_RCU_FREEset when it is added to thelistening_hashin__inet_hash(). It is done for TCP (and dccp).3. However, not all
TCP_LISTENsocket is inSOCK_RCU_FREE. e.g.af_unix.c. It may be accessible throughskb->sk. IfWARN_ON_ONCE()was used, it would be a false alarm.For those non TCP sk,
sk_to_full_sk(sk)is a no-op. If its passed-in sk is not already inTCP_LISTEN,bpf_get_listener_sock()has to return NULL anyway.
Quoted 6 lines
It is not ideal for those sk because this helper still returns NULL if its passed-in sk is already
TCP_LISTENbecause of thesock_flag(sk, SOCK_RCU_FREE)check.On the other hand, there is
sk->sk_statefor thebpf_progto check first before calling this helper.
all of the above makes sense to me. thanks for explaining. With that I think it's best to leave it as-is.
I don't see how any of the below options make it better.
Quoted 4 lines
I have re-thought about (3). How about this:
1. Limit this helper to "sk->sk_protocol==IPPROTO_TCP". Return NULL for others because this helper cannot do anything extra for those sk anyway.
protocol is already in bpf_sock and prog can check it if necessary.
Quoted 7 lines
2. Rename this helper to
bpf_get_tcp_listener_sock(). It still returns "structbpf_sock*" because I think the send_cwnd/srtt_us/rtt_min/... of the listener's "structbpf_tcp_sock*" is not very useful.If needed,
bpf_tcp_sock()can be used to get "structbpf_tcp_sock*".3. Add
WARN_ON_ONCE()on theSOCK_RCU_FREEflag.
not needed as you explained it will be false alarm.