bpf: Extend the bpf_list family of APIs
Subsystems (5)
v11 cover bpf: Extend the bpf_list family of APIs
In BPF, a list can only be used to implement a stack structure. Due to an incomplete API set, only FIFO or LIFO operations are supported. The patches enhance the BPF list API, making it more list-like.
Five new kfuncs have been added:bpf_list_del: remove a node from the listbpf_list_add_impl: insert a node after a given list nodebpf_list_is_first: check if a node is the first in the listbpf_list_is_last: check if a node is the last in the listbpf_list_empty: check if the list is empty
And add test cases for the aforementioned kfuncs.
Changes in v11:
- Move [PATCH v10 7/8] earlier (Eduard Zingerman)
- Fix the synchronization issue in [PATCH v10 2/8] (Eduard Zingerman, Alexei Starovoitov)
Changes in v10:
- Remove the table-driven approach (Ihor Solodrai)
- Use the __nonown_allowed suffix for bpf_list_del/front/back
- Add test cases for __nonown_allowed
Changes in v9:
- Expand table-driven approach coverage (Emil Tsalapatis)
- Clear list node owner and unlink before drop (Emil Tsalapatis)
- Remove warnings caused by WARN_ON_ONCE() (Emil Tsalapatis)
- Introduce the __nonown_allowed suffix (Alexei Starovoitov)
Changes in v8:
- Use [patch v7 5/5] as the start of the patch series (Leon Hwang)
- Introduce double pointer prev_ptr in __bpf_list_del
(Kumar Kartikeya Dwivedi)
- Extract refactored __bpf_list_del/add into separate patches (Leon Hwang)
- Allow bpf_list_front/back result as the prev argument of bpf_list_add
- Split test cases (Leon Hwang)
Changes in v7:
- Replace bpf_list_node_is_edge with bpf_list_is_first/is_last
- Reimplement __bpf_list_del and __bpf_list_add (Kumar Kartikeya Dwivedi)
- Simplify test cases (Mykyta Yatsenko)
Changes in v6:
- Merge [patch v5 (2,4,6)/6] into [patch v6 4/5] (Leon Hwang)
- If list_head was 0-initialized, init it
- refactor kfunc checks to table-driven approach (Leon Hwang)
Changes in v5:
- Fix bpf_obj leak on bpf_list_add_impl error
Changes in v4:
- [patch v3 1/6] Revert to version v1 (Alexei Starovoitov)
- Change the parameters of bpf_list_add_impl to (head, new, prev, ...)
Changes in v3:
- Add a new lock_rec member to struct bpf_reference_state for lock holding detection.
- Add test cases to verify that the verifier correctly restricts calls to bpf_list_del when the spin_lock is not held.
Changes in v2:
- Remove the head parameter from bpf_list_del (Alexei Starovoitov)
- Add bpf_list_add/is_first/is_last/empty to API and test cases (Alexei Starovoitov)
Link to v10:
https://lore.kernel.org/all/[email protected]/
Link to v9:
https://lore.kernel.org/all/[email protected]/
Link to v8:
https://lore.kernel.org/all/[email protected]/
Link to v7:
https://lore.kernel.org/all/[email protected]/
Link to v6:
https://lore.kernel.org/all/[email protected]/
Link to v5:
https://lore.kernel.org/all/[email protected]/
Link to v4:
https://lore.kernel.org/all/[email protected]/
Link to v3:
https://lore.kernel.org/all/[email protected]/
Link to v2:
https://lore.kernel.org/all/[email protected]/
Link to v1:
https://lore.kernel.org/all/[email protected]/
Kaitao Cheng (8):
bpf: refactor __bpf_list_del to take list node pointer
bpf: clear list node owner and unlink before drop
bpf: allow non-owning list-node args via __nonown_allowed
bpf: Introduce the bpf_list_del kfunc.
bpf: refactor __bpf_list_add to take insertion point via **prev_ptr
bpf: Add bpf_list_add to insert node after a given list node
bpf: add bpf_list_is_first/last/empty kfuncs selftests/bpf: Add test cases for
bpf_list_del/add/is_first/is_last/empty
Documentation/bpf/kfuncs.rst | 22 +- kernel/bpf/helpers.c | 147 ++++-- kernel/bpf/verifier.c | 44 +- .../selftests/bpf/progs/refcounted_kptr.c | 421 ++++++++++++++++++ 4 files changed, 601 insertions(+), 33 deletions(-)
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v11 1/8 bpf: refactor __bpf_list_del to take list node pointer
From: Kaitao Cheng <[email protected]>
Refactor __bpf_list_del to accept (head, struct list_head *n) instead of (head, bool tail). The caller now passes the specific node to remove:bpf_list_pop_front passes h->next, bpf_list_pop_back passes h->prev.
Prepares for introducing bpf_list_del(head, node) kfunc to remove an arbitrary node when the user holds ownership.
--- kernel/bpf/helpers.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-)
kernel/bpf/helpers.c +15 -8
@@ -2550,37 +2550,44 @@ __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,return bpf_list_push_back (head, node, meta__ign, off); } -static struct bpf_list_node *__bpf_list_del (struct bpf_list_head *head,bool tail) +static struct bpf_list_node *__bpf_list_del (struct bpf_list_head *head, +struct list_head *n) { -struct list_head *n, *h = (void *)head; +struct list_head *h = (void *)head;struct bpf_list_node_kern *node;/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't * called on its fields, so init here */ -if (unlikely (!h->next)) +if (unlikely (!h->next)) {INIT_LIST_HEAD (h); +return NULL ; + }if (list_empty (h))return NULL ; - n = tail ? h->prev : h->next; node = container_of(n, struct bpf_list_node_kern, list_head); -if (WARN_ON_ONCE (READ_ONCE (node->owner) != head)) +if (unlikely (READ_ONCE (node->owner) != head))return NULL ;list_del_init (n); -WRITE_ONCE (node->owner,NULL ); +/* Ensure __bpf_list_add() sees the node as unlinked. */ +smp_store_release (&node->owner,NULL );return (struct bpf_list_node *)n; } __bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) { -return __bpf_list_del (head, false); +struct list_head *h = (void *)head; + +return __bpf_list_del (head, h->next); } __bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) { -return __bpf_list_del (head, true); +struct list_head *h = (void *)head; + +return __bpf_list_del (head, h->prev); } __bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v11 2/8 bpf: clear list node owner and unlink before drop
In reply to Kaitao ChengFrom: Kaitao Cheng <[email protected]>
The issue only becomes exposed once bpf_list_del() is available: callers can pass an arbitrary bpf_list_head and bpf_list_node pair, including nodes that are not actually linked to the supplied head, or nodes that outlive their original head after refcount-based retention. This was not practically reachable for callers restricted to pop-style helpers alone; bpf_list_del() widens the API surface.
A failure mode appears when bpf_list_head_free() runs while a program still holds an independent refcount on a node (for example via bpf_refcount_acquire()). The list head value embedded in map memory can go away while the node object survives. If node->owner is left pointing at the old head address until drop completes, that pointer becomes stale. If a new bpf_list_head is later allocated at the same address and the stale node is passed to bpf_list_del(), the owner comparison can succeed even though the node is not really linked to the new head, and list_del_init() will follow bogus next/prev pointers with the risk of memory corruption.
When draining a bpf_list_head, mark each node owner with BPF_PTR_POISON under the map spinlock while moving it to a private drain list, then list_del_init() the node and clear owner to NULL before calling __bpf_obj_drop_impl(). Concurrent readers therefore never observe a node that appears linked to a head while its list_head is inconsistent, and surviving refcounted nodes never retain a stale non-NULL owner.
--- kernel/bpf/helpers.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)
kernel/bpf/helpers.c +19 -8
@@ -2247,10 +2247,11 @@ EXPORT_SYMBOL_GPL(bpf_base_func_proto);void bpf_list_head_free (const struct btf_field *field,void *list_head,struct bpf_spin_lock *spin_lock) { -struct list_head *head = list_head, *orig_head = list_head; +struct list_head *head = list_head, drain, *pos, *n;BUILD_BUG_ON (sizeof (struct list_head ) >sizeof (struct bpf_list_head ));BUILD_BUG_ON (__alignof__(struct list_head ) > __alignof__(struct bpf_list_head )); +INIT_LIST_HEAD (&drain);/* Do the actual list draining outside the lock to not hold the lock for * too long, and also prevent deadlocks if tracing programs end up
@@ -2261,20 +2262,30 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head,__bpf_spin_lock_irqsave (spin_lock);if (!head->next ||list_empty (head)) goto unlock; - head = head->next; +list_for_each_safe (pos, n, head) { +struct bpf_list_node_kern *node; + + node = container_of(pos, struct bpf_list_node_kern, list_head); +WRITE_ONCE (node->owner,BPF_PTR_POISON ); +list_move_tail (pos, &drain); + } unlock: -INIT_LIST_HEAD (orig_head); +INIT_LIST_HEAD (head);__bpf_spin_unlock_irqrestore (spin_lock); -while (head != orig_head) { -void *obj = head; +while (!list_empty (&drain)) { +struct bpf_list_node_kern *node; - obj -= field->graph_root.node_offset; - head = head->next; + pos = drain.next; + node = container_of(pos, struct bpf_list_node_kern, list_head); +list_del_init (pos); +/* Ensure __bpf_list_add() sees the node as unlinked. */ +smp_store_release (&node->owner,NULL );/* The contained type can also have resources, including a * bpf_list_head which needs to be freed. */ -__bpf_obj_drop_impl (obj, field->graph_root.value_rec, false); +__bpf_obj_drop_impl ((char *)pos - field->graph_root.node_offset, + field->graph_root.value_rec, false); } }
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v11 3/8 bpf: allow non-owning list-node args via __nonown_allowed
In reply to Kaitao ChengFrom: Kaitao Cheng <[email protected]>
KF_ARG_PTR_TO_LIST_NODE normally requires an owning reference (PTR_TO_BTF_ID | MEM_ALLOC with ref_obj_id). Introduce the __nonown_allowed annotation on selected list-node arguments so non-owning references with ref_obj_id==0 are accepted as well.
This patch only adds the generic verifier support and documents the annotation. Later patches in the series will apply it to bpf_list_add /del(), and bpf_list_is_first/last(), allowing bpf_list_front/back() results to be used as the insertion point, deletion target, or query target for those kfuncs.
Verifier keeps existing owning-ref checks by default; only arguments annotated with __nonown_allowed bypass MEM_ALLOC/ref_obj_id checks and then follow the same list-node validation path.
--- Documentation/bpf/kfuncs.rst | 22 ++++++++++++++++++++-- kernel/bpf/verifier.c | 13 +++++++++++++ 2 files changed, 33 insertions(+), 2 deletions(-)
Documentation/bpf/kfuncs.rst +20 -2
@@ -207,8 +207,26 @@ Here, the buffer may be NULL. If the buffer is not NULL, it must be at least buffer__szk bytes in size. The kfunc is responsible for checking if the buffer is NULL before using it. -2.3.5 __str Annotation ----------------------------- +2.3.5 __nonown_allowed Annotation +--------------------------------- + +This annotation is used to indicate that the parameter may be a non-owning reference. + +An example is given below:: + + __bpf_kfunc int bpf_list_add(..., struct bpf_list_node + *prev__nonown_allowed, ...) + { + ... + } + +For the ``prev__nonown_allowed`` parameter (resolved as ``KF_ARG_PTR_TO_LIST_NODE``), +suffix ``__nonown_allowed`` retains the usual owning-pointer rules and also +permits a non-owning reference with no ref_obj_id (e.g. the return value of +bpf_list_front() / bpf_list_back()). + +2.3.6 __str Annotation +---------------------- This annotation is used to indicate that the argument is a constant string. An example is given below::
kernel/bpf/verifier.c +13 -0
@@ -10714,6 +10714,11 @@ static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_paramreturn btf_param_match_suffix (btf, arg,"__nullable" ); } +static bool is_kfunc_arg_nonown_allowed (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__nonown_allowed" ); +} +static bool is_kfunc_arg_const_str (const struct btf *btf,const struct btf_param *arg) {return btf_param_match_suffix (btf, arg,"__str" );
@@ -12244,6 +12249,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_return ret;break ;case KF_ARG_PTR_TO_LIST_NODE : +if (is_kfunc_arg_nonown_allowed (btf, &args[i]) && +type_is_non_owning_ref (reg->type) && !reg->ref_obj_id) { +/* Allow bpf_list_front/back return value for +* __nonown_allowed list-node arguments. +*/ + goto check_ok; + }if (reg->type != (PTR_TO_BTF_ID |MEM_ALLOC )) {verbose (env,"%s expected pointer to allocated object\n" ,reg_arg_name (env, argno));
@@ -12253,6 +12265,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_verbose (env,"allocated object must be referenced\n" );return -EINVAL ; } +check_ok: ret =process_kf_arg_ptr_to_list_node (env, reg, argno, meta);if (ret <0 )return ret;
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v11 4/8 bpf: Introduce the bpf_list_del kfunc.
In reply to Kaitao ChengFrom: Kaitao Cheng <[email protected]>
Allow users to remove any node from a linked list.
We have added an additional parameter bpf_list_head *head to bpf_list_del, as the verifier requires the head parameter to check whether the lock is being held.
--- kernel/bpf/helpers.c | 10 ++++++++++ kernel/bpf/verifier.c | 6 +++++- 2 files changed, 15 insertions(+), 1 deletion(-)
kernel/bpf/helpers.c +10 -0
@@ -2601,6 +2601,15 @@ __bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)return __bpf_list_del (head, h->prev); } +__bpf_kfunc struct bpf_list_node *bpf_list_del(struct bpf_list_head *head, +struct bpf_list_node *node__nonown_allowed) +{ +struct bpf_list_node_kern *kn = (void *)node__nonown_allowed; + +/* verifier guarantees node is a list node rather than list head */ +return __bpf_list_del (head, &kn->list_head); +} + __bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head) {struct list_head *h = (struct list_head *)head;
@@ -4733,6 +4742,7 @@ BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_list_push_back_impl) BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_list_del, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL) BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)BTF_ID_FLAGS (func, bpf_task_acquire,KF_ACQUIRE |KF_RCU |KF_RET_NULL )
kernel/bpf/verifier.c +5 -1
@@ -10961,6 +10961,7 @@ enum special_kfunc_type { KF_bpf_list_push_back, KF_bpf_list_pop_front, KF_bpf_list_pop_back, + KF_bpf_list_del, KF_bpf_list_front, KF_bpf_list_back, KF_bpf_cast_to_kern_ctx,
@@ -11029,6 +11030,7 @@ BTF_ID(func, bpf_list_push_back_impl) BTF_ID(func, bpf_list_push_back) BTF_ID(func, bpf_list_pop_front) BTF_ID(func, bpf_list_pop_back) +BTF_ID(func, bpf_list_del) BTF_ID(func, bpf_list_front) BTF_ID(func, bpf_list_back)BTF_ID (func, bpf_cast_to_kern_ctx)
@@ -11549,6 +11551,7 @@ static bool is_bpf_list_api_kfunc(u32 btf_id)return is_bpf_list_push_kfunc (btf_id) || btf_id == special_kfunc_list[KF_bpf_list_pop_front] || btf_id == special_kfunc_list[KF_bpf_list_pop_back] || + btf_id == special_kfunc_list[KF_bpf_list_del] || btf_id == special_kfunc_list[KF_bpf_list_front] || btf_id == special_kfunc_list[KF_bpf_list_back]; }
@@ -11671,7 +11674,8 @@ static bool check_kfunc_is_graph_node_api(struct bpf_verifier_env *env,switch (node_field_type) {case BPF_LIST_NODE : - ret =is_bpf_list_push_kfunc (kfunc_btf_id); + ret =is_bpf_list_push_kfunc (kfunc_btf_id) || + kfunc_btf_id == special_kfunc_list[KF_bpf_list_del];break ;case BPF_RB_NODE : ret = (is_bpf_rbtree_add_kfunc(kfunc_btf_id) ||
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v11 5/8 bpf: refactor __bpf_list_add to take insertion point via **prev_ptr
In reply to Kaitao ChengFrom: Kaitao Cheng <[email protected]>
Refactor __bpf_list_add to accept (node, head, struct list_head **prev_ptr, ..) instead of (node, head, bool tail, ..). Load prev from *prev_ptr after INIT_LIST_HEAD(h), so we never dereference an uninitialized h->prev when head was 0-initialized (e.g. push_back passes &h->prev).
When prev is not the list head, validate that prev is in the list via its owner.
Prepares for bpf_list_add(head, new, prev, ..) to insert after a given list node.
--- kernel/bpf/helpers.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-)
kernel/bpf/helpers.c +26 -10
@@ -2478,9 +2478,11 @@ __bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *metastatic int __bpf_list_add (struct bpf_list_node_kern *node,struct bpf_list_head *head, -bool tail,struct btf_record *rec,u64 off) +struct list_head **prev_ptr, +struct btf_record *rec,u64 off) {struct list_head *n = &node->list_head, *h = (void *)head; +struct list_head *prev;/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't * called on its fields, so init here
@@ -2488,19 +2490,31 @@ static int __bpf_list_add(struct bpf_list_node_kern *node,if (unlikely (!h->next))INIT_LIST_HEAD (h); + prev = *prev_ptr; + +/* When prev is not the list head, it must be a node in this list. */ +if (prev != h) { +struct bpf_list_node_kern *prev_kn = + container_of(prev, struct bpf_list_node_kern, list_head); + +if (unlikely (READ_ONCE (prev_kn->owner) != head)) + goto fail; + } +/* node->owner != NULL implies !list_empty(n), no need to separately * check the latter */ -if (cmpxchg (&node->owner,NULL ,BPF_PTR_POISON )) { -/* Only called from BPF prog, no need to migrate_disable */ -__bpf_obj_drop_impl ((void *)n - off, rec, false); -return -EINVAL ; - } +if (cmpxchg (&node->owner,NULL ,BPF_PTR_POISON )) + goto fail; - tail ?list_add_tail (n, h) :list_add (n, h); +list_add (n, prev);WRITE_ONCE (node->owner, head); -return 0 ; + +fail: +/* Only called from BPF prog, no need to migrate_disable */ +__bpf_obj_drop_impl ((void *)n - off, rec, false); +return -EINVAL ; }/**
@@ -2521,8 +2535,9 @@ __bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head, u64 off) {struct bpf_list_node_kern *n = (void *)node; +struct list_head *h = (void *)head; -return __bpf_list_add (n, head, false, meta ? meta->record :NULL , off); +return __bpf_list_add (n, head, &h, meta ? meta->record :NULL , off); } __bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
@@ -2550,8 +2565,9 @@ __bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head, u64 off) {struct bpf_list_node_kern *n = (void *)node; +struct list_head *h = (void *)head; -return __bpf_list_add (n, head, true, meta ? meta->record :NULL , off); +return __bpf_list_add (n, head, &h->prev, meta ? meta->record :NULL , off); } __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v11 6/8 bpf: Add bpf_list_add to insert node after a given list node
In reply to Kaitao ChengFrom: Kaitao Cheng <[email protected]>
Add a new kfunc bpf_list_add(head, new, prev, meta, off) that inserts 'new' after 'prev' in the BPF linked list. Both must be in the same list; 'prev' must already be in the list. The new node must be an owning reference (e.g. from bpf_obj_new); the kfunc consumes that reference and the node becomes non-owning once inserted.
We have added an additional parameter bpf_list_head *head to bpf_list_add, as the verifier requires the head parameter to check whether the lock is being held.
Returns 0 on success, -EINVAL if 'prev' is not in a list or 'new' is already in a list (or duplicate insertion). On failure, the kernel drops the passed-in node.
--- kernel/bpf/helpers.c | 11 +++++++++++ kernel/bpf/verifier.c | 12 +++++++++--- 2 files changed, 20 insertions(+), 3 deletions(-)
kernel/bpf/helpers.c +11 -0
@@ -2577,6 +2577,16 @@ __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,return bpf_list_push_back (head, node, meta__ign, off); } +__bpf_kfunc int bpf_list_add(struct bpf_list_head *head, struct bpf_list_node *new, +struct bpf_list_node *prev__nonown_allowed, +struct btf_struct_meta *meta,u64 off) +{ +struct bpf_list_node_kern *n = (void *)new, *p = (void *)prev__nonown_allowed; +struct list_head *prev_ptr = &p->list_head; + +return __bpf_list_add (n, head, &prev_ptr, meta ? meta->record :NULL , off); +} +static struct bpf_list_node *__bpf_list_del (struct bpf_list_head *head,struct list_head *n) {
@@ -4756,6 +4766,7 @@ BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_list_push_front_impl) BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_list_push_back_impl) +BTF_ID_FLAGS(func, bpf_list_add, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)BTF_ID_FLAGS (func, bpf_list_del,KF_ACQUIRE |KF_RET_NULL )
kernel/bpf/verifier.c +9 -3
@@ -10959,6 +10959,7 @@ enum special_kfunc_type { KF_bpf_list_push_front, KF_bpf_list_push_back_impl, KF_bpf_list_push_back, + KF_bpf_list_add, KF_bpf_list_pop_front, KF_bpf_list_pop_back, KF_bpf_list_del,
@@ -11028,6 +11029,7 @@ BTF_ID(func, bpf_list_push_front_impl) BTF_ID(func, bpf_list_push_front) BTF_ID(func, bpf_list_push_back_impl) BTF_ID(func, bpf_list_push_back) +BTF_ID(func, bpf_list_add) BTF_ID(func, bpf_list_pop_front) BTF_ID(func, bpf_list_pop_back)BTF_ID (func, bpf_list_del)
@@ -11140,7 +11142,8 @@ static bool is_bpf_list_push_kfunc(u32 func_id)return func_id == special_kfunc_list[KF_bpf_list_push_front] || func_id == special_kfunc_list[KF_bpf_list_push_front_impl] || func_id == special_kfunc_list[KF_bpf_list_push_back] || - func_id == special_kfunc_list[KF_bpf_list_push_back_impl]; + func_id == special_kfunc_list[KF_bpf_list_push_back_impl] || + func_id == special_kfunc_list[KF_bpf_list_add]; }static bool is_bpf_rbtree_add_kfunc (u32 func_id)
@@ -19524,8 +19527,11 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,int struct_meta_reg =BPF_REG_3 ;int node_offset_reg =BPF_REG_4 ; -/* rbtree_add has extra 'less' arg, so args-to-fixup are in diff regs */ -if (is_bpf_rbtree_add_kfunc (desc->func_id)) { +/* list_add/rbtree_add have an extra arg (prev/less), +* so args-to-fixup are in diff regs. +*/ +if (desc->func_id == special_kfunc_list[KF_bpf_list_add] || +is_bpf_rbtree_add_kfunc (desc->func_id)) { struct_meta_reg =BPF_REG_4 ; node_offset_reg =BPF_REG_5 ; }
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v11 7/8 bpf: add bpf_list_is_first/last/empty kfuncs
In reply to Kaitao ChengFrom: Kaitao Cheng <[email protected]>
Add three kfuncs for BPF linked list queries:
- bpf_list_is_first(head, node): true if node is the first in the list.
- bpf_list_is_last(head, node): true if node is the last in the list.
- bpf_list_empty(head): true if the list has no entries.
Currently, without these kfuncs, to implement the above functionality it is necessary to first call bpf_list_pop_front/back to retrieve the first or last node before checking whether the passed-in node was the first or last one. After the check, the node had to be pushed back into the list using bpf_list_push_front/back, which was very inefficient.
Now, with the bpf_list_is_first/last/empty kfuncs, we can directly check whether a node is the first, last, or whether the list is empty, without having to first retrieve the node.
--- kernel/bpf/helpers.c | 40 ++++++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 15 +++++++++++++-- 2 files changed, 53 insertions(+), 2 deletions(-)
kernel/bpf/helpers.c +40 -0
@@ -2656,6 +2656,43 @@ __bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head)return (struct bpf_list_node *)h->prev; } +__bpf_kfunc bool bpf_list_is_first(struct bpf_list_head *head, +struct bpf_list_node *node__nonown_allowed) +{ +struct list_head *h = (struct list_head *)head; +struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed; + +if (READ_ONCE (kn->owner) != head) +return false; + +return list_is_first (&kn->list_head, h); +} + +__bpf_kfunc bool bpf_list_is_last(struct bpf_list_head *head, +struct bpf_list_node *node__nonown_allowed) +{ +struct list_head *h = (struct list_head *)head; +struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed; + +if (READ_ONCE (kn->owner) != head) +return false; + +return list_is_last (&kn->list_head, h); +} + +__bpf_kfunc bool bpf_list_empty(struct bpf_list_head *head) +{ +struct list_head *h = (struct list_head *)head; + +/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't +* called on its fields, so init here +*/ +if (unlikely (!h->next)) +INIT_LIST_HEAD (h); + +return list_empty (h); +} + __bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,struct bpf_rb_node *node) {
@@ -4772,6 +4809,9 @@ BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_list_del, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL) BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_list_is_first) +BTF_ID_FLAGS(func, bpf_list_is_last) +BTF_ID_FLAGS(func, bpf_list_empty) BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)BTF_ID_FLAGS (func, bpf_rbtree_remove,KF_ACQUIRE |KF_RET_NULL )
kernel/bpf/verifier.c +13 -2
@@ -10965,6 +10965,9 @@ enum special_kfunc_type { KF_bpf_list_del, KF_bpf_list_front, KF_bpf_list_back, + KF_bpf_list_is_first, + KF_bpf_list_is_last, + KF_bpf_list_empty, KF_bpf_cast_to_kern_ctx, KF_bpf_rdonly_cast, KF_bpf_rcu_read_lock,
@@ -11035,6 +11038,9 @@ BTF_ID(func, bpf_list_pop_back) BTF_ID(func, bpf_list_del) BTF_ID(func, bpf_list_front) BTF_ID(func, bpf_list_back) +BTF_ID(func, bpf_list_is_first) +BTF_ID(func, bpf_list_is_last) +BTF_ID(func, bpf_list_empty) BTF_ID(func, bpf_cast_to_kern_ctx) BTF_ID(func, bpf_rdonly_cast)BTF_ID (func, bpf_rcu_read_lock)
@@ -11556,7 +11562,10 @@ static bool is_bpf_list_api_kfunc(u32 btf_id) btf_id == special_kfunc_list[KF_bpf_list_pop_back] || btf_id == special_kfunc_list[KF_bpf_list_del] || btf_id == special_kfunc_list[KF_bpf_list_front] || - btf_id == special_kfunc_list[KF_bpf_list_back]; + btf_id == special_kfunc_list[KF_bpf_list_back] || + btf_id == special_kfunc_list[KF_bpf_list_is_first] || + btf_id == special_kfunc_list[KF_bpf_list_is_last] || + btf_id == special_kfunc_list[KF_bpf_list_empty]; }static bool is_bpf_rbtree_api_kfunc (u32 btf_id)
@@ -11678,7 +11687,9 @@ static bool check_kfunc_is_graph_node_api(struct bpf_verifier_env *env,switch (node_field_type) {case BPF_LIST_NODE : ret =is_bpf_list_push_kfunc (kfunc_btf_id) || - kfunc_btf_id == special_kfunc_list[KF_bpf_list_del]; + kfunc_btf_id == special_kfunc_list[KF_bpf_list_del] || + kfunc_btf_id == special_kfunc_list[KF_bpf_list_is_first] || + kfunc_btf_id == special_kfunc_list[KF_bpf_list_is_last];break ;case BPF_RB_NODE : ret = (is_bpf_rbtree_add_kfunc(kfunc_btf_id) ||
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v11 8/8 selftests/bpf: Add test cases for bpf_list_del/add/is_first/is_last/empty
In reply to Kaitao ChengFrom: Kaitao Cheng <[email protected]>
Extend refcounted_kptr with tests for bpf_list_add (including prev from bpf_list_front and bpf_refcount_acquire), bpf_list_del (including node from bpf_list_front, bpf_rbtree_remove and bpf_refcount_acquire), bpf_list_empty, bpf_list_is_first/last, and push_back on uninit head.
To verify the validity of bpf_list_del/add, the test also expects the verifier to reject calls to bpf_list_del/add made without holding the spin_lock.
--- .../selftests/bpf/progs/refcounted_kptr.c | 421 ++++++++++++++++++ 1 file changed, 421 insertions(+)
tools/testing/selftests/bpf/progs/refcounted_kptr.c +421 -0
@@ -367,6 +367,427 @@ long insert_rbtree_and_stash__del_tree_##rem_tree(void *ctx) \INSERT_STASH_READ (true,"insert_stash_read: remove from tree" );INSERT_STASH_READ (false,"insert_stash_read: don't remove from tree" ); +SEC("tc") +__description("list_empty_test: list empty before add, non-empty after add") +__success __retval(0) +int list_empty_test(void *ctx) +{ +struct node_data *node_new; + +bpf_spin_lock (&lock); +if (!bpf_list_empty (&head)) { +bpf_spin_unlock (&lock); +return -1 ; + } +bpf_spin_unlock (&lock); + + node_new =bpf_obj_new (typeof (*node_new)); +if (!node_new) +return -2 ; + +bpf_spin_lock (&lock); +bpf_list_push_front (&head, &node_new->l); + +if (bpf_list_empty (&head)) { +bpf_spin_unlock (&lock); +return -3 ; + } +bpf_spin_unlock (&lock); +return 0 ; +} + +static struct node_data *__add_in_list (struct bpf_list_head *head, +struct bpf_spin_lock *lock) +{ +struct node_data *node_new, *node_ref; + + node_new =bpf_obj_new (typeof (*node_new)); +if (!node_new) +return NULL ; + + node_ref =bpf_refcount_acquire (node_new); + +bpf_spin_lock (lock); +bpf_list_push_front (head, &node_new->l); +bpf_spin_unlock (lock); +return node_ref; +} + +SEC("tc") +__description("list_is_edge_test1: is_first on first node, is_last on last node") +__success __retval(0) +int list_is_edge_test1(void *ctx) +{ +struct node_data *node_first, *node_last; +int err =0 ; + + node_last =__add_in_list (&head, &lock); +if (!node_last) +return -1 ; + + node_first =__add_in_list (&head, &lock); +if (!node_first) { +bpf_obj_drop (node_last); +return -2 ; + } + +bpf_spin_lock (&lock); +if (!bpf_list_is_first (&head, &node_first->l)) { + err =-3 ; + goto fail; + } +if (!bpf_list_is_last (&head, &node_last->l)) + err =-4 ; + +fail: +bpf_spin_unlock (&lock); +bpf_obj_drop (node_first); +bpf_obj_drop (node_last); +return err; +} + +SEC("tc") +__description("list_is_edge_test2: accept list_front/list_back return value") +__success __retval(0) +int list_is_edge_test2(void *ctx) +{ +struct bpf_list_node *front, *back; +struct node_data *a, *b; +long err =0 ; + + a =__add_in_list (&head, &lock); +if (!a) +return -1 ; + + b =__add_in_list (&head, &lock); +if (!b) { +bpf_obj_drop (a); +return -2 ; + } + +bpf_spin_lock (&lock); + front =bpf_list_front (&head); + back =bpf_list_back (&head); +if (!front || !back) { + err =-3 ; + goto out_unlock; + } + +if (!bpf_list_is_first (&head, front) ||bpf_list_is_last (&head, front)) { + err =-4 ; + goto out_unlock; + } + +if (!bpf_list_is_last (&head, back) ||bpf_list_is_first (&head, back)) { + err =-5 ; + goto out_unlock; + } + +out_unlock: +bpf_spin_unlock (&lock); +bpf_obj_drop (a); +bpf_obj_drop (b); +return err; +} + +SEC("tc") +__description("list_is_edge_test3: single node is both first and last") +__success __retval(0) +int list_is_edge_test3(void *ctx) +{ +struct node_data *tmp; +struct bpf_list_node *node; +long err =0 ; + + tmp =__add_in_list (&head, &lock); +if (!tmp) +return -1 ; + +bpf_spin_lock (&lock); + node =bpf_list_front (&head); +if (!node) { +bpf_spin_unlock (&lock); +bpf_obj_drop (tmp); +return -2 ; + } + +if (!bpf_list_is_first (&head, node) || !bpf_list_is_last (&head, node)) + err =-3 ; +bpf_spin_unlock (&lock); + +bpf_obj_drop (tmp); +return err; +} + +SEC("tc") +__description("list_del_test1: del returns removed nodes") +__success __retval(0) +int list_del_test1(void *ctx) +{ +struct node_data *node_first, *node_last; +struct bpf_list_node *bpf_node_first, *bpf_node_last; +int err =0 ; + + node_last =__add_in_list (&head, &lock); +if (!node_last) +return -1 ; + + node_first =__add_in_list (&head, &lock); +if (!node_first) { +bpf_obj_drop (node_last); +return -2 ; + } + +bpf_spin_lock (&lock); + bpf_node_last =bpf_list_del (&head, &node_last->l); + bpf_node_first =bpf_list_del (&head, &node_first->l); +bpf_spin_unlock (&lock); + +if (bpf_node_first) + bpf_obj_drop(container_of(bpf_node_first, struct node_data, l)); +else + err =-3 ; + +if (bpf_node_last) + bpf_obj_drop(container_of(bpf_node_last, struct node_data, l)); +else + err =-4 ; + +bpf_obj_drop (node_first); +bpf_obj_drop (node_last); +return err; +} + +SEC("tc") +__description("list_del_test2: remove an arbitrary node from the list") +__success __retval(0) +int list_del_test2(void *ctx) +{ +struct bpf_rb_node *rb; +struct bpf_list_node *l; +struct node_data *n; +long err; + + err =__insert_in_tree_and_list (&head, &root, &lock); +if (err) +return err; + +bpf_spin_lock (&lock); + rb =bpf_rbtree_first (&root); +if (!rb) { +bpf_spin_unlock (&lock); +return -4 ; + } + + rb =bpf_rbtree_remove (&root, rb); +if (!rb) { +bpf_spin_unlock (&lock); +return -5 ; + } + + n = container_of(rb, struct node_data, r); + l =bpf_list_del (&head, &n->l); +bpf_spin_unlock (&lock); +bpf_obj_drop (n); +if (!l) +return -6 ; + +bpf_obj_drop (container_of(l ,struct node_data ,l )); +return 0 ; +} + +SEC("tc") +__description("list_del_test3: list_del accepts list_front return value as node") +__success __retval(0) +int list_del_test3(void *ctx) +{ +struct node_data *tmp; +struct bpf_list_node *bpf_node, *l; +long err =0 ; + + tmp =__add_in_list (&head, &lock); +if (!tmp) +return -1 ; + +bpf_spin_lock (&lock); + bpf_node =bpf_list_front (&head); +if (!bpf_node) { +bpf_spin_unlock (&lock); + err =-2 ; + goto fail; + } + + l =bpf_list_del (&head, bpf_node); +bpf_spin_unlock (&lock); +if (!l) { + err =-3 ; + goto fail; + } + +bpf_obj_drop (container_of(l ,struct node_data ,l )); +bpf_obj_drop (tmp); +return 0 ; + +fail: +bpf_obj_drop (tmp); +return err; +} + +SEC("tc") +__description("list_add_test1: insert new node after prev") +__success __retval(0) +int list_add_test1(void *ctx) +{ +struct node_data *node_first; +struct node_data *new_node; +long err =0 ; + + node_first =__add_in_list (&head, &lock); +if (!node_first) +return -1 ; + + new_node =bpf_obj_new (typeof (*new_node)); +if (!new_node) { + err =-2 ; + goto fail; + } + +bpf_spin_lock (&lock); + err =bpf_list_add (&head, &new_node->l, &node_first->l); +bpf_spin_unlock (&lock); +if (err) { + err =-3 ; + goto fail; + } + +fail: +bpf_obj_drop (node_first); +return err; +} + +SEC("tc") +__description("list_add_test2: list_add accepts list_front return value as prev") +__success __retval(0) +int list_add_test2(void *ctx) +{ +struct node_data *new_node, *tmp; +struct bpf_list_node *bpf_node; +long err =0 ; + + tmp =__add_in_list (&head, &lock); +if (!tmp) +return -1 ; + + new_node =bpf_obj_new (typeof (*new_node)); +if (!new_node) { + err =-2 ; + goto fail; + } + +bpf_spin_lock (&lock); + bpf_node =bpf_list_front (&head); +if (!bpf_node) { +bpf_spin_unlock (&lock); +bpf_obj_drop (new_node); + err =-3 ; + goto fail; + } + + err =bpf_list_add (&head, &new_node->l, bpf_node); +bpf_spin_unlock (&lock); +if (err) { + err =-4 ; + goto fail; + } + +fail: +bpf_obj_drop (tmp); +return err; +} + +struct uninit_head_val { +struct bpf_spin_lock lock; + struct bpf_list_head head __contains(node_data, l); +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, int); + __type(value, struct uninit_head_val); + __uint(max_entries, 1); +} uninit_head_map SEC(".maps"); + +SEC("tc") +__description("list_push_back_uninit_head: push_back on 0-initialized list head") +__success __retval(0) +int list_push_back_uninit_head(void *ctx) +{ +struct uninit_head_val *st; +struct node_data *node; +int ret =-1 , key =0 ; + + st =bpf_map_lookup_elem (&uninit_head_map, &key); +if (!st) +return -1 ; + + node =bpf_obj_new (typeof (*node)); +if (!node) +return -1 ; + +bpf_spin_lock (&st->lock); + ret =bpf_list_push_back (&st->head, &node->l); +bpf_spin_unlock (&st->lock); + +return ret; +} + +SEC("?tc") +__failure __msg("bpf_spin_lock at off=32 must be held for bpf_list_head") +long list_del_without_lock_fail(void *ctx) +{ +struct node_data *n; +struct bpf_list_node *l; + + n =bpf_obj_new (typeof (*n)); +if (!n) +return -1 ; + +/* Error case: delete list node without holding lock */ + l =bpf_list_del (&head, &n->l); +bpf_obj_drop (n); +if (!l) +return -2 ; +bpf_obj_drop (container_of(l ,struct node_data ,l )); + +return 0 ; +} + +SEC("?tc") +__failure __msg("bpf_spin_lock at off=32 must be held for bpf_list_head") +long list_add_without_lock_fail(void *ctx) +{ +struct node_data *n, *prev; +long err; + + n =bpf_obj_new (typeof (*n)); +if (!n) +return -1 ; + + prev =bpf_obj_new (typeof (*prev)); +if (!prev) { +bpf_obj_drop (n); +return -1 ; + } + +/* Error case: add list node without holding lock */ + err =bpf_list_add (&head, &n->l, &prev->l); +bpf_obj_drop (prev); +if (err) +return -2 ; + +return 0 ; +} + SEC("tc") __successlong rbtree_refcounted_node_ref_escapes (void *ctx)
Signature · 1 line
2.50.1 (Apple Git-155)
[email protected]
AI reviewRe: [PATCH bpf-next v11 8/8] selftests/bpf: Add test cases for bpf_list_del/add/is_first/is_last/empty
commit 75130d2969168e50252cc473cfaa880a9795c0ef Author: Kaitao Cheng <[email protected]>
commit 75130d2969168e50252cc473cfaa880a9795c0ef
Author: Kaitao Cheng <[email protected]>
selftests/bpf: Add test cases for bpf_list_del/add/is_first/is_last/empty
Extend refcounted_kptr with tests for bpf_list_add (including prev from bpf_list_front and bpf_refcount_acquire), bpf_list_del (including node from bpf_list_front, bpf_rbtree_remove and bpf_refcount_acquire), bpf_list_empty, bpf_list_is_first/last, and push_back on uninit head.
To verify the validity of bpf_list_del/add, the test also expects the verifier to reject calls to bpf_list_del/add made without holding the spin_lock.
The commit message mentions adding test cases for bpf_list_is_first, bpf_list_is_last, and bpf_list_empty, but these test functions already exist in the file and are not added by this patch. The existing tests are list_empty_test, list_is_edge_test1, list_is_edge_test2, and list_is_edge_test3.
This patch actually adds:
- list_del_test3 (new test for bpf_list_del with list_front return value)
- list_add_test1 (new test for bpf_list_add after prev node)
- list_add_test2 (new test for bpf_list_add with list_front as prev)
- list_push_back_uninit_head (new test for push_back on uninitialized head)
- list_del_without_lock_fail (negative test for del without lock)
- list_add_without_lock_fail (negative test for add without lock)
Should the commit message be adjusted to accurately reflect which tests are being added?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/26204125015
[email protected]
AI reviewRe: [PATCH bpf-next v11 3/8] bpf: allow non-owning list-node args via __nonown_allowed
In reply to Kaitao ChengThis isn't a bug, but the multi-line comment doesn't match the BPF subsystem style. The BPF guide requires multi-line comments to have the opening /* on its own line for files under kernel/bpf/.…
Quoted 4 lines
Documentation/bpf/kfuncs.rst +0
-0
[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
Quoted 15 lines
kernel/bpf/verifier.c +7
-0@@ -12244,6 +12249,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_return ret;break ;case KF_ARG_PTR_TO_LIST_NODE : +if (is_kfunc_arg_nonown_allowed (btf, &args[i]) && +type_is_non_owning_ref (reg->type) && !reg->ref_obj_id) { +/* Allow bpf_list_front/back return value for +* __nonown_allowed list-node arguments. +*/ + goto check_ok; + }if (reg->type != (PTR_TO_BTF_ID |MEM_ALLOC )) {verbose (env,"%s expected pointer to allocated object\n" ,reg_arg_name (env, argno));@@ -12253,6 +12265,7 @@ static int check_kfunc_args(struct
bpf_verifier_env*env, structbpf_kfunc_call_
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 12271
Quoted 7 lines
kernel/bpf/verifier.c +1
-0verbose (env,"allocated object must be referenced\n" );return -EINVAL ; } +check_ok: ret =process_kf_arg_ptr_to_list_node (env, reg, argno, meta);if (ret <0 )return ret;
This isn't a bug, but the multi-line comment doesn't match the BPF subsystem style. The BPF guide requires multi-line comments to have the opening /* on its own line for files under kernel/bpf/. Should this be:
/*
* Allow bpf_list_front/back return value for
* __nonown_allowed list-node arguments.
*/
Eduard Zingerman raised a related nit in v10 RESEND: the check '!reg->ref_obj_id' appears redundant. Would type_is_non_owning_ref() be sufficient here?
Reference: https://lore.kernel.org/bpf/[email protected]/
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/26204125015
[email protected]
AI reviewRe: [PATCH bpf-next v11 2/8] bpf: clear list node owner and unlink before drop
In reply to Kaitao ChengDoes this introduce an unbounded delay with interrupts disabled?
Quoted 19 lines
kernel/bpf/helpers.c +2
-1@@ -2247,10 +2247,11 @@ EXPORT_SYMBOL_GPL(bpf_base_func_proto);void bpf_list_head_free (const struct btf_field *field,void *list_head,struct bpf_spin_lock *spin_lock) { -struct list_head *head = list_head, *orig_head = list_head; +struct list_head *head = list_head, drain, *pos, *n;BUILD_BUG_ON (sizeof (struct list_head ) >sizeof (struct bpf_list_head ));BUILD_BUG_ON (__alignof__(struct list_head ) > __alignof__(struct bpf_list_head )); +INIT_LIST_HEAD (&drain);/* Do the actual list draining outside the lock to not hold the lock for * too long, and also prevent deadlocks if tracing programs end up @@ -2261,20 +2262,30 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head,__bpf_spin_lock_irqsave (spin_lock);
kernel/bpf/helpers.c void bpf_list_head_free(const struct btf_field *field, void *list_head, · line 2271
Quoted 10 lines
kernel/bpf/helpers.c +7
-1if (!head->next ||list_empty (head)) goto unlock; - head = head->next; +list_for_each_safe (pos, n, head) { +struct bpf_list_node_kern *node; + + node = container_of(pos, struct bpf_list_node_kern, list_head); +WRITE_ONCE (node->owner,BPF_PTR_POISON ); +list_move_tail (pos, &drain); + }
Does this introduce an unbounded delay with interrupts disabled?
The original code detached the head in O(1) time. Now there is an O(N) loop inside __bpf_spin_lock_irqsave(). Since BPF lists can grow quite large, could this cause soft lockups?
This also appears to contradict the comment just above the lock which explicitly mentions doing the actual list draining outside the lock to not hold it for too long.
Lore reference:
https://lore.kernel.org/bpf/[email protected]/
Reviewer: [email protected]
Date: Wed, 13 May 2026 06:02:21 +0000
Quoted 15 lines
kernel/bpf/helpers.c +6
-5unlock: -INIT_LIST_HEAD (orig_head); +INIT_LIST_HEAD (head);__bpf_spin_unlock_irqrestore (spin_lock); -while (head != orig_head) { -void *obj = head; +while (!list_empty (&drain)) { +struct bpf_list_node_kern *node; - obj -= field->graph_root.node_offset; - head = head->next; + pos = drain.next; + node = container_of(pos, struct bpf_list_node_kern, list_head); +list_del_init (pos);
kernel/bpf/helpers.c void bpf_list_head_free(const struct btf_field *field, void *list_head, · line 2290
Quoted 10 lines
kernel/bpf/helpers.c +4
-1+/* Ensure __bpf_list_add() sees the node as unlinked. */ +smp_store_release (&node->owner,NULL );/* The contained type can also have resources, including a * bpf_list_head which needs to be freed. */ -__bpf_obj_drop_impl (obj, field->graph_root.value_rec, false); +__bpf_obj_drop_impl ((char *)pos - field->graph_root.node_offset, + field->graph_root.value_rec, false); } }
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/26204125015
[email protected]
AI reviewRe: [PATCH bpf-next v11 4/8] bpf: Introduce the bpf_list_del kfunc.
In reply to Kaitao ChengThis isn't a bug, but Eduard Zingerman suggested in v10 adding a is_bpf_list_del_kfunc() helper for consistency with the existing code style (similar to is_bpf_list_push_kfunc()). This would…
Quoted 4 lines
kernel/bpf/helpers.c +0
-0
[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
kernel/bpf/verifier.c static bool check_kfunc_is_graph_node_api(struct bpf_verifier_env *env, · line 11678
Quoted 7 lines
kernel/bpf/verifier.c +2
-1@@ -11671,7 +11674,8 @@ static bool check_kfunc_is_graph_node_api(struct bpf_verifier_env *env,switch (node_field_type) {case BPF_LIST_NODE : - ret =is_bpf_list_push_kfunc (kfunc_btf_id); + ret =is_bpf_list_push_kfunc (kfunc_btf_id) || + kfunc_btf_id == special_kfunc_list[KF_bpf_list_del];
This isn't a bug, but Eduard Zingerman suggested in v10 adding a is_bpf_list_del_kfunc() helper for consistency with the existing code style (similar to is_bpf_list_push_kfunc()). This would maintain the pattern used throughout the codebase for checking kfunc types.
Reference: https://lore.kernel.org/bpf/[email protected]/
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/26204125015
[email protected]
AI reviewRe: [PATCH bpf-next v11 6/8] bpf: Add bpf_list_add to insert node after a given list node
In reply to Kaitao ChengIf __bpf_list_add() fails at runtime (e.g., because prev is not in the list indicated by head), it drops the 'new' node via __bpf_obj_drop_impl in the fail path.
Quoted 7 lines
kernel/bpf/helpers.c +0
-0@@ -2577,6 +2577,16 @@ __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,return bpf_list_push_back (head, node, meta__ign, off); }
kernel/bpf/helpers.c __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head, · line 2588
Quoted 10 lines
kernel/bpf/helpers.c +9
-0+__bpf_kfunc int bpf_list_add(struct bpf_list_head *head, struct bpf_list_node *new, +struct bpf_list_node *prev__nonown_allowed, +struct btf_struct_meta *meta,u64 off) +{ +struct bpf_list_node_kern *n = (void *)new, *p = (void *)prev__nonown_allowed; +struct list_head *prev_ptr = &p->list_head; + +return __bpf_list_add (n, head, &prev_ptr, meta ? meta->record :NULL , off); +}
If __bpf_list_add() fails at runtime (e.g., because prev is not in the list indicated by head), it drops the 'new' node via __bpf_obj_drop_impl in the fail path.
However, the verifier unconditionally converts 'new' into a non-owning reference during check_kfunc_call(). If __bpf_list_add() fails, the BPF program retains this non-owning reference to a freed node.
Could this allow the program to trigger a use-after-free within the lock critical section?
([email protected], v10 RESEND)
https://lore.kernel.org/bpf/[email protected]/
Quoted 3 lines
kernel/bpf/helpers.c +0
-0static struct bpf_list_node *__bpf_list_del (struct bpf_list_head *head,struct list_head *n)
[ ... ]
kernel/bpf/helpers.c BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) · line 4769
Quoted 5 lines
kernel/bpf/helpers.c +1
-0@@ -4756,6 +4766,7 @@ BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_list_push_front_impl) BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_list_push_back_impl) +BTF_ID_FLAGS (func, bpf_list_add,KF_IMPLICIT_ARGS )
Since bpf_list_add is marked with KF_IMPLICIT_ARGS, won't the verifier look for a bpf_list_add_impl counterpart during program load?
Without defining and registering a bpf_list_add_impl function, find_kfunc_impl_proto() will fail to find the required _impl suffix in the vmlinux BTF.
Will this cause the verifier to unconditionally reject the kfunc?
([email protected], v10 RESEND)
https://lore.kernel.org/bpf/[email protected]/
Quoted 3 lines
kernel/bpf/helpers.c +0
-0BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)BTF_ID_FLAGS (func, bpf_list_del,KF_ACQUIRE |KF_RET_NULL )
[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 19532
Quoted 9 lines
kernel/bpf/verifier.c +3
-2@@ -19524,8 +19527,11 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,int struct_meta_reg =BPF_REG_3 ;int node_offset_reg =BPF_REG_4 ; -/* rbtree_add has extra 'less' arg, so args-to-fixup are in diff regs */ -if (is_bpf_rbtree_add_kfunc (desc->func_id)) { +/* list_add/rbtree_add have an extra arg (prev/less), +* so args-to-fixup are in diff regs. +*/
This isn't a bug, but the multi-line comment format doesn't follow the BPF subsystem style. The preferred form in kernel/bpf/ has the opening /* on its own line.
Additionally, consider adding an is_bpf_list_add_kfunc() helper instead of direct comparison with special_kfunc_list[KF_bpf_list_add].
([email protected] and Eduard Zingerman, v10 RESEND)
https://lore.kernel.org/bpf/f573f7e0a5140fe6254d2ddfd3f69add455fa25c67c020c0312e2c2ceeca7c69@mail.kernel.org/
kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 19537
Quoted 5 lines
kernel/bpf/verifier.c +2
-0+if (desc->func_id == special_kfunc_list[KF_bpf_list_add] || +is_bpf_rbtree_add_kfunc (desc->func_id)) { struct_meta_reg =BPF_REG_4 ; node_offset_reg =BPF_REG_5 ; }
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/26204125015
[email protected]
AI reviewRe: [PATCH bpf-next v11 2/8] bpf: clear list node owner and unlink before drop
In reply to Kaitao ChengThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] Potentially unbounded O(N) list traversal with interrupts disabled in bpf_list_head_free().
Pre-existing issues:
- [High] The patch fixes the stale node->owner issue for bpf_list_head_free() but fails to apply the equivalent fix for bpf_rb_root_free(), leaving bpf_rbtree_remove() vulnerable to the same memory corruption. --
commit 59439e3ee3f8d57af8635b521393d245d26bbdf4
Author: Kaitao Cheng <[email protected]>
bpf: clear list node owner and unlink before drop
This commit fixes an issue where refcounted BPF list nodes can outlive their list head and retain a stale owner pointer. To prevent memory corruption if a new list is allocated at the same address, it moves nodes to a private drain list under the spinlock, poisons their owner pointers, and unlinks them before dropping the objects.
Quoted 4 lines
kernel/bpf/helpers.c +0
-0
[ ... ]
kernel/bpf/helpers.c void bpf_list_head_free(const struct btf_field *field, void *list_head, · line 2271
Quoted 12 lines
kernel/bpf/helpers.c +7
-1@@ -2261,20 +2262,30 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head,__bpf_spin_lock_irqsave (spin_lock);if (!head->next ||list_empty (head)) goto unlock; - head = head->next; +list_for_each_safe (pos, n, head) { +struct bpf_list_node_kern *node; + + node = container_of(pos, struct bpf_list_node_kern, list_head); +WRITE_ONCE (node->owner,BPF_PTR_POISON ); +list_move_tail (pos, &drain); + }
[Severity: High]
Could this new loop potentially cause a long CPU stall?
Since __bpf_spin_lock_irqsave() disables local interrupts, this O(N) list traversal executes with interrupts disabled.
BPF lists do not have a hard size limit, so if a program accumulates a very large number of elements in a single list, could iterating over them here trigger the NMI watchdog or cause severe latency spikes?
Quoted 15 lines
kernel/bpf/helpers.c +6
-5unlock: -INIT_LIST_HEAD (orig_head); +INIT_LIST_HEAD (head);__bpf_spin_unlock_irqrestore (spin_lock); -while (head != orig_head) { -void *obj = head; +while (!list_empty (&drain)) { +struct bpf_list_node_kern *node; - obj -= field->graph_root.node_offset; - head = head->next; + pos = drain.next; + node = container_of(pos, struct bpf_list_node_kern, list_head); +list_del_init (pos);
kernel/bpf/helpers.c void bpf_list_head_free(const struct btf_field *field, void *list_head, · line 2290
Quoted 10 lines
kernel/bpf/helpers.c +4
-1+/* Ensure __bpf_list_add() sees the node as unlinked. */ +smp_store_release (&node->owner,NULL );/* The contained type can also have resources, including a * bpf_list_head which needs to be freed. */ -__bpf_obj_drop_impl (obj, field->graph_root.value_rec, false); +__bpf_obj_drop_impl ((char *)pos - field->graph_root.node_offset, + field->graph_root.value_rec, false); } }
[Severity: High]
This isn't a bug introduced by this patch, but does bpf_rb_root_free() need the exact same fix?
When draining an rbtree, bpf_rb_root_free() calls __bpf_obj_drop_impl() on each node without clearing the node_internal->owner or unlinking the nodes.
If a surviving refcounted rb-node retains a pointer to the freed rb_root, and a new bpf_rb_root is allocated at the old address, wouldn't a call to bpf_rbtree_remove(new_root, stale_node) erroneously bypass the ownership check and invoke rb_erase_cached() with stale tree pointers?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH bpf-next v11 2/8] bpf: clear list node owner and unlink before drop
Quoted 43 lines · 在 2026/5/21 12:30, [email protected] 写道:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] Potentially unbounded O(N) list traversal with interrupts disabled inbpf_list_head_free().Pre-existing issues:
- [High] The patch fixes the stalenode->ownerissue forbpf_list_head_free()but fails to apply the equivalent fix forbpf_rb_root_free(), leavingbpf_rbtree_remove()vulnerable to the same memory corruption. --commit
59439e3ee3f8d57af8635b521393d245d26bbdf4
Author: Kaitao Cheng <[email protected]>bpf: clear list node owner and unlink before drop
This commit fixes an issue where refcounted BPF list nodes can outlive their list head and retain a stale owner pointer. To prevent memory corruption if a new list is allocated at the same address, it moves nodes to a private drain list under the spinlock, poisons their owner pointers, and unlinks them before dropping the objects.
Quoted 4 lines
kernel/bpf/helpers.c +0
-0[ ... ]
kernel/bpf/helpers.c void bpf_list_head_free(const struct btf_field *field, void *list_head, · line 2271
Quoted 12 lines
kernel/bpf/helpers.c +7
-1@@ -2261,20 +2262,30 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head,__bpf_spin_lock_irqsave (spin_lock);if (!head->next ||list_empty (head)) goto unlock; - head = head->next; +list_for_each_safe (pos, n, head) { +struct bpf_list_node_kern *node; + + node = container_of(pos, struct bpf_list_node_kern, list_head); +WRITE_ONCE (node->owner,BPF_PTR_POISON ); +list_move_tail (pos, &drain); + }[Severity: High]
Could this new loop potentially cause a long CPU stall?
Quoted 6 lines
Since
__bpf_spin_lock_irqsave()disables local interrupts, this O(N) list traversal executes with interrupts disabled.BPF lists do not have a hard size limit, so if a program accumulates a very large number of elements in a single list, could iterating over them here trigger the NMI watchdog or cause severe latency spikes?
This is merely address-taking and assignment, so it is almost negligible in cost, and the likelihood of causing a soft lockup is very low.
Quoted 30 lines
Quoted 15 lines
kernel/bpf/helpers.c +6
-5unlock: -INIT_LIST_HEAD (orig_head); +INIT_LIST_HEAD (head);__bpf_spin_unlock_irqrestore (spin_lock); -while (head != orig_head) { -void *obj = head; +while (!list_empty (&drain)) { +struct bpf_list_node_kern *node; - obj -= field->graph_root.node_offset; - head = head->next; + pos = drain.next; + node = container_of(pos, struct bpf_list_node_kern, list_head); +list_del_init (pos);kernel/bpf/helpers.c void bpf_list_head_free(const struct btf_field *field, void *list_head, · line 2290
Quoted 10 lines
kernel/bpf/helpers.c +4
-1+/* Ensure __bpf_list_add() sees the node as unlinked. */ +smp_store_release (&node->owner,NULL );/* The contained type can also have resources, including a * bpf_list_head which needs to be freed. */ -__bpf_obj_drop_impl (obj, field->graph_root.value_rec, false); +__bpf_obj_drop_impl ((char *)pos - field->graph_root.node_offset, + field->graph_root.value_rec, false); } }[Severity: High]
This isn't a bug introduced by this patch, but doesbpf_rb_root_free()need the exact same fix?
Quoted 7 lines
When draining an rbtree,
bpf_rb_root_free()calls__bpf_obj_drop_impl()on each node without clearing thenode_internal->owneror unlinking the nodes.If a surviving refcounted rb-node retains a pointer to the freed
rb_root, and a newbpf_rb_rootis allocated at the old address, wouldn't a call tobpf_rbtree_remove(new_root, stale_node)erroneously bypass the ownership check and invokerb_erase_cached()with stale tree pointers?
I will probably send a separate fix patch later; this issue does not need to be addressed in this patch series.
Signature · 2 lines
Thanks Kaitao Cheng
Re: [PATCH bpf-next v11 3/8] bpf: allow non-owning list-node args via __nonown_allowed
In reply to [email protected]Quoted 41 lines · 在 2026/5/21 12:08, [email protected] 写道:
Quoted 4 lines
Documentation/bpf/kfuncs.rst +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0[ ... ]
Quoted 15 lines
kernel/bpf/verifier.c +7
-0@@ -12244,6 +12249,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_return ret;break ;case KF_ARG_PTR_TO_LIST_NODE : +if (is_kfunc_arg_nonown_allowed (btf, &args[i]) && +type_is_non_owning_ref (reg->type) && !reg->ref_obj_id) { +/* Allow bpf_list_front/back return value for +* __nonown_allowed list-node arguments. +*/ + goto check_ok; + }if (reg->type != (PTR_TO_BTF_ID |MEM_ALLOC )) {verbose (env,"%s expected pointer to allocated object\n" ,reg_arg_name (env, argno));@@ -12253,6 +12265,7 @@ static int check_kfunc_args(struct
bpf_verifier_env*env, structbpf_kfunc_call_kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 12271
Quoted 7 lines
kernel/bpf/verifier.c +1
-0verbose (env,"allocated object must be referenced\n" );return -EINVAL ; } +check_ok: ret =process_kf_arg_ptr_to_list_node (env, reg, argno, meta);if (ret <0 )return ret;This isn't a bug, but the multi-line comment doesn't match the BPF subsystem style. The BPF guide requires multi-line comments to have the opening /* on its own line for files under
kernel/bpf/. Should this be:
Quoted 4 lines
/*
* Allow bpf_list_front/back return value for
*__nonown_allowedlist-node arguments.
*/
This file uses this comment style extensively, so I followed the existing convention here.
Quoted 6 lines
Eduard Zingerman raised a related nit in v10 RESEND: the check '!reg->ref_obj_id' appears redundant. Would
type_is_non_owning_ref()be sufficient here?
Sorry, I missed that. If needed, I can fix it in the next version.
Quoted 4 lines
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/26204125015
Signature · 2 lines
Thanks Kaitao Cheng
Re: [PATCH bpf-next v11 4/8] bpf: Introduce the bpf_list_del kfunc.
In reply to [email protected]Quoted 27 lines · 在 2026/5/21 12:08, [email protected] 写道:
Quoted 4 lines
kernel/bpf/helpers.c +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0[ ... ]
kernel/bpf/verifier.c static bool check_kfunc_is_graph_node_api(struct bpf_verifier_env *env, · line 11678
Quoted 7 lines
kernel/bpf/verifier.c +2
-1@@ -11671,7 +11674,8 @@ static bool check_kfunc_is_graph_node_api(struct bpf_verifier_env *env,switch (node_field_type) {case BPF_LIST_NODE : - ret =is_bpf_list_push_kfunc (kfunc_btf_id); + ret =is_bpf_list_push_kfunc (kfunc_btf_id) || + kfunc_btf_id == special_kfunc_list[KF_bpf_list_del];This isn't a bug, but Eduard Zingerman suggested in v10 adding a
is_bpf_list_del_kfunc()helper for consistency with the existing code style (similar tois_bpf_list_push_kfunc()). This would maintain the pattern used throughout the codebase for checking kfunc types.
Quoted 1 line
The reason for using is_bpf_list_push_kfunc() is that it groups multiple parallel "kfunc_btf_id == special_kfunc_list[*]" checks into one helper, making the code easier to reuse. If we added is_bpf_list_del_kfunc() here, it would only wrap a single KF_bpf_list_del check, which is unnecessary. Also, more helpers such as is_bpf_list_first/last may be added later, and overusing is_bpf_list_* helpers would make the code more redundant.
Quoted 5 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/26204125015
Signature · 2 lines
Thanks Kaitao Cheng
Re: [PATCH bpf-next v11 6/8] bpf: Add bpf_list_add to insert node after a given list node
In reply to [email protected]Quoted 26 lines · 在 2026/5/21 12:08, [email protected] 写道:
Quoted 7 lines
kernel/bpf/helpers.c +0
-0@@ -2577,6 +2577,16 @@ __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,return bpf_list_push_back (head, node, meta__ign, off); }kernel/bpf/helpers.c __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head, · line 2588
Quoted 10 lines
kernel/bpf/helpers.c +9
-0+__bpf_kfunc int bpf_list_add(struct bpf_list_head *head, struct bpf_list_node *new, +struct bpf_list_node *prev__nonown_allowed, +struct btf_struct_meta *meta,u64 off) +{ +struct bpf_list_node_kern *n = (void *)new, *p = (void *)prev__nonown_allowed; +struct list_head *prev_ptr = &p->list_head; + +return __bpf_list_add (n, head, &prev_ptr, meta ? meta->record :NULL , off); +}If
__bpf_list_add()fails at runtime (e.g., because prev is not in the list indicated by head), it drops the 'new' node via__bpf_obj_drop_implin the fail path.However, the verifier unconditionally converts 'new' into a non-owning reference during
check_kfunc_call(). If__bpf_list_add()fails, the BPF program retains this non-owning reference to a freed node.
Quoted 5 lines
Could this allow the program to trigger a use-after-free within the lock critical section?
([email protected], v10 RESEND)
https://lore.kernel.org/bpf/[email protected]/
This should be an acceptable case within the current design framework.
Quoted 14 lines
Quoted 2 lines
kernel/bpf/helpers.c +0
-0static struct bpf_list_node *__bpf_list_del (struct bpf_list_head *head,struct list_head *n)[ ... ]
kernel/bpf/helpers.c BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) · line 4769
Quoted 5 lines
kernel/bpf/helpers.c +1
-0@@ -4756,6 +4766,7 @@ BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_list_push_front_impl) BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_list_push_back_impl) +BTF_ID_FLAGS (func, bpf_list_add,KF_IMPLICIT_ARGS )Since
bpf_list_addis marked withKF_IMPLICIT_ARGS, won't the verifier look for abpf_list_add_implcounterpart during program load?
Quoted 8 lines
Without defining and registering a
bpf_list_add_implfunction,find_kfunc_impl_proto()will fail to find the required_implsuffix in the vmlinux BTF.Will this cause the verifier to unconditionally reject the kfunc?
([email protected], v10 RESEND)
https://lore.kernel.org/bpf/[email protected]/
The target prototype with the _impl suffix has already been deprecated.
Quoted 27 lines
Quoted 3 lines
kernel/bpf/helpers.c +0
-0BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)BTF_ID_FLAGS (func, bpf_list_del,KF_ACQUIRE |KF_RET_NULL )[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0[ ... ]
kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 19532
Quoted 9 lines
kernel/bpf/verifier.c +3
-2@@ -19524,8 +19527,11 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,int struct_meta_reg =BPF_REG_3 ;int node_offset_reg =BPF_REG_4 ; -/* rbtree_add has extra 'less' arg, so args-to-fixup are in diff regs */ -if (is_bpf_rbtree_add_kfunc (desc->func_id)) { +/* list_add/rbtree_add have an extra arg (prev/less), +* so args-to-fixup are in diff regs. +*/ This isn't a bug, but the multi-line comment format doesn't follow the BPF subsystem style. The preferred form in
kernel/bpf/has the opening /* on its own line.
Quoted 5 lines
Additionally, consider adding an
is_bpf_list_add_kfunc()helper instead of direct comparison with special_kfunc_list[KF_bpf_list_add].([email protected] and Eduard Zingerman, v10 RESEND)
https://lore.kernel.org/bpf/f573f7e0a5140fe6254d2ddfd3f69add455fa25c67c020c0312e2c2ceeca7c69@mail.kernel.org/
I provided a more detailed explanation in the link below.
https://lore.kernel.org/all/[email protected]/
Quoted 12 lines
kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 19537
Quoted 5 lines
kernel/bpf/verifier.c +2
-0+if (desc->func_id == special_kfunc_list[KF_bpf_list_add] || +is_bpf_rbtree_add_kfunc (desc->func_id)) { struct_meta_reg =BPF_REG_4 ; node_offset_reg =BPF_REG_5 ; }---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/26204125015
Signature · 2 lines
Thanks Kaitao Cheng
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 5 lines · On Thu, 21 May 2026 11:22:58 +0800 you wrote:
In BPF, a list can only be used to implement a stack structure. Due to an incomplete API set, only FIFO or LIFO operations are supported. The patches enhance the BPF list API, making it more list-like.
Quoted 8 lines
Five new kfuncs have been added:
bpf_list_del: remove a node from the listbpf_list_add_impl: insert a node after a given list nodebpf_list_is_first: check if a node is the first in the listbpf_list_is_last: check if a node is the last in the listbpf_list_empty: check if the list is empty[...]
Here is the summary with links:
- [bpf-next,v11,1/8] bpf: refactor __bpf_list_del to take list node pointer
https://git.kernel.org/bpf/bpf-next/c/cb339ac61d72 - [bpf-next,v11,2/8] bpf: clear list node owner and unlink before drop https://git.kernel.org/bpf/bpf-next/c/cfa6afa4b931 - [bpf-next,v11,3/8] bpf: allow non-owning list-node args via __nonown_allowed https://git.kernel.org/bpf/bpf-next/c/7c8c71591b76 - [bpf-next,v11,4/8] bpf: Introduce the bpf_list_del kfunc. https://git.kernel.org/bpf/bpf-next/c/187baa10963a - [bpf-next,v11,5/8] bpf: refactor __bpf_list_add to take insertion point via **prev_ptr https://git.kernel.org/bpf/bpf-next/c/e6919ff67c1e - [bpf-next,v11,6/8] bpf: Add bpf_list_add to insert node after a given list node https://git.kernel.org/bpf/bpf-next/c/a3493ca504f1 - [bpf-next,v11,7/8] bpf: add bpf_list_is_first/last/empty kfuncs https://git.kernel.org/bpf/bpf-next/c/745515d386eb - [bpf-next,v11,8/8] selftests/bpf: Add test cases for bpf_list_del/add/is_first/is_last/empty https://git.kernel.org/bpf/bpf-next/c/ba3dc064f406
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
[email protected]
AI reviewRe: [PATCH bpf-next v11 6/8] bpf: Add bpf_list_add to insert node after a given list node
In reply to Kaitao ChengThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] The `bpf_list_add` kfunc frees the inserted node on runtime failure, but the BPF verifier…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] The bpf_list_add kfunc frees the inserted node on runtime failure, but the BPF verifier unconditionally converts it to a non-owning reference, leading to a Use-After-Free (UAF) vulnerability.
- [High] The bpf_list_add kfunc is registered with KF_IMPLICIT_ARGS but lacks the required _impl function and BTF ID, causing the verifier to reject all programs attempting to call it. --
commit 41c84b8182697296804a152b66297f50e3a604cb
Author: Kaitao Cheng <[email protected]>
bpf: Add bpf_list_add to insert node after a given list node
This commit adds a new kfunc bpf_list_add() that inserts 'new' after 'prev' in a BPF linked list. The new node must be an owning reference, which the kfunc consumes and converts to a non-owning reference upon insertion.
Quoted 4 lines
kernel/bpf/helpers.c +0
-0
[ ... ]
kernel/bpf/helpers.c __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head, · line 2588
Quoted 13 lines
kernel/bpf/helpers.c +9
-0@@ -2577,6 +2577,16 @@ __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,return bpf_list_push_back (head, node, meta__ign, off); } +__bpf_kfunc int bpf_list_add(struct bpf_list_head *head, struct bpf_list_node *new, +struct bpf_list_node *prev__nonown_allowed, +struct btf_struct_meta *meta,u64 off) +{ +struct bpf_list_node_kern *n = (void *)new, *p = (void *)prev__nonown_allowed; +struct list_head *prev_ptr = &p->list_head; + +return __bpf_list_add (n, head, &prev_ptr, meta ? meta->record :NULL , off); +}
[Severity: Critical]
Is it possible for this to result in a use-after-free if __bpf_list_add() fails?
Looking at __bpf_list_add(), it introduces a runtime check that fails if the prev node doesn't belong to the target list. When it fails, the function calls __bpf_obj_drop_impl() to free the new node and returns -EINVAL.
However, the BPF verifier treats bpf_list_add() as a list push operation. During check_kfunc_call(), it unconditionally consumes the owning reference to the new node and converts it to a non-owning reference, assuming static success.
If the runtime check fails, the BPF program would be granted a valid non-owning reference to the freed new node inside the spinlock critical section, potentially leading to memory corruption.
[ ... ]
kernel/bpf/helpers.c BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) · line 4769
Quoted 5 lines
kernel/bpf/helpers.c +1
-0@@ -4756,6 +4766,7 @@ BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_list_push_front_impl) BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_list_push_back_impl) +BTF_ID_FLAGS (func, bpf_list_add,KF_IMPLICIT_ARGS )
[Severity: High]
Does this kfunc lack a required _impl function?
When a kfunc is marked with KF_IMPLICIT_ARGS, the verifier's find_kfunc_impl_proto() explicitly requires a corresponding _impl function to resolve the full prototype.
Since there is no bpf_list_add_impl() defined in kernel/bpf/helpers.c and registered with BTF, fetch_kfunc_meta() will return -EINVAL. Will this cause the verifier to unconditionally reject any BPF program attempting to use this kfunc?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
[email protected]
AI reviewRe: [PATCH bpf-next v11 7/8] bpf: add bpf_list_is_first/last/empty kfuncs
In reply to Kaitao ChengThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Multi-line comment uses deprecated style in `bpf_list_empty`. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Multi-line comment uses deprecated style in bpf_list_empty.
--
commit 22a1ce2a5267998c134b07d40be7e92224236cc6
Author: Kaitao Cheng <[email protected]>
bpf: add bpf_list_is_first/last/empty kfuncs
This commit adds three new kfuncs to allow BPF programs to directly query whether a node is the first or last in a linked list, or if the list is empty. This avoids the inefficiency of having to pop and push nodes just to check these properties.
Quoted 4 lines
kernel/bpf/helpers.c +0
-0
[ ... ]
Quoted 1 line
kernel/bpf/helpers.c +0
-0@@ -2656,6 +2656,43 @@ __bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head)
[ ... ]
kernel/bpf/helpers.c __bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head) · line 2689
Quoted 7 lines
kernel/bpf/helpers.c +7
-0+__bpf_kfunc bool bpf_list_empty(struct bpf_list_head *head) +{ +struct list_head *h = (struct list_head *)head; + +/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't +* called on its fields, so init here +*/
[Severity: Low]
This isn't a bug, but the multi-line comment uses a deprecated style. According to BPF subsystem guidelines, multi-line comments should have the opening /* on its own line.
Quoted 5 lines
kernel/bpf/helpers.c +5
-0+if (unlikely (!h->next)) +INIT_LIST_HEAD (h); + +return list_empty (h); +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7