bpf: Use separate RCU callbacks for freeing selem
Kumar Kartikeya Dwivedi
maintainerPATCH 1/1 bpf: Use separate RCU callbacks for freeing selem
Martin suggested that instead of using a byte in the hole (which he has a use for in his future patch) in bpf_local_storage_elem, we can dispatch a different call_rcu callback based on whether we need to free special fields in bpf_local_storage_elem data. The free path, described in commit 9db44fdd8105 ("bpf: Support kptrs in local storage maps"), only waits for call_rcu callbacks when there are special (kptrs, etc.) fields in the map value, hence it is necessary that we only access smap in this case.
Therefore, dispatch different RCU callbacks based on the BPF map has a valid btf_record, which dereference and use smap's btf_record only when it is valid.
--- include/linux/bpf_local_storage.h | 6 --- kernel/bpf/bpf_local_storage.c | 76 +++++++++++++++++++------------ 2 files changed, 46 insertions(+), 36 deletions(-)
include/linux/bpf_local_storage.h +0 -6
@@ -74,12 +74,6 @@ struct bpf_local_storage_elem {struct hlist_node snode;/* Linked to bpf_local_storage */ struct bpf_local_storage __rcu *local_storage;struct rcu_head rcu; -bool can_use_smap;/* Is it safe to access smap in bpf_selem_free_* RCU -* callbacks? bpf_local_storage_map_free only -* executes rcu_barrier when there are special -* fields, this field remembers that to ensure we -* don't access already freed smap in sdata. -*/ /* 8 bytes hole */ /* The data is stored in another cacheline to minimize * the number of cachelines access during a cache hit.
kernel/bpf/bpf_local_storage.c +46 -30
122 lines not on this page · raw message
--
2.39.2
Martin KaFai Lau
reviewerQuoted 9 lines · Kumar Kartikeya Dwivedi
Martin suggested that instead of using a byte in the hole (which he has a use for in his future patch) in
bpf_local_storage_elem, we can dispatch a differentcall_rcucallback based on whether we need to free special fields inbpf_local_storage_elemdata. The free path, described in commit9db44fdd8105("bpf: Support kptrs in local storage maps"), only waits forcall_rcucallbacks when there are special (kptrs, etc.) fields in the map value, hence it is necessary that we only access smap in this case.
Quoted 5 lines
Therefore, dispatch different RCU callbacks based on the BPF map has a valid
btf_record, which dereference and use smap'sbtf_recordonly when it is valid.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>
Thanks for your patch. I have already made a similar change in my local branch which has some differences like refactored it a little for my work. The set is almost ready. Do you mind if I include your patch in my set and keep your SOB?
Kumar Kartikeya Dwivedi
maintainerQuoted 15 lines · Martin KaFai Lau
Quoted 13 lines · Kumar Kartikeya Dwivedi
Martin suggested that instead of using a byte in the hole (which he has a use for in his future patch) in
bpf_local_storage_elem, we can dispatch a differentcall_rcucallback based on whether we need to free special fields inbpf_local_storage_elemdata. The free path, described in commit9db44fdd8105("bpf: Support kptrs in local storage maps"), only waits forcall_rcucallbacks when there are special (kptrs, etc.) fields in the map value, hence it is necessary that we only access smap in this case.Therefore, dispatch different RCU callbacks based on the BPF map has a valid
btf_record, which dereference and use smap'sbtf_recordonly when it is valid.
Quoted 6 lines
Quoted 1 line
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>Thanks for your patch. I have already made a similar change in my local branch which has some differences like refactored it a little for my work. The set is almost ready. Do you mind if I include your patch in my set and keep your SOB?
No problem, please do.
Martin KaFai Lau
reviewerQuoted 16 lines · Kumar Kartikeya Dwivedi
Quoted 15 lines · Martin KaFai Lau
Quoted 13 lines · Kumar Kartikeya Dwivedi
Martin suggested that instead of using a byte in the hole (which he has a use for in his future patch) in
bpf_local_storage_elem, we can dispatch a differentcall_rcucallback based on whether we need to free special fields inbpf_local_storage_elemdata. The free path, described in commit9db44fdd8105("bpf: Support kptrs in local storage maps"), only waits forcall_rcucallbacks when there are special (kptrs, etc.) fields in the map value, hence it is necessary that we only access smap in this case.Therefore, dispatch different RCU callbacks based on the BPF map has a valid
btf_record, which dereference and use smap'sbtf_recordonly when it is valid.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>Thanks for your patch. I have already made a similar change in my local branch which has some differences like refactored it a little for my work. The set is almost ready. Do you mind if I include your patch in my set and keep your SOB?
No problem, please do.
Please ignore my previous message. I will make some adjustments on my set.
Applied with some changes on de-referencing SDATA(selem)->smap to address these warnings:
# this is addressed by rcu_dereference_protected(..., true) ../kernel/bpf/bpf_local_storage.c:117:41: warning: dereference of noderef expression
# this is addressed by directly using the earlier dereferenced smap pointer ../kernel/bpf/bpf_local_storage.c:200:27: warning: dereference of noderef expression