bpf: use RCU-safe iteration in dev_map_redirect_multi() SKB path
Subsystems (4)
PATCH 1/1 bpf: use RCU-safe iteration in dev_map_redirect_multi() SKB path
The DEVMAP_HASH branch in dev_map_redirect_multi() uses hlist_for_each_entry_safe() to iterate hash buckets, but this function runs under RCU protection (called from xdp_do_generic_redirect_map() in softirq context). Concurrent writers (__dev_map_hash_update_elem, dev_map_hash_delete_elem) modify the list using RCU primitives (hlist_add_head_rcu, hlist_del_rcu).
hlist_for_each_entry_safe() performs plain pointer dereferences without rcu_dereference(), missing the acquire barrier needed to pair with writers' rcu_assign_pointer(). On weakly-ordered architectures (ARM64, POWER), a reader can observe a partially-constructed node. It also defeats CONFIG_PROVE_RCU lockdep validation and KCSAN data-race detection.
Replace with hlist_for_each_entry_rcu() using rcu_read_lock_bh_held() as the lockdep condition, consistent with the rcu_dereference_check() used in the DEVMAP (non-hash) branch of the same functions. Also fix the same incorrect lockdep_is_held(&dtab->index_lock) condition in dev_map_enqueue_multi(), where the lock is not held either.
--- kernel/bpf/devmap.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
kernel/bpf/devmap.c +2 -3
@@ -665,7 +665,7 @@ int dev_map_enqueue_multi(struct xdp_frame *xdpf, struct net_device *dev_rx,for (i =0 ; i < dtab->n_buckets; i++) { head =dev_map_index_hash (dtab, i);hlist_for_each_entry_rcu (dst, head, index_hlist, -lockdep_is_held (&dtab->index_lock)) { +rcu_read_lock_bh_held ()) {if (!is_valid_dst (dst, xdpf))continue ;
@@ -747,7 +747,6 @@ int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb,struct bpf_dtab_netdev *dst, *last_dst =NULL ;int excluded_devices[1 +MAX_NEST_DEV ];struct hlist_head *head; -struct hlist_node *next;int num_excluded =0 ;unsigned int i;int err;
@@ -787,7 +786,7 @@ int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, }else {/* BPF_MAP_TYPE_DEVMAP_HASH */ for (i =0 ; i < dtab->n_buckets; i++) { head =dev_map_index_hash (dtab, i); -hlist_for_each_entry_safe (dst, next, head, index_hlist) { +hlist_for_each_entry_rcu (dst, head, index_hlist,rcu_read_lock_bh_held ()) {if (is_ifindex_excluded (excluded_devices, num_excluded, dst->dev->ifindex))continue ;
Signature · 1 line
2.53.0
Quoted 14 lines · David Carlier
The
DEVMAP_HASHbranch indev_map_redirect_multi()useshlist_for_each_entry_safe()to iterate hash buckets, but this function runs under RCU protection (called fromxdp_do_generic_redirect_map()in softirq context). Concurrent writers (__dev_map_hash_update_elem,dev_map_hash_delete_elem) modify the list using RCU primitives (hlist_add_head_rcu,hlist_del_rcu).
hlist_for_each_entry_safe()performs plain pointer dereferences withoutrcu_dereference(), missing the acquire barrier needed to pair with writers'rcu_assign_pointer(). On weakly-ordered architectures (ARM64, POWER), a reader can observe a partially-constructed node. It also defeatsCONFIG_PROVE_RCUlockdep validation and KCSAN data-race detection.
Quoted 5 lines
Replace with
hlist_for_each_entry_rcu()usingrcu_read_lock_bh_held()as the lockdep condition, consistent with thercu_dereference_check()used in the DEVMAP (non-hash) branch of the same functions. Also fix the same incorrectlockdep_is_held(&dtab->index_lock)condition indev_map_enqueue_multi(), where the lock is not held either.
Applied. Thanks.
Take a look at the review here:
https://sashiko.dev/#/patchset/20260320072645.16731-1-devnexen%40gmail.com
The first comment is legit and please followup. The second comment looks unrelated.
This patch was applied to bpf/bpf-next.git (master) by Martin KaFai Lau <[email protected]>:
Hello:
This patch was applied to bpf/bpf-next.git (master) by Martin KaFai Lau <[email protected]>:
Quoted 8 lines · On Fri, 20 Mar 2026 07:26:45 +0000 you wrote:
The
DEVMAP_HASHbranch indev_map_redirect_multi()useshlist_for_each_entry_safe()to iterate hash buckets, but this function runs under RCU protection (called fromxdp_do_generic_redirect_map()in softirq context). Concurrent writers (__dev_map_hash_update_elem,dev_map_hash_delete_elem) modify the list using RCU primitives (hlist_add_head_rcu,hlist_del_rcu).[...]
Here is the summary with links:
- [bpf-next] bpf: use RCU-safe iteration in dev_map_redirect_multi() SKB path
https://git.kernel.org/bpf/bpf-next/c/8ed82f807bb0
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html