bpf: Fix a race in reuseport_array_free()
Subsystems (4)
PATCH 1/1 bpf: Fix a race in reuseport_array_free()
In reuseport_array_free(), the rcu_read_lock() cannot ensure sk is still valid. It is because bpf_sk_reuseport_detach() can be called from __sk_destruct() which is invoked through call_rcu(..., __sk_destruct).
This patch takes the reuseport_lock in reuseport_array_free() which is not the fast path. The lock is taken inside the loop in case that the bpf map is big.
--- kernel/bpf/reuseport_array.c | 27 +++++---------------------- 1 file changed, 5 insertions(+), 22 deletions(-)
kernel/bpf/reuseport_array.c +5 -22
@@ -103,29 +103,11 @@ static void reuseport_array_free(struct bpf_map *map)* array now. Hence, this function only races with * bpf_sk_reuseport_detach() which was triggerred by * close() or disconnect(). -* -* This function and bpf_sk_reuseport_detach() are -* both removing sk from "array". Who removes it -* first does not matter. -* -* The only concern here is bpf_sk_reuseport_detach() -* may access "array" which is being freed here. -* bpf_sk_reuseport_detach() access this "array" -* through sk->sk_user_data _and_ with sk->sk_callback_lock -* held which is enough because this "array" is not freed -* until all sk->sk_user_data has stopped referencing this "array". -* -* Hence, due to the above, taking "reuseport_lock" is not -* needed here. */ - -/* -* Since reuseport_lock is not taken, sk is accessed under -* rcu_read_lock() -*/ -rcu_read_lock ();for (i =0 ; i < map->max_entries; i++) { - sk =rcu_dereference (array->ptrs[i]); +spin_lock_bh (&reuseport_lock); + sk =rcu_dereference_protected (array->ptrs[i], +lockdep_is_held (&reuseport_lock));if (sk) {write_lock_bh (&sk->sk_callback_lock);/*
@@ -137,8 +119,9 @@ static void reuseport_array_free(struct bpf_map *map)write_unlock_bh (&sk->sk_callback_lock);RCU_INIT_POINTER (array->ptrs[i],NULL ); } +spin_unlock_bh (&reuseport_lock); +cond_resched (); } -rcu_read_unlock ();/* * Once reaching here, all sk->sk_user_data is not
Signature · 1 line
2.17.1
Quoted 3 lines · Martin KaFai Lau
In
reuseport_array_free(), thercu_read_lock()cannot ensure sk is still valid. It is becausebpf_sk_reuseport_detach()can be called from__sk_destruct()which is invoked throughcall_rcu(..., __sk_destruct).
We could question why reuseport_detach_sock(sk) is called from __sk_destruct() (after the rcu grace period) instead of sk_destruct() ?
Quoted 6 lines
This patch takes the
reuseport_lockinreuseport_array_free()which is not the fast path. The lock is taken inside the loop in case that the bpf map is big.Signed-off-by Martin KaFai Lau <[email protected]>
Quoted 9 lines · Eric Dumazet
Quoted 3 lines · Martin KaFai Lau
In
reuseport_array_free(), thercu_read_lock()cannot ensure sk is still valid. It is becausebpf_sk_reuseport_detach()can be called from__sk_destruct()which is invoked throughcall_rcu(..., __sk_destruct).We could question why
reuseport_detach_sock(sk)is called from__sk_destruct()(after the rcu grace period) instead ofsk_destruct()?
Agree. It is another way to fix it.
In this patch, I chose to avoid the need to single out a special treatment for reuseport_detach_sock() in sk_destruct().
I am happy either way. What do you think?
Quoted 9 lines
Quoted 7 lines
This patch takes the
reuseport_lockinreuseport_array_free()which is not the fast path. The lock is taken inside the loop in case that the bpf map is big.Signed-off-by Martin KaFai Lau <[email protected]>Fixes 5dc4c4b7d4e8 ("bpf: Introduce BPF_MAP_TYPE_REUSEPORT_SOCKARRAY")
Ah...missed that. Thanks!
Quoted 8 lines · Martin Lau
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Martin KaFai Lau
In
reuseport_array_free(), thercu_read_lock()cannot ensure sk is still valid. It is becausebpf_sk_reuseport_detach()can be called from__sk_destruct()which is invoked throughcall_rcu(..., __sk_destruct).
Quoted 8 lines
Quoted 2 lines
We could question why
reuseport_detach_sock(sk)is called from__sk_destruct()(after the rcu grace period) instead ofsk_destruct()?Agree. It is another way to fix it.
In this patch, I chose to avoid the need to single out a special treatment for
reuseport_detach_sock()insk_destruct().I am happy either way. What do you think?
It seems that since we call reuseport_detach_sock() after the rcu grace period, another cpu could catch the sk pointer in reuse->socks[] array and use it right before our cpu frees the socket.
RCU rules are not properly applied here I think.
The rules for deletion are :
1) unpublish object from various lists/arrays/hashes.
2) rcu_grace_period
3) free the object.
If we fix the unpublish (we need to anyway to make the data path safe), then your patch is not needed ?
What about (totally untested, might be horribly wrong)
net/core/sock.c +7 -3
@@ -1700,8 +1700,6 @@ static void __sk_destruct(struct rcu_head *head)sk_filter_uncharge (sk, filter);RCU_INIT_POINTER (sk->sk_filter,NULL ); } -if (rcu_access_pointer (sk->sk_reuseport_cb)) -reuseport_detach_sock (sk);sock_disable_timestamp (sk,SK_FLAGS_TIMESTAMP );
@@ -1728,7 +1726,13 @@ static void __sk_destruct(struct rcu_head *head)void sk_destruct (struct sock *sk) { -if (sock_flag (sk,SOCK_RCU_FREE )) +bool use_call_rcu =sock_flag (sk,SOCK_RCU_FREE ); + +if (rcu_access_pointer (sk->sk_reuseport_cb)) { +reuseport_detach_sock (sk); + use_call_rcu = true; + } +if (use_call_rcu)call_rcu (&sk->sk_rcu, __sk_destruct);else __sk_destruct (&sk->sk_rcu);
Quoted 24 lines · Eric Dumazet
Quoted 8 lines · Martin Lau
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Martin KaFai Lau
In
reuseport_array_free(), thercu_read_lock()cannot ensure sk is still valid. It is becausebpf_sk_reuseport_detach()can be called from__sk_destruct()which is invoked throughcall_rcu(..., __sk_destruct).Quoted 8 lines
Quoted 2 lines
We could question why
reuseport_detach_sock(sk)is called from__sk_destruct()(after the rcu grace period) instead ofsk_destruct()?Agree. It is another way to fix it.
In this patch, I chose to avoid the need to single out a special treatment for
reuseport_detach_sock()insk_destruct().I am happy either way. What do you think?
It seems that since we call
reuseport_detach_sock()after the rcu grace period, another cpu could catch the sk pointer in reuse->socks[] array and use it right before our cpu frees the socket.
Quoted 5 lines
RCU rules are not properly applied here I think.
The rules for deletion are :
1) unpublish object from various lists/arrays/hashes.
Thanks for the analysis. Agreed. Indeed, there is an issue in reuse->socks[] which is shared with other sockets and they may pick up the destructed sk from reuse->socks[].
Quoted 5 lines
2)
rcu_grace_period
3) free the object.If we fix the unpublish (we need to anyway to make the data path safe), then your patch is not needed ?
Correct, not needed.
Quoted 2 lines
What about (totally untested, might be horribly wrong)
I had something similar in mind also. I will take a closer look and re-spin v2.
Quoted 29 lines
net/core/sock.c +7
-3@@ -1700,8 +1700,6 @@ static void __sk_destruct(struct rcu_head *head)sk_filter_uncharge (sk, filter);RCU_INIT_POINTER (sk->sk_filter,NULL ); } -if (rcu_access_pointer (sk->sk_reuseport_cb)) -reuseport_detach_sock (sk);sock_disable_timestamp (sk,SK_FLAGS_TIMESTAMP );@@ -1728,7 +1726,13 @@ static void __sk_destruct(struct rcu_head *head)void sk_destruct (struct sock *sk) { -if (sock_flag (sk,SOCK_RCU_FREE )) +bool use_call_rcu =sock_flag (sk,SOCK_RCU_FREE ); + +if (rcu_access_pointer (sk->sk_reuseport_cb)) { +reuseport_detach_sock (sk); + use_call_rcu = true; + } +if (use_call_rcu)call_rcu (&sk->sk_rcu, __sk_destruct);else __sk_destruct (&sk->sk_rcu);