bpf: net: Fixes in sk_user_data of reuseport_array
Subsystems (4)
v2 cover bpf: net: Fixes in sk_user_data of reuseport_array
This set fixes two issues on sk_user_data when a sk is added to a reuseport_array.
The first patch is to avoid the sk_user_data being copied
to a cloned sk. The second patch avoids doing bpf_sk_reuseport_detach() on sk_user_data that is not managed by reuseport_array.
Since the changes are mostly related to bpf reuseport_array, so it is currently tagged as bpf fixes.
v2:
- Avoid ~3UL (Andrii)
Martin KaFai Lau (2):
bpf: net: Avoid copying sk_user_data of reuseport_array during
sk_clone
bpf: net: Avoid incorrect bpf_sk_reuseport_detach call
include/net/sock.h | 3 ++- kernel/bpf/reuseport_array.c | 14 ++++++++++---- 2 files changed, 12 insertions(+), 5 deletions(-)
Signature · 1 line
2.24.1
PATCH v2 1/2 bpf: net: Avoid copying sk_user_data of reuseport_array during sk_clone
It makes little sense for copying sk_user_data of reuseport_array during sk_clone_lock(). This patch reuses the SK_USER_DATA_NOCOPY bit introduced in commit f1ff5ce2cd5e ("net, sk_msg: Clear sk_user_data pointer on clone if tagged"). It is used to mark the sk_user_data is not supposed to be copied to its clone.
Although the cloned sk's sk_user_data will not be used/freed in bpf_sk_reuseport_detach(), this change can still allow the cloned sk's sk_user_data to be used by some other means.
Freeing the reuseport_array's sk_user_data does not require a rcu grace period. Thus, the existing rcu_assign_sk_user_data_nocopy() is not used.
--- kernel/bpf/reuseport_array.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/bpf/reuseport_array.c +9 -4
@@ -20,11 +20,14 @@ static struct reuseport_array *reuseport_array(struct bpf_map *map)/* The caller must hold the reuseport_lock */ void bpf_sk_reuseport_detach (struct sock *sk) { -struct sock __rcu **socks; +uintptr_t sk_user_data;write_lock_bh (&sk->sk_callback_lock); - socks = sk->sk_user_data; -if (socks) { + sk_user_data = (uintptr_t )sk->sk_user_data; +if (sk_user_data) { +struct sock __rcu **socks; + + socks = (void *)(sk_user_data &SK_USER_DATA_PTRMASK );WRITE_ONCE (sk->sk_user_data,NULL );/* * Do not move this NULL assignment outside of
@@ -252,6 +255,7 @@ int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,struct sock *free_osk =NULL , *osk, *nsk;struct sock_reuseport *reuse;u32 index = *(u32 *)key; +uintptr_t sk_user_data;struct socket *socket;int err, fd;
@@ -305,7 +309,8 @@ int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,if (err) goto put_file_unlock; -WRITE_ONCE (nsk->sk_user_data, &array->ptrs[index]); + sk_user_data = (uintptr_t )&array->ptrs[index] |SK_USER_DATA_NOCOPY ; +WRITE_ONCE (nsk->sk_user_data, (void *)sk_user_data);rcu_assign_pointer (array->ptrs[index], nsk); free_osk = osk; err =0 ;
Signature · 1 line
2.24.1
Re: [PATCH v2 bpf 1/2] bpf: net: Avoid copying sk_user_data of reuseport_array during sk_clone
Quoted 50 lines · On Thu, 09 Jul 2020 06:11:04 +0000 Martin KaFai Lau <[email protected]> wrote:
It makes little sense for copying
sk_user_dataofreuseport_arrayduringsk_clone_lock(). This patch reuses theSK_USER_DATA_NOCOPYbit introduced in commitf1ff5ce2cd5e("net,sk_msg: Clearsk_user_datapointer on clone if tagged"). It is used to mark thesk_user_datais not supposed to be copied to its clone.Although the cloned sk's
sk_user_datawill not be used/freed inbpf_sk_reuseport_detach(), this change can still allow the cloned sk'ssk_user_datato be used by some other means.Freeing the reuseport_array's
sk_user_datadoes not require a rcu grace period. Thus, the existingrcu_assign_sk_user_data_nocopy()is not used.Fixes 5dc4c4b7d4e8 ("bpf: Introduce BPF_MAP_TYPE_REUSEPORT_SOCKARRAY")Signed-off-by Martin KaFai Lau <[email protected]>--- kernel/bpf/reuseport_array.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)kernel/bpf/reuseport_array.c +7
-3@@ -20,11 +20,14 @@ static struct reuseport_array *reuseport_array(struct bpf_map *map)/* The caller must hold the reuseport_lock */ void bpf_sk_reuseport_detach (struct sock *sk) { -struct sock __rcu **socks; +uintptr_t sk_user_data;write_lock_bh (&sk->sk_callback_lock); - socks = sk->sk_user_data; -if (socks) { + sk_user_data = (uintptr_t )sk->sk_user_data; +if (sk_user_data) { +struct sock __rcu **socks; + + socks = (void *)(sk_user_data &SK_USER_DATA_PTRMASK );WRITE_ONCE (sk->sk_user_data,NULL );/* * Do not move this NULL assignment outside of @@ -252,6 +255,7 @@ int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,struct sock *free_osk =NULL , *osk, *nsk;struct sock_reuseport *reuse;u32 index = *(u32 *)key; +uintptr_t sk_user_data;struct socket *socket;int err, fd;
kernel/bpf/reuseport_array.c int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key, · line 316
Quoted 9 lines
kernel/bpf/reuseport_array.c +2
-1@@ -305,7 +309,8 @@ int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,if (err) goto put_file_unlock; -WRITE_ONCE (nsk->sk_user_data, &array->ptrs[index]); + sk_user_data = (uintptr_t )&array->ptrs[index] |SK_USER_DATA_NOCOPY ; +WRITE_ONCE (nsk->sk_user_data, (void *)sk_user_data);rcu_assign_pointer (array->ptrs[index], nsk); free_osk = osk; err =0 ;
Thanks for fixing this before I got around to it. Now we can use reuseport with sockmap splicing :-)
Daniel Borkmann
maintainerRe: [PATCH v2 bpf 1/2] bpf: net: Avoid copying sk_user_data of reuseport_array during sk_clone
In reply to Martin KaFai LauQuoted 5 lines · Martin KaFai Lau
It makes little sense for copying
sk_user_dataofreuseport_arrayduringsk_clone_lock(). This patch reuses theSK_USER_DATA_NOCOPYbit introduced in commitf1ff5ce2cd5e("net,sk_msg: Clearsk_user_datapointer on clone if tagged"). It is used to mark thesk_user_datais not supposed to be copied to its clone.
Quoted 7 lines
Although the cloned sk's
sk_user_datawill not be used/freed inbpf_sk_reuseport_detach(), this change can still allow the cloned sk'ssk_user_datato be used by some other means.Freeing the reuseport_array's
sk_user_datadoes not require a rcu grace period. Thus, the existingrcu_assign_sk_user_data_nocopy()is not used.
nit: Would have been nice though to add a nonrcu API for this nevertheless instead of open coding.
Quoted 2 lines
Fixes 5dc4c4b7d4e8 ("bpf: Introduce BPF_MAP_TYPE_REUSEPORT_SOCKARRAY")Signed-off-by Martin KaFai Lau <[email protected]>
Re: [PATCH v2 bpf 1/2] bpf: net: Avoid copying sk_user_data of reuseport_array during sk_clone
Quoted 10 lines · Daniel Borkmann
Quoted 9 lines · Martin KaFai Lau
It makes little sense for copying
sk_user_dataofreuseport_arrayduringsk_clone_lock(). This patch reuses theSK_USER_DATA_NOCOPYbit introduced in commitf1ff5ce2cd5e("net,sk_msg: Clearsk_user_datapointer on clone if tagged"). It is used to mark thesk_user_datais not supposed to be copied to its clone.Although the cloned sk's
sk_user_datawill not be used/freed inbpf_sk_reuseport_detach(), this change can still allow the cloned sk'ssk_user_datato be used by some other means.
Quoted 6 lines
Quoted 3 lines
Freeing the reuseport_array's
sk_user_datadoes not require a rcu grace period. Thus, the existingrcu_assign_sk_user_data_nocopy()is not used.nit: Would have been nice though to add a nonrcu API for this nevertheless instead of open coding.
Agreed. I will create a follow-up patch to bpf-next later.
I had created (READ|WRITE)_ONCE_SK_USER_DATA() earlier but then noticed there is no use on the READ_ONCE in that particular call, so ditched the idea. I think it does not matter much and should just use READ_ONCE also.
PATCH v2 2/2 bpf: net: Avoid incorrect bpf_sk_reuseport_detach call
In reply to Martin KaFai Laubpf_sk_reuseport_detach is currently called when sk->sk_user_data is not NULL. It is incorrect because sk->sk_user_data may not be managed by the bpf's reuseport_array. It has been reported in [1] that, the bpf_sk_reuseport_detach() which is called from udp_lib_unhash() has corrupted the sk_user_data managed by l2tp.
This patch solves it by using another bit (defined as SK_USER_DATA_BPF) of the sk_user_data pointer value. It marks that a sk_user_data is managed/owned by BPF.
The patch depends on a PTRMASK introduced in
commit f1ff5ce2cd5e ("net, sk_msg: Clear sk_user_data pointer on clone if tagged").
[ Note: sk->sk_user_data is used by bpf's reuseport_array only when a sk is
added to the bpf's reuseport_array.
i.e. doing setsockopt(SO_REUSEPORT) and having "sk->sk_reuseport == 1" alone will not stop sk->sk_user_data being used by other means. ]
[1]: https://lore.kernel.org/netdev/[email protected]/
--- include/net/sock.h | 3 ++- kernel/bpf/reuseport_array.c | 5 +++-- 2 files changed, 5 insertions(+), 3 deletions(-)
include/net/sock.h +2 -1
@@ -533,7 +533,8 @@ enum sk_pacing {* be copied. */ #define SK_USER_DATA_NOCOPY 1UL -#define SK_USER_DATA_PTRMASK ~(SK_USER_DATA_NOCOPY) +#define SK_USER_DATA_BPF 2UL/* Managed by BPF */ +#define SK_USER_DATA_PTRMASK ~(SK_USER_DATA_NOCOPY | SK_USER_DATA_BPF)/** * sk_user_data_is_nocopy - Test if sk_user_data pointer must not be copied
kernel/bpf/reuseport_array.c +3 -2
@@ -24,7 +24,7 @@ void bpf_sk_reuseport_detach(struct sock *sk)write_lock_bh (&sk->sk_callback_lock); sk_user_data = (uintptr_t )sk->sk_user_data; -if (sk_user_data) { +if (sk_user_data &SK_USER_DATA_BPF ) {struct sock __rcu **socks; socks = (void *)(sk_user_data &SK_USER_DATA_PTRMASK );
@@ -309,7 +309,8 @@ int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,if (err) goto put_file_unlock; - sk_user_data = (uintptr_t )&array->ptrs[index] |SK_USER_DATA_NOCOPY ; + sk_user_data = (uintptr_t )&array->ptrs[index] |SK_USER_DATA_NOCOPY | +SK_USER_DATA_BPF ;WRITE_ONCE (nsk->sk_user_data, (void *)sk_user_data);rcu_assign_pointer (array->ptrs[index], nsk); free_osk = osk;
Signature · 1 line
2.24.1
Re: [PATCH v2 bpf 2/2] bpf: net: Avoid incorrect bpf_sk_reuseport_detach call
Quoted 9 lines · Martin KaFai Lau
bpf_sk_reuseport_detachis currently called whensk->sk_user_datais not NULL. It is incorrect becausesk->sk_user_datamay not be managed by the bpf'sreuseport_array. It has been reported in [1] that, thebpf_sk_reuseport_detach()which is called fromudp_lib_unhash()has corrupted thesk_user_datamanaged by l2tp.This patch solves it by using another bit (defined as
SK_USER_DATA_BPF) of thesk_user_datapointer value. It marks that ask_user_datais managed/owned by BPF.
I have reservations about using a bit in sk_user_data to indicate ownership of that pointer. But putting that aside, I confirm that the patch fixes the problem.
Re: [PATCH v2 bpf 2/2] bpf: net: Avoid incorrect bpf_sk_reuseport_detach call
Quoted 11 lines · James Chapman
Quoted 9 lines · Martin KaFai Lau
bpf_sk_reuseport_detachis currently called whensk->sk_user_datais not NULL. It is incorrect becausesk->sk_user_datamay not be managed by the bpf'sreuseport_array. It has been reported in [1] that, thebpf_sk_reuseport_detach()which is called fromudp_lib_unhash()has corrupted thesk_user_datamanaged by l2tp.This patch solves it by using another bit (defined as
SK_USER_DATA_BPF) of thesk_user_datapointer value. It marks that ask_user_datais managed/owned by BPF.
Quoted 7 lines
I have reservations about using a bit in
sk_user_datato indicate ownership of that pointer. But putting that aside, I confirm that the patch fixes the problem.Acked-by James Chapman <[email protected]>Tested-by James Chapman <[email protected]>Reported-by [email protected]
Thanks for the test!
One bit of sk_user_data has already been used to indicate SK_USER_DATA_NOCOPY.
I think using another bit is the cleanest fix for the bpf/net branch instead of tracking this somewhere else.
Quoted 13 lines · Martin KaFai Lau
This set fixes two issues on
sk_user_datawhen a sk is added to areuseport_array.The first patch is to avoid the
sk_user_databeing copied
to a cloned sk. The second patch avoids doingbpf_sk_reuseport_detach()onsk_user_datathat is not managed byreuseport_array.Since the changes are mostly related to bpf
reuseport_array, so it is currently tagged as bpf fixes.v2:
- Avoid ~3UL (Andrii)
Quoted 9 lines
Martin KaFai Lau (2):
bpf: net: Avoid copyingsk_user_dataofreuseport_arrayduring
sk_clone
bpf: net: Avoid incorrectbpf_sk_reuseport_detachcallinclude/net/sock.h | 3 ++- kernel/bpf/reuseport_array.c | 14 ++++++++++---- 2 files changed, 12 insertions(+), 5 deletions(-)
Applied, thanks!