Skip to contentSkip to conversation

bpf: net: Fixes in sk_user_data of reuseport_array

PATCH Applied Martin KaFai Lau 4 participants 9 messages Jul 9, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover bpf: net: Fixes in sk_user_data of reuseport_array

This set fixes two issues on sk_user_data when a sk is added to a reuseport_array.

The first patch is to avoid the sk_user_data being copied
to a cloned sk. The second patch avoids doing bpf_sk_reuseport_detach() on sk_user_data that is not managed by reuseport_array.

Since the changes are mostly related to bpf reuseport_array, so it is currently tagged as bpf fixes.

v2:
- Avoid ~3UL (Andrii)

Martin KaFai Lau (2):
bpf: net: Avoid copying sk_user_data of reuseport_array during
sk_clone
bpf: net: Avoid incorrect bpf_sk_reuseport_detach call

 include/net/sock.h           |  3 ++-
 kernel/bpf/reuseport_array.c | 14 ++++++++++----
 2 files changed, 12 insertions(+), 5 deletions(-)
Signature · 1 line
2.24.1
#1 lore raw

PATCH v2 1/2 bpf: net: Avoid copying sk_user_data of reuseport_array during sk_clone

It makes little sense for copying sk_user_data of reuseport_array during sk_clone_lock(). This patch reuses the SK_USER_DATA_NOCOPY bit introduced in commit f1ff5ce2cd5e ("net, sk_msg: Clear sk_user_data pointer on clone if tagged"). It is used to mark the sk_user_data is not supposed to be copied to its clone.

Although the cloned sk's sk_user_data will not be used/freed in bpf_sk_reuseport_detach(), this change can still allow the cloned sk's sk_user_data to be used by some other means.

Freeing the reuseport_array's sk_user_data does not require a rcu grace period. Thus, the existing rcu_assign_sk_user_data_nocopy() is not used.

Fixes 5dc4c4b7d4e8 ("bpf: Introduce BPF_MAP_TYPE_REUSEPORT_SOCKARRAY")
---
 kernel/bpf/reuseport_array.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/bpf/reuseport_array.c +9 -4
index 21cde24386db..a95bc8d7e812 100644
@@ -20,11 +20,14 @@ static struct reuseport_array *reuseport_array(struct bpf_map *map)
 /* The caller must hold the reuseport_lock */
 void bpf_sk_reuseport_detach(struct sock *sk)
 {
-	struct sock __rcu **socks;
+	uintptr_t sk_user_data;
 
 	write_lock_bh(&sk->sk_callback_lock);
-	socks = sk->sk_user_data;
-	if (socks) {
+	sk_user_data = (uintptr_t)sk->sk_user_data;
+	if (sk_user_data) {
+		struct sock __rcu **socks;
+
+		socks = (void *)(sk_user_data & SK_USER_DATA_PTRMASK);
 		WRITE_ONCE(sk->sk_user_data, NULL);
 		/*
 		 * Do not move this NULL assignment outside of
@@ -252,6 +255,7 @@ int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,
 	struct sock *free_osk = NULL, *osk, *nsk;
 	struct sock_reuseport *reuse;
 	u32 index = *(u32 *)key;
+	uintptr_t sk_user_data;
 	struct socket *socket;
 	int err, fd;
 
@@ -305,7 +309,8 @@ int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,
 	if (err)
 		goto put_file_unlock;
 
-	WRITE_ONCE(nsk->sk_user_data, &array->ptrs[index]);
+	sk_user_data = (uintptr_t)&array->ptrs[index] | SK_USER_DATA_NOCOPY;
+	WRITE_ONCE(nsk->sk_user_data, (void *)sk_user_data);
 	rcu_assign_pointer(array->ptrs[index], nsk);
 	free_osk = osk;
 	err = 0;
Signature · 1 line
2.24.1
Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH v2 bpf 1/2] bpf: net: Avoid copying sk_user_data of reuseport_array during sk_clone

Quoted 50 lines · On Thu, 09 Jul 2020 06:11:04 +0000 Martin KaFai Lau <[email protected]> wrote:

It makes little sense for copying sk_user_data of reuseport_array during sk_clone_lock(). This patch reuses the SK_USER_DATA_NOCOPY bit introduced in commit f1ff5ce2cd5e ("net, sk_msg: Clear sk_user_data pointer on clone if tagged"). It is used to mark the sk_user_data is not supposed to be copied to its clone.

Although the cloned sk's sk_user_data will not be used/freed in bpf_sk_reuseport_detach(), this change can still allow the cloned sk's sk_user_data to be used by some other means.

Freeing the reuseport_array's sk_user_data does not require a rcu grace period. Thus, the existing rcu_assign_sk_user_data_nocopy() is not used.

Fixes 5dc4c4b7d4e8 ("bpf: Introduce BPF_MAP_TYPE_REUSEPORT_SOCKARRAY")
---
 kernel/bpf/reuseport_array.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/bpf/reuseport_array.c +7 -3
index 21cde24386db..a95bc8d7e812 100644
@@ -20,11 +20,14 @@ static struct reuseport_array *reuseport_array(struct bpf_map *map)
 /* The caller must hold the reuseport_lock */
 void bpf_sk_reuseport_detach(struct sock *sk)
 {
-	struct sock __rcu **socks;
+	uintptr_t sk_user_data;
 
 	write_lock_bh(&sk->sk_callback_lock);
-	socks = sk->sk_user_data;
-	if (socks) {
+	sk_user_data = (uintptr_t)sk->sk_user_data;
+	if (sk_user_data) {
+		struct sock __rcu **socks;
+
+		socks = (void *)(sk_user_data & SK_USER_DATA_PTRMASK);
 		WRITE_ONCE(sk->sk_user_data, NULL);
 		/*
 		 * Do not move this NULL assignment outside of
@@ -252,6 +255,7 @@ int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,
 	struct sock *free_osk = NULL, *osk, *nsk;
 	struct sock_reuseport *reuse;
 	u32 index = *(u32 *)key;
+	uintptr_t sk_user_data;
 	struct socket *socket;
 	int err, fd;
 

kernel/bpf/reuseport_array.c int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key, · line 316

Quoted 9 lines
kernel/bpf/reuseport_array.c +2 -1
@@ -305,7 +309,8 @@ int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,
 	if (err)
 		goto put_file_unlock;
 
-	WRITE_ONCE(nsk->sk_user_data, &array->ptrs[index]);
+	sk_user_data = (uintptr_t)&array->ptrs[index] | SK_USER_DATA_NOCOPY;
+	WRITE_ONCE(nsk->sk_user_data, (void *)sk_user_data);
 	rcu_assign_pointer(array->ptrs[index], nsk);
 	free_osk = osk;
 	err = 0;

Thanks for fixing this before I got around to it. Now we can use reuseport with sockmap splicing :-)

maintainer
#3 lore raw

Re: [PATCH v2 bpf 1/2] bpf: net: Avoid copying sk_user_data of reuseport_array during sk_clone

In reply to Martin KaFai Lau
Quoted 5 lines · Martin KaFai Lau

It makes little sense for copying sk_user_data of reuseport_array during sk_clone_lock(). This patch reuses the SK_USER_DATA_NOCOPY bit introduced in commit f1ff5ce2cd5e ("net, sk_msg: Clear sk_user_data pointer on clone if tagged"). It is used to mark the sk_user_data is not supposed to be copied to its clone.

Quoted 7 lines

Although the cloned sk's sk_user_data will not be used/freed in bpf_sk_reuseport_detach(), this change can still allow the cloned sk's sk_user_data to be used by some other means.

Freeing the reuseport_array's sk_user_data does not require a rcu grace period. Thus, the existing rcu_assign_sk_user_data_nocopy() is not used.

nit: Would have been nice though to add a nonrcu API for this nevertheless instead of open coding.

Quoted 2 lines
Fixes 5dc4c4b7d4e8 ("bpf: Introduce BPF_MAP_TYPE_REUSEPORT_SOCKARRAY")
#4 lore raw

Re: [PATCH v2 bpf 1/2] bpf: net: Avoid copying sk_user_data of reuseport_array during sk_clone

Quoted 10 lines · Daniel Borkmann
Quoted 9 lines · Martin KaFai Lau

It makes little sense for copying sk_user_data of reuseport_array during sk_clone_lock(). This patch reuses the SK_USER_DATA_NOCOPY bit introduced in commit f1ff5ce2cd5e ("net, sk_msg: Clear sk_user_data pointer on clone if tagged"). It is used to mark the sk_user_data is not supposed to be copied to its clone.

Although the cloned sk's sk_user_data will not be used/freed in bpf_sk_reuseport_detach(), this change can still allow the cloned sk's sk_user_data to be used by some other means.

Quoted 6 lines
Quoted 3 lines

Freeing the reuseport_array's sk_user_data does not require a rcu grace period. Thus, the existing rcu_assign_sk_user_data_nocopy() is not used.

nit: Would have been nice though to add a nonrcu API for this nevertheless instead of open coding.

Agreed. I will create a follow-up patch to bpf-next later.

I had created (READ|WRITE)_ONCE_SK_USER_DATA() earlier but then noticed there is no use on the READ_ONCE in that particular call, so ditched the idea. I think it does not matter much and should just use READ_ONCE also.

#5 lore raw

PATCH v2 2/2 bpf: net: Avoid incorrect bpf_sk_reuseport_detach call

In reply to Martin KaFai Lau

bpf_sk_reuseport_detach is currently called when sk->sk_user_data is not NULL. It is incorrect because sk->sk_user_data may not be managed by the bpf's reuseport_array. It has been reported in [1] that, the bpf_sk_reuseport_detach() which is called from udp_lib_unhash() has corrupted the sk_user_data managed by l2tp.

This patch solves it by using another bit (defined as SK_USER_DATA_BPF) of the sk_user_data pointer value. It marks that a sk_user_data is managed/owned by BPF.

The patch depends on a PTRMASK introduced in
commit f1ff5ce2cd5e ("net, sk_msg: Clear sk_user_data pointer on clone if tagged").

[ Note: sk->sk_user_data is used by bpf's reuseport_array only when a sk is
added to the bpf's reuseport_array.
i.e. doing setsockopt(SO_REUSEPORT) and having "sk->sk_reuseport == 1" alone will not stop sk->sk_user_data being used by other means. ]

[1]: https://lore.kernel.org/netdev/[email protected]/

Cc James Chapman <[email protected]>
Fixes 5dc4c4b7d4e8 ("bpf: Introduce BPF_MAP_TYPE_REUSEPORT_SOCKARRAY")
---
 include/net/sock.h           | 3 ++-
 kernel/bpf/reuseport_array.c | 5 +++--
 2 files changed, 5 insertions(+), 3 deletions(-)
include/net/sock.h +2 -1
index 3428619faae4..1183507df95b 100644
@@ -533,7 +533,8 @@ enum sk_pacing {
  * be copied.
  */
 #define SK_USER_DATA_NOCOPY	1UL
-#define SK_USER_DATA_PTRMASK	~(SK_USER_DATA_NOCOPY)
+#define SK_USER_DATA_BPF	2UL	/* Managed by BPF */
+#define SK_USER_DATA_PTRMASK	~(SK_USER_DATA_NOCOPY | SK_USER_DATA_BPF)
 
 /**
  * sk_user_data_is_nocopy - Test if sk_user_data pointer must not be copied
kernel/bpf/reuseport_array.c +3 -2
index a95bc8d7e812..cae9d505e04a 100644
@@ -24,7 +24,7 @@ void bpf_sk_reuseport_detach(struct sock *sk)
 
 	write_lock_bh(&sk->sk_callback_lock);
 	sk_user_data = (uintptr_t)sk->sk_user_data;
-	if (sk_user_data) {
+	if (sk_user_data & SK_USER_DATA_BPF) {
 		struct sock __rcu **socks;
 
 		socks = (void *)(sk_user_data & SK_USER_DATA_PTRMASK);
@@ -309,7 +309,8 @@ int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,
 	if (err)
 		goto put_file_unlock;
 
-	sk_user_data = (uintptr_t)&array->ptrs[index] | SK_USER_DATA_NOCOPY;
+	sk_user_data = (uintptr_t)&array->ptrs[index] | SK_USER_DATA_NOCOPY |
+		SK_USER_DATA_BPF;
 	WRITE_ONCE(nsk->sk_user_data, (void *)sk_user_data);
 	rcu_assign_pointer(array->ptrs[index], nsk);
 	free_osk = osk;
Signature · 1 line
2.24.1
Acked-by Tested-by on v2 2/2
#6 lore raw

Re: [PATCH v2 bpf 2/2] bpf: net: Avoid incorrect bpf_sk_reuseport_detach call

Quoted 9 lines · Martin KaFai Lau

bpf_sk_reuseport_detach is currently called when sk->sk_user_data is not NULL. It is incorrect because sk->sk_user_data may not be managed by the bpf's reuseport_array. It has been reported in [1] that, the bpf_sk_reuseport_detach() which is called from udp_lib_unhash() has corrupted the sk_user_data managed by l2tp.

This patch solves it by using another bit (defined as SK_USER_DATA_BPF) of the sk_user_data pointer value. It marks that a sk_user_data is managed/owned by BPF.

I have reservations about using a bit in sk_user_data to indicate ownership of that pointer. But putting that aside, I confirm that the patch fixes the problem.

#7 lore raw

Re: [PATCH v2 bpf 2/2] bpf: net: Avoid incorrect bpf_sk_reuseport_detach call

Quoted 11 lines · James Chapman
Quoted 9 lines · Martin KaFai Lau

bpf_sk_reuseport_detach is currently called when sk->sk_user_data is not NULL. It is incorrect because sk->sk_user_data may not be managed by the bpf's reuseport_array. It has been reported in [1] that, the bpf_sk_reuseport_detach() which is called from udp_lib_unhash() has corrupted the sk_user_data managed by l2tp.

This patch solves it by using another bit (defined as SK_USER_DATA_BPF) of the sk_user_data pointer value. It marks that a sk_user_data is managed/owned by BPF.

Quoted 7 lines

I have reservations about using a bit in sk_user_data to indicate ownership of that pointer. But putting that aside, I confirm that the patch fixes the problem.

Thanks for the test!

One bit of sk_user_data has already been used to indicate SK_USER_DATA_NOCOPY.
I think using another bit is the cleanest fix for the bpf/net branch instead of tracking this somewhere else.

maintainer Applied
#8 lore raw
In reply to Martin KaFai Lau
Quoted 13 lines · Martin KaFai Lau

This set fixes two issues on sk_user_data when a sk is added to a reuseport_array.

The first patch is to avoid the sk_user_data being copied
to a cloned sk. The second patch avoids doing bpf_sk_reuseport_detach() on sk_user_data that is not managed by reuseport_array.

Since the changes are mostly related to bpf reuseport_array, so it is currently tagged as bpf fixes.

v2:
- Avoid ~3UL (Andrii)

Quoted 9 lines

Martin KaFai Lau (2):
bpf: net: Avoid copying sk_user_data of reuseport_array during
sk_clone
bpf: net: Avoid incorrect bpf_sk_reuseport_detach call

  include/net/sock.h           |  3 ++-
  kernel/bpf/reuseport_array.c | 14 ++++++++++----
  2 files changed, 12 insertions(+), 5 deletions(-)

Applied, thanks!