bpf: sockmap fixes
Subsystems (5)
John Fastabend
maintainerv2 cover bpf: sockmap fixes
A set of fixes for sockmap to resolve map/prog not being cleaned up when maps are not cleaned up from the program side.
For this we pull in the ULP infrastructure to hook into the close() hook of the sock layer. This seemed natural because we have additional sockmap features (to add support for TX hooks) that will also use the ULP infrastructure. This allows us to cleanup entries in the map when socks are closed() and avoid trying to get the sk_state_change() hook to fire in all cases.
The second issue resolved here occurs when users don't detach programs. The gist is a refcnt issue resolved by implementing the release callback. See patch for details.
For testing I ran both sample/sockmap and selftests bpf/test_maps.c. Dave Watson ran TLS test suite on v1 version of the patches without the put_module error path change.
v2 changes rebased onto bpf-next with the following small updates
Patch 1/3 Dave Watson pointed out that we should do a module put in case future users use this with another ULP other than sockmap. And module_put is a nop when owner = NULL (sockmap case) so no harm here and code is more robust.
Patch 2/3 Be explicit and set user_visible to false just to avoid any reader confusion.
---
John Fastabend (3):
net: add a UID to use for ULP socket assignment
bpf: sockmap, add sock close() hook to remove socks
bpf: sockmap, fix leaking maps with attached but not detached progs
include/net/tcp.h | 8 ++ kernel/bpf/sockmap.c | 165 ++++++++++++++++++++++++++++---------------------- net/ipv4/tcp_ulp.c | 57 ++++++++++++++++- net/tls/tls_main.c | 2 + 4 files changed, 155 insertions(+), 77 deletions(-)
John Fastabend
maintainerPATCH v2 1/3 net: add a UID to use for ULP socket assignment
Create a UID field and enum that can be used to assign ULPs to sockets. This saves a set of string comparisons if the ULP id is known.
For sockmap, which is added in the next patches, a ULP is used to hook into TCP sockets close state. In this case the ULP being added is done at map insert time and the ULP is known and done on the kernel side. In this case the named lookup is not needed. Because we don't want to expose psock internals to user space socket options a user visible flag is also added. For TLS this is set for BPF it will be cleared.
Alos remove pr_notice, user gets an error code back and should check that rather than rely on logs.
--- include/net/tcp.h | 6 +++++ net/ipv4/tcp_ulp.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++----- net/tls/tls_main.c | 2 ++ 3 files changed, 60 insertions(+), 5 deletions(-)
include/net/tcp.h +6 -0
@@ -1983,6 +1983,10 @@ static inline void tcp_listendrop(const struct sock *sk)#define TCP_ULP_MAX 128#define TCP_ULP_BUF_MAX (TCP_ULP_NAME_MAX*TCP_ULP_MAX) +enum { +TCP_ULP_TLS , +}; +struct tcp_ulp_ops {struct list_head list;
@@ -1991,7 +1995,9 @@ struct tcp_ulp_ops {/* cleanup ulp */ void (*release)(struct sock *sk); +int uid;char name[TCP_ULP_NAME_MAX ]; +bool user_visible;struct module *owner; };int tcp_register_ulp (struct tcp_ulp_ops *type);
net/ipv4/tcp_ulp.c +52 -5
83 lines not on this page · raw message
net/tls/tls_main.c +2 -0
@@ -484,6 +484,8 @@ static int tls_init(struct sock *sk)static struct tcp_ulp_ops tcp_tls_ulp_ops__read_mostly = { .name ="tls" , + .uid =TCP_ULP_TLS , + .user_visible = true, .owner =THIS_MODULE , .init = tls_init, };
John Fastabend
maintainerPATCH v2 2/3 bpf: sockmap, add sock close() hook to remove socks
In reply to John FastabendThe selftests test_maps program was leaving dangling BPF sockmap programs around because not all psock elements were removed from the map. The elements in turn hold a reference on the BPF program they are attached to causing BPF programs to stay open even after test_maps has completed.
The original intent was that sk_state_change() would be called when TCP socks went through TCP_CLOSE state. However, because socks may be in SOCK_DEAD state or the sock may be a listening socket the event is not always triggered.
To resolve this use the ULP infrastructure and register our own proto close() handler. This fixes the above case.
--- include/net/tcp.h | 2 + kernel/bpf/sockmap.c | 146 +++++++++++++++++++++++++++----------------------- 2 files changed, 81 insertions(+), 67 deletions(-)
include/net/tcp.h +2 -0
@@ -1985,6 +1985,7 @@ static inline void tcp_listendrop(const struct sock *sk)enum {TCP_ULP_TLS , +TCP_ULP_BPF , };struct tcp_ulp_ops {
@@ -2003,6 +2004,7 @@ struct tcp_ulp_ops {int tcp_register_ulp (struct tcp_ulp_ops *type);void tcp_unregister_ulp (struct tcp_ulp_ops *type);int tcp_set_ulp (struct sock *sk,const char *name); +int tcp_set_ulp_id (struct sock *sk,const int ulp);void tcp_get_available_ulp (char *buf,size_t len);void tcp_cleanup_ulp (struct sock *sk);
kernel/bpf/sockmap.c +79 -67
206 lines not on this page · raw message
John Fastabend
maintainerRe: [bpf-next PATCH v2 2/3] bpf: sockmap, add sock close() hook to remove socks
Quoted 11 lines · John Fastabend
The selftests
test_mapsprogram was leaving dangling BPF sockmap programs around because not all psock elements were removed from the map. The elements in turn hold a reference on the BPF program they are attached to causing BPF programs to stay open even aftertest_mapshas completed.The original intent was that
sk_state_change()would be called when TCP socks went throughTCP_CLOSEstate. However, because socks may be inSOCK_DEADstate or the sock may be a listening socket the event is not always triggered.
Quoted 7 lines
To resolve this use the ULP infrastructure and register our own proto
close()handler. This fixes the above case.Fixes 174a79ff9515 ("bpf: sockmap with sk redirect support")Reported-by Prashant Bhole <[email protected]>Signed-off-by John Fastabend <[email protected]>---
[...]
v3 will be needed.
Quoted 10 lines
kernel/bpf/sockmap.c +10
-0+void bpf_tcp_close (struct sock *sk,long timeout) +{ +struct smap_psock_map_entry *e, *tmp; +struct smap_psock *psock; +struct sock *osk; + + psock =smap_psock_sk (sk); +if (unlikely (!psock)) +return sk->sk_prot->close (sk, timeout); +
kernel/bpf/sockmap.c static inline struct smap_psock *smap_psock_sk(const struct sock *sk) · line 149
Quoted 10 lines
kernel/bpf/sockmap.c +10
-0+write_lock_bh (&sk->sk_callback_lock); +list_for_each_entry_safe (e, tmp, &psock->maps, list) { + osk =cmpxchg (e->entry, sk,NULL ); +if (osk == sk) { +list_del (&e->list); +smap_release_sock (psock, sk); + } + } +write_unlock_bh (&sk->sk_callback_lock); +return psock->save_close (sk, timeout);
We need this to be in an RCU critical section. Else the release op could free the psock immediately presumably. I've not actually triggered this case, but seems possible. Also the save_close call can not be done inside RCU so we need to cache the value and run it at the end. This is OK because we have the sock lock held.
Probably like this,
void bpf_tcp_close(struct sock *sk, long timeout)
{
struct smap_psock_map_entry *e, *tmp;
struct smap_psock *psock;
struct sock *osk;
void (*close_fun)(struct sock *sk, long timeout);
rcu_read_lock();
psock = smap_psock_sk(sk);
if (unlikely(!psock))
return sk->sk_prot->close(sk, timeout);
/* Although the psock may be destroyed, after RCU grace period, the
* sk will not because we are holding the sock lock here. So we can
* call the original close routine outside the RCU critical section.
*/
close_fun = psock->save_close;
write_lock_bh(&sk->sk_callback_lock);
list_for_each_entry_safe(e, tmp, &psock->maps, list) {
osk = cmpxchg(e->entry, sk, NULL);
if (osk == sk) {
list_del(&e->list);
smap_release_sock(psock, sk);
}
}
write_unlock_bh(&sk->sk_callback_lock);
rcu_read_unlock();
close_fun(sk, timeout);
}John Fastabend
maintainerPATCH v2 3/3 bpf: sockmap, fix leaking maps with attached but not detached progs
In reply to John FastabendWhen a program is attached to a map we increment the program refcnt to ensure that the program is not removed while it is potentially being referenced from sockmap side. However, if this same program also references the map (this is a reasonably common pattern in my programs) then the verifier will also increment the maps refcnt from the verifier. This is to ensure the map doesn't get garbage collected while the program has a reference to it.
So we are left in a state where the map holds the refcnt on the program stopping it from being removed and releasing the map refcnt. And vice versa the program holds a refcnt on the map stopping it from releasing the refcnt on the prog.
All this is fine as long as users detach the program while the map fd is still around. But, if the user omits this detach command we are left with a dangling map we can no longer release.
To resolve this when the map fd is released decrement the program references and remove any reference from the map to the program. This fixes the issue with possibly dangling map and creates a user side API constraint. That is, the map fd must be held open for programs to be attached to a map.
--- kernel/bpf/sockmap.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
kernel/bpf/sockmap.c +14 -5
@@ -598,11 +598,6 @@ static void sock_map_free(struct bpf_map *map) }rcu_read_unlock (); -if (stab->bpf_verdict) -bpf_prog_put (stab->bpf_verdict); -if (stab->bpf_parse) -bpf_prog_put (stab->bpf_parse); -sock_map_remove_complete (stab); }
@@ -878,6 +873,19 @@ static int sock_map_update_elem(struct bpf_map *map,return err; } +static void sock_map_release (struct bpf_map *map,struct file *map_file) +{ + struct bpf_stab *stab = container_of(map, struct bpf_stab, map); +struct bpf_prog *orig; + + orig =xchg (&stab->bpf_parse,NULL ); +if (orig) +bpf_prog_put (orig); + orig =xchg (&stab->bpf_verdict,NULL ); +if (orig) +bpf_prog_put (orig); +} +const struct bpf_map_ops sock_map_ops = { .map_alloc = sock_map_alloc, .map_free = sock_map_free,
@@ -885,6 +893,7 @@ static int sock_map_update_elem(struct bpf_map *map, .map_get_next_key = sock_map_get_next_key, .map_update_elem = sock_map_update_elem, .map_delete_elem = sock_map_delete_elem, + .map_release = sock_map_release, }; BPF_CALL_4(bpf_sock_map_update, struct bpf_sock_ops_kern *, bpf_sock,