bpf: misc performance improvements for cgroup hooks
Subsystems (8)
v3 cover bpf: misc performance improvements for cgroup hooks
First patch tries to remove kzalloc/kfree from getsockopt for the common cases.
Second patch switches cgroup_bpf_enabled to be per-attach to to add only overhead for the cgroup attach types used on the system.
No visible user-side changes.
v3:
- remove extra newline, add comment about sizeof tcp_zerocopy_receive (Martin KaFai Lau)
- add another patch to remove lock_sock overhead from TCP_ZEROCOPY_RECEIVE; technically, this makes patch #1 obsolete, but I'd still prefer to keep it to help with other socket options
v2:
- perf numbers for getsockopt kmalloc reduction (Song Liu)
- (sk) in BPF_CGROUP_PRE_CONNECT_ENABLED (Song Liu)
- 128 -> 64 buffer size, BUILD_BUG_ON (Martin KaFai Lau)
Stanislav Fomichev (3):
bpf: try to avoid kzalloc in cgroup/{s,g}etsockopt
bpf: split cgroup_bpf_enabled per attach type
bpf: remove extra lock_sock for TCP_ZEROCOPY_RECEIVE
include/linux/bpf-cgroup.h | 52 ++++--- include/linux/filter.h | 5 + kernel/bpf/cgroup.c | 132 +++++++++++++++--- net/ipv4/af_inet.c | 9 +- net/ipv4/tcp.c | 2 + net/ipv4/udp.c | 7 +- net/ipv6/af_inet6.c | 9 +- net/ipv6/udp.c | 7 +- .../selftests/bpf/prog_tests/sockopt_sk.c | 22 +++ .../testing/selftests/bpf/progs/sockopt_sk.c | 15 ++ 10 files changed, 205 insertions(+), 55 deletions(-)
Signature · 1 line
2.29.2.729.g45daf8777d-goog
PATCH v3 1/3 bpf: try to avoid kzalloc in cgroup/{s,g}etsockopt
When we attach a bpf program to cgroup/getsockopt any other getsockopt() syscall starts incurring kzalloc/kfree cost. While, in general, it's not an issue, sometimes it is, like in the case of TCP_ZEROCOPY_RECEIVE. TCP_ZEROCOPY_RECEIVE (ab)uses getsockopt system call to implement fastpath for incoming TCP, we don't want to have extra allocations in there.
Let add a small buffer on the stack and use it for small (majority) {s,g}etsockopt values. I've started with 128 bytes to cover the options we care about (TCP_ZEROCOPY_RECEIVE which is 32 bytes currently, with some planned extension to 64).
It seems natural to do the same for setsockopt, but it's a bit more involved when the BPF program modifies the data (where we have to kmalloc). The assumption is that for the majority of setsockopt calls (which are doing pure BPF options or apply policy) this will bring some benefit as well.
Collected some performance numbers using (on a 65k MTU localhost in a VM):
$ perf record -g -- ./tcp_mmap -s -z $ ./tcp_mmap -H ::1 -z $ ... $ perf report --symbol-filter=__cgroup_bpf_run_filter_getsockopt
Without this patch:
4.81% 0.07% tcp_mmap [kernel.kallsyms] [k] __cgroup_bpf_run_filter_>
|
--4.74%--__cgroup_bpf_run_filter_getsockopt
|
|--1.06%--__kmalloc
|
|--0.71%--lock_sock_nested
|
|--0.62%--__might_fault
|
--0.52%--release_sockWith the patch applied:
3.29% 0.07% tcp_mmap [kernel.kallsyms] [k] __cgroup_bpf_run_filter_getsockopt
|
--3.22%--__cgroup_bpf_run_filter_getsockopt
|
|--0.66%--lock_sock_nested
|
|--0.57%--__might_fault
|
--0.56%--release_sockSo it saves about 1% of the system call. Unfortunately, we still get 2-3% of overhead due to another socket lock/unlock :-(
--- include/linux/filter.h | 3 +++ kernel/bpf/cgroup.c | 43 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 45 insertions(+), 1 deletion(-)
include/linux/filter.h +3 -0
@@ -1281,6 +1281,8 @@ struct bpf_sysctl_kern {u64 tmp_reg; }; +#define BPF_SOCKOPT_KERN_BUF_SIZE 64 +struct bpf_sockopt_kern {struct sock *sk;u8 *optval;
@@ -1289,6 +1291,7 @@ struct bpf_sockopt_kern {s32 optname;s32 optlen;s32 retval; +u8 buf[BPF_SOCKOPT_KERN_BUF_SIZE ]; };int copy_bpf_fprog_from_user (struct sock_fprog *dst,sockptr_t src,int len);
kernel/bpf/cgroup.c +42 -1
70 lines not on this page · raw message
Signature · 1 line
2.29.2.729.g45daf8777d-goog
PATCH v3 2/3 bpf: split cgroup_bpf_enabled per attach type
In reply to Stanislav FomichevWhen we attach any cgroup hook, the rest (even if unused/unattached) start to contribute small overhead. In particular, the one we want to avoid is __cgroup_bpf_run_filter_skb which does two redirections to get to the cgroup and pushes/pulls skb.
Let's split cgroup_bpf_enabled to be per-attach to make sure only used attach types trigger.
I've dropped some existing high-level cgroup_bpf_enabled in some places because BPF_PROG_CGROUP_XXX_RUN macros usually have another cgroup_bpf_enabled check.
I also had to copy-paste BPF_CGROUP_RUN_SA_PROG_LOCK for GETPEERNAME/GETSOCKNAME because type for cgroup_bpf_enabled[type] has to be constant and known at compile time.
--- include/linux/bpf-cgroup.h | 36 +++++++++++++++++++----------------- kernel/bpf/cgroup.c | 14 ++++++-------- net/ipv4/af_inet.c | 9 +++++---- net/ipv4/udp.c | 7 +++---- net/ipv6/af_inet6.c | 9 +++++---- net/ipv6/udp.c | 7 +++---- 6 files changed, 41 insertions(+), 41 deletions(-)
include/linux/bpf-cgroup.h +19 -17
@@ -23,8 +23,8 @@ struct ctl_table_header;#ifdef CONFIG_CGROUP_BPF -extern struct static_key_false cgroup_bpf_enabled_key; -#define cgroup_bpf_enabled static_branch_unlikely(&cgroup_bpf_enabled_key) +extern struct static_key_false cgroup_bpf_enabled_key[MAX_BPF_ATTACH_TYPE ]; +#define cgroup_bpf_enabled (type) static_branch_unlikely(&cgroup_bpf_enabled_key[type]) DECLARE_PER_CPU(struct bpf_cgroup_storage*, bpf_cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE ]);
@@ -185,7 +185,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,#define BPF_CGROUP_RUN_PROG_INET_INGRESS (sk, skb) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled) \ + if (cgroup_bpf_enabled(BPF_CGROUP_INET_INGRESS)) \ __ret = __cgroup_bpf_run_filter_skb(sk, skb, \ BPF_CGROUP_INET_INGRESS); \ \
@@ -195,7 +195,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,#define BPF_CGROUP_RUN_PROG_INET_EGRESS (sk, skb) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled && sk && sk == skb->sk) { \ + if (cgroup_bpf_enabled(BPF_CGROUP_INET_EGRESS) && sk && sk == skb->sk) { \ typeof(sk) __sk = sk_to_full_sk(sk); \ if (sk_fullsock(__sk)) \ __ret = __cgroup_bpf_run_filter_skb(__sk, skb, \
@@ -207,7 +207,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,#define BPF_CGROUP_RUN_SK_PROG (sk, type) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled) { \ + if (cgroup_bpf_enabled(type)) { \ __ret = __cgroup_bpf_run_filter_sk(sk, type); \ } \ __ret; \
@@ -228,7 +228,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,#define BPF_CGROUP_RUN_SA_PROG (sk, uaddr, type) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled) \ + if (cgroup_bpf_enabled(type)) \ __ret = __cgroup_bpf_run_filter_sock_addr(sk, uaddr, type, \ NULL); \ __ret; \
@@ -237,7 +237,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,#define BPF_CGROUP_RUN_SA_PROG_LOCK (sk, uaddr, type, t_ctx) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled) { \ + if (cgroup_bpf_enabled(type)) { \ lock_sock(sk); \ __ret = __cgroup_bpf_run_filter_sock_addr(sk, uaddr, type, \ t_ctx); \
@@ -252,8 +252,10 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,#define BPF_CGROUP_RUN_PROG_INET6_BIND_LOCK (sk, uaddr) \ BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, BPF_CGROUP_INET6_BIND, NULL) -#define BPF_CGROUP_PRE_CONNECT_ENABLED (sk) (cgroup_bpf_enabled && \ - sk->sk_prot->pre_connect) +#define BPF_CGROUP_PRE_CONNECT_ENABLED (sk) \ + ((cgroup_bpf_enabled(BPF_CGROUP_INET4_CONNECT) || \ + cgroup_bpf_enabled(BPF_CGROUP_INET6_CONNECT)) && \ + (sk)->sk_prot->pre_connect)#define BPF_CGROUP_RUN_PROG_INET4_CONNECT (sk, uaddr) \ BPF_CGROUP_RUN_SA_PROG(sk, uaddr, BPF_CGROUP_INET4_CONNECT)
@@ -297,7 +299,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,#define BPF_CGROUP_RUN_PROG_SOCK_OPS_SK (sock_ops, sk) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled) \ + if (cgroup_bpf_enabled(BPF_CGROUP_SOCK_OPS)) \ __ret = __cgroup_bpf_run_filter_sock_ops(sk, \ sock_ops, \ BPF_CGROUP_SOCK_OPS); \
@@ -307,7 +309,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,#define BPF_CGROUP_RUN_PROG_SOCK_OPS (sock_ops) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled && (sock_ops)->sk) { \ + if (cgroup_bpf_enabled(BPF_CGROUP_SOCK_OPS) && (sock_ops)->sk) { \ typeof(sk) __sk = sk_to_full_sk((sock_ops)->sk); \ if (__sk && sk_fullsock(__sk)) \ __ret = __cgroup_bpf_run_filter_sock_ops(__sk, \
@@ -320,7 +322,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,#define BPF_CGROUP_RUN_PROG_DEVICE_CGROUP (type, major, minor, access) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled) \ + if (cgroup_bpf_enabled(BPF_CGROUP_DEVICE)) \ __ret = __cgroup_bpf_check_dev_permission(type, major, minor, \ access, \ BPF_CGROUP_DEVICE); \
@@ -332,7 +334,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,#define BPF_CGROUP_RUN_PROG_SYSCTL (head, table, write, buf, count, pos) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled) \ + if (cgroup_bpf_enabled(BPF_CGROUP_SYSCTL)) \ __ret = __cgroup_bpf_run_filter_sysctl(head, table, write, \ buf, count, pos, \ BPF_CGROUP_SYSCTL); \
@@ -343,7 +345,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key, kernel_optval) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled) \ + if (cgroup_bpf_enabled(BPF_CGROUP_SETSOCKOPT)) \ __ret = __cgroup_bpf_run_filter_setsockopt(sock, level, \ optname, optval, \ optlen, \
@@ -354,7 +356,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key,#define BPF_CGROUP_GETSOCKOPT_MAX_OPTLEN (optlen) \ ({ \ int __ret = 0; \ - if (cgroup_bpf_enabled) \ + if (cgroup_bpf_enabled(BPF_CGROUP_GETSOCKOPT)) \ get_user(__ret, optlen); \ __ret; \ })
@@ -363,7 +365,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key, max_optlen, retval) \ ({ \ int __ret = retval; \ - if (cgroup_bpf_enabled) \ + if (cgroup_bpf_enabled(BPF_CGROUP_GETSOCKOPT)) \ __ret = __cgroup_bpf_run_filter_getsockopt(sock, level, \ optname, optval, \ optlen, max_optlen, \
@@ -427,7 +429,7 @@ static inline int bpf_percpu_cgroup_storage_update(struct bpf_map *map,return 0 ; } -#define cgroup_bpf_enabled (0) +#define cgroup_bpf_enabled (type) (0)#define BPF_CGROUP_RUN_SA_PROG_LOCK (sk, uaddr, type, t_ctx) ({ 0; })#define BPF_CGROUP_PRE_CONNECT_ENABLED (sk) (0)#define BPF_CGROUP_RUN_PROG_INET_INGRESS (sk,skb) ({ 0; })
kernel/bpf/cgroup.c +6 -8
@@ -20,7 +20,7 @@#include "../cgroup/cgroup-internal.h" -DEFINE_STATIC_KEY_FALSE (cgroup_bpf_enabled_key); +DEFINE_STATIC_KEY_ARRAY_FALSE (cgroup_bpf_enabled_key,MAX_BPF_ATTACH_TYPE );EXPORT_SYMBOL (cgroup_bpf_enabled_key);void cgroup_bpf_offline (struct cgroup *cgrp)
@@ -129,7 +129,7 @@ static void cgroup_bpf_release(struct work_struct *work)if (pl->link)bpf_cgroup_link_auto_detach (pl->link);kfree (pl); -static_branch_dec (&cgroup_bpf_enabled_key); +static_branch_dec (&cgroup_bpf_enabled_key[type]); } old_array = rcu_dereference_protected( cgrp->bpf.effective[type],
@@ -500,7 +500,7 @@ int __cgroup_bpf_attach(struct cgroup *cgrp,if (old_prog)bpf_prog_put (old_prog);else -static_branch_inc (&cgroup_bpf_enabled_key); +static_branch_inc (&cgroup_bpf_enabled_key[type]);bpf_cgroup_storages_link (new_storage, cgrp, type);return 0 ;
@@ -699,7 +699,7 @@ int __cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog, cgrp->bpf.flags[type] =0 ;if (old_prog)bpf_prog_put (old_prog); -static_branch_dec (&cgroup_bpf_enabled_key); +static_branch_dec (&cgroup_bpf_enabled_key[type]);return 0 ; cleanup:
@@ -1364,8 +1364,7 @@ int __cgroup_bpf_run_filter_setsockopt(struct sock *sk, int *level,* attached to the hook so we don't waste time allocating * memory and locking the socket. */ -if (!cgroup_bpf_enabled || -__cgroup_bpf_prog_array_is_empty (cgrp,BPF_CGROUP_SETSOCKOPT )) +if (__cgroup_bpf_prog_array_is_empty (cgrp,BPF_CGROUP_SETSOCKOPT ))return 0 ;/* Allocate a bit more than the initial user buffer for
@@ -1459,8 +1458,7 @@ int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level,* attached to the hook so we don't waste time allocating * memory and locking the socket. */ -if (!cgroup_bpf_enabled || -__cgroup_bpf_prog_array_is_empty (cgrp,BPF_CGROUP_GETSOCKOPT )) +if (__cgroup_bpf_prog_array_is_empty (cgrp,BPF_CGROUP_GETSOCKOPT ))return retval; ctx.optlen = max_optlen;
net/ipv4/af_inet.c +5 -4
@@ -777,18 +777,19 @@ int inet_getname(struct socket *sock, struct sockaddr *uaddr,return -ENOTCONN ; sin->sin_port = inet->inet_dport; sin->sin_addr.s_addr = inet->inet_daddr; +BPF_CGROUP_RUN_SA_PROG_LOCK (sk, (struct sockaddr *)sin, +BPF_CGROUP_INET4_GETPEERNAME , +NULL ); }else {__be32 addr = inet->inet_rcv_saddr;if (!addr) addr = inet->inet_saddr; sin->sin_port = inet->inet_sport; sin->sin_addr.s_addr = addr; - } -if (cgroup_bpf_enabled)BPF_CGROUP_RUN_SA_PROG_LOCK (sk, (struct sockaddr *)sin, - peer ?BPF_CGROUP_INET4_GETPEERNAME : -BPF_CGROUP_INET4_GETSOCKNAME , +BPF_CGROUP_INET4_GETSOCKNAME ,NULL ); + }memset (sin->sin_zero,0 ,sizeof (sin->sin_zero));return sizeof (*sin); }
net/ipv4/udp.c +3 -4
@@ -1124,7 +1124,7 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)rcu_read_unlock (); } -if (cgroup_bpf_enabled && !connected) { +if (cgroup_bpf_enabled (BPF_CGROUP_UDP4_SENDMSG ) && !connected) { err =BPF_CGROUP_RUN_PROG_UDP4_SENDMSG_LOCK (sk, (struct sockaddr *)usin, &ipc.addr);if (err)
@@ -1858,9 +1858,8 @@ int udp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int noblock,memset (sin->sin_zero,0 ,sizeof (sin->sin_zero)); *addr_len =sizeof (*sin); -if (cgroup_bpf_enabled) -BPF_CGROUP_RUN_PROG_UDP4_RECVMSG_LOCK (sk, - (struct sockaddr *)sin); +BPF_CGROUP_RUN_PROG_UDP4_RECVMSG_LOCK (sk, + (struct sockaddr *)sin); }if (udp_sk (sk)->gro_enabled)
net/ipv6/af_inet6.c +5 -4
@@ -527,18 +527,19 @@ int inet6_getname(struct socket *sock, struct sockaddr *uaddr, sin->sin6_addr = sk->sk_v6_daddr;if (np->sndflow) sin->sin6_flowinfo = np->flow_label; +BPF_CGROUP_RUN_SA_PROG_LOCK (sk, (struct sockaddr *)sin, +BPF_CGROUP_INET6_GETPEERNAME , +NULL ); }else {if (ipv6_addr_any (&sk->sk_v6_rcv_saddr)) sin->sin6_addr = np->saddr;else sin->sin6_addr = sk->sk_v6_rcv_saddr; sin->sin6_port = inet->inet_sport; - } -if (cgroup_bpf_enabled)BPF_CGROUP_RUN_SA_PROG_LOCK (sk, (struct sockaddr *)sin, - peer ?BPF_CGROUP_INET6_GETPEERNAME : -BPF_CGROUP_INET6_GETSOCKNAME , +BPF_CGROUP_INET6_GETSOCKNAME ,NULL ); + } sin->sin6_scope_id =ipv6_iface_scope_id (&sin->sin6_addr, sk->sk_bound_dev_if);return sizeof (*sin);
net/ipv6/udp.c +3 -4
@@ -409,9 +409,8 @@ int udpv6_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, } *addr_len =sizeof (*sin6); -if (cgroup_bpf_enabled) -BPF_CGROUP_RUN_PROG_UDP6_RECVMSG_LOCK (sk, - (struct sockaddr *)sin6); +BPF_CGROUP_RUN_PROG_UDP6_RECVMSG_LOCK (sk, + (struct sockaddr *)sin6); }if (udp_sk (sk)->gro_enabled)
@@ -1462,7 +1461,7 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) fl6.saddr = np->saddr; fl6.fl6_sport = inet->inet_sport; -if (cgroup_bpf_enabled && !connected) { +if (cgroup_bpf_enabled (BPF_CGROUP_UDP6_SENDMSG ) && !connected) { err =BPF_CGROUP_RUN_PROG_UDP6_SENDMSG_LOCK (sk, (struct sockaddr *)sin6, &fl6.saddr);if (err)
Signature · 1 line
2.29.2.729.g45daf8777d-goog
PATCH v3 3/3 bpf: remove extra lock_sock for TCP_ZEROCOPY_RECEIVE
In reply to Stanislav FomichevAdd custom implementation of getsockopt hook for TCP_ZEROCOPY_RECEIVE. We skip generic hooks for TCP_ZEROCOPY_RECEIVE and have a custom call in do_tcp_getsockopt using the on-stack data. This removes 3% overhead for locking/unlocking the socket.
Also:
- Removed BUILD_BUG_ON (zerocopy doesn't depend on the buf size anymore)
- Separated on-stack buffer into bpf_sockopt_buf and downsized to 32 bytes (let's keep it to help with the other options)
(I can probably split this patch into two: add new features and rework
bpf_sockopt_buf; can follow up if the approach in general sounds good).
Without this patch:
3.29% 0.07% tcp_mmap [kernel.kallsyms] [k] __cgroup_bpf_run_filter_getsockopt
|
--3.22%--__cgroup_bpf_run_filter_getsockopt
|
|--0.66%--lock_sock_nested
|
|--0.57%--__might_fault
|
--0.56%--release_sockWith the patch applied:
0.42% 0.10% tcp_mmap [kernel.kallsyms] [k] __cgroup_bpf_run_filter_getsockopt_kern
0.02% 0.02% tcp_mmap [kernel.kallsyms] [k] __cgroup_bpf_run_filter_getsockopt--- include/linux/bpf-cgroup.h | 16 +++ include/linux/filter.h | 6 +- kernel/bpf/cgroup.c | 103 +++++++++++++----- net/ipv4/tcp.c | 2 + .../selftests/bpf/prog_tests/sockopt_sk.c | 22 ++++ .../testing/selftests/bpf/progs/sockopt_sk.c | 15 +++ 6 files changed, 135 insertions(+), 29 deletions(-)
include/linux/bpf-cgroup.h +16 -0
@@ -147,6 +147,10 @@ int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level, int __user *optlen, int max_optlen, int retval); +int __cgroup_bpf_run_filter_getsockopt_kern (struct sock *sk,int level, +int optname,void *optval, +int *optlen,int retval); +static inline enum bpf_cgroup_storage_type cgroup_storage_type (struct bpf_map *map) {
@@ -373,6 +377,16 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key, __ret; \ }) +#define BPF_CGROUP_RUN_PROG_GETSOCKOPT_KERN (sock, level, optname, optval, \ + optlen, retval) \ +({ \ + int __ret = retval; \ + if (cgroup_bpf_enabled(BPF_CGROUP_GETSOCKOPT)) \ + __ret = __cgroup_bpf_run_filter_getsockopt_kern( \ + sock, level, optname, optval, optlen, retval); \ + __ret; \ +}) +int cgroup_bpf_prog_attach (const union bpf_attr *attr,enum bpf_prog_type ptype,struct bpf_prog *prog); int cgroup_bpf_prog_detach(const union bpf_attr *attr,
@@ -454,6 +468,8 @@ static inline int bpf_percpu_cgroup_storage_update(struct bpf_map *map,#define BPF_CGROUP_GETSOCKOPT_MAX_OPTLEN (optlen) ({ 0; })#define BPF_CGROUP_RUN_PROG_GETSOCKOPT (sock, level, optname, optval, \ optlen, max_optlen, retval) ({ retval; }) +#define BPF_CGROUP_RUN_PROG_GETSOCKOPT_KERN (sock, level, optname, optval, \ + optlen, retval) ({ retval; })#define BPF_CGROUP_RUN_PROG_SETSOCKOPT (sock, level, optname, optval, optlen, \ kernel_optval) ({ 0; })
include/linux/filter.h +4 -2
@@ -1281,7 +1281,10 @@ struct bpf_sysctl_kern {u64 tmp_reg; }; -#define BPF_SOCKOPT_KERN_BUF_SIZE 64 +#define BPF_SOCKOPT_KERN_BUF_SIZE 32 +struct bpf_sockopt_buf { +u8 data[BPF_SOCKOPT_KERN_BUF_SIZE ]; +};struct bpf_sockopt_kern {struct sock *sk;
@@ -1291,7 +1294,6 @@ struct bpf_sockopt_kern {s32 optname;s32 optlen;s32 retval; -u8 buf[BPF_SOCKOPT_KERN_BUF_SIZE ]; };int copy_bpf_fprog_from_user (struct sock_fprog *dst,sockptr_t src,int len);
kernel/bpf/cgroup.c +76 -27
182 lines not on this page · raw message
net/ipv4/tcp.c +2 -0
@@ -4098,6 +4098,8 @@ static int do_tcp_getsockopt(struct sock *sk, int level,return -EFAULT ;lock_sock (sk); err =tcp_zerocopy_receive (sk, &zc); + err =BPF_CGROUP_RUN_PROG_GETSOCKOPT_KERN (sk, level, optname, + &zc, &len, err);release_sock (sk); if (len >= offsetofend(struct tcp_zerocopy_receive, err)) goto zerocopy_rcv_sk_err;
tools/testing/selftests/bpf/prog_tests/sockopt_sk.c +22 -0
@@ -11,6 +11,7 @@ static int getsetsockopt(void)char u8[4 ];__u32 u32;char cc[16 ];/* TCP_CA_NAME_MAX */ +struct tcp_zerocopy_receive zc; } buf = {};socklen_t optlen;char *big_buf =NULL ;
@@ -154,6 +155,27 @@ static int getsetsockopt(void) goto err; } +/* TCP_ZEROCOPY_RECEIVE triggers */ +memset (&buf,0 ,sizeof (buf)); + optlen =sizeof (buf.zc); + err =getsockopt (fd,SOL_TCP ,TCP_ZEROCOPY_RECEIVE , &buf, &optlen); +if (err) { +log_err ("Unexpected getsockopt(TCP_ZEROCOPY_RECEIVE) err=%d errno=%d" , + err, errno); + goto err; + } + +memset (&buf,0 ,sizeof (buf)); + buf.zc.address =12345 ;/* rejected by BPF */ + optlen =sizeof (buf.zc); + errno =0 ; + err =getsockopt (fd,SOL_TCP ,TCP_ZEROCOPY_RECEIVE , &buf, &optlen); +if (errno !=EPERM ) { +log_err ("Unexpected getsockopt(TCP_ZEROCOPY_RECEIVE) err=%d errno=%d" , + err, errno); + goto err; + } +free (big_buf);close (fd);return 0 ;
tools/testing/selftests/bpf/progs/sockopt_sk.c +15 -0
@@ -57,6 +57,21 @@ int _getsockopt(struct bpf_sockopt *ctx)return 1 ; } +if (ctx->level ==SOL_TCP && ctx->optname ==TCP_ZEROCOPY_RECEIVE ) { +/* Verify that TCP_ZEROCOPY_RECEIVE triggers. +* It has a custom implementation for performance +* reasons. +*/ + +if (optval +sizeof (struct tcp_zerocopy_receive ) > optval_end) +return 0 ;/* EPERM, bounds check */ + +if (((struct tcp_zerocopy_receive *)optval)->address !=0 ) +return 0 ;/* EPERM, unexpected data */ + +return 1 ; + } +if (ctx->level ==SOL_IP && ctx->optname ==IP_FREEBIND ) {if (optval +1 > optval_end)return 0 ;/* EPERM, bounds check */
Signature · 1 line
2.29.2.729.g45daf8777d-goog
Re: [PATCH bpf-next v3 3/3] bpf: remove extra lock_sock for TCP_ZEROCOPY_RECEIVE
Quoted 14 lines · Stanislav Fomichev
Add custom implementation of getsockopt hook for
TCP_ZEROCOPY_RECEIVE. We skip generic hooks forTCP_ZEROCOPY_RECEIVEand have a custom call indo_tcp_getsockoptusing the on-stack data. This removes 3% overhead for locking/unlocking the socket.Also:
- RemovedBUILD_BUG_ON(zerocopy doesn't depend on the buf size anymore)
- Separated on-stack buffer intobpf_sockopt_bufand downsized to 32 bytes (let's keep it to help with the other options)(I can probably split this patch into two: add new features and rework
bpf_sockopt_buf; can follow up if the approach in general sounds good).
Quoted 6 lines
Without this patch:
3.29% 0.07% tcp_mmap [kernel.kallsyms] [k] __cgroup_bpf_run_filter_getsockopt | --3.22%--__cgroup_bpf_run_filter_getsockopt | |--0.66%--lock_sock_nested
A general question for sockopt prog, why the BPF_CGROUP_(GET|SET)SOCKOPT prog has to run under lock_sock()?
Quoted 9 lines
| |--0.57%--__might_fault | --0.56%--release_sockWith the patch applied:
0.42% 0.10% tcp_mmap [kernel.kallsyms] [k] __cgroup_bpf_run_filter_getsockopt_kern 0.02% 0.02% tcp_mmap [kernel.kallsyms] [k] __cgroup_bpf_run_filter_getsockopt
[ ... ]
Quoted 14 lines
kernel/bpf/cgroup.c +3
-7@@ -1445,15 +1442,29 @@ int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level, int __user *optlen, int max_optlen, int retval) { - struct cgroup *cgrp = sock_cgroup_ptr(&sk->sk_cgrp_data); -struct bpf_sockopt_kern ctx = { - .sk = sk, - .level = level, - .optname = optname, - .retval = retval, - }; + struct bpf_sockopt_kern ctx; +struct bpf_sockopt_buf buf; +struct cgroup *cgrp;
kernel/bpf/cgroup.c int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level, · line 1457
Quoted 10 lines
kernel/bpf/cgroup.c +8
-0int ret; +#ifdef CONFIG_INET +/* TCP do_tcp_getsockopt has optimized getsockopt implementation +* to avoid extra socket lock for TCP_ZEROCOPY_RECEIVE. +*/ +if (sk->sk_prot->getsockopt == tcp_getsockopt && + level ==SOL_TCP && optname ==TCP_ZEROCOPY_RECEIVE ) +return retval; +#endif
That seems too much protocol details and not very scalable.
It is not very related to kernel/bpf/cgroup.c which has very little idea whether a specific protocol has optimized things in some ways (e.g. by directly calling cgroup's bpf prog at some strategic places in this patch). Lets see if it can be done better.
At least, these protocol checks belong to the net's socket.c more than the bpf's cgroup.c here. If it also looks like layering breakage in socket.c, may be adding a signal in sk_prot (for example) to tell if the sk_prot->getsockopt has already called the cgroup's bpf prog? (e.g. tcp_getsockopt() can directly call the cgroup's bpf for all optname instead of only TCP_ZEROCOPY_RECEIVE).
For example:
int __sys_getsockopt(...)
{
/* ... */
if (!sk_prot->bpf_getsockopt_handled)
BPF_CGROUP_RUN_PROG_GETSOCKOPT(...);
}Thoughts?
Quoted 13 lines
kernel/bpf/cgroup.c +10
-0+ +memset (&buf,0 ,sizeof (buf)); +memset (&ctx,0 ,sizeof (ctx)); + + cgrp =sock_cgroup_ptr (&sk->sk_cgrp_data); + ctx.sk = sk; + ctx.level = level; + ctx.optname = optname; + ctx.retval = retval; +/* Opportunistic check to see whether we have any BPF program * attached to the hook so we don't waste time allocating * memory and locking the socket.
Re: [PATCH bpf-next v3 3/3] bpf: remove extra lock_sock for TCP_ZEROCOPY_RECEIVE
Quoted 18 lines · Martin KaFai Lau
Quoted 8 lines · Stanislav Fomichev
Add custom implementation of getsockopt hook for
TCP_ZEROCOPY_RECEIVE. We skip generic hooks forTCP_ZEROCOPY_RECEIVEand have a custom call indo_tcp_getsockoptusing the on-stack data. This removes 3% overhead for locking/unlocking the socket.Also:
- RemovedBUILD_BUG_ON(zerocopy doesn't depend on the buf size anymore)
- Separated on-stack buffer intobpf_sockopt_bufand downsized to 32bytes
Quoted 8 lines
(let's keep it to help with the other options)
(I can probably split this patch into two: add new features and rework
bpf_sockopt_buf; can follow up if the approach in general sounds good).Without this patch:
3.29% 0.07% tcp_mmap [kernel.kallsyms] [k]
Quoted 7 lines
__cgroup_bpf_run_filter_getsockoptQuoted 4 lines
| --3.22%--__cgroup_bpf_run_filter_getsockopt | |--0.66%--lock_sock_nestedA general question for sockopt prog, why the BPF_CGROUP_(GET|SET)SOCKOPT prog has to run under
lock_sock()?
I don't think there is a strong reason. We expose sk to the BPF program, but mainly for the socket storage map (which, afaik, doesn't require socket to be locked). OTOH, it seems that providing a consistent view of the sk to the BPF is a good idea.
Eric has suggested to try to use fast socket lock. It helps a bit, but it doesn't remove the issue completely because
we do a bunch of copy_{to,from}_user in the generic __cgroup_bpf_run_filter_getsockopt as well :-(
Quoted 54 lines
Quoted 7 lines
| |--0.57%--__might_fault | --0.56%--release_sockWith the patch applied:
0.42% 0.10% tcp_mmap [kernel.kallsyms] [k]
__cgroup_bpf_run_filter_getsockopt_kernQuoted 1 line
0.02% 0.02% tcp_mmap [kernel.kallsyms] [k]
__cgroup_bpf_run_filter_getsockoptQuoted 1 line
[ ... ]
Review int __cgroup_bpf_run_filter_getsockopt(struct
Quoted 1 line
(quoted diff) +0
-0@@ -1445,15 +1442,29 @@ int __cgroup_bpf_run_filter_getsockopt(structsock *sk, int level,
Quoted 13 lines
kernel/bpf/cgroup.c +3
-7int __user *optlen, int max_optlen, int retval) { - struct cgroup *cgrp = sock_cgroup_ptr(&sk->sk_cgrp_data); -struct bpf_sockopt_kern ctx = { - .sk = sk, - .level = level, - .optname = optname, - .retval = retval, - }; + struct bpf_sockopt_kern ctx; +struct bpf_sockopt_buf buf; +struct cgroup *cgrp;kernel/bpf/cgroup.c int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level, · line 1457
Quoted 10 lines
kernel/bpf/cgroup.c +8
-0int ret; +#ifdef CONFIG_INET +/* TCP do_tcp_getsockopt has optimized getsockopt implementation +* to avoid extra socket lock for TCP_ZEROCOPY_RECEIVE. +*/ +if (sk->sk_prot->getsockopt == tcp_getsockopt && + level ==SOL_TCP && optname ==TCP_ZEROCOPY_RECEIVE ) +return retval; +#endif That seems too much protocol details and not very scalable.
It is not very related tokernel/bpf/cgroup.cwhich has very little idea whether a specific protocol has optimized things in some ways (e.g. by directly calling cgroup's bpf prog at some strategic places in this patch). Lets see if it can be done better.At least, these protocol checks belong to the net's
socket.cmore than the bpf'scgroup.chere. If it also looks like layering breakage insocket.c, may be adding a signal insk_prot(for example) to tell if thesk_prot->getsockopthas already called the cgroup's bpf prog? (e.g.tcp_getsockopt()can directly call the cgroup's bpf for all optname instead of onlyTCP_ZEROCOPY_RECEIVE).For example:
int
__sys_getsockopt(...)
Quoted 8 lines
{ /* ... */ if (!sk_prot->bpf_getsockopt_handled) BPF_CGROUP_RUN_PROG_GETSOCKOPT(...); }Thoughts?
Sounds good. I didn't go that far because I don't expect there to be a lot of special cases like that. But it might be worth supporting it in a generic way from the beginning.
I was thinking about something simpler:
int __cgroup_bpf_run_filter_getsockopt(sk, ...)
{
if (sk->sk_prot->bypass_bpf_getsockopt(level, optlen)) {
return retval;
} // ...
}
Not sure it's worth exposing it to the __sys_getsockopt. WDYT?
Re: [PATCH bpf-next v3 3/3] bpf: remove extra lock_sock for TCP_ZEROCOPY_RECEIVE
Quoted 24 lines · On Wed, Jan 06, 2021 at 02:45:56PM -0800, [email protected] wrote:
Quoted 23 lines · Martin KaFai Lau
Quoted 8 lines · Stanislav Fomichev
Add custom implementation of getsockopt hook for
TCP_ZEROCOPY_RECEIVE. We skip generic hooks forTCP_ZEROCOPY_RECEIVEand have a custom call indo_tcp_getsockoptusing the on-stack data. This removes 3% overhead for locking/unlocking the socket.Also:
- RemovedBUILD_BUG_ON(zerocopy doesn't depend on the buf size anymore)
- Separated on-stack buffer intobpf_sockopt_bufand downsized to 32bytes
Quoted 8 lines
(let's keep it to help with the other options)
(I can probably split this patch into two: add new features and rework
bpf_sockopt_buf; can follow up if the approach in general sounds good).Without this patch:
3.29% 0.07% tcp_mmap [kernel.kallsyms] [k]
__cgroup_bpf_run_filter_getsockoptQuoted 4 lines
| --3.22%--__cgroup_bpf_run_filter_getsockopt | |--0.66%--lock_sock_nested
Quoted 7 lines
Quoted 3 lines
A general question for sockopt prog, why the BPF_CGROUP_(GET|SET)SOCKOPT prog
has to run underlock_sock()?I don't think there is a strong reason. We expose sk to the BPF program, but mainly for the socket storage map (which, afaik, doesn't require socket to be locked). OTOH, it seems that providing a consistent view of the sk to the BPF is a good idea.
hmm... most of the bpf prog also does not require a locked sock. For example, the __sk_buff->sk. If a bpf prog needs a locked view of sk, a more generic solution is desired. Anyhow, I guess the train has sort of sailed for sockopt bpf.
Quoted 8 lines
Eric has suggested to try to use fast socket lock. It helps a bit, but it doesn't remove the issue completely because
we do a bunch of copy_{to,from}_user in the generic__cgroup_bpf_run_filter_getsockoptas well :-(Quoted 2 lines
Quoted 2 lines
| |--0.57%--__might_fault
Is it a debug kernel?
Quoted 66 lines
Quoted 53 lines
Quoted 5 lines
| --0.56%--release_sockWith the patch applied:
0.42% 0.10% tcp_mmap [kernel.kallsyms] [k]
__cgroup_bpf_run_filter_getsockopt_kernQuoted 1 line
0.02% 0.02% tcp_mmap [kernel.kallsyms] [k]
__cgroup_bpf_run_filter_getsockoptQuoted 1 line
[ ... ]
Review int __cgroup_bpf_run_filter_getsockopt(struct
Quoted 1 line
(quoted diff) +0
-0@@ -1445,15 +1442,29 @@ int __cgroup_bpf_run_filter_getsockopt(structsock *sk, int level,
Quoted 13 lines
kernel/bpf/cgroup.c +3
-7int __user *optlen, int max_optlen, int retval) { - struct cgroup *cgrp = sock_cgroup_ptr(&sk->sk_cgrp_data); -struct bpf_sockopt_kern ctx = { - .sk = sk, - .level = level, - .optname = optname, - .retval = retval, - }; + struct bpf_sockopt_kern ctx; +struct bpf_sockopt_buf buf; +struct cgroup *cgrp;kernel/bpf/cgroup.c int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level, · line 1457
Quoted 10 lines
kernel/bpf/cgroup.c +8
-0int ret; +#ifdef CONFIG_INET +/* TCP do_tcp_getsockopt has optimized getsockopt implementation +* to avoid extra socket lock for TCP_ZEROCOPY_RECEIVE. +*/ +if (sk->sk_prot->getsockopt == tcp_getsockopt && + level ==SOL_TCP && optname ==TCP_ZEROCOPY_RECEIVE ) +return retval; +#endif That seems too much protocol details and not very scalable. It is not very related to
kernel/bpf/cgroup.cwhich has very little idea whether a specific protocol has optimized things in some ways (e.g. by directly calling cgroup's bpf prog at some strategic places in this patch).
Lets see if it can be done better.At least, these protocol checks belong to the net's
socket.cmore than the bpf'scgroup.chere. If it also looks like layering breakage insocket.c, may be adding a signal insk_prot(for example) to tell if thesk_prot->getsockopthas already called the cgroup's bpf prog? (e.g.tcp_getsockopt()can directly call the cgroup's bpf for all optname instead of onlyTCP_ZEROCOPY_RECEIVE).For example:
int
__sys_getsockopt(...)Quoted 8 lines
{ /* ... */ if (!sk_prot->bpf_getsockopt_handled) BPF_CGROUP_RUN_PROG_GETSOCKOPT(...); }Thoughts?
Sounds good. I didn't go that far because I don't expect there to be a lot of special cases like that. But it might be worth supporting it in a generic way from the beginning.
Quoted 5 lines
I was thinking about something simpler:
int
__cgroup_bpf_run_filter_getsockopt(sk, ...){ if (sk->sk_prot->bypass_bpf_getsockopt(level, optlen)) {
I think it meant s/optlen/optname/ which is not __user. Yeah, I think that can provide a more generic solution and also abstract things away.
Please add a details comment in this function.
Quoted 7 lines
return retval; }// ...
}Not sure it's worth exposing it to the
__sys_getsockopt. WDYT?
or call that in BPF_CGROUP_RUN_PROG_GETSOCKOPT(). then the changes in __cgroup_bpf_run_filter_getsockopt() in this patch should go away?
Re: [PATCH bpf-next v3 3/3] bpf: remove extra lock_sock for TCP_ZEROCOPY_RECEIVE
Quoted 42 lines · Martin KaFai Lau
Quoted 3 lines · On Wed, Jan 06, 2021 at 02:45:56PM -0800, [email protected] wrote:
Quoted 2 lines · Martin KaFai Lau
Quoted 1 line · Stanislav Fomichev
Add custom implementation of getsockopt hook for
TCP_ZEROCOPY_RECEIVE.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
We skip generic hooks for
TCP_ZEROCOPY_RECEIVEand have a custom call indo_tcp_getsockoptusing the on-stack data. This removes 3% overhead for locking/unlocking the socket.Also:
- RemovedBUILD_BUG_ON(zerocopy doesn't depend on the buf sizeanymore)
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
- Separated on-stack buffer into
bpf_sockopt_bufand downsized to 32bytes
Quoted 3 lines
(let's keep it to help with the other options)
(I can probably split this patch into two: add new features and
rework
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
bpf_sockopt_buf; can follow up if the approach in general sounds good).Without this patch:
3.29% 0.07% tcp_mmap [kernel.kallsyms] [k]
__cgroup_bpf_run_filter_getsockoptQuoted 4 lines
| --3.22%--__cgroup_bpf_run_filter_getsockopt | |--0.66%--lock_sock_nestedA general question for sockopt prog, why the BPF_CGROUP_(GET|
SET)SOCKOPT
Quoted 6 lines
Quoted 2 lines
prog
has to run underlock_sock()?I don't think there is a strong reason. We expose sk to the BPF program, but mainly for the socket storage map (which, afaik, doesn't require socket to be locked). OTOH, it seems that providing a consistent view of the sk to the BPF is a good idea.
hmm... most of the bpf prog also does not require a locked sock. For example, the
__sk_buff->sk. If a bpf prog needs a locked view of sk, a more generic solution is desired. Anyhow, I guess the train has sort of sailed for sockopt bpf.Quoted 1 line
Quoted 8 lines
Quoted 7 lines
Eric has suggested to try to use fast socket lock. It helps a bit, but it doesn't remove the issue completely because
we do a bunch of copy_{to,from}_user in the generic__cgroup_bpf_run_filter_getsockoptas well :-(Quoted 2 lines
Quoted 2 lines
| |--0.57%--__might_faultIs it a debug kernel?
Yeah, I think I did have CONFIG_DEBUG_ATOMIC_SLEEP=y for this particular run, but I don't think I have anything else debugging related (although, it might bring in DEBUG_KERNEL and some other crap). Let me check if something else has crept in and rerun the benchmarks without it. I'll respin with the updated data if nothing serious pop up.
Quoted 71 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
| --0.56%--release_sockWith the patch applied:
0.42% 0.10% tcp_mmap [kernel.kallsyms] [k]Quoted 7 lines
Quoted 7 lines
__cgroup_bpf_run_filter_getsockopt_kernQuoted 1 line
0.02% 0.02% tcp_mmap [kernel.kallsyms] [k]
__cgroup_bpf_run_filter_getsockoptQuoted 1 line
[ ... ]
Review int
Quoted 1 line
(quoted diff) +0
-0@@ -1445,15 +1442,29 @@ int__cgroup_bpf_run_filter_getsockopt(struct
Quoted 16 lines
Quoted 16 lines
sock *sk, int level,
Quoted 15 lines
kernel/bpf/cgroup.c +3
-7int __user *optlen, int max_optlen, int retval) { - struct cgroup *cgrp = sock_cgroup_ptr(&sk->sk_cgrp_data); -struct bpf_sockopt_kern ctx = { - .sk = sk, - .level = level, - .optname = optname, - .retval = retval, - }; + struct bpf_sockopt_kern ctx; +struct bpf_sockopt_buf buf; +struct cgroup *cgrp;int ret;Quoted 10 lines
Quoted 10 lines
kernel/bpf/cgroup.c int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level, · line 1457
Quoted 8 lines
kernel/bpf/cgroup.c +8
-0+#ifdef CONFIG_INET +/* TCP do_tcp_getsockopt has optimized getsockopt implementation +* to avoid extra socket lock for TCP_ZEROCOPY_RECEIVE. +*/ +if (sk->sk_prot->getsockopt == tcp_getsockopt && + level ==SOL_TCP && optname ==TCP_ZEROCOPY_RECEIVE ) +return retval; +#endif That seems too much protocol details and not very scalable. It is not very related to
kernel/bpf/cgroup.cwhich has very littleidea
Quoted 10 lines
Quoted 10 lines
whether a specific protocol has optimized things in some ways (e.g. by directly calling cgroup's bpf prog at some strategic places in this patch).
Lets see if it can be done better.At least, these protocol checks belong to the net's
socket.cmore than the bpf'scgroup.chere. If it also looks like layering breakage insocket.c, may be adding a signal insk_prot(for example) to tell if thesk_prot->getsockopthas already called the cgroup's bpf prog? (e.g.tcp_getsockopt()can directly call the cgroup's bpf forall
Quoted 20 lines
Quoted 5 lines
optname instead of only
TCP_ZEROCOPY_RECEIVE).For example:
int
__sys_getsockopt(...)Quoted 8 lines
{ /* ... */ if (!sk_prot->bpf_getsockopt_handled) BPF_CGROUP_RUN_PROG_GETSOCKOPT(...); }Thoughts?
Sounds good. I didn't go that far because I don't expect there to be a lot of special cases like that. But it might be worth supporting it in a generic way from the beginning.
I was thinking about something simpler:
Quoted 7 lines
Quoted 3 lines
int
__cgroup_bpf_run_filter_getsockopt(sk, ...){ if (sk->sk_prot->bypass_bpf_getsockopt(level, optlen)) {I think it meant s/optlen/optname/ which is not
__user. Yeah, I think that can provide a more generic solution and also abstract things away.
Please add a details comment in this function.
Sure, will do!
Quoted 10 lines
Quoted 7 lines
return retval; }// ...
}Not sure it's worth exposing it to the
__sys_getsockopt. WDYT?or call that in
BPF_CGROUP_RUN_PROG_GETSOCKOPT(). then the changes in__cgroup_bpf_run_filter_getsockopt()in this patch should go away?
SG, will add to BPF_CGROUP_RUN_PROG_GETSOCKOPT.