tcp: increase flexibility of EBPF congestion control initialization
Subsystems (5)
PATCH v3 1/5 tcp: only init congestion control if not initialized already
In reply to Neal CardwellFrom: Neal Cardwell <[email protected]>
Change tcp_init_transfer() to only initialize congestion control if it has not been initialized already.
With this new approach, we can arrange things so that if the EBPF code sets the congestion control by calling setsockopt(TCP_CONGESTION) then tcp_init_transfer() will not re-initialize the CC module.
This is an approach that has the following beneficial properties:
(1) This allows CC module customizations made by the EBPF called in tcp_init_transfer() to persist, and not be wiped out by a later call to tcp_init_congestion_control() in tcp_init_transfer().
(2) Does not flip the order of EBPF and CC init, to avoid causing bugs for existing code upstream that depends on the current order.
(3) Does not cause 2 initializations for for CC in the case where the EBPF called in tcp_init_transfer() wants to set the CC to a new CC algorithm.
(4) Allows follow-on simplifications to the code in net/core/filter.c and net/ipv4/tcp_cong.c, which currently both have some complexity to special-case CC initialization to avoid double CC initialization if EBPF sets the CC.
--- include/net/inet_connection_sock.h | 3 ++- net/ipv4/tcp.c | 1 + net/ipv4/tcp_cong.c | 3 ++- net/ipv4/tcp_input.c | 4 +++- 4 files changed, 8 insertions(+), 3 deletions(-)
include/net/inet_connection_sock.h +2 -1
@@ -96,7 +96,8 @@ struct inet_connection_sock {void (*icsk_clean_acked)(struct sock *sk,u32 acked_seq);struct hlist_node icsk_listen_portaddr_node;unsigned int (*icsk_sync_mss)(struct sock *sk,u32 pmtu); - __u8 icsk_ca_state:6, + __u8 icsk_ca_state:5, + icsk_ca_initialized:1, icsk_ca_setsockopt:1, icsk_ca_dst_locked:1;__u8 icsk_retransmits;
net/ipv4/tcp.c +1 -0
@@ -2698,6 +2698,7 @@ int tcp_disconnect(struct sock *sk, int flags)if (icsk->icsk_ca_ops->release) icsk->icsk_ca_ops->release (sk);memset (icsk->icsk_ca_priv,0 ,sizeof (icsk->icsk_ca_priv)); + icsk->icsk_ca_initialized =0 ;tcp_set_ca_state (sk, TCP_CA_Open); tp->is_sack_reneg =0 ;tcp_clear_retrans (tp);
net/ipv4/tcp_cong.c +2 -1
@@ -176,7 +176,7 @@ void tcp_assign_congestion_control(struct sock *sk)void tcp_init_congestion_control (struct sock *sk) { -const struct inet_connection_sock *icsk =inet_csk (sk); +struct inet_connection_sock *icsk =inet_csk (sk);tcp_sk (sk)->prior_ssthresh =0 ;if (icsk->icsk_ca_ops->init)
@@ -185,6 +185,7 @@ void tcp_init_congestion_control(struct sock *sk)INET_ECN_xmit (sk); elseINET_ECN_dontxmit (sk ); + icsk->icsk_ca_initialized =1 ; } static void tcp_reinit_congestion_control(struct sock *sk,
net/ipv4/tcp_input.c +3 -1
@@ -5894,8 +5894,10 @@ void tcp_init_transfer(struct sock *sk, int bpf_op, struct sk_buff *skb) tp->snd_cwnd =tcp_init_cwnd (tp,__sk_dst_get (sk)); tp->snd_cwnd_stamp = tcp_jiffies32; + icsk->icsk_ca_initialized =0 ;bpf_skops_established (sk, bpf_op, skb); -tcp_init_congestion_control (sk); +if (!icsk->icsk_ca_initialized) +tcp_init_congestion_control (sk);tcp_init_buffer_space (sk); }
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
PATCH v3 3/5 tcp: simplify tcp_set_congestion_control(): always reinitialize
In reply to Neal CardwellFrom: Neal Cardwell <[email protected]>
Now that the previous patches ensure that all call sites for tcp_set_congestion_control() want to initialize congestion control, we can simplify tcp_set_congestion_control() by removing the reinit argument and the code to support it.
--- include/net/tcp.h | 2 +- net/core/filter.c | 3 +-- net/ipv4/tcp.c | 2 +- net/ipv4/tcp_cong.c | 11 ++--------- 4 files changed, 5 insertions(+), 13 deletions(-)
include/net/tcp.h +1 -1
@@ -1104,7 +1104,7 @@ void tcp_get_available_congestion_control(char *buf, size_t len);void tcp_get_allowed_congestion_control (char *buf,size_t len);int tcp_set_allowed_congestion_control (char *allowed);int tcp_set_congestion_control (struct sock *sk,const char *name,bool load, -bool reinit,bool cap_net_admin); +bool cap_net_admin);u32 tcp_slow_start (struct tcp_sock *tp,u32 acked);void tcp_cong_avoid_ai (struct tcp_sock *tp,u32 w,u32 acked);
net/core/filter.c +1 -2
@@ -4451,8 +4451,7 @@ static int _bpf_setsockopt(struct sock *sk, int level, int optname,strncpy (name, optval,min_t (long, optlen,TCP_CA_NAME_MAX -1 )); name[TCP_CA_NAME_MAX -1 ] =0 ; - ret =tcp_set_congestion_control (sk, name, false, - true, true); + ret =tcp_set_congestion_control (sk, name, false, true); }else {struct inet_connection_sock *icsk =inet_csk (sk);struct tcp_sock *tp =tcp_sk (sk);
net/ipv4/tcp.c +1 -1
@@ -3050,7 +3050,7 @@ static int do_tcp_setsockopt(struct sock *sk, int level, int optname, name[val] =0 ;lock_sock (sk); - err =tcp_set_congestion_control (sk, name, true, true, + err =tcp_set_congestion_control (sk, name, true,ns_capable (sock_net (sk)->user_ns,CAP_NET_ADMIN ));release_sock (sk);
net/ipv4/tcp_cong.c +2 -9
@@ -341,7 +341,7 @@ int tcp_set_allowed_congestion_control(char *val)* already initialized. */ int tcp_set_congestion_control (struct sock *sk,const char *name,bool load, -bool reinit,bool cap_net_admin) +bool cap_net_admin) {struct inet_connection_sock *icsk =inet_csk (sk);const struct tcp_congestion_ops *ca;
@@ -365,15 +365,8 @@ int tcp_set_congestion_control(struct sock *sk, const char *name, bool load,if (!ca) { err = -ENOENT ; }else if (!load) { -const struct tcp_congestion_ops *old_ca = icsk->icsk_ca_ops; -if (bpf_try_module_get (ca, ca->owner)) { -if (reinit) { -tcp_reinit_congestion_control (sk, ca); - }else { - icsk->icsk_ca_ops = ca; -bpf_module_put (old_ca, old_ca->owner); - } +tcp_reinit_congestion_control (sk, ca); }else { err = -EBUSY ; }
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
PATCH v3 4/5 tcp: simplify _bpf_setsockopt(): remove flags argument
In reply to Neal CardwellFrom: Neal Cardwell <[email protected]>
Now that the previous patches have removed the code that uses the flags argument to _bpf_setsockopt(), we can remove that argument.
--- net/core/filter.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)
net/core/filter.c +3 -7
@@ -4314,7 +4314,7 @@ static const struct bpf_func_proto bpf_get_socket_uid_proto = { };static int _bpf_setsockopt (struct sock *sk,int level,int optname, -char *optval,int optlen,u32 flags) +char *optval,int optlen) {char devname[IFNAMSIZ ];int val, valbool;
@@ -4611,9 +4611,7 @@ static int _bpf_getsockopt(struct sock *sk, int level, int optname, BPF_CALL_5(bpf_sock_addr_setsockopt, struct bpf_sock_addr_kern *, ctx, int, level, int, optname, char *, optval, int, optlen) { -u32 flags =0 ; -return _bpf_setsockopt (ctx->sk, level, optname, optval, optlen, - flags); +return _bpf_setsockopt (ctx->sk, level, optname, optval, optlen); }static const struct bpf_func_proto bpf_sock_addr_setsockopt_proto = {
@@ -4647,9 +4645,7 @@ static const struct bpf_func_proto bpf_sock_addr_getsockopt_proto = { BPF_CALL_5(bpf_sock_ops_setsockopt, struct bpf_sock_ops_kern *, bpf_sock, int, level, int, optname, char *, optval, int, optlen) { -u32 flags =0 ; -return _bpf_setsockopt (bpf_sock->sk, level, optname, optval, optlen, - flags); +return _bpf_setsockopt (bpf_sock->sk, level, optname, optval, optlen); }static const struct bpf_func_proto bpf_sock_ops_setsockopt_proto = {
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
PATCH v3 2/5 tcp: simplify EBPF TCP_CONGESTION to always init CC
In reply to Neal CardwellFrom: Neal Cardwell <[email protected]>
Now that the previous patch ensures we don't initialize the congestion control twice, when EBPF sets the congestion control algorithm at connection establishment we can simplify the code by simply initializing the congestion control module at that time.
--- net/core/filter.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
net/core/filter.c +1 -6
@@ -4313,8 +4313,6 @@ static const struct bpf_func_proto bpf_get_socket_uid_proto = { .arg1_type = ARG_PTR_TO_CTX, }; -#define SOCKOPT_CC_REINIT (1 << 0) -static int _bpf_setsockopt (struct sock *sk,int level,int optname,char *optval,int optlen,u32 flags) {
@@ -4449,13 +4447,12 @@ static int _bpf_setsockopt(struct sock *sk, int level, int optname, sk->sk_prot->setsockopt == tcp_setsockopt) {if (optname ==TCP_CONGESTION ) {char name[TCP_CA_NAME_MAX ]; -bool reinit = flags &SOCKOPT_CC_REINIT ;strncpy (name, optval,min_t (long, optlen,TCP_CA_NAME_MAX -1 )); name[TCP_CA_NAME_MAX -1 ] =0 ; ret =tcp_set_congestion_control (sk, name, false, - reinit, true); + true, true); }else {struct inet_connection_sock *icsk =inet_csk (sk);struct tcp_sock *tp =tcp_sk (sk);
@@ -4652,8 +4649,6 @@ BPF_CALL_5(bpf_sock_ops_setsockopt, struct bpf_sock_ops_kern *, bpf_sock, int, level, int, optname, char *, optval, int, optlen) { u32 flags = 0; -if (bpf_sock->op >BPF_SOCK_OPS_NEEDS_ECN ) - flags |=SOCKOPT_CC_REINIT ;return _bpf_setsockopt (bpf_sock->sk, level, optname, optval, optlen, flags); }
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
v3 cover tcp: increase flexibility of EBPF congestion control initialization
From: Neal Cardwell <[email protected]>
This patch series reorganizes TCP congestion control initialization so that if EBPF code called by tcp_init_transfer() sets the congestion control algorithm by calling setsockopt(TCP_CONGESTION) then the TCP stack initializes the congestion control module immediately, instead of having tcp_init_transfer() later initialize the congestion control module.
This increases flexibility for the EBPF code that runs at connection establishment time, and simplifies the code.
This has the following benefits:
(1) This allows CC module customizations made by the EBPF called in tcp_init_transfer() to persist, and not be wiped out by a later call to tcp_init_congestion_control() in tcp_init_transfer().
(2) Does not flip the order of EBPF and CC init, to avoid causing bugs for existing code upstream that depends on the current order.
(3) Does not cause 2 initializations for for CC in the case where the EBPF called in tcp_init_transfer() wants to set the CC to a new CC algorithm.
(4) Allows follow-on simplifications to the code in net/core/filter.c and net/ipv4/tcp_cong.c, which currently both have some complexity to special-case CC initialization to avoid double CC initialization if EBPF sets the CC.
changes in v2:
o rebase onto bpf-next
o add another follow-on simplification suggested by Martin KaFai Lau:
"tcp: simplify tcp_set_congestion_control() load=false case"
changes in v3:
o no change in commits
o resent patch series from @gmail.com, since mail from [email protected] stopped being accepted at [email protected] mid-way through processing the v2 patch series (between patches 2 and 3), confusing patchwork about which patches belonged to the v2 patch series
Neal Cardwell (5):
tcp: only init congestion control if not initialized already
tcp: simplify EBPF TCP_CONGESTION to always init CC
tcp: simplify tcp_set_congestion_control(): always reinitialize
tcp: simplify _bpf_setsockopt(): remove flags argument
tcp: simplify tcp_set_congestion_control() load=false case
include/net/inet_connection_sock.h | 3 ++- include/net/tcp.h | 2 +- net/core/filter.c | 18 ++++-------------- net/ipv4/tcp.c | 3 ++- net/ipv4/tcp_cong.c | 27 +++++++-------------------- net/ipv4/tcp_input.c | 4 +++- 6 files changed, 19 insertions(+), 38 deletions(-)
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
PATCH v3 5/5 tcp: simplify tcp_set_congestion_control() load=false case
From: Neal Cardwell <[email protected]>
Simplify tcp_set_congestion_control() by removing the initialization code path for the !load case.
There are only two call sites for tcp_set_congestion_control(). The EBPF call site is the only one that passes load=false; it also passes cap_net_admin=true. Because of that, the exact same behavior can be achieved by removing the special if (!load) branch of the logic. Both before and after this commit, the EBPF case will call bpf_try_module_get(), and if that succeeds then call tcp_reinit_congestion_control() or if that fails then return EBUSY.
Note that this returns the logic to a structure very similar to the structure before:
commit 91b5b21c7c16 ("bpf: Add support for changing congestion control")
except that the CAP_NET_ADMIN status is passed in as a function argument.
This clean-up was suggested by Martin KaFai Lau.
--- net/ipv4/tcp_cong.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)
net/ipv4/tcp_cong.c +4 -11
@@ -362,21 +362,14 @@ int tcp_set_congestion_control(struct sock *sk, const char *name, bool load, goto out; } -if (!ca) { +if (!ca) err = -ENOENT ; - }else if (!load) { -if (bpf_try_module_get (ca, ca->owner)) { -tcp_reinit_congestion_control (sk, ca); - }else { - err = -EBUSY ; - } - }else if (!((ca->flags &TCP_CONG_NON_RESTRICTED ) || cap_net_admin)) { +else if (!((ca->flags &TCP_CONG_NON_RESTRICTED ) || cap_net_admin)) err = -EPERM ; - }else if (!bpf_try_module_get (ca, ca->owner)) { +else if (!bpf_try_module_get (ca, ca->owner)) err = -EBUSY ; - }else { +else tcp_reinit_congestion_control (sk, ca); - } out:rcu_read_unlock ();return err;
Signature · 1 line
2.28.0.618.gf4bc123cb7-goog
Quoted 36 lines · Neal Cardwell
From: Neal Cardwell <[email protected]>
This patch series reorganizes TCP congestion control initialization so that if EBPF code called by
tcp_init_transfer()sets the congestion control algorithm by callingsetsockopt(TCP_CONGESTION)then the TCP stack initializes the congestion control module immediately, instead of havingtcp_init_transfer()later initialize the congestion control module.This increases flexibility for the EBPF code that runs at connection establishment time, and simplifies the code.
This has the following benefits:
(1) This allows CC module customizations made by the EBPF called in
tcp_init_transfer()to persist, and not be wiped out by a later call totcp_init_congestion_control()intcp_init_transfer().(2) Does not flip the order of EBPF and CC init, to avoid causing bugs for existing code upstream that depends on the current order.
(3) Does not cause 2 initializations for for CC in the case where the EBPF called in
tcp_init_transfer()wants to set the CC to a new CC algorithm.(4) Allows follow-on simplifications to the code in
net/core/filter.candnet/ipv4/tcp_cong.c, which currently both have some complexity to special-case CC initialization to avoid double CC initialization if EBPF sets the CC.changes in v2:
o rebase onto bpf-next
o add another follow-on simplification suggested by Martin KaFai Lau:
"tcp: simplifytcp_set_congestion_control()load=false case"
Quoted 8 lines
changes in v3:
o no change in commits
o resent patch series from @gmail.com, since mail from [email protected] stopped being accepted at [email protected] mid-way through processing the v2 patch series (between patches 2 and 3), confusing patchwork about which patches belonged to the v2 patch series
Alexei Starovoitov
AppliedQuoted 40 lines · Martin KaFai Lau
Quoted 38 lines · Neal Cardwell
From: Neal Cardwell <[email protected]>
This patch series reorganizes TCP congestion control initialization so that if EBPF code called by
tcp_init_transfer()sets the congestion control algorithm by callingsetsockopt(TCP_CONGESTION)then the TCP stack initializes the congestion control module immediately, instead of havingtcp_init_transfer()later initialize the congestion control module.This increases flexibility for the EBPF code that runs at connection establishment time, and simplifies the code.
This has the following benefits:
(1) This allows CC module customizations made by the EBPF called in
tcp_init_transfer()to persist, and not be wiped out by a later call totcp_init_congestion_control()intcp_init_transfer().(2) Does not flip the order of EBPF and CC init, to avoid causing bugs for existing code upstream that depends on the current order.
(3) Does not cause 2 initializations for for CC in the case where the EBPF called in
tcp_init_transfer()wants to set the CC to a new CC algorithm.(4) Allows follow-on simplifications to the code in
net/core/filter.candnet/ipv4/tcp_cong.c, which currently both have some complexity to special-case CC initialization to avoid double CC initialization if EBPF sets the CC.changes in v2:
o rebase onto bpf-next
o add another follow-on simplification suggested by Martin KaFai Lau:
"tcp: simplifytcp_set_congestion_control()load=false case"changes in v3:
Quoted 7 lines
Quoted 6 lines
o no change in commits
o resent patch series from @gmail.com, since mail from [email protected] stopped being accepted at [email protected] mid-way through processing the v2 patch series (between patches 2 and 3), confusing patchwork about which patches belonged to the v2 patch series
Acked-by Martin KaFai Lau <[email protected]>
Applied.
Martin, thanks for the review.
Neal, please keep Acks when you resubmit patches without changes in the future. Also please follow up with a selftests/bpf based on test_progs to cover new functionality.
Thanks