tcp: Introduce optional per-netns ehash.
Subsystems (13)
v5 cover tcp: Introduce optional per-netns ehash.
The more sockets we have in the hash table, the longer we spend looking up the socket. While running a number of small workloads on the same host, they penalise each other and cause performance degradation.
The root cause might be a single workload that consumes much more resources than the others. It often happens on a cloud service where different workloads share the same computing resource.
On EC2 c5.24xlarge instance (196 GiB memory and 524288 (1Mi / 2) ehash entries), after running iperf3 in different netns, creating 24Mi sockets without data transfer in the root netns causes about 10% performance regression for the iperf3's connection.
thash_entries sockets length Gbps
524288 1 1 50.7 24Mi 48 45.1
It is basically related to the length of the list of each hash bucket. For testing purposes to see how performance drops along the length, I set 131072 (1Mi / 8) to thash_entries, and here's the result.
thash_entries sockets length Gbps
131072 1 1 50.7 1Mi 8 49.9 2Mi 16 48.9 4Mi 32 47.3 8Mi 64 44.6 16Mi 128 40.6 24Mi 192 36.3 32Mi 256 32.5 40Mi 320 27.0 48Mi 384 25.0
To resolve the socket lookup degradation, we introduce an optional per-netns hash table for TCP, but it's just ehash, and we still share the global bhash, bhash2 and lhash2.
With a smaller ehash, we can look up non-listener sockets faster and isolate such noisy neighbours. Also, we can reduce lock contention.
For details, please see the last patch.
patch 1 - 4: prep for per-netns ehash
patch 5: small optimisation for netns dismantle without TIME_WAIT sockets
patch 6: add per-netns ehash
Changes:
v5:
* Patch 2
* Keep the tw_refcount base value at 1 (Eric Dumazet)
* Add WARN_ON_ONCE() for tw_refcount (Eric Dumazet)
* Patch 5
* Test tw_refcount against 1 in tcp_twsk_purge()
v4: https://lore.kernel.org/netdev/[email protected]/
* Add Patch 2
* Patch 1
* Add cleanups in tcp_time_wait() and tcp_v[46]_connect()
* Patch 3
* /tcp_death_row/s/->/./
* Patch 4
* Add mellanox and netronome driver changes back (Paolo Abeni, Jakub Kicinski)
* /tcp_death_row/s/->/./
* Patch 5
* Simplify tcp_twsk_purge()
* Patch 6
* Move inet_pernet_hashinfo_free() into tcp_sk_exit_batch()
v3: https://lore.kernel.org/netdev/[email protected]/
* Patch 3
* Drop mellanox and netronome driver changes (Eric Dumazet)
* Patch 4
* Add test results in the changelog
* Patch 5
* Use roundup_pow_of_two() in tcp_set_hashinfo() (Eric Dumazet)
* Remove proc_tcp_child_ehash_entries() and use proc_douintvec_minmax()
v2: https://lore.kernel.org/netdev/[email protected]/
* Drop flock() and UDP stuff
* Patch 2
* Rename inet_get_hashinfo() to tcp_or_dccp_get_hashinfo() (Eric Dumazet)
* Patch 4
* Remove unnecessary inet_twsk_purge() calls for unshare()
* Factorise inet_twsk_purge() calls (Eric Dumazet)
* Patch 5
* Change max buckets size as 16Mi
* Use unsigned int for ehash size (Eric Dumazet)
* Use GFP_KERNEL_ACCOUNT for the per-netns ehash allocation (Eric Dumazet)
* Use current->nsproxy->net_ns for parent netns (Eric Dumazet)
v1: https://lore.kernel.org/netdev/[email protected]/
Kuniyuki Iwashima (6):
tcp: Clean up some functions.
tcp: Don't allocate tcp_death_row outside of struct netns_ipv4.
tcp: Set NULL to sk->sk_prot->h.hashinfo.
tcp: Access &tcp_hashinfo via net.
tcp: Save unnecessary inet_twsk_purge() calls.
tcp: Introduce optional per-netns ehash.
Documentation/networking/ip-sysctl.rst | 23 +++ .../chelsio/inline_crypto/chtls/chtls_cm.c | 5 +- .../mellanox/mlx5/core/en_accel/ktls_rx.c | 5 +- .../net/ethernet/netronome/nfp/crypto/tls.c | 5 +- include/net/inet_hashtables.h | 16 ++ include/net/netns/ipv4.h | 4 +- include/net/tcp.h | 1 + net/core/filter.c | 5 +- net/dccp/proto.c | 2 + net/ipv4/af_inet.c | 2 +- net/ipv4/esp4.c | 3 +- net/ipv4/inet_connection_sock.c | 22 ++- net/ipv4/inet_hashtables.c | 102 ++++++++++--- net/ipv4/inet_timewait_sock.c | 4 +- net/ipv4/netfilter/nf_socket_ipv4.c | 4 +- net/ipv4/netfilter/nf_tproxy_ipv4.c | 16 +- net/ipv4/proc.c | 2 +- net/ipv4/sysctl_net_ipv4.c | 47 +++++- net/ipv4/tcp.c | 1 + net/ipv4/tcp_diag.c | 18 ++- net/ipv4/tcp_ipv4.c | 143 +++++++++++------- net/ipv4/tcp_minisocks.c | 28 +++- net/ipv6/esp6.c | 3 +- net/ipv6/inet6_hashtables.c | 4 +- net/ipv6/netfilter/nf_socket_ipv6.c | 4 +- net/ipv6/netfilter/nf_tproxy_ipv6.c | 8 +- net/ipv6/tcp_ipv6.c | 42 ++--- net/mptcp/mptcp_diag.c | 7 +- 28 files changed, 365 insertions(+), 161 deletions(-)
Signature · 1 line
2.30.2
PATCH v5 1/6 tcp: Clean up some functions.
This patch adds no functional change and cleans up some functions that the following patches touch around so that we make them tidy and easy to review/revert. The changes are
- Keep reverse christmas tree order
- Remove unnecessary init of port in inet_csk_find_open_port()
- Use req_to_sk() once in reqsk_queue_unlink()
- Use sock_net(sk) once in tcp_time_wait() and tcp_v[46]_connect()
--- net/ipv4/inet_connection_sock.c | 21 ++++++++++----------- net/ipv4/inet_hashtables.c | 29 +++++++++++++++-------------- net/ipv4/tcp_ipv4.c | 18 +++++++++--------- net/ipv4/tcp_minisocks.c | 6 +++--- net/ipv6/tcp_ipv6.c | 17 ++++++++--------- 5 files changed, 45 insertions(+), 46 deletions(-)
net/ipv4/inet_connection_sock.c +10 -11
@@ -286,15 +286,13 @@ inet_csk_find_open_port(const struct sock *sk, struct inet_bind_bucket **tb_ret, struct inet_bind_hashbucket **head2_ret, int *port_ret) { struct inet_hashinfo *hinfo = sk->sk_prot->h.hashinfo; -int port =0 ; +int i, low, high, attempt_half, port, l3mdev;struct inet_bind_hashbucket *head, *head2;struct net *net =sock_net (sk); -bool relax = false; -int i, low, high, attempt_half;struct inet_bind2_bucket *tb2;struct inet_bind_bucket *tb;u32 remaining, offset; -int l3mdev; +bool relax = false; l3mdev =inet_sk_bound_l3mdev (sk); ports_exhausted:
@@ -471,15 +469,14 @@ int inet_csk_get_port(struct sock *sk, unsigned short snum) {bool reuse = sk->sk_reuse && sk->sk_state !=TCP_LISTEN ;struct inet_hashinfo *hinfo = sk->sk_prot->h.hashinfo; -int ret =1 , port = snum; -struct net *net =sock_net (sk);bool found_port = false, check_bind_conflict = true;bool bhash_created = false, bhash2_created = false;struct inet_bind_hashbucket *head, *head2;struct inet_bind2_bucket *tb2 =NULL ;struct inet_bind_bucket *tb =NULL ;bool head2_lock_acquired = false; -int l3mdev; +int ret =1 , port = snum, l3mdev; +struct net *net =sock_net (sk); l3mdev =inet_sk_bound_l3mdev (sk);
@@ -909,14 +906,16 @@ static void reqsk_migrate_reset(struct request_sock *req)/* return true if req was found in the ehash table */ static bool reqsk_queue_unlink (struct request_sock *req) { -struct inet_hashinfo *hashinfo =req_to_sk (req)->sk_prot->h.hashinfo; +struct sock *sk =req_to_sk (req);bool found = false; -if (sk_hashed (req_to_sk (req))) { -spinlock_t *lock =inet_ehash_lockp (hashinfo, req->rsk_hash); +if (sk_hashed (sk)) { +struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo; +spinlock_t *lock; + lock =inet_ehash_lockp (hashinfo, req->rsk_hash);spin_lock (lock); - found =__sk_nulls_del_node_init_rcu (req_to_sk (req)); + found =__sk_nulls_del_node_init_rcu (sk);spin_unlock (lock); }if (timer_pending (&req->rsk_timer) &&del_timer_sync (&req->rsk_timer))
net/ipv4/inet_hashtables.c +15 -14
@@ -169,13 +169,14 @@ void inet_bind_hash(struct sock *sk, struct inet_bind_bucket *tb,static void __inet_put_port (struct sock *sk) {struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo; -const int bhash =inet_bhashfn (sock_net (sk),inet_sk (sk)->inet_num, - hashinfo->bhash_size); -struct inet_bind_hashbucket *head = &hashinfo->bhash[bhash]; -struct inet_bind_hashbucket *head2 = -inet_bhashfn_portaddr (hashinfo, sk,sock_net (sk), -inet_sk (sk)->inet_num); +struct inet_bind_hashbucket *head, *head2; +struct net *net =sock_net (sk);struct inet_bind_bucket *tb; +int bhash; + + bhash =inet_bhashfn (net,inet_sk (sk)->inet_num, hashinfo->bhash_size); + head = &hashinfo->bhash[bhash]; + head2 =inet_bhashfn_portaddr (hashinfo, sk, net,inet_sk (sk)->inet_num);spin_lock (&head->lock); tb =inet_csk (sk)->icsk_bind_hash;
@@ -209,17 +210,17 @@ int __inet_inherit_port(const struct sock *sk, struct sock *child) {struct inet_hashinfo *table = sk->sk_prot->h.hashinfo;unsigned short port =inet_sk (child)->inet_num; -const int bhash =inet_bhashfn (sock_net (sk), port, - table->bhash_size); -struct inet_bind_hashbucket *head = &table->bhash[bhash]; -struct inet_bind_hashbucket *head2 = -inet_bhashfn_portaddr (table, child,sock_net (sk), port); +struct inet_bind_hashbucket *head, *head2;bool created_inet_bind_bucket = false; -bool update_fastreuse = false;struct net *net =sock_net (sk); +bool update_fastreuse = false;struct inet_bind2_bucket *tb2;struct inet_bind_bucket *tb; -int l3mdev; +int bhash, l3mdev; + + bhash =inet_bhashfn (net, port, table->bhash_size); + head = &table->bhash[bhash]; + head2 =inet_bhashfn_portaddr (table, child, net, port);spin_lock (&head->lock);spin_lock (&head2->lock);
@@ -629,8 +630,8 @@ static bool inet_ehash_lookup_by_sk(struct sock *sk,bool inet_ehash_insert (struct sock *sk,struct sock *osk,bool *found_dup_sk) {struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo; -struct hlist_nulls_head *list;struct inet_ehash_bucket *head; +struct hlist_nulls_head *list;spinlock_t *lock;bool ret = true;
net/ipv4/tcp_ipv4.c +9 -9
@@ -201,15 +201,16 @@ int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len) {struct inet_bind_hashbucket *prev_addr_hashbucket =NULL ;struct sockaddr_in *usin = (struct sockaddr_in *)uaddr; +struct inet_timewait_death_row *tcp_death_row;__be32 daddr, nexthop, prev_sk_rcv_saddr;struct inet_sock *inet =inet_sk (sk);struct tcp_sock *tp =tcp_sk (sk); +struct ip_options_rcu *inet_opt; +struct net *net =sock_net (sk);__be16 orig_sport, orig_dport;struct flowi4 *fl4;struct rtable *rt;int err; -struct ip_options_rcu *inet_opt; -struct inet_timewait_death_row *tcp_death_row =sock_net (sk)->ipv4.tcp_death_row;if (addr_len <sizeof (struct sockaddr_in ))return -EINVAL ;
@@ -235,7 +236,7 @@ int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)if (IS_ERR (rt)) { err =PTR_ERR (rt);if (err == -ENETUNREACH ) -IP_INC_STATS (sock_net (sk),IPSTATS_MIB_OUTNOROUTES ); +IP_INC_STATS (net,IPSTATS_MIB_OUTNOROUTES );return err; }
@@ -250,8 +251,7 @@ int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)if (!inet->inet_saddr) {if (inet_csk (sk)->icsk_bind2_hash) { prev_addr_hashbucket =inet_bhashfn_portaddr (&tcp_hashinfo, - sk,sock_net (sk), - inet->inet_num); + sk, net, inet->inet_num); prev_sk_rcv_saddr = sk->sk_rcv_saddr; } inet->inet_saddr = fl4->saddr;
@@ -292,6 +292,7 @@ int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)* complete initialization after this. */ tcp_set_state (sk,TCP_SYN_SENT ); + tcp_death_row = net->ipv4.tcp_death_row; err =inet_hash_connect (tcp_death_row, sk);if (err) goto failure;
@@ -317,8 +318,7 @@ int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len) inet->inet_daddr, inet->inet_sport, usin->sin_port)); - tp->tsoffset =secure_tcp_ts_off (sock_net (sk), - inet->inet_saddr, + tp->tsoffset =secure_tcp_ts_off (net, inet->inet_saddr, inet->inet_daddr); }
@@ -2406,9 +2406,9 @@ static void *established_get_first(struct seq_file *seq)static void *established_get_next (struct seq_file *seq,void *cur) { -struct sock *sk = cur; -struct hlist_nulls_node *node;struct tcp_iter_state *st = seq->private; +struct hlist_nulls_node *node; +struct sock *sk = cur; ++st->num; ++st->offset;
net/ipv4/tcp_minisocks.c +3 -3
@@ -247,10 +247,10 @@ void tcp_time_wait(struct sock *sk, int state, int timeo) {const struct inet_connection_sock *icsk =inet_csk (sk);const struct tcp_sock *tp =tcp_sk (sk); +struct net *net =sock_net (sk);struct inet_timewait_sock *tw; -struct inet_timewait_death_row *tcp_death_row =sock_net (sk)->ipv4.tcp_death_row; - tw =inet_twsk_alloc (sk, tcp_death_row, state); + tw =inet_twsk_alloc (sk, net->ipv4.tcp_death_row, state);if (tw) {struct tcp_timewait_sock *tcptw =tcp_twsk ((struct sock *)tw);
@@ -326,7 +326,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)* socket up. We've got bigger problems than * non-graceful socket closings. */ -NET_INC_STATS (sock_net (sk),LINUX_MIB_TCPTIMEWAITOVERFLOW ); +NET_INC_STATS (net,LINUX_MIB_TCPTIMEWAITOVERFLOW ); }tcp_update_metrics (sk);
net/ipv6/tcp_ipv6.c +8 -9
@@ -146,15 +146,16 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len) {struct sockaddr_in6 *usin = (struct sockaddr_in6 *) uaddr; -struct inet_sock *inet =inet_sk (sk);struct inet_connection_sock *icsk =inet_csk (sk); +struct in6_addr *saddr =NULL , *final_p, final;struct inet_timewait_death_row *tcp_death_row;struct ipv6_pinfo *np =tcp_inet6_sk (sk); +struct inet_sock *inet =inet_sk (sk);struct tcp_sock *tp =tcp_sk (sk); -struct in6_addr *saddr =NULL , *final_p, final; +struct net *net =sock_net (sk);struct ipv6_txoptions *opt; -struct flowi6 fl6;struct dst_entry *dst; +struct flowi6 fl6;int addr_type;int err;
@@ -280,7 +281,7 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr,security_sk_classify_flow (sk,flowi6_to_flowi_common (&fl6)); - dst =ip6_dst_lookup_flow (sock_net (sk), sk, &fl6, final_p); + dst =ip6_dst_lookup_flow (net, sk, &fl6, final_p);if (IS_ERR (dst)) { err =PTR_ERR (dst); goto failure;
@@ -292,8 +293,7 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr,if (icsk->icsk_bind2_hash) { prev_addr_hashbucket =inet_bhashfn_portaddr (&tcp_hashinfo, - sk,sock_net (sk), - inet->inet_num); + sk, net, inet->inet_num); prev_v6_rcv_saddr = sk->sk_v6_rcv_saddr; } saddr = &fl6.saddr;
@@ -325,7 +325,7 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr, inet->inet_dport = usin->sin6_port;tcp_set_state (sk,TCP_SYN_SENT ); - tcp_death_row =sock_net (sk)->ipv4.tcp_death_row; + tcp_death_row = net->ipv4.tcp_death_row; err =inet6_hash_connect (tcp_death_row, sk);if (err) goto late_failure;
@@ -339,8 +339,7 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr, sk->sk_v6_daddr.s6_addr32, inet->inet_sport, inet->inet_dport)); - tp->tsoffset =secure_tcpv6_ts_off (sock_net (sk), - np->saddr.s6_addr32, + tp->tsoffset =secure_tcpv6_ts_off (net, np->saddr.s6_addr32, sk->sk_v6_daddr.s6_addr32); }
Signature · 1 line
2.30.2
PATCH v5 2/6 tcp: Don't allocate tcp_death_row outside of struct netns_ipv4.
In reply to Kuniyuki IwashimaWe will soon introduce an optional per-netns ehash and access hash tables via net->ipv4.tcp_death_row->hashinfo instead of &tcp_hashinfo in most places.
It could harm the fast path because dereferences of two fields in net and tcp_death_row might incur two extra cache line misses. To save one dereference, let's place tcp_death_row back in netns_ipv4 and fetch hashinfo via net->ipv4.tcp_death_row"."hashinfo.
Note tcp_death_row was initially placed in netns_ipv4, and commit fbb8295248e1 ("tcp: allocate tcp_death_row outside of struct netns_ipv4") changed it to a pointer so that we can fire TIME_WAIT timers after freeing net. However, we don't do so after commit 04c494e68a13 ("Revert "tcp/dccp:
get rid of inet_twsk_purge()""), so we need not define tcp_death_row as a pointer.
Also, we move refcount_dec_and_test(&tw_refcount) from tcp_sk_exit() to tcp_sk_exit_batch() as a debug check.
--- include/net/netns/ipv4.h | 3 ++- net/ipv4/inet_timewait_sock.c | 4 +--- net/ipv4/proc.c | 2 +- net/ipv4/sysctl_net_ipv4.c | 8 ++------ net/ipv4/tcp_ipv4.c | 19 +++++++------------ net/ipv4/tcp_minisocks.c | 2 +- net/ipv6/tcp_ipv6.c | 2 +- 7 files changed, 15 insertions(+), 25 deletions(-)
include/net/netns/ipv4.h +2 -1
@@ -34,6 +34,7 @@ struct inet_hashinfo;struct inet_timewait_death_row {refcount_t tw_refcount; +/* Padding to avoid false sharing, tw_refcount can be often written */ struct inet_hashinfo *hashinfo____cacheline_aligned_in_smp ;int sysctl_max_tw_buckets; };
@@ -41,7 +42,7 @@ struct inet_timewait_death_row {struct tcp_fastopen_context ;struct netns_ipv4 { -struct inet_timewait_death_row *tcp_death_row; +struct inet_timewait_death_row tcp_death_row;#ifdef CONFIG_SYSCTL struct ctl_table_header *forw_hdr;
net/ipv4/inet_timewait_sock.c +1 -3
@@ -59,9 +59,7 @@ static void inet_twsk_kill(struct inet_timewait_sock *tw)inet_twsk_bind_unhash (tw, hashinfo);spin_unlock (&bhead->lock); -if (refcount_dec_and_test (&tw->tw_dr->tw_refcount)) -kfree (tw->tw_dr); - +refcount_dec (&tw->tw_dr->tw_refcount);inet_twsk_put (tw); }
net/ipv4/proc.c +1 -1
@@ -59,7 +59,7 @@ static int sockstat_seq_show(struct seq_file *seq, void *v)socket_seq_show (seq);seq_printf (seq,"TCP: inuse %d orphan %d tw %d alloc %d mem %ld\n" ,sock_prot_inuse_get (net, &tcp_prot), orphans, -refcount_read (&net->ipv4.tcp_death_row->tw_refcount) -1 , +refcount_read (&net->ipv4.tcp_death_row.tw_refcount) -1 , sockets,proto_memory_allocated (&tcp_prot)); seq_printf(seq, "UDP: inuse %d mem %ld\n", sock_prot_inuse_get(net, &udp_prot),
net/ipv4/sysctl_net_ipv4.c +2 -6
@@ -530,10 +530,9 @@ static struct ctl_table ipv4_table[] = { };static struct ctl_table ipv4_net_table[] = { -/* tcp_max_tw_buckets must be first in this table. */ { .procname ="tcp_max_tw_buckets" , -/* .data = &init_net.ipv4.tcp_death_row.sysctl_max_tw_buckets, */ + .data = &init_net.ipv4.tcp_death_row.sysctl_max_tw_buckets, .maxlen =sizeof (int ), .mode =0644 , .proc_handler = proc_dointvec
@@ -1361,8 +1360,7 @@ static __net_init int ipv4_sysctl_init_net(struct net *net)if (!table) goto err_alloc; -/* skip first entry (sysctl_max_tw_buckets) */ -for (i =1 ; i <ARRAY_SIZE (ipv4_net_table) -1 ; i++) { +for (i =0 ; i <ARRAY_SIZE (ipv4_net_table) -1 ; i++) {if (table[i].data) {/* Update the variables to point into * the current struct net
@@ -1377,8 +1375,6 @@ static __net_init int ipv4_sysctl_init_net(struct net *net) } } - table[0 ].data = &net->ipv4.tcp_death_row->sysctl_max_tw_buckets; - net->ipv4.ipv4_hdr =register_net_sysctl (net,"net/ipv4" , table);if (!net->ipv4.ipv4_hdr) goto err_reg;
net/ipv4/tcp_ipv4.c +7 -12
@@ -292,7 +292,7 @@ int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)* complete initialization after this. */ tcp_set_state (sk,TCP_SYN_SENT ); - tcp_death_row = net->ipv4.tcp_death_row; + tcp_death_row = &net->ipv4.tcp_death_row; err =inet_hash_connect (tcp_death_row, sk);if (err) goto failure;
@@ -3091,13 +3091,9 @@ EXPORT_SYMBOL(tcp_prot); static void __net_exit tcp_sk_exit(struct net *net) { -struct inet_timewait_death_row *tcp_death_row = net->ipv4.tcp_death_row; -if (net->ipv4.tcp_congestion_control)bpf_module_put (net->ipv4.tcp_congestion_control, net->ipv4.tcp_congestion_control->owner); -if (refcount_dec_and_test (&tcp_death_row->tw_refcount)) -kfree (tcp_death_row); }static int __net_init tcp_sk_init (struct net *net)
@@ -3129,13 +3125,10 @@ static int __net_init tcp_sk_init(struct net *net) net->ipv4.sysctl_tcp_tw_reuse =2 ; net->ipv4.sysctl_tcp_no_ssthresh_metrics_save =1 ; - net->ipv4.tcp_death_row =kzalloc (sizeof (struct inet_timewait_death_row ),GFP_KERNEL ); -if (!net->ipv4.tcp_death_row) -return -ENOMEM ; -refcount_set (&net->ipv4.tcp_death_row->tw_refcount,1 ); +refcount_set (&net->ipv4.tcp_death_row.tw_refcount,1 ); cnt = tcp_hashinfo.ehash_mask +1 ; - net->ipv4.tcp_death_row->sysctl_max_tw_buckets = cnt /2 ; - net->ipv4.tcp_death_row->hashinfo = &tcp_hashinfo; + net->ipv4.tcp_death_row.sysctl_max_tw_buckets = cnt /2 ; + net->ipv4.tcp_death_row.hashinfo = &tcp_hashinfo; net->ipv4.sysctl_max_syn_backlog =max (128 , cnt /128 ); net->ipv4.sysctl_tcp_sack =1 ;
@@ -3201,8 +3194,10 @@ static void __net_exit tcp_sk_exit_batch(struct list_head *net_exit_list)inet_twsk_purge (&tcp_hashinfo,AF_INET ); -list_for_each_entry (net, net_exit_list, exit_list) +list_for_each_entry (net, net_exit_list, exit_list) { +WARN_ON_ONCE (!refcount_dec_and_test (&net->ipv4.tcp_death_row.tw_refcount));tcp_fastopen_ctx_destroy (net); + } }static struct pernet_operations __net_initdata tcp_sk_ops = {
net/ipv4/tcp_minisocks.c +1 -1
@@ -250,7 +250,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)struct net *net =sock_net (sk);struct inet_timewait_sock *tw; - tw =inet_twsk_alloc (sk, net->ipv4.tcp_death_row, state); + tw =inet_twsk_alloc (sk, &net->ipv4.tcp_death_row, state);if (tw) {struct tcp_timewait_sock *tcptw =tcp_twsk ((struct sock *)tw);
net/ipv6/tcp_ipv6.c +1 -1
@@ -325,7 +325,7 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr, inet->inet_dport = usin->sin6_port;tcp_set_state (sk,TCP_SYN_SENT ); - tcp_death_row = net->ipv4.tcp_death_row; + tcp_death_row = &net->ipv4.tcp_death_row; err =inet6_hash_connect (tcp_death_row, sk);if (err) goto late_failure;
Signature · 1 line
2.30.2
PATCH v5 3/6 tcp: Set NULL to sk->sk_prot->h.hashinfo.
In reply to Kuniyuki IwashimaWe will soon introduce an optional per-netns ehash.
This means we cannot use the global sk->sk_prot->h.hashinfo to fetch a TCP hashinfo.
Instead, set NULL to sk->sk_prot->h.hashinfo for TCP and get a proper hashinfo from net->ipv4.tcp_death_row.hashinfo.
Note that we need not use sk->sk_prot->h.hashinfo if DCCP is disabled.
--- include/net/inet_hashtables.h | 10 ++++++++++ net/ipv4/af_inet.c | 2 +- net/ipv4/inet_connection_sock.c | 9 ++++----- net/ipv4/inet_hashtables.c | 14 +++++++------- net/ipv4/tcp_ipv4.c | 2 +- net/ipv6/tcp_ipv6.c | 2 +- 6 files changed, 24 insertions(+), 15 deletions(-)
include/net/inet_hashtables.h +10 -0
@@ -170,6 +170,16 @@ struct inet_hashinfo {struct inet_listen_hashbucket *lhash2; }; +static inline struct inet_hashinfo *tcp_or_dccp_get_hashinfo (const struct sock *sk) +{ +#if IS_ENABLED (CONFIG_IP_DCCP ) +return sk->sk_prot->h.hashinfo ? : +sock_net (sk)->ipv4.tcp_death_row.hashinfo; +#else +return sock_net (sk)->ipv4.tcp_death_row.hashinfo; +#endif +} +static inline struct inet_listen_hashbucket *inet_lhash2_bucket (struct inet_hashinfo *h,u32 hash) {
net/ipv4/af_inet.c +1 -1
@@ -1250,7 +1250,7 @@ static int inet_sk_reselect_saddr(struct sock *sk) } prev_addr_hashbucket = -inet_bhashfn_portaddr (sk->sk_prot->h.hashinfo, sk, +inet_bhashfn_portaddr (tcp_or_dccp_get_hashinfo (sk), sk,sock_net (sk), inet->inet_num); inet->inet_saddr = inet->inet_rcv_saddr = new_saddr;
net/ipv4/inet_connection_sock.c +4 -5
@@ -285,7 +285,7 @@ inet_csk_find_open_port(const struct sock *sk, struct inet_bind_bucket **tb_ret,struct inet_bind2_bucket **tb2_ret, struct inet_bind_hashbucket **head2_ret, int *port_ret) { - struct inet_hashinfo *hinfo = sk->sk_prot->h.hashinfo; + struct inet_hashinfo *hinfo = tcp_or_dccp_get_hashinfo(sk);int i, low, high, attempt_half, port, l3mdev;struct inet_bind_hashbucket *head, *head2;struct net *net =sock_net (sk);
@@ -467,8 +467,8 @@ void inet_csk_update_fastreuse(struct inet_bind_bucket *tb,*/ int inet_csk_get_port (struct sock *sk,unsigned short snum) { +struct inet_hashinfo *hinfo =tcp_or_dccp_get_hashinfo (sk);bool reuse = sk->sk_reuse && sk->sk_state !=TCP_LISTEN ; -struct inet_hashinfo *hinfo = sk->sk_prot->h.hashinfo;bool found_port = false, check_bind_conflict = true;bool bhash_created = false, bhash2_created = false;struct inet_bind_hashbucket *head, *head2;
@@ -910,10 +910,9 @@ static bool reqsk_queue_unlink(struct request_sock *req)bool found = false;if (sk_hashed (sk)) { -struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo; -spinlock_t *lock; +struct inet_hashinfo *hashinfo =tcp_or_dccp_get_hashinfo (sk); +spinlock_t *lock =inet_ehash_lockp (hashinfo, req->rsk_hash); - lock =inet_ehash_lockp (hashinfo, req->rsk_hash);spin_lock (lock); found =__sk_nulls_del_node_init_rcu (sk);spin_unlock (lock);
net/ipv4/inet_hashtables.c +7 -7
@@ -168,7 +168,7 @@ void inet_bind_hash(struct sock *sk, struct inet_bind_bucket *tb,*/ static void __inet_put_port (struct sock *sk) { -struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo; +struct inet_hashinfo *hashinfo =tcp_or_dccp_get_hashinfo (sk);struct inet_bind_hashbucket *head, *head2;struct net *net =sock_net (sk);struct inet_bind_bucket *tb;
@@ -208,7 +208,7 @@ EXPORT_SYMBOL(inet_put_port);int __inet_inherit_port (const struct sock *sk,struct sock *child) { -struct inet_hashinfo *table = sk->sk_prot->h.hashinfo; +struct inet_hashinfo *table =tcp_or_dccp_get_hashinfo (sk);unsigned short port =inet_sk (child)->inet_num;struct inet_bind_hashbucket *head, *head2;bool created_inet_bind_bucket = false;
@@ -629,7 +629,7 @@ static bool inet_ehash_lookup_by_sk(struct sock *sk,*/ bool inet_ehash_insert (struct sock *sk,struct sock *osk,bool *found_dup_sk) { -struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo; +struct inet_hashinfo *hashinfo =tcp_or_dccp_get_hashinfo (sk);struct inet_ehash_bucket *head;struct hlist_nulls_head *list;spinlock_t *lock;
@@ -701,7 +701,7 @@ static int inet_reuseport_add_sock(struct sock *sk,int __inet_hash (struct sock *sk,struct sock *osk) { -struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo; +struct inet_hashinfo *hashinfo =tcp_or_dccp_get_hashinfo (sk);struct inet_listen_hashbucket *ilb2;int err =0 ;
@@ -747,7 +747,7 @@ EXPORT_SYMBOL_GPL(inet_hash);void inet_unhash (struct sock *sk) { -struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo; +struct inet_hashinfo *hashinfo =tcp_or_dccp_get_hashinfo (sk);if (sk_unhashed (sk))return ;
@@ -834,7 +834,7 @@ inet_bind2_bucket_find(const struct inet_bind_hashbucket *head, const struct netstruct inet_bind_hashbucket *inet_bhash2_addr_any_hashbucket (const struct sock *sk,const struct net *net,int port) { -struct inet_hashinfo *hinfo = sk->sk_prot->h.hashinfo; +struct inet_hashinfo *hinfo =tcp_or_dccp_get_hashinfo (sk);u32 hash;#if IS_ENABLED (CONFIG_IPV6 )struct in6_addr addr_any = {};
@@ -850,7 +850,7 @@ inet_bhash2_addr_any_hashbucket(const struct sock *sk, const struct net *net, inint inet_bhash2_update_saddr (struct inet_bind_hashbucket *prev_saddr,struct sock *sk) { -struct inet_hashinfo *hinfo = sk->sk_prot->h.hashinfo; +struct inet_hashinfo *hinfo =tcp_or_dccp_get_hashinfo (sk);struct inet_bind2_bucket *tb2, *new_tb2;int l3mdev =inet_sk_bound_l3mdev (sk);struct inet_bind_hashbucket *head2;
net/ipv4/tcp_ipv4.c +1 -1
@@ -3083,7 +3083,7 @@ struct proto tcp_prot = { .slab_flags = SLAB_TYPESAFE_BY_RCU, .twsk_prot = &tcp_timewait_sock_ops, .rsk_prot = &tcp_request_sock_ops, - .h.hashinfo = &tcp_hashinfo, + .h.hashinfo = NULL, .no_autobind = true, .diag_destroy = tcp_abort, };
net/ipv6/tcp_ipv6.c +1 -1
@@ -2194,7 +2194,7 @@ struct proto tcpv6_prot = { .slab_flags = SLAB_TYPESAFE_BY_RCU, .twsk_prot = &tcp6_timewait_sock_ops, .rsk_prot = &tcp6_request_sock_ops, - .h.hashinfo = &tcp_hashinfo, + .h.hashinfo = NULL, .no_autobind = true, .diag_destroy = tcp_abort, };
Signature · 1 line
2.30.2
PATCH v5 4/6 tcp: Access &tcp_hashinfo via net.
In reply to Kuniyuki IwashimaWe will soon introduce an optional per-netns ehash.
This means we cannot use tcp_hashinfo directly in most places.
Instead, access it via net->ipv4.tcp_death_row.hashinfo.
The access will be valid only while initialising tcp_hashinfo itself and creating/destroying each netns.
--- .../chelsio/inline_crypto/chtls/chtls_cm.c | 5 +- .../mellanox/mlx5/core/en_accel/ktls_rx.c | 5 +- .../net/ethernet/netronome/nfp/crypto/tls.c | 5 +- net/core/filter.c | 5 +- net/ipv4/esp4.c | 3 +- net/ipv4/inet_hashtables.c | 2 +- net/ipv4/netfilter/nf_socket_ipv4.c | 4 +- net/ipv4/netfilter/nf_tproxy_ipv4.c | 16 ++--- net/ipv4/tcp_diag.c | 18 ++++- net/ipv4/tcp_ipv4.c | 72 +++++++++++-------- net/ipv4/tcp_minisocks.c | 2 +- net/ipv6/esp6.c | 3 +- net/ipv6/inet6_hashtables.c | 4 +- net/ipv6/netfilter/nf_socket_ipv6.c | 4 +- net/ipv6/netfilter/nf_tproxy_ipv6.c | 8 +-- net/ipv6/tcp_ipv6.c | 23 +++--- net/mptcp/mptcp_diag.c | 7 +- 17 files changed, 108 insertions(+), 78 deletions(-)
drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.c +2 -3
@@ -1069,8 +1069,7 @@ static void chtls_pass_accept_rpl(struct sk_buff *skb,cxgb4_l2t_send (csk->egress_dev, skb, csk->l2t_entry); } -static void inet_inherit_port (struct inet_hashinfo *hash_info, -struct sock *lsk,struct sock *newsk) +static void inet_inherit_port (struct sock *lsk,struct sock *newsk) {local_bh_disable ();__inet_inherit_port (lsk, newsk);
@@ -1240,7 +1239,7 @@ static struct sock *chtls_recv_sock(struct sock *lsk, ipv4.sysctl_tcp_window_scaling), tp->window_clamp);neigh_release (n); -inet_inherit_port (&tcp_hashinfo, lsk, newsk); +inet_inherit_port (lsk, newsk);csk_set_flag (csk,CSK_CONN_INLINE );bh_unlock_sock (newsk);/* tcp_create_openreq_child ->sk_clone_lock */
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c +3 -2
@@ -461,6 +461,7 @@ static void resync_update_sn(struct mlx5e_rq *rq, struct sk_buff *skb) {struct ethhdr *eth = (struct ethhdr *)(skb->data);struct net_device *netdev = rq->netdev; +struct net *net =dev_net (netdev);struct sock *sk =NULL ;unsigned int datalen;struct iphdr *iph;
@@ -475,7 +476,7 @@ static void resync_update_sn(struct mlx5e_rq *rq, struct sk_buff *skb) depth +=sizeof (struct iphdr ); th = (void *)iph +sizeof (struct iphdr ); - sk =inet_lookup_established (dev_net (netdev), &tcp_hashinfo, + sk =inet_lookup_established (net, net->ipv4.tcp_death_row.hashinfo, iph->saddr, th->source, iph->daddr, th->dest, netdev->ifindex);#if IS_ENABLED (CONFIG_IPV6 )
@@ -485,7 +486,7 @@ static void resync_update_sn(struct mlx5e_rq *rq, struct sk_buff *skb) depth +=sizeof (struct ipv6hdr ); th = (void *)ipv6h +sizeof (struct ipv6hdr ); - sk =__inet6_lookup_established (dev_net (netdev), &tcp_hashinfo, + sk =__inet6_lookup_established (net, net->ipv4.tcp_death_row.hashinfo, &ipv6h->saddr, th->source, &ipv6h->daddr,ntohs (th->dest), netdev->ifindex,0 );
drivers/net/ethernet/netronome/nfp/crypto/tls.c +3 -2
@@ -474,6 +474,7 @@ int nfp_net_tls_rx_resync_req(struct net_device *netdev, {struct nfp_net *nn =netdev_priv (netdev);struct nfp_net_tls_offload_ctx *ntls; +struct net *net =dev_net (netdev);struct ipv6hdr *ipv6h;struct tcphdr *th;struct iphdr *iph;
@@ -494,13 +495,13 @@ int nfp_net_tls_rx_resync_req(struct net_device *netdev,switch (ipv6h->version) {case 4 : - sk =inet_lookup_established (dev_net (netdev), &tcp_hashinfo, + sk =inet_lookup_established (net, net->ipv4.tcp_death_row.hashinfo, iph->saddr, th->source, iph->daddr, th->dest, netdev->ifindex);break ;#if IS_ENABLED (CONFIG_IPV6 )case 6 : - sk =__inet6_lookup_established (dev_net (netdev), &tcp_hashinfo, + sk =__inet6_lookup_established (net, net->ipv4.tcp_death_row.hashinfo, &ipv6h->saddr, th->source, &ipv6h->daddr,ntohs (th->dest), netdev->ifindex,0 );
net/core/filter.c +3 -2
@@ -6373,6 +6373,7 @@ static const struct bpf_func_proto bpf_lwt_seg6_adjust_srh_proto = {static struct sock *sk_lookup (struct net *net,struct bpf_sock_tuple *tuple,int dif,int sdif,u8 family,u8 proto) { +struct inet_hashinfo *hinfo = net->ipv4.tcp_death_row.hashinfo;bool refcounted = false;struct sock *sk =NULL ;
@@ -6381,7 +6382,7 @@ static struct sock *sk_lookup(struct net *net, struct bpf_sock_tuple *tuple,__be32 dst4 = tuple->ipv4.daddr;if (proto ==IPPROTO_TCP ) - sk =__inet_lookup (net, &tcp_hashinfo,NULL ,0 , + sk =__inet_lookup (net, hinfo,NULL ,0 , src4, tuple->ipv4.sport, dst4, tuple->ipv4.dport, dif, sdif, &refcounted);
@@ -6395,7 +6396,7 @@ static struct sock *sk_lookup(struct net *net, struct bpf_sock_tuple *tuple,struct in6_addr *dst6 = (struct in6_addr *)&tuple->ipv6.daddr;if (proto ==IPPROTO_TCP ) - sk =__inet6_lookup (net, &tcp_hashinfo,NULL ,0 , + sk =__inet6_lookup (net, hinfo,NULL ,0 , src6, tuple->ipv6.sport, dst6,ntohs (tuple->ipv6.dport), dif, sdif, &refcounted);
net/ipv4/esp4.c +2 -1
@@ -134,6 +134,7 @@ static void esp_free_tcp_sk(struct rcu_head *head)static struct sock *esp_find_tcp_sk (struct xfrm_state *x) {struct xfrm_encap_tmpl *encap = x->encap; +struct net *net =xs_net (x);struct esp_tcp_sk *esk;__be16 sport, dport;struct sock *nsk;
@@ -160,7 +161,7 @@ static struct sock *esp_find_tcp_sk(struct xfrm_state *x) }spin_unlock_bh (&x->lock); - sk =inet_lookup_established (xs_net (x), &tcp_hashinfo, x->id.daddr.a4, + sk =inet_lookup_established (net, net->ipv4.tcp_death_row.hashinfo, x->id.daddr.a4, dport, x->props.saddr.a4, sport,0 );if (!sk)return ERR_PTR (-ENOENT );
net/ipv4/inet_hashtables.c +1 -1
@@ -386,7 +386,7 @@ static inline struct sock *inet_lookup_run_bpf(struct net *net,struct sock *sk, *reuse_sk;bool no_reuseport; -if (hashinfo != &tcp_hashinfo) +if (hashinfo != net->ipv4.tcp_death_row.hashinfo)return NULL ;/* only TCP is supported */ no_reuseport = bpf_sk_lookup_run_v4(net, IPPROTO_TCP, saddr, sport,
net/ipv4/netfilter/nf_socket_ipv4.c +2 -2
@@ -71,8 +71,8 @@ nf_socket_get_sock_v4(struct net *net, struct sk_buff *skb, const int doff, {switch (protocol) {case IPPROTO_TCP : -return inet_lookup (net, &tcp_hashinfo, skb, doff, - saddr, sport, daddr, dport, +return inet_lookup (net, net->ipv4.tcp_death_row.hashinfo, + skb, doff, saddr, sport, daddr, dport, in->ifindex);case IPPROTO_UDP : return udp4_lib_lookup(net, saddr, sport, daddr, dport,
net/ipv4/netfilter/nf_tproxy_ipv4.c +7 -9
@@ -79,6 +79,7 @@ nf_tproxy_get_sock_v4(struct net *net, struct sk_buff *skb,const struct net_device *in, const enum nf_tproxy_lookup_t lookup_type) { + struct inet_hashinfo *hinfo = net->ipv4.tcp_death_row.hashinfo;struct sock *sk;switch (protocol) {
@@ -92,12 +93,10 @@ nf_tproxy_get_sock_v4(struct net *net, struct sk_buff *skb,switch (lookup_type) {case NF_TPROXY_LOOKUP_LISTENER : - sk =inet_lookup_listener (net, &tcp_hashinfo, skb, -ip_hdrlen (skb) + -__tcp_hdrlen (hp), - saddr, sport, - daddr, dport, - in->ifindex,0 ); + sk =inet_lookup_listener (net, hinfo, skb, +ip_hdrlen (skb) +__tcp_hdrlen (hp), + saddr, sport, daddr, dport, + in->ifindex,0 );if (sk && !refcount_inc_not_zero (&sk->sk_refcnt)) sk =NULL ;
@@ -108,9 +107,8 @@ nf_tproxy_get_sock_v4(struct net *net, struct sk_buff *skb,*/ break ;case NF_TPROXY_LOOKUP_ESTABLISHED : - sk =inet_lookup_established (net, &tcp_hashinfo, - saddr, sport, daddr, dport, - in->ifindex); + sk =inet_lookup_established (net, hinfo, saddr, sport, + daddr, dport, in->ifindex);break ;default :BUG ();
net/ipv4/tcp_diag.c +15 -3
@@ -181,13 +181,21 @@ static size_t tcp_diag_get_aux_size(struct sock *sk, bool net_admin)static void tcp_diag_dump (struct sk_buff *skb,struct netlink_callback *cb,const struct inet_diag_req_v2 *r) { -inet_diag_dump_icsk (&tcp_hashinfo, skb, cb, r); +struct inet_hashinfo *hinfo; + + hinfo =sock_net (cb->skb->sk)->ipv4.tcp_death_row.hashinfo; + +inet_diag_dump_icsk (hinfo, skb, cb, r); }static int tcp_diag_dump_one (struct netlink_callback *cb,const struct inet_diag_req_v2 *req) { -return inet_diag_dump_one_icsk (&tcp_hashinfo, cb, req); +struct inet_hashinfo *hinfo; + + hinfo =sock_net (cb->skb->sk)->ipv4.tcp_death_row.hashinfo; + +return inet_diag_dump_one_icsk (hinfo, cb, req); }#ifdef CONFIG_INET_DIAG_DESTROY
@@ -195,9 +203,13 @@ static int tcp_diag_destroy(struct sk_buff *in_skb, const struct inet_diag_req_v2 *req) {struct net *net =sock_net (in_skb->sk); -struct sock *sk =inet_diag_find_one_icsk (net, &tcp_hashinfo, req); +struct inet_hashinfo *hinfo; +struct sock *sk;int err; + hinfo = net->ipv4.tcp_death_row.hashinfo; + sk =inet_diag_find_one_icsk (net, hinfo, req); +if (IS_ERR (sk))return PTR_ERR (sk);
net/ipv4/tcp_ipv4.c +43 -29
250 lines not on this page · raw message
net/ipv4/tcp_minisocks.c +1 -1
@@ -319,7 +319,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)/* Linkage updates. * Note that access to tw after this point is illegal. */ -inet_twsk_hashdance (tw, sk, &tcp_hashinfo); +inet_twsk_hashdance (tw, sk, net->ipv4.tcp_death_row.hashinfo);local_bh_enable (); }else {/* Sorry, if we're out of memory, just CLOSE this
net/ipv6/esp6.c +2 -1
@@ -151,6 +151,7 @@ static void esp_free_tcp_sk(struct rcu_head *head)static struct sock *esp6_find_tcp_sk (struct xfrm_state *x) {struct xfrm_encap_tmpl *encap = x->encap; +struct net *net =xs_net (x);struct esp_tcp_sk *esk;__be16 sport, dport;struct sock *nsk;
@@ -177,7 +178,7 @@ static struct sock *esp6_find_tcp_sk(struct xfrm_state *x) }spin_unlock_bh (&x->lock); - sk =__inet6_lookup_established (xs_net (x), &tcp_hashinfo, &x->id.daddr.in6, + sk =__inet6_lookup_established (net, net->ipv4.tcp_death_row.hashinfo, &x->id.daddr.in6, dport, &x->props.saddr.in6,ntohs (sport),0 ,0 );if (!sk)return ERR_PTR (-ENOENT );
net/ipv6/inet6_hashtables.c +1 -3
@@ -21,8 +21,6 @@#include <net/ip.h> #include <net/sock_reuseport.h> -extern struct inet_hashinfo tcp_hashinfo; -u32 inet6_ehashfn (const struct net *net,const struct in6_addr *laddr,const u16 lport,const struct in6_addr *faddr,const __be16 fport)
@@ -169,7 +167,7 @@ static inline struct sock *inet6_lookup_run_bpf(struct net *net,struct sock *sk, *reuse_sk;bool no_reuseport; -if (hashinfo != &tcp_hashinfo) +if (hashinfo != net->ipv4.tcp_death_row.hashinfo)return NULL ;/* only TCP is supported */ no_reuseport = bpf_sk_lookup_run_v6(net, IPPROTO_TCP, saddr, sport,
net/ipv6/netfilter/nf_socket_ipv6.c +2 -2
@@ -83,8 +83,8 @@ nf_socket_get_sock_v6(struct net *net, struct sk_buff *skb, int doff, {switch (protocol) {case IPPROTO_TCP : -return inet6_lookup (net, &tcp_hashinfo, skb, doff, - saddr, sport, daddr, dport, +return inet6_lookup (net, net->ipv4.tcp_death_row.hashinfo, + skb, doff, saddr, sport, daddr, dport, in->ifindex);case IPPROTO_UDP : return udp6_lib_lookup(net, saddr, sport, daddr, dport,
net/ipv6/netfilter/nf_tproxy_ipv6.c +4 -4
@@ -80,6 +80,7 @@ nf_tproxy_get_sock_v6(struct net *net, struct sk_buff *skb, int thoff,const struct net_device *in, const enum nf_tproxy_lookup_t lookup_type) { + struct inet_hashinfo *hinfo = net->ipv4.tcp_death_row.hashinfo;struct sock *sk;switch (protocol) {
@@ -93,7 +94,7 @@ nf_tproxy_get_sock_v6(struct net *net, struct sk_buff *skb, int thoff,switch (lookup_type) {case NF_TPROXY_LOOKUP_LISTENER : - sk = inet6_lookup_listener(net, &tcp_hashinfo, skb, + sk = inet6_lookup_listener(net, hinfo, skb, thoff + __tcp_hdrlen(hp), saddr, sport, daddr, ntohs(dport),
@@ -108,9 +109,8 @@ nf_tproxy_get_sock_v6(struct net *net, struct sk_buff *skb, int thoff,*/ break ;case NF_TPROXY_LOOKUP_ESTABLISHED : - sk =__inet6_lookup_established (net, &tcp_hashinfo, - saddr, sport, daddr,ntohs (dport), - in->ifindex,0 ); + sk =__inet6_lookup_established (net, hinfo, saddr, sport, daddr, +ntohs (dport), in->ifindex,0 );break ;default :BUG ();
net/ipv6/tcp_ipv6.c +12 -11
@@ -287,12 +287,14 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr, goto failure; } + tcp_death_row = &sock_net (sk)->ipv4.tcp_death_row; +if (!saddr) {struct inet_bind_hashbucket *prev_addr_hashbucket =NULL ;struct in6_addr prev_v6_rcv_saddr;if (icsk->icsk_bind2_hash) { - prev_addr_hashbucket =inet_bhashfn_portaddr (&tcp_hashinfo, + prev_addr_hashbucket =inet_bhashfn_portaddr (tcp_death_row->hashinfo, sk, net, inet->inet_num); prev_v6_rcv_saddr = sk->sk_v6_rcv_saddr; }
@@ -325,7 +327,6 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr, inet->inet_dport = usin->sin6_port;tcp_set_state (sk,TCP_SYN_SENT ); - tcp_death_row = &net->ipv4.tcp_death_row; err =inet6_hash_connect (tcp_death_row, sk);if (err) goto late_failure;
@@ -402,7 +403,7 @@ static int tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,bool fatal;int err; - sk =__inet6_lookup_established (net, &tcp_hashinfo, + sk =__inet6_lookup_established (net, net->ipv4.tcp_death_row.hashinfo, &hdr->daddr, th->dest, &hdr->saddr,ntohs (th->source), skb->dev->ifindex,inet6_sdif (skb));
@@ -1035,11 +1036,10 @@ static void tcp_v6_send_reset(const struct sock *sk, struct sk_buff *skb)* Incoming packet is checked with md5 hash with finding key, * no RST generated if md5 hash doesn't match. */ - sk1 =inet6_lookup_listener (net, - &tcp_hashinfo,NULL ,0 , - &ipv6h->saddr, - th->source, &ipv6h->daddr, -ntohs (th->source), dif, sdif); + sk1 =inet6_lookup_listener (net, net->ipv4.tcp_death_row.hashinfo, +NULL ,0 , &ipv6h->saddr, th->source, + &ipv6h->daddr,ntohs (th->source), + dif, sdif);if (!sk1) goto out;
@@ -1637,7 +1637,7 @@ INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb) hdr =ipv6_hdr (skb); lookup: - sk =__inet6_lookup_skb (&tcp_hashinfo, skb,__tcp_hdrlen (th), + sk =__inet6_lookup_skb (net->ipv4.tcp_death_row.hashinfo, skb,__tcp_hdrlen (th), th->source, th->dest,inet6_iif (skb), sdif, &refcounted);if (!sk)
@@ -1812,7 +1812,7 @@ INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb) {struct sock *sk2; - sk2 = inet6_lookup_listener(dev_net(skb->dev), &tcp_hashinfo, + sk2 = inet6_lookup_listener(net, net->ipv4.tcp_death_row.hashinfo, skb, __tcp_hdrlen(th), &ipv6_hdr(skb)->saddr, th->source, &ipv6_hdr(skb)->daddr,
@@ -1845,6 +1845,7 @@ INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb)void tcp_v6_early_demux (struct sk_buff *skb) { +struct net *net =dev_net (skb->dev);const struct ipv6hdr *hdr;const struct tcphdr *th;struct sock *sk;
@@ -1862,7 +1863,7 @@ void tcp_v6_early_demux(struct sk_buff *skb)return ;/* Note : We use inet6_iif() here, not tcp_v6_iif() */ - sk =__inet6_lookup_established (dev_net (skb->dev), &tcp_hashinfo, + sk =__inet6_lookup_established (net, net->ipv4.tcp_death_row.hashinfo, &hdr->saddr, th->source, &hdr->daddr,ntohs (th->dest),inet6_iif (skb),inet6_sdif (skb));
net/mptcp/mptcp_diag.c +5 -2
@@ -81,15 +81,18 @@ static void mptcp_diag_dump_listeners(struct sk_buff *skb, struct netlink_callbastruct mptcp_diag_ctx *diag_ctx = (void *)cb->ctx;struct nlattr *bc = cb_data->inet_diag_nla_bc;struct net *net =sock_net (skb->sk); +struct inet_hashinfo *hinfo;int i; -for (i = diag_ctx->l_slot; i <= tcp_hashinfo.lhash2_mask; i++) { + hinfo = net->ipv4.tcp_death_row.hashinfo; + +for (i = diag_ctx->l_slot; i <= hinfo->lhash2_mask; i++) {struct inet_listen_hashbucket *ilb;struct hlist_nulls_node *node;struct sock *sk;int num =0 ; - ilb = &tcp_hashinfo.lhash2[i]; + ilb = &hinfo->lhash2[i];rcu_read_lock ();spin_lock (&ilb->lock);
Signature · 1 line
2.30.2
PATCH v5 5/6 tcp: Save unnecessary inet_twsk_purge() calls.
In reply to Kuniyuki IwashimaWhile destroying netns, we call inet_twsk_purge() in tcp_sk_exit_batch() and tcpv6_net_exit_batch() for AF_INET and AF_INET6. These commands trigger the kernel to walk through the potentially big ehash twice even though the netns has no TIME_WAIT sockets.
# ip netns add test # ip netns del test
or
# unshare -n /bin/true >/dev/null
When tw_refcount is 1, we need not call inet_twsk_purge() at least for the net. We can save such unneeded iterations if all netns in net_exit_list have no TIME_WAIT sockets. This change eliminates the tax by the additional unshare() described in the next patch to guarantee the per-netns ehash size.
Tested:
# mount -t debugfs none /sys/kernel/debug/
# echo cleanup_net > /sys/kernel/debug/tracing/set_ftrace_filter
# echo inet_twsk_purge >> /sys/kernel/debug/tracing/set_ftrace_filter
# echo function > /sys/kernel/debug/tracing/current_tracer
# cat ./add_del_unshare.sh
for i in `seq 1 40`
do
(for j in `seq 1 100` ; do unshare -n /bin/true >/dev/null ; done) &
done
wait;
# ./add_del_unshare.shBefore the patch:
# cat /sys/kernel/debug/tracing/trace_pipe
kworker/u128:0-8 [031] ...1. 174.162765: cleanup_net <-process_one_work
kworker/u128:0-8 [031] ...1. 174.240796: inet_twsk_purge <-cleanup_net
kworker/u128:0-8 [032] ...1. 174.244759: inet_twsk_purge <-tcp_sk_exit_batch
kworker/u128:0-8 [034] ...1. 174.290861: cleanup_net <-process_one_work
kworker/u128:0-8 [039] ...1. 175.245027: inet_twsk_purge <-cleanup_net
kworker/u128:0-8 [046] ...1. 175.290541: inet_twsk_purge <-tcp_sk_exit_batch
kworker/u128:0-8 [037] ...1. 175.321046: cleanup_net <-process_one_work
kworker/u128:0-8 [024] ...1. 175.941633: inet_twsk_purge <-cleanup_net
kworker/u128:0-8 [025] ...1. 176.242539: inet_twsk_purge <-tcp_sk_exit_batchAfter:
# cat /sys/kernel/debug/tracing/trace_pipe
kworker/u128:0-8 [038] ...1. 428.116174: cleanup_net <-process_one_work
kworker/u128:0-8 [038] ...1. 428.262532: cleanup_net <-process_one_work
kworker/u128:0-8 [030] ...1. 429.292645: cleanup_net <-process_one_work--- include/net/tcp.h | 1 + net/ipv4/tcp_ipv4.c | 2 +- net/ipv4/tcp_minisocks.c | 15 +++++++++++++++ net/ipv6/tcp_ipv6.c | 2 +- 4 files changed, 18 insertions(+), 2 deletions(-)
include/net/tcp.h +1 -0
@@ -346,6 +346,7 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb);void tcp_rcv_space_adjust (struct sock *sk);int tcp_twsk_unique (struct sock *sk,struct sock *sktw,void *twp);void tcp_twsk_destructor (struct sock *sk); +void tcp_twsk_purge (struct list_head *net_exit_list,int family);ssize_t tcp_splice_read (struct socket *sk,loff_t *ppos,struct pipe_inode_info *pipe,size_t len,unsigned int flags);
net/ipv4/tcp_ipv4.c +1 -1
@@ -3206,7 +3206,7 @@ static void __net_exit tcp_sk_exit_batch(struct list_head *net_exit_list) {struct net *net; -inet_twsk_purge (&tcp_hashinfo,AF_INET ); +tcp_twsk_purge (net_exit_list,AF_INET );list_for_each_entry (net, net_exit_list, exit_list) {WARN_ON_ONCE (!refcount_dec_and_test (&net->ipv4.tcp_death_row.tw_refcount));
net/ipv4/tcp_minisocks.c +15 -0
@@ -347,6 +347,21 @@ void tcp_twsk_destructor(struct sock *sk) }EXPORT_SYMBOL_GPL (tcp_twsk_destructor); +void tcp_twsk_purge (struct list_head *net_exit_list,int family) +{ +struct net *net; + +list_for_each_entry (net, net_exit_list, exit_list) { +/* The last refcount is decremented in tcp_sk_exit_batch() */ +if (refcount_read (&net->ipv4.tcp_death_row.tw_refcount) ==1 ) +continue ; + +inet_twsk_purge (&tcp_hashinfo, family); +break ; + } +} +EXPORT_SYMBOL_GPL (tcp_twsk_purge); +/* Warning : This function is called without sk_listener being locked. * Be sure to read socket fields once, as their value could change under us. */
net/ipv6/tcp_ipv6.c +1 -1
@@ -2229,7 +2229,7 @@ static void __net_exit tcpv6_net_exit(struct net *net) static void __net_exit tcpv6_net_exit_batch(struct list_head *net_exit_list) { -inet_twsk_purge (&tcp_hashinfo,AF_INET6 ); +tcp_twsk_purge (net_exit_list,AF_INET6 ); }static struct pernet_operations tcpv6_net_ops = {
Signature · 1 line
2.30.2
PATCH v5 6/6 tcp: Introduce optional per-netns ehash.
In reply to Kuniyuki IwashimaThe more sockets we have in the hash table, the longer we spend looking up the socket. While running a number of small workloads on the same host, they penalise each other and cause performance degradation.
The root cause might be a single workload that consumes much more resources than the others. It often happens on a cloud service where different workloads share the same computing resource.
On EC2 c5.24xlarge instance (196 GiB memory and 524288 (1Mi / 2) ehash entries), after running iperf3 in different netns, creating 24Mi sockets without data transfer in the root netns causes about 10% performance regression for the iperf3's connection.
thash_entries sockets length Gbps
524288 1 1 50.7 24Mi 48 45.1
It is basically related to the length of the list of each hash bucket. For testing purposes to see how performance drops along the length, I set 131072 (1Mi / 8) to thash_entries, and here's the result.
thash_entries sockets length Gbps
131072 1 1 50.7 1Mi 8 49.9 2Mi 16 48.9 4Mi 32 47.3 8Mi 64 44.6 16Mi 128 40.6 24Mi 192 36.3 32Mi 256 32.5 40Mi 320 27.0 48Mi 384 25.0
To resolve the socket lookup degradation, we introduce an optional per-netns hash table for TCP, but it's just ehash, and we still share the global bhash, bhash2 and lhash2.
With a smaller ehash, we can look up non-listener sockets faster and isolate such noisy neighbours. In addition, we can reduce lock contention.
We can control the ehash size by a new sysctl knob. However, depending on workloads, it will require very sensitive tuning, so we disable the feature by default (net.ipv4.tcp_child_ehash_entries == 0). Moreover, we can fall back to using the global ehash in case we fail to allocate enough memory for a new ehash. The maximum size is 16Mi, which is large enough that even if we have 48Mi sockets, the average list length is 3, and regression would be less than 1%.
We can check the current ehash size by another read-only sysctl knob, net.ipv4.tcp_ehash_entries. A negative value means the netns shares the global ehash (per-netns ehash is disabled or failed to allocate memory).
# dmesg | cut -d ' ' -f 5- | grep "established hash" TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)
# sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = 524288 # can be changed by thash_entries
# sysctl net.ipv4.tcp_child_ehash_entries net.ipv4.tcp_child_ehash_entries = 0 # disabled by default
# ip netns add test1 # ip netns exec test1 sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = -524288 # share the global ehash
# sysctl -w net.ipv4.tcp_child_ehash_entries=100 net.ipv4.tcp_child_ehash_entries = 100
# ip netns add test2 # ip netns exec test2 sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = 128 # own a per-netns ehash with 2^n buckets
When more than two processes in the same netns create per-netns ehash concurrently with different sizes, we need to guarantee the size in one of the following ways:
1) Share the global ehash and create per-netns ehash
First, unshare() with tcp_child_ehash_entries==0. It creates dedicated netns sysctl knobs where we can safely change tcp_child_ehash_entries and clone()/unshare() to create a per-netns ehash.
2) Control write on sysctl by BPF
We can use BPF_PROG_TYPE_CGROUP_SYSCTL to allow/deny read/write on sysctl knobs.
Note the default values of two sysctl knobs depend on the ehash size and should be tuned carefully:
tcp_max_tw_buckets : tcp_child_ehash_entries / 2 tcp_max_syn_backlog : max(128, tcp_child_ehash_entries / 128)
As a bonus, we can dismantle netns faster. Currently, while destroying netns, we call inet_twsk_purge(), which walks through the global ehash. It can be potentially big because it can have many sockets other than TIME_WAIT in all netns. Splitting ehash changes that situation, where it's only necessary for inet_twsk_purge() to clean up TIME_WAIT sockets in each netns.
With regard to this, we do not free the per-netns ehash in inet_twsk_kill() to avoid UAF while iterating the per-netns ehash in inet_twsk_purge(). Instead, we do it in tcp_sk_exit_batch() after calling tcp_twsk_purge() to keep it protocol-family-independent.
In the future, we could optimise ehash lookup/iteration further by removing netns comparison for the per-netns ehash.
--- Documentation/networking/ip-sysctl.rst | 23 +++++++++++ include/net/inet_hashtables.h | 6 +++ include/net/netns/ipv4.h | 1 + net/dccp/proto.c | 2 + net/ipv4/inet_hashtables.c | 57 ++++++++++++++++++++++++++ net/ipv4/sysctl_net_ipv4.c | 39 ++++++++++++++++++ net/ipv4/tcp.c | 1 + net/ipv4/tcp_ipv4.c | 38 ++++++++++++++--- net/ipv4/tcp_minisocks.c | 9 +++- 9 files changed, 168 insertions(+), 8 deletions(-)
Documentation/networking/ip-sysctl.rst +23 -0
@@ -1040,6 +1040,29 @@ tcp_challenge_ack_limit - INTEGER TCP stack implements per TCP socket limits anyway. Default: INT_MAX (unlimited) +tcp_ehash_entries - INTEGER + Show the number of hash buckets for TCP sockets in the current + networking namespace. + + A negative value means the networking namespace does not own its + hash buckets and shares the initial networking namespace's one. + +tcp_child_ehash_entries - INTEGER + Control the number of hash buckets for TCP sockets in the child + networking namespace, which must be set before clone() or unshare(). + + If the value is not 0, the kernel uses a value rounded up to 2^n + as the actual hash bucket size. 0 is a special value, meaning + the child networking namespace will share the initial networking + namespace's hash buckets. + + Note that the child will use the global one in case the kernel + fails to allocate enough memory. + + Possible values: 0, 2^n (n: 0 - 24 (16Mi)) + + Default: 0 + UDP variables =============
include/net/inet_hashtables.h +6 -0
@@ -168,6 +168,8 @@ struct inet_hashinfo {/* The 2nd listener table hashed by local port and address */ unsigned int lhash2_mask;struct inet_listen_hashbucket *lhash2; + +bool pernet; };static inline struct inet_hashinfo *tcp_or_dccp_get_hashinfo (const struct sock *sk)
@@ -214,6 +216,10 @@ static inline void inet_ehash_locks_free(struct inet_hashinfo *hashinfo) hashinfo->ehash_locks =NULL ; } +struct inet_hashinfo *inet_pernet_hashinfo_alloc (struct inet_hashinfo *hashinfo, +unsigned int ehash_entries); +void inet_pernet_hashinfo_free (struct inet_hashinfo *hashinfo); +struct inet_bind_bucket * inet_bind_bucket_create(struct kmem_cache *cachep, struct net *net, struct inet_bind_hashbucket *head,
include/net/netns/ipv4.h +1 -0
@@ -171,6 +171,7 @@ struct netns_ipv4 {int sysctl_tcp_pacing_ca_ratio;int sysctl_tcp_wmem[3 ];int sysctl_tcp_rmem[3 ]; +unsigned int sysctl_tcp_child_ehash_entries;unsigned long sysctl_tcp_comp_sack_delay_ns;unsigned long sysctl_tcp_comp_sack_slack_ns;int sysctl_max_syn_backlog;
net/dccp/proto.c +2 -0
@@ -1197,6 +1197,8 @@ static int __init dccp_init(void)INIT_HLIST_HEAD (&dccp_hashinfo.bhash2[i].chain); } + dccp_hashinfo.pernet = false; + rc =dccp_mib_init ();if (rc) goto out_free_dccp_bhash2;
net/ipv4/inet_hashtables.c +57 -0
60 lines not on this page · raw message
net/ipv4/sysctl_net_ipv4.c +39 -0
@@ -39,6 +39,7 @@ static u32 u32_max_div_HZ = UINT_MAX / HZ;static int one_day_secs =24 *3600 ;static u32 fib_multipath_hash_fields_all_mask__maybe_unused =FIB_MULTIPATH_HASH_FIELD_ALL_MASK ; +static unsigned int tcp_child_ehash_entries_max =16 *1024 *1024 ;/* obsolete */ static int sysctl_tcp_low_latency__read_mostly ;
@@ -382,6 +383,29 @@ static int proc_tcp_available_ulp(struct ctl_table *ctl,return ret; } +static int proc_tcp_ehash_entries (struct ctl_table *table,int write, +void *buffer,size_t *lenp,loff_t *ppos) +{ + struct net *net = container_of(table->data, struct net, + ipv4.sysctl_tcp_child_ehash_entries); +struct inet_hashinfo *hinfo = net->ipv4.tcp_death_row.hashinfo; +int tcp_ehash_entries; +struct ctl_table tbl; + + tcp_ehash_entries = hinfo->ehash_mask +1 ; + +/* A negative number indicates that the child netns +* shares the global ehash. +*/ +if (!net_eq (net, &init_net) && !hinfo->pernet) + tcp_ehash_entries *=-1 ; + + tbl.data = &tcp_ehash_entries; + tbl.maxlen =sizeof (int ); + +return proc_dointvec (&tbl, write, buffer, lenp, ppos); +} +#ifdef CONFIG_IP_ROUTE_MULTIPATH static int proc_fib_multipath_hash_policy(struct ctl_table *table, int write, void *buffer, size_t *lenp,
@@ -1320,6 +1344,21 @@ static struct ctl_table ipv4_net_table[] = { .extra1 = SYSCTL_ZERO, .extra2 = SYSCTL_ONE, }, + { + .procname = "tcp_ehash_entries", + .data = &init_net.ipv4.sysctl_tcp_child_ehash_entries, + .mode = 0444, + .proc_handler = proc_tcp_ehash_entries, + }, + { + .procname = "tcp_child_ehash_entries", + .data = &init_net.ipv4.sysctl_tcp_child_ehash_entries, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_douintvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = &tcp_child_ehash_entries_max, + }, { .procname = "udp_rmem_min", .data = &init_net.ipv4.sysctl_udp_rmem_min,
net/ipv4/tcp.c +1 -0
@@ -4790,6 +4790,7 @@ void __init tcp_init(void)INIT_HLIST_HEAD (&tcp_hashinfo.bhash2[i].chain); } + tcp_hashinfo.pernet = false; cnt = tcp_hashinfo.ehash_mask +1 ; sysctl_tcp_max_orphans = cnt /2 ;
net/ipv4/tcp_ipv4.c +32 -6
@@ -3110,10 +3110,38 @@ static void __net_exit tcp_sk_exit(struct net *net) net->ipv4.tcp_congestion_control->owner); } -static int __net_init tcp_sk_init (struct net *net) +static void __net_init tcp_set_hashinfo (struct net *net) { -int cnt; +struct inet_hashinfo *hinfo; +unsigned int ehash_entries; +struct net *old_net; + +if (net_eq (net, &init_net)) + goto fallback; + + old_net = current->nsproxy->net_ns; + ehash_entries =READ_ONCE (old_net->ipv4.sysctl_tcp_child_ehash_entries); +if (!ehash_entries) + goto fallback; + + ehash_entries =roundup_pow_of_two (ehash_entries); + hinfo =inet_pernet_hashinfo_alloc (&tcp_hashinfo, ehash_entries); +if (!hinfo) { +pr_warn ("Failed to allocate TCP ehash (entries: %u) " +"for a netns, fallback to the global one\n" , + ehash_entries); +fallback: + hinfo = &tcp_hashinfo; + ehash_entries = tcp_hashinfo.ehash_mask +1 ; + } + + net->ipv4.tcp_death_row.hashinfo = hinfo; + net->ipv4.tcp_death_row.sysctl_max_tw_buckets = ehash_entries /2 ; + net->ipv4.sysctl_max_syn_backlog =max (128U , ehash_entries /128 ); +} +static int __net_init tcp_sk_init (struct net *net) +{ net->ipv4.sysctl_tcp_ecn =2 ; net->ipv4.sysctl_tcp_ecn_fallback =1 ;
@@ -3140,11 +3168,8 @@ static int __net_init tcp_sk_init(struct net *net) net->ipv4.sysctl_tcp_no_ssthresh_metrics_save =1 ;refcount_set (&net->ipv4.tcp_death_row.tw_refcount,1 ); - cnt = tcp_hashinfo.ehash_mask +1 ; - net->ipv4.tcp_death_row.sysctl_max_tw_buckets = cnt /2 ; - net->ipv4.tcp_death_row.hashinfo = &tcp_hashinfo; +tcp_set_hashinfo (net); - net->ipv4.sysctl_max_syn_backlog =max (128 , cnt /128 ); net->ipv4.sysctl_tcp_sack =1 ; net->ipv4.sysctl_tcp_window_scaling =1 ; net->ipv4.sysctl_tcp_timestamps =1 ;
@@ -3209,6 +3234,7 @@ static void __net_exit tcp_sk_exit_batch(struct list_head *net_exit_list)tcp_twsk_purge (net_exit_list,AF_INET );list_for_each_entry (net, net_exit_list, exit_list) { +inet_pernet_hashinfo_free (net->ipv4.tcp_death_row.hashinfo);WARN_ON_ONCE (!refcount_dec_and_test (&net->ipv4.tcp_death_row.tw_refcount));tcp_fastopen_ctx_destroy (net); }
net/ipv4/tcp_minisocks.c +7 -2
@@ -349,6 +349,7 @@ EXPORT_SYMBOL_GPL(tcp_twsk_destructor);void tcp_twsk_purge (struct list_head *net_exit_list,int family) { +bool purged_once = false;struct net *net;list_for_each_entry (net, net_exit_list, exit_list) {
@@ -356,8 +357,12 @@ void tcp_twsk_purge(struct list_head *net_exit_list, int family)if (refcount_read (&net->ipv4.tcp_death_row.tw_refcount) ==1 )continue ; -inet_twsk_purge (&tcp_hashinfo, family); -break ; +if (net->ipv4.tcp_death_row.hashinfo->pernet) { +inet_twsk_purge (net->ipv4.tcp_death_row.hashinfo, family); + }else if (!purged_once) { +inet_twsk_purge (&tcp_hashinfo, family); + purged_once = true; + } } }EXPORT_SYMBOL_GPL (tcp_twsk_purge);
Signature · 1 line
2.30.2
Quoted 111 lines · Kuniyuki Iwashima
The more sockets we have in the hash table, the longer we spend looking up the socket. While running a number of small workloads on the same host, they penalise each other and cause performance degradation.
The root cause might be a single workload that consumes much more resources than the others. It often happens on a cloud service where different workloads share the same computing resource.
On EC2 c5.24xlarge instance (196 GiB memory and 524288 (1Mi / 2) ehash entries), after running iperf3 in different netns, creating 24Mi sockets without data transfer in the root netns causes about 10% performance regression for the iperf3's connection.
thash_entries sockets length Gbps 524288 1 1 50.7 24Mi 48 45.1It is basically related to the length of the list of each hash bucket. For testing purposes to see how performance drops along the length, I set 131072 (1Mi / 8) to
thash_entries, and here's the result.thash_entries sockets length Gbps 131072 1 1 50.7 1Mi 8 49.9 2Mi 16 48.9 4Mi 32 47.3 8Mi 64 44.6 16Mi 128 40.6 24Mi 192 36.3 32Mi 256 32.5 40Mi 320 27.0 48Mi 384 25.0To resolve the socket lookup degradation, we introduce an optional per-netns hash table for TCP, but it's just ehash, and we still share the global bhash, bhash2 and lhash2.
With a smaller ehash, we can look up non-listener sockets faster and isolate such noisy neighbours. In addition, we can reduce lock contention.
We can control the ehash size by a new sysctl knob. However, depending on workloads, it will require very sensitive tuning, so we disable the feature by default (net.ipv4.tcp_child_ehash_entries == 0). Moreover, we can fall back to using the global ehash in case we fail to allocate enough memory for a new ehash. The maximum size is 16Mi, which is large enough that even if we have 48Mi sockets, the average list length is 3, and regression would be less than 1%.
We can check the current ehash size by another read-only sysctl knob, net.ipv4.tcp_ehash_entries. A negative value means the netns shares the global ehash (per-netns ehash is disabled or failed to allocate memory).
# dmesg | cut -d ' ' -f 5- | grep "established hash" TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)# sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = 524288 # can be changed by thash_entries# sysctl net.ipv4.tcp_child_ehash_entries net.ipv4.tcp_child_ehash_entries = 0 # disabled by default# ip netns add test1 # ip netns exec test1 sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = -524288 # share the global ehash# sysctl -w net.ipv4.tcp_child_ehash_entries=100 net.ipv4.tcp_child_ehash_entries = 100# ip netns add test2 # ip netns exec test2 sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = 128 # own a per-netns ehash with 2^n bucketsWhen more than two processes in the same netns create per-netns ehash concurrently with different sizes, we need to guarantee the size in one of the following ways:
1) Share the global ehash and create per-netns ehash
First,
unshare()with tcp_child_ehash_entries==0. It creates dedicated netns sysctl knobs where we can safely changetcp_child_ehash_entriesand clone()/unshare() to create a per-netns ehash.2) Control write on sysctl by BPF
We can use
BPF_PROG_TYPE_CGROUP_SYSCTLto allow/deny read/write on sysctl knobs.Note the default values of two sysctl knobs depend on the ehash size and should be tuned carefully:
tcp_max_tw_buckets:tcp_child_ehash_entries/ 2tcp_max_syn_backlog:max(128, tcp_child_ehash_entries / 128)As a bonus, we can dismantle netns faster. Currently, while destroying netns, we call
inet_twsk_purge(), which walks through the global ehash. It can be potentially big because it can have many sockets other thanTIME_WAITin all netns. Splitting ehash changes that situation, where it's only necessary forinet_twsk_purge()to clean upTIME_WAITsockets in each netns.With regard to this, we do not free the per-netns ehash in
inet_twsk_kill()to avoid UAF while iterating the per-netns ehash ininet_twsk_purge(). Instead, we do it intcp_sk_exit_batch()after callingtcp_twsk_purge()to keep it protocol-family-independent.In the future, we could optimise ehash lookup/iteration further by removing netns comparison for the per-netns ehash.
Signed-off-by Kuniyuki Iwashima <[email protected]>
...
Quoted 12 lines
net/ipv4/inet_hashtables.c +4
-0@@ -1145,3 +1145,60 @@ int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo)return 0 ; }EXPORT_SYMBOL_GPL (inet_ehash_locks_alloc); + +struct inet_hashinfo *inet_pernet_hashinfo_alloc (struct inet_hashinfo *hashinfo, +unsigned int ehash_entries) +{
net/ipv4/inet_hashtables.c int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo) · line 1161
Quoted 10 lines
net/ipv4/inet_hashtables.c +10
-0+struct inet_hashinfo *new_hashinfo; +int i; + + new_hashinfo =kmalloc (sizeof (*new_hashinfo),GFP_KERNEL ); +if (!new_hashinfo) + goto err; + + new_hashinfo->ehash =kvmalloc_array (ehash_entries, +sizeof (struct inet_ehash_bucket ), +GFP_KERNEL_ACCOUNT );
Note that in current kernel, init_net ehash table is using hugepages:
# dmesg | grep "TCP established hash table" [ 17.512756] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)
As this is very desirable, I would suggest using the following to avoid possible performance regression,
especially for workload wanting a big ehash, as hinted by your changelog.
new_hashinfo->ehash = vmalloc_huge(ehash_entries * sizeof(struct inet_ehash_bucket), GFP_KERNEL_ACCOUNT);
(No overflow can happen in the multiply, as ehash_entries < 16M)
Another point is that on NUMA, init_net ehash table is spread over available NUMA nodes.
While net_pernet_hashinfo_alloc() will allocate pages depending on current process NUMA policy.
Maybe worth noting this in the changelog, because it is very possible that new nets
is created with default NUMA policy, and depending on which cpu current thread is
running, hash table will fully reside on a 'random' node, with very different performance
results for highly optimized networking applications.
Quoted 20 lines
net/ipv4/inet_hashtables.c +20
-0+if (!new_hashinfo->ehash) + goto free_hashinfo; + + new_hashinfo->ehash_mask = ehash_entries -1 ; + +if (inet_ehash_locks_alloc (new_hashinfo)) + goto free_ehash; + +for (i =0 ; i < ehash_entries; i++) +INIT_HLIST_NULLS_HEAD (&new_hashinfo->ehash[i].chain, i); + + new_hashinfo->bind_bucket_cachep = hashinfo->bind_bucket_cachep; + new_hashinfo->bhash = hashinfo->bhash; + new_hashinfo->bind2_bucket_cachep = hashinfo->bind2_bucket_cachep; + new_hashinfo->bhash2 = hashinfo->bhash2; + new_hashinfo->bhash_size = hashinfo->bhash_size; + + new_hashinfo->lhash2_mask = hashinfo->lhash2_mask; + new_hashinfo->lhash2 = hashinfo->lhash2; +
From: Eric Dumazet <[email protected]> Date: Wed, 7 Sep 2022 13:55:08 -0700
Quoted 144 lines · Kuniyuki Iwashima
Quoted 111 lines
The more sockets we have in the hash table, the longer we spend looking up the socket. While running a number of small workloads on the same host, they penalise each other and cause performance degradation.
The root cause might be a single workload that consumes much more resources than the others. It often happens on a cloud service where different workloads share the same computing resource.
On EC2 c5.24xlarge instance (196 GiB memory and 524288 (1Mi / 2) ehash entries), after running iperf3 in different netns, creating 24Mi sockets without data transfer in the root netns causes about 10% performance regression for the iperf3's connection.
thash_entries sockets length Gbps 524288 1 1 50.7 24Mi 48 45.1It is basically related to the length of the list of each hash bucket. For testing purposes to see how performance drops along the length, I set 131072 (1Mi / 8) to
thash_entries, and here's the result.thash_entries sockets length Gbps 131072 1 1 50.7 1Mi 8 49.9 2Mi 16 48.9 4Mi 32 47.3 8Mi 64 44.6 16Mi 128 40.6 24Mi 192 36.3 32Mi 256 32.5 40Mi 320 27.0 48Mi 384 25.0To resolve the socket lookup degradation, we introduce an optional per-netns hash table for TCP, but it's just ehash, and we still share the global bhash, bhash2 and lhash2.
With a smaller ehash, we can look up non-listener sockets faster and isolate such noisy neighbours. In addition, we can reduce lock contention.
We can control the ehash size by a new sysctl knob. However, depending on workloads, it will require very sensitive tuning, so we disable the feature by default (net.ipv4.tcp_child_ehash_entries == 0). Moreover, we can fall back to using the global ehash in case we fail to allocate enough memory for a new ehash. The maximum size is 16Mi, which is large enough that even if we have 48Mi sockets, the average list length is 3, and regression would be less than 1%.
We can check the current ehash size by another read-only sysctl knob, net.ipv4.tcp_ehash_entries. A negative value means the netns shares the global ehash (per-netns ehash is disabled or failed to allocate memory).
# dmesg | cut -d ' ' -f 5- | grep "established hash" TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)# sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = 524288 # can be changed by thash_entries# sysctl net.ipv4.tcp_child_ehash_entries net.ipv4.tcp_child_ehash_entries = 0 # disabled by default# ip netns add test1 # ip netns exec test1 sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = -524288 # share the global ehash# sysctl -w net.ipv4.tcp_child_ehash_entries=100 net.ipv4.tcp_child_ehash_entries = 100# ip netns add test2 # ip netns exec test2 sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = 128 # own a per-netns ehash with 2^n bucketsWhen more than two processes in the same netns create per-netns ehash concurrently with different sizes, we need to guarantee the size in one of the following ways:
1) Share the global ehash and create per-netns ehash
First,
unshare()with tcp_child_ehash_entries==0. It creates dedicated netns sysctl knobs where we can safely changetcp_child_ehash_entriesand clone()/unshare() to create a per-netns ehash.2) Control write on sysctl by BPF
We can use
BPF_PROG_TYPE_CGROUP_SYSCTLto allow/deny read/write on sysctl knobs.Note the default values of two sysctl knobs depend on the ehash size and should be tuned carefully:
tcp_max_tw_buckets:tcp_child_ehash_entries/ 2tcp_max_syn_backlog:max(128, tcp_child_ehash_entries / 128)As a bonus, we can dismantle netns faster. Currently, while destroying netns, we call
inet_twsk_purge(), which walks through the global ehash. It can be potentially big because it can have many sockets other thanTIME_WAITin all netns. Splitting ehash changes that situation, where it's only necessary forinet_twsk_purge()to clean upTIME_WAITsockets in each netns.With regard to this, we do not free the per-netns ehash in
inet_twsk_kill()to avoid UAF while iterating the per-netns ehash ininet_twsk_purge(). Instead, we do it intcp_sk_exit_batch()after callingtcp_twsk_purge()to keep it protocol-family-independent.In the future, we could optimise ehash lookup/iteration further by removing netns comparison for the per-netns ehash.
Signed-off-by Kuniyuki Iwashima <[email protected]>...
Quoted 12 lines
net/ipv4/inet_hashtables.c +4
-0@@ -1145,3 +1145,60 @@ int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo)return 0 ; }EXPORT_SYMBOL_GPL (inet_ehash_locks_alloc); + +struct inet_hashinfo *inet_pernet_hashinfo_alloc (struct inet_hashinfo *hashinfo, +unsigned int ehash_entries) +{net/ipv4/inet_hashtables.c int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo) · line 1161
Quoted 10 lines
net/ipv4/inet_hashtables.c +10
-0+struct inet_hashinfo *new_hashinfo; +int i; + + new_hashinfo =kmalloc (sizeof (*new_hashinfo),GFP_KERNEL ); +if (!new_hashinfo) + goto err; + + new_hashinfo->ehash =kvmalloc_array (ehash_entries, +sizeof (struct inet_ehash_bucket ), +GFP_KERNEL_ACCOUNT );Note that in current kernel,
init_netehash table is using hugepages:# dmesg | grep "TCP established hash table" [ 17.512756] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)
Quoted 8 lines
As this is very desirable, I would suggest using the following to avoid possible performance regression,
especially for workload wanting a big ehash, as hinted by your changelog.
new_hashinfo->ehash=vmalloc_huge(ehash_entries * sizeof(struct inet_ehash_bucket), GFP_KERNEL_ACCOUNT);(No overflow can happen in the multiply, as
ehash_entries< 16M)
Do we need 'get_order(size) >= MAX_ORDER' check or just use it? Due to the test in alloc_large_system_hash(), on a machine where the calculted bucket size is not large enough, we don't use hugepages for init_net.
Quoted 6 lines
Another point is that on NUMA,
init_netehash table is spread over available NUMA nodes.While
net_pernet_hashinfo_alloc()will allocate pages depending on current process NUMA policy.
Quoted 7 lines
Maybe worth noting this in the changelog, because it is very possible that new nets
is created with default NUMA policy, and depending on which cpu current thread is
running, hash table will fully reside on a 'random' node, with very different performance
results for highly optimized networking applications.
Sounds great!
But I'm not familiar with mm, so let me confirm a bit more.
It seems vmalloc_huge() always pass NUMA_NO_NODE to __vmalloc_node_range(), so if we use vmalloc_huge(), the per-net ehash will be spread on each NUMA nodes unless vmap_allow_huge is disabled in the kernel parameters, right?
Or, even if we use vmalloc_huge(), the ehash could be controlled by the current process's NUMA policy? (Sorry I'm not sure where the policy is applied..)
Quoted 20 lines
Quoted 20 lines
net/ipv4/inet_hashtables.c +20
-0+if (!new_hashinfo->ehash) + goto free_hashinfo; + + new_hashinfo->ehash_mask = ehash_entries -1 ; + +if (inet_ehash_locks_alloc (new_hashinfo)) + goto free_ehash; + +for (i =0 ; i < ehash_entries; i++) +INIT_HLIST_NULLS_HEAD (&new_hashinfo->ehash[i].chain, i); + + new_hashinfo->bind_bucket_cachep = hashinfo->bind_bucket_cachep; + new_hashinfo->bhash = hashinfo->bhash; + new_hashinfo->bind2_bucket_cachep = hashinfo->bind2_bucket_cachep; + new_hashinfo->bhash2 = hashinfo->bhash2; + new_hashinfo->bhash_size = hashinfo->bhash_size; + + new_hashinfo->lhash2_mask = hashinfo->lhash2_mask; + new_hashinfo->lhash2 = hashinfo->lhash2; +
Quoted 151 lines · Kuniyuki Iwashima
From: Eric Dumazet <[email protected]> Date: Wed, 7 Sep 2022 13:55:08 -0700Quoted 148 lines · Kuniyuki Iwashima
Quoted 111 lines
The more sockets we have in the hash table, the longer we spend looking up the socket. While running a number of small workloads on the same host, they penalise each other and cause performance degradation.
The root cause might be a single workload that consumes much more resources than the others. It often happens on a cloud service where different workloads share the same computing resource.
On EC2 c5.24xlarge instance (196 GiB memory and 524288 (1Mi / 2) ehash entries), after running iperf3 in different netns, creating 24Mi sockets without data transfer in the root netns causes about 10% performance regression for the iperf3's connection.
thash_entries sockets length Gbps 524288 1 1 50.7 24Mi 48 45.1It is basically related to the length of the list of each hash bucket. For testing purposes to see how performance drops along the length, I set 131072 (1Mi / 8) to
thash_entries, and here's the result.thash_entries sockets length Gbps 131072 1 1 50.7 1Mi 8 49.9 2Mi 16 48.9 4Mi 32 47.3 8Mi 64 44.6 16Mi 128 40.6 24Mi 192 36.3 32Mi 256 32.5 40Mi 320 27.0 48Mi 384 25.0To resolve the socket lookup degradation, we introduce an optional per-netns hash table for TCP, but it's just ehash, and we still share the global bhash, bhash2 and lhash2.
With a smaller ehash, we can look up non-listener sockets faster and isolate such noisy neighbours. In addition, we can reduce lock contention.
We can control the ehash size by a new sysctl knob. However, depending on workloads, it will require very sensitive tuning, so we disable the feature by default (net.ipv4.tcp_child_ehash_entries == 0). Moreover, we can fall back to using the global ehash in case we fail to allocate enough memory for a new ehash. The maximum size is 16Mi, which is large enough that even if we have 48Mi sockets, the average list length is 3, and regression would be less than 1%.
We can check the current ehash size by another read-only sysctl knob, net.ipv4.tcp_ehash_entries. A negative value means the netns shares the global ehash (per-netns ehash is disabled or failed to allocate memory).
# dmesg | cut -d ' ' -f 5- | grep "established hash" TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)# sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = 524288 # can be changed by thash_entries# sysctl net.ipv4.tcp_child_ehash_entries net.ipv4.tcp_child_ehash_entries = 0 # disabled by default# ip netns add test1 # ip netns exec test1 sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = -524288 # share the global ehash# sysctl -w net.ipv4.tcp_child_ehash_entries=100 net.ipv4.tcp_child_ehash_entries = 100# ip netns add test2 # ip netns exec test2 sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = 128 # own a per-netns ehash with 2^n bucketsWhen more than two processes in the same netns create per-netns ehash concurrently with different sizes, we need to guarantee the size in one of the following ways:
1) Share the global ehash and create per-netns ehash
First,
unshare()with tcp_child_ehash_entries==0. It creates dedicated netns sysctl knobs where we can safely changetcp_child_ehash_entriesand clone()/unshare() to create a per-netns ehash.2) Control write on sysctl by BPF
We can use
BPF_PROG_TYPE_CGROUP_SYSCTLto allow/deny read/write on sysctl knobs.Note the default values of two sysctl knobs depend on the ehash size and should be tuned carefully:
tcp_max_tw_buckets:tcp_child_ehash_entries/ 2tcp_max_syn_backlog:max(128, tcp_child_ehash_entries / 128)As a bonus, we can dismantle netns faster. Currently, while destroying netns, we call
inet_twsk_purge(), which walks through the global ehash. It can be potentially big because it can have many sockets other thanTIME_WAITin all netns. Splitting ehash changes that situation, where it's only necessary forinet_twsk_purge()to clean upTIME_WAITsockets in each netns.With regard to this, we do not free the per-netns ehash in
inet_twsk_kill()to avoid UAF while iterating the per-netns ehash ininet_twsk_purge(). Instead, we do it intcp_sk_exit_batch()after callingtcp_twsk_purge()to keep it protocol-family-independent.In the future, we could optimise ehash lookup/iteration further by removing netns comparison for the per-netns ehash.
Signed-off-by Kuniyuki Iwashima <[email protected]>...
Quoted 12 lines
net/ipv4/inet_hashtables.c +4
-0@@ -1145,3 +1145,60 @@ int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo)return 0 ; }EXPORT_SYMBOL_GPL (inet_ehash_locks_alloc); + +struct inet_hashinfo *inet_pernet_hashinfo_alloc (struct inet_hashinfo *hashinfo, +unsigned int ehash_entries) +{net/ipv4/inet_hashtables.c int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo) · line 1161
Quoted 10 lines
net/ipv4/inet_hashtables.c +10
-0+struct inet_hashinfo *new_hashinfo; +int i; + + new_hashinfo =kmalloc (sizeof (*new_hashinfo),GFP_KERNEL ); +if (!new_hashinfo) + goto err; + + new_hashinfo->ehash =kvmalloc_array (ehash_entries, +sizeof (struct inet_ehash_bucket ), +GFP_KERNEL_ACCOUNT );Note that in current kernel,
init_netehash table is using hugepages:# dmesg | grep "TCP established hash table" [ 17.512756] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)As this is very desirable, I would suggest using the following to avoid possible performance regression,
especially for workload wanting a big ehash, as hinted by your changelog.
Quoted 6 lines
Quoted 4 lines
new_hashinfo->ehash=vmalloc_huge(ehash_entries * sizeof(struct inet_ehash_bucket), GFP_KERNEL_ACCOUNT);(No overflow can happen in the multiply, as
ehash_entries< 16M)Do we need '
get_order(size)>=MAX_ORDER' check or just use it?
No need, just use it.
If you happen to allocate 1MB, vmalloc_huge() will simply use 256 4k pages, not 2MB.
Quoted 3 lines
Due to the test in
alloc_large_system_hash(), on a machine where the calculted bucket size is not large enough, we don't use hugepages forinit_net.
I can tell that even my laptop allocates hugepage...
And it will all depend on the user-configured ehash table size.
Quoted 16 lines
Quoted 8 lines
Another point is that on NUMA,
init_netehash table is spread over available NUMA nodes.While
net_pernet_hashinfo_alloc()will allocate pages depending on current process NUMA policy.Quoted 7 lines
Maybe worth noting this in the changelog, because it is very possible that new nets
is created with default NUMA policy, and depending on which cpu current thread is
running, hash table will fully reside on a 'random' node, with very different performance
results for highly optimized networking applications.
Quoted 6 lines
Sounds great!
But I'm not familiar with mm, so let me confirm a bit more.It seems
vmalloc_huge()always passNUMA_NO_NODEto__vmalloc_node_range(), so if we usevmalloc_huge(), the per-net ehash will be spread on each NUMA nodes unlessvmap_allow_hugeis disabled in the kernel parameters, right?
No, it depends on current NUMA policy
"man numa"
By default, at system init time, NUMA policy spreads allocations on all memory nodes.
After system has booted, default NUMA policy allocates memory on your local node only.
You can check on a NUMA host how vmalloc regions are spread
grep N0= /proc/vmallocinfo
grep N1= /proc/vmallocinfo
For instance, all large system hashes are evenly spread:
grep alloc_large_system_hash /proc/vmallocinfo
...
0x00000000c13cf72b-0x0000000068e51b86 4198400 alloc_large_system_hash+0x160/0x3aa pages=1024 vmalloc vpages N0=512 N1=512
...
while:
# echo 200000 >/proc/sys/net/ipv4/tcp_child_ehash_entries # unshare -n # grep inet_pernet_hashinfo_alloc /proc/vmallocinfo 0x00000000980d41fd-0x00000000ea510502 2101248 inet_pernet_hashinfo_alloc+0x79/0x910 pages=512 vmalloc N1=512
(everything has been allocated into N1, because "unshare -n" probably was scheduled on a cpu in NUMA node1)
Quoted 26 lines
Or, even if we use
vmalloc_huge(), the ehash could be controlled by the current process's NUMA policy? (Sorry I'm not sure where the policy is applied..)Quoted 20 lines
Quoted 20 lines
net/ipv4/inet_hashtables.c +20
-0+if (!new_hashinfo->ehash) + goto free_hashinfo; + + new_hashinfo->ehash_mask = ehash_entries -1 ; + +if (inet_ehash_locks_alloc (new_hashinfo)) + goto free_ehash; + +for (i =0 ; i < ehash_entries; i++) +INIT_HLIST_NULLS_HEAD (&new_hashinfo->ehash[i].chain, i); + + new_hashinfo->bind_bucket_cachep = hashinfo->bind_bucket_cachep; + new_hashinfo->bhash = hashinfo->bhash; + new_hashinfo->bind2_bucket_cachep = hashinfo->bind2_bucket_cachep; + new_hashinfo->bhash2 = hashinfo->bhash2; + new_hashinfo->bhash_size = hashinfo->bhash_size; + + new_hashinfo->lhash2_mask = hashinfo->lhash2_mask; + new_hashinfo->lhash2 = hashinfo->lhash2; +
From: Eric Dumazet <[email protected]> Date: Wed, 7 Sep 2022 14:57:14 -0700
Quoted 164 lines · Kuniyuki Iwashima
Quoted 151 lines
From: Eric Dumazet <[email protected]> Date: Wed, 7 Sep 2022 13:55:08 -0700Quoted 148 lines · Kuniyuki Iwashima
Quoted 111 lines
The more sockets we have in the hash table, the longer we spend looking up the socket. While running a number of small workloads on the same host, they penalise each other and cause performance degradation.
The root cause might be a single workload that consumes much more resources than the others. It often happens on a cloud service where different workloads share the same computing resource.
On EC2 c5.24xlarge instance (196 GiB memory and 524288 (1Mi / 2) ehash entries), after running iperf3 in different netns, creating 24Mi sockets without data transfer in the root netns causes about 10% performance regression for the iperf3's connection.
thash_entries sockets length Gbps 524288 1 1 50.7 24Mi 48 45.1It is basically related to the length of the list of each hash bucket. For testing purposes to see how performance drops along the length, I set 131072 (1Mi / 8) to
thash_entries, and here's the result.thash_entries sockets length Gbps 131072 1 1 50.7 1Mi 8 49.9 2Mi 16 48.9 4Mi 32 47.3 8Mi 64 44.6 16Mi 128 40.6 24Mi 192 36.3 32Mi 256 32.5 40Mi 320 27.0 48Mi 384 25.0To resolve the socket lookup degradation, we introduce an optional per-netns hash table for TCP, but it's just ehash, and we still share the global bhash, bhash2 and lhash2.
With a smaller ehash, we can look up non-listener sockets faster and isolate such noisy neighbours. In addition, we can reduce lock contention.
We can control the ehash size by a new sysctl knob. However, depending on workloads, it will require very sensitive tuning, so we disable the feature by default (net.ipv4.tcp_child_ehash_entries == 0). Moreover, we can fall back to using the global ehash in case we fail to allocate enough memory for a new ehash. The maximum size is 16Mi, which is large enough that even if we have 48Mi sockets, the average list length is 3, and regression would be less than 1%.
We can check the current ehash size by another read-only sysctl knob, net.ipv4.tcp_ehash_entries. A negative value means the netns shares the global ehash (per-netns ehash is disabled or failed to allocate memory).
# dmesg | cut -d ' ' -f 5- | grep "established hash" TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)# sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = 524288 # can be changed by thash_entries# sysctl net.ipv4.tcp_child_ehash_entries net.ipv4.tcp_child_ehash_entries = 0 # disabled by default# ip netns add test1 # ip netns exec test1 sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = -524288 # share the global ehash# sysctl -w net.ipv4.tcp_child_ehash_entries=100 net.ipv4.tcp_child_ehash_entries = 100# ip netns add test2 # ip netns exec test2 sysctl net.ipv4.tcp_ehash_entries net.ipv4.tcp_ehash_entries = 128 # own a per-netns ehash with 2^n bucketsWhen more than two processes in the same netns create per-netns ehash concurrently with different sizes, we need to guarantee the size in one of the following ways:
1) Share the global ehash and create per-netns ehash
First,
unshare()with tcp_child_ehash_entries==0. It creates dedicated netns sysctl knobs where we can safely changetcp_child_ehash_entriesand clone()/unshare() to create a per-netns ehash.2) Control write on sysctl by BPF
We can use
BPF_PROG_TYPE_CGROUP_SYSCTLto allow/deny read/write on sysctl knobs.Note the default values of two sysctl knobs depend on the ehash size and should be tuned carefully:
tcp_max_tw_buckets:tcp_child_ehash_entries/ 2tcp_max_syn_backlog:max(128, tcp_child_ehash_entries / 128)As a bonus, we can dismantle netns faster. Currently, while destroying netns, we call
inet_twsk_purge(), which walks through the global ehash. It can be potentially big because it can have many sockets other thanTIME_WAITin all netns. Splitting ehash changes that situation, where it's only necessary forinet_twsk_purge()to clean upTIME_WAITsockets in each netns.With regard to this, we do not free the per-netns ehash in
inet_twsk_kill()to avoid UAF while iterating the per-netns ehash ininet_twsk_purge(). Instead, we do it intcp_sk_exit_batch()after callingtcp_twsk_purge()to keep it protocol-family-independent.In the future, we could optimise ehash lookup/iteration further by removing netns comparison for the per-netns ehash.
Signed-off-by Kuniyuki Iwashima <[email protected]>...
Quoted 12 lines
net/ipv4/inet_hashtables.c +4
-0@@ -1145,3 +1145,60 @@ int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo)return 0 ; }EXPORT_SYMBOL_GPL (inet_ehash_locks_alloc); + +struct inet_hashinfo *inet_pernet_hashinfo_alloc (struct inet_hashinfo *hashinfo, +unsigned int ehash_entries) +{net/ipv4/inet_hashtables.c int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo) · line 1161
Quoted 10 lines
net/ipv4/inet_hashtables.c +10
-0+struct inet_hashinfo *new_hashinfo; +int i; + + new_hashinfo =kmalloc (sizeof (*new_hashinfo),GFP_KERNEL ); +if (!new_hashinfo) + goto err; + + new_hashinfo->ehash =kvmalloc_array (ehash_entries, +sizeof (struct inet_ehash_bucket ), +GFP_KERNEL_ACCOUNT );Note that in current kernel,
init_netehash table is using hugepages:# dmesg | grep "TCP established hash table" [ 17.512756] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc hugepage)As this is very desirable, I would suggest using the following to avoid possible performance regression,
especially for workload wanting a big ehash, as hinted by your changelog.Quoted 6 lines
Quoted 4 lines
new_hashinfo->ehash=vmalloc_huge(ehash_entries * sizeof(struct inet_ehash_bucket), GFP_KERNEL_ACCOUNT);(No overflow can happen in the multiply, as
ehash_entries< 16M)Do we need '
get_order(size)>=MAX_ORDER' check or just use it?No need, just use it.
If you happen to allocate 1MB,
vmalloc_huge()will simply use 256 4k pages, not 2MB.
Quoted 7 lines
Quoted 3 lines
Due to the test in
alloc_large_system_hash(), on a machine where the calculted bucket size is not large enough, we don't use hugepages forinit_net.I can tell that even my laptop allocates hugepage...
And it will all depend on the user-configured ehash table size.
Ok, I'll use vmalloc_huge().
Quoted 47 lines
Quoted 14 lines
Quoted 6 lines
Another point is that on NUMA,
init_netehash table is spread over available NUMA nodes.While
net_pernet_hashinfo_alloc()will allocate pages depending on current process NUMA policy.Quoted 7 lines
Maybe worth noting this in the changelog, because it is very possible that new nets
is created with default NUMA policy, and depending on which cpu current thread is
running, hash table will fully reside on a 'random' node, with very different performance
results for highly optimized networking applications.Quoted 6 lines
Sounds great!
But I'm not familiar with mm, so let me confirm a bit more.It seems
vmalloc_huge()always passNUMA_NO_NODEto__vmalloc_node_range(), so if we usevmalloc_huge(), the per-net ehash will be spread on each NUMA nodes unlessvmap_allow_hugeis disabled in the kernel parameters, right?No, it depends on current NUMA policy
"man numa"
By default, at system init time, NUMA policy spreads allocations on all memory nodes.
After system has booted, default NUMA policy allocates memory on your local node only.
You can check on a NUMA host how vmalloc regions are spread
grep N0=
/proc/vmallocinfo
grep N1=/proc/vmallocinfoFor instance, all large system hashes are evenly spread:
grep
alloc_large_system_hash/proc/vmallocinfo
...
0x00000000c13cf72b-0x0000000068e51b86 4198400alloc_large_system_hash+0x160/0x3aapages=1024 vmalloc vpages N0=512 N1=512
...while:
Quoted 8 lines
# echo 200000 >/proc/sys/net/ipv4/tcp_child_ehash_entries # unshare -n # grep inet_pernet_hashinfo_alloc /proc/vmallocinfo 0x00000000980d41fd-0x00000000ea510502 2101248 inet_pernet_hashinfo_alloc+0x79/0x910 pages=512 vmalloc N1=512(everything has been allocated into N1, because "unshare -n" probably was scheduled on a cpu in NUMA node1)
Thanks for explanation!
I'll note the difference with the global ehash and NUMA policy in the changelog.
Quoted 25 lines
Quoted 25 lines
Or, even if we use
vmalloc_huge(), the ehash could be controlled by the current process's NUMA policy? (Sorry I'm not sure where the policy is applied..)Quoted 20 lines
Quoted 20 lines
net/ipv4/inet_hashtables.c +20
-0+if (!new_hashinfo->ehash) + goto free_hashinfo; + + new_hashinfo->ehash_mask = ehash_entries -1 ; + +if (inet_ehash_locks_alloc (new_hashinfo)) + goto free_ehash; + +for (i =0 ; i < ehash_entries; i++) +INIT_HLIST_NULLS_HEAD (&new_hashinfo->ehash[i].chain, i); + + new_hashinfo->bind_bucket_cachep = hashinfo->bind_bucket_cachep; + new_hashinfo->bhash = hashinfo->bhash; + new_hashinfo->bind2_bucket_cachep = hashinfo->bind2_bucket_cachep; + new_hashinfo->bhash2 = hashinfo->bhash2; + new_hashinfo->bhash_size = hashinfo->bhash_size; + + new_hashinfo->lhash2_mask = hashinfo->lhash2_mask; + new_hashinfo->lhash2 = hashinfo->lhash2; +
net/ipv4/inet_hashtables.c int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo) · line 1155
Quoted 13 lines · Kuniyuki Iwashima
The more sockets we have in the hash table, the longer we spend looking up the socket. While running a number of small workloads on the same host, they penalise each other and cause performance degradation.
net/ipv4/inet_hashtables.c +8
-0+ +struct inet_hashinfo *inet_pernet_hashinfo_alloc (struct inet_hashinfo *hashinfo, +unsigned int ehash_entries) +{ +struct inet_hashinfo *new_hashinfo; +int i; + + new_hashinfo =kmalloc (sizeof (*new_hashinfo),GFP_KERNEL );
You probably could use kmemdup(hashinfo, sizeof(*hashinfo), GFP_KERNEL);
Quoted 15 lines
net/ipv4/inet_hashtables.c +15
-0+if (!new_hashinfo) + goto err; + + new_hashinfo->ehash =kvmalloc_array (ehash_entries, +sizeof (struct inet_ehash_bucket ), +GFP_KERNEL_ACCOUNT ); +if (!new_hashinfo->ehash) + goto free_hashinfo; + + new_hashinfo->ehash_mask = ehash_entries -1 ; + +if (inet_ehash_locks_alloc (new_hashinfo)) + goto free_ehash; + +for (i =0 ; i < ehash_entries; i++)
net/ipv4/inet_hashtables.c int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo) · line 1180
Quoted 10 lines
net/ipv4/inet_hashtables.c +10
-0+INIT_HLIST_NULLS_HEAD (&new_hashinfo->ehash[i].chain, i); + + new_hashinfo->bind_bucket_cachep = hashinfo->bind_bucket_cachep; + new_hashinfo->bhash = hashinfo->bhash; + new_hashinfo->bind2_bucket_cachep = hashinfo->bind2_bucket_cachep; + new_hashinfo->bhash2 = hashinfo->bhash2; + new_hashinfo->bhash_size = hashinfo->bhash_size; + + new_hashinfo->lhash2_mask = hashinfo->lhash2_mask; + new_hashinfo->lhash2 = hashinfo->lhash2;
This would avoid copying all these @hashinfo fields.
From: Eric Dumazet <[email protected]> Date: Wed, 7 Sep 2022 16:45:17 -0700
Quoted 6 lines · Kuniyuki Iwashima
Quoted 5 lines
The more sockets we have in the hash table, the longer we spend looking up the socket. While running a number of small workloads on the same host, they penalise each other and cause performance degradation.
Quoted 10 lines
net/ipv4/inet_hashtables.c int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo) · line 1155
Quoted 8 lines
net/ipv4/inet_hashtables.c +8
-0+ +struct inet_hashinfo *inet_pernet_hashinfo_alloc (struct inet_hashinfo *hashinfo, +unsigned int ehash_entries) +{ +struct inet_hashinfo *new_hashinfo; +int i; + + new_hashinfo =kmalloc (sizeof (*new_hashinfo),GFP_KERNEL );You probably could use
kmemdup(hashinfo, sizeof(*hashinfo), GFP_KERNEL);
Exactly, I'll use it and remove the manual copy.
Thank you!
Quoted 28 lines
Quoted 15 lines
net/ipv4/inet_hashtables.c +15
-0+if (!new_hashinfo) + goto err; + + new_hashinfo->ehash =kvmalloc_array (ehash_entries, +sizeof (struct inet_ehash_bucket ), +GFP_KERNEL_ACCOUNT ); +if (!new_hashinfo->ehash) + goto free_hashinfo; + + new_hashinfo->ehash_mask = ehash_entries -1 ; + +if (inet_ehash_locks_alloc (new_hashinfo)) + goto free_ehash; + +for (i =0 ; i < ehash_entries; i++)net/ipv4/inet_hashtables.c int inet_ehash_locks_alloc(struct inet_hashinfo *hashinfo) · line 1180
Quoted 10 lines
net/ipv4/inet_hashtables.c +10
-0+INIT_HLIST_NULLS_HEAD (&new_hashinfo->ehash[i].chain, i); + + new_hashinfo->bind_bucket_cachep = hashinfo->bind_bucket_cachep; + new_hashinfo->bhash = hashinfo->bhash; + new_hashinfo->bind2_bucket_cachep = hashinfo->bind2_bucket_cachep; + new_hashinfo->bhash2 = hashinfo->bhash2; + new_hashinfo->bhash_size = hashinfo->bhash_size; + + new_hashinfo->lhash2_mask = hashinfo->lhash2_mask; + new_hashinfo->lhash2 = hashinfo->lhash2;This would avoid copying all these @hashinfo fields.