Skip to contentSkip to conversation

tcp/dccp: Add another way to allocate local ports in connect()

PATCH In review [email protected] 4 participants 6 messages Jul 29, 2024 – Jul 30, 2024 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tcp/dccp: Add another way to allocate local ports in connect()

From: Linkui Xiao <[email protected]>

Commit 07f4c90062f8 ("tcp/dccp: try to not exhaust ip_local_port_range in connect()") allocates even ports for connect() first while leaving odd ports for bind() and this works well in busy servers.

But this strategy causes severe performance degradation in busy clients. when a client has used more than half of the local ports setted in proc/sys/net/ipv4/ip_local_port_range, if this client try to connect to a server again, the connect time increases rapidly since it will traverse all the even ports though they are exhausted.

So this path provides another strategy by introducing a system option:
local_port_allocation. If it is a busy client, users should set it to 1 to use sequential allocation while it should be set to 0 in other situations. Its default value is 0.

In commit 207184853dbd ("tcp/dccp: change source port selection at connect() time"), tell users that they can access all odd and even ports by using IP_LOCAL_PORT_RANGE. But this requires users to modify the socket application. When even numbered ports are not sufficient, use the sysctl parameter to achieve the same effect:
sysctl -w net.ipv4.local_port_allocation=1

---
 include/net/tcp.h          |  1 +
 net/ipv4/inet_hashtables.c | 12 ++++++++----
 net/ipv4/sysctl_net_ipv4.c |  8 ++++++++
 3 files changed, 17 insertions(+), 4 deletions(-)
include/net/tcp.h +1 -0
index 2aac11e7e1cc..99969b8e5183 100644
@@ -269,6 +269,7 @@ DECLARE_PER_CPU(int, tcp_memory_per_cpu_fw_alloc);
 
 extern struct percpu_counter tcp_sockets_allocated;
 extern unsigned long tcp_memory_pressure;
+extern bool sysctl_local_port_allocation;
 
 /* optimized version of sk_under_memory_pressure() for TCP sockets */
 static inline bool tcp_under_memory_pressure(const struct sock *sk)
net/ipv4/inet_hashtables.c +8 -4
index 48d0d494185b..e572f8b21b95 100644
@@ -1020,11 +1020,15 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,
 	l3mdev = inet_sk_bound_l3mdev(sk);
 
 	local_ports = inet_sk_get_local_port_range(sk, &low, &high);
-	step = local_ports ? 1 : 2;
+	/* local_port_allocation 0 means even and odd port allocation strategy
+	 * will be applied, so step is 2; otherwise sequential allocation will
+	 * be used and step is 1. Default value is 0.
+	 */
+	step = sysctl_local_port_allocation ? 1 : 2;
 
 	high++; /* [32768, 60999] -> [32768, 61000[ */
 	remaining = high - low;
-	if (!local_ports && remaining > 1)
+	if (!sysctl_local_port_allocation && remaining > 1)
 		remaining &= ~1U;
 
 	get_random_sleepable_once(table_perturb,
@@ -1037,7 +1041,7 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,
 	/* In first pass we try ports of @low parity.
 	 * inet_csk_get_port() does the opposite choice.
 	 */
-	if (!local_ports)
+	if (!sysctl_local_port_allocation)
 		offset &= ~1U;
 other_parity_scan:
 	port = low + offset;
@@ -1081,7 +1085,7 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,
 		cond_resched();
 	}
 
-	if (!local_ports) {
+	if (!sysctl_local_port_allocation) {
 		offset++;
 		if ((offset & 1) && remaining > 1)
 			goto other_parity_scan;
net/ipv4/sysctl_net_ipv4.c +8 -0
index 9140d20eb2d4..1f6bf3a73516 100644
@@ -45,6 +45,7 @@ static unsigned int tcp_child_ehash_entries_max = 16 * 1024 * 1024;
 static unsigned int udp_child_hash_entries_max = UDP_HTABLE_SIZE_MAX;
 static int tcp_plb_max_rounds = 31;
 static int tcp_plb_max_cong_thresh = 256;
+bool sysctl_local_port_allocation;
 
 /* obsolete */
 static int sysctl_tcp_low_latency __read_mostly;
@@ -632,6 +633,13 @@ static struct ctl_table ipv4_table[] = {
 		.extra1		= &sysctl_fib_sync_mem_min,
 		.extra2		= &sysctl_fib_sync_mem_max,
 	},
+	{
+		.procname	= "local_port_allocation",
+		.data		= &sysctl_local_port_allocation,
+		.maxlen		= sizeof(sysctl_local_port_allocation),
+		.mode		= 0644,
+		.proc_handler	= proc_dobool,
+	},
 };
 
 static struct ctl_table ipv4_net_table[] = {
Signature · 1 line
2.17.1
#1 lore raw
Quoted 18 lines · On Mon, Jul 29, 2024 at 11:56 AM <[email protected]> wrote:

From: Linkui Xiao <[email protected]>

Commit 07f4c90062f8 ("tcp/dccp: try to not exhaust ip_local_port_range in connect()") allocates even ports for connect() first while leaving odd ports for bind() and this works well in busy servers.

But this strategy causes severe performance degradation in busy clients. when a client has used more than half of the local ports setted in proc/sys/net/ipv4/ip_local_port_range, if this client try to connect to a server again, the connect time increases rapidly since it will traverse all the even ports though they are exhausted.

So this path provides another strategy by introducing a system option:
local_port_allocation. If it is a busy client, users should set it to 1 to use sequential allocation while it should be set to 0 in other situations. Its default value is 0.

Quoted 8 lines

In commit 207184853dbd ("tcp/dccp: change source port selection at connect() time"), tell users that they can access all odd and even ports by using IP_LOCAL_PORT_RANGE. But this requires users to modify the socket application. When even numbered ports are not sufficient, use the sysctl parameter to achieve the same effect:
sysctl -w net.ipv4.local_port_allocation=1

Too many errors in this patch...

Lack of READ_ONCE() when reading a sysctl.
Lack or per-netns sysctl.
No documentation.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 32 lines · xiaolinkui

Thanks for your reply.

Quoted 28 lines · Eric Dumazet
Quoted 26 lines · On Mon, Jul 29, 2024 at 11:56 AM <[email protected]> wrote:

From: Linkui Xiao <[email protected]>

Commit 07f4c90062f8 ("tcp/dccp: try to not exhaust ip_local_port_range in connect()") allocates even ports for connect() first while leaving odd ports for bind() and this works well in busy servers.

But this strategy causes severe performance degradation in busy clients. when a client has used more than half of the local ports setted in proc/sys/net/ipv4/ip_local_port_range, if this client try to connect to a server again, the connect time increases rapidly since it will traverse all the even ports though they are exhausted.

So this path provides another strategy by introducing a system option:
local_port_allocation. If it is a busy client, users should set it to 1 to use sequential allocation while it should be set to 0 in other situations. Its default value is 0.

In commit 207184853dbd ("tcp/dccp: change source port selection at connect() time"), tell users that they can access all odd and even ports by using IP_LOCAL_PORT_RANGE. But this requires users to modify the socket application. When even numbered ports are not sufficient, use the sysctl parameter to achieve the same effect:
sysctl -w net.ipv4.local_port_allocation=1

Quoted 8 lines
Quoted 5 lines

Too many errors in this patch...

Lack of READ_ONCE() when reading a sysctl.
Lack or per-netns sysctl.
No documentation.

Yes, it was my negligence.But do you think it's necessary for me to send a v2 version?Or should we maintain the previous state here and put the sysctl setting method in my private kernel?

I think that adding a sysctl because some applications "can not be changed" is not convincing.

Another useful socket option is IP_BIND_ADDRESS_NO_PORT, which also is a choice made by applications.

#4 lore raw
In reply to [email protected]

Hi Linkui,

Quoted 51 lines · On Mon, Jul 29, 2024 at 5:58 PM <[email protected]> wrote:

From: Linkui Xiao <[email protected]>

Commit 07f4c90062f8 ("tcp/dccp: try to not exhaust ip_local_port_range in connect()") allocates even ports for connect() first while leaving odd ports for bind() and this works well in busy servers.

But this strategy causes severe performance degradation in busy clients. when a client has used more than half of the local ports setted in proc/sys/net/ipv4/ip_local_port_range, if this client try to connect to a server again, the connect time increases rapidly since it will traverse all the even ports though they are exhausted.

So this path provides another strategy by introducing a system option:
local_port_allocation. If it is a busy client, users should set it to 1 to use sequential allocation while it should be set to 0 in other situations. Its default value is 0.

In commit 207184853dbd ("tcp/dccp: change source port selection at connect() time"), tell users that they can access all odd and even ports by using IP_LOCAL_PORT_RANGE. But this requires users to modify the socket application. When even numbered ports are not sufficient, use the sysctl parameter to achieve the same effect:
sysctl -w net.ipv4.local_port_allocation=1

---
 include/net/tcp.h          |  1 +
 net/ipv4/inet_hashtables.c | 12 ++++++++----
 net/ipv4/sysctl_net_ipv4.c |  8 ++++++++
 3 files changed, 17 insertions(+), 4 deletions(-)
include/net/tcp.h +1 -0
index 2aac11e7e1cc..99969b8e5183 100644
@@ -269,6 +269,7 @@ DECLARE_PER_CPU(int, tcp_memory_per_cpu_fw_alloc);
 
 extern struct percpu_counter tcp_sockets_allocated;
 extern unsigned long tcp_memory_pressure;
+extern bool sysctl_local_port_allocation;
 
 /* optimized version of sk_under_memory_pressure() for TCP sockets */
 static inline bool tcp_under_memory_pressure(const struct sock *sk)
net/ipv4/inet_hashtables.c +0 -0
index 48d0d494185b..e572f8b21b95 100644
@@ -1020,11 +1020,15 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,
        l3mdev = inet_sk_bound_l3mdev(sk);
 

[...]

net/ipv4/inet_hashtables.c int __inet_hash_connect(struct inet_timewait_death_row *death_row, · line 1027

Quoted 7 lines
        local_ports = inet_sk_get_local_port_range(sk, &low, &high);
net/ipv4/inet_hashtables.c +5 -1
-       step = local_ports ? 1 : 2;
+       /* local_port_allocation 0 means even and odd port allocation strategy
+        * will be applied, so step is 2; otherwise sequential allocation will
+        * be used and step is 1. Default value is 0.
+        */
+       step = sysctl_local_port_allocation ? 1 : 2;

Apart from what Eric said, it seems you break the setsockopt method...? Please look at the 'local_ports'.

Once again, I'm confused about whether we should introduce such a sysctl knob to do the same thing as an existing method does. Maybe it suits a private kernel. I'm just saying.

Thanks,
Jason

Quoted 56 lines
        high++; /* [32768, 60999] -> [32768, 61000[ */
        remaining = high - low;
net/ipv4/inet_hashtables.c +3 -3
-       if (!local_ports && remaining > 1)
+       if (!sysctl_local_port_allocation && remaining > 1)
                remaining &= ~1U;
 
        get_random_sleepable_once(table_perturb,
@@ -1037,7 +1041,7 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,
        /* In first pass we try ports of @low parity.
         * inet_csk_get_port() does the opposite choice.
         */
-       if (!local_ports)
+       if (!sysctl_local_port_allocation)
                offset &= ~1U;
 other_parity_scan:
        port = low + offset;
@@ -1081,7 +1085,7 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,
                cond_resched();
        }
 
-       if (!local_ports) {
+       if (!sysctl_local_port_allocation) {
                offset++;
                if ((offset & 1) && remaining > 1)
                        goto other_parity_scan;
net/ipv4/sysctl_net_ipv4.c +8 -1
index 9140d20eb2d4..1f6bf3a73516 100644
@@ -45,6 +45,7 @@ static unsigned int tcp_child_ehash_entries_max = 16 * 1024 * 1024;
 static unsigned int udp_child_hash_entries_max = UDP_HTABLE_SIZE_MAX;
 static int tcp_plb_max_rounds = 31;
 static int tcp_plb_max_cong_thresh = 256;
+bool sysctl_local_port_allocation;
 
 /* obsolete */
 static int sysctl_tcp_low_latency __read_mostly;
@@ -632,6 +633,13 @@ static struct ctl_table ipv4_table[] = {
                .extra1         = &sysctl_fib_sync_mem_min,
                .extra2         = &sysctl_fib_sync_mem_max,
        },
+       {
+               .procname       = "local_port_allocation",
+               .data           = &sysctl_local_port_allocation,
+               .maxlen         = sizeof(sysctl_local_port_allocation),
+               .mode           = 0644,
+               .proc_handler   = proc_dobool,
+       },
 };
 
 static struct ctl_table ipv4_net_table[] = {
--

2.17.1

#5 lore raw
In reply to [email protected]

From: [email protected]
Date: Mon, 29 Jul 2024 17:55:54 +0800

Quoted 17 lines

From: Linkui Xiao <[email protected]>

Commit 07f4c90062f8 ("tcp/dccp: try to not exhaust ip_local_port_range in connect()") allocates even ports for connect() first while leaving odd ports for bind() and this works well in busy servers.

But this strategy causes severe performance degradation in busy clients. when a client has used more than half of the local ports setted in proc/sys/net/ipv4/ip_local_port_range, if this client try to connect to a server again, the connect time increases rapidly since it will traverse all the even ports though they are exhausted.

So this path provides another strategy by introducing a system option:
local_port_allocation. If it is a busy client, users should set it to 1 to use sequential allocation while it should be set to 0 in other situations. Its default value is 0.

Quoted 4 lines

In commit 207184853dbd ("tcp/dccp: change source port selection at connect() time"), tell users that they can access all odd and even ports by using IP_LOCAL_PORT_RANGE. But this requires users to modify the socket application.

The application should be changed, or probably you can put your application into a cgroup and hook connect() to call bpf_setsockopt() and silently enable IP_LOCAL_PORT_RANGE.

#6 lore raw
Quoted 20 lines · Kuniyuki Iwashima

From: [email protected]
Date: Mon, 29 Jul 2024 17:55:54 +0800

Quoted 17 lines

From: Linkui Xiao <[email protected]>

Commit 07f4c90062f8 ("tcp/dccp: try to not exhaust ip_local_port_range in connect()") allocates even ports for connect() first while leaving odd ports for bind() and this works well in busy servers.

But this strategy causes severe performance degradation in busy clients. when a client has used more than half of the local ports setted in proc/sys/net/ipv4/ip_local_port_range, if this client try to connect to a server again, the connect time increases rapidly since it will traverse all the even ports though they are exhausted.

So this path provides another strategy by introducing a system option:
local_port_allocation. If it is a busy client, users should set it to 1 to use sequential allocation while it should be set to 0 in other situations. Its default value is 0.

Quoted 8 lines
Quoted 4 lines

In commit 207184853dbd ("tcp/dccp: change source port selection at connect() time"), tell users that they can access all odd and even ports by using IP_LOCAL_PORT_RANGE. But this requires users to modify the socket application.

The application should be changed, or probably you can put your application into a cgroup and hook connect() to call bpf_setsockopt() and silently enable IP_LOCAL_PORT_RANGE.

LD_PRELOAD can also be used for non eBPF users.