Skip to contentSkip to conversation

tcp: do not allow to connect with the four-tuple symmetry socket

PATCH In review Jason Xing 3 participants 14 messages Aug 18, 2024 – Aug 20, 2024

Subsystems (3)
lore
Share & tools
lore

Jason Xing

author
#0 lore raw

PATCH 1/1 tcp: do not allow to connect with the four-tuple symmetry socket

From: Jason Xing <[email protected]>

Four-tuple symmetry here means the socket has the same remote/local port and ipaddr, like this, 127.0.0.1:8000 -> 127.0.0.1:8000.

$ ss -nat | grep 8000
ESTAB      0      0          127.0.0.1:8000       127.0.0.1:8000

Before this patch, one client could start a connection successfully as above even without a listener, which means, the socket connects to its self. Then every time other threads trying to bind/listen on this port will encounter a failure surely, unless the thread owning the socket exits.

It can rarely happen on the loopback device when the connect() finds the same port as its remote port while listener is not running. It has the side-effect on other threads. Besides, this solo flow has no merit, no significance at all.

After this patch, the moment we try to connect with a 4-tuple symmetry socket, we will get an error "connect: Cannot assign requested address".

---
 net/ipv4/inet_hashtables.c | 31 +++++++++++++++++++++++++++----
 1 file changed, 27 insertions(+), 4 deletions(-)
net/ipv4/inet_hashtables.c +27 -4
index 9bfcfd016e18..2f8f34ee62fb 100644
@@ -978,6 +978,21 @@ void inet_bhash2_reset_saddr(struct sock *sk)
 }
 EXPORT_SYMBOL_GPL(inet_bhash2_reset_saddr);
 
+/* SYMMETRY means the socket has the same local and remote port/ipaddr */
+#define INET_ADDR_SYMMETRY(sk) (inet_sk(sk)->inet_rcv_saddr == \
+				inet_sk(sk)->inet_daddr)
+#define INET_PORT_SYMMETRY(sk) (inet_sk(sk)->inet_num == \
+				ntohs(inet_sk(sk)->inet_dport))
+#define INET_PORT_SYMMETRY_MATCH(sk, port) (port == \
+					    ntohs(inet_sk(sk)->inet_dport))
+static inline int inet_tuple_symmetry(struct sock *sk)
+{
+	if (INET_ADDR_SYMMETRY(sk) && INET_PORT_SYMMETRY(sk))
+		return -EADDRNOTAVAIL;
+
+	return 0;
+}
+
 /* RFC 6056 3.3.4.  Algorithm 4: Double-Hash Port Selection Algorithm
  * Note that we use 32bit integers (vs RFC 'short integers')
  * because 2^16 is not a multiple of num_ephemeral and this
@@ -997,13 +1012,13 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,
 			struct sock *, __u16, struct inet_timewait_sock **))
 {
 	struct inet_hashinfo *hinfo = death_row->hashinfo;
+	bool tb_created = false, symmetry_test = false;
 	struct inet_bind_hashbucket *head, *head2;
 	struct inet_timewait_sock *tw = NULL;
 	int port = inet_sk(sk)->inet_num;
 	struct net *net = sock_net(sk);
 	struct inet_bind2_bucket *tb2;
 	struct inet_bind_bucket *tb;
-	bool tb_created = false;
 	u32 remaining, offset;
 	int ret, i, low, high;
 	bool local_ports;
@@ -1011,12 +1026,18 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,
 	u32 index;
 
 	if (port) {
-		local_bh_disable();
-		ret = check_established(death_row, sk, port, NULL);
-		local_bh_enable();
+		ret = inet_tuple_symmetry(sk);
+		if (!ret) {
+			local_bh_disable();
+			ret = check_established(death_row, sk, port, NULL);
+			local_bh_enable();
+		}
 		return ret;
 	}
 
+	if (INET_ADDR_SYMMETRY(sk))
+		symmetry_test = true;
+
 	l3mdev = inet_sk_bound_l3mdev(sk);
 
 	local_ports = inet_sk_get_local_port_range(sk, &low, &high);
@@ -1046,6 +1067,8 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,
 			port -= remaining;
 		if (inet_is_local_reserved_port(net, port))
 			continue;
+		if (symmetry_test && INET_PORT_SYMMETRY_MATCH(sk, port))
+			continue;
 		head = &hinfo->bhash[inet_bhashfn(net, port,
 						  hinfo->bhash_size)];
 		spin_lock_bh(&head->lock);
Signature · 1 line
2.37.3
author
#1 lore raw
Quoted 33 lines · Jason Xing

From: Jason Xing <[email protected]>

Four-tuple symmetry here means the socket has the same remote/local port and ipaddr, like this, 127.0.0.1:8000 -> 127.0.0.1:8000.

$ ss -nat | grep 8000
ESTAB      0      0          127.0.0.1:8000       127.0.0.1:8000

Before this patch, one client could start a connection successfully as above even without a listener, which means, the socket connects to its self. Then every time other threads trying to bind/listen on this port will encounter a failure surely, unless the thread owning the socket exits.

It can rarely happen on the loopback device when the connect() finds the same port as its remote port while listener is not running. It has the side-effect on other threads. Besides, this solo flow has no merit, no significance at all.

After this patch, the moment we try to connect with a 4-tuple symmetry socket, we will get an error "connect: Cannot assign requested address".

---
 net/ipv4/inet_hashtables.c | 31 +++++++++++++++++++++++++++----
 1 file changed, 27 insertions(+), 4 deletions(-)
net/ipv4/inet_hashtables.c +0 -0
index 9bfcfd016e18..2f8f34ee62fb 100644
@@ -978,6 +978,21 @@ void inet_bhash2_reset_saddr(struct sock *sk)
 }

net/ipv4/inet_hashtables.c void inet_bhash2_reset_saddr(struct sock *sk) · line 988

Quoted 10 lines
net/ipv4/inet_hashtables.c +8 -0
 EXPORT_SYMBOL_GPL(inet_bhash2_reset_saddr);
 
+/* SYMMETRY means the socket has the same local and remote port/ipaddr */
+#define INET_ADDR_SYMMETRY(sk) (inet_sk(sk)->inet_rcv_saddr == \
+                               inet_sk(sk)->inet_daddr)
+#define INET_PORT_SYMMETRY(sk) (inet_sk(sk)->inet_num == \
+                               ntohs(inet_sk(sk)->inet_dport))
+#define INET_PORT_SYMMETRY_MATCH(sk, port) (port == \
+                                           ntohs(inet_sk(sk)->inet_dport))
+static inline int inet_tuple_symmetry(struct sock *sk)

Patchwork shows to me that I should not use inline in the .c file. I will change it in due course in v2 patch.

author
#2 lore raw
Quoted 9 lines · Jason Xing
Quoted 7 lines · Jason Xing

From: Jason Xing <[email protected]>

Four-tuple symmetry here means the socket has the same remote/local port and ipaddr, like this, 127.0.0.1:8000 -> 127.0.0.1:8000.

$ ss -nat | grep 8000
ESTAB      0      0          127.0.0.1:8000       127.0.0.1:8000

Thanks to the failed tests appearing in patchwork, now I'm aware of the technical term called "self-connection" in English to describe this case. I will update accordingly the title, body messages, function name by introducing "self-connection" words like this in the next submission.

Following this clue, I saw many reports happening in these years, like [1][2]. Users are often astonished about this phenomenon and lost and have to find various ways to workaround it. Since, in my opinion, the self-connection doesn't have any advantage and usefulness, why not avoid it in the kernel? Could networking experts enlighten me? Thanks.

+ Dmitry
Hello Dmitry, do you know why the self-connect_ipv4/6 was introduced in the selftests which the patch I wrote failed? Thanks.

[1]: https://adil.medium.com/what-is-tcp-self-connect-issue-be7d7b5f9f59
[2]: https://stackoverflow.com/questions/5139808/tcp-simultaneous-open-and-self-connect-prevention

Thanks,
Jason

#3 lore raw

From: Jason Xing <[email protected]>
Date: Sun, 18 Aug 2024 21:50:51 +0800

Quoted 17 lines · Jason Xing
Quoted 9 lines · Jason Xing
Quoted 7 lines

From: Jason Xing <[email protected]>

Four-tuple symmetry here means the socket has the same remote/local port and ipaddr, like this, 127.0.0.1:8000 -> 127.0.0.1:8000.

$ ss -nat | grep 8000
ESTAB      0      0          127.0.0.1:8000       127.0.0.1:8000

Thanks to the failed tests appearing in patchwork, now I'm aware of the technical term called "self-connection" in English to describe this case. I will update accordingly the title, body messages, function name by introducing "self-connection" words like this in the next submission.

Quoted 4 lines

Following this clue, I saw many reports happening in these years, like [1][2]. Users are often astonished about this phenomenon and lost and have to find various ways to workaround it. Since, in my opinion, the self-connection doesn't have any advantage and usefulness,

It's useful if you want to test simultaneous connect (SYN_SENT -> SYN_RECV) path as you see in TCP-AO tests. See RFC 9293 and the (!ack && syn) case in tcp_rcv_synsent_state_process().

https://www.rfc-editor.org/rfc/rfc9293.html#section-3.5-7

So you can't remove self-connect functionality, the recent main user is syzkaller though.

author
#4 lore raw

Hello Kuniyuki,

Quoted 25 lines · Kuniyuki Iwashima

From: Jason Xing <[email protected]>
Date: Sun, 18 Aug 2024 21:50:51 +0800

Quoted 17 lines · Jason Xing
Quoted 9 lines · Jason Xing
Quoted 7 lines

From: Jason Xing <[email protected]>

Four-tuple symmetry here means the socket has the same remote/local port and ipaddr, like this, 127.0.0.1:8000 -> 127.0.0.1:8000.

$ ss -nat | grep 8000
ESTAB      0      0          127.0.0.1:8000       127.0.0.1:8000

Thanks to the failed tests appearing in patchwork, now I'm aware of the technical term called "self-connection" in English to describe this case. I will update accordingly the title, body messages, function name by introducing "self-connection" words like this in the next submission.

Quoted 4 lines

Following this clue, I saw many reports happening in these years, like [1][2]. Users are often astonished about this phenomenon and lost and have to find various ways to workaround it. Since, in my opinion, the self-connection doesn't have any advantage and usefulness,

Quoted 5 lines

It's useful if you want to test simultaneous connect (SYN_SENT -> SYN_RECV) path as you see in TCP-AO tests. See RFC 9293 and the (!ack && syn) case in tcp_rcv_synsent_state_process().

https://www.rfc-editor.org/rfc/rfc9293.html#section-3.5-7

Yes, I noticed this one: self-connection is one particular case among simultaneously open cases. Honestly, it's really strange that client and server uses a single socket.

Quoted 3 lines

So you can't remove self-connect functionality, the recent main user is syzkaller though.

Ah, thanks for reminding me. It seems that I have to drop this patch and there is no good way to resolve the issue in the kernel.

Thanks,
Jason

author
#5 lore raw
Quoted 40 lines · Jason Xing

Hello Kuniyuki,

Quoted 25 lines · Kuniyuki Iwashima

From: Jason Xing <[email protected]>
Date: Sun, 18 Aug 2024 21:50:51 +0800

Quoted 17 lines · Jason Xing
Quoted 9 lines · Jason Xing
Quoted 7 lines

From: Jason Xing <[email protected]>

Four-tuple symmetry here means the socket has the same remote/local port and ipaddr, like this, 127.0.0.1:8000 -> 127.0.0.1:8000.

$ ss -nat | grep 8000
ESTAB      0      0          127.0.0.1:8000       127.0.0.1:8000

Thanks to the failed tests appearing in patchwork, now I'm aware of the technical term called "self-connection" in English to describe this case. I will update accordingly the title, body messages, function name by introducing "self-connection" words like this in the next submission.

Quoted 4 lines

Following this clue, I saw many reports happening in these years, like [1][2]. Users are often astonished about this phenomenon and lost and have to find various ways to workaround it. Since, in my opinion, the self-connection doesn't have any advantage and usefulness,

Quoted 5 lines

It's useful if you want to test simultaneous connect (SYN_SENT -> SYN_RECV) path as you see in TCP-AO tests. See RFC 9293 and the (!ack && syn) case in tcp_rcv_synsent_state_process().

https://www.rfc-editor.org/rfc/rfc9293.html#section-3.5-7

Yes, I noticed this one: self-connection is one particular case among simultaneously open cases. Honestly, it's really strange that client and server uses a single socket.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

So you can't remove self-connect functionality, the recent main user is syzkaller though.

Ah, thanks for reminding me. It seems that I have to drop this patch and there is no good way to resolve the issue in the kernel.

Can we introduce one sysctl knob to control it since we can tell there are many user reports/complaints through the internet? Default setting of the new knob is to allow users to connect to itself like right now, not interfering with many years of habits, like what the test tools currently use.

Can I give it a shot?

Thanks,
Jason

#6 lore raw
Quoted 49 lines · Jason Xing
Quoted 40 lines · Jason Xing

Hello Kuniyuki,

Quoted 25 lines · Kuniyuki Iwashima

From: Jason Xing <[email protected]>
Date: Sun, 18 Aug 2024 21:50:51 +0800

Quoted 17 lines · Jason Xing
Quoted 9 lines · Jason Xing
Quoted 7 lines

From: Jason Xing <[email protected]>

Four-tuple symmetry here means the socket has the same remote/local port and ipaddr, like this, 127.0.0.1:8000 -> 127.0.0.1:8000.

$ ss -nat | grep 8000
ESTAB      0      0          127.0.0.1:8000       127.0.0.1:8000

Thanks to the failed tests appearing in patchwork, now I'm aware of the technical term called "self-connection" in English to describe this case. I will update accordingly the title, body messages, function name by introducing "self-connection" words like this in the next submission.

Quoted 4 lines

Following this clue, I saw many reports happening in these years, like [1][2]. Users are often astonished about this phenomenon and lost and have to find various ways to workaround it. Since, in my opinion, the self-connection doesn't have any advantage and usefulness,

Quoted 5 lines

It's useful if you want to test simultaneous connect (SYN_SENT -> SYN_RECV) path as you see in TCP-AO tests. See RFC 9293 and the (!ack && syn) case in tcp_rcv_synsent_state_process().

https://www.rfc-editor.org/rfc/rfc9293.html#section-3.5-7

Yes, I noticed this one: self-connection is one particular case among simultaneously open cases. Honestly, it's really strange that client and server uses a single socket.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

So you can't remove self-connect functionality, the recent main user is syzkaller though.

Ah, thanks for reminding me. It seems that I have to drop this patch and there is no good way to resolve the issue in the kernel.

Quoted 7 lines

Can we introduce one sysctl knob to control it since we can tell there are many user reports/complaints through the internet? Default setting of the new knob is to allow users to connect to itself like right now, not interfering with many years of habits, like what the test tools currently use.

Can I give it a shot?

No you can not.

netfilter can probably do this.

If it can not, it could be augmented I think.

author
#7 lore raw

Hello Eric,

Quoted 57 lines · Eric Dumazet
Quoted 55 lines · Jason Xing
Quoted 40 lines · Jason Xing

Hello Kuniyuki,

Quoted 25 lines · Kuniyuki Iwashima

From: Jason Xing <[email protected]>
Date: Sun, 18 Aug 2024 21:50:51 +0800

Quoted 17 lines · Jason Xing
Quoted 9 lines · Jason Xing
Quoted 7 lines

From: Jason Xing <[email protected]>

Four-tuple symmetry here means the socket has the same remote/local port and ipaddr, like this, 127.0.0.1:8000 -> 127.0.0.1:8000.

$ ss -nat | grep 8000
ESTAB      0      0          127.0.0.1:8000       127.0.0.1:8000

Thanks to the failed tests appearing in patchwork, now I'm aware of the technical term called "self-connection" in English to describe this case. I will update accordingly the title, body messages, function name by introducing "self-connection" words like this in the next submission.

Quoted 4 lines

Following this clue, I saw many reports happening in these years, like [1][2]. Users are often astonished about this phenomenon and lost and have to find various ways to workaround it. Since, in my opinion, the self-connection doesn't have any advantage and usefulness,

Quoted 5 lines

It's useful if you want to test simultaneous connect (SYN_SENT -> SYN_RECV) path as you see in TCP-AO tests. See RFC 9293 and the (!ack && syn) case in tcp_rcv_synsent_state_process().

https://www.rfc-editor.org/rfc/rfc9293.html#section-3.5-7

Yes, I noticed this one: self-connection is one particular case among simultaneously open cases. Honestly, it's really strange that client and server uses a single socket.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

So you can't remove self-connect functionality, the recent main user is syzkaller though.

Ah, thanks for reminding me. It seems that I have to drop this patch and there is no good way to resolve the issue in the kernel.

Can we introduce one sysctl knob to control it since we can tell there are many user reports/complaints through the internet? Default setting of the new knob is to allow users to connect to itself like right now, not interfering with many years of habits, like what the test tools currently use.

Quoted 3 lines
Quoted 1 line

Can I give it a shot?

No you can not.

May I ask why? Is it because self-connection adheres to the simultaneously open part in RFC 9293?

I feel this case is very particular, not explained well in the RFC. Usually, we don't consider one socket to act as client and server unless in debug or test circumstances. As you can see, some people have encountered the issue for a long time.

Quoted 2 lines

netfilter can probably do this.

Sure, It can do. It can be a little bit helpful, but clumsy. We have to set specific rules for each possible listener and then drop those SYN packets if they carry the same remote and local port/ip.

Thanks,
Jason

#8 lore raw
Quoted 61 lines · Jason Xing

Hello Eric,

Quoted 57 lines · Eric Dumazet
Quoted 55 lines · Jason Xing
Quoted 40 lines · Jason Xing

Hello Kuniyuki,

Quoted 25 lines · Kuniyuki Iwashima

From: Jason Xing <[email protected]>
Date: Sun, 18 Aug 2024 21:50:51 +0800

Quoted 17 lines · Jason Xing
Quoted 9 lines · Jason Xing
Quoted 7 lines

From: Jason Xing <[email protected]>

Four-tuple symmetry here means the socket has the same remote/local port and ipaddr, like this, 127.0.0.1:8000 -> 127.0.0.1:8000.

$ ss -nat | grep 8000
ESTAB      0      0          127.0.0.1:8000       127.0.0.1:8000

Thanks to the failed tests appearing in patchwork, now I'm aware of the technical term called "self-connection" in English to describe this case. I will update accordingly the title, body messages, function name by introducing "self-connection" words like this in the next submission.

Quoted 4 lines

Following this clue, I saw many reports happening in these years, like [1][2]. Users are often astonished about this phenomenon and lost and have to find various ways to workaround it. Since, in my opinion, the self-connection doesn't have any advantage and usefulness,

Quoted 5 lines

It's useful if you want to test simultaneous connect (SYN_SENT -> SYN_RECV) path as you see in TCP-AO tests. See RFC 9293 and the (!ack && syn) case in tcp_rcv_synsent_state_process().

https://www.rfc-editor.org/rfc/rfc9293.html#section-3.5-7

Yes, I noticed this one: self-connection is one particular case among simultaneously open cases. Honestly, it's really strange that client and server uses a single socket.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

So you can't remove self-connect functionality, the recent main user is syzkaller though.

Ah, thanks for reminding me. It seems that I have to drop this patch and there is no good way to resolve the issue in the kernel.

Can we introduce one sysctl knob to control it since we can tell there are many user reports/complaints through the internet? Default setting of the new knob is to allow users to connect to itself like right now, not interfering with many years of habits, like what the test tools currently use.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Can I give it a shot?

No you can not.

May I ask why? Is it because self-connection adheres to the simultaneously open part in RFC 9293?

This will break some user programs, obviously.

I will ask you the opposite : What RFC prevents the current situation ?

Quoted 5 lines

I feel this case is very particular, not explained well in the RFC. Usually, we don't consider one socket to act as client and server unless in debug or test circumstances. As you can see, some people have encountered the issue for a long time.

Can you provide links to the issues ? How can a programmer hit this using standard and documented ways (passive, active flows) ?

Quoted 10 lines
Quoted 3 lines

netfilter can probably do this.

Sure, It can do. It can be a little bit helpful, but clumsy. We have to set specific rules for each possible listener and then drop those SYN packets if they carry the same remote and local port/ip.

Thanks,
Jason

author
#9 lore raw
Quoted 63 lines · Eric Dumazet
Quoted 61 lines · Jason Xing

Hello Eric,

Quoted 57 lines · Eric Dumazet
Quoted 55 lines · Jason Xing
Quoted 40 lines · Jason Xing

Hello Kuniyuki,

Quoted 25 lines · Kuniyuki Iwashima

From: Jason Xing <[email protected]>
Date: Sun, 18 Aug 2024 21:50:51 +0800

Quoted 17 lines · Jason Xing
Quoted 9 lines · Jason Xing
Quoted 7 lines

From: Jason Xing <[email protected]>

Four-tuple symmetry here means the socket has the same remote/local port and ipaddr, like this, 127.0.0.1:8000 -> 127.0.0.1:8000.

$ ss -nat | grep 8000
ESTAB      0      0          127.0.0.1:8000       127.0.0.1:8000

Thanks to the failed tests appearing in patchwork, now I'm aware of the technical term called "self-connection" in English to describe this case. I will update accordingly the title, body messages, function name by introducing "self-connection" words like this in the next submission.

Quoted 4 lines

Following this clue, I saw many reports happening in these years, like [1][2]. Users are often astonished about this phenomenon and lost and have to find various ways to workaround it. Since, in my opinion, the self-connection doesn't have any advantage and usefulness,

Quoted 5 lines

It's useful if you want to test simultaneous connect (SYN_SENT -> SYN_RECV) path as you see in TCP-AO tests. See RFC 9293 and the (!ack && syn) case in tcp_rcv_synsent_state_process().

https://www.rfc-editor.org/rfc/rfc9293.html#section-3.5-7

Yes, I noticed this one: self-connection is one particular case among simultaneously open cases. Honestly, it's really strange that client and server uses a single socket.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

So you can't remove self-connect functionality, the recent main user is syzkaller though.

Ah, thanks for reminding me. It seems that I have to drop this patch and there is no good way to resolve the issue in the kernel.

Can we introduce one sysctl knob to control it since we can tell there are many user reports/complaints through the internet? Default setting of the new knob is to allow users to connect to itself like right now, not interfering with many years of habits, like what the test tools currently use.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Can I give it a shot?

No you can not.

May I ask why? Is it because self-connection adheres to the simultaneously open part in RFC 9293?

This will break some user programs, obviously.

I agree. It'a headache.

Quoted 2 lines

I will ask you the opposite : What RFC prevents the current situation ?

Not really, actually. The reason why I wrote the patch is because it indeed happened.

Quoted 10 lines
Quoted 6 lines

I feel this case is very particular, not explained well in the RFC. Usually, we don't consider one socket to act as client and server unless in debug or test circumstances. As you can see, some people have encountered the issue for a long time.

Can you provide links to the issues ? How can a programmer hit this using standard and documented ways (passive, active flows) ?

I've listed some of them in the previous discussion through googling
"tcp self connection" or something like that. Let me copy here:
[1]: https://adil.medium.com/what-is-tcp-self-connect-issue-be7d7b5f9f59
[2]: https://stackoverflow.com/questions/5139808/tcp-simultaneous-open-and-self-connect-prevention

After investigating such an issue more deeply in the customers' machines, the main reason why it can happen is the listener exits while another thread starts to connect, which can cause self-connection, even though the chance is slim. Later, the listener tries to listen and for sure it will fail due to that single self-connection.

Thanks,
Jason

#10 lore raw
Quoted 6 lines · Jason Xing

After investigating such an issue more deeply in the customers' machines, the main reason why it can happen is the listener exits while another thread starts to connect, which can cause self-connection, even though the chance is slim. Later, the listener tries to listen and for sure it will fail due to that single self-connection.

This would happen if the range of ephemeral ports include the listening port, which is discouraged.

ip_local_reserved_ports is supposed to help.

This looks like a security issue to me, and netfilter can handle it.

author
#11 lore raw
Quoted 10 lines · Eric Dumazet
Quoted 6 lines · Jason Xing

After investigating such an issue more deeply in the customers' machines, the main reason why it can happen is the listener exits while another thread starts to connect, which can cause self-connection, even though the chance is slim. Later, the listener tries to listen and for sure it will fail due to that single self-connection.

Quoted 2 lines

This would happen if the range of ephemeral ports include the listening port, which is discouraged.

Yes.

Quoted 2 lines

ip_local_reserved_ports is supposed to help.

Sure, I workarounded it by using this and it worked.

Quoted 2 lines

This looks like a security issue to me, and netfilter can handle it.

I have to admit setting netfilter rules for each flow is not a very user-friendly way.

Anyway, really thanks for your suggestion :)

Thanks,
Jason

#12 lore raw

From: Jason Xing <[email protected]>
Date: Mon, 19 Aug 2024 17:41:32 +0800

Quoted 19 lines · Eric Dumazet
Quoted 10 lines · Jason Xing
Quoted 7 lines

After investigating such an issue more deeply in the customers' machines, the main reason why it can happen is the listener exits while another thread starts to connect, which can cause self-connection, even though the chance is slim. Later, the listener tries to listen and for sure it will fail due to that single self-connection.

Quoted 2 lines

This would happen if the range of ephemeral ports include the listening port, which is discouraged.

Yes.

Quoted 2 lines

ip_local_reserved_ports is supposed to help.

Quoted 7 lines

Sure, I workarounded it by using this and it worked.

Quoted 2 lines

This looks like a security issue to me, and netfilter can handle it.

I have to admit setting netfilter rules for each flow is not a very user-friendly way.

I think even netfilter is not needed.

It sounds like the server application needs to implement graceful shutdown. The server should not release the port if there are on-going clients. The server should spin up a new process and use the following to transfer the remaining connections:

* FD passing
* SO_REUSEPORT w/ (net.ipv4.tcp_migrate_req or BPF)

Then, no client can occupy the server's port even without ip_local_reserved_ports.

But I still recommend using ip_local_reserved_ports unless the server port is random.

author
#13 lore raw
Quoted 37 lines · Kuniyuki Iwashima

From: Jason Xing <[email protected]>
Date: Mon, 19 Aug 2024 17:41:32 +0800

Quoted 19 lines · Eric Dumazet
Quoted 10 lines · Jason Xing
Quoted 7 lines

After investigating such an issue more deeply in the customers' machines, the main reason why it can happen is the listener exits while another thread starts to connect, which can cause self-connection, even though the chance is slim. Later, the listener tries to listen and for sure it will fail due to that single self-connection.

Quoted 2 lines

This would happen if the range of ephemeral ports include the listening port, which is discouraged.

Yes.

Quoted 2 lines

ip_local_reserved_ports is supposed to help.

Quoted 7 lines

Sure, I workarounded it by using this and it worked.

Quoted 2 lines

This looks like a security issue to me, and netfilter can handle it.

I have to admit setting netfilter rules for each flow is not a very user-friendly way.

I think even netfilter is not needed.

It sounds like the server application needs to implement graceful shutdown. The server should not release the port if there are on-going clients. The server should spin up a new process and use the following to transfer the remaining connections:

Quoted 8 lines

* FD passing
* SO_REUSEPORT w/ (net.ipv4.tcp_migrate_req or BPF)

Then, no client can occupy the server's port even without ip_local_reserved_ports.

But I still recommend using ip_local_reserved_ports unless the server port is random.

Yes, there are some ways that can mitigate or solve the issue.

The reason why I wrote the patch is because at the beginning I don't think the self-connection feature is useful which you reminded me of some test tools like syzbot could use. That's not what I was aware of. Thanks for your reply.

Thanks,
Jason