Skip to contentSkip to conversation

tcp: tcp_v4_err() changes

PATCH In review Eric Dumazet 6 participants 23 messages Apr 17, 2024 – Apr 18, 2024

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover tcp: tcp_v4_err() changes

First patch fixes a bug in tcp_v4_err().

Fixing this old bug allows us to bring back a patch that we had to revert. Hopefully linux TCP can better meet RFC 1122 requirements.

Eric Dumazet (2):
tcp: conditionally call ip_icmp_error() from tcp_v4_err()
tcp: no longer abort SYN_SENT when receiving some ICMP (II)

 net/ipv4/tcp_ipv4.c | 9 ++++++++-
 net/ipv6/tcp_ipv6.c | 9 ++++++---
 2 files changed, 14 insertions(+), 4 deletions(-)
Signature · 1 line
2.44.0.683.g7961c838ac-goog
#1 lore raw

PATCH 1/2 tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>
---
 net/ipv4/tcp_ipv4.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/ipv4/tcp_ipv4.c +2 -1
index 88c83ac4212957f19efad0f967952d2502bdbc7f..a717db99972d977a64178d7ed1109325d64a6d51 100644
@@ -602,7 +602,8 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
 		if (fastopen && !fastopen->sk)
 			break;
 
-		ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
+		if (inet_test_bit(RECVERR, sk))
+			ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
 
 		if (!sock_owned_by_user(sk)) {
 			WRITE_ONCE(sk->sk_err, err);
Signature · 1 line
2.44.0.683.g7961c838ac-goog
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 62 lines · Eric Dumazet

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>
---
 net/ipv4/tcp_ipv4.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/ipv4/tcp_ipv4.c +2 -2
index 88c83ac4212957f19efad0f967952d2502bdbc7f..a717db99972d977a64178d7ed1109325d64a6d51 100644
@@ -602,7 +602,8 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
                if (fastopen && !fastopen->sk)
                        break;
 
-               ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
+               if (inet_test_bit(RECVERR, sk))
+                       ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
 
                if (!sock_owned_by_user(sk)) {
                        WRITE_ONCE(sk->sk_err, err);
--
Quoted 2 lines

2.44.0.683.g7961c838ac-goog

Makes sense to me.

Reviewed-by on v1 1/2
#3 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

In reply to Eric Dumazet
Quoted 35 lines · Eric Dumazet

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Quoted 8 lines
Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>

I wonder if we're supposed to move this check into ip_icmp_error() like ipv6_icmp_error() does, because I notice one caller rxrpc_encap_err_rcv() without checking RECVERR bit reuses the ICMP error logic which is introduced in commit b6c66c4324e7 ("rxrpc: Use the core ICMP/ICMP6 parsers'')?

Or should it be a follow-up patch (moving it inside of ip_icmp_error()) to handle the rxrpc case and also prevent future misuse for other people?

Thanks,
Jason

#4 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 46 lines · Jason Xing
Quoted 35 lines · Eric Dumazet

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Quoted 8 lines
Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>
Quoted 7 lines

I wonder if we're supposed to move this check into ip_icmp_error() like ipv6_icmp_error() does, because I notice one caller rxrpc_encap_err_rcv() without checking RECVERR bit reuses the ICMP error logic which is introduced in commit b6c66c4324e7 ("rxrpc: Use the core ICMP/ICMP6 parsers'')?

I tried to focus on the TCP issues, and to have a stable candidate for patch #1.

The refactoring can wait.

Quoted 4 lines

Or should it be a follow-up patch (moving it inside of ip_icmp_error()) to handle the rxrpc case and also prevent future misuse for other people?

#5 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 56 lines · Eric Dumazet
Quoted 46 lines · Jason Xing
Quoted 35 lines · Eric Dumazet

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Quoted 8 lines
Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>
Quoted 7 lines

I wonder if we're supposed to move this check into ip_icmp_error() like ipv6_icmp_error() does, because I notice one caller rxrpc_encap_err_rcv() without checking RECVERR bit reuses the ICMP error logic which is introduced in commit b6c66c4324e7 ("rxrpc: Use the core ICMP/ICMP6 parsers'')?

Quoted 3 lines

I tried to focus on the TCP issues, and to have a stable candidate for patch #1.

The refactoring can wait.

Got it. It's clear.

After this patch is applied, I can adjust a little bit (only by moving it into ip_icmp_error()).

Thanks,
Jason

Quoted 5 lines
Quoted 5 lines

Or should it be a follow-up patch (moving it inside of ip_icmp_error()) to handle the rxrpc case and also prevent future misuse for other people?

maintainer
#6 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

In reply to Eric Dumazet

Hi,

Quoted 51 lines · Eric Dumazet

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>
---
 net/ipv4/tcp_ipv4.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/ipv4/tcp_ipv4.c +0 -0
index 88c83ac4212957f19efad0f967952d2502bdbc7f..a717db99972d977a64178d7ed1109325d64a6d51 100644

net/ipv4/tcp_ipv4.c int tcp_v4_err(struct sk_buff *skb, u32 info) · line 609

Quoted 9 lines
net/ipv4/tcp_ipv4.c +2 -1
@@ -602,7 +602,8 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
 		if (fastopen && !fastopen->sk)
 			break;
 
-		ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
+		if (inet_test_bit(RECVERR, sk))
+			ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
 
 		if (!sock_owned_by_user(sk)) {
 			WRITE_ONCE(sk->sk_err, err);

We have a fcnal-test.sh self-test failure:

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-04-18--06-00&test=fcnal-test-sh

that I suspect are related to this patch (or the following one): the test case creates a TCP connection on loopback and this is the only patchseries touching the related code, included in the relevant patch burst.

Could you please have a look?

Thanks!

Paolo

#7 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 67 lines · Paolo Abeni

Hi,

Quoted 51 lines · Eric Dumazet

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>
---
 net/ipv4/tcp_ipv4.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/ipv4/tcp_ipv4.c +0 -0
index 88c83ac4212957f19efad0f967952d2502bdbc7f..a717db99972d977a64178d7ed1109325d64a6d51 100644

net/ipv4/tcp_ipv4.c int tcp_v4_err(struct sk_buff *skb, u32 info) · line 609

Quoted 9 lines
net/ipv4/tcp_ipv4.c +2 -1
@@ -602,7 +602,8 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
              if (fastopen && !fastopen->sk)
                      break;
 
-             ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
+             if (inet_test_bit(RECVERR, sk))
+                     ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
 
              if (!sock_owned_by_user(sk)) {
                      WRITE_ONCE(sk->sk_err, err);

We have a fcnal-test.sh self-test failure:

Quoted 8 lines

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-04-18--06-00&test=fcnal-test-sh

that I suspect are related to this patch (or the following one): the test case creates a TCP connection on loopback and this is the only patchseries touching the related code, included in the relevant patch burst.

Could you please have a look?

Sure, thanks Paolo !

#8 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 71 lines · Eric Dumazet
Quoted 69 lines · Paolo Abeni

Hi,

Quoted 51 lines · Eric Dumazet

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>
---
 net/ipv4/tcp_ipv4.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/ipv4/tcp_ipv4.c +0 -0
index 88c83ac4212957f19efad0f967952d2502bdbc7f..a717db99972d977a64178d7ed1109325d64a6d51 100644

net/ipv4/tcp_ipv4.c int tcp_v4_err(struct sk_buff *skb, u32 info) · line 609

Quoted 9 lines
net/ipv4/tcp_ipv4.c +2 -1
@@ -602,7 +602,8 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
              if (fastopen && !fastopen->sk)
                      break;
 
-             ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
+             if (inet_test_bit(RECVERR, sk))
+                     ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
 
              if (!sock_owned_by_user(sk)) {
                      WRITE_ONCE(sk->sk_err, err);

We have a fcnal-test.sh self-test failure:

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-04-18--06-00&test=fcnal-test-sh

Quoted 8 lines
Quoted 6 lines

that I suspect are related to this patch (or the following one): the test case creates a TCP connection on loopback and this is the only patchseries touching the related code, included in the relevant patch burst.

Could you please have a look?

Sure, thanks Paolo !

First patch is fine, I see no failure from fcnal-test.sh (as I would expect)

For the second one, I am not familiar enough with this very slow test suite (all these "sleep 1" ... oh well)

I guess "failing tests" depended on TCP connect() to immediately abort on one ICMP message,
depending on old kernel behavior.

I do not know how to launch a subset of the tests, and trace these.

"./fcnal-test.sh -t ipv4_tcp" alone takes more than 9 minutes [1] in a VM running a non debug kernel :/

David, do you have an idea how to proceed ?

Thanks.

[1]
Tests passed: 134
Tests failed: 0

real 9m33.085s
user 0m40.159s
sys 0m30.098s

maintainer
#9 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 77 lines · Eric Dumazet
Quoted 76 lines · Eric Dumazet
Quoted 74 lines · Paolo Abeni

Hi,

Quoted 51 lines · Eric Dumazet

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>
---
 net/ipv4/tcp_ipv4.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/ipv4/tcp_ipv4.c +0 -0
index 88c83ac4212957f19efad0f967952d2502bdbc7f..a717db99972d977a64178d7ed1109325d64a6d51 100644

net/ipv4/tcp_ipv4.c int tcp_v4_err(struct sk_buff *skb, u32 info) · line 609

Quoted 9 lines
net/ipv4/tcp_ipv4.c +2 -1
@@ -602,7 +602,8 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
              if (fastopen && !fastopen->sk)
                      break;
 
-             ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
+             if (inet_test_bit(RECVERR, sk))
+                     ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
 
              if (!sock_owned_by_user(sk)) {
                      WRITE_ONCE(sk->sk_err, err);

We have a fcnal-test.sh self-test failure:

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-04-18--06-00&test=fcnal-test-sh

that I suspect are related to this patch (or the following one): the test case creates a TCP connection on loopback and this is the only patchseries touching the related code, included in the relevant patch burst.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Could you please have a look?

Sure, thanks Paolo !

First patch is fine, I see no failure from fcnal-test.sh (as I would expect)

For the second one, I am not familiar enough with this very slow test suite (all these "sleep 1" ... oh well)

@David, some of them could be replaced with loopy_wait calls

Quoted 10 lines

I guess "failing tests" depended on TCP connect() to immediately abort on one ICMP message,
depending on old kernel behavior.

I do not know how to launch a subset of the tests, and trace these.

"./fcnal-test.sh -t ipv4_tcp" alone takes more than 9 minutes [1] in a VM running a non debug kernel :/

David, do you have an idea how to proceed ?

One very dumb thing I do in that cases is commenting out the other tests, something alike (completely untested!):
---

tools/testing/selftests/net/fcnal-test.sh +4 -0
index 386ebd829df5..494932aa99b2 100755
@@ -1186,6 +1186,7 @@ ipv4_tcp_novrf()
 {
 	local a
 
+if false; then
 	#
 	# server tests
 	#
@@ -1271,6 +1272,7 @@ ipv4_tcp_novrf()
 		log_test_addr ${a} $? 1 "Device server, unbound client, local connection"
 	done
 
+fi
 	a=${NSA_IP}
 	log_start
 	run_cmd nettest -s &
@@ -1487,12 +1489,14 @@ ipv4_tcp()
 	set_sysctl net.ipv4.tcp_l3mdev_accept=0
 	ipv4_tcp_novrf
 	log_subsection "tcp_l3mdev_accept enabled"
+if false; then
 	set_sysctl net.ipv4.tcp_l3mdev_accept=1
 	ipv4_tcp_novrf
 
 	log_subsection "With VRF"
 	setup "yes"
 	ipv4_tcp_vrf
+fi
 }
 
 ################################################################################
#10 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 127 lines · Paolo Abeni
Quoted 77 lines · Eric Dumazet
Quoted 76 lines · Eric Dumazet
Quoted 74 lines · Paolo Abeni

Hi,

Quoted 51 lines · Eric Dumazet

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>
---
 net/ipv4/tcp_ipv4.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/ipv4/tcp_ipv4.c +0 -0
index 88c83ac4212957f19efad0f967952d2502bdbc7f..a717db99972d977a64178d7ed1109325d64a6d51 100644

net/ipv4/tcp_ipv4.c int tcp_v4_err(struct sk_buff *skb, u32 info) · line 609

Quoted 9 lines
net/ipv4/tcp_ipv4.c +2 -1
@@ -602,7 +602,8 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
              if (fastopen && !fastopen->sk)
                      break;
 
-             ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
+             if (inet_test_bit(RECVERR, sk))
+                     ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
 
              if (!sock_owned_by_user(sk)) {
                      WRITE_ONCE(sk->sk_err, err);

We have a fcnal-test.sh self-test failure:

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-04-18--06-00&test=fcnal-test-sh

that I suspect are related to this patch (or the following one): the test case creates a TCP connection on loopback and this is the only patchseries touching the related code, included in the relevant patch burst.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Could you please have a look?

Sure, thanks Paolo !

First patch is fine, I see no failure from fcnal-test.sh (as I would expect)

For the second one, I am not familiar enough with this very slow test suite (all these "sleep 1" ... oh well)

@David, some of them could be replaced with loopy_wait calls

Quoted 10 lines

I guess "failing tests" depended on TCP connect() to immediately abort on one ICMP message,
depending on old kernel behavior.

I do not know how to launch a subset of the tests, and trace these.

"./fcnal-test.sh -t ipv4_tcp" alone takes more than 9 minutes [1] in a VM running a non debug kernel :/

David, do you have an idea how to proceed ?

One very dumb thing I do in that cases is commenting out the other tests, something alike (completely untested!):
---

tools/testing/selftests/net/fcnal-test.sh +2 -0
index 386ebd829df5..494932aa99b2 100755
@@ -1186,6 +1186,7 @@ ipv4_tcp_novrf()
 {
        local a
 
+if false; then
        #
        # server tests
        #
@@ -1271,6 +1272,7 @@ ipv4_tcp_novrf()
                log_test_addr ${a} $? 1 "Device server, unbound client, local connection"
        done
 
+fi
        a=${NSA_IP}
        log_start
        run_cmd nettest -s &
@@ -1487,12 +1489,14 @@ ipv4_tcp()
        set_sysctl net.ipv4.tcp_l3mdev_accept=0
        ipv4_tcp_novrf

tools/testing/selftests/net/fcnal-test.sh ipv4_tcp() · line 1500

Quoted 10 lines
tools/testing/selftests/net/fcnal-test.sh +2 -0
        log_subsection "tcp_l3mdev_accept enabled"
+if false; then
        set_sysctl net.ipv4.tcp_l3mdev_accept=1
        ipv4_tcp_novrf
 
        log_subsection "With VRF"
        setup "yes"
        ipv4_tcp_vrf
+fi
 }

Thanks Paolo

I found that the following patch is fixing the issue for me.

(quoted diff) +0 -0

b/tools/testing/selftests/net/nettest.c
index cd8a580974480212b45d86f35293b77f3d033473..ff25e53024ef6d4101f251c8a8a5e936e44e280f 100644

tools/testing/selftests/net/nettest.c +0 -0
@@ -1744,6 +1744,7 @@ static int connectsock(void *addr, socklen_t

alen, struct sock_args *args)

        if (args->bind_test_only)
                goto out;
+       set_recv_attr(sd, args->version);
        if (connect(sd, addr, alen) < 0) {
                if (errno != EINPROGRESS) {
                        log_err_errno("Failed to connect to remote host");
#11 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 158 lines · Eric Dumazet
Quoted 127 lines · Paolo Abeni
Quoted 77 lines · Eric Dumazet
Quoted 76 lines · Eric Dumazet
Quoted 74 lines · Paolo Abeni

Hi,

Quoted 51 lines · Eric Dumazet

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>
---
 net/ipv4/tcp_ipv4.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/ipv4/tcp_ipv4.c +0 -0
index 88c83ac4212957f19efad0f967952d2502bdbc7f..a717db99972d977a64178d7ed1109325d64a6d51 100644

net/ipv4/tcp_ipv4.c int tcp_v4_err(struct sk_buff *skb, u32 info) · line 609

Quoted 9 lines
net/ipv4/tcp_ipv4.c +2 -1
@@ -602,7 +602,8 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
              if (fastopen && !fastopen->sk)
                      break;
 
-             ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
+             if (inet_test_bit(RECVERR, sk))
+                     ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
 
              if (!sock_owned_by_user(sk)) {
                      WRITE_ONCE(sk->sk_err, err);

We have a fcnal-test.sh self-test failure:

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-04-18--06-00&test=fcnal-test-sh

that I suspect are related to this patch (or the following one): the test case creates a TCP connection on loopback and this is the only patchseries touching the related code, included in the relevant patch burst.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Could you please have a look?

Sure, thanks Paolo !

First patch is fine, I see no failure from fcnal-test.sh (as I would expect)

For the second one, I am not familiar enough with this very slow test suite (all these "sleep 1" ... oh well)

@David, some of them could be replaced with loopy_wait calls

Quoted 10 lines

I guess "failing tests" depended on TCP connect() to immediately abort on one ICMP message,
depending on old kernel behavior.

I do not know how to launch a subset of the tests, and trace these.

"./fcnal-test.sh -t ipv4_tcp" alone takes more than 9 minutes [1] in a VM running a non debug kernel :/

David, do you have an idea how to proceed ?

One very dumb thing I do in that cases is commenting out the other tests, something alike (completely untested!):
---

tools/testing/selftests/net/fcnal-test.sh +2 -0
index 386ebd829df5..494932aa99b2 100755
@@ -1186,6 +1186,7 @@ ipv4_tcp_novrf()
 {
        local a
 
+if false; then
        #
        # server tests
        #
@@ -1271,6 +1272,7 @@ ipv4_tcp_novrf()
                log_test_addr ${a} $? 1 "Device server, unbound client, local connection"
        done
 
+fi
        a=${NSA_IP}
        log_start
        run_cmd nettest -s &
@@ -1487,12 +1489,14 @@ ipv4_tcp()
        set_sysctl net.ipv4.tcp_l3mdev_accept=0
        ipv4_tcp_novrf

tools/testing/selftests/net/fcnal-test.sh ipv4_tcp() · line 1500

Quoted 10 lines
tools/testing/selftests/net/fcnal-test.sh +2 -0
        log_subsection "tcp_l3mdev_accept enabled"
+if false; then
        set_sysctl net.ipv4.tcp_l3mdev_accept=1
        ipv4_tcp_novrf
 
        log_subsection "With VRF"
        setup "yes"
        ipv4_tcp_vrf
+fi
 }

Thanks Paolo

I found that the following patch is fixing the issue for me.

tools/testing/selftests/net/nettest.c +0 -0
index cd8a580974480212b45d86f35293b77f3d033473..ff25e53024ef6d4101f251c8a8a5e936e44e280f

100644

tools/testing/selftests/net/nettest.c +1 -0
@@ -1744,6 +1744,7 @@ static int connectsock(void *addr, socklen_t
 alen, struct sock_args *args)
        if (args->bind_test_only)
                goto out;
 
+       set_recv_attr(sd, args->version);
        if (connect(sd, addr, alen) < 0) {
                if (errno != EINPROGRESS) {
Quoted 1 line
                        log_err_errno("Failed to connect to remote host");

When tracing nettest we now have EHOSTUNREACH

3343  setsockopt(3, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0 <0.000210>
3343  setsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, "eth1\0", 5) = 0 <0.000170>
3343  setsockopt(3, SOL_IP, IP_PKTINFO, [1], 4) = 0 <0.000161>
3343  setsockopt(3, SOL_IP, IP_RECVERR, [1], 4) = 0 <0.000181>
3343  connect(3, {sa_family=AF_INET, sin_port=htons(12345),

sin_addr=inet_addr("172.16.2.1")}, 16) = -1 EINPROGRESS (Operation now in progress) <0.000874>
3343 pselect6(1024, NULL, [3], NULL, {tv_sec=5, tv_nsec=0}, NULL) = 1 (out [3], left {tv_sec=1, tv_nsec=930762080}) <3.069673>
3343 getsockopt(3, SOL_SOCKET, SO_ERROR, [EHOSTUNREACH], [4]) = 0 <0.000270>

As mentioned in net/ipv4/icmp.c :
RFC 1122: 3.2.2.1 States that NET_UNREACH, HOST_UNREACH and SR_FAILED
MUST be considered 'transient errs'.

Maybe another way to fix nettest would be to change wait_for_connect() to pass a non NULL fdset in 4th argument of select()

select(FD_SETSIZE, NULL, &wfd, NULL /* here */, tv);

#12 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 175 lines · Eric Dumazet
Quoted 158 lines · Eric Dumazet
Quoted 127 lines · Paolo Abeni
Quoted 77 lines · Eric Dumazet
Quoted 76 lines · Eric Dumazet
Quoted 74 lines · Paolo Abeni

Hi,

Quoted 51 lines · Eric Dumazet

Blamed commit claimed in its changelog that the new functionality was guarded by IP_RECVERR/IPV6_RECVERR :

Note that applications need to set IP_RECVERR/IPV6_RECVERR option to enable this feature, and that the error message is only queued while in SYN_SNT state.

This was true only for IPv6, because ipv6_icmp_error() has the following check:

if (!inet6_test_bit(RECVERR6, sk))

    return;

Other callers check IP_RECVERR by themselves, it is unclear if we could factorize these checks in ip_icmp_error()

For stable backports, I chose to add the missing check in tcp_v4_err()

We think this missing check was the root cause for commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") breakage, leading to a revert.

Many thanks to Dragos Tatulea for conducting the investigations.

As Jakub said :

The suspicion is that SSH sees the ICMP report on the socket error queue and tries to connect() again, but due to the patch the socket isn't disconnected, so it gets EALREADY, and throws its hands up...

The error bubbles up to Vagrant which also becomes unhappy.

Can we skip the call to ip_icmp_error() for non-fatal ICMP errors?

Fixes 45af29ca761c ("tcp: allow traceroute -Mtcp for unpriv users")
Cc Dragos Tatulea <[email protected]>
Cc Maciej Żenczykowski <[email protected]>
Cc Willem de Bruijn <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Shachar Kagan <[email protected]>
---
 net/ipv4/tcp_ipv4.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/ipv4/tcp_ipv4.c +0 -0
index 88c83ac4212957f19efad0f967952d2502bdbc7f..a717db99972d977a64178d7ed1109325d64a6d51 100644

net/ipv4/tcp_ipv4.c int tcp_v4_err(struct sk_buff *skb, u32 info) · line 609

Quoted 9 lines
net/ipv4/tcp_ipv4.c +2 -1
@@ -602,7 +602,8 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
              if (fastopen && !fastopen->sk)
                      break;
 
-             ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
+             if (inet_test_bit(RECVERR, sk))
+                     ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
 
              if (!sock_owned_by_user(sk)) {
                      WRITE_ONCE(sk->sk_err, err);

We have a fcnal-test.sh self-test failure:

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-04-18--06-00&test=fcnal-test-sh

that I suspect are related to this patch (or the following one): the test case creates a TCP connection on loopback and this is the only patchseries touching the related code, included in the relevant patch burst.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Could you please have a look?

Sure, thanks Paolo !

First patch is fine, I see no failure from fcnal-test.sh (as I would expect)

For the second one, I am not familiar enough with this very slow test suite (all these "sleep 1" ... oh well)

@David, some of them could be replaced with loopy_wait calls

Quoted 10 lines

I guess "failing tests" depended on TCP connect() to immediately abort on one ICMP message,
depending on old kernel behavior.

I do not know how to launch a subset of the tests, and trace these.

"./fcnal-test.sh -t ipv4_tcp" alone takes more than 9 minutes [1] in a VM running a non debug kernel :/

David, do you have an idea how to proceed ?

One very dumb thing I do in that cases is commenting out the other tests, something alike (completely untested!):
---

tools/testing/selftests/net/fcnal-test.sh +2 -0
index 386ebd829df5..494932aa99b2 100755
@@ -1186,6 +1186,7 @@ ipv4_tcp_novrf()
 {
        local a
 
+if false; then
        #
        # server tests
        #
@@ -1271,6 +1272,7 @@ ipv4_tcp_novrf()
                log_test_addr ${a} $? 1 "Device server, unbound client, local connection"
        done
 
+fi
        a=${NSA_IP}
        log_start
        run_cmd nettest -s &
@@ -1487,12 +1489,14 @@ ipv4_tcp()
        set_sysctl net.ipv4.tcp_l3mdev_accept=0
        ipv4_tcp_novrf

tools/testing/selftests/net/fcnal-test.sh ipv4_tcp() · line 1500

Quoted 10 lines
tools/testing/selftests/net/fcnal-test.sh +2 -0
        log_subsection "tcp_l3mdev_accept enabled"
+if false; then
        set_sysctl net.ipv4.tcp_l3mdev_accept=1
        ipv4_tcp_novrf
 
        log_subsection "With VRF"
        setup "yes"
        ipv4_tcp_vrf
+fi
 }

Thanks Paolo

I found that the following patch is fixing the issue for me.

tools/testing/selftests/net/nettest.c +0 -0
index cd8a580974480212b45d86f35293b77f3d033473..ff25e53024ef6d4101f251c8a8a5e936e44e280f

100644

tools/testing/selftests/net/nettest.c +1 -0
@@ -1744,6 +1744,7 @@ static int connectsock(void *addr, socklen_t
 alen, struct sock_args *args)
        if (args->bind_test_only)
                goto out;
 
+       set_recv_attr(sd, args->version);
        if (connect(sd, addr, alen) < 0) {
                if (errno != EINPROGRESS) {
Quoted 1 line
                        log_err_errno("Failed to connect to remote host");

When tracing nettest we now have EHOSTUNREACH

3343  setsockopt(3, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0 <0.000210>
3343  setsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, "eth1\0", 5) = 0 <0.000170>
3343  setsockopt(3, SOL_IP, IP_PKTINFO, [1], 4) = 0 <0.000161>
3343  setsockopt(3, SOL_IP, IP_RECVERR, [1], 4) = 0 <0.000181>
3343  connect(3, {sa_family=AF_INET, sin_port=htons(12345),

sin_addr=inet_addr("172.16.2.1")}, 16) = -1 EINPROGRESS (Operation now in progress) <0.000874>
3343 pselect6(1024, NULL, [3], NULL, {tv_sec=5, tv_nsec=0}, NULL) = 1 (out [3], left {tv_sec=1, tv_nsec=930762080}) <3.069673>
3343 getsockopt(3, SOL_SOCKET, SO_ERROR, [EHOSTUNREACH], [4]) = 0 <0.000270>

Quoted 8 lines

As mentioned in net/ipv4/icmp.c :
RFC 1122: 3.2.2.1 States that NET_UNREACH, HOST_UNREACH and SR_FAILED
MUST be considered 'transient errs'.

Maybe another way to fix nettest would be to change wait_for_connect() to pass a non NULL fdset in 4th argument of select()

select(FD_SETSIZE, NULL, &wfd, NULL /* here */, tv);

This change in wait_for_connect() does not help.

I am guessing set_recv_attr(sd, args->version); is what we need.

I am running all ./fcnal-test.sh tests to make sure everything is green.

#13 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

In reply to Eric Dumazet
Quoted 19 lines · Eric Dumazet

Thanks Paolo

I found that the following patch is fixing the issue for me.

tools/testing/selftests/net/nettest.c +0 -0
index cd8a580974480212b45d86f35293b77f3d033473..ff25e53024ef6d4101f251c8a8a5e936e44e280f

100644

tools/testing/selftests/net/nettest.c +1 -0
@@ -1744,6 +1744,7 @@ static int connectsock(void *addr, socklen_t
 alen, struct sock_args *args)
        if (args->bind_test_only)
                goto out;
 
+       set_recv_attr(sd, args->version);
        if (connect(sd, addr, alen) < 0) {
                if (errno != EINPROGRESS) {
Quoted 1 line
                        log_err_errno("Failed to connect to remote host");

You have a kernel patch that makes a test fail, and your solution is changing userspace? The tests are examples of userspace applications and how they can use APIs, so if the patch breaks a test it is by definition breaking userspace which is not allowed.

#14 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 18 lines · David Ahern
Quoted 16 lines · Eric Dumazet

Thanks Paolo

I found that the following patch is fixing the issue for me.

tools/testing/selftests/net/nettest.c +0 -0
index cd8a580974480212b45d86f35293b77f3d033473..ff25e53024ef6d4101f251c8a8a5e936e44e280f

100644

tools/testing/selftests/net/nettest.c +0 -0
@@ -1744,6 +1744,7 @@ static int connectsock(void *addr, socklen_t
 alen, struct sock_args *args)
        if (args->bind_test_only)
                goto out;
 
Quoted 9 lines
Quoted 4 lines
tools/testing/selftests/net/nettest.c +1 -0
+       set_recv_attr(sd, args->version);
        if (connect(sd, addr, alen) < 0) {
                if (errno != EINPROGRESS) {
                        log_err_errno("Failed to connect to remote host");

You have a kernel patch that makes a test fail, and your solution is changing userspace? The tests are examples of userspace applications and how they can use APIs, so if the patch breaks a test it is by definition breaking userspace which is not allowed.

I think the userspace program relied on a bug added in linux in 2020

Jakub, I will stop trying to push the patches, this is a lost battle.

#15 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 1 line · Eric Dumazet

I think the userspace program relied on a bug added in linux in 2020

which test - when was it added? nettest.c and the fcnal script were merged in 2019.

#16 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 6 lines · David Ahern
Quoted 1 line · Eric Dumazet

I think the userspace program relied on a bug added in linux in 2020

which test - when was it added? nettest.c and the fcnal script were merged in 2019.

The test relies on connect() being aborted by EHOSTUNREACH

This is in complete violation with RFC

Even our own net.ipv4/icmp.c states this clearly.

RFC 1122: 3.2.2.1 States that NET_UNREACH, HOST_UNREACH and SR_FAILED
MUST be considered 'transient errs'.

Your program wants to use RECVERR instead of depending on a bug that we are trying very hard to solve.

maintainer
#17 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

In reply to Eric Dumazet
Quoted 4 lines · On Thu, 18 Apr 2024 19:47:51 +0200 Eric Dumazet wrote:
Quoted 4 lines

You have a kernel patch that makes a test fail, and your solution is changing userspace? The tests are examples of userspace applications and how they can use APIs, so if the patch breaks a test it is by definition breaking userspace which is not allowed.

Tests are often overly sensitive to kernel behavior, while this is obviously a red flag it's not an automatic nack. The more tests we have the more often we'll catch tiny changes. A lot of tests started flaking with 6.9 because of the optimizations in the timer subsystem. You know where I'm going with this..

Quoted 3 lines

I think the userspace program relied on a bug added in linux in 2020

Jakub, I will stop trying to push the patches, this is a lost battle.

If you have the patches ready - please post them.
I'm happy to take the blame if they actually regress something in the wild :(

We're pursuing this because real application suffer real problems when routing changes cause transient ICMP errors. Users read the RFC and then come shouting at us that Linux is buggy.

#18 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

Quoted 13 lines · Jakub Kicinski
Quoted 4 lines · On Thu, 18 Apr 2024 19:47:51 +0200 Eric Dumazet wrote:
Quoted 4 lines

You have a kernel patch that makes a test fail, and your solution is changing userspace? The tests are examples of userspace applications and how they can use APIs, so if the patch breaks a test it is by definition breaking userspace which is not allowed.

Tests are often overly sensitive to kernel behavior, while this is obviously a red flag it's not an automatic nack. The more tests we have the more often we'll catch tiny changes. A lot of tests started flaking with 6.9 because of the optimizations in the timer subsystem. You know where I'm going with this..

Quoted 7 lines
Quoted 3 lines

I think the userspace program relied on a bug added in linux in 2020

Jakub, I will stop trying to push the patches, this is a lost battle.

If you have the patches ready - please post them.
I'm happy to take the blame if they actually regress something in the wild :(

The series with the 2 patches has been posted already.

The remaining part is a nettest.c fix, that David is not happy with.

(quoted diff) +0 -0

b/tools/testing/selftests/net/nettest.c
index cd8a580974480212b45d86f35293b77f3d033473..23d56797900f19890923028af0b7ba162d9d5794 100644

tools/testing/selftests/net/nettest.c +0 -0
@@ -1744,6 +1744,8 @@ static int connectsock(void *addr, socklen_t

alen, struct sock_args *args)

        if (args->bind_test_only)
                goto out;

+ set_recv_attr(sd, args->version);
+

        if (connect(sd, addr, alen) < 0) {
                if (errno != EINPROGRESS) {
                        log_err_errno("Failed to connect to remote host");
Quoted 4 lines

We're pursuing this because real application suffer real problems when routing changes cause transient ICMP errors. Users read the RFC and then come shouting at us that Linux is buggy.

#19 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

In reply to Jakub Kicinski
Quoted 7 lines · Jakub Kicinski
Quoted 4 lines · On Thu, 18 Apr 2024 19:47:51 +0200 Eric Dumazet wrote:
Quoted 4 lines

You have a kernel patch that makes a test fail, and your solution is changing userspace? The tests are examples of userspace applications and how they can use APIs, so if the patch breaks a test it is by definition breaking userspace which is not allowed.

Tests are often overly sensitive to kernel behavior, while this is

That test script is a fairly comprehensive sweep of uAPIs and kernel behavior. Its sole job is to detect user visible changes and breakage from patches. It has done its job here. Do not shoot or criticize the messenger because you do not like the message.

Quoted 5 lines

obviously a red flag it's not an automatic nack. The more tests we have the more often we'll catch tiny changes. A lot of tests started flaking with 6.9 because of the optimizations in the timer subsystem. You know where I'm going with this..

Quoted 7 lines
Quoted 3 lines

I think the userspace program relied on a bug added in linux in 2020

Jakub, I will stop trying to push the patches, this is a lost battle.

If you have the patches ready - please post them.
I'm happy to take the blame if they actually regress something in the wild :(

And because of this test suite you are making a conscious decision to merge a patch that is making a user visible change. That is part of the tough decisions a maintainer has to make; I have no problem with that.

#20 lore raw

Re: [PATCH net-next 1/2] tcp: conditionally call ip_icmp_error() from tcp_v4_err()

In reply to Eric Dumazet
Quoted 16 lines · Eric Dumazet

For the second one, I am not familiar enough with this very slow test suite (all these "sleep 1" ... oh well)

I guess "failing tests" depended on TCP connect() to immediately abort on one ICMP message,
depending on old kernel behavior.

I do not know how to launch a subset of the tests, and trace these.

"./fcnal-test.sh -t ipv4_tcp" alone takes more than 9 minutes [1] in a VM running a non debug kernel :/

David, do you have an idea how to proceed ?

Thanks.

Quoted 7 lines

[1]
Tests passed: 134
Tests failed: 0

real 9m33.085s
user 0m40.159s
sys 0m30.098s

The test suite was started in 2013 and has evolved to cover the many permutations of APIs -- setsockopts, cmsg, VRF, routing, ip rules, netfilter, etc. There are a lot of combinations. They verify not just control path or socket bind succeeds, but data path works as expected which means do a connect and packet transfer.

Some years back nettest gained support to change namespaces and run both client and server in a single instance. I started converting the tests to use that capability and remove the sleeps, but it did not speed up the tests in any significant way.

The tests are in blocks and the blocks can be split out to separate scripts or kept in one to retain the common setup code and launched in parallel. Splitting out to any lower level does not make sense.

If someone wants to pick up the task of splitting the tests or running them in parallel, please do. I have zero time right now. That the suite detects changes in kernel behavior shows it is doing what it is designed to do and proves its value.

#21 lore raw

PATCH 2/2 tcp: no longer abort SYN_SENT when receiving some ICMP (II)

In reply to Eric Dumazet

Notes:

- A prior version of this patch in commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") had to be reverted.

- We found the root cause, and fixed it in prior patch in the series.

- Many thanks to Dragos Tatulea !

Currently, non fatal ICMP messages received on behalf of SYN_SENT sockets do call tcp_ld_RTO_revert() to implement RFC 6069, but immediately call tcp_done(), thus aborting the connect() attempt.

This violates RFC 1122 following requirement:

4.2.3.9 ICMP Messages
...
o Destination Unreachable -- codes 0, 1, 5

Since these Unreachable messages indicate soft error conditions, TCP MUST NOT abort the connection, and it SHOULD make the information available to the application.

This patch makes sure non 'fatal' ICMP[v6] messages do not abort the connection attempt.

It enables RFC 6069 for SYN_SENT sockets as a result.

---
 net/ipv4/tcp_ipv4.c | 6 ++++++
 net/ipv6/tcp_ipv6.c | 9 ++++++---
 2 files changed, 12 insertions(+), 3 deletions(-)
net/ipv4/tcp_ipv4.c +6 -0
index a717db99972d977a64178d7ed1109325d64a6d51..4b50d09f84b9ae7fec98a71bedf39594ab85e5ea 100644
@@ -482,6 +482,7 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
 	const int code = icmp_hdr(skb)->code;
 	struct sock *sk;
 	struct request_sock *fastopen;
+	bool harderr = false;
 	u32 seq, snd_una;
 	int err;
 	struct net *net = dev_net(skb->dev);
@@ -555,6 +556,7 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
 		goto out;
 	case ICMP_PARAMETERPROB:
 		err = EPROTO;
+		harderr = true;
 		break;
 	case ICMP_DEST_UNREACH:
 		if (code > NR_ICMP_UNREACH)
@@ -579,6 +581,7 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
 		}
 
 		err = icmp_err_convert[code].errno;
+		harderr = icmp_err_convert[code].fatal;
 		/* check if this ICMP message allows revert of backoff.
 		 * (see RFC 6069)
 		 */
@@ -605,6 +608,9 @@ int tcp_v4_err(struct sk_buff *skb, u32 info)
 		if (inet_test_bit(RECVERR, sk))
 			ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
 
+		if (!harderr)
+			break;
+
 		if (!sock_owned_by_user(sk)) {
 			WRITE_ONCE(sk->sk_err, err);
 
net/ipv6/tcp_ipv6.c +6 -3
index bb7c3caf4f8536dabdcb3dbe7c90aff9c8985c90..f31527f0a13dee9488ce72834d89524e83f61e5f 100644
@@ -382,7 +382,7 @@ static int tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,
 	struct tcp_sock *tp;
 	__u32 seq, snd_una;
 	struct sock *sk;
-	bool fatal;
+	bool harderr;
 	int err;
 
 	sk = __inet6_lookup_established(net, net->ipv4.tcp_death_row.hashinfo,
@@ -403,9 +403,9 @@ static int tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,
 		return 0;
 	}
 	seq = ntohl(th->seq);
-	fatal = icmpv6_err_convert(type, code, &err);
+	harderr = icmpv6_err_convert(type, code, &err);
 	if (sk->sk_state == TCP_NEW_SYN_RECV) {
-		tcp_req_err(sk, seq, fatal);
+		tcp_req_err(sk, seq, harderr);
 		return 0;
 	}
 
@@ -490,6 +490,9 @@ static int tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,
 
 		ipv6_icmp_error(sk, skb, err, th->dest, ntohl(info), (u8 *)th);
 
+		if (!harderr)
+			break;
+
 		if (!sock_owned_by_user(sk)) {
 			WRITE_ONCE(sk->sk_err, err);
 			sk_error_report(sk);		/* Wake people up to see the error (see connect in sock.c) */
Signature · 1 line
2.44.0.683.g7961c838ac-goog
Reviewed-by on v1 2/2
#22 lore raw

Re: [PATCH net-next 2/2] tcp: no longer abort SYN_SENT when receiving some ICMP (II)

Quoted 28 lines · Eric Dumazet

Notes:

- A prior version of this patch in commit 0a8de364ff7a ("tcp: no longer abort SYN_SENT when receiving some ICMP") had to be reverted.

- We found the root cause, and fixed it in prior patch in the series.

- Many thanks to Dragos Tatulea !

Currently, non fatal ICMP messages received on behalf of SYN_SENT sockets do call tcp_ld_RTO_revert() to implement RFC 6069, but immediately call tcp_done(), thus aborting the connect() attempt.

This violates RFC 1122 following requirement:

4.2.3.9 ICMP Messages
...
o Destination Unreachable -- codes 0, 1, 5

Since these Unreachable messages indicate soft error conditions, TCP MUST NOT abort the connection, and it SHOULD make the information available to the application.

Quoted 8 lines

This patch makes sure non 'fatal' ICMP[v6] messages do not abort the connection attempt.

It enables RFC 6069 for SYN_SENT sockets as a result.

Finally!!