Skip to contentSkip to conversation

Improve stability of big_tcp_tunnels.sh selftest

PATCH Applied Alice Mikityanska 4 participants 9 messages Aug 22 – Aug 27

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover Improve stability of big_tcp_tunnels.sh selftest

From: Alice Mikityanska <[email protected]>

This series addresses flakiness of big_tcp_tunnels.sh discussed at:

https://lore.kernel.org/netdev/[email protected]/

Lower the thresholds to avoid failures like this (number of packets below the threshold):

https://netdev-ctrl.bots.linux.dev/logview.html?f=/logs/vmksft/net-dbg/results/754376/28-big-tcp-tunnels-sh/stdout

Tweak tcp_min_tso_segs and initcwnd to avoid failures like this (BIG TCP doesn't kick in in slow configurations):

https://netdev-ctrl.bots.linux.dev/logview.html?f=/logs/vmksft/net-dbg/results/751124/21-big-tcp-tunnels-sh/stdout

v2 changes: Use $KSFT_MACHINE_SLOW instead of probing kernel config.

v1: https://lore.kernel.org/netdev/[email protected]/

Alice Mikityanska (4):
selftests: net: Wait for netserver to launch
selftests: net: Lower threshold on debug kernels for
big_tcp_tunnels.sh
selftests: net: Lower threshold with csum offload off in
big_tcp_tunnels.sh
selftests: net: Fix slow configurations in big_tcp_tunnels.sh

 .../testing/selftests/net/big_tcp_tunnels.sh  | 38 +++++++++++++++++--
 1 file changed, 34 insertions(+), 4 deletions(-)
Signature · 1 line
2.55.0
#1 lore raw

PATCH v2 1/4 selftests: net: Wait for netserver to launch

From: Alice Mikityanska <[email protected]>

Use wait_local_port_listen after starting netserver in big_tcp_tunnels.sh to ensure it's listening when the test starts.

Fixes 5cb53743e1ff ("selftests: net: Add a test for BIG TCP in UDP tunnels")
---
 tools/testing/selftests/net/big_tcp_tunnels.sh | 3 +++
 1 file changed, 3 insertions(+)
tools/testing/selftests/net/big_tcp_tunnels.sh +3 -0
index d6513ed8d4e8..9b7d0456bcc1 100755
@@ -3,6 +3,8 @@
 #
 # Testing for IPv4 and IPv6 BIG TCP over VXLAN and GENEVE tunnels.
 
+source "$(dirname "$0")/lib.sh"
+
 SERVER_NS=$(mktemp -u server-XXXXXXXX)
 SERVER_IP4="192.168.1.1"
 SERVER_IP6="2001:db8::1:1"
@@ -39,6 +41,7 @@ setup() {
 		gro_max_size 196608 gro_ipv4_max_size 196608
 
 	ip netns exec "$SERVER_NS" netserver >/dev/null
+	wait_local_port_listen "$SERVER_NS" 12865 tcp
 }
 
 setup_tunnel() {
Signature · 1 line
2.55.0
#2 lore raw

PATCH v2 2/4 selftests: net: Lower threshold on debug kernels for big_tcp_tunnels.sh

In reply to Alice Mikityanska

From: Alice Mikityanska <[email protected]>

Debug kernels on upstream CI runners run slower and generate fewer BIG TCP packets, making the test flaky on upstream CI runners. Lower the default threshold for those kernels.

Fixes 5cb53743e1ff ("selftests: net: Add a test for BIG TCP in UDP tunnels")
---
 tools/testing/selftests/net/big_tcp_tunnels.sh | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
tools/testing/selftests/net/big_tcp_tunnels.sh +9 -2
index 9b7d0456bcc1..e0b41f394ebb 100755
@@ -17,11 +17,18 @@ CLIENT_IP6="2001:db8::1:2"
 CLIENT_IP4_TUN="192.168.2.2"
 CLIENT_IP6_TUN="2001:db8::2:2"
 
-: "${PACKETS_THRESHOLD:=1000}"
-
 # Kselftest framework requirement - SKIP code is 4.
 ksft_skip=4
 
+if [ -z "$PACKETS_THRESHOLD" ]; then
+	if [ "$KSFT_MACHINE_SLOW" = yes ]; then
+		echo 'Debug kernel detected, lowering the default threshold'
+		PACKETS_THRESHOLD=100
+	else
+		PACKETS_THRESHOLD=1000
+	fi
+fi
+
 setup() {
 	ip netns add "$SERVER_NS"
 	ip netns add "$CLIENT_NS"
Signature · 1 line
2.55.0
#3 lore raw

PATCH v2 3/4 selftests: net: Lower threshold with csum offload off in big_tcp_tunnels.sh

In reply to Alice Mikityanska

From: Alice Mikityanska <[email protected]>

With checksum offload disabled, much fewer BIG TCP packets are generated due to overall loss of throughput. Use a separate threshold in these tests, which is 1/10 of the threshold set for the rest of tests.

Fixes 5cb53743e1ff ("selftests: net: Add a test for BIG TCP in UDP tunnels")
---
 tools/testing/selftests/net/big_tcp_tunnels.sh | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
tools/testing/selftests/net/big_tcp_tunnels.sh +5 -2
index e0b41f394ebb..003878df7415 100755
@@ -107,6 +107,8 @@ cleanup() {
 }
 
 do_test() {
+	local packets_threshold="$PACKETS_THRESHOLD"
+
 	# When tx csum offload is off, software GSO is performed before passing the
 	# packet to veth. Check BIG TCP packets inside the VXLAN tunnel to verify
 	# the software checksum path: if the checksum code is broken, these packets
@@ -125,6 +127,7 @@ do_test() {
 		else
 			IPTABLES=ip6tables
 		fi
+		packets_threshold=$(( PACKETS_THRESHOLD / 10 ))
 	fi
 	if [ "$2" = 4 ]; then
 		IPTABLES_SACK=iptables
@@ -157,8 +160,8 @@ do_test() {
 	echo "Captured BIG TCP RX packets: $PACKETS_SERVER"
 	echo "Captured BIG TCP TX packets: $PACKETS_CLIENT"
 	echo "Captured TCP SACK packets: $PACKETS_SACK"
-	[ "$PACKETS_SERVER" -gt "$PACKETS_THRESHOLD" ] || return 1
-	[ "$PACKETS_CLIENT" -gt "$PACKETS_THRESHOLD" ] || return 1
+	[ "$PACKETS_SERVER" -gt "$packets_threshold" ] || return 1
+	[ "$PACKETS_CLIENT" -gt "$packets_threshold" ] || return 1
 	[ "$PACKETS_SACK" -lt "$(( PACKETS_CLIENT / 2 ))" ] || return 1
 }
 
Signature · 1 line
2.55.0
#4 lore raw

PATCH v2 4/4 selftests: net: Fix slow configurations in big_tcp_tunnels.sh

In reply to Alice Mikityanska

From: Alice Mikityanska <[email protected]>

The combination of checksum offload disabled (that causes software GSO) and a debug kernel is inherently slow. Depending on the CPU power and load, RTT may increase, limiting sk_pacing_rate, so tcp_tso_autosize caps SKBs at around 40 segments, and zero BIG TCP packets are produced.

Increase sysctl net.ipv4.tcp_min_tso_segs and set a bigger initial value of CWND in these configurations to force BIG TCP.

Fixes 5cb53743e1ff ("selftests: net: Add a test for BIG TCP in UDP tunnels")
---
 tools/testing/selftests/net/big_tcp_tunnels.sh | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
tools/testing/selftests/net/big_tcp_tunnels.sh +17 -0
index 003878df7415..cc0875e52fb9 100755
@@ -49,6 +49,8 @@ setup() {
 
 	ip netns exec "$SERVER_NS" netserver >/dev/null
 	wait_local_port_listen "$SERVER_NS" 12865 tcp
+
+	DEFAULT_TCP_MIN_TSO_SEGS=$(ip netns exec "$CLIENT_NS" sysctl -n net.ipv4.tcp_min_tso_segs)
 }
 
 setup_tunnel() {
@@ -135,6 +137,21 @@ do_test() {
 		IPTABLES_SACK=ip6tables
 	fi
 
+	if [ "$3" != 'on' ] && [ "$KSFT_MACHINE_SLOW" = yes ]; then
+		echo 'Slow configuration; increasing net.ipv4.tcp_min_tso_segs and initcwnd'
+		ip netns exec "$CLIENT_NS" sysctl -w net.ipv4.tcp_min_tso_segs=52
+		if [ "$2" = 4 ]; then
+			ip -netns "$CLIENT_NS" \
+			    route change 192.168.2.0/24 dev tun0 initcwnd 100
+		else
+			ip -netns "$CLIENT_NS" -6 \
+			    route change 2001:db8::2:0/112 dev tun0 initcwnd 100
+		fi
+	else
+		ip netns exec "$CLIENT_NS" \
+		    sysctl -w net.ipv4.tcp_min_tso_segs="$DEFAULT_TCP_MIN_TSO_SEGS"
+	fi
+
 	ip netns exec "$SERVER_NS" "$IPTABLES" -w -t raw -I PREROUTING -i "${CAPTURE_IFACE}1" -m length ! --length 0:65535 -m comment --comment "bigtcp"
 	ip netns exec "$CLIENT_NS" "$IPTABLES" -w -t raw -I OUTPUT -o "${CAPTURE_IFACE}0" -m length ! --length 0:65535 -m comment --comment "bigtcp"
 	ip netns exec "$SERVER_NS" "$IPTABLES_SACK" -w -t raw -I OUTPUT -o "tun1" -p tcp -m tcp --tcp-flags ACK ACK --tcp-option 5 -m comment --comment "sack"
Signature · 1 line
2.55.0
maintainer
#6 lore raw
In reply to Alice Mikityanska
Quoted 11 lines · Alice Mikityanska

From: Alice Mikityanska <[email protected]>

This series addresses flakiness of big_tcp_tunnels.sh discussed at:

https://lore.kernel.org/netdev/[email protected]/

Lower the thresholds to avoid failures like this (number of packets below the threshold):

https://netdev-ctrl.bots.linux.dev/logview.html?f=/logs/vmksft/net-dbg/results/754376/28-big-tcp-tunnels-sh/stdout

Quoted 8 lines

Tweak tcp_min_tso_segs and initcwnd to avoid failures like this (BIG TCP doesn't kick in in slow configurations):

https://netdev-ctrl.bots.linux.dev/logview.html?f=/logs/vmksft/net-dbg/results/751124/21-big-tcp-tunnels-sh/stdout

v2 changes: Use $KSFT_MACHINE_SLOW instead of probing kernel config.

v1: https://lore.kernel.org/netdev/[email protected]/

Any special reason for moving the target tree to 'net'? Selftests stability improvements should go via the net-next tree (without the fixes tag).

/P

#7 lore raw

Hi Paolo,

Quoted 15 lines · Paolo Abeni
Quoted 14 lines · Alice Mikityanska

From: Alice Mikityanska <[email protected]>

This series addresses flakiness of big_tcp_tunnels.sh discussed at:

https://lore.kernel.org/netdev/[email protected]/

Lower the thresholds to avoid failures like this (number of packets below the threshold):

https://netdev-ctrl.bots.linux.dev/logview.html?f=/logs/vmksft/net-dbg/results/754376/28-big-tcp-tunnels-sh/stdout

Tweak tcp_min_tso_segs and initcwnd to avoid failures like this (BIG TCP doesn't kick in in slow configurations):

Quoted 8 lines
Quoted 5 lines

https://netdev-ctrl.bots.linux.dev/logview.html?f=/logs/vmksft/net-dbg/results/751124/21-big-tcp-tunnels-sh/stdout

v2 changes: Use $KSFT_MACHINE_SLOW instead of probing kernel config.

v1: https://lore.kernel.org/netdev/[email protected]/

Any special reason for moving the target tree to 'net'? Selftests stability improvements should go via the net-next tree (without the fixes tag).

Sorry, I suggested to switch the target tree now that the selftest being fixed here has landed there. TBH, I think it is unclear what to do in this case: I agree that these modifications are not critical fixes, but they do help other people validating changes, plus they don't introduce regressions in the kernel code. If they go to net-next, that means other CIs (and developers) will potentially see false positives for ~2 months.

Personnaly, I think such fixes should go to net, but that's fine if the stable team don't backport them.

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
#8 lore raw
In reply to Alice Mikityanska
This series was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 10 lines · On Sat, 22 Aug 2026 15:03:04 +0300 you wrote:

From: Alice Mikityanska <[email protected]>

This series addresses flakiness of big_tcp_tunnels.sh discussed at:

https://lore.kernel.org/netdev/[email protected]/

Lower the thresholds to avoid failures like this (number of packets below the threshold):

[...]

Here is the summary with links:
- [net,v2,1/4] selftests: net: Wait for netserver to launch

    https://git.kernel.org/netdev/net/c/dc170da3347e
  - [net,v2,2/4] selftests: net: Lower threshold on debug kernels for big_tcp_tunnels.sh
    https://git.kernel.org/netdev/net/c/bb42c16f489f
  - [net,v2,3/4] selftests: net: Lower threshold with csum offload off in big_tcp_tunnels.sh
    https://git.kernel.org/netdev/net/c/f7d0400bd345
  - [net,v2,4/4] selftests: net: Fix slow configurations in big_tcp_tunnels.sh
    https://git.kernel.org/netdev/net/c/1d62b83fb751

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html