Improve stability of big_tcp_tunnels.sh selftest
Subsystems (3)
v2 cover Improve stability of big_tcp_tunnels.sh selftest
From: Alice Mikityanska <[email protected]>
This series addresses flakiness of big_tcp_tunnels.sh discussed at:
https://lore.kernel.org/netdev/[email protected]/
Lower the thresholds to avoid failures like this (number of packets below the threshold):
Tweak tcp_min_tso_segs and initcwnd to avoid failures like this (BIG TCP doesn't kick in in slow configurations):
v2 changes: Use $KSFT_MACHINE_SLOW instead of probing kernel config.
v1: https://lore.kernel.org/netdev/[email protected]/
Alice Mikityanska (4):
selftests: net: Wait for netserver to launch
selftests: net: Lower threshold on debug kernels for
big_tcp_tunnels.sh
selftests: net: Lower threshold with csum offload off in
big_tcp_tunnels.sh
selftests: net: Fix slow configurations in big_tcp_tunnels.sh
.../testing/selftests/net/big_tcp_tunnels.sh | 38 +++++++++++++++++-- 1 file changed, 34 insertions(+), 4 deletions(-)
Signature · 1 line
2.55.0
PATCH v2 1/4 selftests: net: Wait for netserver to launch
From: Alice Mikityanska <[email protected]>
Use wait_local_port_listen after starting netserver in big_tcp_tunnels.sh to ensure it's listening when the test starts.
--- tools/testing/selftests/net/big_tcp_tunnels.sh | 3 +++ 1 file changed, 3 insertions(+)
tools/testing/selftests/net/big_tcp_tunnels.sh +3 -0
@@ -3,6 +3,8 @@ # # Testing for IPv4 and IPv6 BIG TCP over VXLAN and GENEVE tunnels. +source "$(dirname "$0")/lib.sh" + SERVER_NS=$(mktemp -u server-XXXXXXXX) SERVER_IP4="192.168.1.1" SERVER_IP6="2001:db8::1:1"
@@ -39,6 +41,7 @@ setup() { gro_max_size 196608 gro_ipv4_max_size 196608 ip netns exec "$SERVER_NS" netserver >/dev/null + wait_local_port_listen "$SERVER_NS" 12865 tcp } setup_tunnel() {
Signature · 1 line
2.55.0
PATCH v2 2/4 selftests: net: Lower threshold on debug kernels for big_tcp_tunnels.sh
In reply to Alice MikityanskaFrom: Alice Mikityanska <[email protected]>
Debug kernels on upstream CI runners run slower and generate fewer BIG TCP packets, making the test flaky on upstream CI runners. Lower the default threshold for those kernels.
--- tools/testing/selftests/net/big_tcp_tunnels.sh | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
tools/testing/selftests/net/big_tcp_tunnels.sh +9 -2
@@ -17,11 +17,18 @@ CLIENT_IP6="2001:db8::1:2" CLIENT_IP4_TUN="192.168.2.2" CLIENT_IP6_TUN="2001:db8::2:2" -: "${PACKETS_THRESHOLD:=1000}" - # Kselftest framework requirement - SKIP code is 4. ksft_skip=4 +if [ -z "$PACKETS_THRESHOLD" ]; then + if [ "$KSFT_MACHINE_SLOW" = yes ]; then + echo 'Debug kernel detected, lowering the default threshold' + PACKETS_THRESHOLD=100 + else + PACKETS_THRESHOLD=1000 + fi +fi + setup() { ip netns add "$SERVER_NS" ip netns add "$CLIENT_NS"
Signature · 1 line
2.55.0
PATCH v2 3/4 selftests: net: Lower threshold with csum offload off in big_tcp_tunnels.sh
In reply to Alice MikityanskaFrom: Alice Mikityanska <[email protected]>
With checksum offload disabled, much fewer BIG TCP packets are generated due to overall loss of throughput. Use a separate threshold in these tests, which is 1/10 of the threshold set for the rest of tests.
--- tools/testing/selftests/net/big_tcp_tunnels.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
tools/testing/selftests/net/big_tcp_tunnels.sh +5 -2
@@ -107,6 +107,8 @@ cleanup() { } do_test() { + local packets_threshold="$PACKETS_THRESHOLD" + # When tx csum offload is off, software GSO is performed before passing the # packet to veth. Check BIG TCP packets inside the VXLAN tunnel to verify # the software checksum path: if the checksum code is broken, these packets
@@ -125,6 +127,7 @@ do_test() { else IPTABLES=ip6tables fi + packets_threshold=$(( PACKETS_THRESHOLD / 10 )) fi if [ "$2" = 4 ]; then IPTABLES_SACK=iptables
@@ -157,8 +160,8 @@ do_test() { echo "Captured BIG TCP RX packets: $PACKETS_SERVER" echo "Captured BIG TCP TX packets: $PACKETS_CLIENT" echo "Captured TCP SACK packets: $PACKETS_SACK" - [ "$PACKETS_SERVER" -gt "$PACKETS_THRESHOLD" ] || return 1 - [ "$PACKETS_CLIENT" -gt "$PACKETS_THRESHOLD" ] || return 1 + [ "$PACKETS_SERVER" -gt "$packets_threshold" ] || return 1 + [ "$PACKETS_CLIENT" -gt "$packets_threshold" ] || return 1 [ "$PACKETS_SACK" -lt "$(( PACKETS_CLIENT / 2 ))" ] || return 1 }
Signature · 1 line
2.55.0
PATCH v2 4/4 selftests: net: Fix slow configurations in big_tcp_tunnels.sh
In reply to Alice MikityanskaFrom: Alice Mikityanska <[email protected]>
The combination of checksum offload disabled (that causes software GSO) and a debug kernel is inherently slow. Depending on the CPU power and load, RTT may increase, limiting sk_pacing_rate, so tcp_tso_autosize caps SKBs at around 40 segments, and zero BIG TCP packets are produced.
Increase sysctl net.ipv4.tcp_min_tso_segs and set a bigger initial value of CWND in these configurations to force BIG TCP.
--- tools/testing/selftests/net/big_tcp_tunnels.sh | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
tools/testing/selftests/net/big_tcp_tunnels.sh +17 -0
@@ -49,6 +49,8 @@ setup() { ip netns exec "$SERVER_NS" netserver >/dev/null wait_local_port_listen "$SERVER_NS" 12865 tcp + + DEFAULT_TCP_MIN_TSO_SEGS=$(ip netns exec "$CLIENT_NS" sysctl -n net.ipv4.tcp_min_tso_segs) } setup_tunnel() {
@@ -135,6 +137,21 @@ do_test() { IPTABLES_SACK=ip6tables fi + if [ "$3" != 'on' ] && [ "$KSFT_MACHINE_SLOW" = yes ]; then + echo 'Slow configuration; increasing net.ipv4.tcp_min_tso_segs and initcwnd' + ip netns exec "$CLIENT_NS" sysctl -w net.ipv4.tcp_min_tso_segs=52 + if [ "$2" = 4 ]; then + ip -netns "$CLIENT_NS" \ + route change 192.168.2.0/24 dev tun0 initcwnd 100 + else + ip -netns "$CLIENT_NS" -6 \ + route change 2001:db8::2:0/112 dev tun0 initcwnd 100 + fi + else + ip netns exec "$CLIENT_NS" \ + sysctl -w net.ipv4.tcp_min_tso_segs="$DEFAULT_TCP_MIN_TSO_SEGS" + fi + ip netns exec "$SERVER_NS" "$IPTABLES" -w -t raw -I PREROUTING -i "${CAPTURE_IFACE}1" -m length ! --length 0:65535 -m comment --comment "bigtcp" ip netns exec "$CLIENT_NS" "$IPTABLES" -w -t raw -I OUTPUT -o "${CAPTURE_IFACE}0" -m length ! --length 0:65535 -m comment --comment "bigtcp" ip netns exec "$SERVER_NS" "$IPTABLES_SACK" -w -t raw -I OUTPUT -o "tun1" -p tcp -m tcp --tcp-flags ACK ACK --tcp-option 5 -m comment --comment "sack"
Signature · 1 line
2.55.0
Hi Alice,
Quoted 3 lines · Alice Mikityanska
From: Alice Mikityanska <[email protected]>
This series addresses flakiness of
big_tcp_tunnels.shdiscussed at:
Thank you for this v2. It looks good to me:
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Paolo Abeni
maintainerQuoted 11 lines · Alice Mikityanska
From: Alice Mikityanska <[email protected]>
This series addresses flakiness of
big_tcp_tunnels.shdiscussed at:https://lore.kernel.org/netdev/[email protected]/
Lower the thresholds to avoid failures like this (number of packets below the threshold):
Quoted 8 lines
Tweak
tcp_min_tso_segsand initcwnd to avoid failures like this (BIG TCP doesn't kick in in slow configurations):v2 changes: Use $KSFT_MACHINE_SLOW instead of probing kernel config.
Any special reason for moving the target tree to 'net'? Selftests stability improvements should go via the net-next tree (without the fixes tag).
/P
Hi Paolo,
Quoted 15 lines · Paolo Abeni
Quoted 14 lines · Alice Mikityanska
From: Alice Mikityanska <[email protected]>
This series addresses flakiness of
big_tcp_tunnels.shdiscussed at:https://lore.kernel.org/netdev/[email protected]/
Lower the thresholds to avoid failures like this (number of packets below the threshold):
Tweak
tcp_min_tso_segsand initcwnd to avoid failures like this (BIG TCP doesn't kick in in slow configurations):
Quoted 8 lines
Quoted 5 lines
v2 changes: Use $KSFT_MACHINE_SLOW instead of probing kernel config.
Any special reason for moving the target tree to 'net'? Selftests stability improvements should go via the net-next tree (without the fixes tag).
Sorry, I suggested to switch the target tree now that the selftest being fixed here has landed there. TBH, I think it is unclear what to do in this case: I agree that these modifications are not critical fixes, but they do help other people validating changes, plus they don't introduce regressions in the kernel code. If they go to net-next, that means other CIs (and developers) will potentially see false positives for ~2 months.
Personnaly, I think such fixes should go to net, but that's fine if the stable team don't backport them.
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
This series was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:
Quoted 10 lines · On Sat, 22 Aug 2026 15:03:04 +0300 you wrote:
From: Alice Mikityanska <[email protected]>
This series addresses flakiness of
big_tcp_tunnels.shdiscussed at:https://lore.kernel.org/netdev/[email protected]/
Lower the thresholds to avoid failures like this (number of packets below the threshold):
[...]
Here is the summary with links:
- [net,v2,1/4] selftests: net: Wait for netserver to launch
https://git.kernel.org/netdev/net/c/dc170da3347e - [net,v2,2/4] selftests: net: Lower threshold on debug kernels for big_tcp_tunnels.sh https://git.kernel.org/netdev/net/c/bb42c16f489f - [net,v2,3/4] selftests: net: Lower threshold with csum offload off in big_tcp_tunnels.sh https://git.kernel.org/netdev/net/c/f7d0400bd345 - [net,v2,4/4] selftests: net: Fix slow configurations in big_tcp_tunnels.sh https://git.kernel.org/netdev/net/c/1d62b83fb751
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html