Skip to contentSkip to conversation

selftests: net: packetdrill: xfail all problems on slow machines

PATCH Applied Jakub Kicinski 4 participants 6 messages Aug 1, 2025 – Aug 5, 2025

Subsystems (3)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

PATCH 1/1 selftests: net: packetdrill: xfail all problems on slow machines

We keep seeing flakes on packetdrill on debug kernels, while non-debug kernels are stable, not a single flake in 200 runs. Time to give up, debug kernels appear to suffer from 10msec latency spikes and any timing-sensitive test is bound to flake.

---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]

---
 .../selftests/net/packetdrill/ksft_runner.sh  | 19 +------------------
 1 file changed, 1 insertion(+), 18 deletions(-)
tools/testing/selftests/net/packetdrill/ksft_runner.sh +1 -18
index c5b01e1bd4c7..a7e790af38ff 100755
@@ -35,24 +35,7 @@ failfunc=ktap_test_fail
 
 if [[ -n "${KSFT_MACHINE_SLOW}" ]]; then
 	optargs+=('--tolerance_usecs=14000')
-
-	# xfail tests that are known flaky with dbg config, not fixable.
-	# still run them for coverage (and expect 100% pass without dbg).
-	declare -ar xfail_list=(
-		"tcp_blocking_blocking-connect.pkt"
-		"tcp_blocking_blocking-read.pkt"
-		"tcp_eor_no-coalesce-retrans.pkt"
-		"tcp_fast_recovery_prr-ss.*.pkt"
-		"tcp_sack_sack-route-refresh-ip-tos.pkt"
-		"tcp_slow_start_slow-start-after-win-update.pkt"
-		"tcp_timestamping.*.pkt"
-		"tcp_user_timeout_user-timeout-probe.pkt"
-		"tcp_zerocopy_cl.*.pkt"
-		"tcp_zerocopy_epoll_.*.pkt"
-		"tcp_tcp_info_tcp-info-.*-limited.pkt"
-	)
-	readonly xfail_regex="^($(printf '%s|' "${xfail_list[@]}"))$"
-	[[ "$script" =~ ${xfail_regex} ]] && failfunc=ktap_test_xfail
+	failfunc=ktap_test_xfail
 fi
 
 ktap_print_header
Signature · 1 line
2.50.1
author maintainer
#2 lore raw
Quoted 9 lines · On Fri, 01 Aug 2025 17:00:35 -0400 Willem de Bruijn wrote:
Quoted 6 lines · Jakub Kicinski

We keep seeing flakes on packetdrill on debug kernels, while non-debug kernels are stable, not a single flake in 200 runs. Time to give up, debug kernels appear to suffer from 10msec latency spikes and any timing-sensitive test is bound to flake.

I should have added "Willem was right" 'cause you suggested this a while back. But didn't know how to phrase it in the commit msg :)

#3 lore raw
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Fri, 01 Aug 2025 17:00:35 -0400 Willem de Bruijn wrote:
Quoted 5 lines · Jakub Kicinski

We keep seeing flakes on packetdrill on debug kernels, while non-debug kernels are stable, not a single flake in 200 runs. Time to give up, debug kernels appear to suffer from 10msec latency spikes and any timing-sensitive test is bound to flake.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

I should have added "Willem was right" 'cause you suggested this a while back. But didn't know how to phrase it in the commit msg :)

Ha, did I? I was hoping that the short allow-list would work. But if latency spikes can happen anytime, then that clearly not.

Acked-by on v1 1/1
#4 lore raw
In reply to Jakub Kicinski

Hi Jakub, Willem,

Quoted 4 lines · Jakub Kicinski

We keep seeing flakes on packetdrill on debug kernels, while non-debug kernels are stable, not a single flake in 200 runs. Time to give up, debug kernels appear to suffer from 10msec latency spikes and any timing-sensitive test is bound to flake.

Thank you for the patch!

Another solution might be to increase the tolerance, but I don't think it will fix all issues. I quickly looked at the last 100 runs, and I think most failures might be fixed by a higher tolerance, e.g.

Quoted 1 line

# tcp_ooo-before-and-after-accept.pkt:19: timing error: expected inbound packet at 0.101619 sec but happened at 0.115894 sec; tolerance 0.014000 sec

(0.275ms above the limit!)

On MPTCP, we used to have a very high tolerance with debug kernels (>0.5s) when public CIs were very limited in terms of CPU resources. I guess having a tolerance of 0.1s would be enough, but for these MPTCP packetdrill tests, I put 0.2s for the tolerance with a debug kernel, just to be on the safe side.

Still, I think increasing the tolerance would not fix all issues. On MPTCP side, the latency introduced by debug kernel caused unexpected retransmissions due to too low RTO. I took the time to make sure injected packets were always done with enough delay, but with the TCP packetdrill tests here, it is possibly not enough to do that when I look at some recent errors, e.g.

Quoted 1 line

tcp_zerocopy_batch.pkt:26: error handling packet: live packet payload: expected 4000 bytes vs actual 5000 bytes

At the end, and as previously mentioned, these adaptations for debug kernel are perhaps not worth it: in this environment, it is probably enough to ignore packetdrill results and focus on kernel warnings.

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
#5 lore raw
In reply to Jakub Kicinski
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 8 lines · On Fri, 1 Aug 2025 11:16:38 -0700 you wrote:

We keep seeing flakes on packetdrill on debug kernels, while non-debug kernels are stable, not a single flake in 200 runs. Time to give up, debug kernels appear to suffer from 10msec latency spikes and any timing-sensitive test is bound to flake.

[...]

Here is the summary with links:
- [net] selftests: net: packetdrill: xfail all problems on slow machines

    https://git.kernel.org/netdev/net/c/5ef7fdf52c0f

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html