Skip to contentSkip to conversation

selftests/net: packetdrill: make tcp buf limited timing tests benign

PATCH Applied Jakub Kicinski 4 participants 5 messages Jan 15, 2025 – Jan 17, 2025

Subsystems (3)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

PATCH 1/1 selftests/net: packetdrill: make tcp buf limited timing tests benign

The following tests are failing on debug kernels:

tcp_tcp_info_tcp-info-rwnd-limited.pkt
tcp_tcp_info_tcp-info-sndbuf-limited.pkt

with reports like:

      assert 19000 <= tcpi_sndbuf_limited <= 21000, tcpi_sndbuf_limited; \

AssertionError: 18000

and:

      assert 348000 <= tcpi_busy_time <= 360000, tcpi_busy_time

AssertionError: 362000

Extend commit 912d6f669725 ("selftests/net: packetdrill: report benign debug flakes as xfail") to cover them.

---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]

---
 tools/testing/selftests/net/packetdrill/ksft_runner.sh | 1 +
 1 file changed, 1 insertion(+)
tools/testing/selftests/net/packetdrill/ksft_runner.sh +1 -0
index ff989c325eef..e15c43b7359b 100755
@@ -43,6 +43,7 @@ if [[ -n "${KSFT_MACHINE_SLOW}" ]]; then
 		"tcp_timestamping.*.pkt"
 		"tcp_user_timeout_user-timeout-probe.pkt"
 		"tcp_zerocopy_epoll_.*.pkt"
+		"tcp_tcp_info_tcp-info-*-limited.pkt"
 	)
 	readonly xfail_regex="^($(printf '%s|' "${xfail_list[@]}"))$"
 	[[ "$script" =~ ${xfail_regex} ]] && failfunc=ktap_test_xfail
Signature · 1 line
2.48.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 12 lines · Jakub Kicinski

The following tests are failing on debug kernels:

tcp_tcp_info_tcp-info-rwnd-limited.pkt
tcp_tcp_info_tcp-info-sndbuf-limited.pkt

with reports like:

      assert 19000 <= tcpi_sndbuf_limited <= 21000, tcpi_sndbuf_limited; \

AssertionError: 18000

and:

Quoted 7 lines
      assert 348000 <= tcpi_busy_time <= 360000, tcpi_busy_time

AssertionError: 362000

Extend commit 912d6f669725 ("selftests/net: packetdrill: report benign debug flakes as xfail") to cover them.

Thanks.

I see that we'll still have a few flakes on dbg. Perhaps one total failure a day. From the following.

tcp-close-close-local-close-then-remote-fin-pkt
tcp-ecn-ecn-uses-ect0-pkt
tcp-eor-no-coalesce-retrans-pkt
tcp-slow-start-slow-start-after-win-update-pkt
tcp-sack-sack-route-refresh-ip-tos-pkt
tcp-ts-recent-reset-tsval-pkt
tcp-zerocopy-closed-pkt

We'll take a look after this change whether we can make these more resilient. But likely also allow-list or even xfail for everything in dbg.

author maintainer
#2 lore raw
Quoted 25 lines · On Thu, 16 Jan 2025 08:05:57 -0500 Willem de Bruijn wrote:
Quoted 12 lines · Jakub Kicinski

The following tests are failing on debug kernels:

tcp_tcp_info_tcp-info-rwnd-limited.pkt
tcp_tcp_info_tcp-info-sndbuf-limited.pkt

with reports like:

      assert 19000 <= tcpi_sndbuf_limited <= 21000, tcpi_sndbuf_limited; \

AssertionError: 18000

and:

Quoted 7 lines
      assert 348000 <= tcpi_busy_time <= 360000, tcpi_busy_time

AssertionError: 362000

Extend commit 912d6f669725 ("selftests/net: packetdrill: report benign debug flakes as xfail") to cover them.

Thanks.

Quoted 7 lines

I see that we'll still have a few flakes on dbg. Perhaps one total failure a day. From the following.

tcp-close-close-local-close-then-remote-fin-pkt
tcp-ecn-ecn-uses-ect0-pkt
tcp-eor-no-coalesce-retrans-pkt
tcp-slow-start-slow-start-after-win-update-pkt

Argh, I missed the two above, I had the ignored cases filtered out when I was looking :(

Quoted 7 lines

tcp-sack-sack-route-refresh-ip-tos-pkt
tcp-ts-recent-reset-tsval-pkt
tcp-zerocopy-closed-pkt

We'll take a look after this change whether we can make these more resilient. But likely also allow-list or even xfail for everything in dbg.

Okay.

#4 lore raw
In reply to Jakub Kicinski
This patch was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 8 lines · On Wed, 15 Jan 2025 15:21:29 -0800 you wrote:

The following tests are failing on debug kernels:

tcp_tcp_info_tcp-info-rwnd-limited.pkt
tcp_tcp_info_tcp-info-sndbuf-limited.pkt

with reports like:

[...]

Here is the summary with links:
- [net-next] selftests/net: packetdrill: make tcp buf limited timing tests benign

    https://git.kernel.org/netdev/net-next/c/3030e3d57ba8

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
#3 lore raw
In reply to Willem de Bruijn

Hi Willem, Jakub,

Quoted 5 lines · Willem de Bruijn
Quoted 4 lines · Jakub Kicinski

The following tests are failing on debug kernels:

tcp_tcp_info_tcp-info-rwnd-limited.pkt
tcp_tcp_info_tcp-info-sndbuf-limited.pkt

(...)

Quoted 3 lines

We'll take a look after this change whether we can make these more resilient. But likely also allow-list or even xfail for everything in dbg.

On MPTCP side, I spent quite a bit of time trying to improve the situation on debug kernels. Sure it feels good and reassuring to have spent this time understanding the instabilities. Most issues were due to spurious retransmissions, because Packetdrill was "too slow" to inject
replies: so more like an issue in the tests. But I don't know if having these tests running in such slow environments helped to find bugs directly, e.g. catching unexpected packets. Maybe once? But at what cost?

Still it is good to run them on debug kernels to have extra verifications on the kernel side. As Ido mentioned last summer, perhaps we can ignore the test results, but keep logging them, and only look at the kernel warnings?

So yes, I agree with Willem: if that cannot easily be fixed, ignoring packetdrill err code for everything in debug sounds like the right direction.

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.