selftests/net: packetdrill: report benign debug flakes as xfail
Subsystems (3)
PATCH 1/1 selftests/net: packetdrill: report benign debug flakes as xfail
From: Willem de Bruijn <[email protected]>
A few recently added packetdrill tests that are known time sensitive (e.g., because testing timestamping) occasionally fail in debug mode:
https://netdev.bots.linux.dev/contest.html?executor=vmksft-packetdrill-dbg
These failures are well understood. Correctness of the tests is verified in non-debug mode. Continue running in debug mode also, to keep coverage with debug instrumentation.
But, only in debug mode, mark these tests with well understood timing issues as XFAIL (known failing) rather than FAIL when failing.
Introduce an allow list xfail_list with known cases.
Expand the ktap infrastructure with XFAIL support.
--- .../selftests/kselftest/ktap_helpers.sh | 15 ++++++++++-- .../selftests/net/packetdrill/ksft_runner.sh | 23 +++++++++++++++---- 2 files changed, 31 insertions(+), 7 deletions(-)
tools/testing/selftests/kselftest/ktap_helpers.sh +13 -2
@@ -7,6 +7,7 @@ KTAP_TESTNO=1 KTAP_CNT_PASS=0 KTAP_CNT_FAIL=0 +KTAP_CNT_XFAIL=0 KTAP_CNT_SKIP=0 KSFT_PASS=0
@@ -69,6 +70,16 @@ ktap_test_skip() { KTAP_CNT_SKIP=$((KTAP_CNT_SKIP+1)) } +ktap_test_xfail() { + description="$1" + + result="ok" + directive="XFAIL" + __ktap_test "$result" "$description" "$directive" + + KTAP_CNT_XFAIL=$((KTAP_CNT_XFAIL+1)) +} + ktap_test_fail() { description="$1"
@@ -99,7 +110,7 @@ ktap_exit_fail_msg() { ktap_finished() { ktap_print_totals - if [ $((KTAP_CNT_PASS + KTAP_CNT_SKIP)) -eq "$KSFT_NUM_TESTS" ]; then + if [ $((KTAP_CNT_PASS + KTAP_CNT_SKIP + KTAP_CNT_XFAIL)) -eq "$KSFT_NUM_TESTS" ]; then exit "$KSFT_PASS" else exit "$KSFT_FAIL"
@@ -107,5 +118,5 @@ ktap_finished() { } ktap_print_totals() { - echo "# Totals: pass:$KTAP_CNT_PASS fail:$KTAP_CNT_FAIL xfail:0 xpass:0 skip:$KTAP_CNT_SKIP error:0" + echo "# Totals: pass:$KTAP_CNT_PASS fail:$KTAP_CNT_FAIL xfail:$KTAP_CNT_XFAIL xpass:0 skip:$KTAP_CNT_SKIP error:0" }
tools/testing/selftests/net/packetdrill/ksft_runner.sh +18 -5
@@ -23,7 +23,7 @@ if [ $# -ne 1 ]; then ktap_exit_fail_msg "usage: $0 <script>" exit "$KSFT_FAIL" fi -script="$1" +script="$(basename $1)" if [ -z "$(which packetdrill)" ]; then ktap_skip_all "packetdrill not found in PATH"
@@ -31,16 +31,29 @@ if [ -z "$(which packetdrill)" ]; then fi declare -a optargs +failfunc=ktap_test_fail + if [[ -n "${KSFT_MACHINE_SLOW}" ]]; then optargs+=('--tolerance_usecs=14000') + + # xfail tests that are known flaky with dbg config, not fixable. + # still run them for coverage (and expect 100% pass without dbg). + declare -ar xfail_list=( + "tcp_fast_recovery_prr-ss.*.pkt" + "tcp_timestamping.*.pkt" + "tcp_user_timeout_user-timeout-probe.pkt" + "tcp_zerocopy_epoll_.*.pkt" + ) + readonly xfail_regex="^($(printf '%s|' "${xfail_list[@]}"))$" + [[ "$script" =~ ${xfail_regex} ]] && failfunc=ktap_test_xfail fi ktap_print_header ktap_set_plan 2 -unshare -n packetdrill ${ipv4_args[@]} ${optargs[@]} $(basename $script) > /dev/null \ - && ktap_test_pass "ipv4" || ktap_test_fail "ipv4" -unshare -n packetdrill ${ipv6_args[@]} ${optargs[@]} $(basename $script) > /dev/null \ - && ktap_test_pass "ipv6" || ktap_test_fail "ipv6" +unshare -n packetdrill ${ipv4_args[@]} ${optargs[@]} $script > /dev/null \ + && ktap_test_pass "ipv4" || $failfunc "ipv4" +unshare -n packetdrill ${ipv6_args[@]} ${optargs[@]} $script > /dev/null \ + && ktap_test_pass "ipv6" || $failfunc "ipv6" ktap_finished
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
This patch was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Hello:
This patch was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Quoted 11 lines · On Fri, 3 Jan 2025 06:31:14 -0500 you wrote:
From: Willem de Bruijn <[email protected]>
A few recently added packetdrill tests that are known time sensitive (e.g., because testing timestamping) occasionally fail in debug mode:
https://netdev.bots.linux.dev/contest.html?executor=vmksft-packetdrill-dbgThese failures are well understood. Correctness of the tests is verified in non-debug mode. Continue running in debug mode also, to keep coverage with debug instrumentation.
[...]
Here is the summary with links:
- [net-next] selftests/net: packetdrill: report benign debug flakes as xfail
https://git.kernel.org/netdev/net-next/c/912d6f669725
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html