Skip to content

selftests: kselftest_harness: support using xfail

Jakub Kicinski kvm linux-kselftest linux-security-module lkml netdev 4 versions – Applied

Status from this message: This series was applied to netdev/net-next.git (main)

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest12 patches 20 repliesApplied 12/12
  2. v311 patches 11 repliesSuperseded
  3. v24 patches 1 replySuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v14 patches12 repliesSuperseded

Changes in v4 from the cover letter

- rebase on top of Mickael's vfork() changes
v3: https://lore.kernel.org/all/[email protected]/
- combine multiple series
- change to "list of expected failures" rather than SKIP()-like handling
v2: https://lore.kernel.org/all/[email protected]/
- fix alignment
follow up RFC: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/

Patches in v4 All 12 applied

#Subject and filesSizeRepliesApplied
1/12selftests/landlock: Redefine TEST_F() as TEST_F_FORK()tools/testing/selftests/landlock/common.h+5 -10 repliesApplied
2/12selftests/harness: Merge TEST_F_FORK() into TEST_F()tools/testing/selftests/kselftest_harness.h tools/testing/selftests/landlock/common.h+27 -916 repliesApplied
3/12selftests: kselftest_harness: use KSFT_* exit codestools/testing/selftests/kselftest_harness.h+5 -60 repliesApplied
4/12selftests: kselftest_harness: generate test name oncetools/testing/selftests/kselftest_harness.h+10 -60 repliesApplied
5/12selftests: kselftest_harness: save full exit code in metadatatools/testing/selftests/kselftest_harness.h tools/testing/selftests/landlock/base_test.c tools/testing/selftests/landlock/fs_test.c +4 more+41 -350 repliesApplied
6/12selftests: kselftest_harness: use exit code to store skiptools/testing/selftests/kselftest_harness.h+5 -140 repliesApplied
7/12selftests: kselftest: add ksft_test_result_code(), handling all exit codestools/testing/selftests/kselftest.h tools/testing/selftests/kselftest_harness.h+46 -20 repliesApplied
8/12selftests: kselftest_harness: print test name for SKIPtools/testing/selftests/kselftest.h tools/testing/selftests/kselftest_harness.h+6 -40 repliesApplied
9/12selftests: kselftest_harness: separate diagnostic message with # in ksft_test_result_code()tools/testing/selftests/kselftest.h tools/testing/selftests/kselftest_harness.h+6 -10 repliesApplied
10/12selftests: kselftest_harness: let PASS / FAIL provide diagnostictools/testing/selftests/kselftest_harness.h+4 -51 replyApplied
11/12selftests: kselftest_harness: support using xfailtools/testing/selftests/kselftest_harness.h+48 -10 repliesApplied
12/12selftests: ip_local_port_range: use XFAIL instead of SKIPtools/testing/selftests/net/ip_local_port_range.c+3 -33 repliesApplied

Reviews trailers per patch

10 of 12 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchKees CookJakub SitnickiMark Brown
SeriesR-b T-b T-b
1/12
2/12R-b
3/12A-b T-b
4/12A-b
5/12
6/12R-b
7/12R-b
8/12R-b
9/12R-b
10/12R-b
11/12R-b
12/12R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v4 thread

Cover letter [PATCH v4 00/12] selftests: kselftest_harness: support using xfail

Hi!

When running selftests for our subsystem in our CI we'd like all tests to pass. Currently some tests use SKIP for cases they expect to fail, because the kselftest_harness limits the return codes to pass/fail/skip. XFAIL which would be a great match here cannot be used.

Remove the no_print handling and use vfork() to run the test in a different process than the setup. This way we don't need to pass "failing step" via the exit code. Further clean up the exit codes so that we can use all KSFT_* values. Rewrite the result printing to make handling XFAIL/XPASS easier. Support tests declaring combinations of fixture + variant they expect to fail.

Merge plan is to put it on top of -rc6 and merge into net-next. That way others should be able to pull the patches without any networking changes.

v4:
- rebase on top of Mickael's vfork() changes
v3: https://lore.kernel.org/all/[email protected]/
- combine multiple series
- change to "list of expected failures" rather than SKIP()-like handling
v2: https://lore.kernel.org/all/[email protected]/
- fix alignment
follow up RFC: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/

Jakub Kicinski (10):
selftests: kselftest_harness: use KSFT_* exit codes
selftests: kselftest_harness: generate test name once
selftests: kselftest_harness: save full exit code in metadata
selftests: kselftest_harness: use exit code to store skip
selftests: kselftest: add ksft_test_result_code(), handling all exit
codes
selftests: kselftest_harness: print test name for SKIP
selftests: kselftest_harness: separate diagnostic message with # in
ksft_test_result_code()
selftests: kselftest_harness: let PASS / FAIL provide diagnostic
selftests: kselftest_harness: support using xfail
selftests: ip_local_port_range: use XFAIL instead of SKIP

Mickaël Salaün (2):
selftests/landlock: Redefine TEST_F() as TEST_F_FORK() selftests/harness: Merge TEST_F_FORK() into TEST_F()

 tools/testing/selftests/kselftest.h           |  45 +++++
 tools/testing/selftests/kselftest_harness.h   | 182 +++++++++++-------
 tools/testing/selftests/landlock/base_test.c  |   2 +-
 tools/testing/selftests/landlock/common.h     |  58 +-----
 tools/testing/selftests/landlock/fs_test.c    |   4 +-
 tools/testing/selftests/landlock/net_test.c   |   4 +-
 .../testing/selftests/landlock/ptrace_test.c  |   7 +-
 .../selftests/net/ip_local_port_range.c       |   6 +-
 tools/testing/selftests/net/tls.c             |   2 +-
 tools/testing/selftests/seccomp/seccomp_bpf.c |   9 +-
 10 files changed, 178 insertions(+), 141 deletions(-)
Signature · 1 line
2.43.2