Skip to content

selftests: report per-suite build results without failing the build

Sasha Levin linux-arm-kernel linux-doc linux-kselftest linux-riscv linuxppc-dev lkml sparclinux 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest7 patches 7 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/7selftests: keep the build status from wrapping to successtools/testing/selftests/Makefile+2 -22 replies
2/7selftests: optionally record per-suite build and install resultsDocumentation/dev-tools/kselftest.rst tools/testing/selftests/Makefile tools/testing/selftests/build_report.mk+87 -81 reply
3/7selftests/futex: report failures of the functional subdirectorytools/testing/selftests/futex/Makefile+21 -110 replies
4/7kselftest/arm64: keep building and installing after a subdirectory failstools/testing/selftests/arm64/Makefile+29 -143 replies
5/7selftests/riscv: keep building and installing after a subdirectory failstools/testing/selftests/riscv/Makefile+29 -100 replies
6/7selftests/powerpc: report the outcome of each subdirectorytools/testing/selftests/powerpc/Makefile tools/testing/selftests/powerpc/pmu/Makefile+30 -100 replies
7/7selftests/sparc64: report failures of the drivers subdirectorytools/testing/selftests/sparc64/Makefile+22 -110 replies

Reviews trailers per patch

2 of 7 patches have a Reviewed-by or Acked-by.

PatchCatalin MarinasMark Brown
1/7R-b
2/7
3/7
4/7A-b R-b
5/7
6/7
7/7

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 0/7] selftests: report per-suite build results without failing the build

Building the selftests succeeds as long as one target builds, so that a single broken suite does not cost CI systems the coverage of all the others. Commit cad6f32665cb ("selftests: Deescalate error reporting") restored that policy after commit 7e47389142b8 ("selftests: Preserve subtarget failures in all/install") had made any failure fatal. The other side of the policy is that make cannot say which suites failed, and CI systems fall back to guessing from the installed files, which misses suites that installed their scripts but failed to compile and flags suites that legitimately install nothing.

This series keeps the policy and adds an opt-in report instead:

- Patch 1 fixes the exit status of the top-level loops, which wraps to success once eight or more targets fail.

- Patch 2 adds KSFT_BUILD_REPORT. When it names an absolute file, the all and install rules append a "phase<TAB>target<TAB>status" record per target, without changing what is built or installed or the exit status of make.

- Patches 3-7 apply the same policy to the suites that build their subdirectories in shell loops (futex, arm64, riscv, powerpc and sparc64). A failing subdirectory no longer hides behind the status of the last one, nor drops the whole suite from the installation, and each subdirectory is recorded as <suite>/<subdir>. FORCE_TARGETS keeps failing fast, now also inside these suites.

Patches 3-7 depend on patch 2 for the recording helper, so the series is easiest to take as a whole; acks from the architecture and futex maintainers would be appreciated.

Tested with mock suites for every converted loop and with native and cross (arm64, riscv, powerpc) builds of the affected suites against the base tree. A kselftest-install of 20 suites gives the same installation and runlist as before, plus the report.

Sasha Levin (7):
selftests: keep the build status from wrapping to success
selftests: optionally record per-suite build and install results selftests/futex: report failures of the functional subdirectory kselftest/arm64: keep building and installing after a subdirectory
fails
selftests/riscv: keep building and installing after a subdirectory
fails
selftests/powerpc: report the outcome of each subdirectory selftests/sparc64: report failures of the drivers subdirectory

 Documentation/dev-tools/kselftest.rst        | 38 +++++++++++++++++
 tools/testing/selftests/Makefile             | 31 ++++++++++----
 tools/testing/selftests/arm64/Makefile       | 43 +++++++++++++-------
 tools/testing/selftests/build_report.mk      | 26 ++++++++++++
 tools/testing/selftests/futex/Makefile       | 32 ++++++++++-----
 tools/testing/selftests/powerpc/Makefile     | 20 ++++++---
 tools/testing/selftests/powerpc/pmu/Makefile | 20 ++++++---
 tools/testing/selftests/riscv/Makefile       | 39 +++++++++++++-----
 tools/testing/selftests/sparc64/Makefile     | 33 ++++++++++-----
 9 files changed, 218 insertions(+), 64 deletions(-)
 create mode 100644 tools/testing/selftests/build_report.mk

base-commit: 4eb2586bd40db27d77cc5b724ca9386ed0c71d53

Signature · 1 line
2.53.0