selftests: report per-suite build results without failing the build
Subsystems (7)
cover selftests: report per-suite build results without failing the build
Building the selftests succeeds as long as one target builds, so that a single broken suite does not cost CI systems the coverage of all the others. Commit cad6f32665cb ("selftests: Deescalate error reporting") restored that policy after commit 7e47389142b8 ("selftests: Preserve subtarget failures in all/install") had made any failure fatal. The other side of the policy is that make cannot say which suites failed, and CI systems fall back to guessing from the installed files, which misses suites that installed their scripts but failed to compile and flags suites that legitimately install nothing.
This series keeps the policy and adds an opt-in report instead:
- Patch 1 fixes the exit status of the top-level loops, which wraps to success once eight or more targets fail.
- Patch 2 adds KSFT_BUILD_REPORT. When it names an absolute file, the all and install rules append a "phase<TAB>target<TAB>status" record per target, without changing what is built or installed or the exit status of make.
- Patches 3-7 apply the same policy to the suites that build their subdirectories in shell loops (futex, arm64, riscv, powerpc and sparc64). A failing subdirectory no longer hides behind the status of the last one, nor drops the whole suite from the installation, and each subdirectory is recorded as <suite>/<subdir>. FORCE_TARGETS keeps failing fast, now also inside these suites.
Patches 3-7 depend on patch 2 for the recording helper, so the series is easiest to take as a whole; acks from the architecture and futex maintainers would be appreciated.
Tested with mock suites for every converted loop and with native and cross (arm64, riscv, powerpc) builds of the affected suites against the base tree. A kselftest-install of 20 suites gives the same installation and runlist as before, plus the report.
Sasha Levin (7):
selftests: keep the build status from wrapping to success
selftests: optionally record per-suite build and install results selftests/futex: report failures of the functional subdirectory kselftest/arm64: keep building and installing after a subdirectory
fails
selftests/riscv: keep building and installing after a subdirectory
fails
selftests/powerpc: report the outcome of each subdirectory selftests/sparc64: report failures of the drivers subdirectory
Documentation/dev-tools/kselftest.rst | 38 +++++++++++++++++ tools/testing/selftests/Makefile | 31 ++++++++++---- tools/testing/selftests/arm64/Makefile | 43 +++++++++++++------- tools/testing/selftests/build_report.mk | 26 ++++++++++++ tools/testing/selftests/futex/Makefile | 32 ++++++++++----- tools/testing/selftests/powerpc/Makefile | 20 ++++++--- tools/testing/selftests/powerpc/pmu/Makefile | 20 ++++++--- tools/testing/selftests/riscv/Makefile | 39 +++++++++++++----- tools/testing/selftests/sparc64/Makefile | 33 ++++++++++----- 9 files changed, 218 insertions(+), 64 deletions(-) create mode 100644 tools/testing/selftests/build_report.mk
base-commit: 4eb2586bd40db27d77cc5b724ca9386ed0c71d53
Signature · 1 line
2.53.0
PATCH 1/7 selftests: keep the build status from wrapping to success
The top-level all and install rules return failure only when every selected target fails, which commit cad6f32665cb ("selftests: Deescalate error reporting") deliberately kept as the default. They implement this by multiplying the exit statuses of the targets:
ret=$$((ret * $$?))
A failing recursive make exits with status 2, so eight failing targets produce 256, and "exit 256" reports success. Building eight or more targets that all fail therefore succeeds, and install goes on to package an empty installation.
Record a successful target instead of multiplying statuses. The rules still fail only when no target succeeds, and FORCE_TARGETS still stops at the first failure.
--- tools/testing/selftests/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/Makefile +2 -2
@@ -236,7 +236,7 @@ all: $(MAKE) OUTPUT=$$BUILD_TARGET -C $$TARGET \ O=$(abs_objtree) \ $(if $(FORCE_TARGETS),|| exit); \ - ret=$$((ret * $$?)); \ + [ $$? -eq 0 ] && ret=0; \ done; exit $$ret; run_tests: all
@@ -303,7 +303,7 @@ ifdef INSTALL_PATH OBJ_PATH=$(INSTALL_PATH) \ O=$(abs_objtree) \ $(if $(FORCE_TARGETS),|| exit); \ - ret=$$((ret * $$?)); \ + [ $$? -eq 0 ] && ret=0; \ done; exit $$ret;
Signature · 1 line
2.53.0
PATCH 2/7 selftests: optionally record per-suite build and install results
In reply to Sasha LevinBy default the selftests build and installation succeed as long as one target succeeds, so that a single broken suite does not cost CI systems the coverage of all others; commit cad6f32665cb ("selftests: Deescalate error reporting") restored that policy after it had been tightened. The exit status of make therefore cannot tell which suites failed, and inspecting the installation is not reliable either: a suite whose compilation failed can still install its scripts.
Add an opt-in report. When KSFT_BUILD_REPORT names an absolute file, the all and install rules append one tab-separated record per target with the phase, the target name and the exit status of its recursive make. Generating the runlist is recorded as part of the install phase. The records are written without changing which targets are built or installed or the exit status of make, also with FORCE_TARGETS. Nothing is recorded by make -n, -t or -q, including when the selftests are built through the kernel's kselftest targets, which clear MAKEFLAGS.
The recording helper lives in a small fragment so that suites built from subdirectories can add records for them, and the variable is exported to sub-makes. A relative path is rejected, since every recursive make runs in a different directory. Keep the exit status of the runlist generation loop as before, and create an empty runlist for installations in which no target emits tests.
Document the format, version 1, and how to read it.
--- Documentation/dev-tools/kselftest.rst | 38 +++++++++++++++++++++++++ tools/testing/selftests/Makefile | 31 ++++++++++++++------ tools/testing/selftests/build_report.mk | 26 +++++++++++++++++ 3 files changed, 87 insertions(+), 8 deletions(-) create mode 100644 tools/testing/selftests/build_report.mk
Documentation/dev-tools/kselftest.rst +38 -0
@@ -133,6 +133,44 @@ build successfully; make will abort as soon as any target fails:: This applies to both the ``all`` and ``install`` targets and is useful in CI environments where a silent partial build would be misleading. +Recording suite build results +============================= + +Since the build succeeds as long as one target builds, the exit status of +make does not show which targets failed. Set ``KSFT_BUILD_REPORT`` to the +absolute path of a file to have the ``all`` and ``install`` targets append +a tab-separated record for each target they process:: + + $ : > /tmp/ksft-build.tsv + $ make -C tools/testing/selftests O=/tmp/kernel-build \ + KSFT_BUILD_REPORT=/tmp/ksft-build.tsv install + +Each record holds the phase (``all`` or ``install``), a name and the exit +status of that step. Recording does not change which targets are built or +installed, or the exit status of make, also with ``FORCE_TARGETS=1``. The +caller creates or truncates the report before a build; the rules only +append to it. Nothing is recorded by ``make -n``, ``-t`` or ``-q``. + +The same name can appear several times: ``install`` builds the targets +again, and generating ``kselftest-list.txt`` is recorded as part of the +``install`` phase. Targets that are added automatically as installation +dependencies of other targets, such as ``net/lib``, are recorded too. +Suites that are built from several subdirectories may also record each +subdirectory under its path, for example ``arm64/tags``; a failure there +means that the suite was built or installed only partially. A record +belongs to the target with the longest name that is equal to the record +name or followed by ``/`` in it. + +A target was built and installed successfully only if all of its records +have a zero status. Missing records do not show that a target was built. +A successful target that installs no tests has no entries in +``kselftest-list.txt``. Install the kernel headers with ``make headers`` +before building: a suite that fails because the headers are missing is +recorded as failed even if a later step installs them. + +``KSFT_BUILD_REPORT_VERSION`` in tools/testing/selftests/Makefile gives the +version of this format, currently ``1``. + Running the full range hotplug selftests ========================================
tools/testing/selftests/Makefile +23 -8
@@ -180,6 +180,10 @@ endif # of the targets gets built. FORCE_TARGETS ?= +# Remember dry runs for the build report before MAKEFLAGS may be cleared. +KSFT_BUILD_REPORT_DRY_RUN := $(strip $(foreach f,n t q,$(findstring $(f),$(firstword -$(MAKEFLAGS))))) +export KSFT_BUILD_REPORT_DRY_RUN + # Clear LDFLAGS and MAKEFLAGS when implicit rules are missing. This provides # implicit rules to sub-test Makefiles which avoids build failures in test # Makefile that don't have explicit build rules.
@@ -224,6 +228,10 @@ ARCH ?= $(SUBARCH) export BUILD export KHDR_INCLUDES +# Version of the optional KSFT_BUILD_REPORT format written by these rules. +KSFT_BUILD_REPORT_VERSION := 1 +include $(top_srcdir)/tools/testing/selftests/build_report.mk + # set default goal to all, so make without a target runs all, even when # all isn't the first target in the file. .DEFAULT_GOAL := all
@@ -232,11 +240,13 @@ all: @ret=1; \ for TARGET in $(TARGETS) $(INSTALL_DEP_TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ - mkdir $$BUILD_TARGET -p; \ + mkdir -p $$BUILD_TARGET && \ $(MAKE) OUTPUT=$$BUILD_TARGET -C $$TARGET \ - O=$(abs_objtree) \ - $(if $(FORCE_TARGETS),|| exit); \ - [ $$? -eq 0 ] && ret=0; \ + O=$(abs_objtree); \ + status=$$?; \ + $(call ksft_record_build_status,all,$$TARGET); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ done; exit $$ret; run_tests: all
@@ -293,7 +303,7 @@ ifdef INSTALL_PATH install -m 744 kselftest/ktap_helpers.sh $(INSTALL_PATH)/kselftest/ install -m 744 kselftest/ksft.py $(INSTALL_PATH)/kselftest/ install -m 744 run_kselftest.sh $(INSTALL_PATH)/ - rm -f $(TEST_LIST) + : > $(TEST_LIST) @ret=1; \ for TARGET in $(TARGETS) $(INSTALL_DEP_TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \
@@ -301,9 +311,11 @@ ifdef INSTALL_PATH INSTALL_PATH=$(INSTALL_PATH)/$$TARGET \ SRC_PATH=$(shell readlink -e $$(pwd)) \ OBJ_PATH=$(INSTALL_PATH) \ - O=$(abs_objtree) \ - $(if $(FORCE_TARGETS),|| exit); \ - [ $$? -eq 0 ] && ret=0; \ + O=$(abs_objtree); \ + status=$$?; \ + $(call ksft_record_build_status,install,$$TARGET); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ done; exit $$ret;
@@ -317,6 +329,9 @@ ifdef INSTALL_PATH printf "Emit Tests for $$TARGET\n"; \ $(MAKE) -s --no-print-directory OUTPUT=$$BUILD_TARGET COLLECTION=$$TARGET \ -C $$TARGET emit_tests >> $(TEST_LIST); \ + status=$$?; \ + $(call ksft_record_build_status,install,$$TARGET); \ + (exit $$status); \ done; @VERSION=$$(git describe HEAD 2>/dev/null); \ if [ -n "$$VERSION" ]; then \
tools/testing/selftests/build_report.mk new file +26 -0
@@ -0,0 +1,26 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Optional per-suite build report, see "Recording suite build results" in +# Documentation/dev-tools/kselftest.rst. Callers set KSFT_BUILD_REPORT to an +# absolute filename and use ksft_record_build_status from a shell loop that +# has stored the exit status of the recorded step in $status. + +ifneq (x$(KSFT_BUILD_REPORT)x,xx) +ifeq ($(filter x/%,$(firstword x$(KSFT_BUILD_REPORT))),) +$(error KSFT_BUILD_REPORT must be an absolute path: '$(KSFT_BUILD_REPORT)') +endif +endif +export KSFT_BUILD_REPORT + +ksft_build_report_dry_run := $(KSFT_BUILD_REPORT_DRY_RUN) \ + $(foreach f,n t q,$(findstring $(f),$(firstword -$(MAKEFLAGS)))) + +ifneq ($(strip $(ksft_build_report_dry_run)),) +ksft_record_build_status = : +else +define ksft_record_build_status +if [ -n "$$KSFT_BUILD_REPORT" ]; then \ + printf '%s\t%s\t%d\n' '$(1)' "$(2)" "$$status" >> "$$KSFT_BUILD_REPORT" || exit 1; \ +fi +endef +endif
Signature · 1 line
2.53.0
PATCH 3/7 selftests/futex: report failures of the functional subdirectory
In reply to Sasha LevinThe futex suite builds its tests in the functional subdirectory and then copies run.sh into the output directory. The copy is the last command of the loop, so its status replaces that of the build, and "make all" succeeds even when the functional tests fail to compile.
Fold the copy into the status of the subdirectory, count a failure to create the output directory as well, and let the build and installation loops return failure when no subdirectory succeeds, which matches how the top-level selftests Makefile treats its targets, or as soon as one fails when FORCE_TARGETS is set. When a build report is requested, record each subdirectory as futex/<subdir> so that a failure is visible even when make itself succeeds.
--- tools/testing/selftests/futex/Makefile | 32 +++++++++++++++++--------- 1 file changed, 21 insertions(+), 11 deletions(-)
tools/testing/selftests/futex/Makefile +21 -11
@@ -4,26 +4,36 @@ SUBDIRS := functional TEST_PROGS := run.sh include ../lib.mk +include $(selfdir)/build_report.mk all: - @for DIR in $(SUBDIRS); do \ - BUILD_TARGET=$(OUTPUT)/$$DIR; \ - mkdir $$BUILD_TARGET -p; \ - $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;\ - if [ -e $$DIR/$(TEST_PROGS) ]; then \ - rsync -a $$DIR/$(TEST_PROGS) $$BUILD_TARGET/; \ - fi \ - done + @ret=$(if $(SUBDIRS),1,0); for DIR in $(SUBDIRS); do \ + BUILD_TARGET=$(OUTPUT)/$$DIR; \ + mkdir -p $$BUILD_TARGET && \ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ + status=$$?; \ + if [ -e $$DIR/$(TEST_PROGS) ]; then \ + rsync -a $$DIR/$(TEST_PROGS) $$BUILD_TARGET/ || \ + [ $$status -ne 0 ] || status=1; \ + fi; \ + $(call ksft_record_build_status,all,futex/$$DIR); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret override define INSTALL_RULE mkdir -p $(INSTALL_PATH) install -t $(INSTALL_PATH) $(TEST_PROGS) $(TEST_PROGS_EXTENDED) $(TEST_FILES) - @for SUBDIR in $(SUBDIRS); do \ + @ret=$(if $(SUBDIRS),1,0); for SUBDIR in $(SUBDIRS); do \ BUILD_TARGET=$(OUTPUT)/$$SUBDIR; \ - mkdir $$BUILD_TARGET -p; \ + mkdir -p $$BUILD_TARGET && \ $(MAKE) OUTPUT=$$BUILD_TARGET -C $$SUBDIR INSTALL_PATH=$(INSTALL_PATH)/$$SUBDIR install; \ - done; + status=$$?; \ + $(call ksft_record_build_status,install,futex/$$SUBDIR); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret endef override define CLEAN
Signature · 1 line
2.53.0
PATCH 4/7 kselftest/arm64: keep building and installing after a subdirectory fails
In reply to Sasha LevinThe arm64 suite builds and installs its subdirectories in shell loops whose status is that of the last subdirectory. A failure in an earlier subdirectory is therefore lost unless FORCE_TARGETS is set, while a failure in the last one fails "make all" and, through "install: all", drops the installation of every other arm64 test as well.
Make each loop fail only when no subdirectory succeeds, which matches how the top-level selftests Makefile treats its targets, and keep stopping at the first failure when FORCE_TARGETS is set. When a build report is requested, record each subdirectory as arm64/<subdir> so that a partially built suite is visible even though make succeeds.
Invoke the subdirectories with $(MAKE) rather than make. The command line variables already reach them through MAKEFLAGS either way, but $(MAKE) lets them use the parent's jobserver and makes -n, -t and -q apply to them.
--- tools/testing/selftests/arm64/Makefile | 43 +++++++++++++++++--------- 1 file changed, 29 insertions(+), 14 deletions(-)
tools/testing/selftests/arm64/Makefile +29 -14
@@ -26,38 +26,53 @@ OUTPUT ?= $(CURDIR) export CFLAGS export top_srcdir +include $(top_srcdir)/tools/testing/selftests/build_report.mk + all: - @for DIR in $(ARM64_SUBTARGETS); do \ + @ret=$(if $(ARM64_SUBTARGETS),1,0); \ + for DIR in $(ARM64_SUBTARGETS); do \ BUILD_TARGET=$(OUTPUT)/$$DIR; \ - mkdir -p $$BUILD_TARGET; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@ \ - $(if $(FORCE_TARGETS),|| exit); \ - done + mkdir -p $$BUILD_TARGET && \ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ + status=$$?; \ + $(call ksft_record_build_status,all,arm64/$$DIR); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret install: all - @for DIR in $(ARM64_SUBTARGETS); do \ + @ret=$(if $(ARM64_SUBTARGETS),1,0); \ + for DIR in $(ARM64_SUBTARGETS); do \ BUILD_TARGET=$(OUTPUT)/$$DIR; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@ \ - $(if $(FORCE_TARGETS),|| exit); \ - done + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ + status=$$?; \ + $(call ksft_record_build_status,install,arm64/$$DIR); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret run_tests: all @for DIR in $(ARM64_SUBTARGETS); do \ BUILD_TARGET=$(OUTPUT)/$$DIR; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ done # Avoid any output on non arm64 on emit_tests emit_tests: - @for DIR in $(ARM64_SUBTARGETS); do \ + @ret=$(if $(ARM64_SUBTARGETS),1,0); \ + for DIR in $(ARM64_SUBTARGETS); do \ BUILD_TARGET=$(OUTPUT)/$$DIR; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ - done + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ + status=$$?; \ + $(call ksft_record_build_status,install,arm64/$$DIR); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret clean: @for DIR in $(ARM64_SUBTARGETS); do \ BUILD_TARGET=$(OUTPUT)/$$DIR; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ done .PHONY: all clean install run_tests emit_tests
Signature · 1 line
2.53.0
PATCH 5/7 selftests/riscv: keep building and installing after a subdirectory fails
In reply to Sasha LevinThe RISC-V suite builds and installs its subdirectories in shell loops whose status is that of the last subdirectory. A failure in any other subdirectory is lost, while a failure in the last one fails "make all" and, through "install: all", drops the installation of every other RISC-V test as well. FORCE_TARGETS has no effect on the loops either.
Make each loop fail only when no subdirectory succeeds, which matches how the top-level selftests Makefile treats its targets, and stop at the first failure when FORCE_TARGETS is set. When a build report is requested, record each subdirectory as riscv/<subdir> so that a partially built suite is visible even though make succeeds. Default OUTPUT to the current directory, as the arm64 suite does, so that the loops also work when the suite is built directly.
--- tools/testing/selftests/riscv/Makefile | 39 +++++++++++++++++++------- 1 file changed, 29 insertions(+), 10 deletions(-)
tools/testing/selftests/riscv/Makefile +29 -10
@@ -20,21 +20,35 @@ CFLAGS += -I$(top_srcdir)/tools/testing/selftests/ CFLAGS += $(KHDR_INCLUDES) +OUTPUT ?= $(CURDIR) + export CFLAGS export top_srcdir +include $(top_srcdir)/tools/testing/selftests/build_report.mk + all: - @for DIR in $(RISCV_SUBTARGETS); do \ + @ret=$(if $(RISCV_SUBTARGETS),1,0); \ + for DIR in $(RISCV_SUBTARGETS); do \ BUILD_TARGET=$(OUTPUT)/$$DIR; \ - mkdir -p $$BUILD_TARGET; \ - $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ - done + mkdir -p $$BUILD_TARGET && \ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ + status=$$?; \ + $(call ksft_record_build_status,all,riscv/$$DIR); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret install: all - @for DIR in $(RISCV_SUBTARGETS); do \ + @ret=$(if $(RISCV_SUBTARGETS),1,0); \ + for DIR in $(RISCV_SUBTARGETS); do \ BUILD_TARGET=$(OUTPUT)/$$DIR; \ - $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ - done + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ + status=$$?; \ + $(call ksft_record_build_status,install,riscv/$$DIR); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret run_tests: all @for DIR in $(RISCV_SUBTARGETS); do \
@@ -44,10 +58,15 @@ run_tests: all # Avoid any output on non riscv on emit_tests emit_tests: - @for DIR in $(RISCV_SUBTARGETS); do \ + @ret=$(if $(RISCV_SUBTARGETS),1,0); \ + for DIR in $(RISCV_SUBTARGETS); do \ BUILD_TARGET=$(OUTPUT)/$$DIR; \ - $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ - done + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ + status=$$?; \ + $(call ksft_record_build_status,install,riscv/$$DIR); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret clean: @for DIR in $(RISCV_SUBTARGETS); do \
Signature · 1 line
2.53.0
PATCH 6/7 selftests/powerpc: report the outcome of each subdirectory
In reply to Sasha LevinThe powerpc suite and its pmu subdirectory install and list their tests in shell loops whose status is that of the last subdirectory, so a failure in any other subdirectory is lost and FORCE_TARGETS has no effect on them.
Make the installation and runlist loops fail only when no subdirectory succeeds, which matches how the top-level selftests Makefile treats its targets, and stop at the first failure when FORCE_TARGETS is set. When a build report is requested, record each subdirectory as powerpc/<subdir> or powerpc/pmu/<subdir>, including the builds of the subdirectories, so that a failure is visible even when make itself succeeds.
The subdirectories are still built as prerequisites of "all", so a subdirectory that fails to build still fails the build of the suite as before; only the reporting of that failure is added.
--- tools/testing/selftests/powerpc/Makefile | 20 +++++++++++++++----- tools/testing/selftests/powerpc/pmu/Makefile | 20 +++++++++++++++----- 2 files changed, 30 insertions(+), 10 deletions(-)
tools/testing/selftests/powerpc/Makefile +15 -5
@@ -37,9 +37,11 @@ endif all: $(SUB_DIRS) $(SUB_DIRS): - BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET; $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all + BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET && $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all; \ + status=$$?; $(call ksft_record_build_status,all,powerpc/$@); exit $$status include ../lib.mk +include $(selfdir)/build_report.mk include ./flags.mk override define RUN_TESTS
@@ -50,17 +52,25 @@ override define RUN_TESTS endef override define INSTALL_RULE - +@for TARGET in $(SUB_DIRS); do \ + +@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \ BUILD_TARGET=$(OUTPUT)/$$TARGET; \ $(MAKE) OUTPUT=$$BUILD_TARGET INSTALL_PATH=$$INSTALL_PATH/$$TARGET -C $$TARGET install;\ - done; + status=$$?; \ + $(call ksft_record_build_status,install,powerpc/$$TARGET); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret endef emit_tests: - +@for TARGET in $(SUB_DIRS); do \ + +@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \ BUILD_TARGET=$(OUTPUT)/$$TARGET; \ $(MAKE) OUTPUT=$$BUILD_TARGET COLLECTION=$(COLLECTION)/$$TARGET -s -C $$TARGET $@;\ - done; + status=$$?; \ + $(call ksft_record_build_status,install,powerpc/$$TARGET); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret override define CLEAN +@for TARGET in $(SUB_DIRS); do \
tools/testing/selftests/powerpc/pmu/Makefile +15 -5
@@ -7,6 +7,7 @@ EXTRA_SOURCES := ../harness.c event.c lib.c ../utils.c top_srcdir = ../../../../.. include ../../lib.mk +include $(selfdir)/build_report.mk include ../flags.mk SUB_DIRS := ebb sampling_tests event_code_tests
@@ -26,7 +27,8 @@ $(OUTPUT)/count_stcx_fail: loop.S $(EXTRA_SOURCES) $(OUTPUT)/per_event_excludes: ../utils.c $(SUB_DIRS): - BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET; $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all + BUILD_TARGET=$(OUTPUT)/$@; mkdir -p $$BUILD_TARGET && $(MAKE) OUTPUT=$$BUILD_TARGET -k -C $@ all; \ + status=$$?; $(call ksft_record_build_status,all,powerpc/pmu/$@); exit $$status DEFAULT_RUN_TESTS := $(RUN_TESTS) override define RUN_TESTS
@@ -42,18 +44,26 @@ emit_tests: BASENAME_TEST=`basename $$TEST`; \ echo "$(COLLECTION):$$BASENAME_TEST"; \ done - +@for TARGET in $(SUB_DIRS); do \ + +@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \ BUILD_TARGET=$(OUTPUT)/$$TARGET; \ $(MAKE) OUTPUT=$$BUILD_TARGET COLLECTION=$(COLLECTION)/$$TARGET -s -C $$TARGET emit_tests; \ - done; + status=$$?; \ + $(call ksft_record_build_status,install,powerpc/pmu/$$TARGET); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret DEFAULT_INSTALL_RULE := $(INSTALL_RULE) override define INSTALL_RULE $(DEFAULT_INSTALL_RULE) - +@for TARGET in $(SUB_DIRS); do \ + +@ret=$(if $(SUB_DIRS),1,0); for TARGET in $(SUB_DIRS); do \ BUILD_TARGET=$(OUTPUT)/$$TARGET; \ $(MAKE) OUTPUT=$$BUILD_TARGET INSTALL_PATH=$$INSTALL_PATH/$$TARGET -C $$TARGET install; \ - done; + status=$$?; \ + $(call ksft_record_build_status,install,powerpc/pmu/$$TARGET); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret endef DEFAULT_CLEAN := $(CLEAN)
Signature · 1 line
2.53.0
PATCH 7/7 selftests/sparc64: report failures of the drivers subdirectory
In reply to Sasha LevinThe sparc64 suite builds and installs its subdirectories in shell loops whose status is that of the last command. For the build that is the copy of the subdirectory's test script, so "make all" succeeds even when the drivers tests fail to compile, and FORCE_TARGETS has no effect.
Fold the copy into the status of the subdirectory and let both loops return failure when no subdirectory succeeds, which matches how the top-level selftests Makefile treats its targets, or as soon as one fails when FORCE_TARGETS is set. When a build report is requested, record each subdirectory as sparc64/<subdir>. Invoke the subdirectories with $(MAKE) so that they use the parent's jobserver.
--- tools/testing/selftests/sparc64/Makefile | 33 ++++++++++++++++-------- 1 file changed, 22 insertions(+), 11 deletions(-)
tools/testing/selftests/sparc64/Makefile +22 -11
@@ -2,6 +2,8 @@ ARCH ?= $(shell uname -m 2>/dev/null || echo not) override ARCH := $(shell echo $(ARCH) | sed -e s/x86_64/x86/) +include ../build_report.mk + ifneq ($(ARCH),sparc64) nothing: .PHONY: all clean run_tests install
@@ -18,33 +20,42 @@ TEST_PROGS := run.sh include ../lib.mk all: - @for DIR in $(SUBDIRS); do \ - BUILD_TARGET=$(OUTPUT)/$$DIR; \ - mkdir $$BUILD_TARGET -p; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@;\ + @ret=$(if $(SUBDIRS),1,0); for DIR in $(SUBDIRS); do \ + BUILD_TARGET=$(OUTPUT)/$$DIR; \ + mkdir -p $$BUILD_TARGET && \ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ + status=$$?; \ #SUBDIR test prog name should be in the form: SUBDIR_test.sh \ TEST=$$DIR"_test.sh"; \ if [ -e $$DIR/$$TEST ]; then \ - rsync -a $$DIR/$$TEST $$BUILD_TARGET/; \ - fi \ - done + rsync -a $$DIR/$$TEST $$BUILD_TARGET/ || \ + [ $$status -ne 0 ] || status=1; \ + fi; \ + $(call ksft_record_build_status,all,sparc64/$$DIR); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret override define INSTALL_RULE mkdir -p $(INSTALL_PATH) install -t $(INSTALL_PATH) $(TEST_PROGS) $(TEST_PROGS_EXTENDED) $(TEST_FILES) - @for SUBDIR in $(SUBDIRS); do \ + @ret=$(if $(SUBDIRS),1,0); for SUBDIR in $(SUBDIRS); do \ BUILD_TARGET=$(OUTPUT)/$$SUBDIR; \ - mkdir $$BUILD_TARGET -p; \ + mkdir -p $$BUILD_TARGET && \ $(MAKE) OUTPUT=$$BUILD_TARGET -C $$SUBDIR INSTALL_PATH=$(INSTALL_PATH)/$$SUBDIR install; \ - done; + status=$$?; \ + $(call ksft_record_build_status,install,sparc64/$$SUBDIR); \ + $(if $(FORCE_TARGETS),[ $$status -eq 0 ] || exit $$status;) \ + [ $$status -eq 0 ] && ret=0; \ + done; exit $$ret endef override define CLEAN @for DIR in $(SUBDIRS); do \ BUILD_TARGET=$(OUTPUT)/$$DIR; \ mkdir $$BUILD_TARGET -p; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@;\ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;\ done endef endif
Signature · 1 line
2.53.0
Re: [PATCH 4/7] kselftest/arm64: keep building and installing after a subdirectory fails
In reply to Sasha LevinQuoted 12 lines · Sasha Levin
The arm64 suite builds and installs its subdirectories in shell loops whose status is that of the last subdirectory. A failure in an earlier subdirectory is therefore lost unless
FORCE_TARGETSis set, while a failure in the last one fails "make all" and, through "install: all", drops the installation of every other arm64 test as well.Make each loop fail only when no subdirectory succeeds, which matches how the top-level selftests
Makefiletreats its targets, and keep stopping at the first failure whenFORCE_TARGETSis set. When a build report is requested, record each subdirectory as arm64/<subdir> so that a partially built suite is visible even though make succeeds.
Quoted 7 lines
Invoke the subdirectories with $(MAKE) rather than make. The command line variables already reach them through MAKEFLAGS either way, but $(MAKE) lets them use the parent's jobserver and makes -n, -t and -q apply to them.
Fixes 313a4db7f338 ("kselftest: arm64: extend toplevel skeleton Makefile")Signed-off-by Sasha Levin <[email protected]>
Sashiko had a comment about breakage ARM64_SUBTARGETS is empty. I think it's a false positive but I didn't check.
Re: [PATCH 1/7] selftests: keep the build status from wrapping to success
In reply to Sasha LevinQuoted 3 lines · Sasha Levin
Record a successful target instead of multiplying statuses. The rules still fail only when no target succeeds, and
FORCE_TARGETSstill stops at the first failure.
Re: [PATCH 4/7] kselftest/arm64: keep building and installing after a subdirectory fails
In reply to Sasha LevinQuoted 11 lines · Sasha Levin
The arm64 suite builds and installs its subdirectories in shell loops whose status is that of the last subdirectory. A failure in an earlier subdirectory is therefore lost unless
FORCE_TARGETSis set, while a failure in the last one fails "make all" and, through "install: all", drops the installation of every other arm64 test as well.Make each loop fail only when no subdirectory succeeds, which matches how the top-level selftests
Makefiletreats its targets, and keep stopping at the first failure whenFORCE_TARGETSis set. When a build report is requested, record each subdirectory as arm64/<subdir> so that a partially built suite is visible even though make succeeds.
I can't help but think it might be less annoying to convert these into separate TARGETS but that disrupts everyone's results which isn't ideal.
Shuah Khan
maintainerRe: [PATCH 1/7] selftests: keep the build status from wrapping to success
In reply to Mark BrownQuoted 7 lines · Mark Brown
Quoted 3 lines · Sasha Levin
Record a successful target instead of multiplying statuses. The rules still fail only when no target succeeds, and
FORCE_TARGETSstill stops at the first failure.Reviewed-by Mark Brown <[email protected]>
Thank you both. Applied to kselftest next for 7.4-rc1
thanks,
-- Shuah
Shuah Khan
maintainerRe: [PATCH 2/7] selftests: optionally record per-suite build and install results
In reply to Sasha LevinQuoted 26 lines · Sasha Levin
By default the selftests build and installation succeed as long as one target succeeds, so that a single broken suite does not cost CI systems the coverage of all others; commit
cad6f32665cb("selftests: Deescalate error reporting") restored that policy after it had been tightened. The exit status of make therefore cannot tell which suites failed, and inspecting the installation is not reliable either: a suite whose compilation failed can still install its scripts.Add an opt-in report. When
KSFT_BUILD_REPORTnames an absolute file, the all and install rules append one tab-separated record per target with the phase, the target name and the exit status of its recursive make. Generating the runlist is recorded as part of the install phase. The records are written without changing which targets are built or installed or the exit status of make, also withFORCE_TARGETS. Nothing is recorded by make -n, -t or -q, including when the selftests are built through the kernel's kselftest targets, which clear MAKEFLAGS.The recording helper lives in a small fragment so that suites built from subdirectories can add records for them, and the variable is exported to sub-makes. A relative path is rejected, since every recursive make runs in a different directory. Keep the exit status of the runlist generation loop as before, and create an empty runlist for installations in which no target emits tests.
Document the format, version 1, and how to read it.
Quoted 9 lines
Link https://lore.kernel.org/r/20260416-selftests-deescalate-error-reporting-v1-1-38e7c0536227@kernel.org open hereSigned-off-by Sasha Levin <[email protected]>--- Documentation/dev-tools/kselftest.rst | 38 +++++++++++++++++++++++++ tools/testing/selftests/Makefile | 31 ++++++++++++++------ tools/testing/selftests/build_report.mk | 26 +++++++++++++++++ 3 files changed, 87 insertions(+), 8 deletions(-)create mode 100644
tools/testing/selftests/build_report.mk
I picked this up for the next merge window. Patch is in kselftest next.
thanks,
-- Shuah
Re: [PATCH 4/7] kselftest/arm64: keep building and installing after a subdirectory fails
In reply to Catalin MarinasQuoted 2 lines · Catalin Marinas
Sashiko had a comment about breakage
ARM64_SUBTARGETSis empty. I think it's a false positive but I didn't check.
It is a false positive. With ARM64_SUBTARGETS empty the loop expands to "for DIR in ; do ... done", which POSIX allows: the word list after "in" may be empty, so the body never runs. The $(if) sets ret to 0 for that case, so the target succeeds.
I ran the patched Makefile with ARCH=x86_64 for all, install and emit_tests under bash, dash and busybox sh, with and without FORCE_TARGETS=1. Every combination exits 0 with no output.
Signature · 2 lines
Thanks, Sasha
Hi Shuah,
Thanks for taking 1/7 and 2/7. 3/7 through 7/7 are still pending and still apply on 7.4-rc1. Catalin acked and Mark reviewed the arm64 one.
Without them a failed subdirectory in futex, arm64, riscv, powerpc or sparc64 still gets lost, and for arm64 and riscv a failure in the last subdirectory drops the whole suite from the install. Could you pick them up?
Signature · 2 lines
Thanks, Sasha