Skip to contentSkip to conversation

make: Add make test{, -c, -c-run, -shell-run} targets

RFC PATCH New Petr Vorel 3 participants 6 messages Jun 18, 2021 – Jun 22, 2021 Kbuild

lore
Share & tools
lore

Petr Vorel

author
#0 lore raw

RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets

For testing C and shell API.

---
Hi all,

another attempt for make test target.

In a long term, I'd like to wrap the execution with some script, as I suppose there will be some metadata in test, allowing to run tests which don't TPASS or TBROK (we have quite a lot of them). Then the summary would be even more useful.

FIXME: broken in lib/, it runs all test twice.

changes v1->v2:
Not sure if it's a good approach to process each test separately, there is no loop as Cyril suggested in [1], thus there is no summary what happen.

I previously had in similar loop in include/mk/generic_leaf_target.inc:

test-c-run: $(TEST_TARGETS)

	@set -e; echo; echo "===== Test C API ====="; \
	for i in $(TEST_TARGETS); do \
	    echo; echo "* $$i"; \
	    echo "PATH $(top_srcdir)/testcases/lib:$$PATH"; \
	    PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$$i || [ $$? -eq 32 ]; \
	done; echo; \
	echo "All C API tests passed or TCONF"

test-shell-run:

	@set -e; echo; echo "===== Test shell API ====="; \
	for i in $(TEST_SHELL_TARGETS); do \
	    echo; echo "* $$i"; \
	    PATH="$(abs_top_srcdir)/testcases/lib:$(abs_top_builddir)/testcases/lib:$$PATH" $(abs_srcdir)/$$i || [ $$? -eq 32 ]; \
	done; echo; \
	echo "All shell API tests passed or TCONF"

but that was problematic in lib (which uses generic_leaf_target.inc).

Kind regards,
Petr

[1] https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/#2695533

 Makefile                            | 23 ++++++++++++++++++++++-
 include/mk/generic_leaf_target.inc  | 25 ++++++++++++++++++++++---
 include/mk/generic_trunk_target.inc |  2 +-
 lib/newlib_tests/Makefile           | 13 +++++++++++++
 4 files changed, 58 insertions(+), 5 deletions(-)
Makefile +22 -1
index 56812d77b..40cd4b1cb 100644
@@ -75,10 +75,14 @@ endif
 INSTALL_TARGETS		+= $(COMMON_TARGETS)
 CLEAN_TARGETS		+= $(COMMON_TARGETS) lib libs
 BOOTSTRAP_TARGETS	:= $(sort $(COMMON_TARGETS) $(CLEAN_TARGETS) $(INSTALL_TARGETS))
+TEST_TARGETS		:= lib
+TEST_SHELL_TARGETS	:= lib
 
 CLEAN_TARGETS		:= $(addsuffix -clean,$(CLEAN_TARGETS))
 INSTALL_TARGETS		:= $(addsuffix -install,$(INSTALL_TARGETS))
 MAKE_TARGETS		:= $(addsuffix -all,$(filter-out lib,$(COMMON_TARGETS)))
+TEST_TARGETS		:= $(addsuffix -test-c-run,$(TEST_TARGETS))
+TEST_SHELL_TARGETS	:= $(addsuffix -test-shell-run,$(TEST_SHELL_TARGETS))
 
 # There's no reason why we should run `all' twice. Otherwise we're just wasting
 # 3+ mins of useful CPU cycles on a modern machine, and even more time on an
@@ -189,8 +193,25 @@ INSTALL_TARGETS		+= $(addprefix $(DESTDIR)/$(bindir)/,$(BINDIR_INSTALL_SCRIPTS))
 
 $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir)
 
+## Test library API
+.PHONY: test test-c test-c-run test-shell-run
+
+$(TEST_TARGETS): lib-all
+	$(MAKE) -C "$(subst -test-c-run,,$@)" \
+		-f "$(abs_top_srcdir)/$(subst -test-c-run,,$@)/Makefile" test-c-run
+
+$(TEST_SHELL_TARGETS): lib-all
+	$(MAKE) -C "$(subst -test-shell-run,,$@)" \
+		-f "$(abs_top_srcdir)/$(subst -test-shell-run,,$@)/Makefile" test-shell-run
+
+test-c-run: $(TEST_TARGETS)
+
+test-shell-run: $(TEST_SHELL_TARGETS)
+
+test: test-c test-c-run test-shell-run
+
 ## Install
-install: $(INSTALL_TARGETS)
+install: lib-all $(INSTALL_TARGETS)
 
 ## Help
 .PHONY: help
include/mk/generic_leaf_target.inc +22 -3
index 64953f89a..19e238f92 100644
@@ -63,8 +63,12 @@
 #				  real files or directories). This will automatically append
 #				  adds the .o suffix to all files referenced by
 #				  $(MAKE_TARGETS)) to CLEAN_TARGETS, if MAKE_TARGETS wasn't
-#				  defined (see
-#				  $(MAKE_TARGETS)).
+#				  defined (see $(MAKE_TARGETS)).
+#
+# $(TEST_TARGETS)		: LTP C API tests to be runned with test-c-run.
+#
+# $(TEST_SHELL_TARGETS)		: LTP shell API tests to be runned with test-shell-run.
+#
 # $(INSTALL_MODE)		: What mode should we using when calling
 # 				  install(1)?
 #
@@ -92,7 +96,7 @@
 # INSTALL_DIR			:= $(libdir)
 #
 
-.PHONY: all clean install
+.PHONY: all clean install test test-c test-c-run test-shell-run
 
 ifneq ($(strip $(MAKE_TARGETS)),)
 $(MAKE_TARGETS) += $(HOST_MAKE_TARGETS)
@@ -102,11 +106,26 @@ $(MAKE_TARGETS): | $(MAKE_DEPS)
 
 all: $(MAKE_TARGETS)
 
+TEST_TARGETS_RUN := $(addsuffix -run,$(TEST_TARGETS))
+TEST_SHELL_TARGETS_RUN := $(addsuffix -run,$(TEST_SHELL_TARGETS))
+
+$(TEST_TARGETS_RUN): | $(TEST_TARGETS)
+test-c-run: $(TEST_TARGETS_RUN)
+test-shell-run: $(TEST_SHELL_TARGETS_RUN)
+
+.PHONY: $(TEST_TARGETS_RUN) $(TEST_SHELL_TARGETS_RUN)
+$(TEST_TARGETS_RUN) $(TEST_SHELL_TARGETS_RUN):
+	@set -e; echo; echo "* $(subst -run,,$@)"; \
+	PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$(subst -run,,$@) || [ $$? -eq 32 ]; \
+
+test: test-c-run test-shell-run
+
 clean:: $(CLEAN_DEPS)
 	-$(RM) -f -r $(CLEAN_TARGETS)
 
 $(INSTALL_FILES): | $(INSTALL_DEPS)
 
 install: $(INSTALL_FILES)
+test-c: $(TEST_TARGETS)
 
 # vim: syntax=make
include/mk/generic_trunk_target.inc +1 -1
index fc59f944f..4427a0f86 100644
@@ -48,7 +48,7 @@
 
 include $(top_srcdir)/include/mk/functions.mk
 
-RECURSIVE_TARGETS		?= all install
+RECURSIVE_TARGETS		?= all install test test-c test-c-run test-shell-run
 
 $(eval $(get_make_dirs))
 
lib/newlib_tests/Makefile +13 -0
index 30ca6810c..7ac28660d 100644
@@ -19,5 +19,18 @@ ifeq ($(ANDROID),1)
 FILTER_OUT_MAKE_TARGETS	+= test08
 endif
 
+# C API tests
+# NOTE: only those which exit 0 (TPASS) or 32 (TCONF)
+# TODO: commented out for quicker demonstration
+# TEST_TARGETS += test05 test07 test09 test12 test15 test16 test18
+# TEST_TARGETS += test_exec test_timer tst_bool_expr tst_res_hexd
+# TEST_TARGETS += tst_strstatus tst_fuzzy_sync01 tst_fuzzy_sync02
+TEST_TARGETS += tst_strstatus test05
+
+# shell API tests
+# NOTE: only those which exit 0 (TPASS) or 32 (TCONF)
+TEST_SHELL_TARGETS += shell/tst_check_driver.sh
+# TODO: commented out for quicker demonstration
+# TEST_SHELL_TARGETS += shell/net/*.sh
 
 include $(top_srcdir)/include/mk/generic_leaf_target.mk
Signature · 1 line
2.32.0
#1 lore raw

RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets

Hello,

Quoted 6 lines · Petr Vorel

For testing C and shell API.

---
Hi all,

Quoted 6 lines

another attempt for make test target.

In a long term, I'd like to wrap the execution with some script, as I suppose there will be some metadata in test, allowing to run tests which don't TPASS or TBROK (we have quite a lot of them). Then the summary would be even more useful.

For each test you could have an tcl/expect (or equivalent Perl/shell if we don't already require tcl) script which matches the output including return value and stderr/stdout.

For most tests it would just call a common function to check for TPASS/TBROK. For tests where we want to look for a given message, it could match the output. If we sometimes expect TCONF then it could perform a check to verify that it really should return TCONF.

I guess you could just put some data in a comment. I think that is likely to be harder though. At least with the number of tests we current have.

Signature · 2 lines
Thank you,
Richard.
author
#2 lore raw

RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets

Hi Richie,

Quoted 12 lines

Hello,

Quoted 7 lines · Petr Vorel

For testing C and shell API.

---
Hi all,

another attempt for make test target.

Quoted 8 lines
Quoted 4 lines

In a long term, I'd like to wrap the execution with some script, as I suppose there will be some metadata in test, allowing to run tests which don't TPASS or TBROK (we have quite a lot of them). Then the summary would be even more useful.

For each test you could have an tcl/expect (or equivalent Perl/shell if we don't already require tcl) script which matches the output including return value and stderr/stdout.

I'd prefer to avoid tcl/expect. FYI There was some effort [1], based on my previous work [1], I plan to get back to it.

Quoted 4 lines

For most tests it would just call a common function to check for TPASS/TBROK. For tests where we want to look for a given message, it could match the output. If we sometimes expect TCONF then it could perform a check to verify that it really should return TCONF.

As I noted in [2] today, I might create very simple library for these test wrappers (small subset of things needed in test wrappers like lib/newlib_tests/shell/test_timeout.sh).

Quoted 3 lines

I guess you could just put some data in a comment. I think that is likely to be harder though. At least with the number of tests we current have.

Yep, that's what was used in [1].

Kind regards,
Petr

[1] https://patchwork.ozlabs.org/project/ltp/patch/ce675759672af52bea02c11d51bd7d10f0bcb5cb.1566500817.git.clanig@suse.com/
[2] https://lists.linux.it/pipermail/ltp/2021-June/023306.html

#3 lore raw

RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets

In reply to Petr Vorel

Hi!

Quoted 32 lines

For testing C and shell API.

---
Hi all,

another attempt for make test target.

In a long term, I'd like to wrap the execution with some script, as I suppose there will be some metadata in test, allowing to run tests which don't TPASS or TBROK (we have quite a lot of them). Then the summary would be even more useful.

FIXME: broken in lib/, it runs all test twice.

changes v1->v2:
Not sure if it's a good approach to process each test separately, there is no loop as Cyril suggested in [1], thus there is no summary what happen.

I previously had in similar loop in include/mk/generic_leaf_target.inc:

test-c-run: $(TEST_TARGETS)

	@set -e; echo; echo "===== Test C API ====="; \
	for i in $(TEST_TARGETS); do \
	    echo; echo "* $$i"; \
	    echo "PATH $(top_srcdir)/testcases/lib:$$PATH"; \
	    PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$$i || [ $$? -eq 32 ]; \
	done; echo; \
	echo "All C API tests passed or TCONF"

test-shell-run:

Quoted 8 lines
	@set -e; echo; echo "===== Test shell API ====="; \
	for i in $(TEST_SHELL_TARGETS); do \
	    echo; echo "* $$i"; \
	    PATH="$(abs_top_srcdir)/testcases/lib:$(abs_top_builddir)/testcases/lib:$$PATH" $(abs_srcdir)/$$i || [ $$? -eq 32 ]; \
	done; echo; \
	echo "All shell API tests passed or TCONF"

but that was problematic in lib (which uses generic_leaf_target.inc).

Why not just add runtest.sh into newlib_tests/ that would do all the work, then we can forget about all the trickery and just run the scritp on 'make test-c'. Well we would have to call make in newlib_tests first to make sure it was compiled, but that should be it.

And the same for testcases/lib/.

I do not think that adding a target per a testcase is sane.

Signature · 2 lines
Cyril Hrubis
[email protected]
author
#4 lore raw

RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets

Hi Cyril,

Quoted 42 lines

Hi!

Quoted 32 lines

For testing C and shell API.

---
Hi all,

another attempt for make test target.

In a long term, I'd like to wrap the execution with some script, as I suppose there will be some metadata in test, allowing to run tests which don't TPASS or TBROK (we have quite a lot of them). Then the summary would be even more useful.

FIXME: broken in lib/, it runs all test twice.

changes v1->v2:
Not sure if it's a good approach to process each test separately, there is no loop as Cyril suggested in [1], thus there is no summary what happen.

I previously had in similar loop in include/mk/generic_leaf_target.inc:

test-c-run: $(TEST_TARGETS)

	@set -e; echo; echo "===== Test C API ====="; \
	for i in $(TEST_TARGETS); do \
	    echo; echo "* $$i"; \
	    echo "PATH $(top_srcdir)/testcases/lib:$$PATH"; \
	    PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$$i || [ $$? -eq 32 ]; \
	done; echo; \
	echo "All C API tests passed or TCONF"

test-shell-run:

Quoted 8 lines
	@set -e; echo; echo "===== Test shell API ====="; \
	for i in $(TEST_SHELL_TARGETS); do \
	    echo; echo "* $$i"; \
	    PATH="$(abs_top_srcdir)/testcases/lib:$(abs_top_builddir)/testcases/lib:$$PATH" $(abs_srcdir)/$$i || [ $$? -eq 32 ]; \
	done; echo; \
	echo "All shell API tests passed or TCONF"

but that was problematic in lib (which uses generic_leaf_target.inc).

Quoted 8 lines

Why not just add runtest.sh into newlib_tests/ that would do all the work, then we can forget about all the trickery and just run the scritp on 'make test-c'. Well we would have to call make in newlib_tests first to make sure it was compiled, but that should be it.

And the same for testcases/lib/.

I do not think that adding a target per a testcase is sane.

Thanks for your input, makes sense. Actually, adding a script which would run it was my approach in 2 years old patch [1].

I'll add runtest.sh which will run all tests which by default TPASS/TCONF now, but possible to specify just some tests (speedup the development).

After we merge it I'll continue in Christian's effort to add metadata of expected output [2], which allows us to run all tests (including these which fail).

I suggest to add 3 targets: make test test-c test-shell (test will run both test-c test-shell).

Kind regards,
Petr

[1] https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/
[2] https://patchwork.ozlabs.org/project/ltp/patch/ce675759672af52bea02c11d51bd7d10f0bcb5cb.1566500817.git.clanig@suse.com/

author
#5 lore raw

RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets

Hi all,

...

Quoted 7 lines
Quoted 6 lines

Why not just add runtest.sh into newlib_tests/ that would do all the work, then we can forget about all the trickery and just run the scritp on 'make test-c'. Well we would have to call make in newlib_tests first to make sure it was compiled, but that should be it.

And the same for testcases/lib/.

Quoted 7 lines
Quoted 1 line

I do not think that adding a target per a testcase is sane.

Thanks for your input, makes sense. Actually, adding a script which would run it was my approach in 2 years old patch [1].

I'll add runtest.sh which will run all tests which by default TPASS/TCONF now, but possible to specify just some tests (speedup the development).

Also make install works also on lib/newlib_tests. Thus I'll install also runtest.sh and try to setup PATH a way it works on both running from git and running from make install. It might be worth to have lib tests runnable on SUT.

Kind regards,
Petr

Quoted 12 lines

After we merge it I'll continue in Christian's effort to add metadata of expected output [2], which allows us to run all tests (including these which fail).

I suggest to add 3 targets: make test test-c test-shell (test will run both test-c test-shell).

Kind regards,
Petr

[1] https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/
[2] https://patchwork.ozlabs.org/project/ltp/patch/ce675759672af52bea02c11d51bd7d10f0bcb5cb.1566500817.git.clanig@suse.com/