make: Add make test{, -c, -c-run, -shell-run} targets
RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets
For testing C and shell API.
---
Hi all,
another attempt for make test target.
In a long term, I'd like to wrap the execution with some script, as I suppose there will be some metadata in test, allowing to run tests which don't TPASS or TBROK (we have quite a lot of them). Then the summary would be even more useful.
FIXME: broken in lib/, it runs all test twice.
changes v1->v2:
Not sure if it's a good approach to process each test separately, there is no loop as Cyril suggested in [1], thus there is no summary what happen.
I previously had in similar loop in include/mk/generic_leaf_target.inc:
test-c-run: $(TEST_TARGETS)
@set -e; echo; echo "===== Test C API ====="; \ for i in $(TEST_TARGETS); do \ echo; echo "* $$i"; \ echo "PATH $(top_srcdir)/testcases/lib:$$PATH"; \ PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$$i || [ $$? -eq 32 ]; \ done; echo; \ echo "All C API tests passed or TCONF"
test-shell-run:
@set -e; echo; echo "===== Test shell API ====="; \ for i in $(TEST_SHELL_TARGETS); do \ echo; echo "* $$i"; \ PATH="$(abs_top_srcdir)/testcases/lib:$(abs_top_builddir)/testcases/lib:$$PATH" $(abs_srcdir)/$$i || [ $$? -eq 32 ]; \ done; echo; \ echo "All shell API tests passed or TCONF"
but that was problematic in lib (which uses generic_leaf_target.inc).
Kind regards,
Petr
[1] https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/#2695533
Makefile | 23 ++++++++++++++++++++++- include/mk/generic_leaf_target.inc | 25 ++++++++++++++++++++++--- include/mk/generic_trunk_target.inc | 2 +- lib/newlib_tests/Makefile | 13 +++++++++++++ 4 files changed, 58 insertions(+), 5 deletions(-)
Makefile +22 -1
@@ -75,10 +75,14 @@ endif INSTALL_TARGETS += $(COMMON_TARGETS) CLEAN_TARGETS += $(COMMON_TARGETS) lib libs BOOTSTRAP_TARGETS := $(sort $(COMMON_TARGETS) $(CLEAN_TARGETS) $(INSTALL_TARGETS)) +TEST_TARGETS := lib +TEST_SHELL_TARGETS := lib CLEAN_TARGETS := $(addsuffix -clean,$(CLEAN_TARGETS)) INSTALL_TARGETS := $(addsuffix -install,$(INSTALL_TARGETS)) MAKE_TARGETS := $(addsuffix -all,$(filter-out lib,$(COMMON_TARGETS))) +TEST_TARGETS := $(addsuffix -test-c-run,$(TEST_TARGETS)) +TEST_SHELL_TARGETS := $(addsuffix -test-shell-run,$(TEST_SHELL_TARGETS)) # There's no reason why we should run `all' twice. Otherwise we're just wasting # 3+ mins of useful CPU cycles on a modern machine, and even more time on an
@@ -189,8 +193,25 @@ INSTALL_TARGETS += $(addprefix $(DESTDIR)/$(bindir)/,$(BINDIR_INSTALL_SCRIPTS)) $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir) +## Test library API +.PHONY: test test-c test-c-run test-shell-run + +$(TEST_TARGETS): lib-all + $(MAKE) -C "$(subst -test-c-run,,$@)" \ + -f "$(abs_top_srcdir)/$(subst -test-c-run,,$@)/Makefile" test-c-run + +$(TEST_SHELL_TARGETS): lib-all + $(MAKE) -C "$(subst -test-shell-run,,$@)" \ + -f "$(abs_top_srcdir)/$(subst -test-shell-run,,$@)/Makefile" test-shell-run + +test-c-run: $(TEST_TARGETS) + +test-shell-run: $(TEST_SHELL_TARGETS) + +test: test-c test-c-run test-shell-run + ## Install -install: $(INSTALL_TARGETS) +install: lib-all $(INSTALL_TARGETS) ## Help .PHONY: help
include/mk/generic_leaf_target.inc +22 -3
@@ -63,8 +63,12 @@ # real files or directories). This will automatically append # adds the .o suffix to all files referenced by # $(MAKE_TARGETS)) to CLEAN_TARGETS, if MAKE_TARGETS wasn't -# defined (see -# $(MAKE_TARGETS)). +# defined (see $(MAKE_TARGETS)). +# +# $(TEST_TARGETS) : LTP C API tests to be runned with test-c-run. +# +# $(TEST_SHELL_TARGETS) : LTP shell API tests to be runned with test-shell-run. +# # $(INSTALL_MODE) : What mode should we using when calling # install(1)? #
@@ -92,7 +96,7 @@ # INSTALL_DIR := $(libdir) # -.PHONY: all clean install +.PHONY: all clean install test test-c test-c-run test-shell-run ifneq ($(strip $(MAKE_TARGETS)),) $(MAKE_TARGETS) += $(HOST_MAKE_TARGETS)
@@ -102,11 +106,26 @@ $(MAKE_TARGETS): | $(MAKE_DEPS) all: $(MAKE_TARGETS) +TEST_TARGETS_RUN := $(addsuffix -run,$(TEST_TARGETS)) +TEST_SHELL_TARGETS_RUN := $(addsuffix -run,$(TEST_SHELL_TARGETS)) + +$(TEST_TARGETS_RUN): | $(TEST_TARGETS) +test-c-run: $(TEST_TARGETS_RUN) +test-shell-run: $(TEST_SHELL_TARGETS_RUN) + +.PHONY: $(TEST_TARGETS_RUN) $(TEST_SHELL_TARGETS_RUN) +$(TEST_TARGETS_RUN) $(TEST_SHELL_TARGETS_RUN): + @set -e; echo; echo "* $(subst -run,,$@)"; \ + PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$(subst -run,,$@) || [ $$? -eq 32 ]; \ + +test: test-c-run test-shell-run + clean:: $(CLEAN_DEPS) -$(RM) -f -r $(CLEAN_TARGETS) $(INSTALL_FILES): | $(INSTALL_DEPS) install: $(INSTALL_FILES) +test-c: $(TEST_TARGETS) # vim: syntax=make
include/mk/generic_trunk_target.inc +1 -1
@@ -48,7 +48,7 @@ include $(top_srcdir)/include/mk/functions.mk -RECURSIVE_TARGETS ?= all install +RECURSIVE_TARGETS ?= all install test test-c test-c-run test-shell-run $(eval $(get_make_dirs))
lib/newlib_tests/Makefile +13 -0
@@ -19,5 +19,18 @@ ifeq ($(ANDROID),1) FILTER_OUT_MAKE_TARGETS += test08 endif +# C API tests +# NOTE: only those which exit 0 (TPASS) or 32 (TCONF) +# TODO: commented out for quicker demonstration +# TEST_TARGETS += test05 test07 test09 test12 test15 test16 test18 +# TEST_TARGETS += test_exec test_timer tst_bool_expr tst_res_hexd +# TEST_TARGETS += tst_strstatus tst_fuzzy_sync01 tst_fuzzy_sync02 +TEST_TARGETS += tst_strstatus test05 + +# shell API tests +# NOTE: only those which exit 0 (TPASS) or 32 (TCONF) +TEST_SHELL_TARGETS += shell/tst_check_driver.sh +# TODO: commented out for quicker demonstration +# TEST_SHELL_TARGETS += shell/net/*.sh include $(top_srcdir)/include/mk/generic_leaf_target.mk
Signature · 1 line
2.32.0
RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets
Hello,
Quoted 6 lines · Petr Vorel
For testing C and shell API.
Signed-off-by Petr Vorel <[email protected]>---
Hi all,
Quoted 6 lines
another attempt for make test target.
In a long term, I'd like to wrap the execution with some script, as I suppose there will be some metadata in test, allowing to run tests which don't TPASS or TBROK (we have quite a lot of them). Then the summary would be even more useful.
For each test you could have an tcl/expect (or equivalent Perl/shell if we don't already require tcl) script which matches the output including return value and stderr/stdout.
For most tests it would just call a common function to check for TPASS/TBROK. For tests where we want to look for a given message, it could match the output. If we sometimes expect TCONF then it could perform a check to verify that it really should return TCONF.
I guess you could just put some data in a comment. I think that is likely to be harder though. At least with the number of tests we current have.
Signature · 2 lines
Thank you, Richard.
RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets
Hi Richie,
Quoted 12 lines
Hello,
Quoted 7 lines · Petr Vorel
For testing C and shell API.
Signed-off-by Petr Vorel <[email protected]>---
Hi all,another attempt for make test target.
Quoted 8 lines
Quoted 4 lines
In a long term, I'd like to wrap the execution with some script, as I suppose there will be some metadata in test, allowing to run tests which don't TPASS or TBROK (we have quite a lot of them). Then the summary would be even more useful.
For each test you could have an tcl/expect (or equivalent Perl/shell if we don't already require tcl) script which matches the output including return value and stderr/stdout.
I'd prefer to avoid tcl/expect. FYI There was some effort [1], based on my previous work [1], I plan to get back to it.
Quoted 4 lines
For most tests it would just call a common function to check for TPASS/TBROK. For tests where we want to look for a given message, it could match the output. If we sometimes expect TCONF then it could perform a check to verify that it really should return TCONF.
As I noted in [2] today, I might create very simple library for these test wrappers (small subset of things needed in test wrappers like lib/newlib_tests/shell/test_timeout.sh).
Quoted 3 lines
I guess you could just put some data in a comment. I think that is likely to be harder though. At least with the number of tests we current have.
Yep, that's what was used in [1].
Kind regards,
Petr
[1] https://patchwork.ozlabs.org/project/ltp/patch/ce675759672af52bea02c11d51bd7d10f0bcb5cb.1566500817.git.clanig@suse.com/
[2] https://lists.linux.it/pipermail/ltp/2021-June/023306.html
RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets
In reply to Petr VorelHi!
Quoted 32 lines
For testing C and shell API.
Signed-off-by Petr Vorel <[email protected]>---
Hi all,another attempt for make test target.
In a long term, I'd like to wrap the execution with some script, as I suppose there will be some metadata in test, allowing to run tests which don't TPASS or TBROK (we have quite a lot of them). Then the summary would be even more useful.
FIXME: broken in
lib/, it runs all test twice.changes
v1->v2:
Not sure if it's a good approach to process each test separately, there is no loop as Cyril suggested in [1], thus there is no summary what happen.I previously had in similar loop in
include/mk/generic_leaf_target.inc:test-c-run: $(TEST_TARGETS)
@set -e; echo; echo "===== Test C API ====="; \ for i in $(TEST_TARGETS); do \ echo; echo "* $$i"; \ echo "PATH $(top_srcdir)/testcases/lib:$$PATH"; \ PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$$i || [ $$? -eq 32 ]; \ done; echo; \ echo "All C API tests passed or TCONF"test-shell-run:
Quoted 8 lines
@set -e; echo; echo "===== Test shell API ====="; \ for i in $(TEST_SHELL_TARGETS); do \ echo; echo "* $$i"; \ PATH="$(abs_top_srcdir)/testcases/lib:$(abs_top_builddir)/testcases/lib:$$PATH" $(abs_srcdir)/$$i || [ $$? -eq 32 ]; \ done; echo; \ echo "All shell API tests passed or TCONF"but that was problematic in lib (which uses generic_leaf_target.inc).
Why not just add runtest.sh into newlib_tests/ that would do all the work, then we can forget about all the trickery and just run the scritp on 'make test-c'. Well we would have to call make in newlib_tests first to make sure it was compiled, but that should be it.
And the same for testcases/lib/.
I do not think that adding a target per a testcase is sane.
Signature · 2 lines
Cyril Hrubis [email protected]
RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets
Hi Cyril,
Quoted 42 lines
Hi!
Quoted 32 lines
For testing C and shell API.
Signed-off-by Petr Vorel <[email protected]>---
Hi all,another attempt for make test target.
In a long term, I'd like to wrap the execution with some script, as I suppose there will be some metadata in test, allowing to run tests which don't TPASS or TBROK (we have quite a lot of them). Then the summary would be even more useful.
FIXME: broken in
lib/, it runs all test twice.changes
v1->v2:
Not sure if it's a good approach to process each test separately, there is no loop as Cyril suggested in [1], thus there is no summary what happen.I previously had in similar loop in
include/mk/generic_leaf_target.inc:test-c-run: $(TEST_TARGETS)
@set -e; echo; echo "===== Test C API ====="; \ for i in $(TEST_TARGETS); do \ echo; echo "* $$i"; \ echo "PATH $(top_srcdir)/testcases/lib:$$PATH"; \ PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$$i || [ $$? -eq 32 ]; \ done; echo; \ echo "All C API tests passed or TCONF"test-shell-run:
Quoted 8 lines
@set -e; echo; echo "===== Test shell API ====="; \ for i in $(TEST_SHELL_TARGETS); do \ echo; echo "* $$i"; \ PATH="$(abs_top_srcdir)/testcases/lib:$(abs_top_builddir)/testcases/lib:$$PATH" $(abs_srcdir)/$$i || [ $$? -eq 32 ]; \ done; echo; \ echo "All shell API tests passed or TCONF"but that was problematic in lib (which uses generic_leaf_target.inc).
Quoted 8 lines
Why not just add
runtest.shinto newlib_tests/ that would do all the work, then we can forget about all the trickery and just run the scritp on 'make test-c'. Well we would have to call make innewlib_testsfirst to make sure it was compiled, but that should be it.And the same for testcases/lib/.
I do not think that adding a target per a testcase is sane.
Thanks for your input, makes sense. Actually, adding a script which would run it was my approach in 2 years old patch [1].
I'll add runtest.sh which will run all tests which by default TPASS/TCONF now, but possible to specify just some tests (speedup the development).
After we merge it I'll continue in Christian's effort to add metadata of expected output [2], which allows us to run all tests (including these which fail).
I suggest to add 3 targets: make test test-c test-shell (test will run both test-c test-shell).
Kind regards,
Petr
[1] https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/
[2] https://patchwork.ozlabs.org/project/ltp/patch/ce675759672af52bea02c11d51bd7d10f0bcb5cb.1566500817.git.clanig@suse.com/
RFC PATCH v2 1/1 make: Add make test{, -c, -c-run, -shell-run} targets
Hi all,
...
Quoted 7 lines
Quoted 6 lines
Why not just add
runtest.shinto newlib_tests/ that would do all the work, then we can forget about all the trickery and just run the scritp on 'make test-c'. Well we would have to call make innewlib_testsfirst to make sure it was compiled, but that should be it.And the same for testcases/lib/.
Quoted 7 lines
Quoted 1 line
I do not think that adding a target per a testcase is sane.
Thanks for your input, makes sense. Actually, adding a script which would run it was my approach in 2 years old patch [1].
I'll add
runtest.shwhich will run all tests which by default TPASS/TCONF now, but possible to specify just some tests (speedup the development).
Also make install works also on lib/newlib_tests. Thus I'll install also runtest.sh and try to setup PATH a way it works on both running from git and running from make install. It might be worth to have lib tests runnable on SUT.
Kind regards,
Petr
Quoted 12 lines
After we merge it I'll continue in Christian's effort to add metadata of expected output [2], which allows us to run all tests (including these which fail).
I suggest to add 3 targets: make test test-c test-shell (test will run both test-c test-shell).
Kind regards,
Petr[1] https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/
[2] https://patchwork.ozlabs.org/project/ltp/patch/ce675759672af52bea02c11d51bd7d10f0bcb5cb.1566500817.git.clanig@suse.com/