selftests/nolibc: fix up and improve test report
Subsystems (3)
cover selftests/nolibc: fix up and improve test report
Hi, Willy
Thanks very much for your merge of the v3 generic part1 of rv32, just tested your latest 20230604-nolibc-rv32+stkp6 branch, everything work well except a trivial test report regression on the 'run' target.
Besides the fixup, a standalone test-report target added to share them among run, run-user and re-run and allow independent test report check via direct 'make test-report'.
Best regards,
Zhangjin
---
Zhangjin Wu (4):
selftests/nolibc: add a test-report target selftests/nolibc: allow run test-report directly selftests/nolibc: always print the log file selftests/nolibc: fix up test-report for run target
tools/testing/selftests/nolibc/Makefile | 30 ++++++++++++------------- 1 file changed, 15 insertions(+), 15 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/4 selftests/nolibc: add a test-report target
A standalone test-report target is added to let the run, run-user and rerun targets share them.
--- tools/testing/selftests/nolibc/Makefile | 26 ++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
tools/testing/selftests/nolibc/Makefile +13 -13
@@ -127,14 +127,18 @@ nolibc-test: nolibc-test.c sysroot/$(ARCH)/include libc-test: nolibc-test.c $(QUIET_CC)$(CC) -o $@ $< -# qemu user-land test -run-user: nolibc-test - $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : +test-report: $(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \ END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ $(CURDIR)/run.out +# qemu user-land test +_run-user: nolibc-test + $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : + +run-user: _run-user test-report + initramfs: nolibc-test $(QUIET_MKDIR)mkdir -p initramfs $(call QUIET_INSTALL, initramfs/init)
@@ -147,20 +151,16 @@ kernel: initramfs $(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs # run the tests after building the kernel -run: kernel +_run: kernel $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out" - $(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \ - END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ - if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ - $(CURDIR)/run.out + +run: _run test-report # re-run the tests from an existing kernel -rerun: +_rerun: $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out" - $(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \ - END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ - if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ - $(CURDIR)/run.out + +rerun: _rerun test-report clean: $(call QUIET_CLEAN, sysroot)
Signature · 1 line
2.25.1
Willy Tarreau
maintainerRe: [PATCH 1/4] selftests/nolibc: add a test-report target
Quoted 21 lines · Zhangjin Wu
A standalone test-report target is added to let the run, run-user and rerun targets share them.
Signed-off-by Zhangjin Wu <[email protected]>--- tools/testing/selftests/nolibc/Makefile | 26 ++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)tools/testing/selftests/nolibc/Makefile +1
-3@@ -127,14 +127,18 @@ nolibc-test: nolibc-test.c sysroot/$(ARCH)/include libc-test: nolibc-test.c $(QUIET_CC)$(CC) -o $@ $< -# qemu user-land test -run-user: nolibc-test - $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : +test-report: $(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
tools/testing/selftests/nolibc/Makefile nolibc-test: nolibc-test.c sysroot/$(ARCH)/include · line 141
Quoted 10 lines
tools/testing/selftests/nolibc/Makefile +6
-0END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ $(CURDIR)/run.out +# qemu user-land test +_run-user: nolibc-test + $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : + +run-user: _run-user test-report +
This will not reliably work, there's no ordering here, nothing guarantees that test-report will run *after* _run-user (e.g. make -j). Another approach is needed if you want to factor this, but in general creating sequences in makefiles is difficult and often more painful than having 3 times the same 3 lines.
Willy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [PATCH 1/4] selftests/nolibc: add a test-report target
Quoted 26 lines · Zhangjin Wu
Quoted 25 lines
A standalone test-report target is added to let the run, run-user and rerun targets share them.
Signed-off-by Zhangjin Wu <[email protected]>--- tools/testing/selftests/nolibc/Makefile | 26 ++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)tools/testing/selftests/nolibc/Makefile +1
-3@@ -127,14 +127,18 @@ nolibc-test: nolibc-test.c sysroot/$(ARCH)/include libc-test: nolibc-test.c $(QUIET_CC)$(CC) -o $@ $< -# qemu user-land test -run-user: nolibc-test - $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : +test-report: $(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \ END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ $(CURDIR)/run.out
Quoted 13 lines
tools/testing/selftests/nolibc/Makefile nolibc-test: nolibc-test.c sysroot/$(ARCH)/include · line 141
Quoted 6 lines
tools/testing/selftests/nolibc/Makefile +6
-0+# qemu user-land test +_run-user: nolibc-test + $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : + +run-user: _run-user test-report +This will not reliably work, there's no ordering here, nothing guarantees that test-report will run *after* _run-user (e.g. make -j). Another approach is needed if you want to factor this, but in general creating sequences in makefiles is difficult and often more painful than having 3 times the same 3 lines.
Ok, thanks, what about this?
# LOG_REPORT: report the test results
LOG_REPORT := awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
printf(" See all results in %s\n", ARGV[1]); }'
run-user: nolibc-test
$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
$(Q)$(LOG_REPORT) $(CURDIR)/run.out
run: kernel
$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
$(Q)$(LOG_REPORT) $(CURDIR)/run.out
rerun:
$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
$(Q)$(LOG_REPORT) $(CURDIR)/run.outOr we directly add a standalone test report script? something like tools/testing/selftests/nolibc/report.sh
#!/bin/sh
#
# report.sh -- report the test results of nolibc-test
#
LOG_FILE=$1
[ ! -f "$LOG_FILE" ] && echo "Usage: $0 /path/to/run.out"
awk '
/\[OK\][\r]*$$/{ p++ }
/\[FAIL\][\r]*$$/{ f++ }
/\[SKIPPED\][\r]*$$/{ s++ }
END {
printf("%d test(s) passed, %d skipped, %d failed.", p, s, f);
printf(" See all results in %s\n", ARGV[1]);
}' $LOG_FILEAnd use it like this:
LOG_REPORT = $(CURDIR)/report.sh
Best regards,
Zhangjin
Quoted 1 line
Willy
Re: [PATCH 1/4] selftests/nolibc: add a test-report target
Hi, Willy
Quoted 80 lines
Quoted 26 lines · Zhangjin Wu
Quoted 25 lines
A standalone test-report target is added to let the run, run-user and rerun targets share them.
Signed-off-by Zhangjin Wu <[email protected]>--- tools/testing/selftests/nolibc/Makefile | 26 ++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)tools/testing/selftests/nolibc/Makefile +1
-3@@ -127,14 +127,18 @@ nolibc-test: nolibc-test.c sysroot/$(ARCH)/include libc-test: nolibc-test.c $(QUIET_CC)$(CC) -o $@ $< -# qemu user-land test -run-user: nolibc-test - $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : +test-report: $(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \ END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ $(CURDIR)/run.outQuoted 13 lines
tools/testing/selftests/nolibc/Makefile nolibc-test: nolibc-test.c sysroot/$(ARCH)/include · line 141
Quoted 6 lines
tools/testing/selftests/nolibc/Makefile +6
-0+# qemu user-land test +_run-user: nolibc-test + $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : + +run-user: _run-user test-report +This will not reliably work, there's no ordering here, nothing guarantees that test-report will run *after* _run-user (e.g. make -j). Another approach is needed if you want to factor this, but in general creating sequences in makefiles is difficult and often more painful than having 3 times the same 3 lines.
Ok, thanks, what about this?
# LOG_REPORT: report the test results LOG_REPORT := awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \ END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ printf(" See all results in %s\n", ARGV[1]); }' run-user: nolibc-test $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : $(Q)$(LOG_REPORT) $(CURDIR)/run.out run: kernel $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out" $(Q)$(LOG_REPORT) $(CURDIR)/run.out rerun: $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out" $(Q)$(LOG_REPORT) $(CURDIR)/run.outOr we directly add a standalone test report script? something like
tools/testing/selftests/nolibc/report.sh#!/bin/sh # # report.sh -- report the test results of nolibc-test # LOG_FILE=$1 [ ! -f "$LOG_FILE" ] && echo "Usage: $0 /path/to/run.out" awk ' /\[OK\][\r]*$$/{ p++ } /\[FAIL\][\r]*$$/{ f++ } /\[SKIPPED\][\r]*$$/{ s++ } END { printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); printf(" See all results in %s\n", ARGV[1]); }' $LOG_FILE
Quoted 4 lines
And use it like this:
LOG_REPORT = $(CURDIR)/report.sh
I plan to renew this patchset, which one of the above methods do you prefer?
For the always print statement:
printf(" See all results in %s\n", ARGV[1]); }'I will paste the reason why I need it, as mentioned in [1], if you still need a clean test report, I will give up this change ;-)
Thanks,
Zhangjin
---
[1]: https://lore.kernel.org/linux-riscv/[email protected]/
Quoted 4 lines
Best regards,
ZhangjinQuoted 1 line
Willy
Willy Tarreau
maintainerRe: [PATCH 1/4] selftests/nolibc: add a test-report target
Hi Zhangjin,
Quoted 82 lines · Zhangjin Wu
Hi, Willy
Quoted 80 lines
Quoted 26 lines · Zhangjin Wu
Quoted 25 lines
A standalone test-report target is added to let the run, run-user and rerun targets share them.
Signed-off-by Zhangjin Wu <[email protected]>--- tools/testing/selftests/nolibc/Makefile | 26 ++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)tools/testing/selftests/nolibc/Makefile +1
-3@@ -127,14 +127,18 @@ nolibc-test: nolibc-test.c sysroot/$(ARCH)/include libc-test: nolibc-test.c $(QUIET_CC)$(CC) -o $@ $< -# qemu user-land test -run-user: nolibc-test - $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : +test-report: $(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \ END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ $(CURDIR)/run.outQuoted 13 lines
tools/testing/selftests/nolibc/Makefile nolibc-test: nolibc-test.c sysroot/$(ARCH)/include · line 141
Quoted 6 lines
tools/testing/selftests/nolibc/Makefile +6
-0+# qemu user-land test +_run-user: nolibc-test + $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : + +run-user: _run-user test-report +This will not reliably work, there's no ordering here, nothing guarantees that test-report will run *after* _run-user (e.g. make -j). Another approach is needed if you want to factor this, but in general creating sequences in makefiles is difficult and often more painful than having 3 times the same 3 lines.
Ok, thanks, what about this?
# LOG_REPORT: report the test results LOG_REPORT := awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \ END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ printf(" See all results in %s\n", ARGV[1]); }' run-user: nolibc-test $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : $(Q)$(LOG_REPORT) $(CURDIR)/run.out run: kernel $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out" $(Q)$(LOG_REPORT) $(CURDIR)/run.out rerun: $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out" $(Q)$(LOG_REPORT) $(CURDIR)/run.outOr we directly add a standalone test report script? something like
tools/testing/selftests/nolibc/report.sh#!/bin/sh # # report.sh -- report the test results of nolibc-test # LOG_FILE=$1 [ ! -f "$LOG_FILE" ] && echo "Usage: $0 /path/to/run.out" awk ' /\[OK\][\r]*$$/{ p++ } /\[FAIL\][\r]*$$/{ f++ } /\[SKIPPED\][\r]*$$/{ s++ } END { printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); printf(" See all results in %s\n", ARGV[1]); }' $LOG_FILE
Quoted 7 lines
Quoted 4 lines
And use it like this:
LOG_REPORT = $(CURDIR)/report.shI plan to renew this patchset, which one of the above methods do you prefer?
IFF it needs to be done I prefer the macro in the Makefile to avoid depending on external scripts that are useless outside of the makefile. BUT, my point remains that I adopted this so that I could quickly and visually check that everything was OK. I'm fine with any other method but I do not want to have to carefully read all these lines to make sure I'm not mixing a "8" with a "0" (I'm mentioning this one because it's exactly the one I had when I decided to add the extra values). For example if you prepend "FAILURE: ", "WARNING: ", "SUCCESS: " in front of these lines to summarize them depending on the highest level encountered (success, skipped, failed), then I'm fine because it's easy to check that all lines show the same word.
Quoted 3 lines
For the always print statement:
printf(" See all results in %s\n", ARGV[1]); }'
Then please put it on its own line without the leading space, this will be even more readable.
Quoted 2 lines
I will paste the reason why I need it, as mentioned in [1], if you still need a clean test report, I will give up this change ;-)
No worries, I don't want to be annoying if you need something, but I don't want to be annoyed by changes either :-)
thanks,
Willy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [PATCH 1/4] selftests/nolibc: add a test-report target
Quoted 52 lines · Zhangjin Wu
Quoted 2 lines
Hi, Willy
(...)
Quoted 41 lines
Quoted 41 lines
Ok, thanks, what about this?
# LOG_REPORT: report the test results LOG_REPORT := awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \ END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ printf(" See all results in %s\n", ARGV[1]); }' run-user: nolibc-test $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : $(Q)$(LOG_REPORT) $(CURDIR)/run.out run: kernel $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out" $(Q)$(LOG_REPORT) $(CURDIR)/run.out rerun: $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out" $(Q)$(LOG_REPORT) $(CURDIR)/run.outOr we directly add a standalone test report script? something like
tools/testing/selftests/nolibc/report.sh#!/bin/sh # # report.sh -- report the test results of nolibc-test # LOG_FILE=$1 [ ! -f "$LOG_FILE" ] && echo "Usage: $0 /path/to/run.out" awk ' /\[OK\][\r]*$$/{ p++ } /\[FAIL\][\r]*$$/{ f++ } /\[SKIPPED\][\r]*$$/{ s++ } END { printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); printf(" See all results in %s\n", ARGV[1]); }' $LOG_FILEQuoted 7 lines
Quoted 4 lines
And use it like this:
LOG_REPORT = $(CURDIR)/report.shI plan to renew this patchset, which one of the above methods do you prefer?
Quoted 12 lines
IFF it needs to be done I prefer the macro in the
Makefileto avoid depending on external scripts that are useless outside of the makefile. BUT, my point remains that I adopted this so that I could quickly and visually check that everything was OK. I'm fine with any other method but I do not want to have to carefully read all these lines to make sure I'm not mixing a "8" with a "0" (I'm mentioning this one because it's exactly the one I had when I decided to add the extra values). For example if you prepend "FAILURE: ", "WARNING: ", "SUCCESS: " in front of these lines to summarize them depending on the highest level encountered (success, skipped, failed), then I'm fine because it's easy to check that all lines show the same word.
Ok.
Quoted 7 lines
Quoted 3 lines
For the always print statement:
printf(" See all results in %s\n", ARGV[1]); }'Then please put it on its own line without the leading space, this will be even more readable.
It is a good balance.
This may be more useful if we run this from the kselftest framework, seems it is not able to run it from the 'kselftest' target currently, here shares something I have tried:
I tried something like this, seems run-user works, but defconfig and run not.
diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 4a3a105e1fdf..70693f6501c6 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -83,6 +83,8 @@ CFLAGS ?= -Os -fno-ident -fno-asynchronous-unwind-tables -std=c89 \
$(CFLAGS_$(ARCH)) $(CFLAGS_STACKPROTECTOR)
LDFLAGS := -s
+NOLIBC_SUBTARGETS ?= run-user
+
help:
@echo "Supported targets under selftests/nolibc:"
@echo " all call the \"run\" target below"
@@ -110,7 +112,9 @@ help:
@echo " IMAGE_NAME = $(if $(IMAGE_NAME),$(IMAGE_NAME),UNKNOWN_ARCH) [determined from \$$ARCH]"
@echo ""
-all: run
+run_tests: $(NOLIBC_SUBTARGETS)
+
+all: $(NOLIBC_SUBTARGETS)This can be triggered from the top-level kselftest framework:
$ make -C /path/to/linux-stable kselftest TARGETS=nolibc NOLIBC_SUBTARGETS=run-user
And seems we still not support O= currently either.
Quoted 6 lines
Quoted 2 lines
I will paste the reason why I need it, as mentioned in [1], if you still need a clean test report, I will give up this change ;-)
No worries, I don't want to be annoying if you need something, but I don't want to be annoyed by changes either :-)
Thanks,
Zhangjin
Quoted 2 lines
thanks,
Willy
PATCH 2/4 selftests/nolibc: allow run test-report directly
In reply to Zhangjin WuThis allows to re-check the test report via 'make test-report' without the requirement to re-run the dependency targets.
--- tools/testing/selftests/nolibc/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/nolibc/Makefile +2 -2
@@ -128,10 +128,10 @@ libc-test: nolibc-test.c $(QUIET_CC)$(CC) -o $@ $< test-report: - $(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \ + $(Q)[ -f $(CURDIR)/run.out ] && awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \ END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ - $(CURDIR)/run.out + $(CURDIR)/run.out || : # qemu user-land test _run-user: nolibc-test
Signature · 1 line
2.25.1
PATCH 3/4 selftests/nolibc: always print the log file
In reply to Zhangjin WuThis allows to check the other issues of the output file manually even when all of them passed.
--- tools/testing/selftests/nolibc/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/nolibc/Makefile +1 -1
@@ -130,7 +130,7 @@ libc-test: nolibc-test.c test-report: $(Q)[ -f $(CURDIR)/run.out ] && awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \ END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ - if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ + printf(" See all results in %s\n", ARGV[1]); }' \ $(CURDIR)/run.out || : # qemu user-land test
Signature · 1 line
2.25.1
Willy Tarreau
maintainerRe: [PATCH 3/4] selftests/nolibc: always print the log file
Quoted 2 lines · Zhangjin Wu
This allows to check the other issues of the output file manually even when all of them passed.
Till now I preferred not to see it when everything was OK since it was useless and permitted a quick visual check in the reports. Do you really think it's useful ? If others prefer it that way we can change it but I purposely added this test to "improve" the output (for me at least). I'm interested in opinions here.
Willy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [PATCH 3/4] selftests/nolibc: always print the log file
Quoted 10 lines · Zhangjin Wu
Quoted 2 lines
This allows to check the other issues of the output file manually even when all of them passed.
Till now I preferred not to see it when everything was OK since it was useless and permitted a quick visual check in the reports. Do you really think it's useful ? If others prefer it that way we can change it but I purposely added this test to "improve" the output (for me at least). I'm interested in opinions here.
I planed to add detailed potential issues to check in the commit message, but didn't do that eventually.
The potential 'issues' may be:
1. string alignment, I found one character offset in the user-space 'efault' handler patchset
2. duplicated print, the one like '30 fork' we have fixed up
3. kernel messages (may provide some important info)
I did add this manually several times in the past weeks, so, if the path is there, I can simply copy it and cat it, hope we can print the path by default ;-)
The commit message may be changed to something like this:
This allows us to check the details in the log file even when all of them passed, from the log details, we can check the string alignment, duplicated print and kernel messages.
Best regards,
Zhangjin
Quoted 1 line
Willy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH 4/4 selftests/nolibc: fix up test-report for run target
In reply to Zhangjin WuThere is a '\r' at the end of every log line when run nolibc-test on qemu-system (make run), add support for this case, otherwise, the test result will like this:
0 test(s) passed, 0 skipped, 0 failed.
--- tools/testing/selftests/nolibc/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/nolibc/Makefile +1 -1
@@ -128,7 +128,7 @@ libc-test: nolibc-test.c $(QUIET_CC)$(CC) -o $@ $< test-report: - $(Q)[ -f $(CURDIR)/run.out ] && awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \ + $(Q)[ -f $(CURDIR)/run.out ] && awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \ END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ printf(" See all results in %s\n", ARGV[1]); }' \ $(CURDIR)/run.out || :
Signature · 1 line
2.25.1
Willy Tarreau
maintainerRe: [PATCH 4/4] selftests/nolibc: fix up test-report for run target
Quoted 5 lines · Zhangjin Wu
There is a '\r' at the end of every log line when run nolibc-test on qemu-system (make run), add support for this case, otherwise, the test result will like this:
0 test(s) passed, 0 skipped, 0 failed.
Argh, sorry for this one! Note that as a general rule, this should have been the first patch of the series (fixes first so that they can be squashed into a pending series or backported when already merged).
I'll remerge it into the series for now.
Thank you!
Willy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Willy Tarreau
maintainerQuoted 3 lines · Zhangjin Wu
Thanks very much for your merge of the v3 generic part1 of rv32, just tested your latest 20230604-nolibc-rv32+stkp6 branch, everything work well except a trivial test report regression on the 'run' target.
(...)
I've squashed your fix into the pending patch and pushed branch 20230605-nolibc-rv32+stkp7. I have only tested userland (I really need to leave now, no time for a kernel build).
Zhangjin and Thomas, now that your last two fixes are merged, I'm assuming that Paul can take the branch any time. If you're seeing a showstopper that needs to be fixed, please let him know, and I'll deal with it once I'm connected again, but please no more attempts to further improve that branch for now (i.e. consider it merged for any future work so that we can finally settle on something).
Thanks!
Willy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 10 lines · Zhangjin Wu
Quoted 3 lines
Thanks very much for your merge of the v3 generic part1 of rv32, just tested your latest 20230604-nolibc-rv32+stkp6 branch, everything work well except a trivial test report regression on the 'run' target.
(...)
I've squashed your fix into the pending patch and pushed branch 20230605-nolibc-rv32+stkp7. I have only tested userland (I really need to leave now, no time for a kernel build).
Just did kernel build + nolibc test for arm, aarch64 and rv64, no regressions found.
Quoted 6 lines
Zhangjin and Thomas, now that your last two fixes are merged, I'm assuming that Paul can take the branch any time. If you're seeing a showstopper that needs to be fixed, please let him know, and I'll deal with it once I'm connected again, but please no more attempts to further improve that branch for now (i.e. consider it merged for any future work so that we can finally settle on something).
Ok, let's work on v6.6 ;-)
Best regards,
Zhangjin
Quoted 3 lines
Thanks!
Willy
Willy Tarreau
maintainerQuoted 6 lines · Zhangjin Wu
Quoted 6 lines · Zhangjin Wu
Quoted 3 lines
Thanks very much for your merge of the v3 generic part1 of rv32, just tested your latest 20230604-nolibc-rv32+stkp6 branch, everything work well except a trivial test report regression on the 'run' target.
(...)
Quoted 7 lines
Quoted 4 lines
I've squashed your fix into the pending patch and pushed branch 20230605-nolibc-rv32+stkp7. I have only tested userland (I really need to leave now, no time for a kernel build).
Just did kernel build + nolibc test for arm, aarch64 and rv64, no regressions found.
Much appreciated, thank you. This evening I'll try the remaining archs as I already have the cross-compilers, then will give Paul the go.
Cheers,
Willy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv