selftests: kselftest_harness: support using xfail
Subsystems (4)
Jakub Kicinski
maintainerv2 cover selftests: kselftest_harness: support using xfail
Hi!
When running selftests for our subsystem in our CI we'd like all tests to pass. Currently some tests use SKIP for cases they expect to fail, because the kselftest_harness limits the return codes to pass/fail/skip.
Clean up and support the use of the full range of ksft exit codes under kselftest_harness.
Merge plan is to put it on top of -rc4 and merge into net-next. That way others should be able to pull the patches without any networking changes.
v2:
- fix alignment
v1: https://lore.kernel.org/all/[email protected]/
Jakub Kicinski (4):
selftests: kselftest_harness: pass step via shared memory
selftests: kselftest_harness: use KSFT_* exit codes
selftests: kselftest_harness: support using xfail
selftests: ip_local_port_range: use XFAIL instead of SKIP
tools/testing/selftests/kselftest_harness.h | 67 ++++++++++++++----- .../selftests/net/ip_local_port_range.c | 2 +- 2 files changed, 52 insertions(+), 17 deletions(-)
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerPATCH v2 1/4 selftests: kselftest_harness: pass step via shared memory
Commit 0ef67a888375 ("selftests/harness: Report skip reason") added shared memory to communicate between harness and test. Use that instead of exit codes to send the failing step back to the harness. The exit codes are limited and because of the step passing we can't use the full range of KSFT_* exit codes.
--- tools/testing/selftests/kselftest_harness.h | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-)
tools/testing/selftests/kselftest_harness.h +10 -11
@@ -101,8 +101,8 @@* ASSERT_* number for which the test failed. This behavior can be enabled by * writing `_metadata->no_print = true;` before the check sequence that is * unable to print. When an error occur, instead of printing an error message -* and calling `abort(3)`, the test process call `_exit(2)` with the assert -* number as argument, which is then printed by the parent process. +* and calling `abort(3)`, the test process call `_exit(2)` and pass the error +* to be printed to the parent process via shared memory. */ #define TH_LOG (fmt, ...) do { \ if (TH_LOG_ENABLED) \
@@ -695,9 +695,8 @@ __bail(_assert, _metadata))#define __INC_STEP (_metadata) \ -/* Keep "step" below 255 (which is used for "SKIP" reporting). */ \ - if (_metadata->passed && _metadata->step < 253) \ - _metadata->step++; + if (_metadata->passed) \ + _metadata->results->step++;#define is_signed_type (var) (!!(((__typeof__(var))(-1)) < (__typeof__(var))1))
@@ -784,6 +783,7 @@struct __test_results {char reason[1024 ];/* Reason for test result */ +unsigned int step;/* Test step reached without failure */ };struct __test_metadata ;
@@ -837,7 +837,6 @@ struct __test_metadata {int trigger;/* extra handler after the evaluation */ int timeout;/* seconds to wait for test timeout */ bool timed_out;/* did this test timeout instead of exiting? */ -__u8 step;bool no_print;/* manual trigger when TH_LOG_STREAM is not available */ bool aborted;/* stopped test due to failed ASSERT */ bool setup_completed;/* did setup finish? */
@@ -875,7 +874,7 @@ static inline void __test_check_assert(struct __test_metadata *t) {if (t->aborted) {if (t->no_print) -_exit (t->step); +_exit (1 );abort (); } }
@@ -960,7 +959,7 @@ void __wait_for_test(struct __test_metadata *t)fprintf (TH_LOG_STREAM ,"# %s: Test failed at step #%d\n" , t->name, -WEXITSTATUS (status)); + t->results->step); } } }else if (WIFSIGNALED (status)) {
@@ -1114,9 +1113,9 @@ void __run_test(struct __fixture_metadata *f, t->passed =1 ; t->skip =0 ; t->trigger =0 ; - t->step =1 ; t->no_print =0 ;memset (t->results->reason,0 ,sizeof (t->results->reason)); + t->results->step =1 ;ksft_print_msg (" RUN %s%s%s.%s ...\n" , f->name, variant->name[0 ] ?"." :"" , variant->name, t->name);
@@ -1137,8 +1136,8 @@ void __run_test(struct __fixture_metadata *f,/* Pass is exit 0 */ if (t->passed)_exit (0 ); -/* Something else happened, report the step. */ -_exit (t->step); +/* Something else happened. */ +_exit (1 ); }else {__wait_for_test (t); }
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerPATCH v2 2/4 selftests: kselftest_harness: use KSFT_* exit codes
In reply to Jakub KicinskiNow that we no longer need low exit codes to communicate assertion steps - use normal KSFT exit codes.
--- tools/testing/selftests/kselftest_harness.h | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
tools/testing/selftests/kselftest_harness.h +6 -7
@@ -874,7 +874,7 @@ static inline void __test_check_assert(struct __test_metadata *t) {if (t->aborted) {if (t->no_print) -_exit (1 ); +_exit (KSFT_FAIL );abort (); } }
@@ -937,7 +937,7 @@ void __wait_for_test(struct __test_metadata *t)fprintf (TH_LOG_STREAM ,"# %s: Test terminated by timeout\n" , t->name); }else if (WIFEXITED (status)) { -if (WEXITSTATUS (status) ==255 ) { +if (WEXITSTATUS (status) ==KSFT_SKIP ) {/* SKIP */ t->passed =1 ; t->skip =1 ;
@@ -950,7 +950,7 @@ void __wait_for_test(struct __test_metadata *t) }else {switch (WEXITSTATUS (status)) {/* Success */ -case 0 : +case KSFT_PASS : t->passed =1 ;break ;/* Other failure, assume step report. */
@@ -1132,12 +1132,11 @@ void __run_test(struct __fixture_metadata *f,setpgrp (); t->fn (t, variant);if (t->skip) -_exit (255 ); -/* Pass is exit 0 */ +_exit (KSFT_SKIP );if (t->passed) -_exit (0 ); +_exit (KSFT_PASS );/* Something else happened. */ -_exit (1 ); +_exit (KSFT_FAIL ); }else {__wait_for_test (t); }
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerPATCH v2 3/4 selftests: kselftest_harness: support using xfail
In reply to Jakub KicinskiSelftest summary includes XFAIL but there's no way to use it from within the harness. Support it in a similar way to skip.
Currently tests report skip for things they expect to fail e.g. when given combination of parameters is known to be unsupported. This is confusing because in an ideal environment and fully featured kernel no tests should be skipped.
---
v2: fix alignment of the reason (remove one space after XFAIL)
--- tools/testing/selftests/kselftest_harness.h | 37 +++++++++++++++++++++ 1 file changed, 37 insertions(+)
tools/testing/selftests/kselftest_harness.h +37 -0
@@ -141,6 +141,33 @@ statement; \ }while (0 ) +/** +* XFAIL() +* +* @statement: statement to run after reporting XFAIL +* @fmt: format string +* @...: optional arguments +* +* .. code-block:: c +* +* XFAIL(statement, fmt, ...); +* +* This forces a "pass" after reporting why something is expected to fail, +* and runs "statement", which is usually "return" or "goto skip". + */ +#define XFAIL (statement, fmt, ...) do { \ + snprintf(_metadata->results->reason, \ + sizeof(_metadata->results->reason), fmt, ##__VA_ARGS__); \ + if (TH_LOG_ENABLED) { \ + fprintf(TH_LOG_STREAM, "# XFAIL %s\n", \ + _metadata->results->reason); \ + } \ + _metadata->passed = 1; \ + _metadata->xfail = 1; \ + _metadata->trigger = 0; \ + statement; \ +} while (0) +/** * TEST() - Defines the test function and creates the registration * stub
@@ -834,6 +861,7 @@ struct __test_metadata {int termsig;int passed;int skip;/* did SKIP get used? */ +int xfail;/* did XFAIL get used? */ int trigger;/* extra handler after the evaluation */ int timeout;/* seconds to wait for test timeout */ bool timed_out;/* did this test timeout instead of exiting? */
@@ -941,6 +969,9 @@ void __wait_for_test(struct __test_metadata *t)/* SKIP */ t->passed =1 ; t->skip =1 ; + }else if (WEXITSTATUS (status) ==KSFT_XFAIL ) { + t->passed =1 ; + t->xfail =1 ; }else if (t->termsig !=-1 ) { t->passed =0 ; fprintf(TH_LOG_STREAM,
@@ -1112,6 +1143,7 @@ void __run_test(struct __fixture_metadata *f,/* reset test struct */ t->passed =1 ; t->skip =0 ; + t->xfail =0 ; t->trigger =0 ; t->no_print =0 ;memset (t->results->reason,0 ,sizeof (t->results->reason));
@@ -1133,6 +1165,8 @@ void __run_test(struct __fixture_metadata *f, t->fn (t, variant);if (t->skip)_exit (KSFT_SKIP ); +if (t->xfail) +_exit (KSFT_XFAIL );if (t->passed)_exit (KSFT_PASS );/* Something else happened. */
@@ -1146,6 +1180,9 @@ void __run_test(struct __fixture_metadata *f,if (t->skip)ksft_test_result_skip ("%s\n" , t->results->reason[0 ] ? t->results->reason :"unknown" ); +else if (t->xfail) +ksft_test_result_xfail ("%s\n" , t->results->reason[0 ] ? + t->results->reason :"unknown" );else ksft_test_result (t->passed,"%s%s%s.%s\n" , f->name, variant->name[0 ] ?"." :"" , variant->name, t->name);
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerPATCH v2 4/4 selftests: ip_local_port_range: use XFAIL instead of SKIP
In reply to Jakub KicinskiSCTP does not support IP_LOCAL_PORT_RANGE and we know it, so use XFAIL instead of SKIP.
--- tools/testing/selftests/net/ip_local_port_range.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/net/ip_local_port_range.c +1 -1
@@ -366,7 +366,7 @@ TEST_F(ip_local_port_range, late_bind)__u16 port;if (variant->so_protocol ==IPPROTO_SCTP ) -SKIP (return,"SCTP doesn't support IP_BIND_ADDRESS_NO_PORT" ); +XFAIL (return,"SCTP doesn't support IP_BIND_ADDRESS_NO_PORT" ); fd =socket (variant->so_domain, variant->so_type,0 ); ASSERT_GE(fd, 0) TH_LOG("socket failed");
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerQuoted 7 lines · On Thu, 15 Feb 2024 16:26:15 -0800 Jakub Kicinski wrote:
Hi!
When running selftests for our subsystem in our CI we'd like all tests to pass. Currently some tests use SKIP for cases they expect to fail, because the
kselftest_harnesslimits the return codes to pass/fail/skip.
Quoted 6 lines
Clean up and support the use of the full range of ksft exit codes under
kselftest_harness.Merge plan is to put it on top of -rc4 and merge into net-next. That way others should be able to pull the patches without any networking changes.
I need to rejig these to follow Kees's suggestion from:
https://lore.kernel.org/all/[email protected]/
Signature · 1 line
pw-bot: cr