kunit: Add support for skipping entire test suites
v4 cover kunit: Add support for skipping entire test suites
Changelog:
==========
Previous version:
https://lore.kernel.org/all/[email protected]
* Merged the KUnit rust binding updates to Patch-1
Summary of changes
==================
This patch series introduces the ability to skip entire 'kunit_suite' based on runtime conditions, addressing a limitation where test suites could only skip individual test cases or fail when prerequisites were not met.
The motivation for this feature comes from test suites that depend on specific hardware features, kernel capabilities, or runtime conditions. Currently, such suites must either:
* Fail when prerequisites are missing
* Skip each test case individually with redundant checks
* Implement workarounds to avoid running tests
An example of such a requirement came from [1] where the patch author wanted to skip the entire 'kunit_suite' but then had to resort marking all struct 'kunit_case' as skipped by accessing 'kunit_case.status' private struct member. This usecase being addressed in the patch[1] can be better implemented with the changes proposed in this patch series.
Structure of the patch series
=============================
PATCH 1:
* Add a 'status' field to struct kunit_suite that allows 'suite_init' callbacks to mark the entire suite as KUNIT_SKIPPED.
* Modify the KUnit core to check this newly introduced 'status' field and bypass all test cases when a suite is marked as skipped.
* Update debugfs_print_results() to emmit correct logs for skipped 'kunit-suite'
* Init 'kunit_suite.status' to KUNIT_SUCCESS so that kunit_suite can be re-run from debugfs
* Update KUnit Rust binding macro-rule 'kunit_unsafe_test_suite' to include and initialize the newly introduced 'kunit_suite.status'
Patch 3:
* Providing an example in kunit-example-test.c demonstrating the usage pattern.
The implementation is minimal and non-intrusive, adding only a status field to kunit_suite and checks in two key functions. Test suites that don't use this proposed feature should be unaffected.
References
==========
[1]
https://lore.kernel.org/all/[email protected]
Vaibhav Jain (2):
kunit,rust: Add ability to skip entire test suites
kunit: Add example of test suite that can be skipped at runtime
include/kunit/test.h | 1 + lib/kunit/debugfs.c | 30 +++++++++++++++++++++--------- lib/kunit/kunit-example-test.c | 29 +++++++++++++++++++++++++++++ lib/kunit/test.c | 17 ++++++++++++++++- rust/kernel/kunit.rs | 1 + 5 files changed, 68 insertions(+), 10 deletions(-)
Signature · 1 line
2.54.0
PATCH v4 1/2 kunit,rust: Add ability to skip entire test suites
Currently, KUnit provides mechanisms to skip individual test cases, but there is no way to skip an entire test suite based on runtime conditions checked during suite initialization. This limitation forces test suites to either fail or skip tests individually when certain prerequisites are not available.
To address this limitation, the patch adds a 'status' field to struct kunit_suite that allows suite_init callbacks to mark the entire suite as KUNIT_SKIPPED. When a suite is marked as skipped, all test cases within that suite are bypassed without execution.
The patch proposed changes to kunit_suite_has_succeeded() to Check suite status before evaluating individual test case results. Also kunit_run_tests() is updated to skip suite execution if kunit_suite's 'status' is KUNIT_SKIPPED, thats either set before suite_init or by the suite_init callback itself. kunit_init_suite() is updated to initialize the 'status' of kunit_suite to KUNIT_SUCCESS so that any skipped suite's can be restarted from debugfs.
This enables test suites to perform runtime capability checks in their 'suite_init' callback and gracefully skip all tests when prerequisites are not met, rather than reporting failures or requiring each test case to perform redundant checks. In case a kunit-suite is skipped it can be re-run from the kunit's debugfs interface.
Also update debugfs_print_results() to clearly log the kunit-suite as 'SKIP'. kunit_suite_has_succeeded() is also updated on which debugfs_print_results() depends to update 'kunit_suite.status' in case any of the kunit_case has failed.
Finally, update KUnit Rust binding macro-rule 'kunit_unsafe_test_suite' to add and initialize the newly introduced 'kunit_suite.status'. Without this 'kunit_suite.status' field is never initialized which is an error for the Rust compiler.
---
Changelog
=========
V3->V4:
* Merged rust kunit binding related changes with this patch [ David Gow ]
* Added David Gow's RB
* Fixed a indentation issue in rust changes [ David Gow ]
V2->V3:
Link:
https://lore.kernel.org/all/[email protected]
None
V1->V2:
Link:
https://lore.kernel.org/all/[email protected]/
* Fix malformed and missing test-log when skipping kunit-suite. [David Gow]
* Update kunit_init_suite() to reset the kunit-suite so that it can be re-run [David Gow]
* Update kunit_suite_has_succeeded() to check for any failed test-case and update the 'status' for kunit-suite.
--- include/kunit/test.h | 1 + lib/kunit/debugfs.c | 30 +++++++++++++++++++++--------- lib/kunit/test.c | 17 ++++++++++++++++- rust/kernel/kunit.rs | 1 + 4 files changed, 39 insertions(+), 10 deletions(-)
include/kunit/test.h +1 -0
@@ -285,6 +285,7 @@ struct kunit_suite {struct string_stream *log;int suite_init_err;bool is_init; +enum kunit_status status; };/* Stores an array of suites, end points one past the end */
lib/kunit/debugfs.c +21 -9
@@ -76,18 +76,30 @@ static int debugfs_print_results(struct seq_file *seq, void *v)seq_puts (seq,"KTAP version 1\n" );seq_puts (seq,"1..1\n" ); -/* Print suite header because it is not stored in the test logs. */ -seq_puts (seq,KUNIT_SUBTEST_INDENT "KTAP version 1\n" ); -seq_printf (seq,KUNIT_SUBTEST_INDENT "# Subtest: %s\n" , suite->name); -seq_printf (seq,KUNIT_SUBTEST_INDENT "1..%zd\n" ,kunit_suite_num_test_cases (suite)); - -kunit_suite_for_each_test_case (suite, test_case) -debugfs_print_result (seq, test_case->log); +if (suite->status !=KUNIT_SKIPPED ) { +/* Print suite header because it is not stored in the test logs. */ +seq_puts (seq, +KUNIT_SUBTEST_INDENT "KTAP version 1\n" ); +seq_printf (seq, +KUNIT_SUBTEST_INDENT "# Subtest: %s\n" , + suite->name); +seq_printf (seq, +KUNIT_SUBTEST_INDENT "1..%zd\n" , +kunit_suite_num_test_cases (suite)); + +kunit_suite_for_each_test_case (suite, test_case) +debugfs_print_result (seq, test_case->log); + }debugfs_print_result (seq, suite->log); -seq_printf (seq,"%s %d %s\n" , -kunit_status_to_ok_not_ok (success),1 , suite->name); +if (suite->status !=KUNIT_SKIPPED ) +seq_printf (seq,"%s %d %s\n" , +kunit_status_to_ok_not_ok (success),1 , suite->name); +else +seq_printf (seq,"%s %d %s # SKIP %s\n" , +kunit_status_to_ok_not_ok (success),1 , suite->name, + suite->status_comment);return 0 ; }
lib/kunit/test.c +16 -1
@@ -214,12 +214,18 @@ enum kunit_status kunit_suite_has_succeeded(struct kunit_suite *suite)const struct kunit_case *test_case;enum kunit_status status =KUNIT_SKIPPED ; +if (suite->status ==KUNIT_SKIPPED ) +return KUNIT_SKIPPED ; +if (suite->suite_init_err)return KUNIT_FAILURE ;kunit_suite_for_each_test_case (suite, test_case) { -if (test_case->status ==KUNIT_FAILURE ) +if (test_case->status ==KUNIT_FAILURE ) { +/* Update the kunit_suite status also */ + suite->status =KUNIT_FAILURE ;return KUNIT_FAILURE ; + }else if (test_case->status ==KUNIT_SUCCESS ) status =KUNIT_SUCCESS ; }
@@ -795,12 +801,20 @@ int kunit_run_tests(struct kunit_suite *suite)/* Taint the kernel so we know we've run tests. */ add_taint (TAINT_TEST ,LOCKDEP_STILL_OK ); +if (suite->status ==KUNIT_SKIPPED ) + goto suite_end; +if (suite->suite_init) { suite->suite_init_err = suite->suite_init (suite);if (suite->suite_init_err) { + suite->status =KUNIT_FAILURE ;kunit_err (suite,KUNIT_SUBTEST_INDENT "# failed to initialize (%d)" , suite->suite_init_err); goto suite_end; + + }else if (suite->status ==KUNIT_SKIPPED ) { +/* Skip this kunit suite */ + goto suite_end; } }
@@ -825,6 +839,7 @@ static void kunit_init_suite(struct kunit_suite *suite)kunit_debugfs_create_suite (suite); suite->status_comment[0 ] ='\0' ; suite->suite_init_err =0 ; + suite->status =KUNIT_SUCCESS ;if (suite->log)string_stream_clear (suite->log);
rust/kernel/kunit.rs +1 -0
@@ -288,6 +288,7 @@ macro_rules! kunit_unsafe_test_suite { log: ::core::ptr::null_mut(), suite_init_err: 0, is_init: false, + status: kernel::bindings::kunit_status_KUNIT_SUCCESS, };#[used(compiler)]
Signature · 1 line
2.54.0
PATCH v4 2/2 kunit: Add example of test suite that can be skipped at runtime
In reply to Vaibhav JainAdd an example test suite name 'example_test_skip_suite' to 'kunit-example-test.c' that shows how to skip an entire test suite based on runtime conditions.
The example suite 'example_skip_suite' provides a 'suite_init' callback named example_skip_suite_init() which marks the entire suite as skipped using kunit_mark_skipped().
This demonstrates a way for conditionally skipping test suites when any prerequisites for kunit_suite execution are not met. The 'suite_init' callback can perform any necessary checks and mark the suite as skipped, preventing all test cases from executing while also indicating why the suite was skipped.
---
Changelog
=========V3->V4:
Link:
https://lore.kernel.org/all/[email protected]
* Updated patch index in the series
* Added RB from David Gow
V2->V3:
Link:
https://lore.kernel.org/all/[email protected]
* Updated patch index in the series
V1->V2:
Link:
https://lore.kernel.org/all/[email protected]/
* Added RB from 'David Gow'
--- lib/kunit/kunit-example-test.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
lib/kunit/kunit-example-test.c +29 -0
@@ -591,5 +591,34 @@ static struct kunit_suite example_init_test_suite = {*/ kunit_test_init_section_suites (&example_init_test_suite); +/* +* This test should always be skipped. +*/ +static void example_skip_suite_test (struct kunit *test) +{ +/* This line should never be seen */ +KUNIT_FAIL (test,"You should not see a this." ); +} + +static struct kunit_case example_skip_suite_test_cases[] = { +KUNIT_CASE (example_skip_suite_test), + {} +}; + +static int example_skip_suite_init (struct kunit_suite *suite) +{ +kunit_mark_skipped (suite,"Test suite expected to be skipped" ); +return 0 ; +} + +static struct kunit_suite example_test_skip_suite = { + .name ="example_skip_suite" , + .suite_init = example_skip_suite_init, + .test_cases = example_skip_suite_test_cases, +}; + +/* This registers a test suite that will be skipped */ +kunit_test_suite (example_test_skip_suite); +MODULE_DESCRIPTION ("Example KUnit test suite" );MODULE_LICENSE ("GPL v2" );
Signature · 1 line
2.54.0