Skip to contentSkip to conversation

kunit: Add support for skipping entire test suites

PATCH New Vaibhav Jain 1 participant 3 messages Jun 26

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v4 cover kunit: Add support for skipping entire test suites

Changelog:
==========
Previous version:
https://lore.kernel.org/all/[email protected]

* Merged the KUnit rust binding updates to Patch-1

Summary of changes
==================

This patch series introduces the ability to skip entire 'kunit_suite' based on runtime conditions, addressing a limitation where test suites could only skip individual test cases or fail when prerequisites were not met.

The motivation for this feature comes from test suites that depend on specific hardware features, kernel capabilities, or runtime conditions. Currently, such suites must either:
* Fail when prerequisites are missing
* Skip each test case individually with redundant checks
* Implement workarounds to avoid running tests

An example of such a requirement came from [1] where the patch author wanted to skip the entire 'kunit_suite' but then had to resort marking all struct 'kunit_case' as skipped by accessing 'kunit_case.status' private struct member. This usecase being addressed in the patch[1] can be better implemented with the changes proposed in this patch series.

Structure of the patch series
=============================
PATCH 1:
* Add a 'status' field to struct kunit_suite that allows 'suite_init' callbacks to mark the entire suite as KUNIT_SKIPPED.
* Modify the KUnit core to check this newly introduced 'status' field and bypass all test cases when a suite is marked as skipped.
* Update debugfs_print_results() to emmit correct logs for skipped 'kunit-suite'
* Init 'kunit_suite.status' to KUNIT_SUCCESS so that kunit_suite can be re-run from debugfs
* Update KUnit Rust binding macro-rule 'kunit_unsafe_test_suite' to include and initialize the newly introduced 'kunit_suite.status'

Patch 3:
* Providing an example in kunit-example-test.c demonstrating the usage pattern.

The implementation is minimal and non-intrusive, adding only a status field to kunit_suite and checks in two key functions. Test suites that don't use this proposed feature should be unaffected.

References
==========
[1]
https://lore.kernel.org/all/[email protected]

Vaibhav Jain (2):
kunit,rust: Add ability to skip entire test suites
kunit: Add example of test suite that can be skipped at runtime

 include/kunit/test.h           |  1 +
 lib/kunit/debugfs.c            | 30 +++++++++++++++++++++---------
 lib/kunit/kunit-example-test.c | 29 +++++++++++++++++++++++++++++
 lib/kunit/test.c               | 17 ++++++++++++++++-
 rust/kernel/kunit.rs           |  1 +
 5 files changed, 68 insertions(+), 10 deletions(-)
Signature · 1 line
2.54.0
#1 lore raw

PATCH v4 1/2 kunit,rust: Add ability to skip entire test suites

Currently, KUnit provides mechanisms to skip individual test cases, but there is no way to skip an entire test suite based on runtime conditions checked during suite initialization. This limitation forces test suites to either fail or skip tests individually when certain prerequisites are not available.

To address this limitation, the patch adds a 'status' field to struct kunit_suite that allows suite_init callbacks to mark the entire suite as KUNIT_SKIPPED. When a suite is marked as skipped, all test cases within that suite are bypassed without execution.

The patch proposed changes to kunit_suite_has_succeeded() to Check suite status before evaluating individual test case results. Also kunit_run_tests() is updated to skip suite execution if kunit_suite's 'status' is KUNIT_SKIPPED, thats either set before suite_init or by the suite_init callback itself. kunit_init_suite() is updated to initialize the 'status' of kunit_suite to KUNIT_SUCCESS so that any skipped suite's can be restarted from debugfs.

This enables test suites to perform runtime capability checks in their 'suite_init' callback and gracefully skip all tests when prerequisites are not met, rather than reporting failures or requiring each test case to perform redundant checks. In case a kunit-suite is skipped it can be re-run from the kunit's debugfs interface.

Also update debugfs_print_results() to clearly log the kunit-suite as 'SKIP'. kunit_suite_has_succeeded() is also updated on which debugfs_print_results() depends to update 'kunit_suite.status' in case any of the kunit_case has failed.

Finally, update KUnit Rust binding macro-rule 'kunit_unsafe_test_suite' to add and initialize the newly introduced 'kunit_suite.status'. Without this 'kunit_suite.status' field is never initialized which is an error for the Rust compiler.

---
Changelog
=========

V3->V4:

* Merged rust kunit binding related changes with this patch [ David Gow ]
* Added David Gow's RB
* Fixed a indentation issue in rust changes [ David Gow ]

V2->V3:
Link:
https://lore.kernel.org/all/[email protected]

None

V1->V2:
Link:
https://lore.kernel.org/all/[email protected]/

* Fix malformed and missing test-log when skipping kunit-suite. [David Gow]
* Update kunit_init_suite() to reset the kunit-suite so that it can be re-run [David Gow]
* Update kunit_suite_has_succeeded() to check for any failed test-case and update the 'status' for kunit-suite.

---
 include/kunit/test.h |  1 +
 lib/kunit/debugfs.c  | 30 +++++++++++++++++++++---------
 lib/kunit/test.c     | 17 ++++++++++++++++-
 rust/kernel/kunit.rs |  1 +
 4 files changed, 39 insertions(+), 10 deletions(-)
include/kunit/test.h +1 -0
index e52452e58305..da5312e0dfa5 100644
@@ -285,6 +285,7 @@ struct kunit_suite {
 	struct string_stream *log;
 	int suite_init_err;
 	bool is_init;
+	enum kunit_status status;
 };
 
 /* Stores an array of suites, end points one past the end */
lib/kunit/debugfs.c +21 -9
index 9c326f1837bd..442b2ceb955b 100644
@@ -76,18 +76,30 @@ static int debugfs_print_results(struct seq_file *seq, void *v)
 	seq_puts(seq, "KTAP version 1\n");
 	seq_puts(seq, "1..1\n");
 
-	/* Print suite header because it is not stored in the test logs. */
-	seq_puts(seq, KUNIT_SUBTEST_INDENT "KTAP version 1\n");
-	seq_printf(seq, KUNIT_SUBTEST_INDENT "# Subtest: %s\n", suite->name);
-	seq_printf(seq, KUNIT_SUBTEST_INDENT "1..%zd\n", kunit_suite_num_test_cases(suite));
-
-	kunit_suite_for_each_test_case(suite, test_case)
-		debugfs_print_result(seq, test_case->log);
+	if (suite->status != KUNIT_SKIPPED) {
+		/* Print suite header because it is not stored in the test logs. */
+		seq_puts(seq,
+			 KUNIT_SUBTEST_INDENT "KTAP version 1\n");
+		seq_printf(seq,
+			   KUNIT_SUBTEST_INDENT "# Subtest: %s\n",
+			   suite->name);
+		seq_printf(seq,
+			   KUNIT_SUBTEST_INDENT "1..%zd\n",
+			   kunit_suite_num_test_cases(suite));
+
+		kunit_suite_for_each_test_case(suite, test_case)
+			debugfs_print_result(seq, test_case->log);
+	}
 
 	debugfs_print_result(seq, suite->log);
 
-	seq_printf(seq, "%s %d %s\n",
-		   kunit_status_to_ok_not_ok(success), 1, suite->name);
+	if (suite->status != KUNIT_SKIPPED)
+		seq_printf(seq, "%s %d %s\n",
+			   kunit_status_to_ok_not_ok(success), 1, suite->name);
+	else
+		seq_printf(seq, "%s %d %s # SKIP %s\n",
+			   kunit_status_to_ok_not_ok(success), 1, suite->name,
+			   suite->status_comment);
 	return 0;
 }
 
lib/kunit/test.c +16 -1
index 99773e000e1b..09e3dabfac0c 100644
@@ -214,12 +214,18 @@ enum kunit_status kunit_suite_has_succeeded(struct kunit_suite *suite)
 	const struct kunit_case *test_case;
 	enum kunit_status status = KUNIT_SKIPPED;
 
+	if (suite->status == KUNIT_SKIPPED)
+		return KUNIT_SKIPPED;
+
 	if (suite->suite_init_err)
 		return KUNIT_FAILURE;
 
 	kunit_suite_for_each_test_case(suite, test_case) {
-		if (test_case->status == KUNIT_FAILURE)
+		if (test_case->status == KUNIT_FAILURE) {
+			/* Update the kunit_suite status also */
+			suite->status = KUNIT_FAILURE;
 			return KUNIT_FAILURE;
+		}
 		else if (test_case->status == KUNIT_SUCCESS)
 			status = KUNIT_SUCCESS;
 	}
@@ -795,12 +801,20 @@ int kunit_run_tests(struct kunit_suite *suite)
 	/* Taint the kernel so we know we've run tests. */
 	add_taint(TAINT_TEST, LOCKDEP_STILL_OK);
 
+	if (suite->status == KUNIT_SKIPPED)
+		goto suite_end;
+
 	if (suite->suite_init) {
 		suite->suite_init_err = suite->suite_init(suite);
 		if (suite->suite_init_err) {
+			suite->status = KUNIT_FAILURE;
 			kunit_err(suite, KUNIT_SUBTEST_INDENT
 				  "# failed to initialize (%d)", suite->suite_init_err);
 			goto suite_end;
+
+		} else if (suite->status == KUNIT_SKIPPED) {
+			/* Skip this kunit suite */
+			goto suite_end;
 		}
 	}
 
@@ -825,6 +839,7 @@ static void kunit_init_suite(struct kunit_suite *suite)
 	kunit_debugfs_create_suite(suite);
 	suite->status_comment[0] = '\0';
 	suite->suite_init_err = 0;
+	suite->status = KUNIT_SUCCESS;
 
 	if (suite->log)
 		string_stream_clear(suite->log);
rust/kernel/kunit.rs +1 -0
index cdee5f27bd7f..91eaff8c186a 100644
@@ -288,6 +288,7 @@ macro_rules! kunit_unsafe_test_suite {
                     log: ::core::ptr::null_mut(),
                     suite_init_err: 0,
                     is_init: false,
+                    status: kernel::bindings::kunit_status_KUNIT_SUCCESS,
                 };
 
             #[used(compiler)]
Signature · 1 line
2.54.0
#2 lore raw

PATCH v4 2/2 kunit: Add example of test suite that can be skipped at runtime

In reply to Vaibhav Jain

Add an example test suite name 'example_test_skip_suite' to 'kunit-example-test.c' that shows how to skip an entire test suite based on runtime conditions.

The example suite 'example_skip_suite' provides a 'suite_init' callback named example_skip_suite_init() which marks the entire suite as skipped using kunit_mark_skipped().

This demonstrates a way for conditionally skipping test suites when any prerequisites for kunit_suite execution are not met. The 'suite_init' callback can perform any necessary checks and mark the suite as skipped, preventing all test cases from executing while also indicating why the suite was skipped.

---
Changelog
=========
V3->V4:
Link:
https://lore.kernel.org/all/[email protected]
* Updated patch index in the series
* Added RB from David Gow

V2->V3:
Link:
https://lore.kernel.org/all/[email protected]
* Updated patch index in the series

V1->V2:
Link:
https://lore.kernel.org/all/[email protected]/
* Added RB from 'David Gow'

---
 lib/kunit/kunit-example-test.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
lib/kunit/kunit-example-test.c +29 -0
index 0bae7b7ca0b0..b8ded54fa46d 100644
@@ -591,5 +591,34 @@ static struct kunit_suite example_init_test_suite = {
  */
 kunit_test_init_section_suites(&example_init_test_suite);
 
+/*
+ * This test should always be skipped.
+ */
+static void example_skip_suite_test(struct kunit *test)
+{
+	/* This line should never be seen */
+	KUNIT_FAIL(test, "You should not see a this.");
+}
+
+static struct kunit_case  example_skip_suite_test_cases[] = {
+	KUNIT_CASE(example_skip_suite_test),
+	{}
+};
+
+static int example_skip_suite_init(struct kunit_suite *suite)
+{
+	kunit_mark_skipped(suite, "Test suite expected to be skipped");
+	return 0;
+}
+
+static struct kunit_suite example_test_skip_suite = {
+	.name = "example_skip_suite",
+	.suite_init = example_skip_suite_init,
+	.test_cases = example_skip_suite_test_cases,
+};
+
+/* This registers a test suite that will be skipped */
+kunit_test_suite(example_test_skip_suite);
+
 MODULE_DESCRIPTION("Example KUnit test suite");
 MODULE_LICENSE("GPL v2");
Signature · 1 line
2.54.0