Handle faults in KUnit tests
Subsystems (7)
v4 cover Handle faults in KUnit tests
Hi,
This patch series teaches KUnit to handle kthread faults as errors, and it brings a few related fixes and improvements.
Shuah, everything should be OK now, could you please merge this series?
All these tests pass (on top of v6.8):./tools/testing/kunit/kunit.py run --alltests ./tools/testing/kunit/kunit.py run --alltests --arch x86_64 ./tools/testing/kunit/kunit.py run --alltests --arch arm64 \
--cross_compile=aarch64-linux-gnu-
I also built and ran KUnit tests as a kernel module.
A new test case check NULL pointer dereference, which wasn't possible before.
This is useful to test current kernel self-protection mechanisms or future ones such as Heki: https://github.com/heki-linux
Previous versions:
v3: https://lore.kernel.org/r/[email protected]
v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]
Regards,
Mickaël Salaün (7):
kunit: Handle thread creation error
kunit: Fix kthread reference
kunit: Fix timeout message
kunit: Handle test faults
kunit: Fix KUNIT_SUCCESS() calls in iov_iter tests
kunit: Print last test location on fault
kunit: Add tests for fault
include/kunit/test.h | 24 ++++++++++++++++++--- include/kunit/try-catch.h | 3 --- kernel/kthread.c | 1 + lib/kunit/kunit-test.c | 45 ++++++++++++++++++++++++++++++++++++++- lib/kunit/try-catch.c | 38 ++++++++++++++++++++++----------- lib/kunit_iov_iter.c | 18 ++++++++-------- 6 files changed, 101 insertions(+), 28 deletions(-)
base-commit: e8f897f4afef0031fe618a8e94127a0934896aba
Signature · 1 line
2.44.0
PATCH v4 1/7 kunit: Handle thread creation error
Previously, if a thread creation failed (e.g. -ENOMEM), the function was called (kunit_catch_run_case or kunit_catch_run_case_cleanup) without marking the test as failed. Instead, fill try_result with the error code returned by kthread_run(), which will mark the test as failed and print "internal error occurred...".
---
Changes since v2:
* Add Rae's and David's Reviewed-by.
Changes since v1:
* Add Kees's Reviewed-by.
--- lib/kunit/try-catch.c | 1 + 1 file changed, 1 insertion(+)
lib/kunit/try-catch.c +1 -0
@@ -69,6 +69,7 @@ void kunit_try_catch_run(struct kunit_try_catch *try_catch, void *context) try_catch, "kunit_try_catch_thread");if (IS_ERR (task_struct)) { + try_catch->try_result =PTR_ERR (task_struct); try_catch->catch (try_catch->context);return ; }
Signature · 1 line
2.44.0
PATCH v4 2/7 kunit: Fix kthread reference
In reply to Mickaël SalaünThere is a race condition when a kthread finishes after the deadline and before the call to kthread_stop(), which may lead to use after free.
---
Changes since v2:
* Add Fixes tag as suggested by David.
* Add David's and Rae's Reviewed-by.
Changes since v1:
* Add Kees's Reviewed-by.
--- lib/kunit/try-catch.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
lib/kunit/try-catch.c +6 -3
@@ -11,6 +11,7 @@#include <linux/completion.h> #include <linux/kernel.h> #include <linux/kthread.h> +#include <linux/sched/task.h> #include "try-catch-impl.h"
@@ -65,14 +66,15 @@ void kunit_try_catch_run(struct kunit_try_catch *try_catch, void *context) try_catch->context = context; try_catch->try_completion = &try_completion; try_catch->try_result =0 ; - task_struct =kthread_run (kunit_generic_run_threadfn_adapter, - try_catch, -"kunit_try_catch_thread" ); + task_struct =kthread_create (kunit_generic_run_threadfn_adapter, + try_catch,"kunit_try_catch_thread" );if (IS_ERR (task_struct)) { try_catch->try_result =PTR_ERR (task_struct); try_catch->catch (try_catch->context);return ; } +get_task_struct (task_struct); +wake_up_process (task_struct); time_remaining =wait_for_completion_timeout (&try_completion,kunit_test_timeout ());
@@ -82,6 +84,7 @@ void kunit_try_catch_run(struct kunit_try_catch *try_catch, void *context)kthread_stop (task_struct); } +put_task_struct (task_struct); exit_code = try_catch->try_result;if (!exit_code)
Signature · 1 line
2.44.0
PATCH v4 3/7 kunit: Fix timeout message
In reply to Mickaël SalaünThe exit code is always checked, so let's properly handle the -ETIMEDOUT error code.
---
Changes since v2:
* Add Rae's and David's Reviewed-by.
Changes since v1:
* Add Kees's Reviewed-by.
--- lib/kunit/try-catch.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
lib/kunit/try-catch.c +2 -1
@@ -79,7 +79,6 @@ void kunit_try_catch_run(struct kunit_try_catch *try_catch, void *context) time_remaining =wait_for_completion_timeout (&try_completion,kunit_test_timeout ());if (time_remaining ==0 ) { -kunit_err (test,"try timed out\n" ); try_catch->try_result = -ETIMEDOUT ;kthread_stop (task_struct); }
@@ -94,6 +93,8 @@ void kunit_try_catch_run(struct kunit_try_catch *try_catch, void *context) try_catch->try_result =0 ; else if (exit_code == -EINTR) kunit_err(test, "wake_up_process() was never called\n"); + else if (exit_code == -ETIMEDOUT) + kunit_err(test, "try timed out\n"); else if (exit_code) kunit_err(test, "Unknown error: %d\n", exit_code);
Signature · 1 line
2.44.0
PATCH v4 4/7 kunit: Handle test faults
In reply to Mickaël SalaünPreviously, when a kernel test thread crashed (e.g. NULL pointer dereference, general protection fault), the KUnit test hanged for 30 seconds and exited with a timeout error.
Fix this issue by waiting on task_struct->vfork_done instead of the custom kunit_try_catch.try_completion, and track the execution state by initially setting try_result with -EINTR and only setting it to 0 if the test passed.
Fix kunit_generic_run_threadfn_adapter() signature by returning 0 instead of calling kthread_complete_and_exit(). Because thread's exit code is never checked, always set it to 0 to make it clear. To make this explicit, export kthread_exit() for KUnit tests built as module.
Fix the -EINTR error message, which couldn't be reached until now.
This is tested with a following patch.
---
Changes since v3:
* Export kthread_exit() for KUnit tests built as module, as suggested by David.
Changes since v2:
* s/-EFAULT/-EINTR/ in commit message as spotted by Rae.
* Add a comment explaining vfork_done as suggested by David.
* Add David's Reviewed-by.
* Add Rae's Tested-by.
Changes since v1:
* Add Kees's Reviewed-by.
--- include/kunit/try-catch.h | 3 --- kernel/kthread.c | 1 + lib/kunit/try-catch.c | 19 ++++++++++++------- 3 files changed, 13 insertions(+), 10 deletions(-)
include/kunit/try-catch.h +0 -3
@@ -14,13 +14,11 @@typedef void (*kunit_try_catch_func_t )(void *); -struct completion ;struct kunit ;/** * struct kunit_try_catch - provides a generic way to run code which might fail. * @test: The test case that is currently being executed. -* @try_completion: Completion that the control thread waits on while test runs. * @try_result: Contains any errno obtained while running test case. * @try: The function, the test case, to attempt to run. * @catch: The function called if @try bails out.
@@ -46,7 +44,6 @@ struct kunit;struct kunit_try_catch {/* private: internal use only. */ struct kunit *test; -struct completion *try_completion;int try_result;kunit_try_catch_func_t try;kunit_try_catch_func_t catch;
kernel/kthread.c +1 -0
@@ -315,6 +315,7 @@ void __noreturn kthread_exit(long result) kthread->result = result;do_exit (0 ); } +EXPORT_SYMBOL (kthread_exit);/** * kthread_complete_and_exit - Exit the current kthread.
lib/kunit/try-catch.c +12 -7
@@ -18,7 +18,7 @@void __noreturn kunit_try_catch_throw (struct kunit_try_catch *try_catch) { try_catch->try_result = -EFAULT ; -kthread_complete_and_exit (try_catch->try_completion, -EFAULT ); +kthread_exit (0 ); }EXPORT_SYMBOL_GPL (kunit_try_catch_throw);
@@ -26,9 +26,12 @@ static int kunit_generic_run_threadfn_adapter(void *data) {struct kunit_try_catch *try_catch = data; + try_catch->try_result = -EINTR ; try_catch->try (try_catch->context); +if (try_catch->try_result == -EINTR ) + try_catch->try_result =0 ; -kthread_complete_and_exit (try_catch->try_completion,0 ); +return 0 ; }static unsigned long kunit_test_timeout (void )
@@ -58,13 +61,11 @@ static unsigned long kunit_test_timeout(void)void kunit_try_catch_run (struct kunit_try_catch *try_catch,void *context) { -DECLARE_COMPLETION_ONSTACK (try_completion);struct kunit *test = try_catch->test;struct task_struct *task_struct;int exit_code, time_remaining; try_catch->context = context; - try_catch->try_completion = &try_completion; try_catch->try_result =0 ; task_struct =kthread_create (kunit_generic_run_threadfn_adapter, try_catch,"kunit_try_catch_thread" );
@@ -75,8 +76,12 @@ void kunit_try_catch_run(struct kunit_try_catch *try_catch, void *context) }get_task_struct (task_struct);wake_up_process (task_struct); - - time_remaining =wait_for_completion_timeout (&try_completion, +/* +* As for a vfork(2), task_struct->vfork_done (pointing to the +* underlying kthread->exited) can be used to wait for the end of a +* kernel thread. +*/ + time_remaining =wait_for_completion_timeout (task_struct->vfork_done,kunit_test_timeout ());if (time_remaining ==0 ) { try_catch->try_result = -ETIMEDOUT ;
@@ -92,7 +97,7 @@ void kunit_try_catch_run(struct kunit_try_catch *try_catch, void *context)if (exit_code == -EFAULT ) try_catch->try_result =0 ;else if (exit_code == -EINTR ) -kunit_err (test,"wake_up_process() was never called\n" ); +kunit_err (test,"try faulted\n" );else if (exit_code == -ETIMEDOUT )kunit_err (test,"try timed out\n" );else if (exit_code)
Signature · 1 line
2.44.0
PATCH v4 5/7 kunit: Fix KUNIT_SUCCESS() calls in iov_iter tests
In reply to Mickaël SalaünFix KUNIT_SUCCESS() calls to pass a test argument.
This is a no-op for now because this macro does nothing, but it will be required for the next commit.
---
Changes since v2:
* Add David's Reviewed-by.
Changes since v1:
* Add Kees's Reviewed-by.
--- lib/kunit_iov_iter.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
lib/kunit_iov_iter.c +9 -9
@@ -139,7 +139,7 @@ static void __init iov_kunit_copy_to_kvec(struct kunit *test)return ; } -KUNIT_SUCCEED (); +KUNIT_SUCCEED (test); }/*
@@ -194,7 +194,7 @@ static void __init iov_kunit_copy_from_kvec(struct kunit *test)return ; } -KUNIT_SUCCEED (); +KUNIT_SUCCEED (test); }struct bvec_test_range {
@@ -302,7 +302,7 @@ static void __init iov_kunit_copy_to_bvec(struct kunit *test)return ; } -KUNIT_SUCCEED (); +KUNIT_SUCCEED (test); }/*
@@ -359,7 +359,7 @@ static void __init iov_kunit_copy_from_bvec(struct kunit *test)return ; } -KUNIT_SUCCEED (); +KUNIT_SUCCEED (test); }static void iov_kunit_destroy_xarray (void *data)
@@ -453,7 +453,7 @@ static void __init iov_kunit_copy_to_xarray(struct kunit *test)return ; } -KUNIT_SUCCEED (); +KUNIT_SUCCEED (test); }/*
@@ -516,7 +516,7 @@ static void __init iov_kunit_copy_from_xarray(struct kunit *test)return ; } -KUNIT_SUCCEED (); +KUNIT_SUCCEED (test); }/*
@@ -596,7 +596,7 @@ static void __init iov_kunit_extract_pages_kvec(struct kunit *test) stop:KUNIT_EXPECT_EQ (test, size,0 );KUNIT_EXPECT_EQ (test, iter.count,0 ); -KUNIT_SUCCEED (); +KUNIT_SUCCEED (test); }/*
@@ -674,7 +674,7 @@ static void __init iov_kunit_extract_pages_bvec(struct kunit *test) stop:KUNIT_EXPECT_EQ (test, size,0 );KUNIT_EXPECT_EQ (test, iter.count,0 ); -KUNIT_SUCCEED (); +KUNIT_SUCCEED (test); }/*
@@ -753,7 +753,7 @@ static void __init iov_kunit_extract_pages_xarray(struct kunit *test) } stop: -KUNIT_SUCCEED (); +KUNIT_SUCCEED (test); }static struct kunit_case __refdata iov_kunit_cases[] = {
Signature · 1 line
2.44.0
PATCH v4 6/7 kunit: Print last test location on fault
In reply to Mickaël SalaünThis helps identify the location of test faults with opportunistic calls to _KUNIT_SAVE_LOC(). This can be useful while writing tests or debugging them. It is possible to call KUNIT_SUCCESS() to explicit save last location.
---
Changes since v3:
* Improve the try-fault error message as suggested by David.
Changes since v2:
* Extend the commit message according to discussion with David.
Changes since v1:
* Add Kees's Reviewed-by.
--- include/kunit/test.h | 24 +++++++++++++++++++++--- lib/kunit/try-catch.c | 10 +++++++--- 2 files changed, 28 insertions(+), 6 deletions(-)
include/kunit/test.h +21 -3
@@ -301,6 +301,8 @@ struct kunit {struct list_head resources;/* Protected by lock. */ char status_comment[KUNIT_STATUS_COMMENT_SIZE ]; +/* Saves the last seen test. Useful to help with faults. */ +struct kunit_loc last_seen; };static inline void kunit_set_failure (struct kunit *test)
@@ -567,6 +569,15 @@ void __printf(2, 3) kunit_log_append(struct string_stream *log, const char *fmt,#define kunit_err (test, fmt, ...) \ kunit_printk(KERN_ERR, test, fmt, ##__VA_ARGS__) +/* +* Must be called at the beginning of each KUNIT_*_ASSERTION(). +* Cf. KUNIT_CURRENT_LOC. +*/ +#define _KUNIT_SAVE_LOC (test) do { \ + WRITE_ONCE(test->last_seen.file, __FILE__); \ + WRITE_ONCE(test->last_seen.line, __LINE__); \ +} while (0) +/** * KUNIT_SUCCEED() - A no-op expectation. Only exists for code clarity. * @test: The test context object.
@@ -575,7 +586,7 @@ void __printf(2, 3) kunit_log_append(struct string_stream *log, const char *fmt,* words, it does nothing and only exists for code clarity. See * KUNIT_EXPECT_TRUE() for more information. */ -#define KUNIT_SUCCEED (test) do {} while (0) +#define KUNIT_SUCCEED (test) _KUNIT_SAVE_LOC(test)void __noreturn __kunit_abort (struct kunit *test);
@@ -601,14 +612,16 @@ void __kunit_do_failed_assertion(struct kunit *test, } while (0) -#define KUNIT_FAIL_ASSERTION (test, assert_type, fmt, ...) \ +#define KUNIT_FAIL_ASSERTION (test, assert_type, fmt, ...) do { \ + _KUNIT_SAVE_LOC(test); \ _KUNIT_FAILED(test, \ assert_type, \ kunit_fail_assert, \ kunit_fail_assert_format, \ {}, \ fmt, \ - ##__VA_ARGS__) + ##__VA_ARGS__); \ +} while (0)/** * KUNIT_FAIL() - Always causes a test to fail when evaluated.
@@ -637,6 +650,7 @@ void __kunit_do_failed_assertion(struct kunit *test, fmt, \ ...) \ do { \ +_KUNIT_SAVE_LOC (test); \if (likely (!!(condition_) == !!expected_true_)) \break ; \ \
@@ -698,6 +712,7 @@ do { \ .right_text = #right, \ }; \ \ + _KUNIT_SAVE_LOC(test); \ if (likely(__left op __right)) \ break; \ \
@@ -758,6 +773,7 @@ do { \ .right_text = #right, \ }; \ \ + _KUNIT_SAVE_LOC(test); \ if (likely((__left) && (__right) && (strcmp(__left, __right) op 0))) \ break; \ \
@@ -791,6 +807,7 @@ do { \ .right_text = #right, \ }; \ \ + _KUNIT_SAVE_LOC(test); \ if (likely(__left && __right)) \ if (likely(memcmp(__left, __right, __size) op 0)) \ break; \
@@ -815,6 +832,7 @@ do { \ do { \ const typeof(ptr) __ptr = (ptr); \ \ + _KUNIT_SAVE_LOC(test); \ if (!IS_ERR_OR_NULL(__ptr)) \ break; \ \
lib/kunit/try-catch.c +7 -3
@@ -96,9 +96,13 @@ void kunit_try_catch_run(struct kunit_try_catch *try_catch, void *context)if (exit_code == -EFAULT ) try_catch->try_result =0 ; -else if (exit_code == -EINTR ) -kunit_err (test,"try faulted\n" ); -else if (exit_code == -ETIMEDOUT ) +else if (exit_code == -EINTR ) { +if (test->last_seen.file) +kunit_err (test,"try faulted: last line seen %s:%d\n" , + test->last_seen.file, test->last_seen.line); +else +kunit_err (test,"try faulted\n" ); + }else if (exit_code == -ETIMEDOUT )kunit_err (test,"try timed out\n" );else if (exit_code)kunit_err (test,"Unknown error: %d\n" , exit_code);
Signature · 1 line
2.44.0
PATCH v4 7/7 kunit: Add tests for fault
In reply to Mickaël SalaünAdd a test case to check NULL pointer dereference and make sure it would result as a failed test.
The full kunit_fault test suite is marked as skipped when run on UML because it would result to a kernel panic.
Tested with:./tools/testing/kunit/kunit.py run --arch x86_64 kunit_fault ./tools/testing/kunit/kunit.py run --arch arm64 \
--cross_compile=aarch64-linux-gnu- kunit_fault
---
Changes since v2:
* Add David's Reviewed-by.
Changes since v1:
* Remove the rodata and const test cases for now.
* Replace CONFIG_X86 check with !CONFIG_UML, and remove the "_x86" references.
--- lib/kunit/kunit-test.c | 45 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-)
lib/kunit/kunit-test.c +44 -1
@@ -109,6 +109,48 @@ static struct kunit_suite kunit_try_catch_test_suite = { .test_cases = kunit_try_catch_test_cases, }; +#ifndef CONFIG_UML + +static void kunit_test_null_dereference (void *data) +{ +struct kunit *test = data; +int *null =NULL ; + + *null =0 ; + +KUNIT_FAIL (test,"This line should never be reached\n" ); +} + +static void kunit_test_fault_null_dereference (struct kunit *test) +{ +struct kunit_try_catch_test_context *ctx = test->priv; +struct kunit_try_catch *try_catch = ctx->try_catch; + +kunit_try_catch_init (try_catch, + test, + kunit_test_null_dereference, + kunit_test_catch); +kunit_try_catch_run (try_catch, test); + +KUNIT_EXPECT_EQ (test, try_catch->try_result, -EINTR ); +KUNIT_EXPECT_TRUE (test, ctx->function_called); +} + +#endif /* !CONFIG_UML */ + +static struct kunit_case kunit_fault_test_cases[] = { +#ifndef CONFIG_UML + KUNIT_CASE(kunit_test_fault_null_dereference), +#endif /* !CONFIG_UML */ + {} +}; + +static struct kunit_suite kunit_fault_test_suite = { + .name ="kunit_fault" , + .init = kunit_try_catch_test_init, + .test_cases = kunit_fault_test_cases, +}; +/* * Context for testing test managed resources * is_resource_initialized is used to test arbitrary resources
@@ -826,6 +868,7 @@ static struct kunit_suite kunit_current_test_suite = {kunit_test_suites (&kunit_try_catch_test_suite, &kunit_resource_test_suite, &kunit_log_test_suite, &kunit_status_test_suite, - &kunit_current_test_suite, &kunit_device_test_suite); + &kunit_current_test_suite, &kunit_device_test_suite, + &kunit_fault_test_suite);MODULE_LICENSE ("GPL v2" );
Signature · 1 line
2.44.0
Quoted 6 lines · Mickaël Salaün
Hi,
This patch series teaches KUnit to handle kthread faults as errors, and it brings a few related fixes and improvements.
Shuah, everything should be OK now, could you please merge this series?
Please cc linux-kselftest and kunit mailing lists. You got the world cc'ed except for the important ones. :)
thanks,
-- Shuah
Quoted 10 lines · Shuah Khan
Quoted 6 lines · Mickaël Salaün
Hi,
This patch series teaches KUnit to handle kthread faults as errors, and it brings a few related fixes and improvements.
Shuah, everything should be OK now, could you please merge this series?
Please cc linux-kselftest and kunit mailing lists. You got the world cc'ed except for the important ones. :)
Indeed, I don't know how I missed that... Do you want me to send it again?
Quoted 4 lines
thanks,
-- Shuah
Quoted 7 lines · Mickaël Salaün
Quoted 6 lines · Shuah Khan
Quoted 5 lines · Mickaël Salaün
Hi,
This patch series teaches KUnit to handle kthread faults as errors, and it brings a few related fixes and improvements.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Shuah, everything should be OK now, could you please merge this series?
Please cc linux-kselftest and kunit mailing lists. You got the world cc'ed except for the important ones. :)
Indeed, I don't know how I missed that... Do you want me to send it again?
Yes please resend. I will pull these right away.
thanks,
-- Shuah