Skip to contentSkip to conversation

kunit: Fix result propagation for parameterised tests

PATCH In review David Gow 4 participants 6 messages Jun 11, 2021

Subsystems (2)
lore
Share & tools
lore

David Gow

author
#0 lore raw

PATCH 1/1 kunit: Fix result propagation for parameterised tests

When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).

This is because test_case->success was being reset to the test->success result after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable, test_result, but this was disacarded after the status line was printed.

Instead, just propagate the result after each parameter run.

Fixes fadb08e7c750 ("kunit: Support for Parameterized Testing")

---

This is fixing quite a serious bug where some test suites would appear to succeed even if some of their component tests failed. It'd be nice to get this into kunit-fixes ASAP.

(This will require a rework of some of the skip tests work, for which I'll send out a new version soon.)

Cheers,
-- David

 lib/kunit/test.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
lib/kunit/test.c +3 -4
index 2f6cc0123232..17973a4a44c2 100644
@@ -376,7 +376,7 @@ static void kunit_run_case_catch_errors(struct kunit_suite *suite,
 	context.test_case = test_case;
 	kunit_try_catch_run(try_catch, &context);
 
-	test_case->success = test->success;
+	test_case->success &= test->success;
 }
 
 int kunit_run_tests(struct kunit_suite *suite)
@@ -388,7 +388,7 @@ int kunit_run_tests(struct kunit_suite *suite)
 
 	kunit_suite_for_each_test_case(suite, test_case) {
 		struct kunit test = { .param_value = NULL, .param_index = 0 };
-		bool test_success = true;
+		test_case->success = true;
 
 		if (test_case->generate_params) {
 			/* Get initial param. */
@@ -398,7 +398,6 @@ int kunit_run_tests(struct kunit_suite *suite)
 
 		do {
 			kunit_run_case_catch_errors(suite, test_case, &test);
-			test_success &= test_case->success;
 
 			if (test_case->generate_params) {
 				if (param_desc[0] == '\0') {
@@ -420,7 +419,7 @@ int kunit_run_tests(struct kunit_suite *suite)
 			}
 		} while (test.param_value);
 
-		kunit_print_ok_not_ok(&test, true, test_success,
+		kunit_print_ok_not_ok(&test, true, test_case->success,
 				      kunit_test_case_num(suite, test_case),
 				      test_case->name);
 	}
Signature · 1 line
2.32.0.272.g935e593368-goog
Reviewed-by on v1 1/1
#1 lore raw
Quoted 11 lines · David Gow

When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).

This is because test_case->success was being reset to the test->success result after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable, test_result, but this was disacarded after the status line was printed.

Quoted 4 lines

Instead, just propagate the result after each parameter run.

Fixes fadb08e7c750 ("kunit: Support for Parameterized Testing")

Would Cc: stable be appropriate?

Thanks,
-- Marco

Quoted 57 lines

---

This is fixing quite a serious bug where some test suites would appear to succeed even if some of their component tests failed. It'd be nice to get this into kunit-fixes ASAP.

(This will require a rework of some of the skip tests work, for which I'll send out a new version soon.)

Cheers,
-- David

 lib/kunit/test.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
lib/kunit/test.c +3 -5
index 2f6cc0123232..17973a4a44c2 100644
@@ -376,7 +376,7 @@ static void kunit_run_case_catch_errors(struct kunit_suite *suite,
        context.test_case = test_case;
        kunit_try_catch_run(try_catch, &context);
 
-       test_case->success = test->success;
+       test_case->success &= test->success;
 }
 
 int kunit_run_tests(struct kunit_suite *suite)
@@ -388,7 +388,7 @@ int kunit_run_tests(struct kunit_suite *suite)
 
        kunit_suite_for_each_test_case(suite, test_case) {
                struct kunit test = { .param_value = NULL, .param_index = 0 };
-               bool test_success = true;
+               test_case->success = true;
 
                if (test_case->generate_params) {
                        /* Get initial param. */
@@ -398,7 +398,6 @@ int kunit_run_tests(struct kunit_suite *suite)
 
                do {
                        kunit_run_case_catch_errors(suite, test_case, &test);
-                       test_success &= test_case->success;
 
                        if (test_case->generate_params) {
                                if (param_desc[0] == '\0') {
@@ -420,7 +419,7 @@ int kunit_run_tests(struct kunit_suite *suite)
                        }
                } while (test.param_value);
 
-               kunit_print_ok_not_ok(&test, true, test_success,
+               kunit_print_ok_not_ok(&test, true, test_case->success,
                                      kunit_test_case_num(suite, test_case),
                                      test_case->name);
        }
--

2.32.0.272.g935e593368-goog

maintainer
#2 lore raw
Quoted 17 lines · Marco Elver
Quoted 11 lines · David Gow

When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).

This is because test_case->success was being reset to the test->success result after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable, test_result, but this was disacarded after the status line was printed.

Quoted 4 lines

Instead, just propagate the result after each parameter run.

Fixes fadb08e7c750 ("kunit: Support for Parameterized Testing")
Quoted 13 lines

Would Cc: stable be appropriate?

Thanks,
-- Marco

Quoted 6 lines

---

This is fixing quite a serious bug where some test suites would appear to succeed even if some of their component tests failed. It'd be nice to get this into kunit-fixes ASAP.

Will apply this with cc stable.

Quoted 3 lines
Quoted 3 lines

(This will require a rework of some of the skip tests work, for which I'll send out a new version soon.)

Thanks for the heads up. I will wait for new version.

thanks,
-- Shuah

author
#3 lore raw
Quoted 26 lines · Shuah Khan
Quoted 24 lines · Marco Elver
Quoted 11 lines · David Gow

When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).

This is because test_case->success was being reset to the test->success result after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable, test_result, but this was disacarded after the status line was printed.

Quoted 4 lines

Instead, just propagate the result after each parameter run.

Fixes fadb08e7c750 ("kunit: Support for Parameterized Testing")

Would Cc: stable be appropriate?

Thanks,
-- Marco

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines

---

This is fixing quite a serious bug where some test suites would appear to succeed even if some of their component tests failed. It'd be nice to get this into kunit-fixes ASAP.

Will apply this with cc stable.

Thanks!

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

(This will require a rework of some of the skip tests work, for which I'll send out a new version soon.)

Thanks for the heads up. I will wait for new version.

Thanks: I've sent out v4 which fixes this:
https://lore.kernel.org/linux-kselftest/[email protected]/

It's rebased on top of this patch, so depends on it, and also depends on the first two patches in the "Do not typecheck binary assertions" series:
https://lore.kernel.org/linux-kselftest/[email protected]/

Quoted 2 lines

thanks,
-- Shuah

Reviewed-by on v1 1/1
#4 lore raw
In reply to David Gow
Quoted 10 lines · David Gow

When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).

This is because test_case->success was being reset to the test->success result after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable, test_result, but this was disacarded after the status line was printed.

nit: s/disacarded/discarded/g

Quoted 4 lines

Instead, just propagate the result after each parameter run.

Fixes fadb08e7c750 ("kunit: Support for Parameterized Testing")

I tried to reproduce the problem described and was unable to. Anyway, from the code it definitely looks like there is a bug like you describe. And it definitely looks like your change should fix it.

Anyway, I tried testing your fix, but given I was unable to reproduce the failure, I am not super confident in my testing. Still,

author
#5 lore raw
Quoted 15 lines · Brendan Higgins
Quoted 10 lines · David Gow

When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).

This is because test_case->success was being reset to the test->success result after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable, test_result, but this was disacarded after the status line was printed.

nit: s/disacarded/discarded/g

Quoted 8 lines
Quoted 4 lines

Instead, just propagate the result after each parameter run.

Fixes fadb08e7c750 ("kunit: Support for Parameterized Testing")

I tried to reproduce the problem described and was unable to. Anyway, from the code it definitely looks like there is a bug like you describe. And it definitely looks like your change should fix it.

I was able to reproduce this again myself. Note that the kunit_tool wrapper does its own result propagation which doesn't have a similar bug, so you won't see the issue in its parsed results. (Using the --raw_output flag does show it, though).

Here's the output from a patched UUID suite, which deliberately fails the first parameter of the first two tests and passes the other parameters and tests, which exhibits the issue:

TAP version 14
1..1
# Subtest: uuid
1..4
# uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57 Expected uuid_parse(data->uuid, &be) == 0, but

       uuid_parse(data->uuid, &be) == -22

failed to parse 'c33fx4995-3701-450e-9fbf-206a2e98e576'
# uuid_correct_be: not ok 1 - c33fx4995-3701-450e-9fbf-206a2e98e576 # uuid_correct_be: ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b # uuid_correct_be: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84 not ok 1 - uuid_correct_be
# uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46 Expected guid_parse(data->uuid, &le) == 0, but

       guid_parse(data->uuid, &le) == -22

failed to parse 'c33fx4995-3701-450e-9fbf-206a2e98e576'
# uuid_correct_le: not ok 1 - c33fx4995-3701-450e-9fbf-206a2e98e576 # uuid_correct_le: ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b # uuid_correct_le: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84 not ok 2 - uuid_correct_le
# uuid_wrong_be: ok 1 - c33f4995-3701-450e-9fbf206a2e98e576 # uuid_wrong_be: ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b # uuid_wrong_be: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e ok 3 - uuid_wrong_be
# uuid_wrong_le: ok 1 - c33f4995-3701-450e-9fbf206a2e98e576 # uuid_wrong_le: ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b # uuid_wrong_le: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e ok 4 - uuid_wrong_le
ok 1 - uuid

Note the "not ok 1 - uuid_correct_be" line, yet it ending in "ok 1 - uuid".

Quoted 5 lines

Anyway, I tried testing your fix, but given I was unable to reproduce the failure, I am not super confident in my testing. Still,