kunit: Fix result propagation for parameterised tests
PATCH 1/1 kunit: Fix result propagation for parameterised tests
When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).
This is because test_case->success was being reset to the test->success result after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable, test_result, but this was disacarded after the status line was printed.
Instead, just propagate the result after each parameter run.
---
This is fixing quite a serious bug where some test suites would appear to succeed even if some of their component tests failed. It'd be nice to get this into kunit-fixes ASAP.
(This will require a rework of some of the skip tests work, for which I'll send out a new version soon.)
Cheers,
-- David
lib/kunit/test.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
lib/kunit/test.c +3 -4
@@ -376,7 +376,7 @@ static void kunit_run_case_catch_errors(struct kunit_suite *suite, context.test_case = test_case;kunit_try_catch_run (try_catch, &context); - test_case->success = test->success; + test_case->success &= test->success; }int kunit_run_tests (struct kunit_suite *suite)
@@ -388,7 +388,7 @@ int kunit_run_tests(struct kunit_suite *suite)kunit_suite_for_each_test_case (suite, test_case) {struct kunit test = { .param_value =NULL , .param_index =0 }; -bool test_success = true; + test_case->success = true;if (test_case->generate_params) {/* Get initial param. */
@@ -398,7 +398,6 @@ int kunit_run_tests(struct kunit_suite *suite) do { kunit_run_case_catch_errors(suite, test_case, &test); - test_success &= test_case->success; if (test_case->generate_params) { if (param_desc[0] == '\0') {
@@ -420,7 +419,7 @@ int kunit_run_tests(struct kunit_suite *suite) } }while (test.param_value); -kunit_print_ok_not_ok (&test, true, test_success, +kunit_print_ok_not_ok (&test, true, test_case->success,kunit_test_case_num (suite, test_case), test_case->name); }
Signature · 1 line
2.32.0.272.g935e593368-goog
Quoted 11 lines · David Gow
When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).
This is because
test_case->successwas being reset to thetest->successresult after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable,test_result, but this was disacarded after the status line was printed.
Quoted 4 lines
Instead, just propagate the result after each parameter run.
Signed-off-by David Gow <[email protected]>Fixes fadb08e7c750 ("kunit: Support for Parameterized Testing")
Would Cc: stable be appropriate?
Thanks,
-- Marco
Quoted 57 lines
---
This is fixing quite a serious bug where some test suites would appear to succeed even if some of their component tests failed. It'd be nice to get this into kunit-fixes ASAP.
(This will require a rework of some of the skip tests work, for which I'll send out a new version soon.)
Cheers,
-- Davidlib/kunit/test.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)lib/kunit/test.c +3
-5@@ -376,7 +376,7 @@ static void kunit_run_case_catch_errors(struct kunit_suite *suite, context.test_case = test_case;kunit_try_catch_run (try_catch, &context); - test_case->success = test->success; + test_case->success &= test->success; }int kunit_run_tests (struct kunit_suite *suite)@@ -388,7 +388,7 @@ int kunit_run_tests(struct kunit_suite *suite)kunit_suite_for_each_test_case (suite, test_case) {struct kunit test = { .param_value =NULL , .param_index =0 }; -bool test_success = true; + test_case->success = true;if (test_case->generate_params) {/* Get initial param. */ @@ -398,7 +398,6 @@ int kunit_run_tests(struct kunit_suite *suite) do { kunit_run_case_catch_errors(suite, test_case, &test); - test_success &= test_case->success; if (test_case->generate_params) { if (param_desc[0] == '\0') {@@ -420,7 +419,7 @@ int kunit_run_tests(struct kunit_suite *suite) } }while (test.param_value); -kunit_print_ok_not_ok (&test, true, test_success, +kunit_print_ok_not_ok (&test, true, test_case->success,kunit_test_case_num (suite, test_case), test_case->name); } --2.32.0.272.g935e593368-goog
Shuah Khan
maintainerQuoted 17 lines · Marco Elver
Quoted 11 lines · David Gow
When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).
This is because
test_case->successwas being reset to thetest->successresult after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable,test_result, but this was disacarded after the status line was printed.Quoted 4 lines
Instead, just propagate the result after each parameter run.
Signed-off-by David Gow <[email protected]>Fixes fadb08e7c750 ("kunit: Support for Parameterized Testing")
Quoted 13 lines
Reviewed-by Marco Elver <[email protected]>Would Cc: stable be appropriate?
Thanks,
-- MarcoQuoted 6 lines
---
This is fixing quite a serious bug where some test suites would appear to succeed even if some of their component tests failed. It'd be nice to get this into kunit-fixes ASAP.
Will apply this with cc stable.
Quoted 3 lines
Quoted 3 lines
(This will require a rework of some of the skip tests work, for which I'll send out a new version soon.)
Thanks for the heads up. I will wait for new version.
thanks,
-- Shuah
Quoted 26 lines · Shuah Khan
Quoted 24 lines · Marco Elver
Quoted 11 lines · David Gow
When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).
This is because
test_case->successwas being reset to thetest->successresult after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable,test_result, but this was disacarded after the status line was printed.Quoted 4 lines
Instead, just propagate the result after each parameter run.
Signed-off-by David Gow <[email protected]>Fixes fadb08e7c750 ("kunit: Support for Parameterized Testing")Reviewed-by Marco Elver <[email protected]>Would Cc: stable be appropriate?
Thanks,
-- Marco
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
---
This is fixing quite a serious bug where some test suites would appear to succeed even if some of their component tests failed. It'd be nice to get this into kunit-fixes ASAP.
Will apply this with cc stable.
Thanks!
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
(This will require a rework of some of the skip tests work, for which I'll send out a new version soon.)
Thanks for the heads up. I will wait for new version.
Thanks: I've sent out v4 which fixes this:
https://lore.kernel.org/linux-kselftest/[email protected]/
It's rebased on top of this patch, so depends on it, and also depends on the first two patches in the "Do not typecheck binary assertions" series:
https://lore.kernel.org/linux-kselftest/[email protected]/
Quoted 2 lines
thanks,
-- Shuah
Quoted 10 lines · David Gow
When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).
This is because
test_case->successwas being reset to thetest->successresult after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable,test_result, but this was disacarded after the status line was printed.
nit: s/disacarded/discarded/g
Quoted 4 lines
Instead, just propagate the result after each parameter run.
Signed-off-by David Gow <[email protected]>Fixes fadb08e7c750 ("kunit: Support for Parameterized Testing")
I tried to reproduce the problem described and was unable to. Anyway, from the code it definitely looks like there is a bug like you describe. And it definitely looks like your change should fix it.
Anyway, I tried testing your fix, but given I was unable to reproduce the failure, I am not super confident in my testing. Still,
Quoted 15 lines · Brendan Higgins
Quoted 10 lines · David Gow
When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).
This is because
test_case->successwas being reset to thetest->successresult after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable,test_result, but this was disacarded after the status line was printed.nit: s/disacarded/discarded/g
Quoted 8 lines
Quoted 4 lines
Instead, just propagate the result after each parameter run.
Signed-off-by David Gow <[email protected]>Fixes fadb08e7c750 ("kunit: Support for Parameterized Testing")I tried to reproduce the problem described and was unable to. Anyway, from the code it definitely looks like there is a bug like you describe. And it definitely looks like your change should fix it.
I was able to reproduce this again myself. Note that the kunit_tool wrapper does its own result propagation which doesn't have a similar bug, so you won't see the issue in its parsed results. (Using the --raw_output flag does show it, though).
Here's the output from a patched UUID suite, which deliberately fails the first parameter of the first two tests and passes the other parameters and tests, which exhibits the issue:
TAP version 14
1..1
# Subtest: uuid
1..4
# uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57 Expected uuid_parse(data->uuid, &be) == 0, but
uuid_parse(data->uuid, &be) == -22
failed to parse 'c33fx4995-3701-450e-9fbf-206a2e98e576'
# uuid_correct_be: not ok 1 - c33fx4995-3701-450e-9fbf-206a2e98e576 # uuid_correct_be: ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b # uuid_correct_be: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84 not ok 1 - uuid_correct_be
# uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46 Expected guid_parse(data->uuid, &le) == 0, but
guid_parse(data->uuid, &le) == -22
failed to parse 'c33fx4995-3701-450e-9fbf-206a2e98e576'
# uuid_correct_le: not ok 1 - c33fx4995-3701-450e-9fbf-206a2e98e576 # uuid_correct_le: ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b # uuid_correct_le: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84 not ok 2 - uuid_correct_le
# uuid_wrong_be: ok 1 - c33f4995-3701-450e-9fbf206a2e98e576 # uuid_wrong_be: ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b # uuid_wrong_be: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e ok 3 - uuid_wrong_be
# uuid_wrong_le: ok 1 - c33f4995-3701-450e-9fbf206a2e98e576 # uuid_wrong_le: ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b # uuid_wrong_le: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e ok 4 - uuid_wrong_le
ok 1 - uuid
Note the "not ok 1 - uuid_correct_be" line, yet it ending in "ok 1 - uuid".
Quoted 5 lines
Anyway, I tried testing your fix, but given I was unable to reproduce the failure, I am not super confident in my testing. Still,
Reviewed-by Brendan Higgins <[email protected]>