Skip to contentSkip to conversation

kunit: tool: print clearer error message when there's no TAP output

PATCH In review Daniel Latypov 3 participants 4 messages Mar 29, 2022 – May 11, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kunit: tool: print clearer error message when there's no TAP output

Before:

$ ./tools/testing/kunit/kunit.py parse /dev/null
...
[ERROR] Test : invalid KTAP input!

After:

$ ./tools/testing/kunit/kunit.py parse /dev/null
...
[ERROR] Test <missing>: could not find any KTAP output!

This error message gets printed out when extract_tap_output() yielded no lines. So while it could be because of malformed KTAP output from KUnit, it could also be due to to not having any KTAP output at all.

Try and make the error message here more clear.

---
 tools/testing/kunit/kunit_parser.py    | 3 ++-
 tools/testing/kunit/kunit_tool_test.py | 4 ++--
 2 files changed, 4 insertions(+), 3 deletions(-)
tools/testing/kunit/kunit_parser.py +2 -1
index 05ff334761dd..103d95a66a7e 100644
@@ -817,7 +817,8 @@ def parse_run_tests(kernel_output: Iterable[str]) -> Test:
 	lines = extract_tap_lines(kernel_output)
 	test = Test()
 	if not lines:
-		test.add_error('invalid KTAP input!')
+		test.name = '<missing>'
+		test.add_error('could not find any KTAP output!')
 		test.status = TestStatus.FAILURE_TO_PARSE_TESTS
 	else:
 		test = parse_test(lines, 0, [])
tools/testing/kunit/kunit_tool_test.py +2 -2
index 352369dffbd9..f14934853ea1 100755
@@ -226,7 +226,7 @@ class KUnitParserTest(unittest.TestCase):
 		with open(crash_log) as file:
 			result = kunit_parser.parse_run_tests(
 				kunit_parser.extract_tap_lines(file.readlines()))
-		print_mock.assert_any_call(StrContains('invalid KTAP input!'))
+		print_mock.assert_any_call(StrContains('could not find any KTAP output!'))
 		print_mock.stop()
 		self.assertEqual(0, len(result.subtests))
 
@@ -559,7 +559,7 @@ class KUnitMainTest(unittest.TestCase):
 		self.assertEqual(e.exception.code, 1)
 		self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
 		self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1)
-		self.print_mock.assert_any_call(StrContains('invalid KTAP input!'))
+		self.print_mock.assert_any_call(StrContains('could not find any KTAP output!'))
 
 	def test_exec_no_tests(self):
 		self.linux_source_mock.run_kernel = mock.Mock(return_value=['TAP version 14', '1..0'])

base-commit: 13776ebb9964b2ea66ffb8c824c0762eed6da784

Signature · 1 line
2.35.1.1021.g381101b075-goog
Reviewed-by on v1 1/1
#1 lore raw
Quoted 11 lines · Daniel Latypov

Before:

$ ./tools/testing/kunit/kunit.py parse /dev/null
...
[ERROR] Test : invalid KTAP input!

After:

$ ./tools/testing/kunit/kunit.py parse /dev/null
...
[ERROR] Test <missing>: could not find any KTAP output!
Quoted 8 lines

This error message gets printed out when extract_tap_output() yielded no lines. So while it could be because of malformed KTAP output from KUnit, it could also be due to to not having any KTAP output at all.

Try and make the error message here more clear.

---

At first I thought that this was "working as intended", but I agree that it's a bit confusing, so changing it is for the best. (And there's no sense getting too bogged down in the philosophical difference between "invalid TAP" and "not valid TAP" :-))

This works fine here, and the code looks sensible. I tested it with the --json option as well, and the result ("test_cases" being empty) makes sense to me:
---
{
"name": "KUnit Test Group",
"arch": "UM",
"defconfig": "kunit_defconfig",
"build_environment": "",
"sub_groups": [],
"test_cases": [],
"lab_name": null,
"kernel": null,
"job": null,
"git_branch": "kselftest"
}
---

So, this is:

Cheers,
-- David

#2 lore raw
Quoted 19 lines · David Gow
Quoted 17 lines · Daniel Latypov

Before:

$ ./tools/testing/kunit/kunit.py parse /dev/null
...
[ERROR] Test : invalid KTAP input!

After:

$ ./tools/testing/kunit/kunit.py parse /dev/null
...
[ERROR] Test <missing>: could not find any KTAP output!

This error message gets printed out when extract_tap_output() yielded no lines. So while it could be because of malformed KTAP output from KUnit, it could also be due to to not having any KTAP output at all.

Try and make the error message here more clear.

Quoted 7 lines
Quoted 2 lines

---

At first I thought that this was "working as intended", but I agree that it's a bit confusing, so changing it is for the best. (And there's no sense getting too bogged down in the philosophical difference between "invalid TAP" and "not valid TAP" :-))

extract_tap_output() returning something means it just saw a TAP header. So this error realistically only happens when there's no TAP at all :P

My allusion to malformed KTAP is in the sense that
KTA<random stuff>P ver<more stuff>sion 1
is malformed.

We could potentially be more completely precise w/ our error message and say "no K?TAP header found", but I initially thought this might be a bit more user-friendly.

Reviewed-by on v1 1/1
#3 lore raw
In reply to Daniel Latypov
Quoted 11 lines · Daniel Latypov

Before:

$ ./tools/testing/kunit/kunit.py parse /dev/null
...
[ERROR] Test : invalid KTAP input!

After:

$ ./tools/testing/kunit/kunit.py parse /dev/null
...
[ERROR] Test <missing>: could not find any KTAP output!
Quoted 7 lines

This error message gets printed out when extract_tap_output() yielded no lines. So while it could be because of malformed KTAP output from KUnit, it could also be due to to not having any KTAP output at all.

Try and make the error message here more clear.