kunit: tool: Parse and print the reason tests are skipped
Subsystems (4)
David Gow
maintainerPATCH v4 1/2 kunit: tool: Parse and print the reason tests are skipped
When a KUnit test (or other KTAP test) is skipped, a "skip reason" can be provided. kunit.py has never done anything with this, ignoring anything included in the KTAP output after the 'SKIP' directive.
Since we have it, and it's used, print it in a nice friendly yellow in parentheses after a skipped test's name.
(And, by parsing it, it can be included in the JUnit results as well.)
---
There are a few bits of KTAP that kunit.py has never actually parsed, and this is one of them. It's also nice to have a good way of quickly seeing why a test has been skipped, given most tests do provide good reason strings.
Happy to hear comments about the style: yellow in parentheses looks pretty good here, but could be a bit confusing if the test name includes parentheses and there's no colour support.
Changes since v3:
https://lore.kernel.org/all/[email protected]/
- Make the parentheses yellow as well (Thanks, Thomas)
- Add some tests for the actual parsing of the SKIP reason. (Thanks, Thomas)
This patch was new in v2.
--- tools/testing/kunit/kunit_parser.py | 15 ++++++++++----- tools/testing/kunit/kunit_tool_test.py | 17 +++++++++++++++++ 2 files changed, 27 insertions(+), 5 deletions(-)
tools/testing/kunit/kunit_parser.py +10 -5
@@ -44,11 +44,12 @@ class Test: self.subtests = [] # type: List[Test] self.log = [] # type: List[str] self.counts = TestCounts() + self.skip_reason = '' def __str__(self) -> str: """Returns string representation of a Test class object.""" return (f'Test({self.status}, {self.name}, {self.expected_count}, ' - f'{self.subtests}, {self.log}, {self.counts})') + f'{self.subtests}, {self.log}, {self.counts}, {self.skip_reason})') def __repr__(self) -> str: """Returns string representation of a Test class object."""
@@ -352,9 +353,9 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: lines.pop() return True -TEST_RESULT = re.compile(r'^\s*(ok|not ok) ([0-9]+) ?(- )?([^#]*)( # .*)?$') +TEST_RESULT = re.compile(r'^\s*(ok|not ok) ([0-9]+) ?(:?- )?([^#]*)( # .*)?$') -TEST_RESULT_SKIP = re.compile(r'^\s*(ok|not ok) ([0-9]+) ?(- )?(.*) # SKIP ?(.*)$') +TEST_RESULT_SKIP = re.compile(r'^\s*(ok|not ok) ([0-9]+) ?(:?- )?(.*) # SKIP ?(.*)$') def peek_test_name_match(lines: LineStream, test: Test) -> bool: """
@@ -418,7 +419,7 @@ def parse_test_result(lines: LineStream, test: Test, # Set name of test object if skip_match: - test.name = skip_match.group(4) or skip_match.group(5) + test.name = skip_match.group(4) else: test.name = match.group(4)
@@ -431,6 +432,7 @@ def parse_test_result(lines: LineStream, test: Test, status = match.group(1) if skip_match: test.status = TestStatus.SKIPPED + test.skip_reason = skip_match.group(5) or '' elif status == 'ok': test.status = TestStatus.SUCCESS else:
@@ -539,7 +541,10 @@ def format_test_result(test: Test, printer: Printer) -> str: if test.status == TestStatus.SUCCESS: return printer.green('[PASSED] ') + test.name if test.status == TestStatus.SKIPPED: - return printer.yellow('[SKIPPED] ') + test.name + skip_message = printer.yellow('[SKIPPED] ') + test.name + if test.skip_reason != '': + skip_message += printer.yellow(' (' + test.skip_reason + ')') + return skip_message if test.status == TestStatus.NO_TESTS: return printer.yellow('[NO TESTS RUN] ') + test.name if test.status == TestStatus.TEST_CRASHED:
tools/testing/kunit/kunit_tool_test.py +17 -0
@@ -235,10 +235,27 @@ class KUnitParserTest(unittest.TestCase): with open(skipped_log) as file: result = kunit_parser.parse_run_tests(file.readlines(), stdout) + # The test result is skipped, and the skip reason is valid + self.assertEqual(kunit_parser.TestStatus.SKIPPED, result.subtests[1].subtests[1].status) + self.assertEqual("this test should be skipped", result.subtests[1].subtests[1].skip_reason) + # A skipped test does not fail the whole suite. self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) self.assertEqual(result.counts, kunit_parser.TestCounts(passed=4, skipped=1)) + def test_skipped_reason_parse(self): + skipped_log = _test_data_path('test_skip_all_tests.log') + with open(skipped_log) as file: + result = kunit_parser.parse_run_tests(file.readlines(), stdout) + + # The first test is skipped, with the correct reaons + self.assertEqual(kunit_parser.TestStatus.SKIPPED, result.subtests[0].subtests[0].status) + self.assertEqual("all tests skipped", result.subtests[0].subtests[0].skip_reason) + + # The first suite is skipped, with no reason + self.assertEqual(kunit_parser.TestStatus.SKIPPED, result.subtests[0].status) + self.assertEqual("", result.subtests[0].skip_reason) + def test_skipped_all_tests(self): skipped_log = _test_data_path('test_skip_all_tests.log') with open(skipped_log) as file:
Signature · 1 line
2.54.0
David Gow
maintainerPATCH v4 2/2 kunit: tool: Add (primitive) support for outputting JUnit XML
This is used by things like Jenkins and other CI systems, which can pretty-print the test output and potentially provide test-level comparisons between runs.
The implementation here is pretty basic: it only provides the raw results, split into tests and test suites, and doesn't provide any overall metadata. However, CI systems like Jenkins can ingest it and it is already useful.
---
Only patch 1 changed, so v4 is the same as v3. I'm reserving the right to switch to a more event-based XML writer in a follow-up, but that'd probably happen as a part of a more broad refactoring of the various output formats.
No changes since v3:
https://lore.kernel.org/all/[email protected]/
Changes since v2:
https://lore.kernel.org/all/[email protected]/
- Rework to use Python's ElementTree XML writing API (Thanks Thomas)
- Output the reason for a test to be skipped.
Changes since v1:
https://lore.kernel.org/all/[email protected]/
- Use python's provided XML quote escaping, rather than coding our own (Thanks Thomas)
- Output proper <skipped> tags for skipped tests
- Report crashed tests as <error>
- Don't output <system-out> tags if there are no lines of log data
--- Documentation/dev-tools/kunit/run_wrapper.rst | 3 + tools/testing/kunit/kunit.py | 21 ++++++- tools/testing/kunit/kunit_junit.py | 61 +++++++++++++++++++ tools/testing/kunit/kunit_tool_test.py | 39 +++++++++++- 4 files changed, 120 insertions(+), 4 deletions(-) create mode 100644 tools/testing/kunit/kunit_junit.py
Documentation/dev-tools/kunit/run_wrapper.rst +3 -0
@@ -324,6 +324,9 @@ command line arguments: - ``--json``: If set, stores the test results in a JSON format and prints to `stdout` or saves to a file if a filename is specified. +- ``--junit``: If set, stores the test results in JUnit XML format and prints to `stdout` or + saves to a file if a filename is specified. + - ``--filter``: Specifies filters on test attributes, for example, ``speed!=slow``. Multiple filters can be used by wrapping input in quotes and separating filters by commas. Example: ``--filter "speed>slow, module=example"``.
tools/testing/kunit/kunit.py +20 -1
@@ -21,6 +21,7 @@ from enum import Enum, auto from typing import Iterable, List, Optional, Sequence, Tuple import kunit_json +import kunit_junit import kunit_kernel import kunit_parser from kunit_printer import stdout, null_printer
@@ -49,6 +50,7 @@ class KunitBuildRequest(KunitConfigRequest): class KunitParseRequest: raw_output: Optional[str] json: Optional[str] + junit: Optional[str] summary: bool failed: bool
@@ -268,6 +270,13 @@ def parse_tests(request: KunitParseRequest, metadata: kunit_json.Metadata, input stdout.print_with_timestamp("Test results stored in %s" % os.path.abspath(request.json)) + if request.junit: + if request.junit == 'stdout': + kunit_junit.print_junit_result(test=test) + else: + kunit_junit.write_junit_result(test=test,filename=request.junit) + stdout.print_with_timestamp(f"Test results stored in {os.path.abspath(request.junit)}") + if test.status != kunit_parser.TestStatus.SUCCESS: return KunitResult(KunitStatus.TEST_FAILURE, parse_time), test
@@ -309,6 +318,7 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree, # So we hackily automatically rewrite --json => --json=stdout pseudo_bool_flag_defaults = { '--json': 'stdout', + '--junit': 'stdout', '--raw_output': 'kunit', } def massage_argv(argv: Sequence[str]) -> Sequence[str]:
@@ -459,6 +469,11 @@ def add_parse_opts(parser: argparse.ArgumentParser) -> None: help='Prints parsed test results as JSON to stdout or a file if ' 'a filename is specified. Does nothing if --raw_output is set.', type=str, const='stdout', default=None, metavar='FILE') + parser.add_argument('--junit', + nargs='?', + help='Prints parsed test results as JUnit XML to stdout or a file if ' + 'a filename is specified. Does nothing if --raw_output is set.', + type=str, const='stdout', default=None, metavar='FILE') parser.add_argument('--summary', help='Prints only the summary line for parsed test results.' 'Does nothing if --raw_output is set.',
@@ -502,6 +517,7 @@ def run_handler(cli_args: argparse.Namespace) -> None: jobs=cli_args.jobs, raw_output=cli_args.raw_output, json=cli_args.json, + junit=cli_args.junit, summary=cli_args.summary, failed=cli_args.failed, timeout=cli_args.timeout,
@@ -552,6 +568,7 @@ def exec_handler(cli_args: argparse.Namespace) -> None: exec_request = KunitExecRequest(raw_output=cli_args.raw_output, build_dir=cli_args.build_dir, json=cli_args.json, + junit=cli_args.junit, summary=cli_args.summary, failed=cli_args.failed, timeout=cli_args.timeout,
@@ -580,7 +597,9 @@ def parse_handler(cli_args: argparse.Namespace) -> None: # We know nothing about how the result was created! metadata = kunit_json.Metadata() request = KunitParseRequest(raw_output=cli_args.raw_output, - json=cli_args.json, summary=cli_args.summary, + json=cli_args.json, + junit=cli_args.junit, + summary=cli_args.summary, failed=cli_args.failed) result, _ = parse_tests(request, metadata, kunit_output) if result.status != KunitStatus.SUCCESS:
tools/testing/kunit/kunit_junit.py new file +61 -0
61 lines not on this page · raw message
tools/testing/kunit/kunit_tool_test.py +36 -3
@@ -24,6 +24,7 @@ import kunit_config import kunit_parser import kunit_kernel import kunit_json +import kunit_junit import kunit from kunit_printer import stdout
@@ -693,6 +694,38 @@ class StrContains(str): def __eq__(self, other): return self in other +class KUnitJUnitTest(unittest.TestCase): + def setUp(self): + self.print_mock = mock.patch('kunit_printer.Printer.print').start() + self.addCleanup(mock.patch.stopall) + + def _junit_string(self, log_file): + with open(_test_data_path(log_file)) as file: + test_result = kunit_parser.parse_run_tests(file, stdout) + junit_string = kunit_junit.get_junit_result( + test=test_result) + print(junit_string) + return junit_string + + def test_failed_test_junit(self): + result = self._junit_string('test_is_test_passed-failure.log') + self.assertTrue("<failure>" in result) + + def test_skipped_test_junit(self): + result = self._junit_string('test_skip_tests.log') + self.assertTrue("<skipped>" in result) + self.assertTrue("skipped=\"1\"" in result) + + def test_crashed_test_junit(self): + result = self._junit_string('test_kernel_panic_interrupt.log') + self.assertTrue("<error>" in result); + + def test_no_tests_junit(self): + result = self._junit_string('test_is_test_passed-no_tests_run_with_header.log') + self.assertTrue("tests=\"0\"" in result) + self.assertFalse("testcase" in result) + + class KUnitMainTest(unittest.TestCase): def setUp(self): path = _test_data_path('test_is_test_passed-all_passed.log')
@@ -940,7 +973,7 @@ class KUnitMainTest(unittest.TestCase): self.linux_source_mock.run_kernel.return_value = ['TAP version 14', 'init: random output'] + want got = kunit._list_tests(self.linux_source_mock, - kunit.KunitExecRequest(None, None, False, False, '.kunit', 300, 'suite*', '', None, None, 'suite', False, False, False)) + kunit.KunitExecRequest(None, None, None, False, False, '.kunit', 300, 'suite*', '', None, None, 'suite', False, False, False)) self.assertEqual(got, want) # Should respect the user's filter glob when listing tests. self.linux_source_mock.run_kernel.assert_called_once_with(
@@ -953,7 +986,7 @@ class KUnitMainTest(unittest.TestCase): # Should respect the user's filter glob when listing tests. mock_tests.assert_called_once_with(mock.ANY, - kunit.KunitExecRequest(None, None, False, False, '.kunit', 300, 'suite*.test*', '', None, None, 'suite', False, False, False)) + kunit.KunitExecRequest(None, None, None, False, False, '.kunit', 300, 'suite*.test*', '', None, None, 'suite', False, False, False)) self.linux_source_mock.run_kernel.assert_has_calls([ mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', filter='', filter_action=None, timeout=300), mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', filter='', filter_action=None, timeout=300),
@@ -966,7 +999,7 @@ class KUnitMainTest(unittest.TestCase): # Should respect the user's filter glob when listing tests. mock_tests.assert_called_once_with(mock.ANY, - kunit.KunitExecRequest(None, None, False, False, '.kunit', 300, 'suite*', '', None, None, 'test', False, False, False)) + kunit.KunitExecRequest(None, None, None, False, False, '.kunit', 300, 'suite*', '', None, None, 'test', False, False, False)) self.linux_source_mock.run_kernel.assert_has_calls([ mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', filter='', filter_action=None, timeout=300), mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', filter='', filter_action=None, timeout=300),
Signature · 1 line
2.54.0
Quoted 10 lines · David Gow
When a KUnit test (or other KTAP test) is skipped, a "skip reason" can be provided.
kunit.pyhas never done anything with this, ignoring anything included in the KTAP output after the 'SKIP' directive.Since we have it, and it's used, print it in a nice friendly yellow in parentheses after a skipped test's name.
(And, by parsing it, it can be included in the JUnit results as well.)
Signed-off-by David Gow <[email protected]>
Thank you; this makes my life easier so I don't have to use --raw_output in my fortify testing. :)
Signature · 1 line
Kees Cook