kunit: kunit_parser: make parser more robust
PATCH v3 1/2 kunit: kunit_parser: make parser more robust
Previously, kunit_parser did not properly handle kunit TAP output that
- had any prefixes (generated from different configs e.g. CONFIG_PRINTK_TIME)
- had unrelated kernel output mixed in the middle of
it, which has shown up when testing with allyesconfig
To remove prefixes, the parser looks for the first line that includes TAP output, "TAP version 14". It then determines the length of the string before this sequence, and strips that number of characters off the beginning of the following lines until the last KUnit output line is reached.
These fixes have been tested with additional tests in the KUnitParseTest and their associated logs have also been added.
---
Changelog
v3:
- Addressing Shuah's git apply issues with whitespaces. Marked all files in test_data as binaries in new kunit/ .gitattributes file. The irregular whitespaces here are necessary as they are used for testing the KUnit parser.
- Important: please note that now git apply works, however the .log files in /test_data/ may get marked as Untracked files. These should be included in the commit as they are required for kunit_tool_test.py.
tools/testing/kunit/.gitattributes | 1 + tools/testing/kunit/kunit_parser.py | 40 +++++----- tools/testing/kunit/kunit_tool_test.py | 69 ++++++++++++++++++ .../test_data/test_config_printk_time.log | Bin 0 -> 1584 bytes .../test_data/test_interrupted_tap_output.log | Bin 0 -> 1982 bytes .../test_data/test_kernel_panic_interrupt.log | Bin 0 -> 1321 bytes .../test_data/test_multiple_prefixes.log | Bin 0 -> 1832 bytes ..._output_with_prefix_isolated_correctly.log | Bin 0 -> 1655 bytes .../kunit/test_data/test_pound_no_prefix.log | Bin 0 -> 1193 bytes .../kunit/test_data/test_pound_sign.log | Bin 0 -> 1655 bytes 10 files changed, 90 insertions(+), 20 deletions(-) create mode 100644 tools/testing/kunit/.gitattributes create mode 100644 tools/testing/kunit/test_data/test_config_printk_time.log create mode 100644 tools/testing/kunit/test_data/test_interrupted_tap_output.log create mode 100644 tools/testing/kunit/test_data/test_kernel_panic_interrupt.log create mode 100644 tools/testing/kunit/test_data/test_multiple_prefixes.log create mode 100644 tools/testing/kunit/test_data/test_output_with_prefix_isolated_correctly.log create mode 100644 tools/testing/kunit/test_data/test_pound_no_prefix.log create mode 100644 tools/testing/kunit/test_data/test_pound_sign.log
tools/testing/kunit/.gitattributes new file +1 -0
@@ -0,0 +1 @@ +test_data/* binary
tools/testing/kunit/kunit_parser.py +20 -20
@@ -46,19 +46,21 @@ class TestStatus(Enum): TEST_CRASHED = auto() NO_TESTS = auto() -kunit_start_re = re.compile(r'^TAP version [0-9]+$') -kunit_end_re = re.compile('List of all partitions:') +kunit_start_re = re.compile(r'TAP version [0-9]+$') +kunit_end_re = re.compile('(List of all partitions:|' + 'Kernel panic - not syncing: VFS:|reboot: System halted)') def isolate_kunit_output(kernel_output): started = False for line in kernel_output: - if kunit_start_re.match(line): + if kunit_start_re.search(line): + prefix_len = len(line.split('TAP version')[0]) started = True - yield line - elif kunit_end_re.match(line): + yield line[prefix_len:] if prefix_len > 0 else line + elif kunit_end_re.search(line): break elif started: - yield line + yield line[prefix_len:] if prefix_len > 0 else line def raw_output(kernel_output): for line in kernel_output:
@@ -108,18 +110,16 @@ OK_NOT_OK_SUBTEST = re.compile(r'^\t(ok|not ok) [0-9]+ - (.*)$') OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) [0-9]+ - (.*)$') -def parse_ok_not_ok_test_case(lines: List[str], - test_case: TestCase, - expecting_test_case: bool) -> bool: +def parse_ok_not_ok_test_case(lines: List[str], test_case: TestCase) -> bool: save_non_diagnositic(lines, test_case) if not lines: - if expecting_test_case: - test_case.status = TestStatus.TEST_CRASHED - return True - else: - return False + test_case.status = TestStatus.TEST_CRASHED + return True line = lines[0] match = OK_NOT_OK_SUBTEST.match(line) + while not match and lines: + line = lines.pop(0) + match = OK_NOT_OK_SUBTEST.match(line) if match: test_case.log.append(lines.pop(0)) test_case.name = match.group(2)
@@ -150,12 +150,12 @@ def parse_diagnostic(lines: List[str], test_case: TestCase) -> bool: else: return False -def parse_test_case(lines: List[str], expecting_test_case: bool) -> TestCase: +def parse_test_case(lines: List[str]) -> TestCase: test_case = TestCase() save_non_diagnositic(lines, test_case) while parse_diagnostic(lines, test_case): pass - if parse_ok_not_ok_test_case(lines, test_case, expecting_test_case): + if parse_ok_not_ok_test_case(lines, test_case): return test_case else: return None
@@ -234,11 +234,11 @@ def parse_test_suite(lines: List[str]) -> TestSuite: expected_test_case_num = parse_subtest_plan(lines) if not expected_test_case_num: return None - test_case = parse_test_case(lines, expected_test_case_num > 0) - expected_test_case_num -= 1 - while test_case: + while expected_test_case_num > 0: + test_case = parse_test_case(lines) + if not test_case: + break test_suite.cases.append(test_case) - test_case = parse_test_case(lines, expected_test_case_num > 0) expected_test_case_num -= 1 if parse_ok_not_ok_test_suite(lines, test_suite): test_suite.status = bubble_up_test_case_errors(test_suite)
tools/testing/kunit/kunit_tool_test.py +69 -0
81 lines not on this page · raw message
tools/testing/kunit/test_data/test_config_printk_time.log new file binary +0 -0
tools/testing/kunit/test_data/test_interrupted_tap_output.log new file binary +0 -0
tools/testing/kunit/test_data/test_kernel_panic_interrupt.log new file binary +0 -0
tools/testing/kunit/test_data/test_multiple_prefixes.log new file binary +0 -0
tools/testing/kunit/test_data/test_output_with_prefix_isolated_correctly.log new file binary +0 -0
tools/testing/kunit/test_data/test_pound_no_prefix.log new file binary +0 -0
tools/testing/kunit/test_data/test_pound_sign.log new file binary +0 -0
Signature · 1 line
2.25.1.481.gfbce0eb801-goog
PATCH v3 2/2 kunit: Run all KUnit tests through allyesconfig
Implemented the functionality to run all KUnit tests through kunit_tool by specifying an --alltests flag, which builds UML with allyesconfig enabled, and consequently runs every KUnit test. A new function was added to kunit_kernel: make_allyesconfig.
Firstly, if --alltests is specified, kunit.py triggers build_um_kernel which call make_allyesconfig. This function calls the make command, disables the broken configs that would otherwise prevent UML from building, then starts the kernel with all possible configurations enabled. All stdout and stderr is sent to test.log and read from there then fed through kunit_parser to parse the tests to the user. Also added a signal_handler in case kunit is interrupted while running.
Tested: Run under different conditions such as testing with --raw_output, testing program interrupt then immediately running kunit again without --alltests and making sure to clean the console.
---
Changelog:
v3:
- Minor change since v2 in message for program interruption. Changed "Cleaning config." to "Cleaning console."
.../kunit/configs/broken_on_uml.config | 41 ++++++++++++ tools/testing/kunit/kunit.py | 30 +++++---- tools/testing/kunit/kunit_kernel.py | 62 ++++++++++++++----- tools/testing/kunit/kunit_parser.py | 1 + tools/testing/kunit/kunit_tool_test.py | 17 ++--- 5 files changed, 115 insertions(+), 36 deletions(-) create mode 100644 tools/testing/kunit/configs/broken_on_uml.config
tools/testing/kunit/configs/broken_on_uml.config new file +41 -0
@@ -0,0 +1,41 @@ +# These are currently broken on UML and prevent allyesconfig from building +# CONFIG_STATIC_LINK is not set +# CONFIG_UML_NET_VECTOR is not set +# CONFIG_UML_NET_VDE is not set +# CONFIG_UML_NET_PCAP is not set +# CONFIG_NET_PTP_CLASSIFY is not set +# CONFIG_IP_VS is not set +# CONFIG_BRIDGE_EBT_BROUTE is not set +# CONFIG_BRIDGE_EBT_T_FILTER is not set +# CONFIG_BRIDGE_EBT_T_NAT is not set +# CONFIG_MTD_NAND_CADENCE is not set +# CONFIG_MTD_NAND_NANDSIM is not set +# CONFIG_BLK_DEV_NULL_BLK is not set +# CONFIG_BLK_DEV_RAM is not set +# CONFIG_SCSI_DEBUG is not set +# CONFIG_NET_VENDOR_XILINX is not set +# CONFIG_NULL_TTY is not set +# CONFIG_PTP_1588_CLOCK is not set +# CONFIG_PINCTRL_EQUILIBRIUM is not set +# CONFIG_DMABUF_SELFTESTS is not set +# CONFIG_COMEDI is not set +# CONFIG_XIL_AXIS_FIFO is not set +# CONFIG_EXFAT_FS is not set +# CONFIG_STM_DUMMY is not set +# CONFIG_FSI_MASTER_ASPEED is not set +# CONFIG_JFS_FS is not set +# CONFIG_UBIFS_FS is not set +# CONFIG_CRAMFS is not set +# CONFIG_CRYPTO_DEV_SAFEXCEL is not set +# CONFIG_CRYPTO_DEV_AMLOGIC_GXL is not set +# CONFIG_KCOV is not set +# CONFIG_LKDTM is not set +# CONFIG_REED_SOLOMON_TEST is not set +# CONFIG_TEST_RHASHTABLE is not set +# CONFIG_TEST_MEMINIT is not set +# CONFIG_NETWORK_PHY_TIMESTAMPING is not set +# CONFIG_DEBUG_INFO_BTF is not set +# CONFIG_PTP_1588_CLOCK_INES is not set +# CONFIG_QCOM_CPR is not set +# CONFIG_RESET_BRCMSTB_RESCAL is not set +# CONFIG_RESET_INTEL_GW is not set
tools/testing/kunit/kunit.py +18 -12
@@ -22,7 +22,9 @@ import kunit_parser KunitResult = namedtuple('KunitResult', ['status','result']) -KunitRequest = namedtuple('KunitRequest', ['raw_output','timeout', 'jobs', 'build_dir', 'defconfig']) +KunitRequest = namedtuple('KunitRequest', ['raw_output','timeout', 'jobs', + 'build_dir', 'defconfig', + 'alltests']) KernelDirectoryPath = sys.argv[0].split('tools/testing/kunit/')[0]
@@ -55,24 +57,23 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree, kunit_parser.print_with_timestamp('Building KUnit Kernel ...') build_start = time.time() - success = linux.build_um_kernel(request.jobs, request.build_dir) + success = linux.build_um_kernel(request.alltests, + request.jobs, + request.build_dir) build_end = time.time() if not success: return KunitResult(KunitStatus.BUILD_FAILURE, 'could not build kernel') kunit_parser.print_with_timestamp('Starting KUnit Kernel ...') test_start = time.time() - - test_result = kunit_parser.TestResult(kunit_parser.TestStatus.SUCCESS, - [], - 'Tests not Parsed.') + kunit_output = linux.run_kernel( + timeout=None if request.alltests else request.timeout, + build_dir=request.build_dir) if request.raw_output: - kunit_parser.raw_output( - linux.run_kernel(timeout=request.timeout, - build_dir=request.build_dir)) + raw_output = kunit_parser.raw_output(kunit_output) + isolated = list(kunit_parser.isolate_kunit_output(raw_output)) + test_result = kunit_parser.parse_test_result(isolated) else: - kunit_output = linux.run_kernel(timeout=request.timeout, - build_dir=request.build_dir) test_result = kunit_parser.parse_run_tests(kunit_output) test_end = time.time()
@@ -120,6 +121,10 @@ def main(argv, linux=None): help='Uses a default .kunitconfig.', action='store_true') + run_parser.add_argument('--alltests', + help='Run all KUnit tests through allyesconfig', + action='store_true') + cli_args = parser.parse_args(argv) if cli_args.subcommand == 'run':
@@ -143,7 +148,8 @@ def main(argv, linux=None): cli_args.timeout, cli_args.jobs, cli_args.build_dir, - cli_args.defconfig) + cli_args.defconfig, + cli_args.alltests) result = run_tests(linux, request) if result.status != KunitStatus.SUCCESS: sys.exit(1)
tools/testing/kunit/kunit_kernel.py +45 -17
109 lines not on this page · raw message
tools/testing/kunit/kunit_parser.py +1 -0
@@ -65,6 +65,7 @@ def isolate_kunit_output(kernel_output): def raw_output(kernel_output): for line in kernel_output: print(line) + yield line DIVIDER = '=' * 60
tools/testing/kunit/kunit_tool_test.py +10 -7
@@ -243,7 +243,8 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run'], self.linux_source_mock) assert self.linux_source_mock.build_reconfig.call_count == 1 assert self.linux_source_mock.run_kernel.call_count == 1 - self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='', timeout=300) + self.linux_source_mock.run_kernel.assert_called_once_with( + build_dir='', timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_passes_args_fail(self):
@@ -258,25 +259,27 @@ class KUnitMainTest(unittest.TestCase): def test_run_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) - kunit.main(['run', '--raw_output'], self.linux_source_mock) + with self.assertRaises(SystemExit) as e: + kunit.main(['run', '--raw_output'], self.linux_source_mock) + assert type(e.exception) == SystemExit + assert e.exception.code == 1 assert self.linux_source_mock.build_reconfig.call_count == 1 assert self.linux_source_mock.run_kernel.call_count == 1 - for kall in self.print_mock.call_args_list: - assert kall != mock.call(StrContains('Testing complete.')) - assert kall != mock.call(StrContains(' 0 tests run')) def test_run_timeout(self): timeout = 3453 kunit.main(['run', '--timeout', str(timeout)], self.linux_source_mock) assert self.linux_source_mock.build_reconfig.call_count == 1 - self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='', timeout=timeout) + self.linux_source_mock.run_kernel.assert_called_once_with( + build_dir='', timeout=timeout) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_builddir(self): build_dir = '.kunit' kunit.main(['run', '--build_dir', build_dir], self.linux_source_mock) assert self.linux_source_mock.build_reconfig.call_count == 1 - self.linux_source_mock.run_kernel.assert_called_once_with(build_dir=build_dir, timeout=300) + self.linux_source_mock.run_kernel.assert_called_once_with( + build_dir=build_dir, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) if __name__ == '__main__':
Signature · 1 line
2.25.1.481.gfbce0eb801-goog
Re: [PATCH v3 2/2] kunit: Run all KUnit tests through allyesconfig
Quoted 16 lines · Heidi Fahim
Implemented the functionality to run all KUnit tests through
kunit_toolby specifying an --alltests flag, which builds UML with allyesconfig enabled, and consequently runs every KUnit test. A new function was added tokunit_kernel:make_allyesconfig.
Firstly, if --alltests is specified,kunit.pytriggersbuild_um_kernelwhich callmake_allyesconfig. This function calls the make command, disables the broken configs that would otherwise prevent UML from building, then starts the kernel with all possible configurations enabled. All stdout and stderr is sent to test.log and read from there then fed throughkunit_parserto parse the tests to the user. Also added asignal_handlerin case kunit is interrupted while running.
Tested: Run under different conditions such as testing with --raw_output, testing program interrupt then immediately running kunit again without --alltests and making sure to clean the console.
Quoted 1 line
Signed-off-by Heidi Fahim <[email protected]>
Quoted 14 lines · Heidi Fahim
Previously,
kunit_parserdid not properly handle kunit TAP output that
- had any prefixes (generated from different configs e.g.CONFIG_PRINTK_TIME)
- had unrelated kernel output mixed in the middle of
it, which has shown up when testing with allyesconfig
To remove prefixes, the parser looks for the first line that includes TAP output, "TAP version 14". It then determines the length of the string before this sequence, and strips that number of characters off the beginning of the following lines until the last KUnit output line is reached.
These fixes have been tested with additional tests in the KUnitParseTest and their associated logs have also been added.
Quoted 1 line
Signed-off-by Heidi Fahim <[email protected]>