Skip to contentSkip to conversation

kunit: tool: add support for QEMU

PATCH In review Brendan Higgins 3 participants 7 messages May 26, 2021 – May 28, 2021

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v2 cover kunit: tool: add support for QEMU

TL;DR: Add support to kunit_tool to dispatch tests via QEMU. Also add support to immediately shutdown a kernel after running KUnit tests.

Background
----------

KUnit has supported running on all architectures for quite some time; however, kunit_tool - the script commonly used to invoke KUnit tests - has only fully supported KUnit run on UML. Its functionality has been broken up for some time to separate the configure, build, run, and parse phases making it possible to be used in part on other architectures to a small extent. Nevertheless, kunit_tool has not supported running tests on other architectures.

What this patchset does
-----------------------

This patchset introduces first class support to kunit_tool for KUnit to be run on many popular architectures via QEMU. It does this by adding two new flags: --arch and --cross_compile.

--arch allows an architecture to be specified by the name the architecture is given in arch/. It uses the specified architecture to select a minimal amount of Kconfigs and QEMU configs needed for the architecture to run in QEMU and provide a console from which KTAP results can be scraped.

--cross_compile allows a toolchain prefix to be specified to make similar to how CROSS_COMPILE is used.

Additionally, this patchset revives the previously considered "kunit:
tool: add support for QEMU"[1] patchs. The motivation for this new kernel command line flags, kunit_shutdown, is to better support running KUnit tests inside of QEMU. For most popular architectures, QEMU can be made to terminate when the Linux kernel that is being run is reboted, halted, or powered off. As Kees pointed out in a previous discussion[2], it is possible to make a kernel initrd that can reboot the kernel immediately, doing this for every architecture would likely be infeasible. Instead, just having an option for the kernel to shutdown when it is done with testing seems a lot simpler, especially since it is an option which would only available in testing configurations of the kernel anyway.

Changes since last revision
---------------------------

I pulled out the QemuConfigs into their own files; the way in which I did this also allows new QemuConfigs to be added without making any changes to kunit_tool.

I changed how Kconfigs are loaded; they are now merged inside of kunit_tool instead of letting Kbuild do it.

I also made numerous nit fixes.

Finally, I added a new section to the kunit_tool documentation to document the new command line flags I added.

[1] http://patches.linaro.org/patch/208336/
[2] https://lkml.org/lkml/2020/6/26/988

Brendan Higgins (3):
Documentation: Add kunit_shutdown to kernel-parameters.txt
kunit: tool: add support for QEMU
Documentation: kunit: document support for QEMU in kunit_tool

David Gow (1):
kunit: Add 'kunit_shutdown' option

 .../admin-guide/kernel-parameters.txt         |   8 +
 Documentation/dev-tools/kunit/kunit-tool.rst  |  48 +++++
 Documentation/dev-tools/kunit/usage.rst       |  50 +++--
 lib/kunit/executor.c                          |  20 ++
 tools/testing/kunit/kunit.py                  |  57 +++++-
 tools/testing/kunit/kunit_config.py           |   7 +-
 tools/testing/kunit/kunit_kernel.py           | 177 +++++++++++++++---
 tools/testing/kunit/kunit_parser.py           |   2 +-
 tools/testing/kunit/kunit_tool_test.py        |  18 +-
 tools/testing/kunit/qemu_config.py            |  16 ++
 tools/testing/kunit/qemu_configs/alpha.py     |  10 +
 tools/testing/kunit/qemu_configs/arm.py       |  13 ++
 tools/testing/kunit/qemu_configs/arm64.py     |  12 ++
 tools/testing/kunit/qemu_configs/i386.py      |  10 +
 tools/testing/kunit/qemu_configs/powerpc.py   |  12 ++
 tools/testing/kunit/qemu_configs/riscv.py     |  31 +++
 tools/testing/kunit/qemu_configs/s390.py      |  14 ++
 tools/testing/kunit/qemu_configs/sparc.py     |  10 +
 tools/testing/kunit/qemu_configs/x86_64.py    |  10 +
 19 files changed, 471 insertions(+), 54 deletions(-)
 create mode 100644 tools/testing/kunit/qemu_config.py
 create mode 100644 tools/testing/kunit/qemu_configs/alpha.py
 create mode 100644 tools/testing/kunit/qemu_configs/arm.py
 create mode 100644 tools/testing/kunit/qemu_configs/arm64.py
 create mode 100644 tools/testing/kunit/qemu_configs/i386.py
 create mode 100644 tools/testing/kunit/qemu_configs/powerpc.py
 create mode 100644 tools/testing/kunit/qemu_configs/riscv.py
 create mode 100644 tools/testing/kunit/qemu_configs/s390.py
 create mode 100644 tools/testing/kunit/qemu_configs/sparc.py
 create mode 100644 tools/testing/kunit/qemu_configs/x86_64.py

base-commit: d7eab3df8f39b116d934bc17f8070861e18cfb62

Signature · 1 line
2.31.1.818.g46aad6cb9e-goog
#1 lore raw

PATCH v2 1/4 kunit: Add 'kunit_shutdown' option

From: David Gow <[email protected]>

Add a new kernel command-line option, 'kunit_shutdown', which allows the user to specify that the kernel poweroff, halt, or reboot after completing all KUnit tests; this is very handy for running KUnit tests on UML or a VM so that the UML/VM process exits cleanly immediately after running all tests without needing a special initramfs.

---
 lib/kunit/executor.c                | 20 ++++++++++++++++++++
 tools/testing/kunit/kunit_kernel.py |  2 +-
 tools/testing/kunit/kunit_parser.py |  2 +-
 3 files changed, 22 insertions(+), 2 deletions(-)
lib/kunit/executor.c +20 -0
index 15832ed446685..7db619624437c 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 
+#include <linux/reboot.h>
 #include <kunit/test.h>
 #include <linux/glob.h>
 #include <linux/moduleparam.h>
@@ -18,6 +19,9 @@ module_param(filter_glob, charp, 0);
 MODULE_PARM_DESC(filter_glob,
 		"Filter which KUnit test suites run at boot-time, e.g. list*");
 
+static char *kunit_shutdown;
+core_param(kunit_shutdown, kunit_shutdown, charp, 0644);
+
 static struct kunit_suite * const *
 kunit_filter_subsuite(struct kunit_suite * const * const subsuite)
 {
@@ -82,6 +86,20 @@ static struct suite_set kunit_filter_suites(void)
 	return filtered;
 }
 
+static void kunit_handle_shutdown(void)
+{
+	if (!kunit_shutdown)
+		return;
+
+	if (!strcmp(kunit_shutdown, "poweroff"))
+		kernel_power_off();
+	else if (!strcmp(kunit_shutdown, "halt"))
+		kernel_halt();
+	else if (!strcmp(kunit_shutdown, "reboot"))
+		kernel_restart(NULL);
+
+}
+
 static void kunit_print_tap_header(struct suite_set *suite_set)
 {
 	struct kunit_suite * const * const *suites, * const *subsuite;
@@ -112,6 +130,8 @@ int kunit_run_all_tests(void)
 		kfree(suite_set.start);
 	}
 
+	kunit_handle_shutdown();
+
 	return 0;
 }
 
tools/testing/kunit/kunit_kernel.py +1 -1
index 89a7d4024e878..dbbceaee12305 100644
@@ -208,7 +208,7 @@ class LinuxSourceTree(object):
 	def run_kernel(self, args=None, build_dir='', filter_glob='', timeout=None) -> Iterator[str]:
 		if not args:
 			args = []
-		args.extend(['mem=1G', 'console=tty'])
+		args.extend(['mem=1G', 'console=tty', 'kunit_shutdown=halt'])
 		if filter_glob:
 			args.append('kunit.filter_glob='+filter_glob)
 		self._ops.linux_bin(args, timeout, build_dir)
tools/testing/kunit/kunit_parser.py +1 -1
index e8bcc139702e2..8d8d4d70b39dd 100644
@@ -49,7 +49,7 @@ class TestStatus(Enum):
 
 kunit_start_re = re.compile(r'TAP version [0-9]+$')
 kunit_end_re = re.compile('(List of all partitions:|'
-			  'Kernel panic - not syncing: VFS:)')
+			  'Kernel panic - not syncing: VFS:|reboot: System halted)')
 
 def isolate_kunit_output(kernel_output) -> Iterator[str]:
 	started = False
Signature · 1 line
2.31.1.818.g46aad6cb9e-goog
Reviewed-by on v2 1/4
#2 lore raw

Re: [PATCH v2 1/4] kunit: Add 'kunit_shutdown' option

Quoted 9 lines · Brendan Higgins

From: David Gow <[email protected]>

Add a new kernel command-line option, 'kunit_shutdown', which allows the user to specify that the kernel poweroff, halt, or reboot after completing all KUnit tests; this is very handy for running KUnit tests on UML or a VM so that the UML/VM process exits cleanly immediately after running all tests without needing a special initramfs.

Quoted 4 lines

Looks good, one optional nit below.

Also, I tested this again on top of 5.13-r4 (7ac3a1c1ae51 ("Merge tag 'mtd/fixes-for-5.13-rc4' of
git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux"))

$ echo -e "\nCONFIG_DEBUG_KERNEL=y\nCONFIG_DEBUG_INFO=y\nCONFIG_GCOV=y"
Quoted 1 line
Quoted 1 line

.kunit/.kunitconfig

$ run_kunit --make_options=CC=/usr/bin/gcc-6
$ <run coverage generation steps>
Overall coverage rate:
  lines......: 13.8% (17286 of 125320 lines)
  functions..: 15.8% (1790 of 11336 functions)

So coverage still seems to be fixed, no need to comment out the uml_abort() anymore and this sidesteps whatever weird issue I started running into these past few weeks.
So that's very exciting.

Quoted 32 lines
---
 lib/kunit/executor.c                | 20 ++++++++++++++++++++
 tools/testing/kunit/kunit_kernel.py |  2 +-
 tools/testing/kunit/kunit_parser.py |  2 +-
 3 files changed, 22 insertions(+), 2 deletions(-)
lib/kunit/executor.c +5 -0
index 15832ed446685..7db619624437c 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 
+#include <linux/reboot.h>
 #include <kunit/test.h>
 #include <linux/glob.h>
 #include <linux/moduleparam.h>
@@ -18,6 +19,9 @@ module_param(filter_glob, charp, 0);
 MODULE_PARM_DESC(filter_glob,
                "Filter which KUnit test suites run at boot-time, e.g. list*");
 
+static char *kunit_shutdown;
+core_param(kunit_shutdown, kunit_shutdown, charp, 0644);
+
 static struct kunit_suite * const *
 kunit_filter_subsuite(struct kunit_suite * const * const subsuite)
 {
@@ -82,6 +86,20 @@ static struct suite_set kunit_filter_suites(void)
        return filtered;
 }
 
+static void kunit_handle_shutdown(void)

lib/kunit/executor.c static struct suite_set kunit_filter_suites(void) · line 99

Quoted 10 lines
lib/kunit/executor.c +10 -0
+{
+       if (!kunit_shutdown)
+               return;
+
+       if (!strcmp(kunit_shutdown, "poweroff"))
+               kernel_power_off();
+       else if (!strcmp(kunit_shutdown, "halt"))
+               kernel_halt();
+       else if (!strcmp(kunit_shutdown, "reboot"))
+               kernel_restart(NULL);

nit: should we complain about an unknown option here?

Quoted 44 lines
lib/kunit/executor.c +5 -0
+
+}
+
 static void kunit_print_tap_header(struct suite_set *suite_set)
 {
        struct kunit_suite * const * const *suites, * const *subsuite;
@@ -112,6 +130,8 @@ int kunit_run_all_tests(void)
                kfree(suite_set.start);
        }
 
+       kunit_handle_shutdown();
+
        return 0;
 }
 
tools/testing/kunit/kunit_kernel.py +1 -1
index 89a7d4024e878..dbbceaee12305 100644
@@ -208,7 +208,7 @@ class LinuxSourceTree(object):
        def run_kernel(self, args=None, build_dir='', filter_glob='', timeout=None) -> Iterator[str]:
                if not args:
                        args = []
-               args.extend(['mem=1G', 'console=tty'])
+               args.extend(['mem=1G', 'console=tty', 'kunit_shutdown=halt'])
                if filter_glob:
                        args.append('kunit.filter_glob='+filter_glob)
                self._ops.linux_bin(args, timeout, build_dir)
tools/testing/kunit/kunit_parser.py +1 -2
index e8bcc139702e2..8d8d4d70b39dd 100644
@@ -49,7 +49,7 @@ class TestStatus(Enum):
 
 kunit_start_re = re.compile(r'TAP version [0-9]+$')
 kunit_end_re = re.compile('(List of all partitions:|'
-                         'Kernel panic - not syncing: VFS:)')
+                         'Kernel panic - not syncing: VFS:|reboot: System halted)')
 
 def isolate_kunit_output(kernel_output) -> Iterator[str]:
        started = False
--

2.31.1.818.g46aad6cb9e-goog

#3 lore raw

PATCH v2 2/4 Documentation: Add kunit_shutdown to kernel-parameters.txt

In reply to Brendan Higgins

Add kunit_shutdown, an option to specify that the kernel shutsdown after running KUnit tests, to the kernel-parameters.txt documentation.

---
 Documentation/admin-guide/kernel-parameters.txt | 8 ++++++++
 1 file changed, 8 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +8 -0
index cb89dbdedc463..78c22bd55b92e 100644
@@ -2256,6 +2256,14 @@
 			0: force disabled
 			1: force enabled
 
+	kunit_shutdown=[KERNEL UNIT TESTING FRAMEWORK] Shutdown kernel after
+			running built-in tests. Tests configured as modules will
+			not be run.
+			Default:	(flag not present) don't shutdown
+			poweroff:	poweroff the kernel after running tests
+			halt:		halt the kernel after running tests
+			reboot:		reboot the kernel after running tests
+
 	kvm.ignore_msrs=[KVM] Ignore guest accesses to unhandled MSRs.
 			Default is 0 (don't ignore, but inject #GP)
 
Signature · 1 line
2.31.1.818.g46aad6cb9e-goog
#4 lore raw

PATCH v2 3/4 kunit: tool: add support for QEMU

In reply to Brendan Higgins

Add basic support to run QEMU via kunit_tool. Add support for i386, x86_64, arm, arm64, and a bunch more.

---
Changes since last revision:

- I pulled out the QemuConfigs into their own files; the way in which I did this also allows new QemuConfigs to be added without making any changes to kunit_tool.

- I changed how Kconfigs are loaded; they are now merged inside of kunit_tool instead of letting Kbuild do it.

- I also made numerous nit fixes.

---
 tools/testing/kunit/kunit.py                |  57 ++++++-
 tools/testing/kunit/kunit_config.py         |   7 +-
 tools/testing/kunit/kunit_kernel.py         | 175 +++++++++++++++++---
 tools/testing/kunit/kunit_tool_test.py      |  18 +-
 tools/testing/kunit/qemu_config.py          |  16 ++
 tools/testing/kunit/qemu_configs/alpha.py   |  10 ++
 tools/testing/kunit/qemu_configs/arm.py     |  13 ++
 tools/testing/kunit/qemu_configs/arm64.py   |  12 ++
 tools/testing/kunit/qemu_configs/i386.py    |  10 ++
 tools/testing/kunit/qemu_configs/powerpc.py |  12 ++
 tools/testing/kunit/qemu_configs/riscv.py   |  31 ++++
 tools/testing/kunit/qemu_configs/s390.py    |  14 ++
 tools/testing/kunit/qemu_configs/sparc.py   |  10 ++
 tools/testing/kunit/qemu_configs/x86_64.py  |  10 ++
 14 files changed, 354 insertions(+), 41 deletions(-)
 create mode 100644 tools/testing/kunit/qemu_config.py
 create mode 100644 tools/testing/kunit/qemu_configs/alpha.py
 create mode 100644 tools/testing/kunit/qemu_configs/arm.py
 create mode 100644 tools/testing/kunit/qemu_configs/arm64.py
 create mode 100644 tools/testing/kunit/qemu_configs/i386.py
 create mode 100644 tools/testing/kunit/qemu_configs/powerpc.py
 create mode 100644 tools/testing/kunit/qemu_configs/riscv.py
 create mode 100644 tools/testing/kunit/qemu_configs/s390.py
 create mode 100644 tools/testing/kunit/qemu_configs/sparc.py
 create mode 100644 tools/testing/kunit/qemu_configs/x86_64.py
tools/testing/kunit/kunit.py +49 -8
index 5da8fb3762f98..be8d8d4a4e08f 100755
@@ -70,10 +70,10 @@ def build_tests(linux: kunit_kernel.LinuxSourceTree,
 	kunit_parser.print_with_timestamp('Building KUnit Kernel ...')
 
 	build_start = time.time()
-	success = linux.build_um_kernel(request.alltests,
-					request.jobs,
-					request.build_dir,
-					request.make_options)
+	success = linux.build_kernel(request.alltests,
+				     request.jobs,
+				     request.build_dir,
+				     request.make_options)
 	build_end = time.time()
 	if not success:
 		return KunitResult(KunitStatus.BUILD_FAILURE,
@@ -189,6 +189,31 @@ def add_common_opts(parser) -> None:
 			     'will get  automatically appended.',
 			     metavar='kunitconfig')
 
+	parser.add_argument('--arch',
+			    help=('Specifies the architecture to run tests under. '
+				  'The architecture specified here must match the '
+				  'string passed to the ARCH make param, '
+				  'e.g. i386, x86_64, arm, um, etc. Non-UML '
+				  'architectures run on QEMU.'),
+			    type=str, default='um', metavar='arch')
+
+	parser.add_argument('--cross_compile',
+			    help=('Sets make\'s CROSS_COMPILE variable; it should '
+				  'be set to a toolchain path prefix (the prefix '
+				  'of gcc and other tools in your toolchain, for '
+				  'example `sparc64-linux-gnu-` if you have the '
+				  'sparc toolchain installed on your system, or '
+				  '`$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux-` '
+				  'if you have downloaded the microblaze toolchain '
+				  'from the 0-day website to a directory in your '
+				  'home directory called `toolchains`).'),
+			    metavar='cross_compile')
+
+	parser.add_argument('--qemu_config',
+			    help=('Takes a path to a path to a file containing '
+				  'a QemuArchParams object.'),
+			    type=str, metavar='qemu_config')
+
 def add_build_opts(parser) -> None:
 	parser.add_argument('--jobs',
 			    help='As in the make command, "Specifies  the number of '
@@ -270,7 +295,11 @@ def main(argv, linux=None):
 			os.mkdir(cli_args.build_dir)
 
 		if not linux:
-			linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir, kunitconfig_path=cli_args.kunitconfig)
+			linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir,
+					kunitconfig_path=cli_args.kunitconfig,
+					arch=cli_args.arch,
+					cross_compile=cli_args.cross_compile,
+					qemu_config_path=cli_args.qemu_config)
 
 		request = KunitRequest(cli_args.raw_output,
 				       cli_args.timeout,
@@ -289,7 +318,11 @@ def main(argv, linux=None):
 			os.mkdir(cli_args.build_dir)
 
 		if not linux:
-			linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir, kunitconfig_path=cli_args.kunitconfig)
+			linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir,
+					kunitconfig_path=cli_args.kunitconfig,
+					arch=cli_args.arch,
+					cross_compile=cli_args.cross_compile,
+					qemu_config_path=cli_args.qemu_config)
 
 		request = KunitConfigRequest(cli_args.build_dir,
 					     cli_args.make_options)
@@ -301,7 +334,11 @@ def main(argv, linux=None):
 			sys.exit(1)
 	elif cli_args.subcommand == 'build':
 		if not linux:
-			linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir, kunitconfig_path=cli_args.kunitconfig)
+			linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir,
+					kunitconfig_path=cli_args.kunitconfig,
+					arch=cli_args.arch,
+					cross_compile=cli_args.cross_compile,
+					qemu_config_path=cli_args.qemu_config)
 
 		request = KunitBuildRequest(cli_args.jobs,
 					    cli_args.build_dir,
@@ -315,7 +352,11 @@ def main(argv, linux=None):
 			sys.exit(1)
 	elif cli_args.subcommand == 'exec':
 		if not linux:
-			linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir)
+			linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir,
+					kunitconfig_path=cli_args.kunitconfig,
+					arch=cli_args.arch,
+					cross_compile=cli_args.cross_compile,
+					qemu_config_path=cli_args.qemu_config)
 
 		exec_request = KunitExecRequest(cli_args.timeout,
 						cli_args.build_dir,
tools/testing/kunit/kunit_config.py +6 -1
index 1e2683dcc0e7a..c77c7d2ef6220 100644
@@ -52,8 +52,13 @@ class Kconfig(object):
 				return False
 		return True
 
+	def merge_in_entries(self, other: 'Kconfig') -> None:
+		if other.is_subset_of(self):
+			return
+		self._entries = list(self.entries().union(other.entries()))
+
 	def write_to_file(self, path: str) -> None:
-		with open(path, 'w') as f:
+		with open(path, 'a+') as f:
 			for entry in self.entries():
 				f.write(str(entry) + '\n')
 
tools/testing/kunit/kunit_kernel.py +148 -27

269 lines not on this page · raw message

tools/testing/kunit/kunit_tool_test.py +13 -5
index 2e809dd956a77..a3b7f68e1cf9f 100755
@@ -303,7 +303,7 @@ class KUnitMainTest(unittest.TestCase):
 
 		self.linux_source_mock = mock.Mock()
 		self.linux_source_mock.build_reconfig = mock.Mock(return_value=True)
-		self.linux_source_mock.build_um_kernel = mock.Mock(return_value=True)
+		self.linux_source_mock.build_kernel = mock.Mock(return_value=True)
 		self.linux_source_mock.run_kernel = mock.Mock(return_value=all_passed_log)
 
 	def test_config_passes_args_pass(self):
@@ -314,7 +314,7 @@ class KUnitMainTest(unittest.TestCase):
 	def test_build_passes_args_pass(self):
 		kunit.main(['build'], self.linux_source_mock)
 		self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 0)
-		self.linux_source_mock.build_um_kernel.assert_called_once_with(False, 8, '.kunit', None)
+		self.linux_source_mock.build_kernel.assert_called_once_with(False, 8, '.kunit', None)
 		self.assertEqual(self.linux_source_mock.run_kernel.call_count, 0)
 
 	def test_exec_passes_args_pass(self):
@@ -396,7 +396,7 @@ class KUnitMainTest(unittest.TestCase):
 	def test_build_builddir(self):
 		build_dir = '.kunit'
 		kunit.main(['build', '--build_dir', build_dir], self.linux_source_mock)
-		self.linux_source_mock.build_um_kernel.assert_called_once_with(False, 8, build_dir, None)
+		self.linux_source_mock.build_kernel.assert_called_once_with(False, 8, build_dir, None)
 
 	def test_exec_builddir(self):
 		build_dir = '.kunit'
@@ -410,14 +410,22 @@ class KUnitMainTest(unittest.TestCase):
 		mock_linux_init.return_value = self.linux_source_mock
 		kunit.main(['run', '--kunitconfig=mykunitconfig'])
 		# Just verify that we parsed and initialized it correctly here.
-		mock_linux_init.assert_called_once_with('.kunit', kunitconfig_path='mykunitconfig')
+		mock_linux_init.assert_called_once_with('.kunit',
+							kunitconfig_path='mykunitconfig',
+							arch='um',
+							cross_compile=None,
+							qemu_config_path=None)
 
 	@mock.patch.object(kunit_kernel, 'LinuxSourceTree')
 	def test_config_kunitconfig(self, mock_linux_init):
 		mock_linux_init.return_value = self.linux_source_mock
 		kunit.main(['config', '--kunitconfig=mykunitconfig'])
 		# Just verify that we parsed and initialized it correctly here.
-		mock_linux_init.assert_called_once_with('.kunit', kunitconfig_path='mykunitconfig')
+		mock_linux_init.assert_called_once_with('.kunit',
+							kunitconfig_path='mykunitconfig',
+							arch='um',
+							cross_compile=None,
+							qemu_config_path=None)
 
 if __name__ == '__main__':
 	unittest.main()
tools/testing/kunit/qemu_config.py new file +16 -0
new file mode 100644
index 0000000000000..1672f6184e952
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Collection of configs for building non-UML kernels and running them on QEMU.
+#
+# Copyright (C) 2021, Google LLC.
+# Author: Brendan Higgins <[email protected]>
+
+from collections import namedtuple
+
+
+QemuArchParams = namedtuple('QemuArchParams', ['linux_arch',
+					       'kconfig',
+					       'qemu_arch',
+					       'kernel_path',
+					       'kernel_command_line',
+					       'extra_qemu_params'])
tools/testing/kunit/qemu_configs/alpha.py new file +10 -0
new file mode 100644
index 0000000000000..5d0c0cff03bd2
@@ -0,0 +1,10 @@
+from ..qemu_config import QemuArchParams
+
+QEMU_ARCH = QemuArchParams(linux_arch='alpha',
+			   kconfig='''
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y''',
+			   qemu_arch='alpha',
+			   kernel_path='arch/alpha/boot/vmlinux',
+			   kernel_command_line='console=ttyS0',
+			   extra_qemu_params=[''])
tools/testing/kunit/qemu_configs/arm.py new file +13 -0
new file mode 100644
index 0000000000000..b9c2a35e02962
@@ -0,0 +1,13 @@
+from ..qemu_config import QemuArchParams
+
+QEMU_ARCH = QemuArchParams(linux_arch='arm',
+			   kconfig='''
+CONFIG_ARCH_VIRT=y
+CONFIG_SERIAL_AMBA_PL010=y
+CONFIG_SERIAL_AMBA_PL010_CONSOLE=y
+CONFIG_SERIAL_AMBA_PL011=y
+CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''',
+			   qemu_arch='arm',
+			   kernel_path='arch/arm/boot/zImage',
+			   kernel_command_line='console=ttyAMA0',
+			   extra_qemu_params=['-machine virt'])
tools/testing/kunit/qemu_configs/arm64.py new file +12 -0
new file mode 100644
index 0000000000000..517c04459f47a
@@ -0,0 +1,12 @@
+from ..qemu_config import QemuArchParams
+
+QEMU_ARCH = QemuArchParams(linux_arch='arm64',
+			   kconfig='''
+CONFIG_SERIAL_AMBA_PL010=y
+CONFIG_SERIAL_AMBA_PL010_CONSOLE=y
+CONFIG_SERIAL_AMBA_PL011=y
+CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''',
+			   qemu_arch='aarch64',
+			   kernel_path='arch/arm64/boot/Image.gz',
+			   kernel_command_line='console=ttyAMA0',
+			   extra_qemu_params=['-machine virt', '-cpu cortex-a57'])
tools/testing/kunit/qemu_configs/i386.py new file +10 -0
new file mode 100644
index 0000000000000..aed3ffd3937dd
@@ -0,0 +1,10 @@
+from ..qemu_config import QemuArchParams
+
+QEMU_ARCH = QemuArchParams(linux_arch='i386',
+			   kconfig='''
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y''',
+			   qemu_arch='x86_64',
+			   kernel_path='arch/x86/boot/bzImage',
+			   kernel_command_line='console=ttyS0',
+			   extra_qemu_params=[''])
tools/testing/kunit/qemu_configs/powerpc.py new file +12 -0
new file mode 100644
index 0000000000000..35e9de24f0db7
@@ -0,0 +1,12 @@
+from ..qemu_config import QemuArchParams
+
+QEMU_ARCH = QemuArchParams(linux_arch='powerpc',
+			   kconfig='''
+CONFIG_PPC64=y
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_HVC_CONSOLE=y''',
+			   qemu_arch='ppc64',
+			   kernel_path='vmlinux',
+			   kernel_command_line='console=ttyS0',
+			   extra_qemu_params=['-M pseries', '-cpu power8'])
tools/testing/kunit/qemu_configs/riscv.py new file +31 -0
new file mode 100644
index 0000000000000..9e528087cd7c5
@@ -0,0 +1,31 @@
+from ..qemu_config import QemuArchParams
+import os
+import os.path
+import sys
+
+GITHUB_OPENSBI_URL = 'https://github.com/qemu/qemu/raw/master/pc-bios/opensbi-riscv64-generic-fw_dynamic.bin'
+OPENSBI_FILE = os.path.basename(GITHUB_OPENSBI_URL)
+
+if not os.path.isfile(OPENSBI_FILE):
+	print('\n\nOpenSBI file is not in the current working directory.\n'
+	      'Would you like me to download it for you from:\n' + GITHUB_OPENSBI_URL + ' ?\n')
+	response = input('yes/[no]: ')
+	if response.strip() == 'yes':
+		os.system('wget ' + GITHUB_OPENSBI_URL)
+	else:
+		sys.exit()
+
+QEMU_ARCH = QemuArchParams(linux_arch='riscv',
+			   kconfig='''
+CONFIG_SOC_VIRT=y
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_SERIAL_OF_PLATFORM=y
+CONFIG_SERIAL_EARLYCON_RISCV_SBI=y''',
+			   qemu_arch='riscv64',
+			   kernel_path='arch/riscv/boot/Image',
+			   kernel_command_line='console=ttyS0',
+			   extra_qemu_params=[
+					   '-machine virt',
+					   '-cpu rv64',
+					   '-bios opensbi-riscv64-generic-fw_dynamic.bin'])
tools/testing/kunit/qemu_configs/s390.py new file +14 -0
new file mode 100644
index 0000000000000..e310bd5211135
@@ -0,0 +1,14 @@
+from ..qemu_config import QemuArchParams
+
+QEMU_ARCH = QemuArchParams(linux_arch='s390',
+			   kconfig='''
+CONFIG_EXPERT=y
+CONFIG_TUNE_ZEC12=y
+CONFIG_NUMA=y
+CONFIG_MODULES=y''',
+			   qemu_arch='s390x',
+			   kernel_path='arch/s390/boot/bzImage',
+			   kernel_command_line='console=ttyS0',
+			   extra_qemu_params=[
+					   '-machine s390-ccw-virtio',
+					   '-cpu qemu',])
tools/testing/kunit/qemu_configs/sparc.py new file +10 -0
new file mode 100644
index 0000000000000..27f474e7ad6e7
@@ -0,0 +1,10 @@
+from ..qemu_config import QemuArchParams
+
+QEMU_ARCH = QemuArchParams(linux_arch='sparc',
+			   kconfig='''
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y''',
+			   qemu_arch='sparc',
+			   kernel_path='arch/sparc/boot/zImage',
+			   kernel_command_line='console=ttyS0 mem=256M',
+			   extra_qemu_params=['-m 256'])
tools/testing/kunit/qemu_configs/x86_64.py new file +10 -0
new file mode 100644
index 0000000000000..77ab1aeee8a37
@@ -0,0 +1,10 @@
+from ..qemu_config import QemuArchParams
+
+QEMU_ARCH = QemuArchParams(linux_arch='x86_64',
+			   kconfig='''
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y''',
+			   qemu_arch='x86_64',
+			   kernel_path='arch/x86/boot/bzImage',
+			   kernel_command_line='console=ttyS0',
+			   extra_qemu_params=[''])
Signature · 1 line
2.31.1.818.g46aad6cb9e-goog
#5 lore raw

PATCH v2 4/4 Documentation: kunit: document support for QEMU in kunit_tool

In reply to Brendan Higgins

Document QEMU support, what it does, and how to use it in kunit_tool.

---
Changes since last revision:

- Finally, I added a new section to the kunit_tool documentation to document the new command line flags I added.

---
 Documentation/dev-tools/kunit/kunit-tool.rst | 48 +++++++++++++++++++
 Documentation/dev-tools/kunit/usage.rst      | 50 +++++++++++++++-----
 2 files changed, 87 insertions(+), 11 deletions(-)
Documentation/dev-tools/kunit/kunit-tool.rst +48 -0
index 4247b7420e3ba..c7ff9afe407a5 100644
@@ -145,6 +145,54 @@ to run KUnit resource tests, you could use:
 
 This uses the standard glob format for wildcards.
 
+Running Tests on QEMU
+=====================
+
+kunit_tool supports running tests on QEMU as well as via UML (as mentioned
+elsewhere). The default way of running tests on QEMU requires two flags:
+
+``--arch``
+	Selects a collection of configs (Kconfig as well as QEMU configs
+	options, etc) that allow KUnit tests to be run on the specified
+	architecture in a minimal way; this is usually not much slower than
+	using UML. The architecture argument is the same as the name of the
+	option passed to the ``ARCH`` variable used by Kbuild. Not all
+	architectures are currently supported by this flag, but can be handled
+	by the ``--qemu_config`` discussed later. If ``um`` is passed (or this
+	this flag is ignored) the tests will run via UML. Non-UML architectures,
+	e.g. i386, x86_64, arm, um, etc. Non-UML run on QEMU.
+
+``--cross_compile``
+	Specifies the use of a toolchain by Kbuild. The argument passed here is
+	the same passed to the ``CROSS_COMPILE`` variable used by Kbuild. As a
+	reminder this will be the prefix for the toolchain binaries such as gcc
+	for example ``sparc64-linux-gnu-`` if you have the sparc toolchain
+	installed on your system, or
+	``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux-``
+	if you have downloaded the microblaze toolchain from the 0-day website
+	to a directory in your home directory called ``toolchains``.
+
+In many cases it is likely that you may want to run an architecture which is
+not supported by the ``--arch`` flag, or you may want to just run KUnit tests
+on QEMU using a non-default configuration. For this use case, you can write
+your own QemuConfig. These QemuConfigs are written in Python. They must have an
+import line ``from ..qemu_config import QemuArchParams`` at the top of the file
+and the file must contain a variable called ``QEMU_ARCH`` that has an instance
+of ``QemuArchParams`` assigned to it. An example can be seen in
+``tools/testing/kunit/qemu_configs/x86_64.py``.
+
+Once you have a QemuConfig you can pass it into kunit_tool using the
+``--qemu_config`` flag; when used this flag replaces the ``--arch`` flag. If we
+were to do this with the ``x86_64.py`` example from above, the invocation would
+look something like this:
+
+.. code-block:: bash
+
+	./tools/testing/kunit/kunit.py run \
+		--timeout=60 \
+		--jobs=12 \
+		--qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py
+
 Other Useful Options
 ====================
 
Documentation/dev-tools/kunit/usage.rst +39 -11

56 lines not on this page · raw message

Signature · 1 line
2.31.1.818.g46aad6cb9e-goog
Reviewed-by on v2 4/4
#6 lore raw

Re: [PATCH v2 4/4] Documentation: kunit: document support for QEMU in kunit_tool

Quoted 10 lines · Brendan Higgins

Document QEMU support, what it does, and how to use it in kunit_tool.

---
Changes since last revision:

- Finally, I added a new section to the kunit_tool documentation to document the new command line flags I added.

Thanks! This looks good to me, modulo a couple of nitpicky spelling/grammar notes below.

Cheers
-- David

Quoted 12 lines
---
 Documentation/dev-tools/kunit/kunit-tool.rst | 48 +++++++++++++++++++
 Documentation/dev-tools/kunit/usage.rst      | 50 +++++++++++++++-----
 2 files changed, 87 insertions(+), 11 deletions(-)
Documentation/dev-tools/kunit/kunit-tool.rst +0 -0
index 4247b7420e3ba..c7ff9afe407a5 100644
@@ -145,6 +145,54 @@ to run KUnit resource tests, you could use:
 
 This uses the standard glob format for wildcards.

Documentation/dev-tools/kunit/kunit-tool.rst to run KUnit resource tests, you could use: · line 156

Quoted 10 lines
Documentation/dev-tools/kunit/kunit-tool.rst +9 -0
 
+Running Tests on QEMU
+=====================
+
+kunit_tool supports running tests on QEMU as well as via UML (as mentioned
+elsewhere). The default way of running tests on QEMU requires two flags:
+
+``--arch``
+       Selects a collection of configs (Kconfig as well as QEMU configs
+       options, etc) that allow KUnit tests to be run on the specified

Nit: should "configs options" be "config options"?

Documentation/dev-tools/kunit/kunit-tool.rst to run KUnit resource tests, you could use: · line 161

Quoted 5 lines
Documentation/dev-tools/kunit/kunit-tool.rst +5 -0
+       architecture in a minimal way; this is usually not much slower than
+       using UML. The architecture argument is the same as the name of the
+       option passed to the ``ARCH`` variable used by Kbuild. Not all
+       architectures are currently supported by this flag, but can be handled
+       by the ``--qemu_config`` discussed later. If ``um`` is passed (or this

Nit: "by the ``--qemu_config`` _option_", or just "by ``--qemu_config``".

Quoted 48 lines
Documentation/dev-tools/kunit/kunit-tool.rst +34 -0
+       this flag is ignored) the tests will run via UML. Non-UML architectures,
+       e.g. i386, x86_64, arm, um, etc. Non-UML run on QEMU.
+
+``--cross_compile``
+       Specifies the use of a toolchain by Kbuild. The argument passed here is
+       the same passed to the ``CROSS_COMPILE`` variable used by Kbuild. As a
+       reminder this will be the prefix for the toolchain binaries such as gcc
+       for example ``sparc64-linux-gnu-`` if you have the sparc toolchain
+       installed on your system, or
+       ``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux-``
+       if you have downloaded the microblaze toolchain from the 0-day website
+       to a directory in your home directory called ``toolchains``.
+
+In many cases it is likely that you may want to run an architecture which is
+not supported by the ``--arch`` flag, or you may want to just run KUnit tests
+on QEMU using a non-default configuration. For this use case, you can write
+your own QemuConfig. These QemuConfigs are written in Python. They must have an
+import line ``from ..qemu_config import QemuArchParams`` at the top of the file
+and the file must contain a variable called ``QEMU_ARCH`` that has an instance
+of ``QemuArchParams`` assigned to it. An example can be seen in
+``tools/testing/kunit/qemu_configs/x86_64.py``.
+
+Once you have a QemuConfig you can pass it into kunit_tool using the
+``--qemu_config`` flag; when used this flag replaces the ``--arch`` flag. If we
+were to do this with the ``x86_64.py`` example from above, the invocation would
+look something like this:
+
+.. code-block:: bash
+
+       ./tools/testing/kunit/kunit.py run \
+               --timeout=60 \
+               --jobs=12 \
+               --qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py
+
 Other Useful Options
 ====================
 
Documentation/dev-tools/kunit/usage.rst +0 -3
index 650f99590df57..888c341701da4 100644
@@ -609,17 +609,45 @@ non-UML architectures:
 None of these are reasons not to run your KUnit tests on real hardware; they are
 only things to be aware of when doing so.
 
-The biggest impediment will likely be that certain KUnit features and
-infrastructure may not support your target environment. For example, at this
-time the KUnit Wrapper (``tools/testing/kunit/kunit.py``) does not work outside

Documentation/dev-tools/kunit/usage.rst non-UML architectures: · line 613

Quoted 10 lines
Documentation/dev-tools/kunit/usage.rst +2 -8
-of UML. Unfortunately, there is no way around this. Using UML (or even just a
-particular architecture) allows us to make a lot of assumptions that make it
-possible to do things which might otherwise be impossible.
-
-Nevertheless, all core KUnit framework features are fully supported on all
-architectures, and using them is straightforward: all you need to do is to take
-your kunitconfig, your Kconfig options for the tests you would like to run, and
-merge them into whatever config your are using for your platform. That's it!
+Currently, the KUnit Wrapper (``tools/testing/kunit/kunit.py``) (aka
+kunit_tool) only fully supports running tests inside of UML and QEMU; however,

I was going to whinge a bit about the fact that we have three names for the "KUnit Wrapper" here, and "KUnit Wrapper" didn't seem to be used much. Then I looked at the rest of the documentation, and "KUnit Wrapper" is used all over the place.

I think it'd be nice to consolidate on one (or, if we have to, two) names for this, but this doc is probably not the place to do so.

Documentation/dev-tools/kunit/usage.rst non-UML architectures: · line 617

Quoted 4 lines
Documentation/dev-tools/kunit/usage.rst +4 -0
+this is only due to our own time limitations as humans working on KUnit. It is
+entirely possible to support other emulators and even actual hardware, but for
+now QEMU and UML is what is fully supported within the KUnit Wrapper. Again, to
+be clear, this is just the Wrapper. The actualy KUnit tests and the KUnit

This maybe could be a little more succinct.
"for now only QEMU and UML are supported by the KUnit Wrapper. The KUnit library and most KUnit tests are fully architecture agnostic..."

Also, "actualy" -> "actually".

Quoted 23 lines
Documentation/dev-tools/kunit/usage.rst +23 -0
+library they are written in is fully architecture agnostic and can be used in
+virtually any setup, you just won't have the benefit of typing a single command
+out of the box and having everything magically work perfectly.
+
+Again, all core KUnit framework features are fully supported on all
+architectures, and using them is straightforward: Most popular architectures
+are supported directly in the KUnit Wrapper via QEMU. Currently, supported
+architectures on QEMU include:
+
+*   i386
+*   x86_64
+*   arm
+*   arm64
+*   alpha
+*   powerpc
+*   riscv
+*   s390
+*   sparc
+
+In order to run KUnit tests on one of these architectures via QEMU with the
+KUnit wrapper, all you need to do is specify the flags ``--arch`` and
+``--cross_compile`` when invoking the KUnit Wrapper. For example, we could run
+the default KUnit tests on ARM in the following manner (assuming we have an ARM

Documentation/dev-tools/kunit/usage.rst non-UML architectures: · line 650

Quoted 10 lines
Documentation/dev-tools/kunit/usage.rst +10 -0
+toolchain installed):
+
+.. code-block:: bash
+
+       tools/testing/kunit/kunit.py run --timeout=60 --jobs=12 --arch=arm --cross_compile=arm-linux-gnueabihf-
+
+Alternatively, if you want to run your tests on real hardware or in some other
+emulation environment, all you need to do is to take your kunitconfig, your
+Kconfig options for the tests you would like to run, and merge them into
+whatever config your are using for your platform. That's it!

Nit "your are" -> "you are" / "you're"

Quoted 6 lines
Documentation/dev-tools/kunit/usage.rst +0 -1
 For example, let's say you have the following kunitconfig:
 
--

2.31.1.818.g46aad6cb9e-goog