kunit: create a centralized executor to dispatch all KUnit tests
Subsystems (10)
v4 cover kunit: create a centralized executor to dispatch all KUnit tests
## TL;DR
This patchset adds a centralized executor to dispatch tests rather than relying on late_initcall to schedule each test suite separately along with a couple of new features that depend on it.
Also, sorry for the extreme delay in getting this out. Part of the delay came from finding that there were actually several architectures that the previous revision of this patchset didn't work on, so I went through and attempted to test this patchset on every architecture - more on that later.
## What am I trying to do?
Conceptually, I am trying to provide a mechanism by which test suites can be grouped together so that they can be reasoned about collectively. The last two of three patches in this series add features which depend on this:
PATCH 8/11 Prints out a test plan[1] right before KUnit tests are run;
this is valuable because it makes it possible for a test harness to detect whether the number of tests run matches the number of tests expected to be run, ensuring that no tests silently failed. The test plan includes a count of tests that will run. With the centralized executor, the tests are located in a single data structure and thus can be counted.
PATCH 9/11 Add a new kernel command-line option which allows the user to
specify that the kernel poweroff, halt, or reboot after completing all KUnit tests; this is very handy for running KUnit tests on UML or a VM so that the UML/VM process exits cleanly immediately after running all tests without needing a special initramfs. The centralized executor provides a definitive point when all tests have completed and the poweroff, halt, or reboot could occur.
In addition, by dispatching tests from a single location, we can guarantee that all KUnit tests run after late_init is complete, which was a concern during the initial KUnit patchset review (this has not been a problem in practice, but resolving with certainty is nevertheless desirable).
Other use cases for this exist, but the above features should provide an idea of the value that this could provide.
## Changes since last revision:
- On the last revision I got some messages from 0day that showed that this patchset didn't work on several architectures, one issue that this patchset addresses is that we were aligning both memory segments as well as structures in the segments to specific byte boundaries which was incorrect.
- The issue mentioned above also caused me to test on additional architectures which revealed that some architectures other than UML do not use the default init linker section macro that most architectures use. There are now several new patches (2, 3, 4, and 6).
- Fixed a formatting consistency issue in the kernel params documentation patch (9/9).
- Add a brief blurb on how and when the kunit_test_suite macro works.
## Remaining work to be done:
The only architecture for which I was able to get a compiler, but was apparently unable to get KUnit into a section that the executor to see was m68k - not sure why.
Alan Maguire (1):
kunit: test: create a single centralized executor for all tests
Brendan Higgins (10):
vmlinux.lds.h: add linker section for KUnit test suites
arch: arm64: add linker section for KUnit test suites
arch: microblaze: add linker section for KUnit test suites
arch: powerpc: add linker section for KUnit test suites
arch: um: add linker section for KUnit test suites
arch: xtensa: add linker section for KUnit test suites
init: main: add KUnit to kernel init
kunit: test: add test plan to KUnit TAP format
Documentation: Add kunit_shutdown to kernel-parameters.txt
Documentation: kunit: add a brief blurb about kunit_test_suite
.../admin-guide/kernel-parameters.txt | 8 ++ Documentation/dev-tools/kunit/usage.rst | 5 ++ arch/arm64/kernel/vmlinux.lds.S | 3 + arch/microblaze/kernel/vmlinux.lds.S | 4 + arch/powerpc/kernel/vmlinux.lds.S | 4 + arch/um/include/asm/common.lds.S | 4 + arch/xtensa/kernel/vmlinux.lds.S | 4 + include/asm-generic/vmlinux.lds.h | 8 ++ include/kunit/test.h | 73 ++++++++++++----- init/main.c | 4 + lib/kunit/Makefile | 3 +- lib/kunit/executor.c | 63 +++++++++++++++ lib/kunit/test.c | 13 +-- tools/testing/kunit/kunit_kernel.py | 2 +- tools/testing/kunit/kunit_parser.py | 74 +++++++++++++++--- .../test_is_test_passed-all_passed.log | Bin 1562 -> 1567 bytes .../test_data/test_is_test_passed-crash.log | Bin 3016 -> 3021 bytes .../test_data/test_is_test_passed-failure.log | Bin 1700 -> 1705 bytes 18 files changed, 226 insertions(+), 46 deletions(-) create mode 100644 lib/kunit/executor.c
base-commit: 4333a9b0b67bb4e8bcd91bdd80da80b0ec151162
prerequisite-patch-id: 2d4b5aa9fa8ada9ae04c8584b47c299a822b9455
prerequisite-patch-id: 582b6d9d28ce4b71628890ec832df6522ca68de0
These patches are available for download with dependencies here:
https://kunit-review.googlesource.com/c/linux/+/3829
[1] https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan
[2] https://patchwork.kernel.org/patch/11383635/
Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
PATCH v4 1/11 vmlinux.lds.h: add linker section for KUnit test suites
Add a linker section where KUnit can put references to its test suites. This patch is the first step in transitioning to dispatching all KUnit tests from a centralized executor rather than having each as its own separate late_initcall.
--- include/asm-generic/vmlinux.lds.h | 8 ++++++++ 1 file changed, 8 insertions(+)
include/asm-generic/vmlinux.lds.h +8 -0
@@ -881,6 +881,13 @@ KEEP(*(.con_initcall.init)) \ __con_initcall_end = .; +/* Alignment must be consistent with (kunit_suite *) in include/kunit/test.h */ +#define KUNIT_TEST_SUITES \ + . = ALIGN(8); \ + __kunit_suites_start = .; \ + KEEP(*(.kunit_test_suites)) \ + __kunit_suites_end = .; +#ifdef CONFIG_BLK_DEV_INITRD #define INIT_RAM_FS \ . = ALIGN(4); \
@@ -1056,6 +1063,7 @@INIT_CALLS \ CON_INITCALL \INIT_RAM_FS \ + KUNIT_TEST_SUITES \ }#define BSS_SECTION (sbss_align, bss_align, stop_align) \
Signature · 7 lines
2.27.0.212.ge8ba1cc988-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 2/11 arch: arm64: add linker section for KUnit test suites
In reply to Brendan HigginsAdd a linker section to arm64 where KUnit can put references to its test suites. This patch is an early step in transitioning to dispatching all KUnit tests from a centralized executor rather than having each as its own separate late_initcall.
--- arch/arm64/kernel/vmlinux.lds.S | 3 +++ 1 file changed, 3 insertions(+)
arch/arm64/kernel/vmlinux.lds.S +3 -0
@@ -181,6 +181,9 @@ SECTIONS INIT_RAM_FS *(.init.rodata.* .init.bss) /* from the EFI stub */ } + .kunit_test_suites : { + KUNIT_TEST_SUITES + } .exit.data : { EXIT_DATA }
Signature · 7 lines
2.27.0.212.ge8ba1cc988-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 3/11 arch: microblaze: add linker section for KUnit test suites
In reply to Brendan HigginsAdd a linker section to microblaze where KUnit can put references to its test suites. This patch is an early step in transitioning to dispatching all KUnit tests from a centralized executor rather than having each as its own separate late_initcall.
--- arch/microblaze/kernel/vmlinux.lds.S | 4 ++++ 1 file changed, 4 insertions(+)
arch/microblaze/kernel/vmlinux.lds.S +4 -0
@@ -128,6 +128,10 @@ SECTIONS { __init_end = .; + .kunit_test_suites : { + KUNIT_TEST_SUITES + } + .bss ALIGN (PAGE_SIZE) : AT(ADDR(.bss) - LOAD_OFFSET) { /* page aligned when MMU used */ __bss_start = . ;
Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
PATCH v4 4/11 arch: powerpc: add linker section for KUnit test suites
In reply to Brendan HigginsAdd a linker section to powerpc where KUnit can put references to its test suites. This patch is an early step in transitioning to dispatching all KUnit tests from a centralized executor rather than having each as its own separate late_initcall.
--- arch/powerpc/kernel/vmlinux.lds.S | 4 ++++ 1 file changed, 4 insertions(+)
arch/powerpc/kernel/vmlinux.lds.S +4 -0
@@ -202,6 +202,10 @@ SECTIONS CON_INITCALL } + .kunit_test_suites : { + KUNIT_TEST_SUITES + } + . = ALIGN(8); __ftr_fixup : AT(ADDR(__ftr_fixup) - LOAD_OFFSET) { __start___ftr_fixup = .;
Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
PATCH v4 5/11 arch: um: add linker section for KUnit test suites
In reply to Brendan HigginsAdd a linker section to UML where KUnit can put references to its test suites. This patch is an early step in transitioning to dispatching all KUnit tests from a centralized executor rather than having each as its own separate late_initcall.
--- arch/um/include/asm/common.lds.S | 4 ++++ 1 file changed, 4 insertions(+)
arch/um/include/asm/common.lds.S +4 -0
@@ -52,6 +52,10 @@ CON_INITCALL } + .kunit_test_suites : { + KUNIT_TEST_SUITES + } + .exitcall : { __exitcall_begin = .; *(.exitcall.exit)
Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
PATCH v4 6/11 arch: xtensa: add linker section for KUnit test suites
In reply to Brendan HigginsAdd a linker section to xtensa where KUnit can put references to its test suites. This patch is an early step in transitioning to dispatching all KUnit tests from a centralized executor rather than having each as its own separate late_initcall.
--- arch/xtensa/kernel/vmlinux.lds.S | 4 ++++ 1 file changed, 4 insertions(+)
arch/xtensa/kernel/vmlinux.lds.S +4 -0
@@ -216,6 +216,10 @@ SECTIONS INIT_RAM_FS } + .kunit_test_suites : { + KUNIT_TEST_SUITES + } + PERCPU_SECTION(XCHAL_ICACHE_LINESIZE) /* We need this dummy segment here */
Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
PATCH v4 7/11 kunit: test: create a single centralized executor for all tests
In reply to Brendan HigginsFrom: Alan Maguire <[email protected]>
Add a centralized executor to dispatch tests rather than relying on late_initcall to schedule each test suite separately. Centralized execution is for built-in tests only; modules will execute tests when loaded.
--- include/kunit/test.h | 64 ++++++++++++++++++++++++++++++-------------- lib/kunit/Makefile | 3 ++- lib/kunit/executor.c | 28 +++++++++++++++++++ lib/kunit/test.c | 2 +- 4 files changed, 75 insertions(+), 22 deletions(-) create mode 100644 lib/kunit/executor.c
include/kunit/test.h +44 -20
@@ -224,7 +224,7 @@ size_t kunit_suite_num_test_cases(struct kunit_suite *suite);unsigned int kunit_test_case_num (struct kunit_suite *suite,struct kunit_case *test_case); -int __kunit_test_suites_init (struct kunit_suite **suites); +int __kunit_test_suites_init (struct kunit_suite *const *const suites);void __kunit_test_suites_exit (struct kunit_suite **suites);
@@ -237,34 +237,58 @@ void __kunit_test_suites_exit(struct kunit_suite **suites);* Registers @suites_list with the test framework. See &struct kunit_suite for * more information. * -* When builtin, KUnit tests are all run as late_initcalls; this means -* that they cannot test anything where tests must run at a different init -* phase. One significant restriction resulting from this is that KUnit -* cannot reliably test anything that is initialize in the late_init phase; -* another is that KUnit is useless to test things that need to be run in -* an earlier init phase. -* -* An alternative is to build the tests as a module. Because modules -* do not support multiple late_initcall()s, we need to initialize an -* array of suites for a module. -* -* TODO([email protected]): Don't run all KUnit tests as -* late_initcalls. I have some future work planned to dispatch all KUnit -* tests from the same place, and at the very least to do so after -* everything else is definitely initialized. +* If a test suite is built-in, module_init() gets translated into +* an initcall which we don't want as the idea is that for builtins +* the executor will manage execution. So ensure we do not define +* module_{init|exit} functions for the builtin case when registering +* suites via kunit_test_suites() below. */ -#define kunit_test_suites(suites_list...) \ - static struct kunit_suite *suites[] = {suites_list, NULL}; \ - static int kunit_test_suites_init(void) \ +#ifdef MODULE +#define kunit_test_suites_for_module (__suites) \ + static int __init kunit_test_suites_init(void) \ { \ + struct kunit_suite *suites[] = (__suites); \ return __kunit_test_suites_init(suites); \ } \ - late_initcall(kunit_test_suites_init); \ + module_init(kunit_test_suites_init); \ + \ static void __exit kunit_test_suites_exit(void) \ { \ return __kunit_test_suites_exit(suites); \ } \ module_exit(kunit_test_suites_exit) +#else +#define kunit_test_suites_for_module (__suites) +#endif /* MODULE */ + +#define __kunit_test_suites (unique_array, unique_suites, ...) \ + static struct kunit_suite *unique_array[] = { __VA_ARGS__, NULL }; \ + kunit_test_suites_for_module(unique_array); \ + static struct kunit_suite **unique_suites \ + __used __section(.kunit_test_suites) = unique_array + +/** +* kunit_test_suites() - used to register one or more &struct kunit_suite +* with KUnit. +* +* @suites: a statically allocated list of &struct kunit_suite. +* +* Registers @suites with the test framework. See &struct kunit_suite for +* more information. +* +* When builtin, KUnit tests are all run via executor; this is done +* by placing the array of struct kunit_suite * in the .kunit_test_suites +* ELF section. +* +* An alternative is to build the tests as a module. Because modules do not +* support multiple initcall()s, we need to initialize an array of suites for a +* module. +* +*/ +#define kunit_test_suites (...) \ + __kunit_test_suites(__UNIQUE_ID(array), \ + __UNIQUE_ID(suites), \ + __VA_ARGS__)#define kunit_test_suite (suite) kunit_test_suites(&suite)
lib/kunit/Makefile +2 -1
@@ -3,7 +3,8 @@ obj-$(CONFIG_KUNIT) += kunit.o kunit-objs += test.o \ string-stream.o \ assert.o \ - try-catch.o + try-catch.o \ + executor.o ifeq ($(CONFIG_KUNIT_DEBUGFS),y) kunit-objs += debugfs.o
lib/kunit/executor.c new file +28 -0
@@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <kunit/test.h> + +/* +* These symbols point to the .kunit_test_suites section and are defined in +* include/asm-generic/vmlinux.lds.h, and consequently must be extern. +*/ +extern struct kunit_suite *const *const __kunit_suites_start[]; +extern struct kunit_suite *const *const __kunit_suites_end[]; + +#if IS_BUILTIN (CONFIG_KUNIT ) + +static int kunit_run_all_tests (void ) +{ +struct kunit_suite *const *const *suites; + +for (suites = __kunit_suites_start; + suites < __kunit_suites_end; + suites++) +__kunit_test_suites_init (*suites); + +return 0 ; +} + +late_initcall (kunit_run_all_tests); + +#endif /* IS_BUILTIN(CONFIG_KUNIT) */
lib/kunit/test.c +1 -1
@@ -380,7 +380,7 @@ static void kunit_init_suite(struct kunit_suite *suite)kunit_debugfs_create_suite (suite); } -int __kunit_test_suites_init (struct kunit_suite **suites) +int __kunit_test_suites_init (struct kunit_suite *const *const suites) {unsigned int i;
Signature · 7 lines
2.27.0.212.ge8ba1cc988-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
kernel test robot
build botRe: [PATCH v4 07/11] kunit: test: create a single centralized executor for all tests
Thank you for the patch! Yet something to improve:
Hi Brendan,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on 4333a9b0b67bb4e8bcd91bdd80da80b0ec151162]
url: https://github.com/0day-ci/linux/commits/Brendan-Higgins/kunit-create-a-centralized-executor-to-dispatch-all-KUnit-tests/20200625-050434 base: 4333a9b0b67bb4e8bcd91bdd80da80b0ec151162 config: um-allmodconfig (attached as .config) compiler: gcc-9 (Debian 9.3.0-13) 9.3.0 reproduce (this is a W=1 build): # save the attached .config to linux build tree make W=1 ARCH=um
If you fix the issue, kindly add following tag as appropriate
All error/warnings (new ones prefixed by >>):
cc1: warning: arch/um/include/uapi: No such file or directory [-Wmissing-include-dirs] In file included from lib//kunit/kunit-test.c:8:
lib//kunit/kunit-test.c: In function 'kunit_test_suites_init':
>> include/kunit/test.h:250:34: error: invalid initializer
250 | struct kunit_suite *suites[] = (__suites); \
| ^
>> include/kunit/test.h:266:2: note: in expansion of macro 'kunit_test_suites_for_module'
266 | kunit_test_suites_for_module(unique_array); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> include/kunit/test.h:289:2: note: in expansion of macro '__kunit_test_suites'
289 | __kunit_test_suites(__UNIQUE_ID(array), \
| ^~~~~~~~~~~~~~~~~~~
>> lib//kunit/kunit-test.c:369:1: note: in expansion of macro 'kunit_test_suites'
369 | kunit_test_suites(&kunit_try_catch_test_suite, &kunit_resource_test_suite,
| ^~~~~~~~~~~~~~~~~
lib//kunit/kunit-test.c: In function 'kunit_test_suites_exit':
>> include/kunit/test.h:257:35: error: 'suites' undeclared (first use in this function)
257 | return __kunit_test_suites_exit(suites); \
| ^~~~~~
>> include/kunit/test.h:266:2: note: in expansion of macro 'kunit_test_suites_for_module'
266 | kunit_test_suites_for_module(unique_array); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> include/kunit/test.h:289:2: note: in expansion of macro '__kunit_test_suites'
289 | __kunit_test_suites(__UNIQUE_ID(array), \
| ^~~~~~~~~~~~~~~~~~~
>> lib//kunit/kunit-test.c:369:1: note: in expansion of macro 'kunit_test_suites'
369 | kunit_test_suites(&kunit_try_catch_test_suite, &kunit_resource_test_suite,
| ^~~~~~~~~~~~~~~~~
include/kunit/test.h:257:35: note: each undeclared identifier is reported only once for each function it appears in
257 | return __kunit_test_suites_exit(suites); \
| ^~~~~~
>> include/kunit/test.h:266:2: note: in expansion of macro 'kunit_test_suites_for_module'
266 | kunit_test_suites_for_module(unique_array); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> include/kunit/test.h:289:2: note: in expansion of macro '__kunit_test_suites'
289 | __kunit_test_suites(__UNIQUE_ID(array), \
| ^~~~~~~~~~~~~~~~~~~
>> lib//kunit/kunit-test.c:369:1: note: in expansion of macro 'kunit_test_suites' 369 | kunit_test_suites(&kunit_try_catch_test_suite, &kunit_resource_test_suite,
| ^~~~~~~~~~~~~~~~~
--
cc1: warning: arch/um/include/uapi: No such file or directory [-Wmissing-include-dirs] In file included from lib//kunit/kunit-example-test.c:9:
lib//kunit/kunit-example-test.c: In function 'kunit_test_suites_init':
>> include/kunit/test.h:250:34: error: invalid initializer
250 | struct kunit_suite *suites[] = (__suites); \
| ^
>> include/kunit/test.h:266:2: note: in expansion of macro 'kunit_test_suites_for_module'
266 | kunit_test_suites_for_module(unique_array); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> include/kunit/test.h:289:2: note: in expansion of macro '__kunit_test_suites'
289 | __kunit_test_suites(__UNIQUE_ID(array), \
| ^~~~~~~~~~~~~~~~~~~
>> lib//kunit/kunit-example-test.c:88:1: note: in expansion of macro 'kunit_test_suites'
88 | kunit_test_suites(&example_test_suite);
| ^~~~~~~~~~~~~~~~~
lib//kunit/kunit-example-test.c: In function 'kunit_test_suites_exit':
>> include/kunit/test.h:257:35: error: 'suites' undeclared (first use in this function)
257 | return __kunit_test_suites_exit(suites); \
| ^~~~~~
>> include/kunit/test.h:266:2: note: in expansion of macro 'kunit_test_suites_for_module'
266 | kunit_test_suites_for_module(unique_array); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> include/kunit/test.h:289:2: note: in expansion of macro '__kunit_test_suites'
289 | __kunit_test_suites(__UNIQUE_ID(array), \
| ^~~~~~~~~~~~~~~~~~~
>> lib//kunit/kunit-example-test.c:88:1: note: in expansion of macro 'kunit_test_suites'
88 | kunit_test_suites(&example_test_suite);
| ^~~~~~~~~~~~~~~~~
include/kunit/test.h:257:35: note: each undeclared identifier is reported only once for each function it appears in
257 | return __kunit_test_suites_exit(suites); \
| ^~~~~~
>> include/kunit/test.h:266:2: note: in expansion of macro 'kunit_test_suites_for_module'
266 | kunit_test_suites_for_module(unique_array); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> include/kunit/test.h:289:2: note: in expansion of macro '__kunit_test_suites'
289 | __kunit_test_suites(__UNIQUE_ID(array), \
| ^~~~~~~~~~~~~~~~~~~
>> lib//kunit/kunit-example-test.c:88:1: note: in expansion of macro 'kunit_test_suites'
88 | kunit_test_suites(&example_test_suite);
| ^~~~~~~~~~~~~~~~~
--
cc1: warning: arch/um/include/uapi: No such file or directory [-Wmissing-include-dirs]
In file included from include/linux/err.h:5,
from include/kunit/assert.h:12,
from include/kunit/test.h:12,
from fs/ext4/inode-test.c:7:
include/asm-generic/fixmap.h: In function 'fix_to_virt':
include/asm-generic/fixmap.h:32:19: warning: comparison of unsigned expression >= 0 is always true [-Wtype-limits]
32 | BUILD_BUG_ON(idx >= __end_of_fixed_addresses);
| ^~
include/linux/compiler.h:372:9: note: in definition of macro '__compiletime_assert'
372 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler.h:392:2: note: in expansion of macro '_compiletime_assert'
392 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
| ^~~~~~~~~~~~~~~~~~
include/linux/build_bug.h:50:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
50 | BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
| ^~~~~~~~~~~~~~~~
include/asm-generic/fixmap.h:32:2: note: in expansion of macro 'BUILD_BUG_ON'
32 | BUILD_BUG_ON(idx >= __end_of_fixed_addresses);
| ^~~~~~~~~~~~
In file included from include/linux/uaccess.h:11,
from include/linux/sched/task.h:11,
from include/linux/sched/signal.h:9,
from include/linux/rcuwait.h:6,
from include/linux/percpu-rwsem.h:7,
from include/linux/fs.h:33,
from include/linux/huge_mm.h:8,
from include/linux/mm.h:675,
from include/linux/bvec.h:13,
from include/linux/blk_types.h:10,
from include/linux/genhd.h:19,
from include/linux/blkdev.h:11,
from fs/ext4/ext4.h:21,
from fs/ext4/inode-test.c:11:
arch/um/include/asm/uaccess.h: In function '__access_ok':
arch/um/include/asm/uaccess.h:17:29: warning: comparison of unsigned expression >= 0 is always true [-Wtype-limits]
17 | (((unsigned long) (addr) >= FIXADDR_USER_START) && \
| ^~
arch/um/include/asm/uaccess.h:45:3: note: in expansion of macro '__access_ok_vsyscall'
45 | __access_ok_vsyscall(addr, size) ||
| ^~~~~~~~~~~~~~~~~~~~
In file included from fs/ext4/inode-test.c:7:
fs/ext4/inode-test.c: In function 'kunit_test_suites_init':
>> include/kunit/test.h:250:34: error: invalid initializer
250 | struct kunit_suite *suites[] = (__suites); \
| ^
>> include/kunit/test.h:266:2: note: in expansion of macro 'kunit_test_suites_for_module'
266 | kunit_test_suites_for_module(unique_array); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> include/kunit/test.h:289:2: note: in expansion of macro '__kunit_test_suites'
289 | __kunit_test_suites(__UNIQUE_ID(array), \
| ^~~~~~~~~~~~~~~~~~~
>> fs/ext4/inode-test.c:272:1: note: in expansion of macro 'kunit_test_suites'
272 | kunit_test_suites(&ext4_inode_test_suite);
| ^~~~~~~~~~~~~~~~~
fs/ext4/inode-test.c: In function 'kunit_test_suites_exit':
>> include/kunit/test.h:257:35: error: 'suites' undeclared (first use in this function)
257 | return __kunit_test_suites_exit(suites); \
| ^~~~~~
>> include/kunit/test.h:266:2: note: in expansion of macro 'kunit_test_suites_for_module'
266 | kunit_test_suites_for_module(unique_array); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> include/kunit/test.h:289:2: note: in expansion of macro '__kunit_test_suites'
289 | __kunit_test_suites(__UNIQUE_ID(array), \
| ^~~~~~~~~~~~~~~~~~~
>> fs/ext4/inode-test.c:272:1: note: in expansion of macro 'kunit_test_suites'
272 | kunit_test_suites(&ext4_inode_test_suite);
| ^~~~~~~~~~~~~~~~~
include/kunit/test.h:257:35: note: each undeclared identifier is reported only once for each function it appears in
257 | return __kunit_test_suites_exit(suites); \
| ^~~~~~
>> include/kunit/test.h:266:2: note: in expansion of macro 'kunit_test_suites_for_module'
266 | kunit_test_suites_for_module(unique_array); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> include/kunit/test.h:289:2: note: in expansion of macro '__kunit_test_suites'
289 | __kunit_test_suites(__UNIQUE_ID(array), \
| ^~~~~~~~~~~~~~~~~~~
>> fs/ext4/inode-test.c:272:1: note: in expansion of macro 'kunit_test_suites'
272 | kunit_test_suites(&ext4_inode_test_suite);
| ^~~~~~~~~~~~~~~~~vim +250 include/kunit/test.h
230
231 /**
232 * kunit_test_suites() - used to register one or more &struct kunit_suite
233 * with KUnit.
234 *
235 * @suites_list...: a statically allocated list of &struct kunit_suite.
236 *
237 * Registers @suites_list with the test framework. See &struct kunit_suite for
238 * more information.
239 *
240 * If a test suite is built-in, module_init() gets translated into
241 * an initcall which we don't want as the idea is that for builtins
242 * the executor will manage execution. So ensure we do not define
243 * module_{init|exit} functions for the builtin case when registering
244 * suites via kunit_test_suites() below.
245 */
246 #ifdef MODULE
247 #define kunit_test_suites_for_module(__suites) \
248 static int __init kunit_test_suites_init(void) \
249 { \
> 250 struct kunit_suite *suites[] = (__suites); \
251 return __kunit_test_suites_init(suites); \
252 } \
253 module_init(kunit_test_suites_init); \
254 \
255 static void __exit kunit_test_suites_exit(void) \
256 { \
> 257 return __kunit_test_suites_exit(suites); \
258 } \
259 module_exit(kunit_test_suites_exit)
260 #else
261 #define kunit_test_suites_for_module(__suites)
262 #endif /* MODULE */
263
264 #define __kunit_test_suites(unique_array, unique_suites, ...) \
265 static struct kunit_suite *unique_array[] = { __VA_ARGS__, NULL }; \
> 266 kunit_test_suites_for_module(unique_array); \
267 static struct kunit_suite **unique_suites \
268 __used __section(.kunit_test_suites) = unique_array
269
270 /**
271 * kunit_test_suites() - used to register one or more &struct kunit_suite
272 * with KUnit.
273 *
274 * @suites: a statically allocated list of &struct kunit_suite.
275 *
276 * Registers @suites with the test framework. See &struct kunit_suite for
277 * more information.
278 *
279 * When builtin, KUnit tests are all run via executor; this is done
280 * by placing the array of struct kunit_suite * in the .kunit_test_suites
281 * ELF section.
282 *
283 * An alternative is to build the tests as a module. Because modules do not
284 * support multiple initcall()s, we need to initialize an array of suites for a
285 * module.
286 *
287 */
288 #define kunit_test_suites(...) \
> 289 __kunit_test_suites(__UNIQUE_ID(array), \ 290 __UNIQUE_ID(suites), \
291 __VA_ARGS__)
292
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
PATCH v4 8/11 init: main: add KUnit to kernel init
In reply to Brendan HigginsRemove KUnit from init calls entirely, instead call directly from kernel_init().
--- include/kunit/test.h | 9 +++++++++ init/main.c | 4 ++++ lib/kunit/executor.c | 4 +--- 3 files changed, 14 insertions(+), 3 deletions(-)
include/kunit/test.h +9 -0
@@ -228,6 +228,15 @@ int __kunit_test_suites_init(struct kunit_suite * const * const suites);void __kunit_test_suites_exit (struct kunit_suite **suites); +#if IS_BUILTIN (CONFIG_KUNIT ) +int kunit_run_all_tests (void ); +#else +static inline int kunit_run_all_tests (void ) +{ +return 0 ; +} +#endif /* IS_BUILTIN(CONFIG_KUNIT) */ +/** * kunit_test_suites() - used to register one or more &struct kunit_suite * with KUnit.
init/main.c +4 -0
@@ -106,6 +106,8 @@#define CREATE_TRACE_POINTS #include <trace/events/initcall.h> +#include <kunit/test.h> +static int kernel_init (void *);extern void init_IRQ (void );
@@ -1504,6 +1506,8 @@ static noinline void __init kernel_init_freeable(void)do_basic_setup (); +kunit_run_all_tests (); +console_on_rootfs ();/*
lib/kunit/executor.c +1 -3
@@ -11,7 +11,7 @@ extern struct kunit_suite * const * const __kunit_suites_end[];#if IS_BUILTIN (CONFIG_KUNIT ) -static int kunit_run_all_tests (void ) +int kunit_run_all_tests (void ) {struct kunit_suite *const *const *suites;
@@ -23,6 +23,4 @@ static int kunit_run_all_tests(void) return 0; } -late_initcall (kunit_run_all_tests); -#endif /* IS_BUILTIN(CONFIG_KUNIT) */
Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
PATCH v4 9/11 kunit: test: add test plan to KUnit TAP format
In reply to Brendan HigginsTAP 14 allows an optional test plan to be emitted before the start of the start of testing[1]; this is valuable because it makes it possible for a test harness to detect whether the number of tests run matches the number of tests expected to be run, ensuring that no tests silently failed.
Link[1]: https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan
--- lib/kunit/executor.c | 37 +++++++++ lib/kunit/test.c | 11 --- tools/testing/kunit/kunit_kernel.py | 2 +- tools/testing/kunit/kunit_parser.py | 74 +++++++++++++++--- .../test_is_test_passed-all_passed.log | Bin 1562 -> 1567 bytes .../test_data/test_is_test_passed-crash.log | Bin 3016 -> 3021 bytes .../test_data/test_is_test_passed-failure.log | Bin 1700 -> 1705 bytes 7 files changed, 100 insertions(+), 24 deletions(-)
lib/kunit/executor.c +37 -0
@@ -1,5 +1,6 @@// SPDX-License-Identifier: GPL-2.0 +#include <linux/reboot.h> #include <kunit/test.h> /*
@@ -11,15 +12,51 @@ extern struct kunit_suite * const * const __kunit_suites_end[];#if IS_BUILTIN (CONFIG_KUNIT ) +static char *kunit_shutdown; +core_param (kunit_shutdown, kunit_shutdown, charp,0644 ); + +static void kunit_handle_shutdown (void ) +{ +if (!kunit_shutdown) +return ; + +if (!strcmp (kunit_shutdown,"poweroff" )) +kernel_power_off (); +else if (!strcmp (kunit_shutdown,"halt" )) +kernel_halt (); +else if (!strcmp (kunit_shutdown,"reboot" )) +kernel_restart (NULL ); + +} + +static void kunit_print_tap_header (void ) +{ +struct kunit_suite *const *const *suites, *const *subsuite; +int num_of_suites =0 ; + +for (suites = __kunit_suites_start; + suites < __kunit_suites_end; + suites++) +for (subsuite = *suites; *subsuite !=NULL ; subsuite++) + num_of_suites++; + +pr_info ("TAP version 14\n" ); +pr_info ("1..%d\n" , num_of_suites); +} +int kunit_run_all_tests (void ) {struct kunit_suite *const *const *suites; +kunit_print_tap_header (); +for (suites = __kunit_suites_start; suites < __kunit_suites_end; suites++)__kunit_test_suites_init (*suites); +kunit_handle_shutdown (); +return 0 ; }
lib/kunit/test.c +0 -11
@@ -19,16 +19,6 @@ static void kunit_set_failure(struct kunit *test)WRITE_ONCE (test->success, false); } -static void kunit_print_tap_version (void ) -{ -static bool kunit_has_printed_tap_version; - -if (!kunit_has_printed_tap_version) { -pr_info ("TAP version 14\n" ); - kunit_has_printed_tap_version = true; - } -} -/* * Append formatted message to log, size of which is limited to * KUNIT_LOG_SIZE bytes (including null terminating byte).
@@ -68,7 +58,6 @@ EXPORT_SYMBOL_GPL(kunit_suite_num_test_cases);static void kunit_print_subtest_start (struct kunit_suite *suite) { -kunit_print_tap_version ();kunit_log (KERN_INFO , suite,KUNIT_SUBTEST_INDENT "# Subtest: %s" , suite->name); kunit_log(KERN_INFO, suite, KUNIT_SUBTEST_INDENT "1..%zd",
tools/testing/kunit/kunit_kernel.py +1 -1
@@ -172,7 +172,7 @@ class LinuxSourceTree(object): return self.validate_config(build_dir) def run_kernel(self, args=[], build_dir='', timeout=None): - args.extend(['mem=1G']) + args.extend(['mem=1G', 'kunit_shutdown=halt']) outfile = 'test.log' self._ops.linux_bin(args, timeout, build_dir, outfile) subprocess.call(['stty', 'sane'])
tools/testing/kunit/kunit_parser.py +62 -12
133 lines not on this page · raw message
tools/testing/kunit/test_data/test_is_test_passed-all_passed.log binary +0 -0
tools/testing/kunit/test_data/test_is_test_passed-crash.log binary +0 -0
tools/testing/kunit/test_data/test_is_test_passed-failure.log binary +0 -0
Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
PATCH v4 10/11 Documentation: Add kunit_shutdown to kernel-parameters.txt
In reply to Brendan HigginsAdd kunit_shutdown, an option to specify that the kernel shutsdown after running KUnit tests, to the kernel-parameters.txt documentation.
--- Documentation/admin-guide/kernel-parameters.txt | 8 ++++++++ 1 file changed, 8 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +8 -0
@@ -2183,6 +2183,14 @@ 0: force disabled 1: force enabled + kunit_shutdown=[KERNEL UNIT TESTING FRAMEWORK] Shutdown kernel after + running built-in tests. Tests configured as modules will + not be run. + Default: (flag not present) don't shutdown + poweroff: poweroff the kernel after running tests + halt: halt the kernel after running tests + reboot: reboot the kernel after running tests + kvm.ignore_msrs=[KVM] Ignore guest accesses to unhandled MSRs. Default is 0 (don't ignore, but inject #GP)
Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
PATCH v4 11/11 Documentation: kunit: add a brief blurb about kunit_test_suite
In reply to Brendan HigginsAdd a brief blurb saying how and when the kunit_test_suite() macro works to the usage documentation.
--- Documentation/dev-tools/kunit/usage.rst | 5 +++++ 1 file changed, 5 insertions(+)
Documentation/dev-tools/kunit/usage.rst +5 -0
@@ -211,6 +211,11 @@ KUnit test framework. .. note:: A test case will only be run if it is associated with a test suite. +``kunit_test_suite(...)`` is a macro which tells the linker to put the specified +test suite in a special linker section so that it can be run by KUnit either +after late_init, or when the test module is loaded (depending on whether the +test was built in or not). + For more information on these types of things see the :doc:`api/test`. Isolating Behavior
Signature · 1 line
2.27.0.212.ge8ba1cc988-goog
Glad this is back out there: a couple of minor nitpicks below:
Quoted 20 lines · Brendan Higgins
## TL;DR
This patchset adds a centralized executor to dispatch tests rather than relying on
late_initcallto schedule each test suite separately along with a couple of new features that depend on it.Also, sorry for the extreme delay in getting this out. Part of the delay came from finding that there were actually several architectures that the previous revision of this patchset didn't work on, so I went through and attempted to test this patchset on every architecture - more on that later.
## What am I trying to do?
Conceptually, I am trying to provide a mechanism by which test suites can be grouped together so that they can be reasoned about collectively. The last two of three patches in this series add features which depend on this:
Quoted 8 lines
PATCH 8/11 Prints out a test plan[1] right before KUnit tests are run;
this is valuable because it makes it possible for a test harness to detect whether the number of tests run matches the number of tests expected to be run, ensuring that no tests silently failed. The test plan includes a count of tests that will run. With the centralized executor, the tests are located in a single data structure and thus can be counted.
This appears to actually be patch 9/11.
Quoted 8 lines
PATCH 9/11 Add a new kernel command-line option which allows the user to
specify that the kernel poweroff, halt, or reboot after completing all KUnit tests; this is very handy for running KUnit tests on UML or a VM so that the UML/VM process exits cleanly immediately after running all tests without needing a special initramfs. The centralized executor provides a definitive point when all tests have completed and the poweroff, halt, or reboot could occur.
This seems to have been merged into the above patch (9/11).
Quoted 81 lines
In addition, by dispatching tests from a single location, we can guarantee that all KUnit tests run after
late_initis complete, which was a concern during the initial KUnit patchset review (this has not been a problem in practice, but resolving with certainty is nevertheless desirable).Other use cases for this exist, but the above features should provide an idea of the value that this could provide.
## Changes since last revision:
- On the last revision I got some messages from 0day that showed that this patchset didn't work on several architectures, one issue that this patchset addresses is that we were aligning both memory segments as well as structures in the segments to specific byte boundaries which was incorrect.
- The issue mentioned above also caused me to test on additional architectures which revealed that some architectures other than UML do not use the default init linker section macro that most architectures use. There are now several new patches (2, 3, 4, and 6).
- Fixed a formatting consistency issue in the kernel params documentation patch (9/9).
- Add a brief blurb on how and when thekunit_test_suitemacro works.## Remaining work to be done:
The only architecture for which I was able to get a compiler, but was apparently unable to get KUnit into a section that the executor to see was m68k - not sure why.
Alan Maguire (1):
kunit: test: create a single centralized executor for all testsBrendan Higgins (10):
vmlinux.lds.h: add linker section for KUnit test suites
arch: arm64: add linker section for KUnit test suites
arch: microblaze: add linker section for KUnit test suites
arch: powerpc: add linker section for KUnit test suites
arch: um: add linker section for KUnit test suites
arch: xtensa: add linker section for KUnit test suites
init: main: add KUnit to kernel init
kunit: test: add test plan to KUnit TAP format
Documentation: Addkunit_shutdownto kernel-parameters.txt
Documentation: kunit: add a brief blurb aboutkunit_test_suite.../admin-guide/kernel-parameters.txt | 8 ++ Documentation/dev-tools/kunit/usage.rst | 5 ++ arch/arm64/kernel/vmlinux.lds.S | 3 + arch/microblaze/kernel/vmlinux.lds.S | 4 + arch/powerpc/kernel/vmlinux.lds.S | 4 + arch/um/include/asm/common.lds.S | 4 + arch/xtensa/kernel/vmlinux.lds.S | 4 + include/asm-generic/vmlinux.lds.h | 8 ++ include/kunit/test.h | 73 ++++++++++++----- init/main.c | 4 + lib/kunit/Makefile | 3 +- lib/kunit/executor.c | 63 +++++++++++++++ lib/kunit/test.c | 13 +-- tools/testing/kunit/kunit_kernel.py | 2 +- tools/testing/kunit/kunit_parser.py | 74 +++++++++++++++--- .../test_is_test_passed-all_passed.log | Bin 1562 -> 1567 bytes .../test_data/test_is_test_passed-crash.log | Bin 3016 -> 3021 bytes .../test_data/test_is_test_passed-failure.log | Bin 1700 -> 1705 bytes 18 files changed, 226 insertions(+), 46 deletions(-) create mode 100644 lib/kunit/executor.cbase-commit:
4333a9b0b67bb4e8bcd91bdd80da80b0ec151162
prerequisite-patch-id:2d4b5aa9fa8ada9ae04c8584b47c299a822b9455
prerequisite-patch-id:582b6d9d28ce4b71628890ec832df6522ca68de0These patches are available for download with dependencies here:
https://kunit-review.googlesource.com/c/linux/+/3829
[1] https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan
[2] https://patchwork.kernel.org/patch/11383635/--
2.27.0.212.ge8ba1cc988-goog
Quoted 45 lines · David Gow
Glad this is back out there: a couple of minor nitpicks below:
Quoted 20 lines · Brendan Higgins
## TL;DR
This patchset adds a centralized executor to dispatch tests rather than relying on
late_initcallto schedule each test suite separately along with a couple of new features that depend on it.Also, sorry for the extreme delay in getting this out. Part of the delay came from finding that there were actually several architectures that the previous revision of this patchset didn't work on, so I went through and attempted to test this patchset on every architecture - more on that later.
## What am I trying to do?
Conceptually, I am trying to provide a mechanism by which test suites can be grouped together so that they can be reasoned about collectively. The last two of three patches in this series add features which depend on this:
Quoted 8 lines
PATCH 8/11 Prints out a test plan[1] right before KUnit tests are run;
this is valuable because it makes it possible for a test harness to detect whether the number of tests run matches the number of tests expected to be run, ensuring that no tests silently failed. The test plan includes a count of tests that will run. With the centralized executor, the tests are located in a single data structure and thus can be counted.This appears to actually be patch 9/11.
Quoted 8 lines
PATCH 9/11 Add a new kernel command-line option which allows the user to
specify that the kernel poweroff, halt, or reboot after completing all KUnit tests; this is very handy for running KUnit tests on UML or a VM so that the UML/VM process exits cleanly immediately after running all tests without needing a special initramfs. The centralized executor provides a definitive point when all tests have completed and the poweroff, halt, or reboot could occur.
Quoted 1 line
This seems to have been merged into the above patch (9/11).
Whoops, good catch.
Fixed in v5!
Quoted 73 lines
Quoted 73 lines
In addition, by dispatching tests from a single location, we can guarantee that all KUnit tests run after
late_initis complete, which was a concern during the initial KUnit patchset review (this has not been a problem in practice, but resolving with certainty is nevertheless desirable).Other use cases for this exist, but the above features should provide an idea of the value that this could provide.
## Changes since last revision:
- On the last revision I got some messages from 0day that showed that this patchset didn't work on several architectures, one issue that this patchset addresses is that we were aligning both memory segments as well as structures in the segments to specific byte boundaries which was incorrect.
- The issue mentioned above also caused me to test on additional architectures which revealed that some architectures other than UML do not use the default init linker section macro that most architectures use. There are now several new patches (2, 3, 4, and 6).
- Fixed a formatting consistency issue in the kernel params documentation patch (9/9).
- Add a brief blurb on how and when thekunit_test_suitemacro works.## Remaining work to be done:
The only architecture for which I was able to get a compiler, but was apparently unable to get KUnit into a section that the executor to see was m68k - not sure why.
Alan Maguire (1):
kunit: test: create a single centralized executor for all testsBrendan Higgins (10):
vmlinux.lds.h: add linker section for KUnit test suites
arch: arm64: add linker section for KUnit test suites
arch: microblaze: add linker section for KUnit test suites
arch: powerpc: add linker section for KUnit test suites
arch: um: add linker section for KUnit test suites
arch: xtensa: add linker section for KUnit test suites
init: main: add KUnit to kernel init
kunit: test: add test plan to KUnit TAP format
Documentation: Addkunit_shutdownto kernel-parameters.txt
Documentation: kunit: add a brief blurb aboutkunit_test_suite.../admin-guide/kernel-parameters.txt | 8 ++ Documentation/dev-tools/kunit/usage.rst | 5 ++ arch/arm64/kernel/vmlinux.lds.S | 3 + arch/microblaze/kernel/vmlinux.lds.S | 4 + arch/powerpc/kernel/vmlinux.lds.S | 4 + arch/um/include/asm/common.lds.S | 4 + arch/xtensa/kernel/vmlinux.lds.S | 4 + include/asm-generic/vmlinux.lds.h | 8 ++ include/kunit/test.h | 73 ++++++++++++----- init/main.c | 4 + lib/kunit/Makefile | 3 +- lib/kunit/executor.c | 63 +++++++++++++++ lib/kunit/test.c | 13 +-- tools/testing/kunit/kunit_kernel.py | 2 +- tools/testing/kunit/kunit_parser.py | 74 +++++++++++++++--- .../test_is_test_passed-all_passed.log | Bin 1562 -> 1567 bytes .../test_data/test_is_test_passed-crash.log | Bin 3016 -> 3021 bytes .../test_data/test_is_test_passed-failure.log | Bin 1700 -> 1705 bytes 18 files changed, 226 insertions(+), 46 deletions(-) create mode 100644 lib/kunit/executor.cbase-commit:
4333a9b0b67bb4e8bcd91bdd80da80b0ec151162
prerequisite-patch-id:2d4b5aa9fa8ada9ae04c8584b47c299a822b9455
prerequisite-patch-id:582b6d9d28ce4b71628890ec832df6522ca68de0These patches are available for download with dependencies here:
Quoted 8 lines
Quoted 8 lines
https://kunit-review.googlesource.com/c/linux/+/3829
[1] https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan
[2] https://patchwork.kernel.org/patch/11383635/--
2.27.0.212.ge8ba1cc988-goog
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel