Skip to contentSkip to conversation

kunit: Kconfig: enable a KUNIT_RUN_ALL fragment

PATCH In review Anders Roxell 4 participants 6 messages May 1, 2020 – May 5, 2020

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kunit: Kconfig: enable a KUNIT_RUN_ALL fragment

Make it easier to enable all KUnit fragments. This is needed for kernel test-systems, so its easy to get all KUnit tests enabled and if new gets added they will be enabled as well. Fragments that has to be builtin will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.

Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if someone wants that even though KUNIT_RUN_ALL is enabled.

---
 drivers/base/Kconfig      |  3 ++-
 drivers/base/test/Kconfig |  3 ++-
 fs/ext4/Kconfig           |  3 ++-
 lib/Kconfig.debug         |  6 ++++--
 lib/Kconfig.kcsan         |  3 ++-
 lib/kunit/Kconfig         | 15 ++++++++++++---
 security/apparmor/Kconfig |  3 ++-
 7 files changed, 26 insertions(+), 10 deletions(-)
drivers/base/Kconfig +2 -1
index 5f0bc74d2409..c48e6e4ef367 100644
@@ -149,8 +149,9 @@ config DEBUG_TEST_DRIVER_REMOVE
 	  test this functionality.
 
 config PM_QOS_KUNIT_TEST
-	bool "KUnit Test for PM QoS features"
+	bool "KUnit Test for PM QoS features" if !KUNIT_RUN_ALL
 	depends on KUNIT=y
+	default KUNIT_RUN_ALL
 
 config HMEM_REPORTING
 	bool
drivers/base/test/Kconfig +2 -1
index 305c7751184a..0d662d689f6b 100644
@@ -9,5 +9,6 @@ config TEST_ASYNC_DRIVER_PROBE
 
 	  If unsure say N.
 config KUNIT_DRIVER_PE_TEST
-	bool "KUnit Tests for property entry API"
+	bool "KUnit Tests for property entry API" if !KUNIT_RUN_ALL
 	depends on KUNIT=y
+	default KUNIT_RUN_ALL
fs/ext4/Kconfig +2 -1
index 2a592e38cdfe..76785143259d 100644
@@ -103,9 +103,10 @@ config EXT4_DEBUG
 		echo 1 > /sys/module/ext4/parameters/mballoc_debug
 
 config EXT4_KUNIT_TESTS
-	tristate "KUnit tests for ext4"
+	tristate "KUnit tests for ext4" if !KUNIT_RUN_ALL
 	select EXT4_FS
 	depends on KUNIT
+	default KUNIT_RUN_ALL
 	help
 	  This builds the ext4 KUnit tests.
 
lib/Kconfig.debug +4 -2
index 8e4aded46281..993e0c5549bc 100644
@@ -2123,8 +2123,9 @@ config TEST_SYSCTL
 	  If unsure, say N.
 
 config SYSCTL_KUNIT_TEST
-	tristate "KUnit test for sysctl"
+	tristate "KUnit test for sysctl" if !KUNIT_RUN_ALL
 	depends on KUNIT
+	default KUNIT_RUN_ALL
 	help
 	  This builds the proc sysctl unit test, which runs on boot.
 	  Tests the API contract and implementation correctness of sysctl.
@@ -2134,8 +2135,9 @@ config SYSCTL_KUNIT_TEST
 	  If unsure, say N.
 
 config LIST_KUNIT_TEST
-	tristate "KUnit Test for Kernel Linked-list structures"
+	tristate "KUnit Test for Kernel Linked-list structures" if !KUNIT_RUN_ALL
 	depends on KUNIT
+	default KUNIT_RUN_ALL
 	help
 	  This builds the linked list KUnit test suite.
 	  It tests that the API and basic functionality of the list_head type
lib/Kconfig.kcsan +2 -1
index ea28245c6c1d..91398300a1bc 100644
@@ -46,8 +46,9 @@ config KCSAN_SELFTEST
 	  works as intended.
 
 config KCSAN_TEST
-	tristate "KCSAN test for integrated runtime behaviour"
+	tristate "KCSAN test for integrated runtime behaviour" if !KUNIT_RUN_ALL
 	depends on TRACEPOINTS && KUNIT
+	default KUNIT_RUN_ALL
 	select TORTURE_TEST
 	help
 	  KCSAN test focusing on behaviour of the integrated runtime. Tests
lib/kunit/Kconfig +12 -3
index 95d12e3d6d95..d6a912779816 100644
@@ -15,7 +15,8 @@ menuconfig KUNIT
 if KUNIT
 
 config KUNIT_DEBUGFS
-	bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation"
+	bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation" if !KUNIT_RUN_ALL
+	default KUNIT_RUN_ALL
 	help
 	  Enable debugfs representation for kunit.  Currently this consists
 	  of /sys/kernel/debug/kunit/<test_suite>/results files for each
@@ -23,7 +24,8 @@ config KUNIT_DEBUGFS
 	  run that occurred.
 
 config KUNIT_TEST
-	tristate "KUnit test for KUnit"
+	tristate "KUnit test for KUnit" if !KUNIT_RUN_ALL
+	default KUNIT_RUN_ALL
 	help
 	  Enables the unit tests for the KUnit test framework. These tests test
 	  the KUnit test framework itself; the tests are both written using
@@ -32,7 +34,8 @@ config KUNIT_TEST
 	  expected.
 
 config KUNIT_EXAMPLE_TEST
-	tristate "Example test for KUnit"
+	tristate "Example test for KUnit" if !KUNIT_RUN_ALL
+	default KUNIT_RUN_ALL
 	help
 	  Enables an example unit test that illustrates some of the basic
 	  features of KUnit. This test only exists to help new users understand
@@ -41,4 +44,10 @@ config KUNIT_EXAMPLE_TEST
 	  is intended for curious hackers who would like to understand how to
 	  use KUnit for kernel development.
 
+config KUNIT_RUN_ALL
+	tristate "KUnit run all test"
+	help
+	  Enables all KUnit tests. If they can be enabled.
+	  That depends on if KUnit is enabled as a module or builtin.
+
 endif # KUNIT
security/apparmor/Kconfig +2 -1
index 0fe336860773..c4648426ea5d 100644
@@ -70,8 +70,9 @@ config SECURITY_APPARMOR_DEBUG_MESSAGES
 	  the kernel message buffer.
 
 config SECURITY_APPARMOR_KUNIT_TEST
-	bool "Build KUnit tests for policy_unpack.c"
+	bool "Build KUnit tests for policy_unpack.c" if !KUNIT_RUN_ALL
 	depends on KUNIT=y && SECURITY_APPARMOR
+	default KUNIT_RUN_ALL
 	help
 	  This builds the AppArmor KUnit tests.
 
Signature · 1 line
2.20.1
maintainer
#1 lore raw
Quoted 94 lines · Anders Roxell

Make it easier to enable all KUnit fragments. This is needed for kernel test-systems, so its easy to get all KUnit tests enabled and if new gets added they will be enabled as well. Fragments that has to be builtin will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.

Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if someone wants that even though KUNIT_RUN_ALL is enabled.

---
 drivers/base/Kconfig      |  3 ++-
 drivers/base/test/Kconfig |  3 ++-
 fs/ext4/Kconfig           |  3 ++-
 lib/Kconfig.debug         |  6 ++++--
 lib/Kconfig.kcsan         |  3 ++-
 lib/kunit/Kconfig         | 15 ++++++++++++---
 security/apparmor/Kconfig |  3 ++-
 7 files changed, 26 insertions(+), 10 deletions(-)
drivers/base/Kconfig +2 -1
index 5f0bc74d2409..c48e6e4ef367 100644
@@ -149,8 +149,9 @@ config DEBUG_TEST_DRIVER_REMOVE
          test this functionality.
 
 config PM_QOS_KUNIT_TEST
-       bool "KUnit Test for PM QoS features"
+       bool "KUnit Test for PM QoS features" if !KUNIT_RUN_ALL
        depends on KUNIT=y
+       default KUNIT_RUN_ALL
 
 config HMEM_REPORTING
        bool
drivers/base/test/Kconfig +2 -1
index 305c7751184a..0d662d689f6b 100644
@@ -9,5 +9,6 @@ config TEST_ASYNC_DRIVER_PROBE
 
          If unsure say N.
 config KUNIT_DRIVER_PE_TEST
-       bool "KUnit Tests for property entry API"
+       bool "KUnit Tests for property entry API" if !KUNIT_RUN_ALL
        depends on KUNIT=y
+       default KUNIT_RUN_ALL
fs/ext4/Kconfig +2 -1
index 2a592e38cdfe..76785143259d 100644
@@ -103,9 +103,10 @@ config EXT4_DEBUG
                echo 1 > /sys/module/ext4/parameters/mballoc_debug
 
 config EXT4_KUNIT_TESTS
-       tristate "KUnit tests for ext4"
+       tristate "KUnit tests for ext4" if !KUNIT_RUN_ALL
        select EXT4_FS
        depends on KUNIT
+       default KUNIT_RUN_ALL
        help
          This builds the ext4 KUnit tests.
 
lib/Kconfig.debug +4 -2
index 8e4aded46281..993e0c5549bc 100644
@@ -2123,8 +2123,9 @@ config TEST_SYSCTL
          If unsure, say N.
 
 config SYSCTL_KUNIT_TEST
-       tristate "KUnit test for sysctl"
+       tristate "KUnit test for sysctl" if !KUNIT_RUN_ALL
        depends on KUNIT
+       default KUNIT_RUN_ALL
        help
          This builds the proc sysctl unit test, which runs on boot.
          Tests the API contract and implementation correctness of sysctl.
@@ -2134,8 +2135,9 @@ config SYSCTL_KUNIT_TEST
          If unsure, say N.
 
 config LIST_KUNIT_TEST
-       tristate "KUnit Test for Kernel Linked-list structures"
+       tristate "KUnit Test for Kernel Linked-list structures" if !KUNIT_RUN_ALL
        depends on KUNIT
+       default KUNIT_RUN_ALL
        help
          This builds the linked list KUnit test suite.
          It tests that the API and basic functionality of the list_head type
lib/Kconfig.kcsan +0 -0
index ea28245c6c1d..91398300a1bc 100644

lib/Kconfig.kcsan config KCSAN_SELFTEST · line 54

Quoted 10 lines
lib/Kconfig.kcsan +2 -1
@@ -46,8 +46,9 @@ config KCSAN_SELFTEST
          works as intended.
 
 config KCSAN_TEST
-       tristate "KCSAN test for integrated runtime behaviour"
+       tristate "KCSAN test for integrated runtime behaviour" if !KUNIT_RUN_ALL
        depends on TRACEPOINTS && KUNIT
+       default KUNIT_RUN_ALL
        select TORTURE_TEST
        help
          KCSAN test focusing on behaviour of the integrated runtime. Tests

I think if you want this patch to be picked up you need to split it, with one patch for each test that is not yet in mainline or the tree that should pick this patch up.

The KCSAN test is in the -rcu tree, but I don't expect it to be merged before 5.9. Most likely, we would only be able to pick up the patch that would make the chance to the KCSAN Kconfig entry once the rest here made it into mainline.

Thanks,
-- Marco

Quoted 35 lines
lib/kunit/Kconfig +6 -3
index 95d12e3d6d95..d6a912779816 100644
@@ -15,7 +15,8 @@ menuconfig KUNIT
 if KUNIT
 
 config KUNIT_DEBUGFS
-       bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation"
+       bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation" if !KUNIT_RUN_ALL
+       default KUNIT_RUN_ALL
        help
          Enable debugfs representation for kunit.  Currently this consists
          of /sys/kernel/debug/kunit/<test_suite>/results files for each
@@ -23,7 +24,8 @@ config KUNIT_DEBUGFS
          run that occurred.
 
 config KUNIT_TEST
-       tristate "KUnit test for KUnit"
+       tristate "KUnit test for KUnit" if !KUNIT_RUN_ALL
+       default KUNIT_RUN_ALL
        help
          Enables the unit tests for the KUnit test framework. These tests test
          the KUnit test framework itself; the tests are both written using
@@ -32,7 +34,8 @@ config KUNIT_TEST
          expected.
 
 config KUNIT_EXAMPLE_TEST
-       tristate "Example test for KUnit"
+       tristate "Example test for KUnit" if !KUNIT_RUN_ALL
+       default KUNIT_RUN_ALL
        help
          Enables an example unit test that illustrates some of the basic
          features of KUnit. This test only exists to help new users understand

lib/kunit/Kconfig config KUNIT_EXAMPLE_TEST · line 52

Quoted 9 lines
lib/kunit/Kconfig +6 -0
@@ -41,4 +44,10 @@ config KUNIT_EXAMPLE_TEST
          is intended for curious hackers who would like to understand how to
          use KUnit for kernel development.
 
+config KUNIT_RUN_ALL
+       tristate "KUnit run all test"
+       help
+         Enables all KUnit tests. If they can be enabled.
+         That depends on if KUnit is enabled as a module or builtin.
+

s/tests. If/tests, if/ ?

Quoted 19 lines
lib/kunit/Kconfig +0 -0
 endif # KUNIT
security/apparmor/Kconfig +2 -2
index 0fe336860773..c4648426ea5d 100644
@@ -70,8 +70,9 @@ config SECURITY_APPARMOR_DEBUG_MESSAGES
          the kernel message buffer.
 
 config SECURITY_APPARMOR_KUNIT_TEST
-       bool "Build KUnit tests for policy_unpack.c"
+       bool "Build KUnit tests for policy_unpack.c" if !KUNIT_RUN_ALL
        depends on KUNIT=y && SECURITY_APPARMOR
+       default KUNIT_RUN_ALL
        help
          This builds the AppArmor KUnit tests.
 
--

2.20.1

#2 lore raw
Quoted 98 lines · Marco Elver
Quoted 96 lines · Anders Roxell

Make it easier to enable all KUnit fragments. This is needed for kernel test-systems, so its easy to get all KUnit tests enabled and if new gets added they will be enabled as well. Fragments that has to be builtin will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.

Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if someone wants that even though KUNIT_RUN_ALL is enabled.

---
 drivers/base/Kconfig      |  3 ++-
 drivers/base/test/Kconfig |  3 ++-
 fs/ext4/Kconfig           |  3 ++-
 lib/Kconfig.debug         |  6 ++++--
 lib/Kconfig.kcsan         |  3 ++-
 lib/kunit/Kconfig         | 15 ++++++++++++---
 security/apparmor/Kconfig |  3 ++-
 7 files changed, 26 insertions(+), 10 deletions(-)
drivers/base/Kconfig +2 -1
index 5f0bc74d2409..c48e6e4ef367 100644
@@ -149,8 +149,9 @@ config DEBUG_TEST_DRIVER_REMOVE
          test this functionality.
 
 config PM_QOS_KUNIT_TEST
-       bool "KUnit Test for PM QoS features"
+       bool "KUnit Test for PM QoS features" if !KUNIT_RUN_ALL
        depends on KUNIT=y
+       default KUNIT_RUN_ALL
 
 config HMEM_REPORTING
        bool
drivers/base/test/Kconfig +2 -1
index 305c7751184a..0d662d689f6b 100644
@@ -9,5 +9,6 @@ config TEST_ASYNC_DRIVER_PROBE
 
          If unsure say N.
 config KUNIT_DRIVER_PE_TEST
-       bool "KUnit Tests for property entry API"
+       bool "KUnit Tests for property entry API" if !KUNIT_RUN_ALL
        depends on KUNIT=y
+       default KUNIT_RUN_ALL
fs/ext4/Kconfig +2 -1
index 2a592e38cdfe..76785143259d 100644
@@ -103,9 +103,10 @@ config EXT4_DEBUG
                echo 1 > /sys/module/ext4/parameters/mballoc_debug
 
 config EXT4_KUNIT_TESTS
-       tristate "KUnit tests for ext4"
+       tristate "KUnit tests for ext4" if !KUNIT_RUN_ALL
        select EXT4_FS
        depends on KUNIT
+       default KUNIT_RUN_ALL
        help
          This builds the ext4 KUnit tests.
 
lib/Kconfig.debug +4 -2
index 8e4aded46281..993e0c5549bc 100644
@@ -2123,8 +2123,9 @@ config TEST_SYSCTL
          If unsure, say N.
 
 config SYSCTL_KUNIT_TEST
-       tristate "KUnit test for sysctl"
+       tristate "KUnit test for sysctl" if !KUNIT_RUN_ALL
        depends on KUNIT
+       default KUNIT_RUN_ALL
        help
          This builds the proc sysctl unit test, which runs on boot.
          Tests the API contract and implementation correctness of sysctl.
@@ -2134,8 +2135,9 @@ config SYSCTL_KUNIT_TEST
          If unsure, say N.
 
 config LIST_KUNIT_TEST
-       tristate "KUnit Test for Kernel Linked-list structures"
+       tristate "KUnit Test for Kernel Linked-list structures" if !KUNIT_RUN_ALL
        depends on KUNIT
+       default KUNIT_RUN_ALL
        help
          This builds the linked list KUnit test suite.
          It tests that the API and basic functionality of the list_head type
lib/Kconfig.kcsan +0 -0
index ea28245c6c1d..91398300a1bc 100644
@@ -46,8 +46,9 @@ config KCSAN_SELFTEST
          works as intended.
 
Quoted 12 lines

lib/Kconfig.kcsan config KCSAN_SELFTEST · line 54

Quoted 8 lines
lib/Kconfig.kcsan +2 -1
 config KCSAN_TEST
-       tristate "KCSAN test for integrated runtime behaviour"
+       tristate "KCSAN test for integrated runtime behaviour" if !KUNIT_RUN_ALL
        depends on TRACEPOINTS && KUNIT
+       default KUNIT_RUN_ALL
        select TORTURE_TEST
        help
          KCSAN test focusing on behaviour of the integrated runtime. Tests

I think if you want this patch to be picked up you need to split it, with one patch for each test that is not yet in mainline or the tree that should pick this patch up.

OK, would it be ok to do one patch per subsystem if it's in the mainline tree, and another patch if it's only in the next tree for the same subsystem?

Quoted 47 lines

The KCSAN test is in the -rcu tree, but I don't expect it to be merged before 5.9. Most likely, we would only be able to pick up the patch that would make the chance to the KCSAN Kconfig entry once the rest here made it into mainline.

Thanks,
-- Marco

Quoted 38 lines
lib/kunit/Kconfig +6 -3
index 95d12e3d6d95..d6a912779816 100644
@@ -15,7 +15,8 @@ menuconfig KUNIT
 if KUNIT
 
 config KUNIT_DEBUGFS
-       bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation"
+       bool "KUnit - Enable /sys/kernel/debug/kunit debugfs representation" if !KUNIT_RUN_ALL
+       default KUNIT_RUN_ALL
        help
          Enable debugfs representation for kunit.  Currently this consists
          of /sys/kernel/debug/kunit/<test_suite>/results files for each
@@ -23,7 +24,8 @@ config KUNIT_DEBUGFS
          run that occurred.
 
 config KUNIT_TEST
-       tristate "KUnit test for KUnit"
+       tristate "KUnit test for KUnit" if !KUNIT_RUN_ALL
+       default KUNIT_RUN_ALL
        help
          Enables the unit tests for the KUnit test framework. These tests test
          the KUnit test framework itself; the tests are both written using
@@ -32,7 +34,8 @@ config KUNIT_TEST
          expected.
 
 config KUNIT_EXAMPLE_TEST
-       tristate "Example test for KUnit"
+       tristate "Example test for KUnit" if !KUNIT_RUN_ALL
+       default KUNIT_RUN_ALL
        help
          Enables an example unit test that illustrates some of the basic
          features of KUnit. This test only exists to help new users understand
@@ -41,4 +44,10 @@ config KUNIT_EXAMPLE_TEST
          is intended for curious hackers who would like to understand how to
          use KUnit for kernel development.
 
Quoted 8 lines

lib/kunit/Kconfig config KUNIT_EXAMPLE_TEST · line 52

Quoted 6 lines
lib/kunit/Kconfig +6 -0
+config KUNIT_RUN_ALL
+       tristate "KUnit run all test"
+       help
+         Enables all KUnit tests. If they can be enabled.
+         That depends on if KUnit is enabled as a module or builtin.
+

s/tests. If/tests, if/ ?

correct, I will fix that.

Cheers,
Anders

Quoted 20 lines
Quoted 20 lines
lib/kunit/Kconfig +0 -0
 endif # KUNIT
security/apparmor/Kconfig +2 -2
index 0fe336860773..c4648426ea5d 100644
@@ -70,8 +70,9 @@ config SECURITY_APPARMOR_DEBUG_MESSAGES
          the kernel message buffer.
 
 config SECURITY_APPARMOR_KUNIT_TEST
-       bool "Build KUnit tests for policy_unpack.c"
+       bool "Build KUnit tests for policy_unpack.c" if !KUNIT_RUN_ALL
        depends on KUNIT=y && SECURITY_APPARMOR
+       default KUNIT_RUN_ALL
        help
          This builds the AppArmor KUnit tests.
 
--

2.20.1

#3 lore raw
In reply to Anders Roxell
Quoted 8 lines · Anders Roxell

Make it easier to enable all KUnit fragments. This is needed for kernel test-systems, so its easy to get all KUnit tests enabled and if new gets added they will be enabled as well. Fragments that has to be builtin will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.

Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if someone wants that even though KUNIT_RUN_ALL is enabled.

I would LOVE IT, if you could make this work! I have been trying to figure out the best way to run all KUnit tests for a long time now.

That being said, I am a bit skeptical that this approach will be much more successful than just using allyesconfig. Either way, there are tests coming down the pipeline that are incompatible with each other (the KASAN test and the KCSAN test will be incompatible). Even so, tests like the apparmor test require a lot of non-default configuration to compile. In the end, I am not sure how many tests we will really be able to turn on this way.

Thoughts?

#4 lore raw
Quoted 22 lines · Brendan Higgins
Quoted 8 lines · Anders Roxell

Make it easier to enable all KUnit fragments. This is needed for kernel test-systems, so its easy to get all KUnit tests enabled and if new gets added they will be enabled as well. Fragments that has to be builtin will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.

Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if someone wants that even though KUNIT_RUN_ALL is enabled.

I would LOVE IT, if you could make this work! I have been trying to figure out the best way to run all KUnit tests for a long time now.

That being said, I am a bit skeptical that this approach will be much more successful than just using allyesconfig. Either way, there are tests coming down the pipeline that are incompatible with each other (the KASAN test and the KCSAN test will be incompatible). Even so, tests like the apparmor test require a lot of non-default configuration to compile. In the end, I am not sure how many tests we will really be able to turn on this way.

Quoted 1 line

Thoughts?

I think there's still some value in this which the allyesconfig option doesn't provide. As you point out, it's not possible to have a generic "run all tests" option due to potential conflicting dependencies, but this does provide a way to run all tests for things enabled in the current config. This could be really useful for downstream developers who want a way of running all tests relevant to their config without the overhead of running irrelevant tests (e.g., for drivers they don't build). Using allyesconfig doesn't make that distinction.

Ultimately, we'll probably still want something which enables a broader set of tests for upstream development: whether that's based on this, allyesconfig, or something else entirely remains to be seen, I think. I suspect we're going to end up with something subsystem-specific (having a kunitconfig per subsystem, or a testing line in MAINTAINERS or similar are ideas which have been brought up in the past).

This is a great looking tool to have in the toolbox, though.

-- David

#5 lore raw
Quoted 27 lines · David Gow
Quoted 22 lines · Brendan Higgins
Quoted 8 lines · Anders Roxell

Make it easier to enable all KUnit fragments. This is needed for kernel test-systems, so its easy to get all KUnit tests enabled and if new gets added they will be enabled as well. Fragments that has to be builtin will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.

Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if someone wants that even though KUNIT_RUN_ALL is enabled.

I would LOVE IT, if you could make this work! I have been trying to figure out the best way to run all KUnit tests for a long time now.

That being said, I am a bit skeptical that this approach will be much more successful than just using allyesconfig. Either way, there are tests coming down the pipeline that are incompatible with each other (the KASAN test and the KCSAN test will be incompatible). Even so, tests like the apparmor test require a lot of non-default configuration to compile. In the end, I am not sure how many tests we will really be able to turn on this way.

Quoted 1 line

Thoughts?

Quoted 8 lines

I think there's still some value in this which the allyesconfig option doesn't provide. As you point out, it's not possible to have a generic "run all tests" option due to potential conflicting dependencies, but this does provide a way to run all tests for things enabled in the current config. This could be really useful for downstream developers who want a way of running all tests relevant to their config without the overhead of running irrelevant tests (e.g., for drivers they don't build).

It will solve that as well as for a tester doesn't have to go through all KUnit tests fragments to turn them on.

Quoted 1 line

Using allyesconfig doesn't make that distinction.

We could also create a config fragment file in kernel/configs/kunit.config where we set
------start
CONFIG_KUNIT=y
CONFIG_KUNIT_RUN_ALL=y
CONFIG_SECURITY_APPARMOR=y
------end

So, these two can only be enabled if KUNIT=y
CONFIG_KUNIT_DRIVER_PE_TEST=y
CONFIG_PM_QOS_KUNIT_TEST=y

and for this one we have a pre-request of SECURITY_APPARMOR=y CONFIG_SECURITY_APPARMOR_KUNIT_TEST=y

Other tests solves the dependencies with 'select' like CONFIG_EXT4_KUNIT_TESTS, that adds this row in fs/ext4/Kconfig, 'select EXT4_FS'

But I think we should try to minimize the number of 'select' statements, in order to avoid circular dependencies and unexpected behaviours. Maybe we should add the CONFIG_EXT4_FS=y into the kunit.config file instead ?

Quoted 10 lines

Ultimately, we'll probably still want something which enables a broader set of tests for upstream development: whether that's based on this, allyesconfig, or something else entirely remains to be seen, I think. I suspect we're going to end up with something subsystem-specific (having a kunitconfig per subsystem, or a testing line in MAINTAINERS or similar are ideas which have been brought up in the past).

This is a great looking tool to have in the toolbox, though.

I agree!

I'll prepare a patchset with individual patches as was suggested by Marco shortly.

Cheers,
Anders