Skip to contentSkip to conversation

lib: Convert test_printf.c to KUnit

PATCH Superseded Arpitha Raghunandan 7 participants 12 messages Nov 3, 2020 – Nov 6, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 lib: Convert test_printf.c to KUnit

Convert test lib/test_printf.c to KUnit. More information about KUnit can be found at:
https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html.
KUnit provides a common framework for unit tests in the kernel. KUnit and kselftest are standardizing around KTAP, converting this test to KUnit makes this test output in KTAP which we are trying to make the standard test result format for the kernel. More about the KTAP format can be found at:
https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/.
I ran both the original and converted tests as is to produce the output for success of the test in the two cases. I also ran these tests with a small modification to show the difference in the output for failure of the test in both cases. The modification I made is:

- test("127.000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);
+ test("127-000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);

Original test success:

[    0.540860] test_printf: loaded.
[    0.540863] test_printf: random seed = 0x5c46c33837bc0619
[    0.541022] test_printf: all 388 tests passed

Original test failure:

[    0.537980] test_printf: loaded.
[    0.537983] test_printf: random seed = 0x1bc1efd881954afb
[    0.538029] test_printf: vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538030] test_printf: kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538124] test_printf: failed 2 out of 388 tests
[    0.538125] test_printf: random seed used was 0x1bc1efd881954afb

Converted test success:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
ok 1 - printf

Converted test failure:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    # ip: EXPECTATION FAILED at lib/printf_kunit.c:82
vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' # ip: EXPECTATION FAILED at lib/printf_kunit.c:124
kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' not ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
not ok 1 - printf

This patch is based on top of Andy's series that renames KUnit based tests in lib/ and Ramsus' series on deterministic random testing.

---
Changes v2->v3:
- Split tests into multiple KUNIT_CASE()
- kunittest variable made static
- Allocation and freeing done through init and exit callbacks in KUnit Changes v1->v2:
- Retain the control flow (early returns) in do_test()
- Display less irrelevant information on test failure
- A more detailed commit message

 lib/Kconfig.debug                     |   7 +-
 lib/Makefile                          |   2 +-
 lib/{test_printf.c => printf_kunit.c} | 266 ++++++++++++++------------
 3 files changed, 150 insertions(+), 125 deletions(-)
 rename lib/{test_printf.c => printf_kunit.c} (78%)
lib/Kconfig.debug +4 -3
index c789b39ed527..42de097d4c5f 100644
@@ -2051,9 +2051,6 @@ config TEST_STRSCPY
 config TEST_KSTRTOX
 	tristate "Test kstrto*() family of functions at runtime"
 
-config TEST_PRINTF
-	tristate "Test printf() family of functions at runtime"
-
 config TEST_BITMAP
 	tristate "Test bitmap_*() family of functions at runtime"
 	help
@@ -2280,6 +2277,10 @@ config BITS_TEST
 
 	  If unsure, say N.
 
+config PRINTF_KUNIT_TEST
+	tristate "KUnit test for printf() family of functions at runtime"
+	depends on KUNIT
+
 config TEST_UDELAY
 	tristate "udelay test driver"
 	help
lib/Makefile +1 -1
index dc76e7d8a453..1351d242447c 100644
@@ -84,7 +84,6 @@ obj-$(CONFIG_TEST_SORT) += test_sort.o
 obj-$(CONFIG_TEST_USER_COPY) += test_user_copy.o
 obj-$(CONFIG_TEST_STATIC_KEYS) += test_static_keys.o
 obj-$(CONFIG_TEST_STATIC_KEYS) += test_static_key_base.o
-obj-$(CONFIG_TEST_PRINTF) += test_printf.o
 obj-$(CONFIG_TEST_BITMAP) += test_bitmap.o
 obj-$(CONFIG_TEST_STRSCPY) += test_strscpy.o
 obj-$(CONFIG_TEST_UUID) += test_uuid.o
@@ -352,3 +351,4 @@ obj-$(CONFIG_BITFIELD_KUNIT) += bitfield_kunit.o
 obj-$(CONFIG_BITS_TEST) += bits_kunit.o
 obj-$(CONFIG_LINEAR_RANGES_TEST) += linear_ranges_kunit.o
 obj-$(CONFIG_LIST_KUNIT_TEST) += list_kunit.o
+obj-$(CONFIG_PRINTF_KUNIT_TEST) += printf_kunit.o
lib/printf_kunit.c renamed from lib/test_printf.c +145 -121

499 lines not on this page · raw message

Signature · 1 line
2.25.1
#1 lore raw
Quoted 84 lines · Arpitha Raghunandan

Convert test lib/test_printf.c to KUnit. More information about KUnit can be found at:
https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html.
KUnit provides a common framework for unit tests in the kernel. KUnit and kselftest are standardizing around KTAP, converting this test to KUnit makes this test output in KTAP which we are trying to make the standard test result format for the kernel. More about the KTAP format can be found at:
https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/.
I ran both the original and converted tests as is to produce the output for success of the test in the two cases. I also ran these tests with a small modification to show the difference in the output for failure of the test in both cases. The modification I made is:

- test("127.000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);
+ test("127-000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);

Original test success:

[    0.540860] test_printf: loaded.
[    0.540863] test_printf: random seed = 0x5c46c33837bc0619
[    0.541022] test_printf: all 388 tests passed

Original test failure:

[    0.537980] test_printf: loaded.
[    0.537983] test_printf: random seed = 0x1bc1efd881954afb
[    0.538029] test_printf: vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538030] test_printf: kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538124] test_printf: failed 2 out of 388 tests
[    0.538125] test_printf: random seed used was 0x1bc1efd881954afb

Converted test success:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
ok 1 - printf

Converted test failure:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    # ip: EXPECTATION FAILED at lib/printf_kunit.c:82
vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' # ip: EXPECTATION FAILED at lib/printf_kunit.c:124
kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' not ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format

Quoted 8 lines

    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
not ok 1 - printf

Better, indeed.

But can be this improved to have a cumulative statistics, like showing only number of total, succeeded, failed with details of the latter ones?

Quoted 2 lines

This patch is based on top of Andy's series that renames KUnit based tests in lib/ and Ramsus' series on deterministic random testing.

For the reference:
https://lore.kernel.org/linux-kselftest/[email protected]/

...

lib/printf_kunit.c test_number(void) · line 182

Quoted 4 lines
lib/printf_kunit.c renamed from lib/test_printf.c +2 -2
-static void __init
-test_string(void)
+static void
+test_string(struct kunit *ktest)

I guess no need anymore to keep them on two lines, just combine to one line.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#2 lore raw
Quoted 94 lines · Andy Shevchenko
Quoted 84 lines · Arpitha Raghunandan

Convert test lib/test_printf.c to KUnit. More information about KUnit can be found at:
https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html.
KUnit provides a common framework for unit tests in the kernel. KUnit and kselftest are standardizing around KTAP, converting this test to KUnit makes this test output in KTAP which we are trying to make the standard test result format for the kernel. More about the KTAP format can be found at:
https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/.
I ran both the original and converted tests as is to produce the output for success of the test in the two cases. I also ran these tests with a small modification to show the difference in the output for failure of the test in both cases. The modification I made is:

- test("127.000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);
+ test("127-000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);

Original test success:

[    0.540860] test_printf: loaded.
[    0.540863] test_printf: random seed = 0x5c46c33837bc0619
[    0.541022] test_printf: all 388 tests passed

Original test failure:

[    0.537980] test_printf: loaded.
[    0.537983] test_printf: random seed = 0x1bc1efd881954afb
[    0.538029] test_printf: vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538030] test_printf: kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538124] test_printf: failed 2 out of 388 tests
[    0.538125] test_printf: random seed used was 0x1bc1efd881954afb

Converted test success:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
ok 1 - printf

Converted test failure:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    # ip: EXPECTATION FAILED at lib/printf_kunit.c:82
vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' # ip: EXPECTATION FAILED at lib/printf_kunit.c:124
kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' not ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format

Quoted 8 lines

    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
not ok 1 - printf

Quoted 4 lines

Better, indeed.

But can be this improved to have a cumulative statistics, like showing only number of total, succeeded, failed with details of the latter ones?

Is that the proper test output format? We have a standard...

thanks,

greg k-h

#3 lore raw
Quoted 2 lines · Greg KH
Quoted 1 line · Andy Shevchenko

On Tue, Nov 03, 2020 at 04:40:49PM +0530, Arpitha Raghunandan wrote:

...

Quoted 6 lines
Quoted 4 lines

Better, indeed.

But can be this improved to have a cumulative statistics, like showing only number of total, succeeded, failed with details of the latter ones?

Is that the proper test output format? We have a standard...

I dunno. I'm asking...
We have few possibilities here: a) let it go like above, b) use the cumulative statistics, c) if there is no such available, implement and use it, d) ...

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#4 lore raw
Quoted 11 lines · Andy Shevchenko
Quoted 2 lines · Greg KH
Quoted 1 line · Andy Shevchenko

On Tue, Nov 03, 2020 at 04:40:49PM +0530, Arpitha Raghunandan wrote:

...

Quoted 5 lines
Quoted 4 lines

Better, indeed.

But can be this improved to have a cumulative statistics, like showing only number of total, succeeded, failed with details of the latter ones?

Quoted 6 lines
Quoted 1 line

Is that the proper test output format? We have a standard...

I dunno. I'm asking...
We have few possibilities here: a) let it go like above, b) use the cumulative statistics, c) if there is no such available, implement and use it, d) ...

We have a standard, please use it.

greg k-h

#5 lore raw
In reply to Greg KH
Quoted 95 lines · Greg KH
Quoted 94 lines · Andy Shevchenko
Quoted 92 lines · Arpitha Raghunandan

Convert test lib/test_printf.c to KUnit. More information about KUnit can be found at:
https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html.
KUnit provides a common framework for unit tests in the kernel. KUnit and kselftest are standardizing around KTAP, converting this test to KUnit makes this test output in KTAP which we are trying to make the standard test result format for the kernel. More about the KTAP format can be found at:
https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/.
I ran both the original and converted tests as is to produce the output for success of the test in the two cases. I also ran these tests with a small modification to show the difference in the output for failure of the test in both cases. The modification I made is:

- test("127.000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);
+ test("127-000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);

Original test success:

[    0.540860] test_printf: loaded.
[    0.540863] test_printf: random seed = 0x5c46c33837bc0619
[    0.541022] test_printf: all 388 tests passed

Original test failure:

[    0.537980] test_printf: loaded.
[    0.537983] test_printf: random seed = 0x1bc1efd881954afb
[    0.538029] test_printf: vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538030] test_printf: kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538124] test_printf: failed 2 out of 388 tests
[    0.538125] test_printf: random seed used was 0x1bc1efd881954afb

Converted test success:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
ok 1 - printf

Converted test failure:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    # ip: EXPECTATION FAILED at lib/printf_kunit.c:82
vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' # ip: EXPECTATION FAILED at lib/printf_kunit.c:124
kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' not ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
not ok 1 - printf

Quoted 6 lines
Quoted 4 lines

Better, indeed.

But can be this improved to have a cumulative statistics, like showing only number of total, succeeded, failed with details of the latter ones?

Is that the proper test output format? We have a standard...

What is the standard, please?

The original module produced:

[   48.577739] test_printf: loaded.
[   48.578046] test_printf: all 388 tests passed

It comes from KSTM_MODULE_LOADERS() macro that has been created by the commit eebf4dd452377921e3a26 ("kselftest: Add test module framework header"). There are 3 users:

$> git grep KSTM_MODULE_LOADERS
Documentation/dev-tools/kselftest.rst: KSTM_MODULE_LOADERS(test_foo);
lib/test_bitmap.c:KSTM_MODULE_LOADERS(test_bitmap); lib/test_printf.c:KSTM_MODULE_LOADERS(test_printf); lib/test_strscpy.c:KSTM_MODULE_LOADERS(test_strscpy);
tools/testing/selftests/kselftest_module.h:#define KSTM_MODULE_LOADERS(__module) \

When I looked for similar strings then I see huge creativity:

drivers/firmware/psci/psci_checker.c: pr_info("Hotplug tests passed OK\n");
drivers/firmware/psci/psci_checker.c: pr_info("Suspend tests passed OK\n");
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c: seq_printf(m, "ib ring tests passed.\n");

drivers/mtd/chips/jedec_probe.c:        /* all tests passed - mark  as success */
drivers/usb/host/xhci-mem.c:    xhci_dbg(xhci, "TRB math tests passed.\n");
fs/unicode/utf8-selftest.c:             pr_info("All %u tests passed\n", total_tests);
kernel/kcsan/selftest.c:        pr_info("selftest: %d/%d tests passed\n", passed, total);
lib/crc32test.c:                pr_info("crc32c: self tests passed, processed %d bytes in %lld nsec\n",
lib/crc32test.c:                pr_info("crc32c_combine: %d self tests passed\n", runs);
lib/crc32test.c:                pr_info("crc32: self tests passed, processed %d bytes in %lld nsec\n",
lib/crc32test.c:                pr_info("crc32_combine: %d self tests passed\n", runs);
lib/globtest.c:         KERN_INFO "glob: %u self-tests passed, %u failed\n";
lib/random32.c:         pr_info("prandom: %d self tests passed\n", runs);
lib/test_hash.c:        pr_notice("%u tests passed.", tests);
lib/test_hexdump.c:             pr_info("all %u tests passed\n", total_tests);
lib/test_ida.c: printk("IDA: %u of %u tests passed\n", tests_passed, tests_run);
lib/test_meminit.c:             pr_info("all %d tests passed!\n", num_tests);
lib/test_overflow.c:            pr_info("all tests passed\n");
lib/test_stackinit.c:           pr_info("all tests passed!\n");
lib/test_user_copy.c:           pr_info("tests passed.\n");
lib/test_uuid.c:                pr_info("all %u tests passed\n", total_tests);
lib/test_xarray.c:      printk("XArray: %u of %u tests passed\n", tests_passed, tests_run);

tools/bootconfig/test-bootconfig.sh: echo "All tests passed"
Binary file tools/testing/kunit/test_data/test_is_test_passed-all_passed.log matches Binary file tools/testing/kunit/test_data/test_is_test_passed-crash.log matches Binary file tools/testing/kunit/test_data/test_is_test_passed-failure.log matches Binary file tools/testing/kunit/test_data/test_output_isolated_correctly.log matches tools/testing/selftests/bpf/test_tc_tunnel.sh: echo "OK. All tests passed" tools/testing/selftests/bpf/test_xdping.sh:echo "OK. All tests passed"
tools/testing/selftests/dmabuf-heaps/dmabuf-heap.c: printf("Ioctl compatibility tests passed\n");
tools/testing/selftests/kselftest_harness.h: ksft_print_msg("%s: %u / %u tests passed.\n", ret ? "FAILED" : "PASSED",
tools/testing/selftests/kselftest_module.h: pr_info("all %u tests passed\n", total_tests);
tools/testing/selftests/net/ipv6_flowlabel.sh:echo OK. All tests passed
tools/testing/selftests/net/msg_zerocopy.sh: echo "OK. All tests passed"
tools/testing/selftests/net/psock_fanout.c: printf("OK. All tests passed\n");
tools/testing/selftests/net/psock_snd.sh:echo "OK. All tests passed"
tools/testing/selftests/net/psock_tpacket.c: printf("OK. All tests passed\n");
tools/testing/selftests/net/so_txtime.sh:echo OK. All tests passed
tools/testing/selftests/net/txtimestamp.sh: echo "OK. All tests passed"

From my POV, KUnit is a framework that is supposed to actually unify many tests.

There is alternative framework maintained by Shuah Khan. In compare with KUnit, it prints the results in the userspace. The format is quite similar, for example:

tools/testing/selftests/livepatch/# make run_tests TAP version 13
1..5
# selftests: livepatch: test-livepatch.sh
# TEST: basic function patching ... ok
# TEST: multiple livepatches ... ok
# TEST: atomic replace livepatch ... ok
ok 1 selftests: livepatch: test-livepatch.sh
# selftests: livepatch: test-callbacks.sh
# TEST: target module before livepatch ... ok
# TEST: module_coming notifier ... ok
# TEST: module_going notifier ... ok
# TEST: module_coming and module_going notifiers ... ok # TEST: target module not present ... ok
# TEST: pre-patch callback -ENODEV ... ok
# TEST: module_coming + pre-patch callback -ENODEV ... ok # TEST: multiple target modules ... ok
# TEST: busy target module ... ok
# TEST: multiple livepatches ... ok
# TEST: atomic replace ... ok
ok 2 selftests: livepatch: test-callbacks.sh
# selftests: livepatch: test-shadow-vars.sh
# TEST: basic shadow variable API ... ok
ok 3 selftests: livepatch: test-shadow-vars.sh
# selftests: livepatch: test-state.sh
# TEST: system state modification ... ok
# TEST: taking over system state modification ... ok # TEST: compatible cumulative livepatches ... ok
# TEST: incompatible cumulative livepatches ... ok
ok 4 selftests: livepatch: test-state.sh
# selftests: livepatch: test-ftrace.sh
# TEST: livepatch interaction with ftrace_enabled sysctl ... ok ok 5 selftests: livepatch: test-ftrace.sh

I do not have strong meaning what format is better. But the format:

"doing something ... ok"

is problematic in kernel log because it contains messages from all kernel activity. As a result, many unrelated messages might be added between "doing something ..." and "ok". It is much more reliable with stdout/stderr in userspace.

For kernel, it is more reliable to print results when the test finishes. And it seems to be "easy" to spot an error in KUnit report.

That said, it would be nice to see some summary on the very last line. It would be especially useful when the log is longer than one screen.

The result of every subtest might help to locate the problem when the system freezes. It might be optional.

I am personally fine with the proposed output of printf_kunit.c.

Best Regards,
Petr

#6 lore raw
Quoted 96 lines · Petr Mladek
Quoted 95 lines · Greg KH
Quoted 94 lines · Andy Shevchenko
Quoted 92 lines · Arpitha Raghunandan

Convert test lib/test_printf.c to KUnit. More information about KUnit can be found at:
https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html.
KUnit provides a common framework for unit tests in the kernel. KUnit and kselftest are standardizing around KTAP, converting this test to KUnit makes this test output in KTAP which we are trying to make the standard test result format for the kernel. More about the KTAP format can be found at:
https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/.
I ran both the original and converted tests as is to produce the output for success of the test in the two cases. I also ran these tests with a small modification to show the difference in the output for failure of the test in both cases. The modification I made is:

- test("127.000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);
+ test("127-000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);

Original test success:

[    0.540860] test_printf: loaded.
[    0.540863] test_printf: random seed = 0x5c46c33837bc0619
[    0.541022] test_printf: all 388 tests passed

Original test failure:

[    0.537980] test_printf: loaded.
[    0.537983] test_printf: random seed = 0x1bc1efd881954afb
[    0.538029] test_printf: vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538030] test_printf: kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538124] test_printf: failed 2 out of 388 tests
[    0.538125] test_printf: random seed used was 0x1bc1efd881954afb

Converted test success:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
ok 1 - printf

Converted test failure:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    # ip: EXPECTATION FAILED at lib/printf_kunit.c:82
vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' # ip: EXPECTATION FAILED at lib/printf_kunit.c:124
kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' not ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
not ok 1 - printf

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Better, indeed.

But can be this improved to have a cumulative statistics, like showing only number of total, succeeded, failed with details of the latter ones?

Is that the proper test output format? We have a standard...

What is the standard, please?

The TAP format should be the standard, no reason the kernel can not spit out the same test message format that the userspace tests do, right?

thanks,

greg k-h

#7 lore raw
Quoted 103 lines · Greg KH
Quoted 101 lines · Petr Mladek
Quoted 100 lines · Greg KH
Quoted 98 lines · Andy Shevchenko
Quoted 84 lines · Arpitha Raghunandan

Convert test lib/test_printf.c to KUnit. More information about KUnit can be found at:
https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html.
KUnit provides a common framework for unit tests in the kernel. KUnit and kselftest are standardizing around KTAP, converting this test to KUnit makes this test output in KTAP which we are trying to make the standard test result format for the kernel. More about the KTAP format can be found at:
https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/.
I ran both the original and converted tests as is to produce the output for success of the test in the two cases. I also ran these tests with a small modification to show the difference in the output for failure of the test in both cases. The modification I made is:

- test("127.000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);
+ test("127-000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);

Original test success:

[    0.540860] test_printf: loaded.
[    0.540863] test_printf: random seed = 0x5c46c33837bc0619
[    0.541022] test_printf: all 388 tests passed

Original test failure:

[    0.537980] test_printf: loaded.
[    0.537983] test_printf: random seed = 0x1bc1efd881954afb
[    0.538029] test_printf: vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538030] test_printf: kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538124] test_printf: failed 2 out of 388 tests
[    0.538125] test_printf: random seed used was 0x1bc1efd881954afb

Converted test success:
# Subtest: printf
1..25
ok 1 - test_basic
ok 2 - test_number
ok 3 - test_string
ok 4 - plain
ok 5 - null_pointer
ok 6 - error_pointer
ok 7 - invalid_pointer
ok 8 - symbol_ptr
ok 9 - kernel_ptr
ok 10 - struct_resource
ok 11 - addr
ok 12 - escaped_str
ok 13 - hex_string
ok 14 - mac
ok 15 - ip
ok 16 - uuid
ok 17 - dentry
ok 18 - struct_va_format
ok 19 - time_and_date
ok 20 - struct_clk
ok 21 - bitmap
ok 22 - netdev_features
ok 23 - flags
ok 24 - errptr
ok 25 - fwnode_pointer
ok 1 - printf

Converted test failure:
# Subtest: printf
1..25
ok 1 - test_basic
ok 2 - test_number
ok 3 - test_string
ok 4 - plain
ok 5 - null_pointer
ok 6 - error_pointer
ok 7 - invalid_pointer
ok 8 - symbol_ptr
ok 9 - kernel_ptr
ok 10 - struct_resource
ok 11 - addr
ok 12 - escaped_str
ok 13 - hex_string
ok 14 - mac
# ip: EXPECTATION FAILED at lib/printf_kunit.c:82
vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
# ip: EXPECTATION FAILED at lib/printf_kunit.c:124
kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
not ok 15 - ip
ok 16 - uuid
ok 17 - dentry
ok 18 - struct_va_format

Quoted 8 lines

ok 19 - time_and_date
ok 20 - struct_clk
ok 21 - bitmap
ok 22 - netdev_features
ok 23 - flags
ok 24 - errptr
ok 25 - fwnode_pointer
not ok 1 - printf

Better, indeed.

But can be this improved to have a cumulative statistics, like showing only number of total, succeeded, failed with details of the latter ones?

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Is that the proper test output format? We have a standard...

What is the standard, please?

The TAP format should be the standard, no reason the kernel can not spit out the same test message format that the userspace tests do, right?

KUnit and kselftest both use the TAP format. Well, we both use slight variations on the TAP format. Nevertheless, we are both unifying around a common variation of TAP called KTAP[1]. To my knowledge, neither kselftest or KUnit fully obey the new KTAP spec, but I believe we are both working toward full compliance (as of 5.10 KUnit output should be fully compliant[2]), and both are pretty close.

When I say both kselftest and KUnit are pretty close to obeying KTAP, I mean that for the purpose of Arpitha's change here which we are discussing, the sample segment of output that she shared as the new output of this test *is fully compliant with KTAP and is not expected to change any time soon*. Additionally, most, but not all, of kselftest's tests are compliant as well.

The main visual difference between KUnit and kselftest TAP output comes from two things: kselftests tend to make heavy use of diagnostic lines (basically log lines which serve to communicate arbitrary unstructured information about the test) - KUnit has facilities for devs to use these in tests, but so far they have not gotten much use probably due to the smaller scope nature of KUnit tests. The other difference is because KUnit groups related test cases together into subtests (KUnit calls them test suites, but TAP calls them subtests) whereas kselftest does not - again this largely has to do with the style of tests. Both of these features exist in KTAP, but the output tends to reflect the kinds of tests being written.

Tim, Shuah, David (or anyone else), please let me know if I missed anything.

I hope that answers everyone's questions.

P.S. You can get KUnit results from user space separated out from dmesg if you prefer to run your tests that way[3].

[1] https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/
[2] We now support the top level test plan as specified, but do not
support SKIP tests yet; however, most KUnit tests don't need SKIP
tests anyway. See [1] for more information.
[3] https://www.kernel.org/doc/html/latest/dev-tools/kunit/usage.html#kunit-debugfs-representation

#8 lore raw
In reply to Petr Mladek
Quoted 96 lines · Petr Mladek
Quoted 95 lines · Greg KH
Quoted 94 lines · Andy Shevchenko
Quoted 92 lines · Arpitha Raghunandan

Convert test lib/test_printf.c to KUnit. More information about KUnit can be found at:
https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html.
KUnit provides a common framework for unit tests in the kernel. KUnit and kselftest are standardizing around KTAP, converting this test to KUnit makes this test output in KTAP which we are trying to make the standard test result format for the kernel. More about the KTAP format can be found at:
https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/.
I ran both the original and converted tests as is to produce the output for success of the test in the two cases. I also ran these tests with a small modification to show the difference in the output for failure of the test in both cases. The modification I made is:

- test("127.000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);
+ test("127-000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);

Original test success:

[    0.540860] test_printf: loaded.
[    0.540863] test_printf: random seed = 0x5c46c33837bc0619
[    0.541022] test_printf: all 388 tests passed

Original test failure:

[    0.537980] test_printf: loaded.
[    0.537983] test_printf: random seed = 0x1bc1efd881954afb
[    0.538029] test_printf: vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538030] test_printf: kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538124] test_printf: failed 2 out of 388 tests
[    0.538125] test_printf: random seed used was 0x1bc1efd881954afb

Converted test success:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
ok 1 - printf

Converted test failure:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    # ip: EXPECTATION FAILED at lib/printf_kunit.c:82
vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' # ip: EXPECTATION FAILED at lib/printf_kunit.c:124
kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' not ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
not ok 1 - printf

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Better, indeed.

But can be this improved to have a cumulative statistics, like showing only number of total, succeeded, failed with details of the latter ones?

Is that the proper test output format? We have a standard...

Actually more like xkcd#927.

Quoted 11 lines

What is the standard, please?

The original module produced:

[   48.577739] test_printf: loaded.
[   48.578046] test_printf: all 388 tests passed

It comes from KSTM_MODULE_LOADERS() macro that has been created by the commit eebf4dd452377921e3a26 ("kselftest: Add test module framework header").

That's a bit of a simplification. That code was clearly lifted directly from the original test_printf.c code (707cc7280f452a162c52bc240eae62568b9753c2). test_bitmap.c cloned test_printf.c (including a "Test cases for printf facility" comment...). Later both were converted to use the KSTM header.

As the one coming up with that originally, I certainly endorse its use as a standard way of producing a final, free-form, summary, and I prefer to keep that total tally (i.e. 388) being printed for the reasons I've previously stated. [*]

That said, I have absolutely nothing against _also_ producing machine-parsable TAP output. And the above looks to be a good compromise between spitting out one TAP line for each of the 388 test cases (or checks, or atoms, whatever the indiviual parts are to be called) and treating all of test_printf.c as one single PASS/FAIL test.

[*] If I add some test cases to the time_and_date and run the kernel under virtme, I want to see 388 becoming something else, so that I know that I actually ran the code I just added and not some stale vmlinux - maybe I only did "make lib/test_printf.o" and didn't recreate vmlinux, maybe I did "make vmlinux" but failed to notice the build was broken. BTDT.

And those summary lines are even more important to keep given my "deterministic random testing" series - the seed used _must_ be reported (preferably also in the "all good" case, but certainly in the "some failed" case).

Arpitha, thank you for taking that series into account. Is there some way to keep the print of the total number of "atoms" as well as the reporting of the random seed? Or should the "deterministic random testing" be done in the context of KUNIT rather than KSTM? I'm really not sure why we have both nor which one one is supposed to write against. But I would prefer that the framework, whichever one, provides a means to get a deterministic sequence of random numbers, so that I won't have to eventually copy-paste the boilerplate to test_sort.c and test_list_sort.c - it's also much nicer if all test modules have a somewhat consistent interface in terms of the module parameters they accept.

Rasmus
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees

#9 lore raw
Quoted 157 lines · Rasmus Villemoes
Quoted 96 lines · Petr Mladek
Quoted 95 lines · Greg KH
Quoted 94 lines · Andy Shevchenko
Quoted 92 lines · Arpitha Raghunandan

Convert test lib/test_printf.c to KUnit. More information about KUnit can be found at:
https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html.
KUnit provides a common framework for unit tests in the kernel. KUnit and kselftest are standardizing around KTAP, converting this test to KUnit makes this test output in KTAP which we are trying to make the standard test result format for the kernel. More about the KTAP format can be found at:
https://lore.kernel.org/linux-kselftest/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com/.
I ran both the original and converted tests as is to produce the output for success of the test in the two cases. I also ran these tests with a small modification to show the difference in the output for failure of the test in both cases. The modification I made is:

- test("127.000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);
+ test("127-000.000.001|127.0.0.1", "%pi4|%pI4", &sa.sin_addr, &sa.sin_addr);

Original test success:

[    0.540860] test_printf: loaded.
[    0.540863] test_printf: random seed = 0x5c46c33837bc0619
[    0.541022] test_printf: all 388 tests passed

Original test failure:

[    0.537980] test_printf: loaded.
[    0.537983] test_printf: random seed = 0x1bc1efd881954afb
[    0.538029] test_printf: vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538030] test_printf: kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1'
[    0.538124] test_printf: failed 2 out of 388 tests
[    0.538125] test_printf: random seed used was 0x1bc1efd881954afb

Converted test success:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
ok 1 - printf

Converted test failure:
    # Subtest: printf
    1..25
    ok 1 - test_basic
    ok 2 - test_number
    ok 3 - test_string
    ok 4 - plain
    ok 5 - null_pointer
    ok 6 - error_pointer
    ok 7 - invalid_pointer
    ok 8 - symbol_ptr
    ok 9 - kernel_ptr
    ok 10 - struct_resource
    ok 11 - addr
    ok 12 - escaped_str
    ok 13 - hex_string
    ok 14 - mac
    # ip: EXPECTATION FAILED at lib/printf_kunit.c:82
vsnprintf(buf, 256, "%pi4|%pI4", ...) wrote '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' # ip: EXPECTATION FAILED at lib/printf_kunit.c:124
kvasprintf(..., "%pi4|%pI4", ...) returned '127.000.000.001|127.0.0.1', expected '127-000.000.001|127.0.0.1' not ok 15 - ip
    ok 16 - uuid
    ok 17 - dentry
    ok 18 - struct_va_format
    ok 19 - time_and_date
    ok 20 - struct_clk
    ok 21 - bitmap
    ok 22 - netdev_features
    ok 23 - flags
    ok 24 - errptr
    ok 25 - fwnode_pointer
not ok 1 - printf

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Better, indeed.

But can be this improved to have a cumulative statistics, like showing only number of total, succeeded, failed with details of the latter ones?

Is that the proper test output format? We have a standard...

Actually more like xkcd#927.

Quoted 11 lines

What is the standard, please?

The original module produced:

[   48.577739] test_printf: loaded.
[   48.578046] test_printf: all 388 tests passed

It comes from KSTM_MODULE_LOADERS() macro that has been created by the commit eebf4dd452377921e3a26 ("kselftest: Add test module framework header").

That's a bit of a simplification. That code was clearly lifted directly from the original test_printf.c code (707cc7280f452a162c52bc240eae62568b9753c2). test_bitmap.c cloned test_printf.c (including a "Test cases for printf facility" comment...). Later both were converted to use the KSTM header.

As the one coming up with that originally, I certainly endorse its use as a standard way of producing a final, free-form, summary, and I prefer to keep that total tally (i.e. 388) being printed for the reasons I've previously stated. [*]

That said, I have absolutely nothing against _also_ producing machine-parsable TAP output. And the above looks to be a good compromise between spitting out one TAP line for each of the 388 test cases (or checks, or atoms, whatever the indiviual parts are to be called) and treating all of test_printf.c as one single PASS/FAIL test.

[*] If I add some test cases to the time_and_date and run the kernel under virtme, I want to see 388 becoming something else, so that I know that I actually ran the code I just added and not some stale vmlinux - maybe I only did "make lib/test_printf.o" and didn't recreate vmlinux, maybe I did "make vmlinux" but failed to notice the build was broken. BTDT.

And those summary lines are even more important to keep given my "deterministic random testing" series - the seed used _must_ be reported (preferably also in the "all good" case, but certainly in the "some failed" case).

Arpitha, thank you for taking that series into account. Is there some way to keep the print of the total number of "atoms" as well as the reporting of the random seed? Or should the "deterministic random testing" be done in the context of KUNIT rather than KSTM? I'm really not sure why we have both nor which one one is supposed to write against. But I would prefer that the framework, whichever one, provides a means to get a deterministic sequence of random numbers, so that I won't have to eventually copy-paste the boilerplate to test_sort.c and test_list_sort.c - it's also much nicer if all test modules have a somewhat consistent interface in terms of the module parameters they accept.

Quoted 2 lines

Rasmus

The total number of "atoms" can be printed by maintaining a static variable total_count that can be incremented as is in the original test_printf test. But, the reporting of the random seed currently is done in kselftest and so will not show up with KUnit. I am not really sure which is better in this case.

Thanks!

#10 lore raw
Quoted 5 lines · Arpitha Raghunandan

The total number of "atoms" can be printed by maintaining a static variable total_count that can be incremented as is in the original test_printf test. But, the reporting of the random seed currently is done in kselftest and so will not show up with KUnit. I am not really sure which is better in this case.

So my real questions are: Why do we have both kselftest and kunit? What's the difference? Are there any plans to merge the two frameworks? What's the point of converting from one to the other if not? And if one is to take a currently stand-alone ad hoc test module and place it under one of those umbrellas, which one should one choose?

I don't really care if the "deterministic random testing" prep work goes with kstm or kunit; whichever framework could provide that boilerplate is the framework I'd make sure to use for subsequent work on various tests.

The reporting of the number of "atoms" in the printf test suite is something I'd miss if not preserved, but if there's sufficient good rationale for doing the conversion to Kunit I could live with that. But if kunit then can't provide a per-test-module rnd_state and handle its seeding (and reporting of what seed was used), I won't ack the conversion.

Rasmus

#11 lore raw
Quoted 8 lines · Rasmus Villemoes
Quoted 5 lines · Arpitha Raghunandan

The total number of "atoms" can be printed by maintaining a static variable total_count that can be incremented as is in the original test_printf test. But, the reporting of the random seed currently is done in kselftest and so will not show up with KUnit. I am not really sure which is better in this case.

So my real questions are: Why do we have both kselftest and kunit?

One is testing code within the kernel image testing it within kernelspace, and one is outside the kernel testing it from userspace.

thanks,

greg k-h