docs: Add documentation about tests and SBIUnit
PATCH 1/4 docs: Add documentation about tests and SBIUnit
It is good when the code is covered with tests. Tests help us to keep the code clean and avoid regressions. Also, a good test is always a nice documentation for the code it covers.
This and the subsequent patches in this series introduce SBIUnit - the set of macros and functions which simplify the unit test development for OpenSBI and automate tests execution and evaluation. Also, this patch series contains two of the tests: one which covers functions from lib/sbi_bitmap.c, and another covering a part of functions from lib/sbi_console.c.
This patch contains the documentation for SBIUnit. It describes:
- What is SBIUnit
- Simple test writing scenario
- How we can cover static functions
- How we can "mock" structures in order to test the functions which operate on them
- SBIUnit API Reference
This thing is mainly inspired by the KUnit framework from the Linux Kernel, where the similar unit-test development tooling have been used successfully for a pretty long time now. I believe it would be good to have such a thing in OpenSBI as well.
--- docs/writing_tests.md | 143 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 docs/writing_tests.md
docs/writing_tests.md new file +143 -0
@@ -0,0 +1,143 @@ +Writing tests for OpenSBI +========================= + +SBIUnit +------- +SBIUnit is a set of macros and functions which simplify the test development and automate the +test execution and evaluation. All of the SBIUnit definitions could be found in +`include/sbi/sbi_unit.h` header file, and implementations are available in `lib/sbi/sbi_unit.c`. + +Simple SBIUnit test +------------------- + +For instance, we would like to test the following function from `lib/sbi/sbi_string.c`: + +```c +size_t sbi_strlen(const char *str) +{ + unsigned long ret = 0; + + while (*str != '\0') { + ret++; + str++; + } + + return ret; +} +``` + +Apparently, it calculates the string length. + +Create the file `lib/sbi/sbi_string_test.c` with the following content: + +```c +#include <sbi/sbi_unit.h> +#include <sbi/sbi_string.h> + +static void strlen_test(struct sbiunit_test_case *test) +{ + SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hello"), 5); + SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hell\0o"), 4); +} + +static struct sbiunit_test_case string_test_cases[] = { + SBIUNIT_TEST_CASE(strlen_test), + {}, +}; + +SBIUNIT_TEST_SUITE(string_test_suite, string_test_cases); +``` + +After that, add the corresponding entry to `lib/sbi/sbi_unit.c` and update the `test_suites` array: +```c +... +extern struct sbiunit_test_suite string_test_suite; +... +static struct sbiunit_test_suite *test_suites[] = { + ... + &string_test_suite, +}; +... +``` + +Add the corresponding Makefile entry to `lib/sbi/objects.mk`: +```lang-makefile +... +libsbi-objs-$(CONFIG_SBIUNIT) += sbi_string_test.o +``` + +Now recompile OpenSBI with CONFIG_SBIUNIT option enabled and run it in the QEMU. You will see +something like this: +``` +# make PLATFORM=generic run +... +# Running SBIUNIT tests # +... +## Running test suite: string_test_suite +[OK] strlen_test +1 SUCCESS / 0 FAIL / 1 TOTAL +``` + +Now let's try to change this test in the way that it will fail: + +```c +- SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hello"), 5); ++ SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hello"), 100); +``` + +Compile and run it again: +``` +... +# Running SBIUNIT tests # +... +## Running test suite: string_test_suite +strlen_test: Condition "sbi_strlen("Hello") == 100" expected to be true! +[FAIL] strlen_test +0 SUCCESS / 1 FAIL / 1 TOTAL +``` +Covering the static functions / using the static definitions +------------------------------------------------------------ + +SBIUnit also allows you to test static functions. In order to do so, simply include your test source +in the file you would like to test. Complementing the example above, just add this to the +`lib/sbi/sbi_string.c` file: + +```c +#ifdef CONFIG_SBIUNIT +#include "sbi_string_test.c" +#endif +``` + +In this case you should not add a new entry to `lib/sbi/objects.mk`, because the test code will be +included into the `sbi_string` object file. + +See example in `lib/sbi/sbi_console_test.c`, where statically declared `console_dev` variable is +used to mock the `sbi_console_device` structure. + +"Mocking" the structures +------------------------ +See the example of structure "mocking" in the `lib/sbi/sbi_console_test.c`, where the +sbi_console_device structure was mocked to be used in various console-related functions in order to +test them. + +API Reference +------------- +All of the `SBIUNIT_EXPECT_*` macros will cause a test case to fail if the corresponding conditions +are not met, however, the execution of a particular test case will not be stopped. + +All of the `SBIUNIT_ASSERT_*` macros will cause a test case to fail and stop immediately. + +- `SBIUNIT_EXPECT(test_case, condition)` - sets an expectation that 'condition' is true +- `SBIUNIT_ASSERT(test_case, condition)` - sets an assertion that 'condition' is true +- `SBIUNIT_EXPECT_EQ(test_case, a, b)` - sets an expectation that a = b +- `SBIUNIT_ASSERT_EQ(test_case, a, b)` - sets an assertion that a = b +- `SBIUNIT_EXPECT_NE(test_case, a, b)` - sets an expectation that a != b +- `SBIUNIT_ASSERT_NE(test_case, a, b)` - sets an assertion that a != b +- `SBIUNIT_EXPECT_MEMEQ(test_case, a, b, len)` - performs sbi_memcmp on memory regions a and b with +a length 'len', and sets an expectation that they are equal +- `SBIUNIT_ASSERT_MEMEQ(test_case, a, b, len)` - performs sbi_memcmp on memory regions a and b with +a length 'len', and sets an assertion that they are equal +- `SBIUNIT_EXPECT_STREQ(test_case, a, b)` - performs sbi_strcmp on strings a and b and sets +an expectation that they are equal +- `SBIUNIT_ASSERT_STREQ(test_case, a, b)` - performs sbi_strcmp on strings a and b and sets +an assertion that they are equal
Signature · 1 line
2.34.1
PATCH 2/4 lib: Add SBIUnit testing macros and functions
It is good when the code is covered with tests. Tests help us to keep the code clean and avoid regressions. Also, a good test is always a nice documentation for the code it covers.
This and the subsequent patches in this series introduce SBIUnit - the set of macros and functions which simplify the unit test development for OpenSBI and automate tests execution and evaluation.
This patch introduces all of the SBIUnit macros and functions which can be used during the test development process. Also, it defines the 'run_all_tests' function, which is being called during the 'init_coldboot' right after printing the boot hart information.
Also, add the CONFIG_SBIUNIT Kconfig entry in order to be able to turn the tests on and off. When the CONFIG_SBIUNIT is disabled, the tests and all related code should be excluded completely on the compilation stage (and, apparently, it works in this way).
--- include/sbi/sbi_unit.h | 69 ++++++++++++++++++++++++++++++++++++++++++ lib/sbi/Kconfig | 4 +++ lib/sbi/objects.mk | 1 + lib/sbi/sbi_init.c | 8 +++++ lib/sbi/sbi_unit.c | 44 +++++++++++++++++++++++++++ 5 files changed, 126 insertions(+) create mode 100644 include/sbi/sbi_unit.h create mode 100644 lib/sbi/sbi_unit.c
include/sbi/sbi_unit.h new file +69 -0
@@ -0,0 +1,69 @@ +/* +* SPDX-License-Identifier: BSD-2-Clause +* +* Author: Ivan Orlov <[email protected]> +*/ +#ifndef __SBI_UNIT_H__ +#define __SBI_UNIT_H__ + +extern struct sbiunit_test_suite *__start_sbiunit_test_suites; +extern struct sbiunit_test_suite *__end_sbiunit_test_suites; + +#include <sbi/sbi_types.h> +#include <sbi/sbi_console.h> +#include <sbi/sbi_string.h> + +#define _CONCAT (a, b) a ## b +#define CONCAT (a, b) _CONCAT(a, b) + +struct sbiunit_test_case { +char *name; +bool should_run; +bool result; +void (*test_func)(struct sbiunit_test_case *test); +void (*onerr)(struct sbiunit_test_case *test); +}; + +struct sbiunit_test_suite { +struct sbiunit_test_case *cases; +const char *name; +}; + +#define SBIUNIT_TEST_CASE (func) \ + { .name = #func, .should_run = 1, .result = 0, .test_func = &func } + +#define SBIUNIT_TEST_SUITE (suite_name, cases_arr) \ + struct sbiunit_test_suite suite_name = { .name = #suite_name, .cases = cases_arr } + +#define SBIUNIT_INFO (test, msg) sbi_printf("%s: %s", test->name, msg) + +#define SBIUNIT_EXPECT (test, cond) do { \ + if (!(cond)) { \ + test->result = 0; \ + SBIUNIT_INFO(test, "Condition \"" #cond "\" expected to be true!\n"); \ + } \ +} while (0) + +#define SBIUNIT_ASSERT (test, cond) do { \ + if (!(cond)) { \ + test->result = 0; \ + SBIUNIT_INFO(test, "Condition \"" #cond "\" must be true!\n"); \ + return; \ + } \ +} while (0) + +#define SBIUNIT_EXPECT_EQ (test, a, b) SBIUNIT_EXPECT(test, a == b) +#define SBIUNIT_ASSERT_EQ (test, a, b) SBIUNIT_ASSERT(test, a == b) +#define SBIUNIT_EXPECT_NE (test, a, b) SBIUNIT_EXPECT(test, a != b) +#define SBIUNIT_ASSERT_NE (test, a, b) SBIUNIT_ASSERT(test, a != b) +#define SBIUNIT_EXPECT_MEMEQ (test, a, b, len) \ + SBIUNIT_EXPECT(test, sbi_memcmp(a, b, len) == 0) +#define SBIUNIT_ASSERT_MEMEQ (test, a, b, len) \ + SBIUNIT_ASSERT(test, sbi_memcmp(a, b, len) == 0) +#define SBIUNIT_EXPECT_STREQ (test, a, b) \ + SBIUNIT_EXPECT(test, sbi_strcmp(a, b) == 0) +#define SBIUNIT_ASSERT_STREQ (test, a, b) \ + SBIUNIT_ASSERT(test, sbi_strcmp(a, b) == 0) + +void run_all_tests (void ); +#endif
lib/sbi/Kconfig +4 -0
@@ -50,4 +50,8 @@ config SBI_ECALL_DBTR bool "Debug Trigger Extension" default y +config SBIUNIT + bool "Enable SBIUNIT tests" + default n + endmenu
lib/sbi/objects.mk +1 -0
@@ -11,6 +11,7 @@ libsbi-objs-y += riscv_asm.o libsbi-objs-y += riscv_atomic.o libsbi-objs-y += riscv_hardfp.o libsbi-objs-y += riscv_locks.o +libsbi-objs-y += sbi_unit.o libsbi-objs-y += sbi_ecall.o libsbi-objs-y += sbi_ecall_exts.o
lib/sbi/sbi_init.c +8 -0
@@ -30,6 +30,10 @@#include <sbi/sbi_tlb.h> #include <sbi/sbi_version.h> +#ifdef CONFIG_SBIUNIT +#include <sbi/sbi_unit.h> +#endif +#define BANNER \ " ____ _____ ____ _____\n" \ " / __ \\ / ____| _ \\_ _|\n" \
@@ -398,6 +402,10 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)sbi_boot_print_hart (scratch, hartid); +#ifdef CONFIG_SBIUNIT +run_all_tests (); +#endif +/* * Configure PMP at last because if SMEPMP is detected, * M-mode access to the S/U space will be rescinded.
lib/sbi/sbi_unit.c new file +44 -0
44 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 2/4 lib: Add SBIUnit testing macros and functions
Quoted 7 lines · Ivan Orlov
It is good when the code is covered with tests. Tests help us to keep the code clean and avoid regressions. Also, a good test is always a nice documentation for the code it covers.
This and the subsequent patches in this series introduce SBIUnit - the set of macros and functions which simplify the unit test development for OpenSBI and automate tests execution and evaluation.
The above paragraphs should be dropped.
Quoted 10 lines
This patch introduces all of the SBIUnit macros and functions which can be used during the test development process. Also, it defines the '
run_all_tests' function, which is being called during the 'init_coldboot' right after printing the boot hart information.Also, add the
CONFIG_SBIUNITKconfigentry in order to be able to turn the tests on and off. When theCONFIG_SBIUNITis disabled, the tests and all related code should be excluded completely on the compilation stage (and, apparently, it works in this way).
The '(and, apparently, it works in this way)' should be dropped.
Quoted 18 lines
Signed-off-by Ivan Orlov <[email protected]>--- include/sbi/sbi_unit.h | 69 ++++++++++++++++++++++++++++++++++++++++++ lib/sbi/Kconfig | 4 +++ lib/sbi/objects.mk | 1 + lib/sbi/sbi_init.c | 8 +++++ lib/sbi/sbi_unit.c | 44 +++++++++++++++++++++++++++ 5 files changed, 126 insertions(+) create mode 100644 include/sbi/sbi_unit.h create mode 100644 lib/sbi/sbi_unit.cinclude/sbi/sbi_unit.h new file +0
-0
include/sbi/sbi_unit.h line 10
Quoted 10 lines
include/sbi/sbi_unit.h new file +10
-0@@ -0,0 +1,69 @@ +/* +* SPDX-License-Identifier: BSD-2-Clause +* +* Author: Ivan Orlov <[email protected]> +*/ +#ifndef __SBI_UNIT_H__ +#define __SBI_UNIT_H__ + +extern struct sbiunit_test_suite *__start_sbiunit_test_suites; +extern struct sbiunit_test_suite *__end_sbiunit_test_suites;
These start/end pointers don't appear to be used?
include/sbi/sbi_unit.h line 17
Quoted 7 lines
include/sbi/sbi_unit.h new file +7
-0+ +#include <sbi/sbi_types.h> +#include <sbi/sbi_console.h> +#include <sbi/sbi_string.h> + +#define _CONCAT (a, b) a ## b +#define CONCAT (a, b) _CONCAT(a, b)
This CONCAT macro isn't used.
include/sbi/sbi_unit.h line 20
Quoted 3 lines
include/sbi/sbi_unit.h new file +3
-0+ +struct sbiunit_test_case { +char *name;
const char *name ?
include/sbi/sbi_unit.h line 21
Quoted 1 line
include/sbi/sbi_unit.h new file +1
-0+bool should_run;
Rename 'should_run' to 'enabled'
include/sbi/sbi_unit.h line 29
Quoted 8 lines
include/sbi/sbi_unit.h new file +8
-0+bool result; +void (*test_func)(struct sbiunit_test_case *test); +void (*onerr)(struct sbiunit_test_case *test); +}; + +struct sbiunit_test_suite { +struct sbiunit_test_case *cases; +const char *name;
nit: I'd put name first.
include/sbi/sbi_unit.h line 33
Quoted 4 lines
include/sbi/sbi_unit.h new file +4
-0+}; + +#define SBIUNIT_TEST_CASE (func) \ + { .name = #func, .should_run = 1, .result = 0, .test_func = &func }
nit: Should use true/false instead of 1/0.
nit: No need for the '&' in front of func.
include/sbi/sbi_unit.h line 38
Quoted 5 lines
include/sbi/sbi_unit.h new file +5
-0+ +#define SBIUNIT_TEST_SUITE (suite_name, cases_arr) \ + struct sbiunit_test_suite suite_name = { .name = #suite_name, .cases = cases_arr } + +#define SBIUNIT_INFO (test, msg) sbi_printf("%s: %s", test->name, msg)
Maybe some sort of prefix like "SBIUNIT" should be on this line.
include/sbi/sbi_unit.h line 51
Quoted 13 lines
include/sbi/sbi_unit.h new file +13
-0+ +#define SBIUNIT_EXPECT (test, cond) do { \ + if (!(cond)) { \ + test->result = 0; \ + SBIUNIT_INFO(test, "Condition \"" #cond "\" expected to be true!\n"); \ + } \ +} while (0) + +#define SBIUNIT_ASSERT (test, cond) do { \ + if (!(cond)) { \ + test->result = 0; \ + SBIUNIT_INFO(test, "Condition \"" #cond "\" must be true!\n"); \ + return; \
Unnecessary 'return'. It's strange that an ASSERT macro doesn't result in an sbi_panic().
include/sbi/sbi_unit.h line 58
Quoted 7 lines
include/sbi/sbi_unit.h new file +7
-0+ } \ +} while (0) + +#define SBIUNIT_EXPECT_EQ (test, a, b) SBIUNIT_EXPECT(test, a == b) +#define SBIUNIT_ASSERT_EQ (test, a, b) SBIUNIT_ASSERT(test, a == b) +#define SBIUNIT_EXPECT_NE (test, a, b) SBIUNIT_EXPECT(test, a != b) +#define SBIUNIT_ASSERT_NE (test, a, b) SBIUNIT_ASSERT(test, a != b)
These macros need () around their arguments, e.g.
#define SBIUNIT_EXPECT_EQ(test, a, b) SBIUNIT_EXPECT(test, (a) == (b))
Quoted 41 lines
include/sbi/sbi_unit.h new file +11
-0+#define SBIUNIT_EXPECT_MEMEQ (test, a, b, len) \ + SBIUNIT_EXPECT(test, sbi_memcmp(a, b, len) == 0) +#define SBIUNIT_ASSERT_MEMEQ (test, a, b, len) \ + SBIUNIT_ASSERT(test, sbi_memcmp(a, b, len) == 0) +#define SBIUNIT_EXPECT_STREQ (test, a, b) \ + SBIUNIT_EXPECT(test, sbi_strcmp(a, b) == 0) +#define SBIUNIT_ASSERT_STREQ (test, a, b) \ + SBIUNIT_ASSERT(test, sbi_strcmp(a, b) == 0) + +void run_all_tests (void ); +#endif lib/sbi/Kconfig +4
-0@@ -50,4 +50,8 @@ config SBI_ECALL_DBTR bool "Debug Trigger Extension" default y +config SBIUNIT + bool "Enable SBIUNIT tests" + default n + endmenulib/sbi/objects.mk +1
-0@@ -11,6 +11,7 @@ libsbi-objs-y += riscv_asm.o libsbi-objs-y += riscv_atomic.o libsbi-objs-y += riscv_hardfp.o libsbi-objs-y += riscv_locks.o +libsbi-objs-y += sbi_unit.o libsbi-objs-y += sbi_ecall.o libsbi-objs-y += sbi_ecall_exts.olib/sbi/sbi_init.c +0
-0
lib/sbi/sbi_init.c line 35
Quoted 6 lines
lib/sbi/sbi_init.c +3
-0@@ -30,6 +30,10 @@#include <sbi/sbi_tlb.h> #include <sbi/sbi_version.h> +#ifdef CONFIG_SBIUNIT +#include <sbi/sbi_unit.h> +#endif
We can move the '#ifdef CONFIG_SBIUNIT' into the header and then always include it.
lib/sbi/sbi_init.c static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid) · line 407
Quoted 11 lines
lib/sbi/sbi_init.c +4
-0+#define BANNER \ " ____ _____ ____ _____\n" \ " / __ \\ / ____| _ \\_ _|\n" \@@ -398,6 +402,10 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)sbi_boot_print_hart (scratch, hartid); +#ifdef CONFIG_SBIUNIT +run_all_tests (); +#endif
I was going to suggest that we could provide a stub for !defined(CONFIG_SBIUNIT), but I sort like that the #ifdef annotates that this function is only called with that config enabled. So, either way.
Quoted 19 lines
lib/sbi/sbi_init.c +1
-0+/* * Configure PMP at last because if SMEPMP is detected, * M-mode access to the S/U space will be rescinded. lib/sbi/sbi_unit.c new file +9
-0@@ -0,0 +1,44 @@ +/* +* SPDX-License-Identifier: BSD-2-Clause +* +* Author: Ivan Orlov <[email protected]> +*/ +#include <sbi/sbi_unit.h> +#include <sbi/sbi_console.h> + +static struct sbiunit_test_suite *test_suites[] = {
lib/sbi/sbi_unit.c line 19
Quoted 10 lines
lib/sbi/sbi_unit.c new file +10
-0+}; + +static void run_test_suite (struct sbiunit_test_suite *suite) +{ +struct sbiunit_test_case *s_case; +u32 count_pass, count_fail; + +sbi_printf ("## Running test suite: %s\n" , suite->name); + count_pass =0 ; + count_fail =0 ;
nit: Can set these zero in the declarations above.
lib/sbi/sbi_unit.c line 23
Quoted 4 lines
lib/sbi/sbi_unit.c new file +4
-0+ + s_case = suite->cases; +while (s_case->should_run) { + s_case->result =1 ;
Why initialize result to pass?
lib/sbi/sbi_unit.c line 29
Quoted 6 lines
lib/sbi/sbi_unit.c new file +6
-0+ s_case->test_func (s_case); +if (s_case->result) + count_pass++; +else + count_fail++; +sbi_printf ("[%s] %s\n" , s_case->result ?"OK" :"FAIL" , s_case->name);
Is OK/FAIL what KUnit outputs? I'd expect OK/NOK or PASS/FAIL. We'll also probably want a SKIP sooner or later.
lib/sbi/sbi_unit.c line 42
Quoted 13 lines
lib/sbi/sbi_unit.c new file +13
-0+ s_case++; + } +sbi_printf ("%u SUCCESS / %u FAIL / %u TOTAL\n" , count_pass, count_fail, + count_pass + count_fail); +} + +void run_all_tests (void ) +{ +u32 i; + +sbi_printf ("\n# Running SBIUNIT tests #\n" ); + +for (i =0 ; i <sizeof (test_suites) /sizeof (test_suites[0 ]); i++)
We have array_size() in include/sbi/sbi_types.h
Quoted 6 lines
lib/sbi/sbi_unit.c new file +2
-0+run_test_suite (test_suites[i]); +}Signature · 1 line
2.34.1
Thanks,
drew
PATCH 2/4 lib: Add SBIUnit testing macros and functions
Quoted 17 lines · Andrew Jones
Quoted 7 lines · Ivan Orlov
It is good when the code is covered with tests. Tests help us to keep the code clean and avoid regressions. Also, a good test is always a nice documentation for the code it covers.
This and the subsequent patches in this series introduce SBIUnit - the set of macros and functions which simplify the unit test development for OpenSBI and automate tests execution and evaluation.
The above paragraphs should be dropped.
Quoted 6 lines
This patch introduces all of the SBIUnit macros and functions which can be used during the test development process. Also, it defines the '
run_all_tests' function, which is being called during the 'init_coldboot' right after printing the boot hart information.
Quoted 7 lines
Quoted 4 lines
Also, add the
CONFIG_SBIUNITKconfigentry in order to be able to turn the tests on and off. When theCONFIG_SBIUNITis disabled, the tests and all related code should be excluded completely on the compilation stage (and, apparently, it works in this way).The '(and, apparently, it works in this way)' should be dropped.
Thanks, I'll put this information in the cover letter of V2.
Quoted 22 lines
Quoted 22 lines
Signed-off-by Ivan Orlov <[email protected]>--- include/sbi/sbi_unit.h | 69 ++++++++++++++++++++++++++++++++++++++++++ lib/sbi/Kconfig | 4 +++ lib/sbi/objects.mk | 1 + lib/sbi/sbi_init.c | 8 +++++ lib/sbi/sbi_unit.c | 44 +++++++++++++++++++++++++++ 5 files changed, 126 insertions(+)create mode 100644
include/sbi/sbi_unit.h
create mode 100644lib/sbi/sbi_unit.cinclude/sbi/sbi_unit.h new file +4
-0@@ -0,0 +1,69 @@ +/* +* SPDX-License-Identifier: BSD-2-Clause +* +* Author: Ivan Orlov <[email protected]>
Quoted 9 lines
include/sbi/sbi_unit.h line 10
Quoted 6 lines
include/sbi/sbi_unit.h new file +6
-0+*/ +#ifndef __SBI_UNIT_H__ +#define __SBI_UNIT_H__ + +extern struct sbiunit_test_suite *__start_sbiunit_test_suites; +extern struct sbiunit_test_suite *__end_sbiunit_test_suites;These start/end pointers don't appear to be used?
Ah, I just forgot to remove them after implementing the SBIUnit version with a dedicated ELF section for the tests... Thanks!
Quoted 16 lines
include/sbi/sbi_unit.h line 17
Quoted 7 lines
include/sbi/sbi_unit.h new file +7
-0+ +#include <sbi/sbi_types.h> +#include <sbi/sbi_console.h> +#include <sbi/sbi_string.h> + +#define _CONCAT (a, b) a ## b +#define CONCAT (a, b) _CONCAT(a, b)This CONCAT macro isn't used.
include/sbi/sbi_unit.h line 20
Quoted 3 lines
include/sbi/sbi_unit.h new file +3
-0+ +struct sbiunit_test_case { +char *name;const char *name ?
Quoted 4 lines
include/sbi/sbi_unit.h line 21
Quoted 1 line
include/sbi/sbi_unit.h new file +1
-0+bool should_run;Rename '
should_run' to 'enabled'
Alright, will be done.
Quoted 27 lines
include/sbi/sbi_unit.h line 29
Quoted 8 lines
include/sbi/sbi_unit.h new file +8
-0+bool result; +void (*test_func)(struct sbiunit_test_case *test); +void (*onerr)(struct sbiunit_test_case *test); +}; + +struct sbiunit_test_suite { +struct sbiunit_test_case *cases; +const char *name;nit: I'd put name first.
include/sbi/sbi_unit.h line 33
Quoted 4 lines
include/sbi/sbi_unit.h new file +4
-0+}; + +#define SBIUNIT_TEST_CASE (func) \ + { .name = #func, .should_run = 1, .result = 0, .test_func = &func }nit: Should use true/false instead of 1/0.
nit: No need for the '&' in front of func.include/sbi/sbi_unit.h line 38
Quoted 5 lines
include/sbi/sbi_unit.h new file +5
-0+ +#define SBIUNIT_TEST_SUITE (suite_name, cases_arr) \ + struct sbiunit_test_suite suite_name = { .name = #suite_name, .cases = cases_arr } + +#define SBIUNIT_INFO (test, msg) sbi_printf("%s: %s", test->name, msg)Maybe some sort of prefix like "SBIUNIT" should be on this line.
Quoted 17 lines
include/sbi/sbi_unit.h line 51
Quoted 13 lines
include/sbi/sbi_unit.h new file +13
-0+ +#define SBIUNIT_EXPECT (test, cond) do { \ + if (!(cond)) { \ + test->result = 0; \ + SBIUNIT_INFO(test, "Condition \"" #cond "\" expected to be true!\n"); \ + } \ +} while (0) + +#define SBIUNIT_ASSERT (test, cond) do { \ + if (!(cond)) { \ + test->result = 0; \ + SBIUNIT_INFO(test, "Condition \"" #cond "\" must be true!\n"); \ + return; \Unnecessary 'return'. It's strange that an ASSERT macro doesn't result in an
sbi_panic().
I thought it would be bad if test results in hang, but if it is ok I will rewrite it.
Quoted 56 lines
include/sbi/sbi_unit.h line 58
Quoted 7 lines
include/sbi/sbi_unit.h new file +7
-0+ } \ +} while (0) + +#define SBIUNIT_EXPECT_EQ (test, a, b) SBIUNIT_EXPECT(test, a == b) +#define SBIUNIT_ASSERT_EQ (test, a, b) SBIUNIT_ASSERT(test, a == b) +#define SBIUNIT_EXPECT_NE (test, a, b) SBIUNIT_EXPECT(test, a != b) +#define SBIUNIT_ASSERT_NE (test, a, b) SBIUNIT_ASSERT(test, a != b)These macros need () around their arguments, e.g.
#define
SBIUNIT_EXPECT_EQ(test, a, b)SBIUNIT_EXPECT(test, (a) == (b))Quoted 44 lines
include/sbi/sbi_unit.h new file +11
-0+#define SBIUNIT_EXPECT_MEMEQ (test, a, b, len) \ + SBIUNIT_EXPECT(test, sbi_memcmp(a, b, len) == 0) +#define SBIUNIT_ASSERT_MEMEQ (test, a, b, len) \ + SBIUNIT_ASSERT(test, sbi_memcmp(a, b, len) == 0) +#define SBIUNIT_EXPECT_STREQ (test, a, b) \ + SBIUNIT_EXPECT(test, sbi_strcmp(a, b) == 0) +#define SBIUNIT_ASSERT_STREQ (test, a, b) \ + SBIUNIT_ASSERT(test, sbi_strcmp(a, b) == 0) + +void run_all_tests (void ); +#endif lib/sbi/Kconfig +4
-0@@ -50,4 +50,8 @@ config SBI_ECALL_DBTR bool "Debug Trigger Extension" default y +config SBIUNIT + bool "Enable SBIUNIT tests" + default n + endmenulib/sbi/objects.mk +1
-0@@ -11,6 +11,7 @@ libsbi-objs-y += riscv_asm.o libsbi-objs-y += riscv_atomic.o libsbi-objs-y += riscv_hardfp.o libsbi-objs-y += riscv_locks.o +libsbi-objs-y += sbi_unit.o libsbi-objs-y += sbi_ecall.o libsbi-objs-y += sbi_ecall_exts.olib/sbi/sbi_init.c +0
-0@@ -30,6 +30,10 @@#include <sbi/sbi_tlb.h> #include <sbi/sbi_version.h>
Quoted 7 lines
lib/sbi/sbi_init.c line 35
Quoted 3 lines
lib/sbi/sbi_init.c +3
-0+#ifdef CONFIG_SBIUNIT +#include <sbi/sbi_unit.h> +#endif We can move the '#ifdef
CONFIG_SBIUNIT' into the header and then always include it.
Yeah, makes sense.
Quoted 8 lines
Quoted 8 lines
lib/sbi/sbi_init.c +1
-0+#define BANNER \ " ____ _____ ____ _____\n" \ " / __ \\ / ____| _ \\_ _|\n" \@@ -398,6 +402,10 @@ static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid)sbi_boot_print_hart (scratch, hartid);
Quoted 8 lines
lib/sbi/sbi_init.c static void __noreturn init_coldboot(struct sbi_scratch *scratch, u32 hartid) · line 407
Quoted 3 lines
lib/sbi/sbi_init.c +3
-0+#ifdef CONFIG_SBIUNIT +run_all_tests (); +#endif I was going to suggest that we could provide a stub for !defined(CONFIG_SBIUNIT), but I sort like that the #ifdef annotates that this function is only called with that config enabled. So, either way.
I think your suggestion is better, it will help avoiding #ifdef "boilerplate" and make the code cleaner.
Quoted 32 lines
Quoted 19 lines
lib/sbi/sbi_init.c +1
-0+/* * Configure PMP at last because if SMEPMP is detected, * M-mode access to the S/U space will be rescinded. lib/sbi/sbi_unit.c new file +9
-0@@ -0,0 +1,44 @@ +/* +* SPDX-License-Identifier: BSD-2-Clause +* +* Author: Ivan Orlov <[email protected]> +*/ +#include <sbi/sbi_unit.h> +#include <sbi/sbi_console.h> + +static struct sbiunit_test_suite *test_suites[] = {lib/sbi/sbi_unit.c line 19
Quoted 10 lines
lib/sbi/sbi_unit.c new file +10
-0+}; + +static void run_test_suite (struct sbiunit_test_suite *suite) +{ +struct sbiunit_test_case *s_case; +u32 count_pass, count_fail; + +sbi_printf ("## Running test suite: %s\n" , suite->name); + count_pass =0 ; + count_fail =0 ;nit: Can set these zero in the declarations above.
Quoted 6 lines
lib/sbi/sbi_unit.c line 23
Quoted 4 lines
lib/sbi/sbi_unit.c new file +4
-0+ + s_case = suite->cases; +while (s_case->should_run) { + s_case->result =1 ;Why initialize result to pass?
The logic was to set the 'result' to 0 in the SBIUNIT_ASSERT or SBIUNIT_EXPECT, but it really seems confusing so I agree that it might be better to create a 'failed' field and set it to 'true' in the assert/expect.
Quoted 11 lines
lib/sbi/sbi_unit.c line 29
Quoted 7 lines
lib/sbi/sbi_unit.c new file +6
-0+ s_case->test_func (s_case); +if (s_case->result) + count_pass++; +else + count_fail++; +sbi_printf ("[%s] %s\n" , s_case->result ?"OK" :"FAIL" , s_case->name);Is OK/FAIL what KUnit outputs? I'd expect OK/NOK or PASS/FAIL. We'll also probably want a SKIP sooner or later.
It looks like KUnit uses 'FAILED' or 'PASSED'. So I will update the messages correspondingly.
Quoted 7 lines
Quoted 7 lines
lib/sbi/sbi_unit.c new file +7
-0+ s_case++; + } +sbi_printf ("%u SUCCESS / %u FAIL / %u TOTAL\n" , count_pass, count_fail, + count_pass + count_fail); +} + +void run_all_tests (void )
Quoted 9 lines
lib/sbi/sbi_unit.c line 42
Quoted 6 lines
lib/sbi/sbi_unit.c new file +6
-0+{ +u32 i; + +sbi_printf ("\n# Running SBIUNIT tests #\n" ); + +for (i =0 ; i <sizeof (test_suites) /sizeof (test_suites[0 ]); i++)We have
array_size()ininclude/sbi/sbi_types.h
Cool, I'll use it instead.
Thank you!
Signature · 2 lines
Kind regards, Ivan Orlov
PATCH 2/4 lib: Add SBIUnit testing macros and functions
Quoted 2 lines · Ivan Orlov
Quoted 1 line · Andrew Jones
On Thu, Feb 08, 2024 at 09:50:48AM +0000, Ivan Orlov wrote:
...
Quoted 10 lines
Quoted 9 lines
include/sbi/sbi_unit.h line 51
Quoted 5 lines
include/sbi/sbi_unit.h new file +5
-0+#define SBIUNIT_ASSERT (test, cond) do { \ + if (!(cond)) { \ + test->result = 0; \ + SBIUNIT_INFO(test, "Condition \"" #cond "\" must be true!\n"); \ + return; \Unnecessary 'return'. It's strange that an ASSERT macro doesn't result in an
sbi_panic().
Quoted 2 lines
I thought it would be bad if test results in hang, but if it is ok I will rewrite it.
I think an assert should output its message and then stop all execution. Asserts should only be used in situations where failures mean the test programmer made an improper assumption. If we don't halt the moment we determine the assumption is wrong, then we'll potentially generate passing test results, but the tests may not be testing what the test programmer thought they were.
Thanks,
drew
PATCH 2/4 lib: Add SBIUnit testing macros and functions
Quoted 17 lines · Andrew Jones
Quoted 2 lines · Ivan Orlov
Quoted 1 line · Andrew Jones
On Thu, Feb 08, 2024 at 09:50:48AM +0000, Ivan Orlov wrote:
...
Quoted 10 lines
Quoted 9 lines
include/sbi/sbi_unit.h line 51
Quoted 5 lines
include/sbi/sbi_unit.h new file +5
-0+#define SBIUNIT_ASSERT (test, cond) do { \ + if (!(cond)) { \ + test->result = 0; \ + SBIUNIT_INFO(test, "Condition \"" #cond "\" must be true!\n"); \ + return; \Unnecessary 'return'. It's strange that an ASSERT macro doesn't result in an
sbi_panic().Quoted 2 lines
I thought it would be bad if test results in hang, but if it is ok I will rewrite it.
Quoted 7 lines
I think an assert should output its message and then stop all execution. Asserts should only be used in situations where failures mean the test programmer made an improper assumption. If we don't halt the moment we determine the assumption is wrong, then we'll potentially generate passing test results, but the tests may not be testing what the test programmer thought they were.
Makes sense, so let's add the sbi_panic in version 2.
Thanks!
Signature · 2 lines
Kind regards, Ivan Orlov
PATCH 3/4 lib: tests: Add a test for sbi_bitmap
In reply to Ivan OrlovAdd test suite covering all of the functions from lib/sbi/sbi_bitmap.c:__bitmap_and, __bitmap_or and __bitmap_xor.
This patch depends on the previous patches in this series as it uses SBIUnit macros, functions and definitions
--- lib/sbi/objects.mk | 2 + lib/sbi/sbi_bitmap_test.c | 104 ++++++++++++++++++++++++++++++++++++++ lib/sbi/sbi_unit.c | 2 + 3 files changed, 108 insertions(+) create mode 100644 lib/sbi/sbi_bitmap_test.c
lib/sbi/objects.mk +2 -0
@@ -56,6 +56,8 @@ libsbi-objs-$(CONFIG_SBI_ECALL_VENDOR) += sbi_ecall_vendor.o carray-sbi_ecall_exts-$(CONFIG_SBI_ECALL_DBTR) += ecall_dbtr libsbi-objs-$(CONFIG_SBI_ECALL_DBTR) += sbi_ecall_dbtr.o +libsbi-objs-$(CONFIG_SBIUNIT) += sbi_bitmap_test.o + libsbi-objs-y += sbi_bitmap.o libsbi-objs-y += sbi_bitops.o libsbi-objs-y += sbi_console.o
lib/sbi/sbi_bitmap_test.c new file +104 -0
104 lines not on this page · raw message
lib/sbi/sbi_unit.c +2 -0
@@ -6,7 +6,9 @@#include <sbi/sbi_unit.h> #include <sbi/sbi_console.h> +extern struct sbiunit_test_suite bitmap_test_suite;static struct sbiunit_test_suite *test_suites[] = { + &bitmap_test_suite, };static void run_test_suite (struct sbiunit_test_suite *suite)
Signature · 1 line
2.34.1
PATCH 3/4 lib: tests: Add a test for sbi_bitmap
Quoted 5 lines · Ivan Orlov
Add test suite covering all of the functions from
lib/sbi/sbi_bitmap.c:__bitmap_and,__bitmap_orand__bitmap_xor.This patch depends on the previous patches in this series as it uses SBIUnit macros, functions and definitions
No need to state that a patch depends on a previous patch, as that's generally the case.
Quoted 31 lines
Signed-off-by Ivan Orlov <[email protected]>--- lib/sbi/objects.mk | 2 + lib/sbi/sbi_bitmap_test.c | 104 ++++++++++++++++++++++++++++++++++++++ lib/sbi/sbi_unit.c | 2 + 3 files changed, 108 insertions(+) create mode 100644 lib/sbi/sbi_bitmap_test.clib/sbi/objects.mk +2
-0@@ -56,6 +56,8 @@ libsbi-objs-$(CONFIG_SBI_ECALL_VENDOR) += sbi_ecall_vendor.o carray-sbi_ecall_exts-$(CONFIG_SBI_ECALL_DBTR) += ecall_dbtr libsbi-objs-$(CONFIG_SBI_ECALL_DBTR) += sbi_ecall_dbtr.o +libsbi-objs-$(CONFIG_SBIUNIT) += sbi_bitmap_test.o + libsbi-objs-y += sbi_bitmap.o libsbi-objs-y += sbi_bitops.o libsbi-objs-y += sbi_console.olib/sbi/sbi_bitmap_test.c new file +3
-0@@ -0,0 +1,104 @@ +/* +* SPDX-License-Identifier: BSD-2-Clause +*
lib/sbi/sbi_bitmap_test.c line 13
Quoted 10 lines
lib/sbi/sbi_bitmap_test.c new file +10
-0+* Author: Ivan Orlov <[email protected]> +*/ +#include <sbi/sbi_bitmap.h> +#include <sbi/sbi_unit.h> +#include <sbi/sbi_console.h> + +static u64 data_a[] = {0xDEADBEEF ,0x00BAB10C ,0x1BADB002 ,0xABADBABE }; +static u64 data_b[] = {0xC00010FF ,0x00BAB10C ,0xBAAAAAAD ,0xBADDCAFE }; +static u64 data_zero[] = {0 ,0 ,0 ,0 }; +
nit: Remove one blank line and the below defines up above the globals.
lib/sbi/sbi_bitmap_test.c line 22
Quoted 9 lines
lib/sbi/sbi_bitmap_test.c new file +9
-0+ +#define DATA_SIZE sizeof(data_zero) +#define DATA_BIT_SIZE (DATA_SIZE * 8) + +static void bitmap_and_test (struct sbiunit_test_case *test) +{ +u64 res[DATA_SIZE ]; +u64 a_and_b[] = {0xDEADBEEF &0xC00010FF ,0x00BAB10C &0x00BAB10C , +0x1BADB002 &0xBAAAAAAD ,0xABADBABE &0xBADDCAFE };
Should refer to the test data by name, e.g. data_a[0] or with a define, '#define DATA_A0 0xDEADBEEFULL'
lib/sbi/sbi_bitmap_test.c line 27
Quoted 5 lines
lib/sbi/sbi_bitmap_test.c new file +5
-0+ +__bitmap_and (res, data_a, data_b,DATA_BIT_SIZE ); +SBIUNIT_EXPECT_MEMEQ (test, res, a_and_b,DATA_SIZE ); + +// a & a = a
Should use /**/ comment format.
Quoted 95 lines
lib/sbi/sbi_bitmap_test.c new file +77
-0+__bitmap_and (res, data_a, data_a,DATA_BIT_SIZE ); +SBIUNIT_ASSERT_MEMEQ (test, res, data_a,DATA_SIZE ); + +// a & 0 = 0 +__bitmap_and (res, data_a, data_zero,DATA_BIT_SIZE ); +SBIUNIT_EXPECT_MEMEQ (test, res, data_zero,DATA_SIZE ); + +// 0 & 0 = 0 +__bitmap_and (res, data_zero, data_zero,DATA_BIT_SIZE ); +SBIUNIT_EXPECT_MEMEQ (test, res, data_zero,DATA_SIZE ); + +sbi_memcpy (res, data_zero,DATA_SIZE ); +// Cover zero 'bits' argument +__bitmap_and (res, data_a, data_b,0 ); +SBIUNIT_EXPECT_MEMEQ (test, res, data_zero,DATA_SIZE ); +} + +static void bitmap_or_test (struct sbiunit_test_case *test) +{ +u64 res[DATA_SIZE ]; +u64 a_or_b[] = {0xDEADBEEF |0xC00010FF ,0x00BAB10C |0x00BAB10C , +0x1BADB002 |0xBAAAAAAD ,0xABADBABE |0xBADDCAFE }; + +__bitmap_or (res, data_a, data_b,DATA_BIT_SIZE ); +SBIUNIT_EXPECT_MEMEQ (test, res, a_or_b,DATA_SIZE ); + +// a | a = a +__bitmap_or (res, data_a, data_a,DATA_BIT_SIZE ); +SBIUNIT_EXPECT_MEMEQ (test, res, data_a,DATA_SIZE ); + +// a | 0 = a +__bitmap_or (res, data_a, data_zero,DATA_BIT_SIZE ); +SBIUNIT_EXPECT_MEMEQ (test, res, data_a,DATA_SIZE ); + +// 0 | 0 = 0 +__bitmap_or (res, data_zero, data_zero,DATA_BIT_SIZE ); +SBIUNIT_EXPECT_MEMEQ (test, res, data_zero,DATA_SIZE ); + +sbi_memcpy (res, data_zero,DATA_SIZE ); +__bitmap_or (res, data_a, data_b,0 ); +SBIUNIT_EXPECT_MEMEQ (test, res, data_zero,DATA_SIZE ); +} + +static void bitmap_xor_test (struct sbiunit_test_case *test) +{ +u64 res[DATA_SIZE ]; +u64 a_xor_b[] = {0xDEADBEEF ^0xC00010FF ,0x00BAB10C ^0x00BAB10C , +0x1BADB002 ^0xBAAAAAAD ,0xABADBABE ^0xBADDCAFE }; + +__bitmap_xor (res, data_a, data_b,DATA_BIT_SIZE ); +SBIUNIT_EXPECT_MEMEQ (test, res, a_xor_b,DATA_SIZE ); + +// a ^ 0 = a +__bitmap_xor (res, data_a, data_zero,DATA_BIT_SIZE ); +SBIUNIT_EXPECT_MEMEQ (test, res, data_a,DATA_SIZE ); + +// a ^ a = 0 +__bitmap_xor (res, data_a, data_a,DATA_BIT_SIZE ); +SBIUNIT_EXPECT_MEMEQ (test, res, data_zero,DATA_SIZE ); + +// 0 ^ 0 = 0 +__bitmap_xor (res, data_zero, data_zero,DATA_BIT_SIZE ); +SBIUNIT_EXPECT_MEMEQ (test, res, data_zero,DATA_SIZE ); + +sbi_memcpy (res, data_zero,DATA_SIZE ); +__bitmap_xor (res, data_a, data_b,0 ); +SBIUNIT_EXPECT_MEMEQ (test, res, data_zero,DATA_SIZE ); +} + +static struct sbiunit_test_case bitmap_test_cases[] = { +SBIUNIT_TEST_CASE (bitmap_and_test), +SBIUNIT_TEST_CASE (bitmap_or_test), +SBIUNIT_TEST_CASE (bitmap_xor_test), + {}, +}; + +SBIUNIT_TEST_SUITE (bitmap_test_suite, bitmap_test_cases);lib/sbi/sbi_unit.c +2
-0@@ -6,7 +6,9 @@#include <sbi/sbi_unit.h> #include <sbi/sbi_console.h> +extern struct sbiunit_test_suite bitmap_test_suite;static struct sbiunit_test_suite *test_suites[] = { + &bitmap_test_suite, };static void run_test_suite (struct sbiunit_test_suite *suite)Signature · 1 line
2.34.1
Thanks,
drew
PATCH 4/4 lib: tests: Add sbi_console test
In reply to Ivan OrlovAdd the test suite covering some of the functions from lib/sbi/sbi_console.c: putc, puts and printf. The test covers a variety of format specifiers for printf and different strings and characters for putc and puts.
In order to do that, the test "mocks" the sbi_console_device structure by setting the 'console_dev' variable to the virtual console.
This patch depends on the previous patches from the series as it uses SBIUnit macros, functions and definitions.
--- lib/sbi/sbi_console.c | 4 ++ lib/sbi/sbi_console_test.c | 104 +++++++++++++++++++++++++++++++++++++ lib/sbi/sbi_unit.c | 2 + 3 files changed, 110 insertions(+) create mode 100644 lib/sbi/sbi_console_test.c
lib/sbi/sbi_console.c +4 -0
@@ -488,3 +488,7 @@ int sbi_console_init(struct sbi_scratch *scratch)return rc; } + +#ifdef CONFIG_SBIUNIT +#include "sbi_console_test.c" +#endif
lib/sbi/sbi_console_test.c new file +104 -0
104 lines not on this page · raw message
lib/sbi/sbi_unit.c +2 -0
@@ -7,8 +7,10 @@#include <sbi/sbi_console.h> extern struct sbiunit_test_suite bitmap_test_suite; +extern struct sbiunit_test_suite console_test_suite;static struct sbiunit_test_suite *test_suites[] = { &bitmap_test_suite, + &console_test_suite, };static void run_test_suite (struct sbiunit_test_suite *suite)
Signature · 1 line
2.34.1
PATCH 4/4 lib: tests: Add sbi_console test
Quoted 10 lines · Ivan Orlov
Add the test suite covering some of the functions from
lib/sbi/sbi_console.c: putc, puts and printf. The test covers a variety of format specifiers for printf and different strings and characters for putc and puts.In order to do that, the test "mocks" the
sbi_console_devicestructure by setting the 'console_dev' variable to the virtual console.This patch depends on the previous patches from the series as it uses SBIUnit macros, functions and definitions.
Again, this last sentence isn't necessary to write in a commit message.
Quoted 35 lines
Signed-off-by Ivan Orlov <[email protected]>--- lib/sbi/sbi_console.c | 4 ++ lib/sbi/sbi_console_test.c | 104 +++++++++++++++++++++++++++++++++++++ lib/sbi/sbi_unit.c | 2 + 3 files changed, 110 insertions(+) create mode 100644 lib/sbi/sbi_console_test.clib/sbi/sbi_console.c +4
-0@@ -488,3 +488,7 @@ int sbi_console_init(struct sbi_scratch *scratch)return rc; } + +#ifdef CONFIG_SBIUNIT +#include "sbi_console_test.c" +#endif lib/sbi/sbi_console_test.c new file +8
-0@@ -0,0 +1,104 @@ +/* +* SPDX-License-Identifier: BSD-2-Clause +* +* Author: Ivan Orlov <[email protected]> +*/ +#include <sbi/sbi_unit.h> +#include <sbi/sbi_heap.h> +
lib/sbi/sbi_console_test.c line 18
Quoted 10 lines
lib/sbi/sbi_console_test.c new file +10
-0+/* +* Console functions are wrapped in order to mock the console object and don't affect the actual +* console output +*/ +#define CONSOLE_DO (action) ({ \ + old_dev = console_dev; \ + console_dev = &new_dev; \ + action; \ + console_dev = old_dev; \ +})
Why not use a local variable for old_dev and pass 'new_dev' in as an argument?
lib/sbi/sbi_console_test.c line 20
Quoted 2 lines
lib/sbi/sbi_console_test.c new file +2
-0+ +// We are using a GCC extension here, which allows us to return a value from a block
No need for this comment.
lib/sbi/sbi_console_test.c line 24
Quoted 4 lines
lib/sbi/sbi_console_test.c new file +4
-0+#define CONSOLE_DO_RET (action) ({ \ + old_dev = console_dev; \ + console_dev = &new_dev; \ + u64 res = action; \
Please use names like __res inside macros to avoid shadowing concerns.
lib/sbi/sbi_console_test.c line 27
Quoted 3 lines
lib/sbi/sbi_console_test.c new file +3
-0+ console_dev = old_dev; \ + res; \ +})
We only need CONSOLE_DO_RET(). The caller can ignore the return value. And its name should include an TEST_ prefix, because this file is included by other files.
lib/sbi/sbi_console_test.c line 29
Quoted 2 lines
lib/sbi/sbi_console_test.c new file +2
-0+ +#define BUF_LEN 1024
TEST_CONSOLE_BUF_LEN
lib/sbi/sbi_console_test.c line 34
Quoted 5 lines
lib/sbi/sbi_console_test.c new file +5
-0+ +static const struct sbi_console_device *old_dev; + +static char buf[BUF_LEN ]; +static u32 pos;
All the above need a test_console_ prefix
lib/sbi/sbi_console_test.c line 44
Quoted 10 lines
lib/sbi/sbi_console_test.c new file +10
-0+static void test_console_putc (char c) +{ + buf[pos] = c; + pos = (pos +1 ) %BUF_LEN ; +} + +static void clear_buf (void ) +{ + pos =0 ; +sbi_memset (buf,0 ,BUF_LEN );
I guess buf[0] = '\0' should be sufficient.
lib/sbi/sbi_console_test.c line 47
Quoted 3 lines
lib/sbi/sbi_console_test.c new file +3
-0+} + +// Mocking the console
/**/ comment style
lib/sbi/sbi_console_test.c line 57
Quoted 10 lines
lib/sbi/sbi_console_test.c new file +10
-0+static const struct sbi_console_device new_dev = { + .name ="Test console device" , + .console_putc = test_console_putc, +}; + +static void putc_test (struct sbiunit_test_case *test) +{ +clear_buf (); + +CONSOLE_DO (sbi_putc ('a' ));
I think I would replace the CONSOLE_DO macro with a _begin() / _end() pair.
test_console_begin();
sbi_putc('a');
test_console_end();
lib/sbi/sbi_console_test.c line 61
Quoted 4 lines
lib/sbi/sbi_console_test.c new file +4
-0+SBIUNIT_ASSERT_EQ (test, buf[0 ],'a' ); +} + +#define PUTS_TEST (test, expected, param) do { \
param is the string, so it should be named 's' or 'str' or similar.
Quoted 62 lines
lib/sbi/sbi_console_test.c new file +43
-0+clear_buf (); \ +CONSOLE_DO (sbi_puts (param)); \ +SBIUNIT_ASSERT_STREQ (test, buf, expected); \ +}while (0 ) + +static void puts_test(struct sbiunit_test_case *test) +{ +PUTS_TEST (test,"Hello, OpenSBI!" ,"Hello, OpenSBI!" ); +PUTS_TEST (test,"Hello,\r\nOpenSBI!" ,"Hello,\nOpenSBI!" ); +PUTS_TEST (test,"Hello," ,"Hello,\0OpenSBI!" ); +} + +#define PRINTF_TEST (test, expected, format, ...) do { \ + clear_buf(); \ + SBIUNIT_ASSERT_EQ(test, CONSOLE_DO_RET(sbi_printf(format, ##__VA_ARGS__)), \ + sbi_strlen(expected)); \ + SBIUNIT_ASSERT_STREQ(test, buf, expected); \ +} while (0) + +static void printf_test (struct sbiunit_test_case *test) +{ +PRINTF_TEST (test,"Hello" ,"Hello" ); +PRINTF_TEST (test,"3 5 7" ,"%d %d %d" ,3 ,5 ,7 ); +PRINTF_TEST (test,"Hello" ,"%s" ,"Hello" ); +PRINTF_TEST (test,"-1" ,"%d" ,-1 ); +PRINTF_TEST (test,"FF" ,"%X" ,255 ); +PRINTF_TEST (test,"ff" ,"%x" ,255 ); +PRINTF_TEST (test,"A" ,"%c" ,'A' ); +PRINTF_TEST (test,"1fe" ,"%p" , (void *)0x1fe ); +PRINTF_TEST (test,"4294967295" ,"%u" ,4294967295U ); +PRINTF_TEST (test,"-2147483647" ,"%ld" ,-2147483647l ); +PRINTF_TEST (test,"-9223372036854775807" ,"%lld" ,-9223372036854775807LL ); +PRINTF_TEST (test,"18446744073709551615" ,"%llu" ,18446744073709551615ULL ); +} + +static struct sbiunit_test_case console_test_cases[] = { +SBIUNIT_TEST_CASE (putc_test), +SBIUNIT_TEST_CASE (puts_test), +SBIUNIT_TEST_CASE (printf_test), + {}, +}; + +SBIUNIT_TEST_SUITE (console_test_suite, console_test_cases);lib/sbi/sbi_unit.c +2
-0@@ -7,8 +7,10 @@#include <sbi/sbi_console.h> extern struct sbiunit_test_suite bitmap_test_suite; +extern struct sbiunit_test_suite console_test_suite;static struct sbiunit_test_suite *test_suites[] = { &bitmap_test_suite, + &console_test_suite, };static void run_test_suite (struct sbiunit_test_suite *suite)Signature · 1 line
2.34.1
Thanks,
drew
PATCH 4/4 lib: tests: Add sbi_console test
Quoted 96 lines · Andrew Jones
Quoted 10 lines · Ivan Orlov
Add the test suite covering some of the functions from
lib/sbi/sbi_console.c: putc, puts and printf. The test covers a variety of format specifiers for printf and different strings and characters for putc and puts.In order to do that, the test "mocks" the
sbi_console_devicestructure by setting the 'console_dev' variable to the virtual console.This patch depends on the previous patches from the series as it uses SBIUnit macros, functions and definitions.
Again, this last sentence isn't necessary to write in a commit message.
Quoted 35 lines
Signed-off-by Ivan Orlov <[email protected]>--- lib/sbi/sbi_console.c | 4 ++ lib/sbi/sbi_console_test.c | 104 +++++++++++++++++++++++++++++++++++++ lib/sbi/sbi_unit.c | 2 + 3 files changed, 110 insertions(+)create mode 100644
lib/sbi/sbi_console_test.clib/sbi/sbi_console.c +4
-0@@ -488,3 +488,7 @@ int sbi_console_init(struct sbi_scratch *scratch)return rc; } + +#ifdef CONFIG_SBIUNIT +#include "sbi_console_test.c" +#endif lib/sbi/sbi_console_test.c new file +8
-0@@ -0,0 +1,104 @@ +/* +* SPDX-License-Identifier: BSD-2-Clause +* +* Author: Ivan Orlov <[email protected]> +*/ +#include <sbi/sbi_unit.h> +#include <sbi/sbi_heap.h> +lib/sbi/sbi_console_test.c line 18
Quoted 10 lines
lib/sbi/sbi_console_test.c new file +10
-0+/* +* Console functions are wrapped in order to mock the console object and don't affect the actual +* console output +*/ +#define CONSOLE_DO (action) ({ \ + old_dev = console_dev; \ + console_dev = &new_dev; \ + action; \ + console_dev = old_dev; \ +})Why not use a local variable for
old_devand pass 'new_dev' in as an argument?lib/sbi/sbi_console_test.c line 20
Quoted 2 lines
lib/sbi/sbi_console_test.c new file +2
-0+ +// We are using a GCC extension here, which allows us to return a value from a block No need for this comment.
lib/sbi/sbi_console_test.c line 24
Quoted 4 lines
lib/sbi/sbi_console_test.c new file +4
-0+#define CONSOLE_DO_RET (action) ({ \ + old_dev = console_dev; \ + console_dev = &new_dev; \ + u64 res = action; \Please use names like
__resinside macros to avoid shadowing concerns.lib/sbi/sbi_console_test.c line 27
Quoted 3 lines
lib/sbi/sbi_console_test.c new file +3
-0+ console_dev = old_dev; \ + res; \ +})We only need
CONSOLE_DO_RET(). The caller can ignore the return value. And its name should include anTEST_prefix, because this file is included by other files.lib/sbi/sbi_console_test.c line 29
Quoted 2 lines
lib/sbi/sbi_console_test.c new file +2
-0+ +#define BUF_LEN 1024
TEST_CONSOLE_BUF_LENlib/sbi/sbi_console_test.c line 34
Quoted 5 lines
lib/sbi/sbi_console_test.c new file +5
-0+ +static const struct sbi_console_device *old_dev; + +static char buf[BUF_LEN ]; +static u32 pos;All the above need a
test_console_prefix
Quoted 13 lines
lib/sbi/sbi_console_test.c line 44
Quoted 10 lines
lib/sbi/sbi_console_test.c new file +10
-0+static void test_console_putc (char c) +{ + buf[pos] = c; + pos = (pos +1 ) %BUF_LEN ; +} + +static void clear_buf (void ) +{ + pos =0 ; +sbi_memset (buf,0 ,BUF_LEN );I guess buf[0] = '\0' should be sufficient.
Sorry for the late reply, I agree on all of the points you mentioned except this one. The 'puts' test stops me from clearing the buffer by setting the first char to zero:
``` PUTS_TEST(test, "Hello,", "Hello,\0OpenSBI!"); ```
This test checks if 'puts' stops printing after facing \0. 'puts' won't print out the \0 to the buffer after printing "Hello,", so in case of the 'lazy' buffer clearing there might be other characters in buffer after 'Hello,'. In this case, the 'sbi_strcmp' won't stop comparing the strings causing the test to fail (despite the behavior is correct).
I reckon that clearing the buffer completely will help avoiding such tricky issues by making the experiment as pure as possible :)
Thank you!
Signature · 2 lines
Kind regards, Ivan Orlov
PATCH 4/4 lib: tests: Add sbi_console test
Quoted 2 lines · Ivan Orlov
Quoted 1 line · Andrew Jones
On Thu, Feb 08, 2024 at 09:50:50AM +0000, Ivan Orlov wrote:
...
Quoted 16 lines
Quoted 7 lines
lib/sbi/sbi_console_test.c line 44
Quoted 4 lines
lib/sbi/sbi_console_test.c new file +4
-0+static void clear_buf (void ) +{ + pos =0 ; +sbi_memset (buf,0 ,BUF_LEN );I guess buf[0] = '\0' should be sufficient.
Sorry for the late reply, I agree on all of the points you mentioned except this one. The 'puts' test stops me from clearing the buffer by setting the first char to zero:
``` PUTS_TEST(test, "Hello,", "Hello,\0OpenSBI!"); ```
Quoted 8 lines
This test checks if 'puts' stops printing after facing \0. 'puts' won't print out the \0 to the buffer after printing "Hello,", so in case of the 'lazy' buffer clearing there might be other characters in buffer after 'Hello,'. In this case, the '
sbi_strcmp' won't stop comparing the strings causing the test to fail (despite the behavior is correct).I reckon that clearing the buffer completely will help avoiding such tricky issues by making the experiment as pure as possible :)
Since the mock console device is a ring buffer it doesn't guarantee zeros will follow the printed strings. I suggest writing tests which only confirm the characters printed are in the stream and in the right order, but not check for anything console device implementation specific.
Thanks,
drew
PATCH 4/4 lib: tests: Add sbi_console test
Quoted 26 lines · Andrew Jones
Quoted 2 lines · Ivan Orlov
Quoted 1 line · Andrew Jones
On Thu, Feb 08, 2024 at 09:50:50AM +0000, Ivan Orlov wrote:
...
Quoted 22 lines
Quoted 7 lines
lib/sbi/sbi_console_test.c line 44
Quoted 4 lines
lib/sbi/sbi_console_test.c new file +4
-0+static void clear_buf (void ) +{ + pos =0 ; +sbi_memset (buf,0 ,BUF_LEN );I guess buf[0] = '\0' should be sufficient.
Sorry for the late reply, I agree on all of the points you mentioned except this one. The 'puts' test stops me from clearing the buffer by setting the first char to zero:
``` PUTS_TEST(test, "Hello,", "Hello,\0OpenSBI!"); ```This test checks if 'puts' stops printing after facing \0. 'puts' won't print out the \0 to the buffer after printing "Hello,", so in case of the 'lazy' buffer clearing there might be other characters in buffer after 'Hello,'. In this case, the '
sbi_strcmp' won't stop comparing the strings causing the test to fail (despite the behavior is correct).
Quoted 8 lines
Quoted 2 lines
I reckon that clearing the buffer completely will help avoiding such tricky issues by making the experiment as pure as possible :)
Since the mock console device is a ring buffer it doesn't guarantee zeros will follow the printed strings. I suggest writing tests which only confirm the characters printed are in the stream and in the right order, but not check for anything console device implementation specific.
Ah, alright, I see. Moreover, this particular test case is just useless since it doesn't increase the coverage. So I guess I will just use the "lazy" buffer clearing and remove this particular 'PUTS_TEST'. Thanks for the clarification!
Signature · 2 lines
Kind regards, Ivan Orlov
PATCH 1/4 docs: Add documentation about tests and SBIUnit
In reply to Ivan OrlovQuoted 10 lines · Ivan Orlov
It is good when the code is covered with tests. Tests help us to keep the code clean and avoid regressions. Also, a good test is always a nice documentation for the code it covers.
This and the subsequent patches in this series introduce SBIUnit - the set of macros and functions which simplify the unit test development for OpenSBI and automate tests execution and evaluation. Also, this patch series contains two of the tests: one which covers functions from
lib/sbi_bitmap.c, and another covering a part of functions fromlib/sbi_console.c.
The above should be in the cover letter, which this series is missing. Cover letters are nice since they give reviewers a place to comment on the overall series.
Quoted 10 lines
This patch contains the documentation for SBIUnit. It describes:
- What is SBIUnit
- Simple test writing scenario
- How we can cover static functions
- How we can "mock" structures in order to test the functions which operate on them
- SBIUnit API Reference
Quoted 4 lines
This thing is mainly inspired by the KUnit framework from the Linux Kernel, where the similar unit-test development tooling have been used successfully for a pretty long time now. I believe it would be good to have such a thing in OpenSBI as well.
The above paragraph belongs in the cover letter along with elaboration on how only certain parts of KUnit were reimplemented and why those certain parts, and not others, were chosen.
Quoted 13 lines
Signed-off-by Ivan Orlov <[email protected]>--- docs/writing_tests.md | 143 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 docs/writing_tests.mddocs/writing_tests.md new file +0
-0
docs/writing_tests.md line 6
Quoted 6 lines
docs/writing_tests.md new file +6
-0@@ -0,0 +1,143 @@ +Writing tests for OpenSBI +========================= + +SBIUnit +------- +SBIUnit is a set of macros and functions which simplify the test development and automate the
We aim for 80 char line length, especially in documentation.
docs/writing_tests.md line 7
Quoted 1 line
docs/writing_tests.md new file +1
-0+test execution and evaluation. All of the SBIUnit definitions could be found in
s/could be found in/are in the/
docs/writing_tests.md line 8
Quoted 1 line
docs/writing_tests.md new file +1
-0+`include/sbi/sbi_unit.h` header file, and implementations are available in `lib/sbi/sbi_unit.c`.
I think the string 'test' should be somewhere in the filenames. 'unit' is too generic.
Quoted 11 lines
docs/writing_tests.md new file +11
-0+ +Simple SBIUnit test +------------------- + +For instance, we would like to test the following function from `lib/sbi/sbi_string.c`: + +```c +size_t sbi_strlen(const char *str) +{ + unsigned long ret = 0; +
docs/writing_tests.md line 29
Quoted 10 lines
docs/writing_tests.md new file +10
-0+ while (*str != '\0') { + ret++; + str++; + } + + return ret; +} +``` + +Apparently, it calculates the string length.
s/Apparently, it/which/
Quoted 12 lines
docs/writing_tests.md new file +12
-0+ +Create the file `lib/sbi/sbi_string_test.c` with the following content: + +```c +#include <sbi/sbi_unit.h> +#include <sbi/sbi_string.h> + +static void strlen_test(struct sbiunit_test_case *test) +{ + SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hello"), 5); + SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hell\0o"), 4); +}
docs/writing_tests.md line 51
Quoted 10 lines
docs/writing_tests.md new file +10
-0+ +static struct sbiunit_test_case string_test_cases[] = { + SBIUNIT_TEST_CASE(strlen_test), + {}, +}; + +SBIUNIT_TEST_SUITE(string_test_suite, string_test_cases); +``` + +After that, add the corresponding entry to `lib/sbi/sbi_unit.c` and update the `test_suites` array:
s/After that,/Then,/
But I think we should be able to add the test suite pointer to an elf section with the SBIUNIT_TEST_SUITE() macro to avoid this step.
Quoted 67 lines
docs/writing_tests.md new file +67
-0+```c +... +extern struct sbiunit_test_suite string_test_suite; +... +static struct sbiunit_test_suite *test_suites[] = { + ... + &string_test_suite, +}; +... +``` + +Add the corresponding Makefile entry to `lib/sbi/objects.mk`: +```lang-makefile +... +libsbi-objs-$(CONFIG_SBIUNIT) += sbi_string_test.o +``` + +Now recompile OpenSBI with CONFIG_SBIUNIT option enabled and run it in the QEMU. You will see +something like this: +``` +# make PLATFORM=generic run +... +# Running SBIUNIT tests # +... +## Running test suite: string_test_suite +[OK] strlen_test +1 SUCCESS / 0 FAIL / 1 TOTAL +``` + +Now let's try to change this test in the way that it will fail: + +```c +- SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hello"), 5); ++ SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hello"), 100); +``` + +Compile and run it again: +``` +... +# Running SBIUNIT tests # +... +## Running test suite: string_test_suite +strlen_test: Condition "sbi_strlen("Hello") == 100" expected to be true! +[FAIL] strlen_test +0 SUCCESS / 1 FAIL / 1 TOTAL +``` +Covering the static functions / using the static definitions +------------------------------------------------------------ + +SBIUnit also allows you to test static functions. In order to do so, simply include your test source +in the file you would like to test. Complementing the example above, just add this to the +`lib/sbi/sbi_string.c` file: + +```c +#ifdef CONFIG_SBIUNIT +#include "sbi_string_test.c" +#endif +``` + +In this case you should not add a new entry to `lib/sbi/objects.mk`, because the test code will be +included into the `sbi_string` object file. + +See example in `lib/sbi/sbi_console_test.c`, where statically declared `console_dev` variable is +used to mock the `sbi_console_device` structure. + +"Mocking" the structures +------------------------
docs/writing_tests.md line 128
Quoted 10 lines
docs/writing_tests.md new file +10
-0+See the example of structure "mocking" in the `lib/sbi/sbi_console_test.c`, where the +sbi_console_device structure was mocked to be used in various console-related functions in order to +test them. + +API Reference +------------- +All of the `SBIUNIT_EXPECT_*` macros will cause a test case to fail if the corresponding conditions +are not met, however, the execution of a particular test case will not be stopped. + +All of the `SBIUNIT_ASSERT_*` macros will cause a test case to fail and stop immediately.
The distinction between SBIUNIT_EXPECT* and SBIUNIT_ASSERT* is good to document, but I don't think we want to list the functions as is done below. We should just reference the header from here, since we'll be lucky to keep the header filename in sync, let alone each function...
Quoted 5 lines
docs/writing_tests.md new file +5
-0+ +- `SBIUNIT_EXPECT(test_case, condition)` - sets an expectation that 'condition' is true +- `SBIUNIT_ASSERT(test_case, condition)` - sets an assertion that 'condition' is true +- `SBIUNIT_EXPECT_EQ(test_case, a, b)` - sets an expectation that a = b +- `SBIUNIT_ASSERT_EQ(test_case, a, b)` - sets an assertion that a = b
docs/writing_tests.md line 143
Quoted 10 lines
docs/writing_tests.md new file +10
-0+- `SBIUNIT_EXPECT_NE(test_case, a, b)` - sets an expectation that a != b +- `SBIUNIT_ASSERT_NE(test_case, a, b)` - sets an assertion that a != b +- `SBIUNIT_EXPECT_MEMEQ(test_case, a, b, len)` - performs sbi_memcmp on memory regions a and b with +a length 'len', and sets an expectation that they are equal +- `SBIUNIT_ASSERT_MEMEQ(test_case, a, b, len)` - performs sbi_memcmp on memory regions a and b with +a length 'len', and sets an assertion that they are equal +- `SBIUNIT_EXPECT_STREQ(test_case, a, b)` - performs sbi_strcmp on strings a and b and sets +an expectation that they are equal +- `SBIUNIT_ASSERT_STREQ(test_case, a, b)` - performs sbi_strcmp on strings a and b and sets +an assertion that they are equal
Thanks,
drew
PATCH 1/4 docs: Add documentation about tests and SBIUnit
Quoted 4 lines · Andrew Jones
The above should be in the cover letter, which this series is missing. Cover letters are nice since they give reviewers a place to comment on the overall series.
Alright, will be moved to the cover letter in V2.
Quoted 4 lines
The above paragraph belongs in the cover letter along with elaboration on how only certain parts of KUnit were reimplemented and why those certain parts, and not others, were chosen.
Makes sense, I'll extend the description.
Quoted 81 lines
Quoted 13 lines
Signed-off-by Ivan Orlov <[email protected]>--- docs/writing_tests.md | 143 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+)create mode 100644
docs/writing_tests.mddocs/writing_tests.md new file +0
-0docs/writing_tests.md line 6
Quoted 6 lines
docs/writing_tests.md new file +6
-0@@ -0,0 +1,143 @@ +Writing tests for OpenSBI +========================= + +SBIUnit +------- +SBIUnit is a set of macros and functions which simplify the test development and automate theWe aim for 80 char line length, especially in documentation.
docs/writing_tests.md line 7
Quoted 1 line
docs/writing_tests.md new file +1
-0+test execution and evaluation. All of the SBIUnit definitions could be found ins/could be found in/are in the/
docs/writing_tests.md line 8
Quoted 1 line
docs/writing_tests.md new file +1
-0+`include/sbi/sbi_unit.h` header file, and implementations are available in `lib/sbi/sbi_unit.c`.I think the string 'test' should be somewhere in the filenames. 'unit' is too generic.
Quoted 11 lines
docs/writing_tests.md new file +11
-0+ +Simple SBIUnit test +------------------- + +For instance, we would like to test the following function from `lib/sbi/sbi_string.c`: + +```c +size_t sbi_strlen(const char *str) +{ + unsigned long ret = 0; +docs/writing_tests.md line 29
Quoted 10 lines
docs/writing_tests.md new file +10
-0+ while (*str != '\0') { + ret++; + str++; + } + + return ret; +} +``` + +Apparently, it calculates the string length.s/Apparently, it/which/
Quoted 12 lines
docs/writing_tests.md new file +12
-0+ +Create the file `lib/sbi/sbi_string_test.c` with the following content: + +```c +#include <sbi/sbi_unit.h> +#include <sbi/sbi_string.h> + +static void strlen_test(struct sbiunit_test_case *test) +{ + SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hello"), 5); + SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hell\0o"), 4); +}docs/writing_tests.md line 51
Quoted 10 lines
docs/writing_tests.md new file +10
-0+ +static struct sbiunit_test_case string_test_cases[] = { + SBIUNIT_TEST_CASE(strlen_test), + {}, +}; + +SBIUNIT_TEST_SUITE(string_test_suite, string_test_cases); +``` + +After that, add the corresponding entry to `lib/sbi/sbi_unit.c` and update the `test_suites` array:s/After that,/Then,/
Quoted 3 lines
But I think we should be able to add the test suite pointer to an elf section with the
SBIUNIT_TEST_SUITE()macro to avoid this step.
That was an initial idea, however I faced some obstacles during the implementation.
I believe we would like to cover the static functions, as well as use static variables in the tests. In this case, we would include the test in the source we are covering (for instance, include "sbi_console_test.c" in "sbi_console.c"). If we use OpenSBI (libplatsbi.a) as a library when linking firmware, and firmware refers to a symbol from "sbi_console.h", it will automatically link the test code too. This means that firmware should have the test ELF section as well. Manual registration of the tests in 'sbi_unit.c', on the other hand, would not require any effort from the firmware developers if they decide to enable tests for OpenSBI.
Moreover, manual test declaration will make sure that we included all of the tests. Otherwise, if we define the test in a separate file, it will be linked out unless we refer to symbols from it somewhere.
What do you think of it?
Quoted 74 lines
Quoted 74 lines
docs/writing_tests.md new file +74
-0+```c +... +extern struct sbiunit_test_suite string_test_suite; +... +static struct sbiunit_test_suite *test_suites[] = { + ... + &string_test_suite, +}; +... +``` + +Add the corresponding Makefile entry to `lib/sbi/objects.mk`: +```lang-makefile +... +libsbi-objs-$(CONFIG_SBIUNIT) += sbi_string_test.o +``` + +Now recompile OpenSBI with CONFIG_SBIUNIT option enabled and run it in the QEMU. You will see +something like this: +``` +# make PLATFORM=generic run +... +# Running SBIUNIT tests # +... +## Running test suite: string_test_suite +[OK] strlen_test +1 SUCCESS / 0 FAIL / 1 TOTAL +``` + +Now let's try to change this test in the way that it will fail: + +```c +- SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hello"), 5); ++ SBIUNIT_EXPECT_EQ(test, sbi_strlen("Hello"), 100); +``` + +Compile and run it again: +``` +... +# Running SBIUNIT tests # +... +## Running test suite: string_test_suite +strlen_test: Condition "sbi_strlen("Hello") == 100" expected to be true! +[FAIL] strlen_test +0 SUCCESS / 1 FAIL / 1 TOTAL +``` +Covering the static functions / using the static definitions +------------------------------------------------------------ + +SBIUnit also allows you to test static functions. In order to do so, simply include your test source +in the file you would like to test. Complementing the example above, just add this to the +`lib/sbi/sbi_string.c` file: + +```c +#ifdef CONFIG_SBIUNIT +#include "sbi_string_test.c" +#endif +``` + +In this case you should not add a new entry to `lib/sbi/objects.mk`, because the test code will be +included into the `sbi_string` object file. + +See example in `lib/sbi/sbi_console_test.c`, where statically declared `console_dev` variable is +used to mock the `sbi_console_device` structure. + +"Mocking" the structures +------------------------ +See the example of structure "mocking" in the `lib/sbi/sbi_console_test.c`, where the +sbi_console_device structure was mocked to be used in various console-related functions in order to +test them. + +API Reference +------------- +All of the `SBIUNIT_EXPECT_*` macros will cause a test case to fail if the corresponding conditions
Quoted 9 lines
docs/writing_tests.md line 128
Quoted 3 lines
docs/writing_tests.md new file +3
-0+are not met, however, the execution of a particular test case will not be stopped. + +All of the `SBIUNIT_ASSERT_*` macros will cause a test case to fail and stop immediately.The distinction between SBIUNIT_EXPECT* and SBIUNIT_ASSERT* is good to document, but I don't think we want to list the functions as is done below. We should just reference the header from here, since we'll be lucky to keep the header filename in sync, let alone each function...
I agree, it is definitely redundant.
Thank you so much for the review!
Signature · 2 lines
Kind regards, Ivan Orlov
PATCH 1/4 docs: Add documentation about tests and SBIUnit
Quoted 1 line · Ivan Orlov
On 2/12/24 17:20, Andrew Jones wrote:
...
Quoted 7 lines
Quoted 3 lines
But I think we should be able to add the test suite pointer to an elf section with the
SBIUNIT_TEST_SUITE()macro to avoid this step.That was an initial idea, however I faced some obstacles during the implementation.
Quoted 9 lines
I believe we would like to cover the static functions, as well as use static variables in the tests. In this case, we would include the test in the source we are covering (for instance, include "
sbi_console_test.c" in "sbi_console.c"). If we use OpenSBI (libplatsbi.a) as a library when linking firmware, and firmware refers to a symbol from "sbi_console.h", it will automatically link the test code too. This means that firmware should have the test ELF section as well. Manual registration of the tests in 'sbi_unit.c', on the other hand, would not require any effort from the firmware developers if they decide to enable tests for OpenSBI.
Hmm, I see. So maybe we can use the build system's carray?
Quoted 4 lines
Moreover, manual test declaration will make sure that we included all of the tests. Otherwise, if we define the test in a separate file, it will be linked out unless we refer to symbols from it somewhere.
This could possibly be worked around with the linker's --whole-archive option.
Thanks,
drew
PATCH 1/4 docs: Add documentation about tests and SBIUnit
Quoted 20 lines · Andrew Jones
Quoted 1 line · Ivan Orlov
On 2/12/24 17:20, Andrew Jones wrote:
...
Quoted 7 lines
Quoted 3 lines
But I think we should be able to add the test suite pointer to an elf section with the
SBIUNIT_TEST_SUITE()macro to avoid this step.That was an initial idea, however I faced some obstacles during the implementation.
Quoted 9 lines
I believe we would like to cover the static functions, as well as use static variables in the tests. In this case, we would include the test in the source we are covering (for instance, include "
sbi_console_test.c" in "sbi_console.c"). If we use OpenSBI (libplatsbi.a) as a library when linking firmware, and firmware refers to a symbol from "sbi_console.h", it will automatically link the test code too. This means that firmware should have the test ELF section as well. Manual registration of the tests in 'sbi_unit.c', on the other hand, would not require any effort from the firmware developers if they decide to enable tests for OpenSBI.
Quoted 2 lines
Hmm, I see. So maybe we can use the build system's carray?
Hmmm, I've never heard about this thing before, could you please point me to a reference or source where I could read about it?
Quoted 7 lines
Quoted 4 lines
Moreover, manual test declaration will make sure that we included all of the tests. Otherwise, if we define the test in a separate file, it will be linked out unless we refer to symbols from it somewhere.
This could possibly be worked around with the linker's --whole-archive option.
In this case we would have to update the makefile logic, because now OpenSBI uses 'compile_elf' makefile procedure for the test payload and the firmware elf files both, and we would not like to use --whole-archive for the test payload. I guess it is an overcomplication...
Signature · 2 lines
Kind regards, Ivan Orlov
PATCH 1/4 docs: Add documentation about tests and SBIUnit
Quoted 21 lines · Ivan Orlov
Quoted 20 lines · Andrew Jones
Quoted 1 line · Ivan Orlov
On 2/12/24 17:20, Andrew Jones wrote:
...
Quoted 16 lines
Quoted 3 lines
But I think we should be able to add the test suite pointer to an elf section with the
SBIUNIT_TEST_SUITE()macro to avoid this step.That was an initial idea, however I faced some obstacles during the implementation.
I believe we would like to cover the static functions, as well as use static variables in the tests. In this case, we would include the test in the source we are covering (for instance, include "
sbi_console_test.c" in "sbi_console.c"). If we use OpenSBI (libplatsbi.a) as a library when linking firmware, and firmware refers to a symbol from "sbi_console.h", it will automatically link the test code too. This means that firmware should have the test ELF section as well. Manual registration of the tests in 'sbi_unit.c', on the other hand, would not require any effort from the firmware developers if they decide to enable tests for OpenSBI.
Quoted 5 lines
Quoted 2 lines
Hmm, I see. So maybe we can use the build system's carray?
Hmmm, I've never heard about this thing before, could you please point me to a reference or source where I could read about it?
It's used in OpenSBI code for arrays such as sbi_ecall_exts. See commit 56bed1a0fe39 ("lib: sbi_ecall: Generate extensions list with carray") for that example.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Moreover, manual test declaration will make sure that we included all of the tests. Otherwise, if we define the test in a separate file, it will be linked out unless we refer to symbols from it somewhere.
Quoted 7 lines
Quoted 2 lines
This could possibly be worked around with the linker's --whole-archive option.
In this case we would have to update the makefile logic, because now OpenSBI uses '
compile_elf' makefile procedure for the test payload and the firmware elf files both, and we would not like to use --whole-archive for the test payload. I guess it is an overcomplication...
Maybe, but I'm not sure how much we'd care about having bloated binaries when building with CONFIG_SBIUNIT.
Thanks,
drew
PATCH 1/4 docs: Add documentation about tests and SBIUnit
Quoted 7 lines · Andrew Jones
Quoted 2 lines
Hmmm, I've never heard about this thing before, could you please point me to a reference or source where I could read about it?
It's used in OpenSBI code for arrays such as
sbi_ecall_exts. See commit56bed1a0fe39("lib:sbi_ecall: Generate extensions list with carray") for that example.
Thanks, I'll take a look and use them in the V2 if it is possible!
Quoted 9 lines
Quoted 5 lines
In this case we would have to update the makefile logic, because now OpenSBI uses '
compile_elf' makefile procedure for the test payload and the firmware elf files both, and we would not like to use --whole-archive for the test payload. I guess it is an overcomplication...Maybe, but I'm not sure how much we'd care about having bloated binaries when building with
CONFIG_SBIUNIT.
The problem is not with the bloated payload binaries: if we use --whole-archive when linking the test payload, it starts giving the 'undefined reference' errors, because some of the object files refer to symbols which don't exist in the test payload. Like:
``` opensbi/platform/generic/andes/sleep.S:63: undefined reference to `_start_warm' ```
_start_warm is defined in fw_base, but it's not accessible when building the test payload (firmware/payloads/).
Probably, we could just disable the test payload building when CONFIG_SBIUNIT is enabled, but I'm not sure if it is a correct approach... Another option would be optional --whole-archive use, so we would use it for linking the fw_* ELF files only.
Signature · 2 lines
Kind regards, Ivan Orlov