test: fix naming of test functions in the log test suite
cover test: fix naming of test functions in the log test suite
Some Python subtests executed via the 'ut' console command were not executed due to not following a required naming convention.
Add a description of the naming convention.
Fix the tests.
Heinrich Schuchardt (2):
test: describe naming conventions for macro UNIT_TEST
test: fix naming of test functions in the log test suite
include/test/test.h | 24 +++++++++++++++++++- test/log/nolog_test.c | 24 ++++++++++---------- test/log/syslog_test.c | 48 ++++++++++++++++++++-------------------- test/py/tests/test_ut.py | 17 +++++++++++++- 4 files changed, 75 insertions(+), 38 deletions(-)
--
2.26.2
PATCH 1/2 test: describe naming conventions for macro UNIT_TEST
Strict naming conventions have to be followed for Python function generate_ut_subtest() to collect C unit tests to be executed via command 'ut'.
Describe the requirements both on the C as well on the Python side.
--- include/test/test.h | 24 +++++++++++++++++++++++- test/py/tests/test_ut.py | 17 ++++++++++++++++- 2 files changed, 39 insertions(+), 2 deletions(-)
include/test/test.h +23 -1
@@ -41,7 +41,29 @@ struct unit_test {int flags; }; -/* Declare a new unit test */ +/** +* UNIT_TEST() - create linker generated list entry for unit a unit test +* +* The macro UNIT_TEST() is used to create a linker generated list entry. These +* list entries are enumerate tests that can be execute using the ut command. +* The list entries are used both by the implementation of the ut command as +* well as in a related Python test. +* +* For Python testing the subtests are collected in Python function +* generate_ut_subtest() by applying a regular expression to the lines of file +* u-boot.sym. The list entries have to follow strict naming conventions to be +* matched by the expression. +* +* Use UNIT_TEST(foo_test_bar, _flags, foo_test) for a test bar in test suite +* foo that can be executed via command 'ut foo bar' and is implemented in +* function foo_test_bar(). +* +* @_name: concatenation of name of the test suite, "_test_", and the name +* of the test +* @_flags: an integer field that can be evaluated by the test suite +* implementation +* @_suite: name of the test suite concatenated with "_test" +*/ #define UNIT_TEST (_name, _flags, _suite) \ ll_entry_declare(struct unit_test, _name, _suite) = { \ .file = __FILE__, \
test/py/tests/test_ut.py +16 -1
@@ -22,7 +22,22 @@ def test_ut_dm_init(u_boot_console): fh.write(data) def test_ut(u_boot_console, ut_subtest): - """Execute a "ut" subtest.""" + """Execute a "ut" subtest. + + The subtests are collected in function generate_ut_subtest() from linker + generated lists by applying a regular expression to the lines of file + u-boot.sym. The list entries are created using the C macro UNIT_TEST(). + + Strict naming conventions have to be followed to match the regular + expression. Use UNIT_TEST(foo_test_bar, _flags, foo_test) for a test bar in + test suite foo that can be executed via command 'ut foo bar' and is + implemented in C function foo_test_bar(). + + Args: + u_boot_console (ConsoleBase): U-Boot console + ut_subtest (str): test to be executed via command ut, e.g 'foo bar' to + execute command 'ut foo bar' + """ output = u_boot_console.run_command('ut ' + ut_subtest) assert output.endswith('Failures: 0')
--
2.26.2
PATCH 1/2 test: describe naming conventions for macro UNIT_TEST
Quoted 21 lines · Heinrich Schuchardt
Strict naming conventions have to be followed for Python function
generate_ut_subtest()to collect C unit tests to be executed via command 'ut'.Describe the requirements both on the C as well on the Python side.
+/**
+ *UNIT_TEST()- create linker generated list entry for unit a unit test
+ *
+ * The macroUNIT_TEST()is used to create a linker generated list entry. These
+ * list entries are enumerate tests that can be execute using the ut command.
+ * The list entries are used both by the implementation of the ut command as
+ * well as in a related Python test.
+ *
+ * For Python testing the subtests are collected in Python function
+ *generate_ut_subtest()by applying a regular expression to the lines of file
+ * u-boot.sym. The list entries have to follow strict naming conventions to be
+ * matched by the expression.
+ *
+ * UseUNIT_TEST(foo_test_bar, _flags, foo_test)for a test bar in test suite
+ * foo that can be executed via command 'ut foo bar' and is implemented in
Quoted 8 lines
+ * function
foo_test_bar().
+ *
+ * @_name: concatenation of name of the test suite, "_test_", and the name
+ * of the test
+ * @_flags: an integer field that can be evaluated by the test suite
+ * implementation
+ * @_suite: name of the test suite concatenated with "_test"
+ */
Perhaps the macro could simply take "foo" and "bar" as parameters, and generate the function name foo_test_bar internally rather than having the user pass it in. That way, compilation will actively fail if the function isn't named correctly, since it won't match the reference created by this macro.
To help make this easier, we could add another macro e.g. UNIT_TEST_FUNC() that evaluates to just the expected function name, so that people wouldn't have to know the naming convention when they implement the function; they'd just write e.g.:
static int UNIT_TEST_FUNC(log, nolog_err)(struct unit_test_state *uts)
But certainly this series is a good first step, and fine even if we don't implement this suggestion.
The series,
PATCH 1/2 test: describe naming conventions for macro UNIT_TEST
Quoted 38 lines · Stephen Warren
Quoted 23 lines · Heinrich Schuchardt
Strict naming conventions have to be followed for Python function
generate_ut_subtest()to collect C unit tests to be executed via command 'ut'.Describe the requirements both on the C as well on the Python side.
+/**
+ *UNIT_TEST()- create linker generated list entry for unit a unit test
+ *
+ * The macroUNIT_TEST()is used to create a linker generated list entry. These
+ * list entries are enumerate tests that can be execute using the ut command.
+ * The list entries are used both by the implementation of the ut command as
+ * well as in a related Python test.
+ *
+ * For Python testing the subtests are collected in Python function
+ *generate_ut_subtest()by applying a regular expression to the lines of file
+ * u-boot.sym. The list entries have to follow strict naming conventions to be
+ * matched by the expression.
+ *
+ * UseUNIT_TEST(foo_test_bar, _flags, foo_test)for a test bar in test suite
+ * foo that can be executed via command 'ut foo bar' and is implemented in
+ * functionfoo_test_bar().
+ *Quoted 6 lines
+ * @_name: concatenation of name of the test suite, "_test_", and the name + * of the test + * @_flags: an integer field that can be evaluated by the test suite + * implementation + * @_suite: name of the test suite concatenated with "_test"+ */
Perhaps the macro could simply take "foo" and "bar" as parameters, and generate the function name
foo_test_barinternally rather than having the user pass it in. That way, compilation will actively fail if the function isn't named correctly, since it won't match the reference created by this macro.
Quoted 6 lines
To help make this easier, we could add another macro e.g.
UNIT_TEST_FUNC()that evaluates to just the expected function name, so that people wouldn't have to know the naming convention when they implement the function; they'd just write e.g.:static int
UNIT_TEST_FUNC(log, nolog_err)(structunit_test_state*uts)
I am not a huge fan of that. It looks weird to have the function name auto-generated, and it defeats ctags, code search, etc.. Another option might be to check for exported test/ functions that don't match and print a warning?
Quoted 3 lines
But certainly this series is a good first step, and fine even if we don't implement this suggestion.
Yes definitely.
Quoted 3 lines
The series,
Reviewed-by: Stephen Warren <swarren at nvidia.com
PATCH 1/2 test: describe naming conventions for macro UNIT_TEST
In reply to Heinrich SchuchardtQuoted 9 lines · Heinrich Schuchardt
Strict naming conventions have to be followed for Python function
generate_ut_subtest()to collect C unit tests to be executed via command 'ut'.Describe the requirements both on the C as well on the Python side.
Signed-off-by Heinrich Schuchardt <[email protected]>Reviewed-by Stephen Warren <[email protected]Reviewed-by Simon Glass <[email protected]>
Applied to u-boot/master, thanks!
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: not available URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200508/783ffa64/attachment.sig>
PATCH 2/2 test: fix naming of test functions in the log test suite
In reply to Heinrich SchuchardtBoth the nolog as well as the syslog tests were not found by Python function generate_ut_subtest() due to not following the nameing requirements imposed by the regular expression used to find linker generated list entries in file u-boot.sym.
Adjust the naming of test functions.
With the patch the following tests are executed successfully for sandbox_defconfig:
test/py/tests/test_ut.py::test_ut[ut_log_syslog_debug] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_err] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_info] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_nodebug] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_notice] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_warning] PASSED
The nolog tests are only executed if CONFIG_LOG=n and CONFIG_CONSOLE_RECORD=y.
--- test/log/nolog_test.c | 24 ++++++++++----------- test/log/syslog_test.c | 48 +++++++++++++++++++++--------------------- 2 files changed, 36 insertions(+), 36 deletions(-)
test/log/nolog_test.c +12 -12
@@ -19,7 +19,7 @@ DECLARE_GLOBAL_DATA_PTR;#define BUFFSIZE 32 -static int nolog_test_log_err (struct unit_test_state *uts) +static int log_test_nolog_err (struct unit_test_state *uts) {char buf[BUFFSIZE ];
@@ -31,9 +31,9 @@ static int nolog_test_log_err(struct unit_test_state *uts)ut_assertok (ut_check_console_end (uts));return 0 ; } -LOG_TEST (nolog_test_log_err); +LOG_TEST (log_test_nolog_err); -static int nolog_test_log_warning (struct unit_test_state *uts) +static int log_test_nolog_warning (struct unit_test_state *uts) {char buf[BUFFSIZE ];
@@ -45,9 +45,9 @@ static int nolog_test_log_warning(struct unit_test_state *uts)ut_assertok (ut_check_console_end (uts));return 0 ; } -LOG_TEST (nolog_test_log_warning); +LOG_TEST (log_test_nolog_warning); -static int nolog_test_log_notice (struct unit_test_state *uts) +static int log_test_nolog_notice (struct unit_test_state *uts) {char buf[BUFFSIZE ];
@@ -59,9 +59,9 @@ static int nolog_test_log_notice(struct unit_test_state *uts)ut_assertok (ut_check_console_end (uts));return 0 ; } -LOG_TEST (nolog_test_log_notice); +LOG_TEST (log_test_nolog_notice); -static int nolog_test_log_info (struct unit_test_state *uts) +static int log_test_nolog_info (struct unit_test_state *uts) {char buf[BUFFSIZE ];
@@ -73,7 +73,7 @@ static int nolog_test_log_info(struct unit_test_state *uts)ut_assertok (ut_check_console_end (uts));return 0 ; } -LOG_TEST (nolog_test_log_info); +LOG_TEST (log_test_nolog_info);#undef _DEBUG#define _DEBUG 0
@@ -90,7 +90,7 @@ static int nolog_test_nodebug(struct unit_test_state *uts) }LOG_TEST (nolog_test_nodebug); -static int nolog_test_log_nodebug (struct unit_test_state *uts) +static int log_test_nolog_nodebug (struct unit_test_state *uts) {char buf[BUFFSIZE ];
@@ -102,7 +102,7 @@ static int nolog_test_log_nodebug(struct unit_test_state *uts)ut_assertok (ut_check_console_end (uts));return 0 ; } -LOG_TEST (nolog_test_log_nodebug); +LOG_TEST (log_test_nolog_nodebug);#undef _DEBUG#define _DEBUG 1
@@ -120,7 +120,7 @@ static int nolog_test_debug(struct unit_test_state *uts) }LOG_TEST (nolog_test_debug); -static int nolog_test_log_debug (struct unit_test_state *uts) +static int log_test_nolog_debug (struct unit_test_state *uts) {char buf[BUFFSIZE ];
@@ -132,4 +132,4 @@ static int nolog_test_log_debug(struct unit_test_state *uts)ut_assertok (ut_check_console_end (uts));return 0 ; } -LOG_TEST (nolog_test_log_debug); +LOG_TEST (log_test_nolog_debug);
test/log/syslog_test.c +24 -24
159 lines not on this page · raw message
--
2.26.2
PATCH 2/2 test: fix naming of test functions in the log test suite
Quoted 17 lines · Heinrich Schuchardt
Both the nolog as well as the syslog tests were not found by Python function
generate_ut_subtest()due to not following the nameing requirements imposed by the regular expression used to find linker generated list entries in file u-boot.sym.Adjust the naming of test functions.
With the patch the following tests are executed successfully for
sandbox_defconfig:test/py/tests/test_ut.py::test_ut[ut_log_syslog_debug] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_err] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_info] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_nodebug] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_notice] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_warning] PASSED
Quoted 5 lines
The nolog tests are only executed if
CONFIG_LOG=nandCONFIG_CONSOLE_RECORD=y.Reported-by Simon Glass <[email protected]>Signed-off-by Heinrich Schuchardt <[email protected]>
Applied to u-boot/master, thanks!
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: not available URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200508/b8a14b92/attachment.sig>