Skip to contentSkip to conversation

test: fix naming of test functions in the log test suite

PATCH New Heinrich Schuchardt 4 participants 7 messages May 6, 2020 – May 8, 2020

lore
Share & tools
lore
#0 lore raw

cover test: fix naming of test functions in the log test suite

Some Python subtests executed via the 'ut' console command were not executed due to not following a required naming convention.

Add a description of the naming convention.
Fix the tests.

Heinrich Schuchardt (2):
test: describe naming conventions for macro UNIT_TEST
test: fix naming of test functions in the log test suite

 include/test/test.h      | 24 +++++++++++++++++++-
 test/log/nolog_test.c    | 24 ++++++++++----------
 test/log/syslog_test.c   | 48 ++++++++++++++++++++--------------------
 test/py/tests/test_ut.py | 17 +++++++++++++-
 4 files changed, 75 insertions(+), 38 deletions(-)

--
2.26.2

#1 lore raw

PATCH 1/2 test: describe naming conventions for macro UNIT_TEST

Strict naming conventions have to be followed for Python function generate_ut_subtest() to collect C unit tests to be executed via command 'ut'.

Describe the requirements both on the C as well on the Python side.

---
 include/test/test.h      | 24 +++++++++++++++++++++++-
 test/py/tests/test_ut.py | 17 ++++++++++++++++-
 2 files changed, 39 insertions(+), 2 deletions(-)
include/test/test.h +23 -1
index 2a75211008..029288de88 100644
@@ -41,7 +41,29 @@ struct unit_test {
 	int flags;
 };
 
-/* Declare a new unit test */
+/**
+ * UNIT_TEST() - create linker generated list entry for unit a unit test
+ *
+ * The macro UNIT_TEST() is used to create a linker generated list entry. These
+ * list entries are enumerate tests that can be execute using the ut command.
+ * The list entries are used both by the implementation of the ut command as
+ * well as in a related Python test.
+ *
+ * For Python testing the subtests are collected in Python function
+ * generate_ut_subtest() by applying a regular expression to the lines of file
+ * u-boot.sym. The list entries have to follow strict naming conventions to be
+ * matched by the expression.
+ *
+ * Use UNIT_TEST(foo_test_bar, _flags, foo_test) for a test bar in test suite
+ * foo that can be executed via command 'ut foo bar' and is implemented in
+ * function foo_test_bar().
+ *
+ * @_name:	concatenation of name of the test suite, "_test_", and the name
+ *		of the test
+ * @_flags:	an integer field that can be evaluated by the test suite
+ *		implementation
+ * @_suite:	name of the test suite concatenated with "_test"
+ */
 #define UNIT_TEST(_name, _flags, _suite)				\
 	ll_entry_declare(struct unit_test, _name, _suite) = {		\
 		.file = __FILE__,					\
test/py/tests/test_ut.py +16 -1
index 6c7b8dd2b3..01c2b3ffa1 100644
@@ -22,7 +22,22 @@ def test_ut_dm_init(u_boot_console):
             fh.write(data)
 
 def test_ut(u_boot_console, ut_subtest):
-    """Execute a "ut" subtest."""
+    """Execute a "ut" subtest.
+
+    The subtests are collected in function generate_ut_subtest() from linker
+    generated lists by applying a regular expression to the lines of file
+    u-boot.sym. The list entries are created using the C macro UNIT_TEST().
+
+    Strict naming conventions have to be followed to match the regular
+    expression. Use UNIT_TEST(foo_test_bar, _flags, foo_test) for a test bar in
+    test suite foo that can be executed via command 'ut foo bar' and is
+    implemented in C function foo_test_bar().
+
+    Args:
+        u_boot_console (ConsoleBase): U-Boot console
+        ut_subtest (str): test to be executed via command ut, e.g 'foo bar' to
+            execute command 'ut foo bar'
+    """
 
     output = u_boot_console.run_command('ut ' + ut_subtest)
     assert output.endswith('Failures: 0')

--
2.26.2

#2 lore raw

PATCH 1/2 test: describe naming conventions for macro UNIT_TEST

Quoted 21 lines · Heinrich Schuchardt

Strict naming conventions have to be followed for Python function generate_ut_subtest() to collect C unit tests to be executed via command 'ut'.

Describe the requirements both on the C as well on the Python side.

+/**
+ * UNIT_TEST() - create linker generated list entry for unit a unit test
+ *
+ * The macro UNIT_TEST() is used to create a linker generated list entry. These
+ * list entries are enumerate tests that can be execute using the ut command.
+ * The list entries are used both by the implementation of the ut command as
+ * well as in a related Python test.
+ *
+ * For Python testing the subtests are collected in Python function
+ * generate_ut_subtest() by applying a regular expression to the lines of file
+ * u-boot.sym. The list entries have to follow strict naming conventions to be
+ * matched by the expression.
+ *
+ * Use UNIT_TEST(foo_test_bar, _flags, foo_test) for a test bar in test suite
+ * foo that can be executed via command 'ut foo bar' and is implemented in

Quoted 8 lines

+ * function foo_test_bar().
+ *
+ * @_name: concatenation of name of the test suite, "_test_", and the name
+ * of the test
+ * @_flags: an integer field that can be evaluated by the test suite
+ * implementation
+ * @_suite: name of the test suite concatenated with "_test"
+ */

Perhaps the macro could simply take "foo" and "bar" as parameters, and generate the function name foo_test_bar internally rather than having the user pass it in. That way, compilation will actively fail if the function isn't named correctly, since it won't match the reference created by this macro.

To help make this easier, we could add another macro e.g. UNIT_TEST_FUNC() that evaluates to just the expected function name, so that people wouldn't have to know the naming convention when they implement the function; they'd just write e.g.:

static int UNIT_TEST_FUNC(log, nolog_err)(struct unit_test_state *uts)

But certainly this series is a good first step, and fine even if we don't implement this suggestion.

The series,

#3 lore raw

PATCH 1/2 test: describe naming conventions for macro UNIT_TEST

Quoted 38 lines · Stephen Warren
Quoted 23 lines · Heinrich Schuchardt

Strict naming conventions have to be followed for Python function generate_ut_subtest() to collect C unit tests to be executed via command 'ut'.

Describe the requirements both on the C as well on the Python side.

+/**
+ * UNIT_TEST() - create linker generated list entry for unit a unit test
+ *
+ * The macro UNIT_TEST() is used to create a linker generated list entry. These
+ * list entries are enumerate tests that can be execute using the ut command.
+ * The list entries are used both by the implementation of the ut command as
+ * well as in a related Python test.
+ *
+ * For Python testing the subtests are collected in Python function
+ * generate_ut_subtest() by applying a regular expression to the lines of file
+ * u-boot.sym. The list entries have to follow strict naming conventions to be
+ * matched by the expression.
+ *
+ * Use UNIT_TEST(foo_test_bar, _flags, foo_test) for a test bar in test suite
+ * foo that can be executed via command 'ut foo bar' and is implemented in
+ * function foo_test_bar().
+ *

Quoted 6 lines
+ * @_name:   concatenation of name of the test suite, "_test_", and the name
+ *           of the test
+ * @_flags:  an integer field that can be evaluated by the test suite
+ *           implementation
+ * @_suite:  name of the test suite concatenated with "_test"

+ */

Perhaps the macro could simply take "foo" and "bar" as parameters, and generate the function name foo_test_bar internally rather than having the user pass it in. That way, compilation will actively fail if the function isn't named correctly, since it won't match the reference created by this macro.

Quoted 6 lines

To help make this easier, we could add another macro e.g. UNIT_TEST_FUNC() that evaluates to just the expected function name, so that people wouldn't have to know the naming convention when they implement the function; they'd just write e.g.:

static int UNIT_TEST_FUNC(log, nolog_err)(struct unit_test_state *uts)

I am not a huge fan of that. It looks weird to have the function name auto-generated, and it defeats ctags, code search, etc.. Another option might be to check for exported test/ functions that don't match and print a warning?

Quoted 3 lines

But certainly this series is a good first step, and fine even if we don't implement this suggestion.

Yes definitely.

Quoted 3 lines

The series,
Reviewed-by: Stephen Warren <swarren at nvidia.com

#4 lore raw

PATCH 1/2 test: describe naming conventions for macro UNIT_TEST

In reply to Heinrich Schuchardt
Quoted 9 lines · Heinrich Schuchardt

Strict naming conventions have to be followed for Python function generate_ut_subtest() to collect C unit tests to be executed via command 'ut'.

Describe the requirements both on the C as well on the Python side.

Applied to u-boot/master, thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200508/783ffa64/attachment.sig>
#5 lore raw

PATCH 2/2 test: fix naming of test functions in the log test suite

In reply to Heinrich Schuchardt

Both the nolog as well as the syslog tests were not found by Python function generate_ut_subtest() due to not following the nameing requirements imposed by the regular expression used to find linker generated list entries in file u-boot.sym.

Adjust the naming of test functions.

With the patch the following tests are executed successfully for sandbox_defconfig:

test/py/tests/test_ut.py::test_ut[ut_log_syslog_debug] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_err] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_info] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_nodebug] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_notice] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_warning] PASSED

The nolog tests are only executed if CONFIG_LOG=n and CONFIG_CONSOLE_RECORD=y.

---
 test/log/nolog_test.c  | 24 ++++++++++-----------
 test/log/syslog_test.c | 48 +++++++++++++++++++++---------------------
 2 files changed, 36 insertions(+), 36 deletions(-)
test/log/nolog_test.c +12 -12
index 84619521c9..c418ed07c9 100644
@@ -19,7 +19,7 @@ DECLARE_GLOBAL_DATA_PTR;
 
 #define BUFFSIZE 32
 
-static int nolog_test_log_err(struct unit_test_state *uts)
+static int log_test_nolog_err(struct unit_test_state *uts)
 {
 	char buf[BUFFSIZE];
 
@@ -31,9 +31,9 @@ static int nolog_test_log_err(struct unit_test_state *uts)
 	ut_assertok(ut_check_console_end(uts));
 	return 0;
 }
-LOG_TEST(nolog_test_log_err);
+LOG_TEST(log_test_nolog_err);
 
-static int nolog_test_log_warning(struct unit_test_state *uts)
+static int log_test_nolog_warning(struct unit_test_state *uts)
 {
 	char buf[BUFFSIZE];
 
@@ -45,9 +45,9 @@ static int nolog_test_log_warning(struct unit_test_state *uts)
 	ut_assertok(ut_check_console_end(uts));
 	return 0;
 }
-LOG_TEST(nolog_test_log_warning);
+LOG_TEST(log_test_nolog_warning);
 
-static int nolog_test_log_notice(struct unit_test_state *uts)
+static int log_test_nolog_notice(struct unit_test_state *uts)
 {
 	char buf[BUFFSIZE];
 
@@ -59,9 +59,9 @@ static int nolog_test_log_notice(struct unit_test_state *uts)
 	ut_assertok(ut_check_console_end(uts));
 	return 0;
 }
-LOG_TEST(nolog_test_log_notice);
+LOG_TEST(log_test_nolog_notice);
 
-static int nolog_test_log_info(struct unit_test_state *uts)
+static int log_test_nolog_info(struct unit_test_state *uts)
 {
 	char buf[BUFFSIZE];
 
@@ -73,7 +73,7 @@ static int nolog_test_log_info(struct unit_test_state *uts)
 	ut_assertok(ut_check_console_end(uts));
 	return 0;
 }
-LOG_TEST(nolog_test_log_info);
+LOG_TEST(log_test_nolog_info);
 
 #undef _DEBUG
 #define _DEBUG 0
@@ -90,7 +90,7 @@ static int nolog_test_nodebug(struct unit_test_state *uts)
 }
 LOG_TEST(nolog_test_nodebug);
 
-static int nolog_test_log_nodebug(struct unit_test_state *uts)
+static int log_test_nolog_nodebug(struct unit_test_state *uts)
 {
 	char buf[BUFFSIZE];
 
@@ -102,7 +102,7 @@ static int nolog_test_log_nodebug(struct unit_test_state *uts)
 	ut_assertok(ut_check_console_end(uts));
 	return 0;
 }
-LOG_TEST(nolog_test_log_nodebug);
+LOG_TEST(log_test_nolog_nodebug);
 
 #undef _DEBUG
 #define _DEBUG 1
@@ -120,7 +120,7 @@ static int nolog_test_debug(struct unit_test_state *uts)
 }
 LOG_TEST(nolog_test_debug);
 
-static int nolog_test_log_debug(struct unit_test_state *uts)
+static int log_test_nolog_debug(struct unit_test_state *uts)
 {
 	char buf[BUFFSIZE];
 
@@ -132,4 +132,4 @@ static int nolog_test_log_debug(struct unit_test_state *uts)
 	ut_assertok(ut_check_console_end(uts));
 	return 0;
 }
-LOG_TEST(nolog_test_log_debug);
+LOG_TEST(log_test_nolog_debug);
test/log/syslog_test.c +24 -24

159 lines not on this page · raw message

--
2.26.2

#6 lore raw

PATCH 2/2 test: fix naming of test functions in the log test suite

Quoted 17 lines · Heinrich Schuchardt

Both the nolog as well as the syslog tests were not found by Python function generate_ut_subtest() due to not following the nameing requirements imposed by the regular expression used to find linker generated list entries in file u-boot.sym.

Adjust the naming of test functions.

With the patch the following tests are executed successfully for sandbox_defconfig:

test/py/tests/test_ut.py::test_ut[ut_log_syslog_debug] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_err] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_info] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_nodebug] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_notice] PASSED test/py/tests/test_ut.py::test_ut[ut_log_syslog_warning] PASSED

Quoted 5 lines

The nolog tests are only executed if CONFIG_LOG=n and CONFIG_CONSOLE_RECORD=y.

Applied to u-boot/master, thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200508/b8a14b92/attachment.sig>