Skip to contentSkip to conversation

Add C tests for hush

RFC PATCH New Francis Laniel 1 participant 7 messages Aug 20, 2021

lore
Share & tools
lore
#0 lore raw

cover Add C tests for hush

Hi everyone.

First, I hope you are fine and the same for your relatives.

This series introduces C tests for hush.
The main goal of these tests is to act as regression tests in case of hush being modified.
During their writings some bugs were found, unfortunately I did not write patches to fix them.
First, I think these bugs were present in Busybox when hush was taken from. Also, as the whole U-Boot board scripts seem to work with these bugs maybe correcting them can lead to breaking things somewhere...
This is why I marked this series as RFC as I would like the hear the opinion of the community on this problem.

Regarding the series, first commit introduces a new subcommand to ut: ut hush. Then, others add tests for the following:
* if test condition: All these tests were taken from test_hush_if_test.py which was then removed.
* variable expansion: These tests ensures variable expansion is correct. While writing them a bug was found in hush where the following:
foo='bar "quux'
echo $foo
prints:
bar quux
instead of:
bar "quux
* commands lists: These tests test several command separated by ';', "&&" and "||". Here also, some bugs were found, indeed the following:
false && false || true
returns 0 in actual (2021) hush's Busybox, while it returns 1 in U-Boot.
* loops: This test is quite simple as we do not have math support in U-Boot, this can be tricky to heavily test while and until loops.

To conclude, here is this series diffstat:
Francis Laniel (6):
test: Add framework to test hush behavior.
test: hush: Test hush if/else.
test/py: hush_if_test: Remove the test file.
test: hush: Test hush variable expansion.
test: hush: Test hush commands list.
test: hush: Test hush loops.

 include/test/hush.h                |  15 ++
 include/test/suites.h              |   1 +
 test/Makefile                      |   3 +
 test/cmd_ut.c                      |   6 +
 test/hush/Makefile                 |  10 +
 test/hush/cmd_ut_hush.c            |  20 ++
 test/hush/dollar.c                 | 188 ++++++++++++++++++
 test/hush/if.c                     | 308 +++++++++++++++++++++++++++++
 test/hush/list.c                   |  79 ++++++++
 test/hush/loop.c                   |  64 ++++++
 test/py/tests/test_hush_if_test.py | 192 ------------------
 11 files changed, 694 insertions(+), 192 deletions(-)
 create mode 100644 include/test/hush.h
 create mode 100644 test/hush/Makefile
 create mode 100644 test/hush/cmd_ut_hush.c
 create mode 100644 test/hush/dollar.c
 create mode 100644 test/hush/if.c
 create mode 100644 test/hush/list.c
 create mode 100644 test/hush/loop.c
 delete mode 100644 test/py/tests/test_hush_if_test.py

Best regards.
--
2.25.1

#1 lore raw

RFC PATCH 1/6 test: Add framework to test hush behavior.

This commit introduces a new subcommand to ut: ut hush.
For the moment, this command does nothing, future commits will add tests which will be run on command call.

Note that CONFIG_HUSH_PARSER must be defined to compile this new subcommand.

---
 include/test/hush.h     | 15 +++++++++++++++
 include/test/suites.h   |  1 +
 test/Makefile           |  3 +++
 test/cmd_ut.c           |  6 ++++++
 test/hush/Makefile      |  6 ++++++
 test/hush/cmd_ut_hush.c | 20 ++++++++++++++++++++
 6 files changed, 51 insertions(+)
 create mode 100644 include/test/hush.h
 create mode 100644 test/hush/Makefile
 create mode 100644 test/hush/cmd_ut_hush.c
include/test/hush.h new file +15 -0
new file mode 100644
index 0000000000..cca66544a0
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * (C) Copyright 2021
+ * Francis Laniel, Amarula Solutions, [email protected]
+ */
+
+#ifndef __TEST_HUSH_H__
+#define __TEST_HUSH_H__
+
+#include <test/test.h>
+
+/* Declare a new environment test */
+#define HUSH_TEST(_name, _flags)	UNIT_TEST(_name, _flags, hush_test)
+
+#endif /* __TEST_HUSH_H__ */
include/test/suites.h +1 -0
index d35cd83a4e..35a8429e18 100644
@@ -36,6 +36,7 @@ int do_ut_compression(struct cmd_tbl *cmdtp, int flag, int argc,
 		      char *const argv[]);
 int do_ut_dm(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]);
 int do_ut_env(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]);
+int do_ut_hush(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]);
 int do_ut_lib(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]);
 int do_ut_log(struct cmd_tbl *cmdtp, int flag, int argc, char * const argv[]);
 int do_ut_mem(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]);
test/Makefile +3 -0
index b3b2902e2e..c471125982 100644
@@ -16,6 +16,9 @@ obj-$(CONFIG_$(SPL_)CMDLINE) += cmd_ut.o
 obj-$(CONFIG_$(SPL_)CMDLINE) += command_ut.o
 obj-$(CONFIG_$(SPL_)UT_COMPRESSION) += compression.o
 obj-y += dm/
+ifneq ($(CONFIG_HUSH_PARSER),)
+obj-$(CONFIG_CMDLINE) += hush/
+endif
 obj-$(CONFIG_$(SPL_)CMDLINE) += print_ut.o
 obj-$(CONFIG_$(SPL_)CMDLINE) += str_ut.o
 obj-$(CONFIG_UT_TIME) += time_ut.o
test/cmd_ut.c +6 -0
index 90b260f72d..714b7e7417 100644
@@ -70,6 +70,9 @@ static struct cmd_tbl cmd_ut_sub[] = {
 #ifdef CONFIG_CMD_ADDRMAP
 	U_BOOT_CMD_MKENT(addrmap, CONFIG_SYS_MAXARGS, 1, do_ut_addrmap, "", ""),
 #endif
+#if CONFIG_IS_ENABLED(HUSH_PARSER)
+	U_BOOT_CMD_MKENT(hush, CONFIG_SYS_MAXARGS, 1, do_ut_hush, "", ""),
+#endif
 };
 
 static int do_ut_all(struct cmd_tbl *cmdtp, int flag, int argc,
@@ -148,6 +151,9 @@ static char ut_help_text[] =
 #endif
 #ifdef CONFIG_CMD_ADDRMAP
 	"ut addrmap - Very basic test of addrmap command\n"
+#endif
+#if CONFIG_IS_ENABLED(HUSH_PARSER)
+	"ut hush [test-name] - Test hush behavior\n"
 #endif
 	;
 #endif /* CONFIG_SYS_LONGHELP */
test/hush/Makefile new file +6 -0
new file mode 100644
index 0000000000..dfa2a92615
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0+
+#
+# (C) Copyright 2021
+# Francis Laniel, Amarula Solutions, [email protected]
+
+obj-y += cmd_ut_hush.o
test/hush/cmd_ut_hush.c new file +20 -0
new file mode 100644
index 0000000000..48a1adbf28
@@ -0,0 +1,20 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * (C) Copyright 2021
+ * Francis Laniel, Amarula Solutions, [email protected]
+ */
+
+#include <common.h>
+#include <command.h>
+#include <test/hush.h>
+#include <test/suites.h>
+#include <test/ut.h>
+
+int do_ut_hush(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[])
+{
+	struct unit_test *tests = UNIT_TEST_SUITE_START(hush_test);
+	const int n_ents = UNIT_TEST_SUITE_COUNT(hush_test);
+
+	return cmd_ut_category("hush", "hush_test_",
+			       tests, n_ents, argc, argv);
+}
Signature · 1 line
2.25.1
#2 lore raw

RFC PATCH 2/6 test: hush: Test hush if/else.

In reply to Francis Laniel

As asked in commit 9c6bf1715f6a ("test/py: hush_if_test: Add tests to cover octal/hex values"), this commit translates test_hush_if_test.py to a C test.

---
 test/hush/Makefile |   1 +
 test/hush/if.c     | 308 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 309 insertions(+)
 create mode 100644 test/hush/if.c
test/hush/Makefile +1 -0
index dfa2a92615..a3c9ae5106 100644
@@ -4,3 +4,4 @@
 # Francis Laniel, Amarula Solutions, [email protected]
 
 obj-y += cmd_ut_hush.o
+obj-y += if.o
test/hush/if.c new file +308 -0

308 lines not on this page · raw message

Signature · 1 line
2.25.1
#3 lore raw

RFC PATCH 3/6 test/py: hush_if_test: Remove the test file.

In reply to Francis Laniel

Commit 9087ab2cc4 ("test/py: hush_if_test: Remove the test file.") translated this test to a C test, so this python file is no more needed.

---
 test/py/tests/test_hush_if_test.py | 192 -----------------------------
 1 file changed, 192 deletions(-)
 delete mode 100644 test/py/tests/test_hush_if_test.py
test/py/tests/test_hush_if_test.py deleted +0 -192
deleted file mode 100644
index d117921a6a..0000000000
@@ -1,192 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
-
-# Test operation of the "if" shell command.
-
-import os
-import os.path
-import pytest
-
-# TODO: These tests should be converted to a C test.
-# For more information please take a look at the thread
-# https://lists.denx.de/pipermail/u-boot/2019-October/388732.html
-
-pytestmark = pytest.mark.buildconfigspec('hush_parser')
-
-# The list of "if test" conditions to test.
-subtests = (
-    # Base if functionality.
-
-    ('true', True),
-    ('false', False),
-
-    # Basic operators.
-
-    ('test aaa = aaa', True),
-    ('test aaa = bbb', False),
-
-    ('test aaa != bbb', True),
-    ('test aaa != aaa', False),
-
-    ('test aaa < bbb', True),
-    ('test bbb < aaa', False),
-
-    ('test bbb > aaa', True),
-    ('test aaa > bbb', False),
-
-    ('test 123 -eq 123', True),
-    ('test 123 -eq 456', False),
-
-    ('test 123 -ne 456', True),
-    ('test 123 -ne 123', False),
-
-    ('test 123 -lt 456', True),
-    ('test 123 -lt 123', False),
-    ('test 456 -lt 123', False),
-
-    ('test 123 -le 456', True),
-    ('test 123 -le 123', True),
-    ('test 456 -le 123', False),
-
-    ('test 456 -gt 123', True),
-    ('test 123 -gt 123', False),
-    ('test 123 -gt 456', False),
-
-    ('test 456 -ge 123', True),
-    ('test 123 -ge 123', True),
-    ('test 123 -ge 456', False),
-
-    # Octal tests
-
-    ('test 010 -eq 010', True),
-    ('test 010 -eq 011', False),
-
-    ('test 010 -ne 011', True),
-    ('test 010 -ne 010', False),
-
-    # Hexadecimal tests
-
-    ('test 0x2000000 -gt 0x2000001', False),
-    ('test 0x2000000 -gt 0x2000000', False),
-    ('test 0x2000000 -gt 0x1ffffff', True),
-
-    # Mixed tests
-
-    ('test 010 -eq 10', False),
-    ('test 010 -ne 10', True),
-    ('test 0xa -eq 10', True),
-    ('test 0xa -eq 012', True),
-
-    ('test 2000000 -gt 0x1ffffff', False),
-    ('test 0x2000000 -gt 1ffffff', True),
-    ('test 0x2000000 -lt 1ffffff', False),
-    ('test 0x2000000 -eq 2000000', False),
-    ('test 0x2000000 -ne 2000000', True),
-
-    ('test -z ""', True),
-    ('test -z "aaa"', False),
-
-    ('test -n "aaa"', True),
-    ('test -n ""', False),
-
-    # Inversion of simple tests.
-
-    ('test ! aaa = aaa', False),
-    ('test ! aaa = bbb', True),
-    ('test ! ! aaa = aaa', True),
-    ('test ! ! aaa = bbb', False),
-
-    # Binary operators.
-
-    ('test aaa != aaa -o bbb != bbb', False),
-    ('test aaa != aaa -o bbb = bbb', True),
-    ('test aaa = aaa -o bbb != bbb', True),
-    ('test aaa = aaa -o bbb = bbb', True),
-
-    ('test aaa != aaa -a bbb != bbb', False),
-    ('test aaa != aaa -a bbb = bbb', False),
-    ('test aaa = aaa -a bbb != bbb', False),
-    ('test aaa = aaa -a bbb = bbb', True),
-
-    # Inversion within binary operators.
-
-    ('test ! aaa != aaa -o ! bbb != bbb', True),
-    ('test ! aaa != aaa -o ! bbb = bbb', True),
-    ('test ! aaa = aaa -o ! bbb != bbb', True),
-    ('test ! aaa = aaa -o ! bbb = bbb', False),
-
-    ('test ! ! aaa != aaa -o ! ! bbb != bbb', False),
-    ('test ! ! aaa != aaa -o ! ! bbb = bbb', True),
-    ('test ! ! aaa = aaa -o ! ! bbb != bbb', True),
-    ('test ! ! aaa = aaa -o ! ! bbb = bbb', True),
-
-    # -z operator.
-
-    ('test -z "$ut_var_nonexistent"', True),
-    ('test -z "$ut_var_exists"', False),
-)
-
-def exec_hush_if(u_boot_console, expr, result):
-    """Execute a shell "if" command, and validate its result."""
-
-    config = u_boot_console.config.buildconfig
-    maxargs = int(config.get('config_sys_maxargs', '0'))
-    args = len(expr.split(' ')) - 1
-    if args > maxargs:
-        u_boot_console.log.warning('CONFIG_SYS_MAXARGS too low; need ' +
-            str(args))
-        pytest.skip()
-
-    cmd = 'if ' + expr + '; then echo true; else echo false; fi'
-    response = u_boot_console.run_command(cmd)
-    assert response.strip() == str(result).lower()
-
-def test_hush_if_test_setup(u_boot_console):
-    """Set up environment variables used during the "if" tests."""
-
-    u_boot_console.run_command('setenv ut_var_nonexistent')
-    u_boot_console.run_command('setenv ut_var_exists 1')
-
[email protected]('cmd_echo')
[email protected]('expr,result', subtests)
-def test_hush_if_test(u_boot_console, expr, result):
-    """Test a single "if test" condition."""
-
-    exec_hush_if(u_boot_console, expr, result)
-
-def test_hush_if_test_teardown(u_boot_console):
-    """Clean up environment variables used during the "if" tests."""
-
-    u_boot_console.run_command('setenv ut_var_exists')
-
-# We might test this on real filesystems via UMS, DFU, 'save', etc.
-# Of those, only UMS currently allows file removal though.
[email protected]('cmd_echo')
[email protected]('sandbox')
-def test_hush_if_test_host_file_exists(u_boot_console):
-    """Test the "if test -e" shell command."""
-
-    test_file = u_boot_console.config.result_dir + \
-        '/creating_this_file_breaks_u_boot_tests'
-
-    try:
-        os.unlink(test_file)
-    except:
-        pass
-    assert not os.path.exists(test_file)
-
-    expr = 'test -e hostfs - ' + test_file
-    exec_hush_if(u_boot_console, expr, False)
-
-    try:
-        with open(test_file, 'wb'):
-            pass
-        assert os.path.exists(test_file)
-
-        expr = 'test -e hostfs - ' + test_file
-        exec_hush_if(u_boot_console, expr, True)
-    finally:
-        os.unlink(test_file)
-
-    expr = 'test -e hostfs - ' + test_file
-    exec_hush_if(u_boot_console, expr, False)
Signature · 1 line
2.25.1
#4 lore raw

RFC PATCH 4/6 test: hush: Test hush variable expansion.

In reply to Francis Laniel

This commit ensures shell variables are replaced by their values.

---
 test/hush/Makefile |   1 +
 test/hush/dollar.c | 188 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 189 insertions(+)
 create mode 100644 test/hush/dollar.c
test/hush/Makefile +1 -0
index a3c9ae5106..feb4f71956 100644
@@ -5,3 +5,4 @@
 
 obj-y += cmd_ut_hush.o
 obj-y += if.o
+obj-y += dollar.o
test/hush/dollar.c new file +188 -0

188 lines not on this page · raw message

Signature · 1 line
2.25.1
#5 lore raw

RFC PATCH 5/6 test: hush: Test hush commands list.

In reply to Francis Laniel

This commit ensures behavior of commands separated by ';', '&&' and '||'.

---
 test/hush/Makefile |  1 +
 test/hush/list.c   | 79 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 80 insertions(+)
 create mode 100644 test/hush/list.c
test/hush/Makefile +1 -0
index feb4f71956..ff4fe7700b 100644
@@ -6,3 +6,4 @@
 obj-y += cmd_ut_hush.o
 obj-y += if.o
 obj-y += dollar.o
+obj-y += list.o
test/hush/list.c new file +79 -0

79 lines not on this page · raw message

Signature · 1 line
2.25.1
#6 lore raw

RFC PATCH 6/6 test: hush: Test hush loops.

In reply to Francis Laniel

The added tests ensure correct behavior of for, while and until loops.

---
 test/hush/Makefile |  1 +
 test/hush/loop.c   | 64 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+)
 create mode 100644 test/hush/loop.c
test/hush/Makefile +1 -0
index ff4fe7700b..a2d98815e5 100644
@@ -7,3 +7,4 @@ obj-y += cmd_ut_hush.o
 obj-y += if.o
 obj-y += dollar.o
 obj-y += list.o
+obj-y += loop.o
test/hush/loop.c new file +64 -0

64 lines not on this page · raw message

Signature · 1 line
2.25.1