Add C tests for hush
cover Add C tests for hush
Hi everyone.
First, I hope you are fine and the same for your relatives.
This series introduces C tests for hush.
The main goal of these tests is to act as regression tests in case of hush being modified.
During their writings some bugs were found, unfortunately I did not write patches to fix them.
First, I think these bugs were present in Busybox when hush was taken from. Also, as the whole U-Boot board scripts seem to work with these bugs maybe correcting them can lead to breaking things somewhere...
This is why I marked this series as RFC as I would like the hear the opinion of the community on this problem.
Regarding the series, first commit introduces a new subcommand to ut: ut hush. Then, others add tests for the following:
* if test condition: All these tests were taken from test_hush_if_test.py which was then removed.
* variable expansion: These tests ensures variable expansion is correct. While writing them a bug was found in hush where the following:
foo='bar "quux'
echo $foo
prints:
bar quux
instead of:
bar "quux
* commands lists: These tests test several command separated by ';', "&&" and "||". Here also, some bugs were found, indeed the following:
false && false || true
returns 0 in actual (2021) hush's Busybox, while it returns 1 in U-Boot.
* loops: This test is quite simple as we do not have math support in U-Boot, this can be tricky to heavily test while and until loops.
To conclude, here is this series diffstat:
Francis Laniel (6):
test: Add framework to test hush behavior.
test: hush: Test hush if/else.
test/py: hush_if_test: Remove the test file.
test: hush: Test hush variable expansion.
test: hush: Test hush commands list.
test: hush: Test hush loops.
include/test/hush.h | 15 ++ include/test/suites.h | 1 + test/Makefile | 3 + test/cmd_ut.c | 6 + test/hush/Makefile | 10 + test/hush/cmd_ut_hush.c | 20 ++ test/hush/dollar.c | 188 ++++++++++++++++++ test/hush/if.c | 308 +++++++++++++++++++++++++++++ test/hush/list.c | 79 ++++++++ test/hush/loop.c | 64 ++++++ test/py/tests/test_hush_if_test.py | 192 ------------------ 11 files changed, 694 insertions(+), 192 deletions(-) create mode 100644 include/test/hush.h create mode 100644 test/hush/Makefile create mode 100644 test/hush/cmd_ut_hush.c create mode 100644 test/hush/dollar.c create mode 100644 test/hush/if.c create mode 100644 test/hush/list.c create mode 100644 test/hush/loop.c delete mode 100644 test/py/tests/test_hush_if_test.py
Best regards.
--
2.25.1
RFC PATCH 1/6 test: Add framework to test hush behavior.
This commit introduces a new subcommand to ut: ut hush.
For the moment, this command does nothing, future commits will add tests which will be run on command call.
Note that CONFIG_HUSH_PARSER must be defined to compile this new subcommand.
--- include/test/hush.h | 15 +++++++++++++++ include/test/suites.h | 1 + test/Makefile | 3 +++ test/cmd_ut.c | 6 ++++++ test/hush/Makefile | 6 ++++++ test/hush/cmd_ut_hush.c | 20 ++++++++++++++++++++ 6 files changed, 51 insertions(+) create mode 100644 include/test/hush.h create mode 100644 test/hush/Makefile create mode 100644 test/hush/cmd_ut_hush.c
include/test/hush.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* (C) Copyright 2021 +* Francis Laniel, Amarula Solutions, [email protected] +*/ + +#ifndef __TEST_HUSH_H__ +#define __TEST_HUSH_H__ + +#include <test/test.h> + +/* Declare a new environment test */ +#define HUSH_TEST (_name, _flags) UNIT_TEST(_name, _flags, hush_test) + +#endif /* __TEST_HUSH_H__ */
include/test/suites.h +1 -0
@@ -36,6 +36,7 @@ int do_ut_compression(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]);int do_ut_dm (struct cmd_tbl *cmdtp,int flag,int argc,char *const argv[]);int do_ut_env (struct cmd_tbl *cmdtp,int flag,int argc,char *const argv[]); +int do_ut_hush (struct cmd_tbl *cmdtp,int flag,int argc,char *const argv[]);int do_ut_lib (struct cmd_tbl *cmdtp,int flag,int argc,char *const argv[]);int do_ut_log (struct cmd_tbl *cmdtp,int flag,int argc,char *const argv[]);int do_ut_mem (struct cmd_tbl *cmdtp,int flag,int argc,char *const argv[]);
test/Makefile +3 -0
@@ -16,6 +16,9 @@ obj-$(CONFIG_$(SPL_)CMDLINE) += cmd_ut.o obj-$(CONFIG_$(SPL_)CMDLINE) += command_ut.o obj-$(CONFIG_$(SPL_)UT_COMPRESSION) += compression.o obj-y += dm/ +ifneq ($(CONFIG_HUSH_PARSER),) +obj-$(CONFIG_CMDLINE) += hush/ +endif obj-$(CONFIG_$(SPL_)CMDLINE) += print_ut.o obj-$(CONFIG_$(SPL_)CMDLINE) += str_ut.o obj-$(CONFIG_UT_TIME) += time_ut.o
test/cmd_ut.c +6 -0
@@ -70,6 +70,9 @@ static struct cmd_tbl cmd_ut_sub[] = {#ifdef CONFIG_CMD_ADDRMAP U_BOOT_CMD_MKENT(addrmap, CONFIG_SYS_MAXARGS, 1, do_ut_addrmap, "", ""),#endif +#if CONFIG_IS_ENABLED (HUSH_PARSER ) + U_BOOT_CMD_MKENT(hush, CONFIG_SYS_MAXARGS, 1, do_ut_hush, "", ""), +#endif }; static int do_ut_all(struct cmd_tbl *cmdtp, int flag, int argc,
@@ -148,6 +151,9 @@ static char ut_help_text[] =#endif #ifdef CONFIG_CMD_ADDRMAP "ut addrmap - Very basic test of addrmap command\n" +#endif +#if CONFIG_IS_ENABLED (HUSH_PARSER ) + "ut hush [test-name] - Test hush behavior\n"#endif ;#endif /* CONFIG_SYS_LONGHELP */
test/hush/Makefile new file +6 -0
@@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0+ +# +# (C) Copyright 2021 +# Francis Laniel, Amarula Solutions, [email protected] + +obj-y += cmd_ut_hush.o
test/hush/cmd_ut_hush.c new file +20 -0
@@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* (C) Copyright 2021 +* Francis Laniel, Amarula Solutions, [email protected] +*/ + +#include <common.h> +#include <command.h> +#include <test/hush.h> +#include <test/suites.h> +#include <test/ut.h> + +int do_ut_hush (struct cmd_tbl *cmdtp,int flag,int argc,char *const argv[]) +{ +struct unit_test *tests =UNIT_TEST_SUITE_START (hush_test); +const int n_ents =UNIT_TEST_SUITE_COUNT (hush_test); + +return cmd_ut_category ("hush" ,"hush_test_" , + tests, n_ents, argc, argv); +}
Signature · 1 line
2.25.1
RFC PATCH 2/6 test: hush: Test hush if/else.
In reply to Francis LanielAs asked in commit 9c6bf1715f6a ("test/py: hush_if_test: Add tests to cover octal/hex values"), this commit translates test_hush_if_test.py to a C test.
--- test/hush/Makefile | 1 + test/hush/if.c | 308 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 309 insertions(+) create mode 100644 test/hush/if.c
test/hush/Makefile +1 -0
@@ -4,3 +4,4 @@ # Francis Laniel, Amarula Solutions, [email protected] obj-y += cmd_ut_hush.o +obj-y += if.o
test/hush/if.c new file +308 -0
308 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC PATCH 3/6 test/py: hush_if_test: Remove the test file.
In reply to Francis LanielCommit 9087ab2cc4 ("test/py: hush_if_test: Remove the test file.") translated this test to a C test, so this python file is no more needed.
--- test/py/tests/test_hush_if_test.py | 192 ----------------------------- 1 file changed, 192 deletions(-) delete mode 100644 test/py/tests/test_hush_if_test.py
test/py/tests/test_hush_if_test.py deleted +0 -192
@@ -1,192 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0 -# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved. - -# Test operation of the "if" shell command. - -import os -import os.path -import pytest - -# TODO: These tests should be converted to a C test. -# For more information please take a look at the thread -# https://lists.denx.de/pipermail/u-boot/2019-October/388732.html - -pytestmark = pytest.mark.buildconfigspec('hush_parser') - -# The list of "if test" conditions to test. -subtests = ( - # Base if functionality. - - ('true', True), - ('false', False), - - # Basic operators. - - ('test aaa = aaa', True), - ('test aaa = bbb', False), - - ('test aaa != bbb', True), - ('test aaa != aaa', False), - - ('test aaa < bbb', True), - ('test bbb < aaa', False), - - ('test bbb > aaa', True), - ('test aaa > bbb', False), - - ('test 123 -eq 123', True), - ('test 123 -eq 456', False), - - ('test 123 -ne 456', True), - ('test 123 -ne 123', False), - - ('test 123 -lt 456', True), - ('test 123 -lt 123', False), - ('test 456 -lt 123', False), - - ('test 123 -le 456', True), - ('test 123 -le 123', True), - ('test 456 -le 123', False), - - ('test 456 -gt 123', True), - ('test 123 -gt 123', False), - ('test 123 -gt 456', False), - - ('test 456 -ge 123', True), - ('test 123 -ge 123', True), - ('test 123 -ge 456', False), - - # Octal tests - - ('test 010 -eq 010', True), - ('test 010 -eq 011', False), - - ('test 010 -ne 011', True), - ('test 010 -ne 010', False), - - # Hexadecimal tests - - ('test 0x2000000 -gt 0x2000001', False), - ('test 0x2000000 -gt 0x2000000', False), - ('test 0x2000000 -gt 0x1ffffff', True), - - # Mixed tests - - ('test 010 -eq 10', False), - ('test 010 -ne 10', True), - ('test 0xa -eq 10', True), - ('test 0xa -eq 012', True), - - ('test 2000000 -gt 0x1ffffff', False), - ('test 0x2000000 -gt 1ffffff', True), - ('test 0x2000000 -lt 1ffffff', False), - ('test 0x2000000 -eq 2000000', False), - ('test 0x2000000 -ne 2000000', True), - - ('test -z ""', True), - ('test -z "aaa"', False), - - ('test -n "aaa"', True), - ('test -n ""', False), - - # Inversion of simple tests. - - ('test ! aaa = aaa', False), - ('test ! aaa = bbb', True), - ('test ! ! aaa = aaa', True), - ('test ! ! aaa = bbb', False), - - # Binary operators. - - ('test aaa != aaa -o bbb != bbb', False), - ('test aaa != aaa -o bbb = bbb', True), - ('test aaa = aaa -o bbb != bbb', True), - ('test aaa = aaa -o bbb = bbb', True), - - ('test aaa != aaa -a bbb != bbb', False), - ('test aaa != aaa -a bbb = bbb', False), - ('test aaa = aaa -a bbb != bbb', False), - ('test aaa = aaa -a bbb = bbb', True), - - # Inversion within binary operators. - - ('test ! aaa != aaa -o ! bbb != bbb', True), - ('test ! aaa != aaa -o ! bbb = bbb', True), - ('test ! aaa = aaa -o ! bbb != bbb', True), - ('test ! aaa = aaa -o ! bbb = bbb', False), - - ('test ! ! aaa != aaa -o ! ! bbb != bbb', False), - ('test ! ! aaa != aaa -o ! ! bbb = bbb', True), - ('test ! ! aaa = aaa -o ! ! bbb != bbb', True), - ('test ! ! aaa = aaa -o ! ! bbb = bbb', True), - - # -z operator. - - ('test -z "$ut_var_nonexistent"', True), - ('test -z "$ut_var_exists"', False), -) - -def exec_hush_if(u_boot_console, expr, result): - """Execute a shell "if" command, and validate its result.""" - - config = u_boot_console.config.buildconfig - maxargs = int(config.get('config_sys_maxargs', '0')) - args = len(expr.split(' ')) - 1 - if args > maxargs: - u_boot_console.log.warning('CONFIG_SYS_MAXARGS too low; need ' + - str(args)) - pytest.skip() - - cmd = 'if ' + expr + '; then echo true; else echo false; fi' - response = u_boot_console.run_command(cmd) - assert response.strip() == str(result).lower() - -def test_hush_if_test_setup(u_boot_console): - """Set up environment variables used during the "if" tests.""" - - u_boot_console.run_command('setenv ut_var_nonexistent') - u_boot_console.run_command('setenv ut_var_exists 1') - [email protected]('cmd_echo') [email protected]('expr,result', subtests) -def test_hush_if_test(u_boot_console, expr, result): - """Test a single "if test" condition.""" - - exec_hush_if(u_boot_console, expr, result) - -def test_hush_if_test_teardown(u_boot_console): - """Clean up environment variables used during the "if" tests.""" - - u_boot_console.run_command('setenv ut_var_exists') - -# We might test this on real filesystems via UMS, DFU, 'save', etc. -# Of those, only UMS currently allows file removal though. [email protected]('cmd_echo') [email protected]('sandbox') -def test_hush_if_test_host_file_exists(u_boot_console): - """Test the "if test -e" shell command.""" - - test_file = u_boot_console.config.result_dir + \ - '/creating_this_file_breaks_u_boot_tests' - - try: - os.unlink(test_file) - except: - pass - assert not os.path.exists(test_file) - - expr = 'test -e hostfs - ' + test_file - exec_hush_if(u_boot_console, expr, False) - - try: - with open(test_file, 'wb'): - pass - assert os.path.exists(test_file) - - expr = 'test -e hostfs - ' + test_file - exec_hush_if(u_boot_console, expr, True) - finally: - os.unlink(test_file) - - expr = 'test -e hostfs - ' + test_file - exec_hush_if(u_boot_console, expr, False)
Signature · 1 line
2.25.1
RFC PATCH 4/6 test: hush: Test hush variable expansion.
In reply to Francis LanielThis commit ensures shell variables are replaced by their values.
--- test/hush/Makefile | 1 + test/hush/dollar.c | 188 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 189 insertions(+) create mode 100644 test/hush/dollar.c
test/hush/Makefile +1 -0
@@ -5,3 +5,4 @@ obj-y += cmd_ut_hush.o obj-y += if.o +obj-y += dollar.o
test/hush/dollar.c new file +188 -0
188 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC PATCH 5/6 test: hush: Test hush commands list.
In reply to Francis LanielThis commit ensures behavior of commands separated by ';', '&&' and '||'.
--- test/hush/Makefile | 1 + test/hush/list.c | 79 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+) create mode 100644 test/hush/list.c
test/hush/Makefile +1 -0
@@ -6,3 +6,4 @@ obj-y += cmd_ut_hush.o obj-y += if.o obj-y += dollar.o +obj-y += list.o
test/hush/list.c new file +79 -0
79 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC PATCH 6/6 test: hush: Test hush loops.
In reply to Francis LanielThe added tests ensure correct behavior of for, while and until loops.
--- test/hush/Makefile | 1 + test/hush/loop.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+) create mode 100644 test/hush/loop.c
test/hush/Makefile +1 -0
@@ -7,3 +7,4 @@ obj-y += cmd_ut_hush.o obj-y += if.o obj-y += dollar.o obj-y += list.o +obj-y += loop.o
test/hush/loop.c new file +64 -0
64 lines not on this page · raw message
Signature · 1 line
2.25.1