test/py: Implement pytest infrastructure
PATCH v4 1/8 test/py: Implement pytest infrastructure
This tool aims to test U-Boot by executing U-Boot shell commands using the console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:
- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself. It is asserted that writing test-related code in Python is simpler and more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.
A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.
The hook scripts, relay control utilities, and udev rules I use for my own HW setup are published at https://github.com/swarren/uboot-test-hooks.
See README.md for more details!
---
Apologies for the resend; I forgot to CC the list last time.
v4:
- Fix all users after ubspawn -> u_boot_spawn module rename.
- Don't escape ^ in the HTML log file.
- Fix u_boot_spawn.py Spawn.__init__ to set SIGHUP handler is set to SIG_DFL. This ensures that sandbox responds to SIGHUP when run non- interactively (e.g. under a daemon continuous integration server).
- Fix logging of all child processes to make sure that all stdout/err, exit code, and exception data from all processes makes its way to both test stdout (for pytest to log to its stdout in case of error) and the HTML test log. This changed multiplexed_log.py RunAndLog.run() (a fair bit of change) and u_boot_console_exec_attach.py ConseolExecAttach.* (just passing sys.stdout to log.get_runner() everywhere).
- Also mention kermit as a possible app to attach to a serial console.
- Fix test_version to work correctly when BUILD_TAG is set, which causes U-Boot to print both the version string and BUILD_TAG value at startup, whereas the "version" shell command only prints the version string.
- Reformat function-level documentation.
v3:
- Rework HTML log generation so that TAB characters render as expected. Suggested by Michal Simek.
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Typo fixes.
- Add more documentation. Suggested by Simon Glass.
- Make "notes" in the log file be <pre> so that their formatting is preserved. This is useful for large notes such as exception dumps.
v2:
- Many fixes and tweaks have been squashed in. Separated out some of the tests into separate commits, and added some more tests.
--- test/py/.gitignore | 1 + test/py/README.md | 300 ++++++++++++++++++++ test/py/conftest.py | 422 ++++++++++++++++++++++++++++ test/py/multiplexed_log.css | 88 ++++++ test/py/multiplexed_log.py | 515 ++++++++++++++++++++++++++++++++++ test/py/pytest.ini | 11 + test/py/test.py | 32 +++ test/py/tests/test_000_version.py | 20 ++ test/py/tests/test_help.py | 9 + test/py/tests/test_unknown_cmd.py | 14 + test/py/u_boot_console_base.py | 360 ++++++++++++++++++++++++ test/py/u_boot_console_exec_attach.py | 65 +++++ test/py/u_boot_console_sandbox.py | 79 ++++++ test/py/u_boot_spawn.py | 174 ++++++++++++ 14 files changed, 2090 insertions(+) create mode 100644 test/py/.gitignore create mode 100644 test/py/README.md create mode 100644 test/py/conftest.py create mode 100644 test/py/multiplexed_log.css create mode 100644 test/py/multiplexed_log.py create mode 100644 test/py/pytest.ini create mode 100755 test/py/test.py create mode 100644 test/py/tests/test_000_version.py create mode 100644 test/py/tests/test_help.py create mode 100644 test/py/tests/test_unknown_cmd.py create mode 100644 test/py/u_boot_console_base.py create mode 100644 test/py/u_boot_console_exec_attach.py create mode 100644 test/py/u_boot_console_sandbox.py create mode 100644 test/py/u_boot_spawn.py
test/py/.gitignore new file +1 -0
@@ -0,0 +1 @@ +*.pyc
test/py/README.md new file +300 -0
300 lines not on this page · raw message
test/py/conftest.py new file +422 -0
422 lines not on this page · raw message
test/py/multiplexed_log.css new file +88 -0
88 lines not on this page · raw message
test/py/multiplexed_log.py new file +515 -0
515 lines not on this page · raw message
test/py/pytest.ini new file +11 -0
@@ -0,0 +1,11 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +# Static configuration data for pytest. pytest reads this at startup time. + +[pytest] +markers = + boardspec: U-Boot: Describes the set of boards a test can/can't run on. + buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.
test/py/test.py new file +32 -0
@@ -0,0 +1,32 @@ +#!/usr/bin/env python + +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +# Wrapper script to invoke pytest with the directory name that contains the +# U-Boot tests. + +import os +import os.path +import sys + +# Get rid of argv[0] +sys.argv.pop(0) + +# argv; py.test test_directory_name user-supplied-arguments +args = ["py.test", os.path.dirname(__file__) + "/tests"] +args.extend(sys.argv) + +try: + os.execvp("py.test", args) +except: + # Log full details of any exception for detailed analysis + import traceback + traceback.print_exc() + # Hint to the user that they likely simply haven't installed the required + # dependencies. + print >>sys.stderr, """ +exec(py.test) failed; perhaps you are missing some dependencies? +See test/py/README.md for the list."""
test/py/tests/test_000_version.py new file +20 -0
@@ -0,0 +1,20 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +# pytest runs tests the order of their module path, which is related to the +# filename containing the test. This file is named such that it is sorted +# first, simply as a very basic sanity check of the functionality of the U-Boot +# command prompt. + +def test_version(u_boot_console): + '''Test that the "version" command prints the U-Boot version.''' + + # "version" prints the U-Boot sign-on message. This is usually considered + # an error, so that any unexpected reboot causes an error. Here, this + # error detection is disabled since the sign-on message is expected. + with u_boot_console.disable_check('main_signon'): + response = u_boot_console.run_command('version') + # Ensure "version" printed what we expected. + u_boot_console.validate_version_string_in_text(response)
test/py/tests/test_help.py new file +9 -0
@@ -0,0 +1,9 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +def test_help(u_boot_console): + '''Test that the "help" command can be executed.''' + + u_boot_console.run_command('help')
test/py/tests/test_unknown_cmd.py new file +14 -0
@@ -0,0 +1,14 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +def test_unknown_command(u_boot_console): + '''Test that executing an unknown command causes U-Boot to print an + error.''' + + # The "unknown command" error is actively expected here, + # so error detection for it is disabled. + with u_boot_console.disable_check('unknown_command'): + response = u_boot_console.run_command('non_existent_cmd') + assert('Unknown command \'non_existent_cmd\' - try \'help\'' in response)
test/py/u_boot_console_base.py new file +360 -0
360 lines not on this page · raw message
test/py/u_boot_console_exec_attach.py new file +65 -0
65 lines not on this page · raw message
test/py/u_boot_console_sandbox.py new file +79 -0
79 lines not on this page · raw message
test/py/u_boot_spawn.py new file +174 -0
174 lines not on this page · raw message
Signature · 1 line
2.7.0
PATCH v4 2/8 test/py: test that sandbox exits when asked
Test the sandbox port's implementation of the reset command and SIGHUP handling. These should both cause the U-Boot process to exit gracefully.
---
v4:
- No changes.
v3:
- Move test scripts into a sub-directory. Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- Typo.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
--- test/py/tests/test_sandbox_exit.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 test/py/tests/test_sandbox_exit.py
test/py/tests/test_sandbox_exit.py new file +24 -0
@@ -0,0 +1,24 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +import pytest +import signal + + at pytest.mark.boardspec('sandbox') + at pytest.mark.buildconfigspec('reset') +def test_reset(u_boot_console): + '''Test that the "reset" command exits sandbox process.''' + + u_boot_console.run_command('reset', wait_for_prompt=False) + assert(u_boot_console.validate_exited()) + u_boot_console.ensure_spawned() + + at pytest.mark.boardspec('sandbox') +def test_ctrl_c(u_boot_console): + '''Test that sending SIGINT to sandbox causes it to exit.''' + + u_boot_console.kill(signal.SIGINT) + assert(u_boot_console.validate_exited()) + u_boot_console.ensure_spawned()
Signature · 1 line
2.7.0
PATCH v4 2/8 test/py: test that sandbox exits when asked
Quoted 17 lines · Stephen Warren
Test the sandbox port's implementation of the reset command and SIGHUP handling. These should both cause the U-Boot process to exit gracefully.
Signed-off-by Stephen Warren <[email protected]>Signed-off-by Stephen Warren <[email protected]>Acked-by Simon Glass <[email protected]>---
v4:
- No changes.v3:
- Move test scripts into a sub-directory. Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- Typo.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
Quoted 4 lines
--- test/py/tests/test_sandbox_exit.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 test/py/tests/test_sandbox_exit.py
Applied to u-boot-dm, thanks!
PATCH v4 3/8 test/py: add test of setenv/printenv/echo
In reply to Stephen WarrenThis tests basic environment variable functionality.
---
v4:
- Reformat function-level documentation.
v3:
- Handle environment values that contain an = sign.
Reported by Michal Simek.
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Simplify command string construction. Suggested by Simon Glass.
- Move relevant edits to command_ut.c into this patch from a later one. Suggested by Simon Glass.
--- test/command_ut.c | 4 - test/py/tests/test_env.py | 221 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 221 insertions(+), 4 deletions(-) create mode 100644 test/py/tests/test_env.py
test/command_ut.c +0 -4
@@ -20,10 +20,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])printf ("%s: Testing commands\n" , __func__);run_command ("env default -f -a" ,0 ); -/* run a single command */ -run_command ("setenv single 1" ,0 ); -assert (!strcmp ("1" ,getenv ("single" ))); -/* make sure that compound statements work */ #ifdef CONFIG_SYS_HUSH_PARSER run_command ("if test -n ${single} ; then setenv check 1; fi" ,0 );
test/py/tests/test_env.py new file +221 -0
221 lines not on this page · raw message
Signature · 1 line
2.7.0
PATCH v4 3/8 test/py: add test of setenv/printenv/echo
Quoted 20 lines · Stephen Warren
This tests basic environment variable functionality.
Signed-off-by Stephen Warren <[email protected]>Signed-off-by Stephen Warren <[email protected]>Acked-by Simon Glass <[email protected]>---
v4:
- Reformat function-level documentation.v3:
- Handle environment values that contain an = sign.
Reported by Michal Simek.
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Simplify command string construction. Suggested by Simon Glass.
- Move relevant edits tocommand_ut.cinto this patch from a later one. Suggested by Simon Glass.
Quoted 5 lines
--- test/command_ut.c | 4 - test/py/tests/test_env.py | 221 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 221 insertions(+), 4 deletions(-) create mode 100644 test/py/tests/test_env.py
Applied to u-boot-dm, thanks!
PATCH v4 4/8 test/py: test the md/mw commands
In reply to Stephen WarrenThis tests whether md/mw work, and affect each-other.
Command repeat is also tested.
test/cmd_repeat.sh is removed, since the new Python-based test does everything it used to.
---
v4:
- No changes.
v3:
- Add extra mw during md test to account for the possibility that the test's write data may already be present in RAM. Suggested by Michal Simek.
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
--- test/cmd_repeat.sh | 29 ----------------------------- test/py/tests/test_md.py | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 29 deletions(-) delete mode 100755 test/cmd_repeat.sh create mode 100644 test/py/tests/test_md.py
test/cmd_repeat.sh deleted +0 -29
@@ -1,29 +0,0 @@ -#!/bin/sh - -# Test for U-Boot cli including command repeat - -BASE="$(dirname $0)" -. $BASE/common.sh - -run_test() { - ./${OUTPUT_DIR}/u-boot <<END -setenv ctrlc_ignore y -md 0 - -reset -END -} -check_results() { - echo "Check results" - - grep -q 00000100 ${tmp} || fail "Command did not repeat" -} - -echo "Test CLI repeat" -echo -tmp="$(tempfile)" -build_uboot -run_test >${tmp} -check_results ${tmp} -rm ${tmp} -echo "Test passed"
test/py/tests/test_md.py new file +36 -0
@@ -0,0 +1,36 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +import pytest + + at pytest.mark.buildconfigspec('cmd_memory') +def test_md(u_boot_console): + '''Test that md reads memory as expected, and that memory can be modified + using the mw command.''' + + ram_base = u_boot_console.find_ram_base() + addr = '%08x' % ram_base + val = 'a5f09876' + expected_response = addr + ': ' + val + u_boot_console.run_command('mw ' + addr + ' 0 10') + response = u_boot_console.run_command('md ' + addr + ' 10') + assert(not (expected_response in response)) + u_boot_console.run_command('mw ' + addr + ' ' + val) + response = u_boot_console.run_command('md ' + addr + ' 10') + assert(expected_response in response) + + at pytest.mark.buildconfigspec('cmd_memory') +def test_md_repeat(u_boot_console): + '''Test command repeat (via executing an empty command) operates correctly + for "md"; the command must repeat and dump an incrementing address.''' + + ram_base = u_boot_console.find_ram_base() + addr_base = '%08x' % ram_base + words = 0x10 + addr_repeat = '%08x' % (ram_base + (words * 4)) + u_boot_console.run_command('md %s %x' % (addr_base, words)) + response = u_boot_console.run_command('') + expected_response = addr_repeat + ': ' + assert(expected_response in response)
Signature · 1 line
2.7.0
PATCH v4 4/8 test/py: test the md/mw commands
Quoted 23 lines · Stephen Warren
This tests whether md/mw work, and affect each-other.
Command repeat is also tested.
test/cmd_repeat.shis removed, since the new Python-based test does everything it used to.Signed-off-by Stephen Warren <[email protected]>Signed-off-by Stephen Warren <[email protected]>Acked-by Simon Glass <[email protected]>---
v4:
- No changes.v3:
- Add extra mw during md test to account for the possibility that the test's write data may already be present in RAM. Suggested by Michal Simek.
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
Quoted 6 lines
--- test/cmd_repeat.sh | 29 ----------------------------- test/py/tests/test_md.py | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 29 deletions(-) delete mode 100755 test/cmd_repeat.sh create mode 100644 test/py/tests/test_md.py
Applied to u-boot-dm, thanks!
PATCH v4 5/8 test/py: add test of basic shell functionality
In reply to Stephen WarrenFrom: Stephen Warren <[email protected]>
This tests whether the following features of the U-Boot shell:
- Execution of a directly entered command.
- Compound commands (; delimiter).
- Quoting of arguments containing spaces.
- Executing commands from environment variables.
---
v4:
- No changes.
v3:
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Move relevant edits to command_ut.c into this patch from a later one. Suggested by Simon Glass.
--- test/command_ut.c | 16 --------------- test/py/tests/test_shell_basics.py | 42 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 16 deletions(-) create mode 100644 test/py/tests/test_shell_basics.py
test/command_ut.c +0 -16
@@ -27,10 +27,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])run_command ("setenv check" ,0 );#endif -/* commands separated by ; */ -run_command_list ("setenv list 1; setenv list ${list}1" ,-1 ,0 ); -assert (!strcmp ("11" ,getenv ("list" ))); -/* commands separated by \n */ run_command_list ("setenv list 1\n setenv list ${list}1" ,-1 ,0 );assert (!strcmp ("11" ,getenv ("list" )));
@@ -39,11 +35,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])run_command_list ("setenv list 1${list}\n" ,-1 ,0 );assert (!strcmp ("111" ,getenv ("list" ))); -/* three commands in a row */ -run_command_list ("setenv list 1\n setenv list ${list}2; " -"setenv list ${list}3" ,-1 ,0 ); -assert (!strcmp ("123" ,getenv ("list" ))); -/* a command string with \0 in it. Stuff after \0 should be ignored */ run_command ("setenv list" ,0 );run_command_list (test_cmd,sizeof (test_cmd),0 );
@@ -62,13 +53,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])assert (run_command_list ("false" ,-1 ,0 ) ==1 );assert (run_command_list ("echo" ,-1 ,0 ) ==0 ); -run_command ("setenv foo 'setenv monty 1; setenv python 2'" ,0 ); -run_command ("run foo" ,0 ); -assert (getenv ("monty" ) !=NULL ); -assert (!strcmp ("1" ,getenv ("monty" ))); -assert (getenv ("python" ) !=NULL ); -assert (!strcmp ("2" ,getenv ("python" ))); -#ifdef CONFIG_SYS_HUSH_PARSER run_command ("setenv foo 'setenv black 1\nsetenv adder 2'" ,0 );run_command ("run foo" ,0 );
test/py/tests/test_shell_basics.py new file +42 -0
42 lines not on this page · raw message
Signature · 1 line
2.7.0
PATCH v4 5/8 test/py: add test of basic shell functionality
Quoted 22 lines · Stephen Warren
From: Stephen Warren <[email protected]>
This tests whether the following features of the U-Boot shell:
- Execution of a directly entered command.
- Compound commands (; delimiter).
- Quoting of arguments containing spaces.
- Executing commands from environment variables.Signed-off-by Stephen Warren <[email protected]>Acked-by Simon Glass <[email protected]>---
v4:
- No changes.v3:
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Move relevant edits tocommand_ut.cinto this patch from a later one. Suggested by Simon Glass.
Quoted 5 lines
--- test/command_ut.c | 16 --------------- test/py/tests/test_shell_basics.py | 42 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 16 deletions(-) create mode 100644 test/py/tests/test_shell_basics.py
Applied to u-boot-dm, thanks!
PATCH v4 6/8 test/py: test the shell if command
In reply to Stephen WarrenFrom: Stephen Warren <[email protected]>
Migrate all most tests from command_ut.c into the Python test system. This allows the tests to be run against any U-Boot binary that supports the if command (i.e. where hush is enabled) without requiring that binary to be permanently bloated with the code from command_ut.
Some tests in command_ut.c can only be executed from C code, since they test internal (more unit-level) features of various U-Boot APIs. The migrated tests can all operate directly from the U-Boot console.
---
v4:
- No changes.
v3:
- Move test scripts into a sub-directory. Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Remove inclusion of <os.h>. Suggested by Simon Glass.
--- test/command_ut.c | 116 ---------------------------- test/py/tests/test_hush_if_test.py | 154 +++++++++++++++++++++++++++++++++++++ 2 files changed, 154 insertions(+), 116 deletions(-) create mode 100644 test/py/tests/test_hush_if_test.py
test/command_ut.c +0 -116
@@ -7,9 +7,6 @@#define DEBUG #include <common.h> -#ifdef CONFIG_SANDBOX -#include <os.h> -#endif static const char test_cmd[] ="setenv list 1\n setenv list ${list}2; " "setenv list ${list}3\0"
@@ -20,13 +17,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])printf ("%s: Testing commands\n" , __func__);run_command ("env default -f -a" ,0 ); -/* make sure that compound statements work */ -#ifdef CONFIG_SYS_HUSH_PARSER -run_command ("if test -n ${single} ; then setenv check 1; fi" ,0 ); -assert (!strcmp ("1" ,getenv ("check" ))); -run_command ("setenv check" ,0 ); -#endif -/* commands separated by \n */ run_command_list ("setenv list 1\n setenv list ${list}1" ,-1 ,0 );assert (!strcmp ("11" ,getenv ("list" )));
@@ -60,112 +50,6 @@ static int do_ut_cmd(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])assert (!strcmp ("1" ,getenv ("black" )));assert (getenv ("adder" ) !=NULL );assert (!strcmp ("2" ,getenv ("adder" ))); - -/* Test the 'test' command */ - -#define HUSH_TEST (name, expr, expected_result) \ - run_command("if test " expr " ; then " \ - "setenv " #name "_" #expected_result " y; else " \ - "setenv " #name "_" #expected_result " n; fi", 0); \ - assert(!strcmp(#expected_result, getenv(#name "_" #expected_result))); \ - setenv(#name "_" #expected_result, NULL); - -/* Basic operators */ -HUSH_TEST (streq,"aaa = aaa" , y); -HUSH_TEST (streq,"aaa = bbb" , n); - -HUSH_TEST (strneq,"aaa != bbb" , y); -HUSH_TEST (strneq,"aaa != aaa" , n); - -HUSH_TEST (strlt,"aaa < bbb" , y); -HUSH_TEST (strlt,"bbb < aaa" , n); - -HUSH_TEST (strgt,"bbb > aaa" , y); -HUSH_TEST (strgt,"aaa > bbb" , n); - -HUSH_TEST (eq,"123 -eq 123" , y); -HUSH_TEST (eq,"123 -eq 456" , n); - -HUSH_TEST (ne,"123 -ne 456" , y); -HUSH_TEST (ne,"123 -ne 123" , n); - -HUSH_TEST (lt,"123 -lt 456" , y); -HUSH_TEST (lt_eq,"123 -lt 123" , n); -HUSH_TEST (lt,"456 -lt 123" , n); - -HUSH_TEST (le,"123 -le 456" , y); -HUSH_TEST (le_eq,"123 -le 123" , y); -HUSH_TEST (le,"456 -le 123" , n); - -HUSH_TEST (gt,"456 -gt 123" , y); -HUSH_TEST (gt_eq,"123 -gt 123" , n); -HUSH_TEST (gt,"123 -gt 456" , n); - -HUSH_TEST (ge,"456 -ge 123" , y); -HUSH_TEST (ge_eq,"123 -ge 123" , y); -HUSH_TEST (ge,"123 -ge 456" , n); - -HUSH_TEST (z,"-z \"\"" , y); -HUSH_TEST (z,"-z \"aaa\"" , n); - -HUSH_TEST (n,"-n \"aaa\"" , y); -HUSH_TEST (n,"-n \"\"" , n); - -/* Inversion of simple tests */ -HUSH_TEST (streq_inv,"! aaa = aaa" , n); -HUSH_TEST (streq_inv,"! aaa = bbb" , y); - -HUSH_TEST (streq_inv_inv,"! ! aaa = aaa" , y); -HUSH_TEST (streq_inv_inv,"! ! aaa = bbb" , n); - -/* Binary operators */ -HUSH_TEST (or_0_0,"aaa != aaa -o bbb != bbb" , n); -HUSH_TEST (or_0_1,"aaa != aaa -o bbb = bbb" , y); -HUSH_TEST (or_1_0,"aaa = aaa -o bbb != bbb" , y); -HUSH_TEST (or_1_1,"aaa = aaa -o bbb = bbb" , y); - -HUSH_TEST (and_0_0,"aaa != aaa -a bbb != bbb" , n); -HUSH_TEST (and_0_1,"aaa != aaa -a bbb = bbb" , n); -HUSH_TEST (and_1_0,"aaa = aaa -a bbb != bbb" , n); -HUSH_TEST (and_1_1,"aaa = aaa -a bbb = bbb" , y); - -/* Inversion within binary operators */ -HUSH_TEST (or_0_0_inv,"! aaa != aaa -o ! bbb != bbb" , y); -HUSH_TEST (or_0_1_inv,"! aaa != aaa -o ! bbb = bbb" , y); -HUSH_TEST (or_1_0_inv,"! aaa = aaa -o ! bbb != bbb" , y); -HUSH_TEST (or_1_1_inv,"! aaa = aaa -o ! bbb = bbb" , n); - -HUSH_TEST (or_0_0_inv_inv,"! ! aaa != aaa -o ! ! bbb != bbb" , n); -HUSH_TEST (or_0_1_inv_inv,"! ! aaa != aaa -o ! ! bbb = bbb" , y); -HUSH_TEST (or_1_0_inv_inv,"! ! aaa = aaa -o ! ! bbb != bbb" , y); -HUSH_TEST (or_1_1_inv_inv,"! ! aaa = aaa -o ! ! bbb = bbb" , y); - -setenv ("ut_var_nonexistent" ,NULL ); -setenv ("ut_var_exists" ,"1" ); -HUSH_TEST (z_varexp_quoted,"-z \"$ut_var_nonexistent\"" , y); -HUSH_TEST (z_varexp_quoted,"-z \"$ut_var_exists\"" , n); -setenv ("ut_var_exists" ,NULL ); - -run_command ("setenv ut_var_space \" \"" ,0 ); -assert (!strcmp (getenv ("ut_var_space" )," " )); -run_command ("setenv ut_var_test $ut_var_space" ,0 ); -assert (!getenv ("ut_var_test" )); -run_command ("setenv ut_var_test \"$ut_var_space\"" ,0 ); -assert (!strcmp (getenv ("ut_var_test" )," " )); -run_command ("setenv ut_var_test \" 1${ut_var_space}${ut_var_space} 2 \"" ,0 ); -assert (!strcmp (getenv ("ut_var_test" )," 1 2 " )); -setenv ("ut_var_space" ,NULL ); -setenv ("ut_var_test" ,NULL ); - -#ifdef CONFIG_SANDBOX -/* File existence */ -HUSH_TEST (e,"-e hostfs - creating_this_file_breaks_uboot_unit_test" , n); -run_command ("sb save hostfs - creating_this_file_breaks_uboot_unit_test 0 1" ,0 ); -HUSH_TEST (e,"-e hostfs - creating_this_file_breaks_uboot_unit_test" , y); -/* Perhaps this could be replaced by an "rm" shell command one day */ -assert (!os_unlink ("creating_this_file_breaks_uboot_unit_test" )); -HUSH_TEST (e,"-e hostfs - creating_this_file_breaks_uboot_unit_test" , n); -#endif #endif assert (run_command ("" ,0 ) ==0 );
test/py/tests/test_hush_if_test.py new file +154 -0
154 lines not on this page · raw message
Signature · 1 line
2.7.0
PATCH v4 6/8 test/py: test the shell if command
Quoted 24 lines · Stephen Warren
From: Stephen Warren <[email protected]>
Migrate all most tests from
command_ut.cinto the Python test system. This allows the tests to be run against any U-Boot binary that supports the if command (i.e. where hush is enabled) without requiring that binary to be permanently bloated with the code fromcommand_ut.Some tests in
command_ut.ccan only be executed from C code, since they test internal (more unit-level) features of various U-Boot APIs. The migrated tests can all operate directly from the U-Boot console.Signed-off-by Stephen Warren <[email protected]>Acked-by Simon Glass <[email protected]>---
v4:
- No changes.v3:
- Move test scripts into a sub-directory. Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Remove inclusion of <os.h>. Suggested by Simon Glass.
Quoted 5 lines
--- test/command_ut.c | 116 ---------------------------- test/py/tests/test_hush_if_test.py | 154 +++++++++++++++++++++++++++++++++++++ 2 files changed, 154 insertions(+), 116 deletions(-) create mode 100644 test/py/tests/test_hush_if_test.py
Applied to u-boot-dm, thanks!
PATCH v4 7/8 test/py: test the ums command
In reply to Stephen WarrenFrom: Stephen Warren <[email protected]>
This test invokes the "ums" command in U-Boot, and validates that a USB storage device is enumerated on the test host system, and can be read from.
---
v4:
- No changes.
v3:
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Rename boardenv USB device node variable to avoid naming conflicts if we add support for other types of USB device later (e.g. CDC ACM, CDC Ethernet, DFU).
--- test/py/tests/test_ums.py | 94 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 test/py/tests/test_ums.py
test/py/tests/test_ums.py new file +94 -0
@@ -0,0 +1,94 @@ +# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +# Test U-Boot's "ums" command. At present, this test only ensures that a UMS +# device can be enumerated by the host/test machine. In the future, this test +# should be enhanced to validate disk IO. + +import os +import pytest +import time + +''' +Note: This test relies on: + +a) boardenv_* to contain configuration values to define which USB ports are +available for testing. Without this, this test will be automatically skipped. +For example: + +env__usb_dev_ports = ( + {'tgt_usb_ctlr': '0', 'host_ums_dev_node': '/dev/disk/by-path/pci-0000:00:14.0-usb-0:13:1.0-scsi-0:0:0:0'}, +) + +env__block_devs = ( + {'type': 'mmc', 'id': '0'}, # eMMC; always present + {'type': 'mmc', 'id': '1'}, # SD card; present since I plugged one in +) + +b) udev rules to set permissions on devices nodes, so that sudo is not +required. For example: + +ACTION=="add", SUBSYSTEM=="block", SUBSYSTEMS=="usb", KERNELS=="3-13", MODE:="666" + +(You may wish to change the group ID instead of setting the permissions wide +open. All that matters is that the user ID running the test can access the +device.) +''' + +def open_ums_device(host_ums_dev_node): + '''Attempt to open a device node, returning either the opened file handle, + or None on any error.''' + + try: + return open(host_ums_dev_node, 'rb') + except: + return None + +def wait_for_ums_device(host_ums_dev_node): + '''Continually attempt to open the device node exported by the "ums" + command, and either return the opened file handle, or raise an exception + after a timeout.''' + + for i in xrange(100): + fh = open_ums_device(host_ums_dev_node) + if fh: + return fh + time.sleep(0.1) + raise Exception('UMS device did not appear') + +def wait_for_ums_device_gone(host_ums_dev_node): + '''Continually attempt to open the device node exported by the "ums" + command, and either return once the device has disappeared, or raise an + exception if it does not before a timeout occurs.''' + + for i in xrange(100): + fh = open_ums_device(host_ums_dev_node) + if not fh: + return + fh.close() + time.sleep(0.1) + raise Exception('UMS device did not disappear') + + at pytest.mark.buildconfigspec('cmd_usb_mass_storage') +def test_ums(u_boot_console, env__usb_dev_port, env__block_devs): + '''Test the "ums" command; the host system must be able to enumerate a UMS + device when "ums" is running, and this device must disappear when "ums" is + aborted.''' + + tgt_usb_ctlr = env__usb_dev_port['tgt_usb_ctlr'] + host_ums_dev_node = env__usb_dev_port['host_ums_dev_node'] + + # We're interested in testing USB device mode on each port, not the cross- + # product of that with each device. So, just pick the first entry in the + # device list here. We'll test each block device somewhere else. + tgt_dev_type = env__block_devs[0]['type'] + tgt_dev_id = env__block_devs[0]['id'] + + cmd = 'ums %s %s %s' % (tgt_usb_ctlr, tgt_dev_type, tgt_dev_id) + u_boot_console.run_command('ums 0 mmc 0', wait_for_prompt=False) + fh = wait_for_ums_device(host_ums_dev_node) + fh.read(4096) + fh.close() + u_boot_console.ctrlc() + wait_for_ums_device_gone(host_ums_dev_node)
Signature · 1 line
2.7.0
PATCH v4 7/8 test/py: test the ums command
Quoted 21 lines · Stephen Warren
From: Stephen Warren <[email protected]>
This test invokes the "ums" command in U-Boot, and validates that a USB storage device is enumerated on the test host system, and can be read from.
Signed-off-by Stephen Warren <[email protected]>Acked-by Simon Glass <[email protected]>---
v4:
- No changes.v3:
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Rename boardenv USB device node variable to avoid naming conflicts if we add support for other types of USB device later (e.g. CDC ACM, CDC Ethernet, DFU).
Quoted 4 lines
--- test/py/tests/test_ums.py | 94 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 test/py/tests/test_ums.py
Applied to u-boot-dm, thanks!
PATCH v4 7/8 test/py: test the ums command
In reply to Stephen WarrenHi Stephen,
Quoted 125 lines
From: Stephen Warren <[email protected]>
This test invokes the "ums" command in U-Boot, and validates that a USB storage device is enumerated on the test host system, and can be read from.
Signed-off-by Stephen Warren <[email protected]>Acked-by Simon Glass <[email protected]>---
v4:
- No changes.v3:
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Add more documentation. Suggested by Simon Glass.
- Rename boardenv USB device node variable to avoid naming conflicts if we add support for other types of USB device later (e.g. CDC ACM, CDC Ethernet, DFU).--- test/py/tests/test_ums.py | 94+++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644
test/py/tests/test_ums.pytest/py/tests/test_ums.py new file +5
-0@@ -0,0 +1,94 @@ +# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +# Test U-Boot's "ums" command. At present, this test only ensuresthat a UMS +# device can be enumerated by the host/test machine. In the future, this test +# should be enhanced to validate disk IO.
test/py/tests/test_ums.py new file +9
-0+ +import os +import pytest +import time + +''' +Note: This test relies on: + +a) boardenv_* to contain configuration values to define which USBports are +available for testing. Without this, this test will be automatically skipped. +For example:
test/py/tests/test_ums.py new file +3
-0+ +env__usb_dev_ports = ( + {'tgt_usb_ctlr': '0', 'host_ums_dev_node':'/dev/disk/by-path/pci-0000:00:14.0-usb-0:13:1.0-scsi-0:0:0:0'}, +)
test/py/tests/test_ums.py new file +4
-0+ +env__block_devs = ( + {'type': 'mmc', 'id': '0'}, # eMMC; always present + {'type': 'mmc', 'id': '1'}, # SD card; present since I pluggedone in +)
test/py/tests/test_ums.py new file +2
-0+ +b) udev rules to set permissions on devices nodes, so that sudo isnot +required. For example:
test/py/tests/test_ums.py new file +2
-0+ +ACTION=="add", SUBSYSTEM=="block", SUBSYSTEMS=="usb",KERNELS=="3-13", MODE:="666" +
+(You may wish to change the group ID instead of setting the permissions wide +open. All that matters is that the user ID running the test can access the +device.)
+'''test/py/tests/test_ums.py new file +3
-0+ +def open_ums_device(host_ums_dev_node): + '''Attempt to open a device node, returning either the openedfile handle,
test/py/tests/test_ums.py new file +9
-0+ or None on any error.''' + + try: + return open(host_ums_dev_node, 'rb') + except: + return None + +def wait_for_ums_device(host_ums_dev_node): + '''Continually attempt to open the device node exported by the"ums"
+ command, and either return the opened file handle, or raise an
exceptiontest/py/tests/test_ums.py new file +11
-0+ after a timeout.''' + + for i in xrange(100): + fh = open_ums_device(host_ums_dev_node) + if fh: + return fh + time.sleep(0.1) + raise Exception('UMS device did not appear') + +def wait_for_ums_device_gone(host_ums_dev_node): + '''Continually attempt to open the device node exported by the"ums"
+ command, and either return once the device has disappeared, or
raise antest/py/tests/test_ums.py new file +13
-0+ exception if it does not before a timeout occurs.''' + + for i in xrange(100): + fh = open_ums_device(host_ums_dev_node) + if not fh: + return + fh.close() + time.sleep(0.1) + raise Exception('UMS device did not disappear') + + at pytest.mark.buildconfigspec('cmd_usb_mass_storage') +def test_ums(u_boot_console, env__usb_dev_port, env__block_devs): + '''Test the "ums" command; the host system must be able toenumerate a UMS
+ device when "ums" is running, and this device must disappear
when "ums" istest/py/tests/test_ums.py new file +6
-0+ aborted.''' + + tgt_usb_ctlr = env__usb_dev_port['tgt_usb_ctlr'] + host_ums_dev_node = env__usb_dev_port['host_ums_dev_node'] + + # We're interested in testing USB device mode on each port, notthe cross-
+ # product of that with each device. So, just pick the first
entry in thetest/py/tests/test_ums.py new file +1
-0+ # device list here. We'll test each block device somewhere else.
test/py/tests/test_ums.py line 94
Quoted 10 lines
test/py/tests/test_ums.py new file +10
-0+ tgt_dev_type = env__block_devs[0]['type'] + tgt_dev_id = env__block_devs[0]['id'] + + cmd = 'ums %s %s %s' % (tgt_usb_ctlr, tgt_dev_type, tgt_dev_id) + u_boot_console.run_command('ums 0 mmc 0', wait_for_prompt=False) + fh = wait_for_ums_device(host_ums_dev_node) + fh.read(4096) + fh.close() + u_boot_console.ctrlc() + wait_for_ums_device_gone(host_ums_dev_node)
Signature · 5 lines
Best regards, Lukasz Majewski Samsung R&D Institute Poland (SRPOL) | Linux Platform Group
PATCH v4 8/8 test/py: add a test for the sleep command
In reply to Stephen WarrenFrom: Stephen Warren <[email protected]>
Execute "sleep", and validate that it sleeps for approximately the correct amount of time.
---
v4: New patch.
Note that this test won't pass for sandbox until the following patches is applied: timer: sandbox: work without device tree
--- test/py/tests/test_sleep.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 test/py/tests/test_sleep.py
test/py/tests/test_sleep.py new file +24 -0
@@ -0,0 +1,24 @@ +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +import pytest +import time + +def test_sleep(u_boot_console): + '''Test the sleep command, and validate that it sleeps for approximately + the correct amount of time.''' + + # Do this before we time anything, to make sure U-Boot is already running. + # Otherwise, the system boot time is included in the time measurement. + u_boot_console.ensure_spawned() + + # 3s isn't too long, but is enough to cross a few second boundaries. + sleep_time = 3 + tstart = time.time() + u_boot_console.run_command('sleep %d' % sleep_time) + tend = time.time() + elapsed = tend - tstart + delta_to_expected = abs(elapsed - sleep_time) + # 0.25s margin is hopefully enough to account for any system overhead. + assert delta_to_expected < 0.25
Signature · 1 line
2.7.0
PATCH v4 8/8 test/py: add a test for the sleep command
Quoted 10 lines · Stephen Warren
From: Stephen Warren <[email protected]>
Execute "sleep", and validate that it sleeps for approximately the correct amount of time.
Signed-off-by Stephen Warren <[email protected]>Acked-by Simon Glass <[email protected]>---
v4: New patch.
Quoted 6 lines
Note that this test won't pass for sandbox until the following patches is applied: timer: sandbox: work without device tree
--- test/py/tests/test_sleep.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 test/py/tests/test_sleep.py
Applied to u-boot-dm, thanks!
PATCH v4 1/8 test/py: Implement pytest infrastructure
In reply to Stephen WarrenQuoted 83 lines · Stephen Warren
This tool aims to test U-Boot by executing U-Boot shell commands using the console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:
- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself. It is asserted that writing test-related code in Python is simpler and more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and
test/cmd_ut.ctoo.The hook scripts, relay control utilities, and udev rules I use for my own HW setup are published at https://github.com/swarren/uboot-test-hooks.
See README.md for more details!
Signed-off-by Stephen Warren <[email protected]>Signed-off-by Stephen Warren <[email protected]>Tested-by Michal Simek <[email protected]>Tested-by Simon Glass <[email protected]>Acked-by Simon Glass <[email protected]> #v3---
Apologies for the resend; I forgot to CC the list last time.v4:
- Fix all users after ubspawn ->u_boot_spawnmodule rename.
- Don't escape ^ in the HTML log file.
- Fixu_boot_spawn.pySpawn.__init__ to set SIGHUP handler is set toSIG_DFL. This ensures that sandbox responds to SIGHUP when run non- interactively (e.g. under a daemon continuous integration server).
- Fix logging of all child processes to make sure that all stdout/err, exit code, and exception data from all processes makes its way to both test stdout (for pytest to log to its stdout in case of error) and the HTML test log. This changedmultiplexed_log.pyRunAndLog.run()(a fair bit of change) andu_boot_console_exec_attach.pyConseolExecAttach.* (just passing sys.stdout tolog.get_runner()everywhere).
- Also mention kermit as a possible app to attach to a serial console.
- Fixtest_versionto work correctly whenBUILD_TAGis set, which causes U-Boot to print both the version string andBUILD_TAGvalue at startup, whereas the "version" shell command only prints the version string.
- Reformat function-level documentation.v3:
- Rework HTML log generation so that TAB characters render as expected. Suggested by Michal Simek.
- Move test scripts into a sub-directory.
Suggested by Michal Simek.
- s/uboot/u[-_]boot/g. Suggested by Simon Glass.
- s/"/'/g. Suggested by Simon Glass.
- Typo fixes.
- Add more documentation. Suggested by Simon Glass.
- Make "notes" in the log file be <pre> so that their formatting is preserved. This is useful for large notes such as exception dumps.v2:
- Many fixes and tweaks have been squashed in. Separated out some of the tests into separate commits, and added some more tests.--- test/py/.gitignore | 1 + test/py/README.md | 300 ++++++++++++++++++++ test/py/conftest.py | 422 ++++++++++++++++++++++++++++ test/py/multiplexed_log.css | 88 ++++++ test/py/multiplexed_log.py | 515 ++++++++++++++++++++++++++++++++++ test/py/pytest.ini | 11 + test/py/test.py | 32 +++ test/py/tests/test_000_version.py | 20 ++ test/py/tests/test_help.py | 9 + test/py/tests/test_unknown_cmd.py | 14 + test/py/u_boot_console_base.py | 360 ++++++++++++++++++++++++ test/py/u_boot_console_exec_attach.py | 65 +++++ test/py/u_boot_console_sandbox.py | 79 ++++++ test/py/u_boot_spawn.py | 174 ++++++++++++ 14 files changed, 2090 insertions(+) create mode 100644 test/py/.gitignore create mode 100644 test/py/README.md create mode 100644 test/py/conftest.py create mode 100644 test/py/multiplexed_log.css create mode 100644 test/py/multiplexed_log.py create mode 100644 test/py/pytest.ini
Quoted 8 lines
create mode 100755 test/py/test.py create mode 100644 test/py/tests/test_000_version.py create mode 100644 test/py/tests/test_help.py create mode 100644 test/py/tests/test_unknown_cmd.py create mode 100644 test/py/u_boot_console_base.py create mode 100644 test/py/u_boot_console_exec_attach.py create mode 100644 test/py/u_boot_console_sandbox.py create mode 100644 test/py/u_boot_spawn.py
Applied to u-boot-dm, thanks!