test/py: Convert vboot tests to use the test/py framework

Add a few notes about how testing works in U-Boot.

Signed-off-by Simon Glass <[email protected]>

---

 test/README | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)
 create mode 100644 test/README

test/README new file +11 -0

new file mode 100644
index 0000000..dfd83d6
@@ -0,0 +1,82 @@
+Testing in U-Boot
+=================
+
+U-Boot has a large amount of code. This file describes how this code is
+tested and what tests you should write when adding a new feature.
+
+
+Sandbox
+-------
+U-Boot can be built as a user-space application (e.g. for Linux). This
+allows test to be executed without needing target hardware. The 'sandbox'

test/README new file +16 -0

+
+You can run the tests on sandbox with:
+
+       ./test/py/test.py --bd sandbox --build
+
+This will produce HTML output in build-sandbox/test-log.html
+
+See test/py/README.md for more information about the pytest suite.
+
+
+tbot
+----
+
+Tbot provides a way to execute tests on target hardware. It is intended for
+trying out both U-Boot and Linux (and potentially other software) on a
+number of boards automatically. It can be used to create a continuous test

test/README new file +30 -1

+
+
+When to write tests
+-------------------
+
+If you add code to U-Boot without a test you are taking a risk. Even if you
+perform thorough manual testing at the time of submission, it may break when
+future changes are made to U-Boot. It may even break when applied to mainline,
+if other changes interact with it. A good mindset is that untested code
+probably doesn't work and should be deleted.
+
+You can assume that the Pytest suite will be run before patches are accepted
+to mainline, so this provides protection against future breakage.
+
+On the other hand there is quite a bit of code that is not covered with tests,
+or is covered sparingly. So here are some suggestions:
+
+- If you are adding a new uclass, add a sandbox driver and a test that uses it
+- If you are modifying code covered by an existing test, add a new test case
+  to cover your changes
+- If the code you are modifying has not tests, consider writing one. Even a
+  very basic test is useful, and may be picked up and enhanced by others. It
+  is much easier to add onto a test - writing a new large test can seem
+  daunting to most contributors.
+
+
+Future work
+-----------
+
+Converting existing shell scripts into pytest tests.
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Hi Simon,

Quoted 3 lines · Simon Glass

Add a few notes about how testing works in U-Boot.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

Quoted 24 lines

---

 test/README | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)
 create mode 100644 test/README

test/README new file +11 -0

new file mode 100644
index 0000000..dfd83d6
@@ -0,0 +1,82 @@
+Testing in U-Boot
+=================
+
+U-Boot has a large amount of code. This file describes how this code is
+tested and what tests you should write when adding a new feature.
+
+
+Sandbox
+-------
+U-Boot can be built as a user-space application (e.g. for Linux). This
+allows test to be executed without needing target hardware. The 'sandbox'

test/README line 21

Quoted 10 lines

test/README new file +10 -0

+target provides this feature and it is widely used in tests.
+
+
+Pytest Suite
+------------
+
+Many tests are available using the pytest suite, in test/py. This can run
+either on sandbox or on real hardware. It relies on the U-Boot console to
+inject test commands and check the result. It is slower to run than C code,
+but provides the ability to unify lots of test and summarise their results.

lots of tests

Quoted 16 lines

test/README new file +16 -0

+
+You can run the tests on sandbox with:
+
+       ./test/py/test.py --bd sandbox --build
+
+This will produce HTML output in build-sandbox/test-log.html
+
+See test/py/README.md for more information about the pytest suite.
+
+
+tbot
+----
+
+Tbot provides a way to execute tests on target hardware. It is intended for
+trying out both U-Boot and Linux (and potentially other software) on a
+number of boards automatically. It can be used to create a continuous test

test/README line 47

Quoted 10 lines

test/README new file +10 -0

+environment. See tools/tbot/README for more information.
+
+
+Ad-hoc tests
+------------
+
+There are several ad-hoc tests which run outside the pytest environment:
+
+   test/fs     - File system test (shell script)
+   test/image  - FIT and lagacy image tests (shell script and Python)

s/lagacy/legacy/

test/README line 52

Quoted 5 lines

test/README new file +5 -0

+   test/stdint - A test that stdint.h can be used in U-Boot (shell script)
+   trace       - Test for the tracing feature (shell script)
+   vboot       - Test for verified boot (shell script)
+
+The above should be converted to run as part of the pytest suite.

Is this a NOTE or a TODO?

Quoted 34 lines

test/README new file +30 -1

+
+
+When to write tests
+-------------------
+
+If you add code to U-Boot without a test you are taking a risk. Even if you
+perform thorough manual testing at the time of submission, it may break when
+future changes are made to U-Boot. It may even break when applied to mainline,
+if other changes interact with it. A good mindset is that untested code
+probably doesn't work and should be deleted.
+
+You can assume that the Pytest suite will be run before patches are accepted
+to mainline, so this provides protection against future breakage.
+
+On the other hand there is quite a bit of code that is not covered with tests,
+or is covered sparingly. So here are some suggestions:
+
+- If you are adding a new uclass, add a sandbox driver and a test that uses it
+- If you are modifying code covered by an existing test, add a new test case
+  to cover your changes
+- If the code you are modifying has not tests, consider writing one. Even a
+  very basic test is useful, and may be picked up and enhanced by others. It
+  is much easier to add onto a test - writing a new large test can seem
+  daunting to most contributors.
+
+
+Future work
+-----------
+
+Converting existing shell scripts into pytest tests.
--

2.8.0.rc3.226.g39d4020

Quoted 4 lines

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Thank for these docs! I wonder if a clone/mirror of the repo on Github can be set up to run the sandbox/Pytest tests in TravisCI with minimum effort? :)

Add a few notes about how testing works in U-Boot.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

A common check before sending patches is to run all available tests on sandbox. But everytime I do this I have to look up the README. This presents quite a barrier to actually doing this.

Add a shell script to help. To run the tests, type:

test/run

in the U-Boot directory, which should be easy to remember.

Signed-off-by Simon Glass <[email protected]>

---

 test/README | 11 +++++++++++
 test/run    |  4 ++++
 2 files changed, 15 insertions(+)
 create mode 100755 test/run

test/README +0 -0

index dfd83d6..6cfee05 100644

test/README +2 -0

+
+
 Sandbox
 -------
 U-Boot can be built as a user-space application (e.g. for Linux). This

test/run new file +4 -1

new file mode 100755
index 0000000..a6dcf8f
@@ -0,0 +1,4 @@
+#!/bin/sh
+
+# Run all tests
+./test/py/test.py --bd sandbox --build
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

A common check before sending patches is to run all available tests on sandbox. But everytime I do this I have to look up the README. This presents quite a barrier to actually doing this.

Add a shell script to help. To run the tests, type:

test/run

in the U-Boot directory, which should be easy to remember.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

At present sandbox exits when the 'bootm' command completes, since it is not actually able to run the OS that is loaded. Normally 'bootm' failure is considered a fatal error in U-Boot.

However this is annoying for tests, which may want to examine the state after a test is complete. In any case there is a 'reset' command which can be used to exit, if required.

Change the behaviour to return normally from the 'bootm' command on sandbox.

Signed-off-by Simon Glass <[email protected]>

---

 arch/sandbox/lib/bootm.c | 2 +-
 common/bootm_os.c        | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

arch/sandbox/lib/bootm.c +1 -1

index d49c927..0c9a797 100644
@@ -56,7 +56,7 @@ int do_bootm_linux(int flag, int argc, char *argv[], bootm_headers_t *images)
                bootstage_mark(BOOTSTAGE_ID_RUN_OS);
                printf("## Transferring control to Linux (at address %08lx)...\n",
                       images->ep);
-               reset_cpu(0);
+               printf("sandbox: continuing, as we cannot run Linux\n");
        }
 
        return 0;

common/bootm_os.c +1 -1

index 9ec84bd..e3f5a46 100644
@@ -481,6 +481,7 @@ int boot_selected_os(int argc, char * const argv[], int state,
 
        /* Stand-alone may return when 'autostart' is 'no' */
        if (images->os.type == IH_TYPE_STANDALONE ||
+           IS_ENABLED(CONFIG_SANDBOX) ||
            state == BOOTM_STATE_OS_FAKE_GO) /* We expect to return */
                return 0;
        bootstage_error(BOOTSTAGE_ID_BOOT_OS_RETURNED);
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

At present sandbox exits when the 'bootm' command completes, since it is not actually able to run the OS that is loaded. Normally 'bootm' failure is considered a fatal error in U-Boot.

However this is annoying for tests, which may want to examine the state after a test is complete. In any case there is a 'reset' command which can be used to exit, if required.

Change the behaviour to return normally from the 'bootm' command on sandbox.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

Normally tests will run with the test.dtb file designed for this purpose. However, the verified boot tests need to run with their own device-tree file, containing a public key.

Make the device-tree file a config option so that it can be adjusted by tests. The default is to keep the current behaviour.

Signed-off-by Simon Glass <[email protected]>

---

 test/py/conftest.py               | 1 +
 test/py/u_boot_console_sandbox.py | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

test/py/conftest.py +1 -0

index 449f98b..5b16456 100644
@@ -179,6 +179,7 @@ def pytest_configure(config):
     ubconfig.board_type = board_type
     ubconfig.board_identity = board_identity
     ubconfig.gdbserver = gdbserver
+    ubconfig.dtb = build_dir + '/arch/sandbox/dts/test.dtb'
 
     env_vars = (
         'board_type',

test/py/u_boot_console_sandbox.py +1 -2

index 04654ae..b440497 100644
@@ -46,7 +46,7 @@ class ConsoleSandbox(ConsoleBase):
             self.config.build_dir + '/u-boot',
             '-v',
             '-d',
-            self.config.build_dir + '/arch/sandbox/dts/test.dtb'
+            self.config.dtb
         ]
         return Spawn(cmd, cwd=self.config.source_dir)
 
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Normally tests will run with the test.dtb file designed for this purpose. However, the verified boot tests need to run with their own device-tree file, containing a public key.

Make the device-tree file a config option so that it can be adjusted by tests. The default is to keep the current behaviour.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

Tests may want to look at the output from running a command. Return it so that this is possible.

Signed-off-by Simon Glass <[email protected]>

---

 test/py/multiplexed_log.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

test/py/multiplexed_log.py +2 -2

index 68917eb..02c44df 100644
@@ -119,7 +119,7 @@ class RunAndLog(object):
                 raised if such problems occur.
 
         Returns:
-            Nothing.
+            The output as a string.
         """
 
         msg = '+' + ' '.join(cmd) + '\n'

@@ -161,6 +161,7 @@ class RunAndLog(object):
             self.chained_file.write(output)
         if exception:
             raise exception
+        return output
 
 class SectionCtxMgr(object):
     """A context manager for Python's "with" statement, which allows a certain
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Tests may want to look at the output from running a command. Return it so that this is possible.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

Tests may want to look at the output from running a command, even if it fails (e.g. with a non-zero return code). Provide a means to obtain this.

Another approach would be to return a class object containing both the output and the exception, but I'm not sure if that would result in a lot of refactoring.

Signed-off-by Simon Glass <[email protected]>

---

 test/py/multiplexed_log.py | 7 +++++++
 1 file changed, 7 insertions(+)

test/py/multiplexed_log.py +7 -1

index 02c44df..35a32fb 100644
@@ -101,6 +101,7 @@ class RunAndLog(object):
         self.logfile = logfile
         self.name = name
         self.chained_file = chained_file
+        self.output = None
 
     def close(self):
         """Clean up any resources managed by this object."""

@@ -109,6 +110,9 @@ class RunAndLog(object):
     def run(self, cmd, cwd=None, ignore_errors=False):
         """Run a command as a sub-process, and log the results.
 
+        The output is available at self.output which can be useful if there is
+        an exception.
+
         Args:
             cmd: The command to execute.
             cwd: The directory to run the command in. Can be None to use the

@@ -159,6 +163,9 @@ class RunAndLog(object):
         self.logfile.write(self, output)
         if self.chained_file:
             self.chained_file.write(output)
+
+        # Store the output so it can be accessed if we raise an exception.
+        self.output = output
         if exception:
             raise exception
         return output
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Tests may want to look at the output from running a command, even if it fails (e.g. with a non-zero return code). Provide a means to obtain this.

Another approach would be to return a class object containing both the output and the exception, but I'm not sure if that would result in a lot of refactoring.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

It is useful to be able to obtain the output from a command. Return it from this function.

Signed-off-by Simon Glass <[email protected]>

---

 test/py/u_boot_utils.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

test/py/u_boot_utils.py +3 -3

index 6a6b2ec..5dc0f71 100644
@@ -165,12 +165,13 @@ def run_and_log(u_boot_console, cmd, ignore_errors=False):
             problems occur.
 
     Returns:
-        Nothing.
+        The output as a string.
     """
 
     runner = u_boot_console.log.get_runner(cmd[0], sys.stdout)
-    runner.run(cmd, ignore_errors=ignore_errors)
+    output = runner.run(cmd, ignore_errors=ignore_errors)
     runner.close()
+    return output
 
 ram_base = None
 def find_ram_base(u_boot_console):
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

It is useful to be able to obtain the output from a command. Return it from this function.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

It is sometimes inconvenient to convert a string into a list for execution with run_and_log(). Provide a helper function to do this.

Signed-off-by Simon Glass <[email protected]>

---

 test/py/u_boot_utils.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

test/py/u_boot_utils.py +0 -0

index 5dc0f71..5d638d9 100644

test/py/u_boot_utils.py +1 -1

+
 ram_base = None
 def find_ram_base(u_boot_console):
     """Find the running U-Boot's RAM location.
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

It is sometimes inconvenient to convert a string into a list for execution with run_and_log(). Provide a helper function to do this.

test/py/u_boot_utils.py +0 -0

It is sometimes inconvenient to convert a string into a list for execution with run_and_log(). Provide a helper function to do this.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

Sometimes we want to run a command and check that it fails. Add a function to handle this. It can check the return code and also make sure that the output contains a given error message.

Signed-off-by Simon Glass <[email protected]>

---

 test/py/u_boot_utils.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

test/py/u_boot_utils.py +0 -0

index 5d638d9..c834e7b 100644

test/py/u_boot_utils.py +3 -1

+    finally:
+        runner.close()
+
 ram_base = None
 def find_ram_base(u_boot_console):
     """Find the running U-Boot's RAM location.
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Sometimes we want to run a command and check that it fails. Add a function to handle this. It can check the return code and also make sure that the output contains a given error message.

test/py/u_boot_utils.py +2 -0

 
+def run_and_log_expect_exception(u_boot_console, cmd, retcode, msg):
+    """Run a command which is expected to fail.

Sometimes we want to run a command and check that it fails. Add a function to handle this. It can check the return code and also make sure that the output contains a given error message.

Signed-off-by Simon Glass <[email protected]>

Some tests want to execute a sequence of commands. Add a helper for this.

Signed-off-by Simon Glass <[email protected]>

---

 test/py/u_boot_console_base.py | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

test/py/u_boot_console_base.py +0 -0

index 815fa64..b5aad7c 100644
@@ -216,6 +216,22 @@ class ConsoleBase(object):
             self.cleanup_spawn()

test/py/u_boot_console_base.py +2 -1

+        return output
+
     def ctrlc(self):
         """Send a CTRL-C character to U-Boot.
 
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Some tests want to execute a sequence of commands. Add a helper for this.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

Some tests want to execute a sequence of commands. Add a helper for this.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

The error code may provide useful information for debugging. Add it to the error string.

Signed-off-by Simon Glass <[email protected]>

---

 tools/fit_image.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

tools/fit_image.c +2 -3

index 58aa8e2..f471982 100644
@@ -650,8 +650,8 @@ static int fit_handle_file(struct image_tool_params *params)
        }
 
        if (ret) {
-               fprintf(stderr, "%s Can't add hashes to FIT blob\n",
-                       params->cmdname);
+               fprintf(stderr, "%s Can't add hashes to FIT blob: %d\n",
+                       params->cmdname, ret);
                goto err_system;
        }
 
--

2.8.0.rc3.226.g39d4020

The error code may provide useful information for debugging. Add it to the error string.

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

We should not be returning -1 as an error code. This can mask a situation where we run out of space adding things to the FIT. By returning the correct error in this case (-ENOSPC) it can be handled by the higher-level code.

This may fix the error reported by Tom Van Deun here:

https://www.mail-archive.com/u-boot at lists.denx.de/msg217417.html

although I am not sure as I cannot actually repeat it.

Signed-off-by Simon Glass <[email protected]>

Reported-by Tom Van Deun <[email protected]>

---

 tools/image-host.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

tools/image-host.c +8 -7

index 7effb6c..3e14fdc 100644
@@ -38,7 +38,7 @@ static int fit_set_hash_value(void *fit, int noffset, uint8_t *value,
                printf("Can't set hash '%s' property for '%s' node(%s)\n",
                       FIT_VALUE_PROP, fit_get_name(fit, noffset, NULL),
                       fdt_strerror(ret));
-               return -1;
+               return ret == -FDT_ERR_NOSPACE ? -ENOSPC : -EIO;
        }
 
        return 0;

@@ -64,25 +64,27 @@ static int fit_image_process_hash(void *fit, const char *image_name,
        const char *node_name;
        int value_len;
        char *algo;
+       int ret;
 
        node_name = fit_get_name(fit, noffset, NULL);
 
        if (fit_image_hash_get_algo(fit, noffset, &algo)) {
                printf("Can't get hash algo property for '%s' hash node in '%s' image node\n",
                       node_name, image_name);
-               return -1;
+               return -ENOENT;
        }
 
        if (calculate_hash(data, size, algo, value, &value_len)) {
                printf("Unsupported hash algorithm (%s) for '%s' hash node in '%s' image node\n",
                       algo, node_name, image_name);
-               return -1;
+               return -EPROTONOSUPPORT;
        }
 
-       if (fit_set_hash_value(fit, noffset, value, value_len)) {
+       ret = fit_set_hash_value(fit, noffset, value, value_len);
+       if (ret) {
                printf("Can't set hash value for '%s' hash node in '%s' image node\n",
                       node_name, image_name);
-               return -1;
+               return ret;
        }
 
        return 0;

@@ -322,7 +324,7 @@ int fit_image_add_verification_data(const char *keydir, void *keydest,
                                comment, require_keys);
                }
                if (ret)
-                       return -1;
+                       return ret;
        }
 
        return 0;
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

We should not be returning -1 as an error code. This can mask a situation where we run out of space adding things to the FIT. By returning the correct error in this case (-ENOSPC) it can be handled by the higher-level code.

This may fix the error reported by Tom Van Deun here:

https://www.mail-archive.com/u-boot at lists.denx.de/msg217417.html

although I am not sure as I cannot actually repeat it.

Signed-off-by Simon Glass <[email protected]>

Reported-by Tom Van Deun <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

At present all the hush tests are skipped on sandbox because the test thinks that this option is disabled. In fact it has just been renamed.

It might be better to use the full CONFIG_xxx name in tests with @pytest.mark.buildconfigspec(), since at present it is not really clear that the options are related.

Fixes f1f9d4fa (hush: complete renaming CONFIG_SYS_HUSH_PARSER to CONFIG_HUSH_PARSER)

Signed-off-by Simon Glass <[email protected]>

---

 test/py/tests/test_hush_if_test.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

test/py/tests/test_hush_if_test.py +4 -5

index 1eeaa5b..b572538 100644
@@ -109,27 +109,27 @@ def exec_hush_if(u_boot_console, expr, result):
     response = u_boot_console.run_command(cmd)
     assert response.strip() == str(result).lower()
 
- at pytest.mark.buildconfigspec('sys_hush_parser')
+ at pytest.mark.buildconfigspec('hush_parser')
 def test_hush_if_test_setup(u_boot_console):
     """Set up environment variables used during the "if" tests."""
 
     u_boot_console.run_command('setenv ut_var_nonexistent')
     u_boot_console.run_command('setenv ut_var_exists 1')
 
- at pytest.mark.buildconfigspec('sys_hush_parser')
+ at pytest.mark.buildconfigspec('hush_parser')
 @pytest.mark.parametrize('expr,result', subtests)
 def test_hush_if_test(u_boot_console, expr, result):
     """Test a single "if test" condition."""
 
     exec_hush_if(u_boot_console, expr, result)
 
- at pytest.mark.buildconfigspec('sys_hush_parser')
+ at pytest.mark.buildconfigspec('hush_parser')
 def test_hush_if_test_teardown(u_boot_console):
     """Clean up environment variables used during the "if" tests."""
 
     u_boot_console.run_command('setenv ut_var_exists')
 
- at pytest.mark.buildconfigspec('sys_hush_parser')
+ at pytest.mark.buildconfigspec('hush_parser')
 # We might test this on real filesystems via UMS, DFU, 'save', etc.
 # Of those, only UMS currently allows file removal though.
 @pytest.mark.boardspec('sandbox')
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

At present all the hush tests are skipped on sandbox because the test thinks that this option is disabled. In fact it has just been renamed.

It might be better to use the full CONFIG_xxx name in tests with @pytest.mark.buildconfigspec(), since at present it is not really clear that the options are related.

Fixes f1f9d4fa (hush: complete renaming CONFIG_SYS_HUSH_PARSER to CONFIG_HUSH_PARSER)

At present all the hush tests are skipped on sandbox because the test thinks that this option is disabled. In fact it has just been renamed.

It might be better to use the full CONFIG_xxx name in tests with @pytest.mark.buildconfigspec(), since at present it is not really clear that the options are related.

Fixes f1f9d4fa (hush: complete renaming CONFIG_SYS_HUSH_PARSER to CONFIG_HUSH_PARSER)

Signed-off-by Simon Glass <[email protected]>

Reviewed-by Teddy Reed <[email protected]>

Now that we have a suitable test framework we should move all tests into it. The vboot test is a suitable candidate. Rewrite it in Python and move the data files into an appropriate directory.

Signed-off-by Simon Glass <[email protected]>

---

 test/README                                       |   1 -
 test/py/tests/test_vboot.py                       | 185 ++++++++++++++++++++++
 test/{ => py/tests}/vboot/sandbox-kernel.dts      |   0
 test/{ => py/tests}/vboot/sandbox-u-boot.dts      |   0
 test/{ => py/tests}/vboot/sign-configs-sha1.its   |   0
 test/{ => py/tests}/vboot/sign-configs-sha256.its |   0
 test/{ => py/tests}/vboot/sign-images-sha1.its    |   0
 test/{ => py/tests}/vboot/sign-images-sha256.its  |   0
 test/vboot/.gitignore                             |   3 -
 test/vboot/vboot_test.sh                          | 151 ------------------
 10 files changed, 185 insertions(+), 155 deletions(-)
 create mode 100644 test/py/tests/test_vboot.py
 rename test/{ => py/tests}/vboot/sandbox-kernel.dts (100%)
 rename test/{ => py/tests}/vboot/sandbox-u-boot.dts (100%)
 rename test/{ => py/tests}/vboot/sign-configs-sha1.its (100%)
 rename test/{ => py/tests}/vboot/sign-configs-sha256.its (100%)
 rename test/{ => py/tests}/vboot/sign-images-sha1.its (100%)
 rename test/{ => py/tests}/vboot/sign-images-sha256.its (100%)
 delete mode 100644 test/vboot/.gitignore
 delete mode 100755 test/vboot/vboot_test.sh

test/README +0 -1

index 6cfee05..ee55972 100644
@@ -58,7 +58,6 @@ There are several ad-hoc tests which run outside the pytest environment:
    test/image  - FIT and lagacy image tests (shell script and Python)
    test/stdint - A test that stdint.h can be used in U-Boot (shell script)
    trace       - Test for the tracing feature (shell script)
-   vboot       - Test for verified boot (shell script)
 
 The above should be converted to run as part of the pytest suite.
 

test/py/tests/test_vboot.py new file +9 -0

new file mode 100644
index 0000000..c779895
@@ -0,0 +1,185 @@
+# Copyright (c) 2013, Google Inc.
+#
+# SPDX-License-Identifier:     GPL-2.0+
+#
+# U-Boot Verified Boot Test
+
+"""
+This tests verified boot in the following ways:
+

test/py/tests/test_vboot.py new file +14 -0

+- Sign the FIT and mark the key as 'required' for verification
+- Check that image verification works
+- Corrupt the signature
+- Check that image verification no-longer works
+
+Tests run with both SHA1 and SHA256 hashing.
+"""
+
+import pytest
+import sys
+import u_boot_utils as util
+
+ at pytest.mark.buildconfigspec('fit_signature')
+def test_vboot(u_boot_console):

test/py/tests/test_vboot.py new file +10 -0

+
+        The output file will be the same as the input file but with a .dtb
+        extension.
+
+        Args:
+            dts: Device tree file to compile.
+        """
+        dtb = dts.replace('.dts', '.dtb')
+        util.cmd(cons, 'dtc %s %s%s -O dtb '
+                       '-o %s%s' % (dtc_args, datadir, dts, tmpdir, dtb))

test/py/tests/test_vboot.py new file +7 -0

+        """
+        cons.cleanup_spawn()
+        cons.ensure_spawned()
+        cons.log.action('%s: Test Verified Boot Run: %s' % (algo, test_type))
+        output = cons.run_command_list(
+            ['sb load hostfs - 100 %stest.fit' % tmpdir,
+             'fdt addr 100',

test/py/tests/test_vboot.py new file +5 -0

+        """
+        util.run_and_log(cons, [mkimage, '-D', dtc_args, '-f',
+                                '%s%s' % (datadir, its), fit])
+
+    def sign_fit():

test/py/tests/test_vboot.py new file +47 -0

+
+        This is the main part of the test code. The same procedure is followed
+        for both hashing algorithms.
+
+        Args:
+            sha: Either 'sha1' or 'sha256', to select the algorithm to use
+        """
+        global algo
+
+        algo = sha
+
+        # Compile our device tree files for kernel and U-Boot
+        dtc('sandbox-kernel.dts')
+        dtc('sandbox-u-boot.dts')
+
+        # Build the FIT, but don't sign anything yet
+        cons.log.action('%s: Test FIT with signed images' % algo)
+        make_fit('sign-images-%s.its' % algo)
+        run_bootm('unsigned images', 'dev-')
+
+        # Sign images with our dev keys
+        sign_fit()
+        run_bootm('signed images', 'dev+')
+
+        # Create a fresh .dtb without the public keys
+        dtc('sandbox-u-boot.dts')
+
+        cons.log.action('%s: Test FIT with signed configuration' % algo)
+        make_fit('sign-configs-%s.its' % algo)
+        run_bootm('unsigned config', '%s+ OK' % algo)
+
+        # Sign images with our dev keys
+        sign_fit()
+        run_bootm('signed config', 'dev+')
+
+        cons.log.action('%s: Check signed config on the host' % algo)
+
+        util.run_and_log(cons, [fit_check_sign, '-f', fit, '-k', tmpdir,
+                                '-k', dtb])
+
+        # Increment the first byte of the signature, which should cause failure
+        sig = util.cmd(cons, 'fdtget -t bx %s %s value' % (fit, sig_node))
+        byte_list = sig.split()
+        byte = int(byte_list[0], 16)
+        byte_list = ['%x' % (byte + 1)] + byte_list[1:]
+        sig = ' '.join(byte_list)
+        util.cmd(cons, 'fdtput -t bx %s %s value %s' % (fit, sig_node, sig))

test/py/tests/test_vboot.py new file +6 -0

+
+    # Create an RSA key pair
+    public_exponent = 65537
+    util.cmd(cons, 'openssl genpkey -algorithm RSA -out %sdev.key '
+                   '-pkeyopt rsa_keygen_bits:2048 '
+                   '-pkeyopt rsa_keygen_pubexp:%d '

test/py/tests/test_vboot.py new file +9 -0

+    try:
+        # We need to use our own device tree file. Remember to restore it
+        # afterwards.
+        old_dtb = cons.config.dtb
+        cons.config.dtb = dtb
+        test_with_algo('sha1')
+        test_with_algo('sha256')
+    finally:
+        cons.config.dtb = old_dtb

test/py/tests/vboot/sandbox-kernel.dts renamed from test/vboot/sandbox-kernel.dts +0 -0

similarity index 100%
rename from test/vboot/sandbox-kernel.dts
rename to test/py/tests/vboot/sandbox-kernel.dts

test/py/tests/vboot/sandbox-u-boot.dts renamed from test/vboot/sandbox-u-boot.dts +0 -0

similarity index 100%
rename from test/vboot/sandbox-u-boot.dts
rename to test/py/tests/vboot/sandbox-u-boot.dts

test/py/tests/vboot/sign-configs-sha1.its renamed from test/vboot/sign-configs-sha1.its +0 -0

similarity index 100%
rename from test/vboot/sign-configs-sha1.its
rename to test/py/tests/vboot/sign-configs-sha1.its

test/py/tests/vboot/sign-configs-sha256.its renamed from test/vboot/sign-configs-sha256.its +0 -0

similarity index 100%
rename from test/vboot/sign-configs-sha256.its
rename to test/py/tests/vboot/sign-configs-sha256.its

test/py/tests/vboot/sign-images-sha1.its renamed from test/vboot/sign-images-sha1.its +0 -0

similarity index 100%
rename from test/vboot/sign-images-sha1.its
rename to test/py/tests/vboot/sign-images-sha1.its

test/py/tests/vboot/sign-images-sha256.its renamed from test/vboot/sign-images-sha256.its +0 -0

similarity index 100%
rename from test/vboot/sign-images-sha256.its
rename to test/py/tests/vboot/sign-images-sha256.its

test/vboot/.gitignore deleted +0 -3

deleted file mode 100644
index 4631242..0000000
@@ -1,3 +0,0 @@
-/*.dtb
-/test.fit
-/dev-keys

test/vboot/vboot_test.sh deleted +0 -152

deleted file mode 100755
index 6d7abb8..0000000
@@ -1,151 +0,0 @@
-#!/bin/bash
-#
-# Copyright (c) 2013, Google Inc.
-#
-# Simple Verified Boot Test Script
-#
-# SPDX-License-Identifier:     GPL-2.0+
-
-set -e
-
-# Run U-Boot and report the result
-# Args:
-#      $1:     Test message
-run_uboot() {
-       echo -n "Test Verified Boot Run: $1: "
-       ${uboot} -d sandbox-u-boot.dtb >${tmp} -c '
-sb load hostfs - 100 test.fit;
-fdt addr 100;
-bootm 100;
-reset'
-       if ! grep -q "$2" ${tmp}; then
-               echo
-               echo "Verified boot key check failed, output follows:"
-               cat ${tmp}
-               false
-       else
-               echo "OK"
-       fi
-}
-
-echo "Simple Verified Boot Test"
-echo "========================="
-echo
-echo "Please see doc/uImage.FIT/verified-boot.txt for more information"
-echo
-
-err=0
-tmp=/tmp/vboot_test.$$
-
-dir=$(dirname $0)
-
-if [ -z ${O} ]; then
-       O=.
-fi
-O=$(readlink -f ${O})
-
-dtc="-I dts -O dtb -p 2000"
-uboot="${O}/u-boot"
-mkimage="${O}/tools/mkimage"
-fit_check_sign="${O}/tools/fit_check_sign"
-keys="${dir}/dev-keys"
-echo ${mkimage} -D "${dtc}"
-
-echo "Build keys"
-mkdir -p ${keys}
-
-PUBLIC_EXPONENT=${1}
-
-if [ -z "${PUBLIC_EXPONENT}" ]; then
-       PUBLIC_EXPONENT=65537
-fi
-
-# Create an RSA key pair
-openssl genpkey -algorithm RSA -out ${keys}/dev.key \
-    -pkeyopt rsa_keygen_bits:2048 \
-    -pkeyopt rsa_keygen_pubexp:${PUBLIC_EXPONENT} 2>/dev/null
-
-# Create a certificate containing the public key
-openssl req -batch -new -x509 -key ${keys}/dev.key -out ${keys}/dev.crt
-
-pushd ${dir} >/dev/null
-
-function do_test {
-       echo do $sha test
-       # Compile our device tree files for kernel and U-Boot
-       dtc -p 0x1000 sandbox-kernel.dts -O dtb -o sandbox-kernel.dtb
-       dtc -p 0x1000 sandbox-u-boot.dts -O dtb -o sandbox-u-boot.dtb
-
-       # Create a number kernel image with zeroes
-       head -c 5000 /dev/zero >test-kernel.bin
-
-       # Build the FIT, but don't sign anything yet
-       echo Build FIT with signed images
-       ${mkimage} -D "${dtc}" -f sign-images-$sha.its test.fit >${tmp}
-
-       run_uboot "unsigned signatures:" "dev-"
-
-       # Sign images with our dev keys
-       echo Sign images
-       ${mkimage} -D "${dtc}" -F -k dev-keys -K sandbox-u-boot.dtb \
-               -r test.fit >${tmp}
-
-       run_uboot "signed images" "dev+"
-
-
-       # Create a fresh .dtb without the public keys
-       dtc -p 0x1000 sandbox-u-boot.dts -O dtb -o sandbox-u-boot.dtb
-
-       echo Build FIT with signed configuration
-       ${mkimage} -D "${dtc}" -f sign-configs-$sha.its test.fit >${tmp}
-
-       run_uboot "unsigned config" $sha"+ OK"
-
-       # Sign images with our dev keys
-       echo Sign images
-       ${mkimage} -D "${dtc}" -F -k dev-keys -K sandbox-u-boot.dtb \
-               -r test.fit >${tmp}
-
-       run_uboot "signed config" "dev+"
-
-       echo check signed config on the host
-       if ! ${fit_check_sign} -f test.fit -k sandbox-u-boot.dtb >${tmp}; then
-               echo
-               echo "Verified boot key check on host failed, output follows:"
-               cat ${tmp}
-               false
-       else
-               if ! grep -q "dev+" ${tmp}; then
-                       echo
-                       echo "Verified boot key check failed, output follows:"
-                       cat ${tmp}
-                       false
-               else
-                       echo "OK"
-               fi
-       fi
-
-       run_uboot "signed config" "dev+"
-
-       # Increment the first byte of the signature, which should cause failure
-       sig=$(fdtget -t bx test.fit /configurations/conf at 1/signature at 1 value)
-       newbyte=$(printf %x $((0x${sig:0:2} + 1)))
-       sig="${newbyte} ${sig:2}"
-       fdtput -t bx test.fit /configurations/conf at 1/signature at 1 value ${sig}
-
-       run_uboot "signed config with bad hash" "Bad Data Hash"
-}
-
-sha=sha1
-do_test
-sha=sha256
-do_test
-
-popd >/dev/null
-
-echo
-if ${ok}; then
-       echo "Test passed"
-else
-       echo "Test failed"
-fi
--

2.8.0.rc3.226.g39d4020

_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Hi Simon,

Quoted 27 lines · Simon Glass

Now that we have a suitable test framework we should move all tests into it. The vboot test is a suitable candidate. Rewrite it in Python and move the data files into an appropriate directory.

Signed-off-by Simon Glass <[email protected]>

---

 test/README                                       |   1 -
 test/py/tests/test_vboot.py                       | 185 ++++++++++++++++++++++
 test/{ => py/tests}/vboot/sandbox-kernel.dts      |   0
 test/{ => py/tests}/vboot/sandbox-u-boot.dts      |   0
 test/{ => py/tests}/vboot/sign-configs-sha1.its   |   0
 test/{ => py/tests}/vboot/sign-configs-sha256.its |   0
 test/{ => py/tests}/vboot/sign-images-sha1.its    |   0
 test/{ => py/tests}/vboot/sign-images-sha256.its  |   0
 test/vboot/.gitignore                             |   3 -
 test/vboot/vboot_test.sh                          | 151 ------------------
 10 files changed, 185 insertions(+), 155 deletions(-)
 create mode 100644 test/py/tests/test_vboot.py
 rename test/{ => py/tests}/vboot/sandbox-kernel.dts (100%)
 rename test/{ => py/tests}/vboot/sandbox-u-boot.dts (100%)
 rename test/{ => py/tests}/vboot/sign-configs-sha1.its (100%)
 rename test/{ => py/tests}/vboot/sign-configs-sha256.its (100%)
 rename test/{ => py/tests}/vboot/sign-images-sha1.its (100%)
 rename test/{ => py/tests}/vboot/sign-images-sha256.its (100%)
 delete mode 100644 test/vboot/.gitignore
 delete mode 100755 test/vboot/vboot_test.sh

Thanks for this refactor! If the comments related to the sh to Python are too nit-picky we can certainly change and expand the test harness within additional patches later.

Hi Simon,

test/py/tests/test_vboot.py line 160

Quoted 11 lines · Simon Glass

Now that we have a suitable test framework we should move all tests into it. The vboot test is a suitable candidate. Rewrite it in Python and move the data files into an appropriate directory.

test/py/tests/test_vboot.py new file +7 -0

+    datadir = 'test/py/tests/vboot/'
+    fit = '%stest.fit' % tmpdir
+    mkimage = cons.config.build_dir + '/tools/mkimage'
+    fit_check_sign = cons.config.build_dir + '/tools/fit_check_sign'
+    dtc_args = '-I dts -O dtb -i %s' % tmpdir
+    dtb = '%ssandbox-u-boot.dtb' % tmpdir
+    sig_node = '/configurations/conf at 1/signature at 1'

If these variables are used throughout the tests like globals, should they be DATADIR, MKIMAGE, etc?

+ byte_list = ['%x' % (byte + 1)] + byte_list[1:]

test/py/tests/test_vboot.py +4 -0

+    # Create an RSA key pair
+    public_exponent = 65537
+    util.cmd(cons, 'openssl genpkey -algorithm RSA -out %sdev.key '
+                   '-pkeyopt rsa_keygen_bits:2048 '

Now that we have a suitable test framework we should move all tests into it. The vboot test is a suitable candidate. Rewrite it in Python and move the data files into an appropriate directory.

Signed-off-by Simon Glass <[email protected]>