Implement pytest-based test infrastructure

This tool aims to test U-Boot by executing U-Boot shell commands using the console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself. It is asserted that writing test-related code in Python is simpler and more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.

In the future, I hope to publish (out-of-tree) the hook scripts, relay control utilities, and udev rules I will use for my own HW setup.

See README.md for more details!

Signed-off-by Stephen Warren <[email protected]>

---
 .gitignore                           |   1 +
 test/py/README.md                    | 287 +++++++++++++++++++++++++++++++++++
 test/py/board_jetson_tk1.py          |   1 +
 test/py/board_sandbox.py             |   1 +
 test/py/board_seaboard.py            |   1 +
 test/py/conftest.py                  | 225 +++++++++++++++++++++++++++
 test/py/multiplexed_log.css          |  70 +++++++++
 test/py/multiplexed_log.py           | 172 +++++++++++++++++++++
 test/py/pytest.ini                   |   5 +
 test/py/soc_tegra124.py              |   1 +
 test/py/soc_tegra20.py               |   1 +
 test/py/test.py                      |  12 ++
 test/py/test_000_version.py          |   9 ++
 test/py/test_env.py                  |  96 ++++++++++++
 test/py/test_help.py                 |   2 +
 test/py/test_md.py                   |  12 ++
 test/py/test_sandbox_exit.py         |  15 ++
 test/py/test_unknown_cmd.py          |   4 +
 test/py/uboot_console_base.py        | 143 +++++++++++++++++
 test/py/uboot_console_exec_attach.py |  28 ++++
 test/py/uboot_console_sandbox.py     |  22 +++
 21 files changed, 1108 insertions(+)
 create mode 100644 test/py/README.md
 create mode 100644 test/py/board_jetson_tk1.py
 create mode 100644 test/py/board_sandbox.py
 create mode 100644 test/py/board_seaboard.py
 create mode 100644 test/py/conftest.py
 create mode 100644 test/py/multiplexed_log.css
 create mode 100644 test/py/multiplexed_log.py
 create mode 100644 test/py/pytest.ini
 create mode 100644 test/py/soc_tegra124.py
 create mode 100644 test/py/soc_tegra20.py
 create mode 100755 test/py/test.py
 create mode 100644 test/py/test_000_version.py

 create mode 100644 test/py/test_env.py
 create mode 100644 test/py/test_help.py
 create mode 100644 test/py/test_md.py
 create mode 100644 test/py/test_sandbox_exit.py
 create mode 100644 test/py/test_unknown_cmd.py
 create mode 100644 test/py/uboot_console_base.py
 create mode 100644 test/py/uboot_console_exec_attach.py
 create mode 100644 test/py/uboot_console_sandbox.py

Hi Stephen,

Quoted 12 lines · Stephen Warren

This tool aims to test U-Boot by executing U-Boot shell commands using the console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself. It is asserted that writing test-related code in Python is simpler and more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

Quoted 2 lines

A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.

It's great to see this and thank you for putting in the effort!

It looks like a good way of doing functional tests. I still see a role for unit tests and things like test/dm. But if we can arrange to call all U-Boot tests (unit and functional) from one 'test.py' command that would be a win.

I'll look more when I can get it to work - see below.

I get this on my Ubuntu 64-bit machine (14.04.3)

$ ./test/py/test.py --bd sandbox --buildTraceback (most recent call last):
   File "./test/py/test.py", line 12, in <module>
     os.execvp("py.test", args)
   File "/usr/lib/python2.7/os.py", line 344, in execvp
     _execvpe(file, args)
   File "/usr/lib/python2.7/os.py", line 380, in _execvpe
     func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

## Requirements

The test suite is implemented using pytest. Interaction with the U-Boot console uses pexpect. Interaction with real hardware uses the tools of your choice; you get to implement various "hook" scripts that are called by the test suite at the appropriate time.

On Debian or Debian-like distributions, the following packages are required. Similar package names should exist in other distributions.

| Package        | Version tested (Ubuntu 14.04) |
| -------------- | ----------------------------- |
| python         | 2.7.5-5ubuntu3                |
| python-pytest  | 2.5.1-1                       |
| python-pexpect | 3.1-1ubuntu0.1                |

Quoted 30 lines · Simon Glass

Hi Stephen,

Quoted 15 lines · Stephen Warren

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler
and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

Quoted 2 lines

A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.

It's great to see this and thank you for putting in the effort!

It looks like a good way of doing functional tests. I still see a role for unit tests and things like test/dm. But if we can arrange to call all U-Boot tests (unit and functional) from one 'test.py' command that would be a win.

I'll look more when I can get it to work - see below.

made it print a message about checking the docs for missing requirements. I can probably patch the top-level test.py to do the same.

Quoted 35 lines · Stephen Warren

Quoted 33 lines · Simon Glass

Hi Stephen,

Quoted 16 lines · Stephen Warren

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler
and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

Quoted 2 lines

A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.

It's great to see this and thank you for putting in the effort!

It looks like a good way of doing functional tests. I still see a role for unit tests and things like test/dm. But if we can arrange to call all U-Boot tests (unit and functional) from one 'test.py' command that would be a win.

I'll look more when I can get it to work - see below.

...

Quoted 3 lines

made it print a message about checking the docs for missing requirements. I can probably patch the top-level test.py to do the same.

I've pushed such a patch to:

git://github.com/swarren/u-boot.git tegra_dev (the separate pytests branch has now been deleted)

There are also a variety of other patches there related to this testing infra-structure. I guess I'll hold off sending them to the list until there's been some general feedback on the patches I've already posted, but feel free to pull the branch down and play with it. Note that it's likely to get rebased as I work.

Hi Stephen,

Quoted 46 lines · Stephen Warren

Quoted 35 lines · Stephen Warren

Quoted 33 lines · Simon Glass

Hi Stephen,

Quoted 16 lines · Stephen Warren

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler
and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

Quoted 2 lines

A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.

It's great to see this and thank you for putting in the effort!

It looks like a good way of doing functional tests. I still see a role for unit tests and things like test/dm. But if we can arrange to call all U-Boot tests (unit and functional) from one 'test.py' command that would be a win.

I'll look more when I can get it to work - see below.

...

Quoted 3 lines

made it print a message about checking the docs for missing requirements. I can probably patch the top-level test.py to do the same.

I've pushed such a patch to:

Quoted 4 lines

git://github.com/swarren/u-boot.git tegra_dev (the separate pytests branch has now been deleted)

There are also a variety of other patches there related to this testing infra-structure. I guess I'll hold off sending them to the list until there's been some general feedback on the patches I've already posted, but feel free to pull the branch down and play with it. Note that it's likely to get rebased as I work.

OK I got it working thank you. It is horribly slow though - do you know what is holding it up? For me to takes 12 seconds to run the (very basic) tests.

Also please see dm_test_usb_tree() which uses a console buffer to check command output.

I wonder if we should use something like that
for simple unit tests, and use python for the more complicated functional tests?

Quoted 49 lines · Simon Glass

Hi Stephen,

Quoted 46 lines · Stephen Warren

Quoted 35 lines · Stephen Warren

Quoted 33 lines · Simon Glass

Hi Stephen,

Quoted 16 lines · Stephen Warren

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler
and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

Quoted 2 lines

A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.

It's great to see this and thank you for putting in the effort!

It looks like a good way of doing functional tests. I still see a role for unit tests and things like test/dm. But if we can arrange to call all U-Boot tests (unit and functional) from one 'test.py' command that would be a win.

I'll look more when I can get it to work - see below.

...

Quoted 3 lines

made it print a message about checking the docs for missing requirements. I can probably patch the top-level test.py to do the same.

I've pushed such a patch to:

Quoted 8 lines

Quoted 4 lines

git://github.com/swarren/u-boot.git tegra_dev (the separate pytests branch has now been deleted)

There are also a variety of other patches there related to this testing infra-structure. I guess I'll hold off sending them to the list until there's been some general feedback on the patches I've already posted, but feel free to pull the branch down and play with it. Note that it's likely to get rebased as I work.

OK I got it working thank you. It is horribly slow though - do you know what is holding it up? For me to takes 12 seconds to run the (very basic) tests.

It looks like pexpect includes a default delay to simulate human interaction. If you edit test/py/uboot_console_base.py ensure_spawned() and add the following somewhere soon after the assignment to self.p:

self.p.delaybeforesend = 0

... that will more than halve the execution time. (8.3 -> 3.5s on my 5-year-old laptop).

That said, even your 12s or my 8.3s doesn't seem like a bad price to pay for some easy-to-use automated testing.

Quoted 3 lines

Also please see dm_test_usb_tree() which uses a console buffer to check command output.

OK, I'll take a look.

Quoted 3 lines

I wonder if we should use something like that
for simple unit tests, and use python for the more complicated functional tests?

I'm not sure that's a good idea; it'd be best to settle on a single way of executing tests so that (a) people don't have to run/implement different kinds of tests in different ways (b) we can leverage test code across as many tests as possible.

(Well, doing unit tests and system level tests differently might be necessary since one calls functions and the other uses the shell "user interface", but having multiple ways of doing e.g. system tests doesn't seem like a good idea.)

Hi Stephen,

Quoted 63 lines · Stephen Warren

Quoted 49 lines · Simon Glass

Hi Stephen,

Quoted 46 lines · Stephen Warren

Quoted 35 lines · Stephen Warren

Quoted 33 lines · Simon Glass

Hi Stephen,

Quoted 16 lines · Stephen Warren

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler
and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

Quoted 2 lines

A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.

It's great to see this and thank you for putting in the effort!

It looks like a good way of doing functional tests. I still see a role for unit tests and things like test/dm. But if we can arrange to call all U-Boot tests (unit and functional) from one 'test.py' command that would be a win.

I'll look more when I can get it to work - see below.

...

Quoted 3 lines

made it print a message about checking the docs for missing requirements. I can probably patch the top-level test.py to do the same.

I've pushed such a patch to:

Quoted 8 lines

Quoted 4 lines

git://github.com/swarren/u-boot.git tegra_dev (the separate pytests branch has now been deleted)

There are also a variety of other patches there related to this testing infra-structure. I guess I'll hold off sending them to the list until there's been some general feedback on the patches I've already posted, but feel free to pull the branch down and play with it. Note that it's likely to get rebased as I work.

OK I got it working thank you. It is horribly slow though - do you know what is holding it up? For me to takes 12 seconds to run the (very basic) tests.

It looks like pexpect includes a default delay to simulate human interaction. If you edit test/py/uboot_console_base.py ensure_spawned() and add the following somewhere soon after the assignment to self.p:

Quoted 7 lines

self.p.delaybeforesend = 0

... that will more than halve the execution time. (8.3 -> 3.5s on my 5-year-old laptop).

That said, even your 12s or my 8.3s doesn't seem like a bad price to pay for some easy-to-use automated testing.

Sure, but my reference is to the difference between a native C test and this framework. As we add more and more tests the overhead will be significant. If it takes 8 seconds to run the current (fairly trivial) tests, it might take a minute to run a larger suite, and to me that is too long (e.g. to bisect for a failing commit).

I wonder what is causing the delay?

Quoted 15 lines

Quoted 3 lines

Also please see dm_test_usb_tree() which uses a console buffer to check command output.

OK, I'll take a look.

Quoted 3 lines

I wonder if we should use something like that
for simple unit tests, and use python for the more complicated functional tests?

I'm not sure that's a good idea; it'd be best to settle on a single way of executing tests so that (a) people don't have to run/implement different kinds of tests in different ways (b) we can leverage test code across as many tests as possible.

Quoted 4 lines

(Well, doing unit tests and system level tests differently might be necessary since one calls functions and the other uses the shell "user interface", but having multiple ways of doing e.g. system tests doesn't seem like a good idea.)

As you found with some of the tests, it is convenient/necessary to be able to call U-Boot C functions in some tests. So I don't see this as a one-size-fits-all solution.

I think it is perfectly reasonable for the python framework to run the existing C tests - there is no need to rewrite them in Python. Also for the driver model tests - we can just run the tests from some sort of python wrapper and get the best of both worlds, right?

Please don't take this to indicate any lack of enthusiasm for what you are doing - it's a great development and I'm sure it will help a lot! We really need to unify all the tests so we can run them all in one step.

I just think we should aim to have the automated tests run in a few seconds (let's say 5-10 at the outside). We need to make sure that the python framework will allow this even when running thousands of tests.

Hi Stephen,

Quoted 116 lines · Simon Glass

Hi Stephen,

Quoted 63 lines · Stephen Warren

Quoted 49 lines · Simon Glass

Hi Stephen,

Quoted 46 lines · Stephen Warren

Quoted 35 lines · Stephen Warren

Quoted 33 lines · Simon Glass

Hi Stephen,

Quoted 16 lines · Stephen Warren

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler
and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

Quoted 2 lines

A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.

It's great to see this and thank you for putting in the effort!

It looks like a good way of doing functional tests. I still see a role for unit tests and things like test/dm. But if we can arrange to call all U-Boot tests (unit and functional) from one 'test.py' command that would be a win.

I'll look more when I can get it to work - see below.

...

Quoted 3 lines

made it print a message about checking the docs for missing requirements. I can probably patch the top-level test.py to do the same.

I've pushed such a patch to:

Quoted 8 lines

Quoted 4 lines

git://github.com/swarren/u-boot.git tegra_dev (the separate pytests branch has now been deleted)

There are also a variety of other patches there related to this testing infra-structure. I guess I'll hold off sending them to the list until there's been some general feedback on the patches I've already posted, but feel free to pull the branch down and play with it. Note that it's likely to get rebased as I work.

OK I got it working thank you. It is horribly slow though - do you know what is holding it up? For me to takes 12 seconds to run the (very basic) tests.

It looks like pexpect includes a default delay to simulate human interaction. If you edit test/py/uboot_console_base.py ensure_spawned() and add the following somewhere soon after the assignment to self.p:

Quoted 7 lines

self.p.delaybeforesend = 0

... that will more than halve the execution time. (8.3 -> 3.5s on my 5-year-old laptop).

That said, even your 12s or my 8.3s doesn't seem like a bad price to pay for some easy-to-use automated testing.

Sure, but my reference is to the difference between a native C test and this framework. As we add more and more tests the overhead will be significant. If it takes 8 seconds to run the current (fairly trivial) tests, it might take a minute to run a larger suite, and to me that is too long (e.g. to bisect for a failing commit).

I wonder what is causing the delay?

Quoted 15 lines

Quoted 3 lines

Also please see dm_test_usb_tree() which uses a console buffer to check command output.

OK, I'll take a look.

Quoted 3 lines

I wonder if we should use something like that
for simple unit tests, and use python for the more complicated functional tests?

I'm not sure that's a good idea; it'd be best to settle on a single way of executing tests so that (a) people don't have to run/implement different kinds of tests in different ways (b) we can leverage test code across as many tests as possible.

Quoted 4 lines

(Well, doing unit tests and system level tests differently might be necessary since one calls functions and the other uses the shell "user interface", but having multiple ways of doing e.g. system tests doesn't seem like a good idea.)

As you found with some of the tests, it is convenient/necessary to be able to call U-Boot C functions in some tests. So I don't see this as a one-size-fits-all solution.

I think it is perfectly reasonable for the python framework to run the existing C tests - there is no need to rewrite them in Python. Also for the driver model tests - we can just run the tests from some sort of python wrapper and get the best of both worlds, right?

Please don't take this to indicate any lack of enthusiasm for what you are doing - it's a great development and I'm sure it will help a lot! We really need to unify all the tests so we can run them all in one step.

Quoted 3 lines

I just think we should aim to have the automated tests run in a few seconds (let's say 5-10 at the outside). We need to make sure that the python framework will allow this even when running thousands of tests.

BTW I would like to see if buildman can run tests automatically on each commit. It's been a long-term goal for a while.

Hi Stephen,

Quoted 63 lines · Stephen Warren

Quoted 49 lines · Simon Glass

Hi Stephen,

Quoted 46 lines · Stephen Warren

Quoted 35 lines · Stephen Warren

Quoted 33 lines · Simon Glass

Hi Stephen,

Quoted 16 lines · Stephen Warren

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler
and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

Quoted 2 lines

A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.

It's great to see this and thank you for putting in the effort!

It looks like a good way of doing functional tests. I still see a role for unit tests and things like test/dm. But if we can arrange to call all U-Boot tests (unit and functional) from one 'test.py' command that would be a win.

I'll look more when I can get it to work - see below.

...

Quoted 3 lines

made it print a message about checking the docs for missing requirements. I can probably patch the top-level test.py to do the same.

I've pushed such a patch to:

Quoted 8 lines

Quoted 4 lines

git://github.com/swarren/u-boot.git tegra_dev (the separate pytests branch has now been deleted)

There are also a variety of other patches there related to this testing infra-structure. I guess I'll hold off sending them to the list until there's been some general feedback on the patches I've already posted, but feel free to pull the branch down and play with it. Note that it's likely to get rebased as I work.

OK I got it working thank you. It is horribly slow though - do you know what is holding it up? For me to takes 12 seconds to run the (very basic) tests.

It looks like pexpect includes a default delay to simulate human interaction. If you edit test/py/uboot_console_base.py ensure_spawned() and add the following somewhere soon after the assignment to self.p:

Quoted 7 lines

self.p.delaybeforesend = 0

... that will more than halve the execution time. (8.3 -> 3.5s on my 5-year-old laptop).

That said, even your 12s or my 8.3s doesn't seem like a bad price to pay for some easy-to-use automated testing.

Sure, but my reference is to the difference between a native C test and this framework. As we add more and more tests the overhead will be significant. If it takes 8 seconds to run the current (fairly trivial) tests, it might take a minute to run a larger suite, and to me that is too long (e.g. to bisect for a failing commit).

I wonder what is causing the delay?

Quoted 14 lines

Quoted 2 lines

Also please see dm_test_usb_tree() which uses a console buffer to check command output.

OK, I'll take a look.

Quoted 3 lines

I wonder if we should use something like that
for simple unit tests, and use python for the more complicated functional tests?

I'm not sure that's a good idea; it'd be best to settle on a single way of executing tests so that (a) people don't have to run/implement different kinds of tests in different ways (b) we can leverage test code across as many tests as possible.

Quoted 4 lines

(Well, doing unit tests and system level tests differently might be necessary since one calls functions and the other uses the shell "user interface", but having multiple ways of doing e.g. system tests doesn't seem like a good idea.)

As you found with some of the tests, it is convenient/necessary to be able to call U-Boot C functions in some tests. So I don't see this as a one-size-fits-all solution.

I think it is perfectly reasonable for the python framework to run the existing C tests

- there is no need to rewrite them in Python.

Also
for the driver model tests - we can just run the tests from some sort of python wrapper and get the best of both worlds, right?

Please don't take this to indicate any lack of enthusiasm for what you are doing - it's a great development and I'm sure it will help a lot! We really need to unify all the tests so we can run them all in one step.

I just think we should aim to have the automated tests run in a few seconds (let's say 5-10 at the outside). We need to make sure that the python framework will allow this even when running thousands of tests.

Quoted 74 lines · Simon Glass

Hi Stephen,

Quoted 63 lines · Stephen Warren

Quoted 49 lines · Simon Glass

Hi Stephen,

Quoted 46 lines · Stephen Warren

Quoted 35 lines · Stephen Warren

Quoted 33 lines · Simon Glass

Hi Stephen,

Quoted 16 lines · Stephen Warren

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler
and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

Quoted 2 lines

A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.

It's great to see this and thank you for putting in the effort!

It looks like a good way of doing functional tests. I still see a role for unit tests and things like test/dm. But if we can arrange to call all U-Boot tests (unit and functional) from one 'test.py' command that would be a win.

I'll look more when I can get it to work - see below.

...

Quoted 3 lines

made it print a message about checking the docs for missing requirements. I can probably patch the top-level test.py to do the same.

I've pushed such a patch to:

Quoted 8 lines

Quoted 4 lines

git://github.com/swarren/u-boot.git tegra_dev (the separate pytests branch has now been deleted)

There are also a variety of other patches there related to this testing infra-structure. I guess I'll hold off sending them to the list until there's been some general feedback on the patches I've already posted, but feel free to pull the branch down and play with it. Note that it's likely to get rebased as I work.

OK I got it working thank you. It is horribly slow though - do you know what is holding it up? For me to takes 12 seconds to run the (very basic) tests.

It looks like pexpect includes a default delay to simulate human interaction. If you edit test/py/uboot_console_base.py ensure_spawned() and add the following somewhere soon after the assignment to self.p:

Quoted 7 lines

self.p.delaybeforesend = 0

... that will more than halve the execution time. (8.3 -> 3.5s on my 5-year-old laptop).

That said, even your 12s or my 8.3s doesn't seem like a bad price to pay for some easy-to-use automated testing.

Quoted 7 lines

Sure, but my reference is to the difference between a native C test and this framework. As we add more and more tests the overhead will be significant. If it takes 8 seconds to run the current (fairly trivial) tests, it might take a minute to run a larger suite, and to me that is too long (e.g. to bisect for a failing commit).

I wonder what is causing the delay?

I actually hope the opposite.

Most of the tests supported today are the most trivial possible tests, i.e. they take very little CPU time on the target to implement. I would naively expect that once we implement more interesting tests (USB Mass Storage, USB enumeration, eMMC/SD/USB data reading, Ethernet DHCP/TFTP, ...) the command invocation overhead will rapidly become insignificant. This certainly seems to be true for the UMS test I have locally, but who knows whether this will be more generally true.

I put a bit of time measurement into run_command() and found that on my system at work, for p.send("the shell command to execute") was actually (marginally) slower on sandbox than on real HW, despite real HW being a 115200 baud serial port, and the code splitting the shell commands into chunks that are sent and waited for synchronously to avoid overflowing UART FIFOs. I'm not sure why this is. Looking at U-Boot's console, it seems to be non-blocking, so I don't think termios VMIN/VTIME come into play (setting them to 0 made no difference), and the two raw modes took the same time. I meant to look into pexpect's termios settings to see if there was anything to tweak there, but forgot today.

I did do one experiment to compare expect (the Tcl version) and pexpect. If I do roughly the following in both:

spawn u-boot (sandbox)
wait for prompt
100 times:
send "echo $foo\n"
wait for "echo $foo"
wait for shell prompt
send "reset"
wait for "reset"
send "\n"

... then Tcl is about 3x faster on my system (IIRC 0.5 vs. 1.5s). If I remove all the "wait"s, then IIRC Tcl was about 15x faster or more. That's a pity. Still, I'm sure as heck not going to rewrite all this in Tcl:-( I wonder if something similar to pexpect but more targetted at simple "interactive shell" cases would remove any of that overhead.

Quoted 15 lines

Quoted 15 lines

Quoted 3 lines

Also please see dm_test_usb_tree() which uses a console buffer to check command output.

OK, I'll take a look.

Quoted 3 lines

I wonder if we should use something like that
for simple unit tests, and use python for the more complicated functional tests?

I'm not sure that's a good idea; it'd be best to settle on a single way of executing tests so that (a) people don't have to run/implement different kinds of tests in different ways (b) we can leverage test code across as many tests as possible.

Quoted 8 lines

Quoted 4 lines

(Well, doing unit tests and system level tests differently might be necessary since one calls functions and the other uses the shell "user interface", but having multiple ways of doing e.g. system tests doesn't seem like a good idea.)

As you found with some of the tests, it is convenient/necessary to be able to call U-Boot C functions in some tests. So I don't see this as a one-size-fits-all solution.

Yes, although I expect the split would be need-to-call-a-C-function -> put the code in U-Boot vs. anything else in Python via the shell prompt.

Quoted 2 lines

I think it is perfectly reasonable for the python framework to run the existing C tests

Yes.

Quoted 1 line

- there is no need to rewrite them in Python.

Probably not as an absolute mandate. Still, consistency would be nice. One advantage of having things as individual pytests is that the status of separate tests doesn't get aggregated; you can see that of 1000 tests, 10 failed, rather than seeing that 1000 logical tests were executed as part of 25 pytests, and 2 of those failed, each only because of 1 subtest with the other hundred subtests passing.

Quoted 4 lines

Also
for the driver model tests - we can just run the tests from some sort of python wrapper and get the best of both worlds, right?

I expect so, yes. I haven't looked at those yet.

Quoted 4 lines

Please don't take this to indicate any lack of enthusiasm for what you are doing - it's a great development and I'm sure it will help a lot! We really need to unify all the tests so we can run them all in one step.

Thanks:-)

Quoted 3 lines

I just think we should aim to have the automated tests run in a few seconds (let's say 5-10 at the outside). We need to make sure that the python framework will allow this even when running thousands of tests.

I'd be happy with something that took minutes, or longer. Given "build all boards" takes a very long time (and I'm sure we'd like everyone to do that, although I imagine few do), something of the same order of magnitude might even be reasonable? Thousands of test sounds like rather a lot; perhaps that number makes sense for tiny unit tests. I was thinking of testing fewer larger user-visible features that generally will have disk/network/... IO rates as the limiting factor. Perhaps one of those tests could indeed be "run 1000 tiny C-based unit tests via a single shell command".

Hi Stephen,

Quoted 7 lines · Stephen Warren

Quoted 6 lines · Simon Glass

Quoted 5 lines · Stephen Warren

Quoted 3 lines · Simon Glass

OK I got it working thank you. It is horribly slow though - do you know what is holding it up? For me to takes 12 seconds to run the (very basic) tests.

Quoted 24 lines

I put a bit of time measurement into run_command() and found that on my system at work, for p.send("the shell command to execute") was actually (marginally) slower on sandbox than on real HW, despite real HW being a 115200 baud serial port, and the code splitting the shell commands into chunks that are sent and waited for synchronously to avoid overflowing UART FIFOs. I'm not sure why this is. Looking at U-Boot's console, it seems to be non-blocking, so I don't think termios VMIN/VTIME come into play (setting them to 0 made no difference), and the two raw modes took the same time. I meant to look into pexpect's termios settings to see if there was anything to tweak there, but forgot today.

I did do one experiment to compare expect (the Tcl version) and pexpect. If I do roughly the following in both:

spawn u-boot (sandbox)
wait for prompt
100 times:
send "echo $foo\n"
wait for "echo $foo"
wait for shell prompt
send "reset"
wait for "reset"
send "\n"

Quoted 5 lines

... then Tcl is about 3x faster on my system (IIRC 0.5 vs. 1.5s). If I remove all the "wait"s, then IIRC Tcl was about 15x faster or more. That's a pity. Still, I'm sure as heck not going to rewrite all this in Tcl:-( I wonder if something similar to pexpect but more targetted at simple "interactive shell" cases would remove any of that overhead.

It is possible that we should use sandbox in 'cooked' mode so that lines an entered synchronously. The -t option might help here, or we may need something else.

Quoted 17 lines · Simon Glass

Hi Stephen,

Quoted 12 lines · Stephen Warren

Quoted 10 lines · Simon Glass

Quoted 7 lines · Stephen Warren

Quoted 3 lines · Simon Glass

OK I got it working thank you. It is horribly slow though - do you know what is holding it up? For me to takes 12 seconds to run the (very basic) tests.

..

Quoted 31 lines

Quoted 29 lines

I put a bit of time measurement into run_command() and found that on my system at work, for p.send("the shell command to execute") was actually (marginally) slower on sandbox than on real HW, despite real HW being a 115200 baud serial port, and the code splitting the shell commands into chunks that are sent and waited for synchronously to avoid overflowing UART FIFOs. I'm not sure why this is. Looking at U-Boot's console, it seems to be non-blocking, so I don't think termios VMIN/VTIME come into play (setting them to 0 made no difference), and the two raw modes took the same time. I meant to look into pexpect's termios settings to see if there was anything to tweak there, but forgot today.

I did do one experiment to compare expect (the Tcl version) and pexpect. If I do roughly the following in both:

spawn u-boot (sandbox)
wait for prompt
100 times:
send "echo $foo\n"
wait for "echo $foo"
wait for shell prompt
send "reset"
wait for "reset"
send "\n"

... then Tcl is about 3x faster on my system (IIRC 0.5 vs. 1.5s). If I remove all the "wait"s, then IIRC Tcl was about 15x faster or more. That's a pity. Still, I'm sure as heck not going to rewrite all this in Tcl:-( I wonder if something similar to pexpect but more targetted at simple "interactive shell" cases would remove any of that overhead.

Quoted 3 lines

It is possible that we should use sandbox in 'cooked' mode so that lines an entered synchronously. The -t option might help here, or we may need something else.

I don't think cooked mode will work, since I believe cooked is line-buffered, yet when U-Boot emits the shell prompt there's no \n printed afterwards.

FWIW, I hacked out pexpect and replaced it with some custom code. That reduced by sandbox execution time from ~5.1s to ~2.3s. Execution time against real HW didn't seem to be affected at all. Some features like timeouts and complete error handling are still missing, but I don't think that would affect the execution time. See my github tree for the WIP patch.

Hi Stephen,

Quoted 57 lines · Stephen Warren

Quoted 17 lines · Simon Glass

Hi Stephen,

Quoted 12 lines · Stephen Warren

Quoted 10 lines · Simon Glass

Quoted 7 lines · Stephen Warren

Quoted 3 lines · Simon Glass

OK I got it working thank you. It is horribly slow though - do you know what is holding it up? For me to takes 12 seconds to run the (very basic) tests.

..

Quoted 34 lines

Quoted 24 lines

I put a bit of time measurement into run_command() and found that on my system at work, for p.send("the shell command to execute") was actually (marginally) slower on sandbox than on real HW, despite real HW being a 115200 baud serial port, and the code splitting the shell commands into chunks that are sent and waited for synchronously to avoid overflowing UART FIFOs. I'm not sure why this is. Looking at U-Boot's console, it seems to be non-blocking, so I don't think termios VMIN/VTIME come into play (setting them to 0 made no difference), and the two raw modes took the same time. I meant to look into pexpect's termios settings to see if there was anything to tweak there, but forgot today.

I did do one experiment to compare expect (the Tcl version) and pexpect. If I do roughly the following in both:

spawn u-boot (sandbox)
wait for prompt
100 times:
send "echo $foo\n"
wait for "echo $foo"
wait for shell prompt
send "reset"
wait for "reset"
send "\n"

Quoted 5 lines

... then Tcl is about 3x faster on my system (IIRC 0.5 vs. 1.5s). If I remove all the "wait"s, then IIRC Tcl was about 15x faster or more. That's a pity. Still, I'm sure as heck not going to rewrite all this in Tcl:-( I wonder if something similar to pexpect but more targetted at simple "interactive shell" cases would remove any of that overhead.

It is possible that we should use sandbox in 'cooked' mode so that lines an entered synchronously. The -t option might help here, or we may need something else.

Quoted 3 lines

I don't think cooked mode will work, since I believe cooked is line-buffered, yet when U-Boot emits the shell prompt there's no \n printed afterwards.

Do you mean we need fflush() after writing the prompt? If so, that should be easy to arrange. We have a similar problem with the LCD, and added lcd_sync().

Quoted 5 lines

FWIW, I hacked out pexpect and replaced it with some custom code. That reduced by sandbox execution time from ~5.1s to ~2.3s. Execution time against real HW didn't seem to be affected at all. Some features like timeouts and complete error handling are still missing, but I don't think that would affect the execution time. See my github tree for the WIP patch.

Interesting, that's a big improvement. I wonder if we should look at building U-Boot with SWIG to remove all these overheads? Then the U-Boot command line (and any other feature we want) could become a Python class. Of course that would only work for sandbox.

Quoted 75 lines · Simon Glass

Hi Stephen,

Quoted 69 lines · Stephen Warren

Quoted 22 lines · Simon Glass

Hi Stephen,

Quoted 16 lines · Stephen Warren

Quoted 13 lines · Simon Glass

Quoted 9 lines · Stephen Warren

Quoted 3 lines · Simon Glass

OK I got it working thank you. It is horribly slow though - do you know what is holding it up? For me to takes 12 seconds to run the (very basic) tests.

..

Quoted 32 lines

Quoted 24 lines

I put a bit of time measurement into run_command() and found that on my system at work, for p.send("the shell command to execute") was actually (marginally) slower on sandbox than on real HW, despite real HW being a 115200 baud serial port, and the code splitting the shell commands into chunks that are sent and waited for synchronously to avoid overflowing UART FIFOs. I'm not sure why this is. Looking at U-Boot's console, it seems to be non-blocking, so I don't think termios VMIN/VTIME come into play (setting them to 0 made no difference), and the two raw modes took the same time. I meant to look into pexpect's termios settings to see if there was anything to tweak there, but forgot today.

I did do one experiment to compare expect (the Tcl version) and pexpect. If I do roughly the following in both:

spawn u-boot (sandbox)
wait for prompt
100 times:
send "echo $foo\n"
wait for "echo $foo"
wait for shell prompt
send "reset"
wait for "reset"
send "\n"

Quoted 5 lines

... then Tcl is about 3x faster on my system (IIRC 0.5 vs. 1.5s). If I remove all the "wait"s, then IIRC Tcl was about 15x faster or more. That's a pity. Still, I'm sure as heck not going to rewrite all this in Tcl:-( I wonder if something similar to pexpect but more targetted at simple "interactive shell" cases would remove any of that overhead.

Quoted 3 lines

It is possible that we should use sandbox in 'cooked' mode so that lines an entered synchronously. The -t option might help here, or we may need something else.

I don't think cooked mode will work, since I believe cooked is line-buffered, yet when U-Boot emits the shell prompt there's no \n printed afterwards.

Quoted 3 lines

Do you mean we need fflush() after writing the prompt? If so, that should be easy to arrange. We have a similar problem with the LCD, and added lcd_sync().

Anything U-Boot does will only affect its own buffer when sending into the PTY.

If the test program used cooked mode for its reading side of the PTY, then even with fflush() on the sending side, I don't believe reading from the PTY would return characters until a \n appeared.

FWIW, passing "-t cooked" to U-Boot (which affects data in the other direction to the discussion above) (plus hacking the code to disable terminal-level input echoing) doesn't make any difference to the test timing. That's not particularly surprising, since the test program sends each command as a single write, so it's likely that U-Boot reads each command into its stdin buffers in one go anyway.

Quoted 8 lines

Quoted 6 lines

FWIW, I hacked out pexpect and replaced it with some custom code. That reduced by sandbox execution time from ~5.1s to ~2.3s. Execution time against real HW didn't seem to be affected at all. Some features like timeouts and complete error handling are still missing, but I don't think that would affect the execution time. See my github tree for the WIP patch.

Quoted 4 lines

Interesting, that's a big improvement. I wonder if we should look at building U-Boot with SWIG to remove all these overheads? Then the U-Boot command line (and any other feature we want) could become a Python class. Of course that would only work for sandbox.

SWIG doesn't seem like a good direction; it would re-introduce different paths between sandbox and non-sandbox again. One of the main benefits of the test/py/ approach is that sandbox and real HW are treated the same.

At present we don't have a sensible test framework for anything other than sandbox, so to me the main benefit is that with your setup, we do.

The benefit of the existing sandbox tests is that they are very fast. We could bisect for a test failure in a few minutes. I'd like to make sure that we can still write C tests (that are called from your framework with results integrated into it) and that the Python tests are also fast.

How do we move this forward? Are you planing to resend the patch with the faster approach?

On 12/01/2015 09:40 AM, Simon Glass wrote:
...

Quoted 5 lines

At present we don't have a sensible test framework for anything other than sandbox, so to me the main benefit is that with your setup, we do.

Quoted 8 lines

The benefit of the existing sandbox tests is that they are very fast. We could bisect for a test failure in a few minutes. I'd like to make sure that we can still write C tests (that are called from your framework with results integrated into it) and that the Python tests are also fast.

How do we move this forward? Are you planing to resend the patch with the faster approach?

I'm tempted to squash down all/most the fixes/enhancements I've made since posting the original into a single commit rather than sending follow-on enhancements, since none of it is applied yet. I can keep the various test implementations etc. in separate commits as a series. Does that seem reasonable?

I need to do some more testing/clean-up of the version that doesn't use pexpect. For example, I have only tested sandbox and not real HW, and also haven't tested (and perhaps implemented some of) the support for matching unexpected error messages in the console log. Still, that all shouldn't

too long.

Quoted 26 lines · Simon Glass

Hi Stephen,

Quoted 12 lines · Stephen Warren

This tool aims to test U-Boot by executing U-Boot shell commands using the console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself. It is asserted that writing test-related code in Python is simpler and more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

Quoted 2 lines

A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too.

It's great to see this and thank you for putting in the effort!

It looks like a good way of doing functional tests. I still see a role for unit tests and things like test/dm. But if we can arrange to call all U-Boot tests (unit and functional) from one 'test.py' command that would be a win.

I'll look more when I can get it to work - see below.

...

Quoted 10 lines

I get this on my Ubuntu 64-bit machine (14.04.3)

$ ./test/py/test.py --bd sandbox --buildTraceback (most recent call last):
  File "./test/py/test.py", line 12, in <module>
    os.execvp("py.test", args)
  File "/usr/lib/python2.7/os.py", line 344, in execvp
    _execvpe(file, args)
  File "/usr/lib/python2.7/os.py", line 380, in _execvpe
    func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

"py.test" isn't in your $PATH. Did you install it? See the following in test/py/README.md:

Quoted 7 lines

## Requirements

The test suite is implemented using pytest. Interaction with the U-Boot console uses pexpect. Interaction with real hardware uses the tools of your choice; you get to implement various "hook" scripts that are called by the test suite at the appropriate time.

Quoted 8 lines

On Debian or Debian-like distributions, the following packages are required. Similar package names should exist in other distributions.

| Package        | Version tested (Ubuntu 14.04) |
| -------------- | ----------------------------- |
| python         | 2.7.5-5ubuntu3                |
| python-pytest  | 2.5.1-1                       |
| python-pexpect | 3.1-1ubuntu0.1                |

In the main Python code, I trapped at least one exception location and made it print a message about checking the docs for missing requirements. I can probably patch the top-level test.py to do the same.

On Thu, Nov 19, 2015 at 10:00:32AM -0700, Stephen Warren wrote:

Quoted 18 lines

See the following in test/py/README.md:

Quoted 15 lines

## Requirements

The test suite is implemented using pytest. Interaction with the U-Boot console uses pexpect. Interaction with real hardware uses the tools of your choice; you get to implement various "hook" scripts that are called by the test suite at the appropriate time.

On Debian or Debian-like distributions, the following packages are required. Similar package names should exist in other distributions.

| Package        | Version tested (Ubuntu 14.04) |
| -------------- | ----------------------------- |
| python         | 2.7.5-5ubuntu3                |
| python-pytest  | 2.5.1-1                       |
| python-pexpect | 3.1-1ubuntu0.1                |

Quoted 4 lines

In the main Python code, I trapped at least one exception location and made it print a message about checking the docs for missing requirements. I can probably patch the top-level test.py to do the same.

Isn't there some way to inject the local to U-Boot copy of the libraries in? I swear I've done something like that before in python..