test: Include Python tools in test coverage

PATCH New Simon Glass 3 participants 15 messages Nov 27, 2017 – Dec 19, 2017

lore

Share & tools

lore

Simon Glass

author

2017-11-27 03:25 UTC #0 lore raw

v2 cover test: Include Python tools in test coverage

At present there are a number of tools in U-Boot which have tests which are not widely used:

patman - prepare, check and send patches
buildman - multi-threaded multi-commit builder
dtoc - convert device tree file to C
binman - produce firmware images

In fact many tests are broken and this has not been noticed.

This series fixes up the tests and updates 'make tests' to run them. This should help ensure that the tests stay current.

Note: This series is available at u-boot-dm/binman-working. It requires Masahiro's pylibfdt series.

Changes in v2:
- Add new patch to correct operation when tests pass
- Adjust PYTHONPATH for new locations
- Adjust PYTHONPATH for new locations
- Drop patches already applied
- Run tests with travis-ci also
- Drop RFC tag from patch for binman code-coverage tests

Simon Glass (7):
test: Correct operation when tests pass
test: Run binman tests
test: Run patman tests
test: Run buildman tests
test: Run dtoc tests
travis.yml: Run tests for tools
binman: Run code coverage tests

 .travis.yml | 15 +++++++++++++++
 test/run    | 16 +++++++++++++++-
 2 files changed, 30 insertions(+), 1 deletion(-)

Signature · 1 line

2.15.0.417.g466bffb3ac-goog

Simon Glass

author

2017-11-27 03:25 UTC #1 lore raw

PATCH v2 1/7 test: Correct operation when tests pass

When tests pass an error message is printed because of a variable that is not initialised. Fix this.

Signed-off-by Simon Glass <[email protected]>

---

Changes in v2:
- Add new patch to correct operation when tests pass

 test/run | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

test/run +2 -1

index caee4f83f2c..56160977a3f 100755
@@ -3,9 +3,10 @@
 run_test() {
 	$@
 	[ $? -ne 0 ] && result=$((result+1))
-	echo "result $result"
 }
 
+result=0
+
 # Run all tests that the standard sandbox build can support
 run_test ./test/py/test.py --bd sandbox --build

Signature · 1 line

2.15.0.417.g466bffb3ac-goog

Tom Rini

2017-11-28 14:19 UTC #2 lore raw

PATCH v2 1/7 test: Correct operation when tests pass

Quoted 4 lines · Simon Glass

When tests pass an error message is printed because of a variable that is not initialised. Fix this.
Signed-off-by Simon Glass <[email protected]>

Applied to u-boot/master, thanks!

Signature · 8 lines

Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20171128/54e11a35/attachment.sig>

Simon Glass

author

2017-11-27 03:25 UTC #3 lore raw

PATCH v2 2/7 test: Run binman tests

In reply to Simon Glass

Update the test script to run the binman tests also.

Signed-off-by Simon Glass <[email protected]>

---

Changes in v2:
- Adjust PYTHONPATH for new locations

 test/run | 3 +++
 1 file changed, 3 insertions(+)

test/run +3 -0

index 56160977a3f..a12ea31bf23 100755
@@ -17,6 +17,9 @@ run_test ./test/py/test.py --bd sandbox_spl --build -k \
 # Run tests for the flat DT version of sandbox
 ./test/py/test.py --bd sandbox_flattree --build
 
+PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \
+	./tools/binman/binman -t
+
 if [ $result == 0 ]; then
 	echo "Tests passed!"
 else

Signature · 1 line

2.15.0.417.g466bffb3ac-goog

Simon Glass

author

2017-12-19 13:18 UTC #4 lore raw

PATCH v2 2/7 test: Run binman tests

Quoted 10 lines · Simon Glass

Update the test script to run the binman tests also.
Signed-off-by Simon Glass <[email protected]>
---
Changes in v2:
- Adjust PYTHONPATH for new locations
 test/run | 3 +++
 1 file changed, 3 insertions(+)

Applied to u-boot-dm

Simon Glass

author

2017-11-27 03:25 UTC #5 lore raw

PATCH v2 3/7 test: Run patman tests

In reply to Simon Glass

Update the test script to run the patman tests also.

Signed-off-by Simon Glass <[email protected]>

---

Changes in v2: None

 test/run | 1 +
 1 file changed, 1 insertion(+)

test/run +1 -0

index a12ea31bf23..8f72a1f7b29 100755
@@ -19,6 +19,7 @@ run_test ./test/py/test.py --bd sandbox_spl --build -k \
 
 PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \
 	./tools/binman/binman -t
+run_test ./tools/patman/patman --test
 
 if [ $result == 0 ]; then
 	echo "Tests passed!"

Signature · 1 line

2.15.0.417.g466bffb3ac-goog

Simon Glass

author

2017-12-19 13:18 UTC #6 lore raw

PATCH v2 3/7 test: Run patman tests

Quoted 9 lines · Simon Glass

Update the test script to run the patman tests also.
Signed-off-by Simon Glass <[email protected]>
---
Changes in v2: None
 test/run | 1 +
 1 file changed, 1 insertion(+)

Applied to u-boot-dm

Simon Glass

author

2017-11-27 03:25 UTC #7 lore raw

PATCH v2 4/7 test: Run buildman tests

In reply to Simon Glass

Update the test script to run the buildman tests also.

Signed-off-by Simon Glass <[email protected]>

---

Changes in v2: None

 test/run | 1 +
 1 file changed, 1 insertion(+)

test/run +1 -0

index 8f72a1f7b29..a53383bb8e8 100755
@@ -20,6 +20,7 @@ run_test ./test/py/test.py --bd sandbox_spl --build -k \
 PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \
 	./tools/binman/binman -t
 run_test ./tools/patman/patman --test
+run_test ./tools/buildman/buildman -t
 
 if [ $result == 0 ]; then
 	echo "Tests passed!"

Signature · 1 line

2.15.0.417.g466bffb3ac-goog

Simon Glass

author

2017-12-19 13:18 UTC #8 lore raw

PATCH v2 4/7 test: Run buildman tests

Quoted 9 lines · Simon Glass

Update the test script to run the buildman tests also.
Signed-off-by Simon Glass <[email protected]>
---
Changes in v2: None
 test/run | 1 +
 1 file changed, 1 insertion(+)

Applied to u-boot-dm

Simon Glass

author

2017-11-27 03:25 UTC #9 lore raw

PATCH v2 5/7 test: Run dtoc tests

In reply to Simon Glass

Update the test script to run the dtoc tests also.

Signed-off-by Simon Glass <[email protected]>

---

Changes in v2:
- Adjust PYTHONPATH for new locations

 test/run | 1 +
 1 file changed, 1 insertion(+)

test/run +1 -0

index a53383bb8e8..af1219fdda9 100755
@@ -21,6 +21,7 @@ PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \
 	./tools/binman/binman -t
 run_test ./tools/patman/patman --test
 run_test ./tools/buildman/buildman -t
+PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test ./tools/dtoc/dtoc -t
 
 if [ $result == 0 ]; then
 	echo "Tests passed!"

Signature · 1 line

2.15.0.417.g466bffb3ac-goog

Simon Glass

author

2017-12-19 13:19 UTC #10 lore raw

PATCH v2 5/7 test: Run dtoc tests

Quoted 10 lines · Simon Glass

Update the test script to run the dtoc tests also.
Signed-off-by Simon Glass <[email protected]>
---
Changes in v2:
- Adjust PYTHONPATH for new locations
 test/run | 1 +
 1 file changed, 1 insertion(+)

Applied to u-boot-dm

Simon Glass

author

2017-11-27 03:26 UTC #11 lore raw

PATCH v2 6/7 travis.yml: Run tests for tools

In reply to Simon Glass

Run tests for the Python tools used by U-Boot.

Signed-off-by Simon Glass <[email protected]>

---

Changes in v2: None

 .travis.yml | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

.travis.yml +15 -0

index 0b7a0622bfd..5afed98bf6b 100644
@@ -113,6 +113,20 @@ script:
      ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
        -k "${TEST_PY_TEST_SPEC:-not a_test_which_does_not_exist}"
        --build-dir "$UBOOT_TRAVIS_BUILD_DIR";
+     ret=$?;
+     if [[ $ret -ne 0 ]]; then
+       exit $ret;
+     fi;
+   fi;
+   if [[ -n "${TEST_PY_TOOLS}" ]]; then
+     PYTHONPATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt"
+     PATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}"
+     ./tools/binman/binman -t &&
+     ./tools/patman/patman --test &&
+     ./tools/buildman/buildman -t &&
+     PYTHONPATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt"
+     PATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}"
+     ./tools/dtoc/dtoc -t;
    fi
 
 matrix:

@@ -283,6 +297,7 @@ matrix:
           TEST_PY_TEST_SPEC="test_ofplatdata"
           BUILDMAN="^sandbox$"
           TOOLCHAIN="x86_64"
+          TEST_PY_TOOLS="yes"
     - env:
         - TEST_PY_BD="sandbox_flattree"
           BUILDMAN="^sandbox_flattree$"

Signature · 1 line

2.15.0.417.g466bffb3ac-goog

sjg at google.com

2017-12-19 13:09 UTC #12 lore raw

PATCH v2 6/7 travis.yml: Run tests for tools

Run tests for the Python tools used by U-Boot.

Signed-off-by Simon Glass <[email protected]>

---

Changes in v2: None

 .travis.yml | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

Applied to u-boot-dm thanks!

Simon Glass

author

2017-11-27 03:26 UTC #13 lore raw

PATCH v2 7/7 binman: Run code coverage tests

In reply to Simon Glass

Binman has 100% test coverage for the code as it is at present. To encourage it to stay that way, run the code-coverage test as part of the normal U-Boot tests.

This is RFC because it requires the Python code coverage tools to be available.

Signed-off-by Simon Glass <[email protected]>

---

Changes in v2:
- Drop patches already applied
- Run tests with travis-ci also
- Drop RFC tag from patch for binman code-coverage tests

 test/run | 7 +++++++
 1 file changed, 7 insertions(+)

test/run +7 -0

index af1219fdda9..d5631b802af 100755
@@ -23,6 +23,13 @@ run_test ./tools/patman/patman --test
 run_test ./tools/buildman/buildman -t
 PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test ./tools/dtoc/dtoc -t
 
+# This needs you to set up Python test coverage tools.
+# To enable Python test coverage on Debian-type distributions (e.g. Ubuntu):
+#   $ sudo apt-get install python-pip python-pytest
+#   $ sudo pip install coverage
+PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \
+	./tools/binman/binman -T
+
 if [ $result == 0 ]; then
 	echo "Tests passed!"
 else

Signature · 1 line

2.15.0.417.g466bffb3ac-goog

sjg at google.com

2017-12-19 13:09 UTC #14 lore raw

PATCH v2 7/7 binman: Run code coverage tests

Binman has 100% test coverage for the code as it is at present. To encourage it to stay that way, run the code-coverage test as part of the normal U-Boot tests.

This is RFC because it requires the Python code coverage tools to be available.

Signed-off-by Simon Glass <[email protected]>

---

Changes in v2:
- Drop patches already applied
- Run tests with travis-ci also
- Drop RFC tag from patch for binman code-coverage tests

 test/run | 7 +++++++
 1 file changed, 7 insertions(+)

Applied to u-boot-dm thanks!