test: make tests should use pytest -ra
PATCH 1/1 test: make tests should use pytest -ra
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1] test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1] test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2] test/py/tests/test_efi_loader.py:115: Network not initialized
--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
test/run +3 -3
@@ -19,17 +19,17 @@ run_test() { failures=0 # Run all tests that the standard sandbox build can support -run_test "sandbox" ./test/py/test.py --bd sandbox --build -m "${mark_expr}" +run_test "sandbox" ./test/py/test.py -ra --bd sandbox --build -m "${mark_expr}" # Run tests which require sandbox_spl -run_test "sandbox_spl" ./test/py/test.py --bd sandbox_spl --build \ +run_test "sandbox_spl" ./test/py/test.py -ra --bd sandbox_spl --build \ -k 'test_ofplatdata or test_handoff' # Run tests for the flat-device-tree version of sandbox. This is a special # build which does not enable CONFIG_OF_LIVE for the live device tree, so we can # check that functionality is the same. The standard sandbox build (above) uses # CONFIG_OF_LIVE. -run_test "sandbox_flattree" ./test/py/test.py --bd sandbox_flattree --build \ +run_test "sandbox_flattree" ./test/py/test.py -ra --bd sandbox_flattree --build \ -k test_ut # Set up a path to dtc (device-tree compiler) and libfdt.py, a library it
--
2.25.1
PATCH 1/1 test: make tests should use pytest -ra
Quoted 12 lines · Heinrich Schuchardt
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1]
test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1]test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2]test/py/tests/test_efi_loader.py:115: Network not initialized
Quoted 5 lines
Signed-off-by Heinrich Schuchardt <[email protected]>--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
This is really noisy - I get lots of extra output. Can we make this an option?
Regards,
Simon
PATCH 1/1 test: make tests should use pytest -ra
Quoted 13 lines · Simon Glass
Quoted 12 lines · Heinrich Schuchardt
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1]
test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1]test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2]test/py/tests/test_efi_loader.py:115: Network not initialized
Quoted 7 lines
Quoted 5 lines
Signed-off-by Heinrich Schuchardt <[email protected]>--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.py sssssssssssssssssssssssssssssssssssssss [ 0%] test/py/tests/test_fs/test_ext.py ssssssssssssssssssssss [ 0%] test/py/tests/test_fs/test_mkdir.py ssssssssssss [ 0%] test/py/tests/test_fs/test_symlink.py ssss [ 0%] test/py/tests/test_fs/test_unlink.py ssssssssssssss [ 0%]
This new line has actionable information:
SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16
Next step is to change this line to provide a more useful output, e.g.
- except CalledProcessError:
- pytest.skip('Setup failed for filesystem: ' + fs_type)
+ except CalledProcessError as err:
+ pytest.skip('Setup failed for filesystem: ' + fs_type + \
+ ', {}'.format(err))
SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.
Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using /usr/sbin/mkfs.vfat or /sbin/mkfs.vfat and we can fix it.
We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Simon, why do you want to remove the actionable information?
Best regards
Heinrich
PATCH 1/1 test: make tests should use pytest -ra
Quoted 56 lines · Heinrich Schuchardt
Quoted 13 lines · Simon Glass
Quoted 12 lines · Heinrich Schuchardt
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1]
test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1]test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2]test/py/tests/test_efi_loader.py:115: Network not initializedQuoted 7 lines
Quoted 5 lines
Signed-off-by Heinrich Schuchardt <[email protected]>--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.pysssssssssssssssssssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_ext.pyssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_mkdir.pyssssssssssss [ 0%]test/py/tests/test_fs/test_symlink.pyssss [ 0%]test/py/tests/test_fs/test_unlink.pyssssssssssssss [ 0%]This new line has actionable information:
SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16Next step is to change this line to provide a more useful output, e.g.
(quoted diff) +3
-2- except CalledProcessError: - pytest.skip('Setup failed for filesystem: ' + fs_type) + except CalledProcessError as err: + pytest.skip('Setup failed for filesystem: ' + fs_type + \ + ', {}'.format(err))SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using
/usr/sbin/mkfs.vfator/sbin/mkfs.vfatand we can fix it.
Quoted 5 lines
We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Simon, why do you want to remove the actionable information?
OK, so I ran this through "qcheck" and I got a bit more output, but some of which indicates we should figure out how to enable these tests for sandbox out of the box.
The follow-up patch I really want to see is passing -ra in the travis/gitlab/azure file as I very much agree we should get everything running in CI or have a good reason why not (ie the MMC tests might be hard to enable on all platforms via qemu but we should make a good try at it).
And giving better error messages too would be very helpful so it's clearer why we aren't (and why some people would not want to) run the FS tests for example as we really haven't figured out a good 100% non-root-user method to do those tests.
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: not available URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200420/4751f612/attachment.sig>
PATCH 1/1 test: make tests should use pytest -ra
Quoted 73 lines · Tom Rini
Quoted 56 lines · Heinrich Schuchardt
Quoted 13 lines · Simon Glass
Quoted 12 lines · Heinrich Schuchardt
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1]
test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1]test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2]test/py/tests/test_efi_loader.py:115: Network not initializedQuoted 7 lines
Quoted 5 lines
Signed-off-by Heinrich Schuchardt <[email protected]>--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.pysssssssssssssssssssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_ext.pyssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_mkdir.pyssssssssssss [ 0%]test/py/tests/test_fs/test_symlink.pyssss [ 0%]test/py/tests/test_fs/test_unlink.pyssssssssssssss [ 0%]This new line has actionable information:
SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16Next step is to change this line to provide a more useful output, e.g.
(quoted diff) +3
-2- except CalledProcessError: - pytest.skip('Setup failed for filesystem: ' + fs_type) + except CalledProcessError as err: + pytest.skip('Setup failed for filesystem: ' + fs_type + \ + ', {}'.format(err))SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using
/usr/sbin/mkfs.vfator/sbin/mkfs.vfatand we can fix it.Quoted 5 lines
We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Simon, why do you want to remove the actionable information?
OK, so I ran this through "qcheck" and I got a bit more output, but some of which indicates we should figure out how to enable these tests for sandbox out of the box.
The follow-up patch I really want to see is passing -ra in the travis/gitlab/azure file as I very much agree we should get everything running in CI or have a good reason why not (ie the MMC tests might be hard to enable on all platforms via qemu but we should make a good try at it).
Quoted 5 lines
And giving better error messages too would be very helpful so it's clearer why we aren't (and why some people would not want to) run the FS tests for example as we really haven't figured out a good 100% non-root-user method to do those tests.
The tests run fine on my Debian system with
https://github.com/xypron2/u-boot/commit/f465d52a5a539761d9e5602331280ce07a1bcbca
where I assume mkfs.* and fsck.ext4 are in /sbin. The same should work on Ubuntu.
On current distributions there are also symlinks in /usr/sbin/ but
https://askubuntu.com/questions/674847/sudo-usr-sbin-mkfs-ext4-command-not-found
indicates that this does not hold true for elder distributions.
I will submit the test once tested in Travis CI and Gitlab CI.
Best regards
Heinrich
PATCH 1/1 test: make tests should use pytest -ra
In reply to Heinrich SchuchardtHi Heinrich,
Quoted 57 lines · Heinrich Schuchardt
Quoted 13 lines · Simon Glass
Quoted 12 lines · Heinrich Schuchardt
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1]
test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1]test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2]test/py/tests/test_efi_loader.py:115: Network not initializedQuoted 7 lines
Quoted 5 lines
Signed-off-by Heinrich Schuchardt <[email protected]>--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.pysssssssssssssssssssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_ext.pyssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_mkdir.pyssssssssssss [ 0%]test/py/tests/test_fs/test_symlink.pyssss [ 0%]test/py/tests/test_fs/test_unlink.pyssssssssssssss [ 0%]This new line has actionable information:
SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16Next step is to change this line to provide a more useful output, e.g.
(quoted diff) +3
-2- except CalledProcessError: - pytest.skip('Setup failed for filesystem: ' + fs_type) + except CalledProcessError as err: + pytest.skip('Setup failed for filesystem: ' + fs_type + \ + ', {}'.format(err))SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using
/usr/sbin/mkfs.vfator/sbin/mkfs.vfatand we can fix it.
Quoted 5 lines
We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Simon, why do you want to remove the actionable information?
I don't want to remove it. It isn't there now!
Let's fix it before we enable it. Otherwise it is just noise. The device tree fiasco is a real pain.
BTW the fs tests are flaky for me so I seldom run them.
Regards,
Simon
PATCH 1/1 test: make tests should use pytest -ra
Quoted 64 lines · Simon Glass
Hi Heinrich,
Quoted 61 lines · Heinrich Schuchardt
Quoted 13 lines · Simon Glass
Quoted 12 lines · Heinrich Schuchardt
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1]
test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1]test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2]test/py/tests/test_efi_loader.py:115: Network not initializedQuoted 7 lines
Quoted 5 lines
Signed-off-by Heinrich Schuchardt <[email protected]>--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.pysssssssssssssssssssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_ext.pyssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_mkdir.pyssssssssssss [ 0%]test/py/tests/test_fs/test_symlink.pyssss [ 0%]test/py/tests/test_fs/test_unlink.pyssssssssssssss [ 0%]This new line has actionable information:
SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16Next step is to change this line to provide a more useful output, e.g.
(quoted diff) +3
-2- except CalledProcessError: - pytest.skip('Setup failed for filesystem: ' + fs_type) + except CalledProcessError as err: + pytest.skip('Setup failed for filesystem: ' + fs_type + \ + ', {}'.format(err))SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using
/usr/sbin/mkfs.vfator/sbin/mkfs.vfatand we can fix it.We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Quoted 8 lines
Quoted 1 line
Simon, why do you want to remove the actionable information?
I don't want to remove it. It isn't there now!
Let's fix it before we enable it. Otherwise it is just noise. The device tree fiasco is a real pain.
BTW the fs tests are flaky for me so I seldom run them.
What do you mean by flaky?
Do you mean unreliable (cf.
https://www.urbandictionary.com/define.php?term=flaky)?
What is unreliable about the tests?
Best regards
Heinrich
Quoted 4 lines
Regards,
Simon
PATCH 1/1 test: make tests should use pytest -ra
Hi Heinrich,
Quoted 73 lines · Heinrich Schuchardt
Quoted 71 lines · Simon Glass
Hi Heinrich,
Quoted 57 lines · Heinrich Schuchardt
Quoted 13 lines · Simon Glass
Quoted 12 lines · Heinrich Schuchardt
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1]
test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1]test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2]test/py/tests/test_efi_loader.py:115: Network not initializedQuoted 7 lines
Quoted 5 lines
Signed-off-by Heinrich Schuchardt <[email protected]>--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.pysssssssssssssssssssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_ext.pyssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_mkdir.pyssssssssssss [ 0%]test/py/tests/test_fs/test_symlink.pyssss [ 0%]test/py/tests/test_fs/test_unlink.pyssssssssssssss [ 0%]This new line has actionable information:
SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16Next step is to change this line to provide a more useful output, e.g.
(quoted diff) +3
-2- except CalledProcessError: - pytest.skip('Setup failed for filesystem: ' + fs_type) + except CalledProcessError as err: + pytest.skip('Setup failed for filesystem: ' + fs_type + \ + ', {}'.format(err))SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using
/usr/sbin/mkfs.vfator/sbin/mkfs.vfatand we can fix it.Quoted 5 lines
We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Simon, why do you want to remove the actionable information?
I don't want to remove it. It isn't there now!
Let's fix it before we enable it. Otherwise it is just noise. The device tree fiasco is a real pain.
Quoted 6 lines
Quoted 1 line
BTW the fs tests are flaky for me so I seldom run them.
What do you mean by flaky?
Do you mean unreliable (cf.
https://www.urbandictionary.com/define.php?term=flaky)?
Yes!
Quoted 2 lines
What is unreliable about the tests?
You have it above - the filesystem tests sometimes fail for me.
I think all the other tests are good, although I think there is one that has a time delay in it that needs to be fixed.
Also we should really run the tests concurrently like binman does (see tools/concurrencytest).
Regards,
Simon
PATCH 1/1 test: make tests should use pytest -ra
Quoted 83 lines · Simon Glass
Hi Heinrich,
Quoted 73 lines · Heinrich Schuchardt
Quoted 71 lines · Simon Glass
Hi Heinrich,
Quoted 57 lines · Heinrich Schuchardt
Quoted 13 lines · Simon Glass
Quoted 12 lines · Heinrich Schuchardt
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1]
test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1]test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2]test/py/tests/test_efi_loader.py:115: Network not initializedQuoted 7 lines
Quoted 5 lines
Signed-off-by Heinrich Schuchardt <[email protected]>--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.pysssssssssssssssssssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_ext.pyssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_mkdir.pyssssssssssss [ 0%]test/py/tests/test_fs/test_symlink.pyssss [ 0%]test/py/tests/test_fs/test_unlink.pyssssssssssssss [ 0%]This new line has actionable information:
SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16Next step is to change this line to provide a more useful output, e.g.
(quoted diff) +3
-2- except CalledProcessError: - pytest.skip('Setup failed for filesystem: ' + fs_type) + except CalledProcessError as err: + pytest.skip('Setup failed for filesystem: ' + fs_type + \ + ', {}'.format(err))SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using
/usr/sbin/mkfs.vfator/sbin/mkfs.vfatand we can fix it.Quoted 5 lines
We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Simon, why do you want to remove the actionable information?
I don't want to remove it. It isn't there now!
Let's fix it before we enable it. Otherwise it is just noise. The device tree fiasco is a real pain.
Quoted 6 lines
Quoted 1 line
BTW the fs tests are flaky for me so I seldom run them.
What do you mean by flaky?
Do you mean unreliable (cf.
https://www.urbandictionary.com/define.php?term=flaky)?
Quoted 6 lines
Yes!
Quoted 2 lines
What is unreliable about the tests?
You have it above - the filesystem tests sometimes fail for me.
I think I've seen some of the FAT tests historically, but not recently. The biggest problem I see with them is that they are skipped or run for seemingly random reasons, under gitlab. And the root patch here would help to see why. For example:
https://gitlab.denx.de/u-boot/u-boot/-/jobs/80886 skips them but
https://gitlab.denx.de/u-boot/u-boot/-/jobs/80850 runs them. May be a runner config problem?
Quoted 5 lines
I think all the other tests are good, although I think there is one that has a time delay in it that needs to be fixed.
Also we should really run the tests concurrently like binman does (see
tools/concurrencytest).
I'm not sure how we could run most tests concurrently and keep things
generic. We can spawn N sandbox binaries but only one physical board.
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: not available URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200421/2a1eff18/attachment.sig>
PATCH 1/1 test: make tests should use pytest -ra
Hi Tom,
Quoted 92 lines · Tom Rini
Quoted 83 lines · Simon Glass
Hi Heinrich,
Quoted 73 lines · Heinrich Schuchardt
Quoted 71 lines · Simon Glass
Hi Heinrich,
Quoted 57 lines · Heinrich Schuchardt
Quoted 13 lines · Simon Glass
Quoted 12 lines · Heinrich Schuchardt
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1]
test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1]test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2]test/py/tests/test_efi_loader.py:115: Network not initializedQuoted 7 lines
Quoted 5 lines
Signed-off-by Heinrich Schuchardt <[email protected]>--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.pysssssssssssssssssssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_ext.pyssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_mkdir.pyssssssssssss [ 0%]test/py/tests/test_fs/test_symlink.pyssss [ 0%]test/py/tests/test_fs/test_unlink.pyssssssssssssss [ 0%]This new line has actionable information:
SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16Next step is to change this line to provide a more useful output, e.g.
(quoted diff) +3
-2- except CalledProcessError: - pytest.skip('Setup failed for filesystem: ' + fs_type) + except CalledProcessError as err: + pytest.skip('Setup failed for filesystem: ' + fs_type + \ + ', {}'.format(err))SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using
/usr/sbin/mkfs.vfator/sbin/mkfs.vfatand we can fix it.Quoted 5 lines
We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Simon, why do you want to remove the actionable information?
I don't want to remove it. It isn't there now!
Let's fix it before we enable it. Otherwise it is just noise. The device tree fiasco is a real pain.
Quoted 6 lines
Quoted 1 line
BTW the fs tests are flaky for me so I seldom run them.
What do you mean by flaky?
Do you mean unreliable (cf.
https://www.urbandictionary.com/define.php?term=flaky)?Quoted 6 lines
Yes!
Quoted 2 lines
What is unreliable about the tests?
You have it above - the filesystem tests sometimes fail for me.
Quoted 7 lines
I think I've seen some of the FAT tests historically, but not recently. The biggest problem I see with them is that they are skipped or run for seemingly random reasons, under gitlab. And the root patch here would help to see why. For example:
https://gitlab.denx.de/u-boot/u-boot/-/jobs/80886 skips them but
https://gitlab.denx.de/u-boot/u-boot/-/jobs/80850 runs them. May be a runner config problem?
Oh gosh that is odd.
Quoted 9 lines
Quoted 6 lines
I think all the other tests are good, although I think there is one that has a time delay in it that needs to be fixed.
Also we should really run the tests concurrently like binman does (see
tools/concurrencytest).I'm not sure how we could run most tests concurrently and keep things
generic. We can spawn N sandbox binaries but only one physical board.
Yes this is only for sandbox. It is pretty easy to turn on when it is allowed.
The current code in binman does this:
use_concurrent = True
try:
from concurrencytest import ConcurrentTestSuite, fork_for_tests
except:
use_concurrent = False
then:
if use_concurrent and processes != 1:
concurrent_suite = ConcurrentTestSuite(suite,
fork_for_tests(processes or multiprocessing.cpu_count()))
concurrent_suite.run(result)
else:
suite.run(result)
Regards,
Simon
PATCH 1/1 test: make tests should use pytest -ra
Quoted 126 lines · Simon Glass
Hi Tom,
Quoted 92 lines · Tom Rini
Quoted 83 lines · Simon Glass
Hi Heinrich,
Quoted 73 lines · Heinrich Schuchardt
Quoted 71 lines · Simon Glass
Hi Heinrich,
Quoted 57 lines · Heinrich Schuchardt
Quoted 13 lines · Simon Glass
Quoted 12 lines · Heinrich Schuchardt
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1]
test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1]test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2]test/py/tests/test_efi_loader.py:115: Network not initializedQuoted 7 lines
Quoted 5 lines
Signed-off-by Heinrich Schuchardt <[email protected]>--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.pysssssssssssssssssssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_ext.pyssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_mkdir.pyssssssssssss [ 0%]test/py/tests/test_fs/test_symlink.pyssss [ 0%]test/py/tests/test_fs/test_unlink.pyssssssssssssss [ 0%]This new line has actionable information:
SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16Next step is to change this line to provide a more useful output, e.g.
(quoted diff) +3
-2- except CalledProcessError: - pytest.skip('Setup failed for filesystem: ' + fs_type) + except CalledProcessError as err: + pytest.skip('Setup failed for filesystem: ' + fs_type + \ + ', {}'.format(err))SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using
/usr/sbin/mkfs.vfator/sbin/mkfs.vfatand we can fix it.Quoted 5 lines
We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Simon, why do you want to remove the actionable information?
I don't want to remove it. It isn't there now!
Let's fix it before we enable it. Otherwise it is just noise. The device tree fiasco is a real pain.
Quoted 6 lines
Quoted 1 line
BTW the fs tests are flaky for me so I seldom run them.
What do you mean by flaky?
Do you mean unreliable (cf.
https://www.urbandictionary.com/define.php?term=flaky)?Quoted 6 lines
Yes!
Quoted 2 lines
What is unreliable about the tests?
You have it above - the filesystem tests sometimes fail for me.
Quoted 7 lines
I think I've seen some of the FAT tests historically, but not recently. The biggest problem I see with them is that they are skipped or run for seemingly random reasons, under gitlab. And the root patch here would help to see why. For example:
https://gitlab.denx.de/u-boot/u-boot/-/jobs/80886 skips them but
https://gitlab.denx.de/u-boot/u-boot/-/jobs/80850 runs them. May be a runner config problem?Oh gosh that is odd.
Quoted 9 lines
Quoted 6 lines
I think all the other tests are good, although I think there is one that has a time delay in it that needs to be fixed.
Also we should really run the tests concurrently like binman does (see
tools/concurrencytest).I'm not sure how we could run most tests concurrently and keep things
generic. We can spawn N sandbox binaries but only one physical board.Yes this is only for sandbox. It is pretty easy to turn on when it is allowed.
The current code in binman does this:
use_concurrent= True
try:
from concurrencytest import ConcurrentTestSuite,fork_for_tests
except:
use_concurrent= False
Quoted 8 lines
then:
if
use_concurrentand processes != 1:
concurrent_suite= ConcurrentTestSuite(suite,
fork_for_tests(processes or multiprocessing.cpu_count()))
concurrent_suite.run(result)
else:
suite.run(result)
Right, OK. But have you proof-of-concepted that around pytest? Rewrite all of our tests in a different python framework seems like a big ask. I can see it being useful if a big part of the bottleneck is running check/qcheck in your own dev cycle. I personally don't try and concurrent my HW tests even if I could for everything-not-Pi.
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: not available URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200421/3b03a824/attachment.sig>
PATCH 1/1 test: make tests should use pytest -ra
Hi Tom,
Quoted 137 lines · Tom Rini
Quoted 126 lines · Simon Glass
Hi Tom,
Quoted 92 lines · Tom Rini
Quoted 83 lines · Simon Glass
Hi Heinrich,
Quoted 73 lines · Heinrich Schuchardt
Quoted 71 lines · Simon Glass
Hi Heinrich,
Quoted 57 lines · Heinrich Schuchardt
Quoted 13 lines · Simon Glass
Quoted 12 lines · Heinrich Schuchardt
By passing -ra to pytest we get a summary indicating which tests failed and why tests were skipped.
Here is an example output:
======================== short test summary info ========================= SKIPPED [1]
test/py/tests/test_efi_loader.py:81: No DHCP server available SKIPPED [1]test/py/tests/test_efi_loader.py:100: No static network configuration is defined
SKIPPED [2]test/py/tests/test_efi_loader.py:115: Network not initializedQuoted 7 lines
Quoted 5 lines
Signed-off-by Heinrich Schuchardt <[email protected]>--- test/run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)This is really noisy - I get lots of extra output. Can we make this an option?
When I run 'make tests' I get 41 out of 199 lines explaining skipped and failed tests.
Lines like these are noise because there is no actionable information:
test/py/tests/test_fs/test_basic.pysssssssssssssssssssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_ext.pyssssssssssssssssssssss [ 0%]test/py/tests/test_fs/test_mkdir.pyssssssssssss [ 0%]test/py/tests/test_fs/test_symlink.pyssss [ 0%]test/py/tests/test_fs/test_unlink.pyssssssssssssss [ 0%]This new line has actionable information:
SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16Next step is to change this line to provide a more useful output, e.g.
(quoted diff) +3
-2- except CalledProcessError: - pytest.skip('Setup failed for filesystem: ' + fs_type) + except CalledProcessError as err: + pytest.skip('Setup failed for filesystem: ' + fs_type + \ + ', {}'.format(err))SKIPPED [13]
test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16, Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit status 127.Now we know that that the test is wrong by assuming that mkfs.vfat is in the path instead of using
/usr/sbin/mkfs.vfator/sbin/mkfs.vfatand we can fix it.Quoted 5 lines
We should get rid of all skipped tests especially on Travis CI and Gitlab CI. Further we should provide instructions to set up a local system to avoid skipping tests.
Simon, why do you want to remove the actionable information?
I don't want to remove it. It isn't there now!
Let's fix it before we enable it. Otherwise it is just noise. The device tree fiasco is a real pain.
Quoted 6 lines
Quoted 1 line
BTW the fs tests are flaky for me so I seldom run them.
What do you mean by flaky?
Do you mean unreliable (cf.
https://www.urbandictionary.com/define.php?term=flaky)?Quoted 6 lines
Yes!
Quoted 2 lines
What is unreliable about the tests?
You have it above - the filesystem tests sometimes fail for me.
Quoted 7 lines
I think I've seen some of the FAT tests historically, but not recently. The biggest problem I see with them is that they are skipped or run for seemingly random reasons, under gitlab. And the root patch here would help to see why. For example:
https://gitlab.denx.de/u-boot/u-boot/-/jobs/80886 skips them but
https://gitlab.denx.de/u-boot/u-boot/-/jobs/80850 runs them. May be a runner config problem?Oh gosh that is odd.
Quoted 9 lines
Quoted 6 lines
I think all the other tests are good, although I think there is one that has a time delay in it that needs to be fixed.
Also we should really run the tests concurrently like binman does (see
tools/concurrencytest).I'm not sure how we could run most tests concurrently and keep things
generic. We can spawn N sandbox binaries but only one physical board.Yes this is only for sandbox. It is pretty easy to turn on when it is allowed.
The current code in binman does this:
use_concurrent= True
try:
from concurrencytest import ConcurrentTestSuite,fork_for_tests
except:
use_concurrent= FalseQuoted 8 lines
then:
if
use_concurrentand processes != 1:
concurrent_suite= ConcurrentTestSuite(suite,
fork_for_tests(processes or multiprocessing.cpu_count()))
concurrent_suite.run(result)
else:
suite.run(result)
Quoted 2 lines
Right, OK. But have you proof-of-concepted that around pytest? Rewrite all of our tests in a different python framework seems like a big ask.
It doesn't require any changes to the tests. It is just a different runner, although if you break the rules (independent tests) you might not get away with it. At least I didn't make any changes in binman. It makes a large difference there:
ellesmere:~/u$ time binman test <unittest.result.TestResult run=262 errors=0 failures=0>
real 0m1.680s
...
ellesmere:~/u$ time binman test -P 1 <unittest.result.TestResult run=262 errors=0 failures=0>
real 0m8.843s
Quoted 3 lines
I can see it being useful if a big part of the bottleneck is running check/qcheck in your own dev cycle. I personally don't try and concurrent my HW tests even if I could for everything-not-Pi.
Yes I don't have a way to run tests concurrently on the lab either. But in principal it could be done, since the host machine is mostly idle. I haven't tried it with gitlab-runner either.
Regards,
Simon