tests/avocado: Disable the test_sbsaref_edk2_firmware in the gitlab-CI
PATCH 1/1 tests/avocado: Disable the test_sbsaref_edk2_firmware in the gitlab-CI
The test fails occasionally, see e.g.:
https://gitlab.com/thuth/qemu/-/jobs/4196177756#L489
https://gitlab.com/thuth/qemu/-/jobs/4623296271#L290
Disable it at least in the gitlab-CI to avoid failing CI pipelines due to this problem.
--- tests/avocado/machine_aarch64_sbsaref.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tests/avocado/machine_aarch64_sbsaref.py +2 -2
@@ -8,8 +8,7 @@ import os -from avocado import skip -from avocado import skipUnless +from avocado import skipIf from avocado.utils import archive from avocado_qemu import QemuSystemTest
@@ -76,6 +75,7 @@ def fetch_firmware(self): "sbsa-ref", ) + @skipIf(os.getenv('GITLAB_CI'), 'Test does not work reliably') def test_sbsaref_edk2_firmware(self): """ :avocado: tags=cpu:cortex-a57
Signature · 1 line
2.39.3
Quoted 8 lines · Thomas Huth
The test fails occasionally, see e.g.:
https://gitlab.com/thuth/qemu/-/jobs/4196177756#L489
https://gitlab.com/thuth/qemu/-/jobs/4623296271#L290Disable it at least in the gitlab-CI to avoid failing CI pipelines due to this problem.
I'm not really a fan of disabling flaky tests only in the gitlab CI. It means that plain "make check-avocado" is useless to the average developer because it can have all manner of broken or flaky tests in it that never get tested by our CI. In fact I've had to develop the habit of running "GITLAB_CI=1 make check-avocado" so I can run the CI tests that we think work and not get distracted by intermittent failures.
If we want "disable this test except for if you're trying to investigate why it's flaky" then we should have the test be opt-in, not opt-out, the same way we do with QEMU_TEST_FLAKY_TESTS in the qtests.
thanks
-- PMM
Quoted 19 lines · Peter Maydell
Quoted 8 lines · Thomas Huth
The test fails occasionally, see e.g.:
https://gitlab.com/thuth/qemu/-/jobs/4196177756#L489
https://gitlab.com/thuth/qemu/-/jobs/4623296271#L290Disable it at least in the gitlab-CI to avoid failing CI pipelines due to this problem.
I'm not really a fan of disabling flaky tests only in the gitlab CI. It means that plain "make check-avocado" is useless to the average developer because it can have all manner of broken or flaky tests in it that never get tested by our CI. In fact I've had to develop the habit of running "GITLAB_CI=1 make check-avocado" so I can run the CI tests that we think work and not get distracted by intermittent failures.
Quoted 4 lines
If we want "disable this test except for if you're trying to investigate why it's flaky" then we should have the test be opt-in, not opt-out, the same way we do with
QEMU_TEST_FLAKY_TESTSin the qtests.
Agreed, this makes perfectly sense. Actually, I was looking for something like this in the tests/avocado directory, but I did not find anything similar, so I went for the GITLAB_CI variable ...
I'll send a v2 with QEMU_TEST_FLAKY_TESTS instead (and document it in the docs directory).
Thomas