tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
PATCH 1/1 tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run. Add a possibility to users to skip this particular test, by setting the AVOCADO_SKIP_SLOW_TESTS environment variable:
$ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance
...
(05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min
...--- tests/acceptance/boot_linux.py | 2 ++ 1 file changed, 2 insertions(+)
tests/acceptance/boot_linux.py +2 -0
@@ -15,6 +15,7 @@ from qemu.accel import kvm_available from qemu.accel import tcg_available +from avocado import skipIf from avocado.utils import cloudinit from avocado.utils import network from avocado.utils import vmimage
@@ -159,6 +160,7 @@ def add_common_args(self): self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom') + @skipIf(os.getenv('AVOCADO_SKIP_SLOW_TESTS'), 'Test takes >7min') def test_virt_tcg(self): """ :avocado: tags=accel:tcg
Signature · 1 line
2.21.3
Quoted 8 lines · Philippe Mathieu-Daudé
The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run. Add a possibility to users to skip this particular test, by setting the
AVOCADO_SKIP_SLOW_TESTSenvironment variable:$ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance ... (05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min ...
7m = 420s - I'm seeing:
hyperfine "./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg" Benchmark #1: ./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg
Time (mean ± σ): 162.179 s ± 3.138 s [User: 204.726 s, System: 9.663 s]
Range (min … max): 158.651 s … 170.036 s 10 runsIs this on very slow hardware?
Quoted 27 lines
Reported-by Peter Maydell <[email protected]>Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- tests/acceptance/boot_linux.py | 2 ++ 1 file changed, 2 insertions(+)tests/acceptance/boot_linux.py +2
-0@@ -15,6 +15,7 @@ from qemu.accel import kvm_available from qemu.accel import tcg_available +from avocado import skipIf from avocado.utils import cloudinit from avocado.utils import network from avocado.utils import vmimage@@ -159,6 +160,7 @@ def add_common_args(self): self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom') + @skipIf(os.getenv('AVOCADO_SKIP_SLOW_TESTS'), 'Test takes >7min') def test_virt_tcg(self): """ :avocado: tags=accel:tcg
Signature · 1 line
Alex Bennée
Quoted 13 lines · Alex Bennée
Quoted 8 lines · Philippe Mathieu-Daudé
The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run. Add a possibility to users to skip this particular test, by setting the
AVOCADO_SKIP_SLOW_TESTSenvironment variable:$ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance ... (05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min ...
Quoted 8 lines
7m = 420s - I'm seeing:
hyperfine "./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg" Benchmark #1: ./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg
Time (mean ± σ): 162.179 s ± 3.138 s [User: 204.726 s, System: 9.663 s] Range (min … max): 158.651 s … 170.036 s 10 runsIs this on very slow hardware?
Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz 16-core clang sanitizer build, which probably slows it down a bit.
But even 200 seconds is an order of magnitude slower than any of the other tests that check-acceptance runs. I think we should be aiming for tests here to be ~30 seconds at most, or the whole thing will take forever by the time we have decent coverage of most machines.
thanks
-- PMM
Quoted 23 lines · Peter Maydell
Quoted 13 lines · Alex Bennée
Quoted 8 lines · Philippe Mathieu-Daudé
The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run. Add a possibility to users to skip this particular test, by setting the
AVOCADO_SKIP_SLOW_TESTSenvironment variable:$ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance ... (05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min ...Quoted 8 lines
7m = 420s - I'm seeing:
hyperfine "./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg" Benchmark #1: ./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg
Time (mean ± σ): 162.179 s ± 3.138 s [User: 204.726 s, System: 9.663 s] Range (min … max): 158.651 s … 170.036 s 10 runsIs this on very slow hardware?
Quoted 8 lines
Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz 16-core clang sanitizer build, which probably slows it down a bit.
But even 200 seconds is an order of magnitude slower than any of the other tests that check-acceptance runs. I think we should be aiming for tests here to be ~30 seconds at most, or the whole thing will take forever by the time we have decent coverage of most machines.
I think we could say the same for a whole bunch of the tests that are based on full distro downloads. What is this one running under the hood anyway?
Quoted 3 lines
thanks
-- PMM
Signature · 1 line
Alex Bennée
Quoted 35 lines · Alex Bennée
Quoted 23 lines · Peter Maydell
Quoted 13 lines · Alex Bennée
Quoted 8 lines · Philippe Mathieu-Daudé
The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run. Add a possibility to users to skip this particular test, by setting the
AVOCADO_SKIP_SLOW_TESTSenvironment variable:$ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance ... (05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min ...Quoted 8 lines
7m = 420s - I'm seeing:
hyperfine "./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg" Benchmark #1: ./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg
Time (mean ± σ): 162.179 s ± 3.138 s [User: 204.726 s, System: 9.663 s] Range (min … max): 158.651 s … 170.036 s 10 runsIs this on very slow hardware?
Quoted 8 lines
Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz 16-core clang sanitizer build, which probably slows it down a bit.
But even 200 seconds is an order of magnitude slower than any of the other tests that check-acceptance runs. I think we should be aiming for tests here to be ~30 seconds at most, or the whole thing will take forever by the time we have decent coverage of most machines.
Quoted 3 lines
I think we could say the same for a whole bunch of the tests that are based on full distro downloads. What is this one running under the hood anyway?
Download time is now counted separately, this was one big feature request after the last KVM forum.
The second request was to improve test timing management, like having different timers tracking if the test is behaving as expected. I.e. you might want to boot a VM and run 'ping -c 3'. The whole test shouldn't take more than 7min, but we want to be sure the 'ping' command doesn't take more than 3.x seconds. I haven't heard about a such feature ready yet.
I really hope you are using a common cache location and now download the artifacts each time...
Quoted 7 lines
Quoted 5 lines
thanks
-- PMM
Quoted 28 lines · On Thu, 7 May 2020 18:22:35 +0200 Philippe Mathieu-Daudé <[email protected]>…
The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run. Add a possibility to users to skip this particular test, by setting the
AVOCADO_SKIP_SLOW_TESTSenvironment variable:$ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance ... (05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min ...Reported-by Peter Maydell <[email protected]>Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- tests/acceptance/boot_linux.py | 2 ++ 1 file changed, 2 insertions(+)tests/acceptance/boot_linux.py +1
-0@@ -15,6 +15,7 @@ from qemu.accel import kvm_available from qemu.accel import tcg_available +from avocado import skipIf from avocado.utils import cloudinit from avocado.utils import network from avocado.utils import vmimage
tests/acceptance/boot_linux.py def add_common_args(self): · line 166
Quoted 7 lines
tests/acceptance/boot_linux.py +1
-0@@ -159,6 +160,7 @@ def add_common_args(self): self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom') + @skipIf(os.getenv('AVOCADO_SKIP_SLOW_TESTS'), 'Test takes >7min') def test_virt_tcg(self): """ :avocado: tags=accel:tcg
Hi,
Why not simply add slow tag to the test. Like:
:avocado: tags=slow
The slow tests can then be skipped with
$ make check-acceptance AVOCADO_TAGS='-t -slow'
Regards,
Lukas Straub
Quoted 6 lines · Lukas Straub
Hi,
Why not simply add slow tag to the test. Like:
:avocado: tags=slowThe slow tests can then be skipped with
$ make check-acceptance AVOCADO_TAGS='-t -slow'
Is it possible to have the default be "do the fast stuff" and only do the slow stuff if the user asks? That's the way round that we do the iotests, I think.
thanks
-- PMM
Quoted 11 lines · On Mon, 11 May 2020 10:11:20 +0100 Peter Maydell <[email protected]>…
Quoted 6 lines · Lukas Straub
Hi,
Why not simply add slow tag to the test. Like:
:avocado: tags=slowThe slow tests can then be skipped with
$ make check-acceptance AVOCADO_TAGS='-t -slow'Is it possible to have the default be "do the fast stuff" and only do the slow stuff if the user asks? That's the way round that we do the iotests, I think.
I'd set AVOCADO_TAGS to '-t -slow' by default, but now that I look at the Makefile it's not that simple:
AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
We'd have to change the tests to skip if their target was not built. Than we can use tags for uses like this (and more like net, disk, ... tags). That would make it easier to filter the tests one wants to run: https://avocado-framework.readthedocs.io/en/78.0/guides/user/chapters/tags.html
Regards,
Lukas Straub
Quoted 2 lines
thanks
-- PMM
+Cleber
Quoted 15 lines · Lukas Straub
Quoted 11 lines · On Mon, 11 May 2020 10:11:20 +0100 Peter Maydell <[email protected]>…
Quoted 6 lines · Lukas Straub
Hi,
Why not simply add slow tag to the test. Like:
:avocado: tags=slowThe slow tests can then be skipped with
$ make check-acceptance AVOCADO_TAGS='-t -slow'Is it possible to have the default be "do the fast stuff" and only do the slow stuff if the user asks? That's the way round that we do the iotests, I think.
Quoted 5 lines
I'd set
AVOCADO_TAGSto '-t -slow' by default, but now that I look at theMakefileit's not that simple:AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
We'd have to change the tests to skip if their target was not built. Than we can use tags for uses like this (and more like net, disk, ... tags). That would make it easier to filter the tests one wants to run: https://avocado-framework.readthedocs.io/en/78.0/guides/user/chapters/tags.html
Cleber what do you think of this approach?
Quoted 7 lines
Regards,
Lukas StraubQuoted 2 lines
thanks
-- PMM