tests/acceptance: Add boot vmlinux test
cover tests/acceptance: Add boot vmlinux test
This series add a new acceptance test: boot an uncompressed Linux kernel built with CONFIG_PVH, so checking the PVH capability introduced in QEMU 4.0 works.
The test implementation depends on [1] which is likely released on next Avocado. So that will need a version 2 of this series to bump Avocado version.
Also I want to use this as an example of a scenario that test assets could be better managed. As you see on patch 01 the kernel is built at test time on localhost. While Avocado provides an API to easily fetch and build it, the whole process takes reasonable time - besides the fact that localhost must have all build dependencies installed. How could it be done better?
Nonetheless I argue in favor of merging this as is, and gradually implement corrections to improve the tests assets management. Also if eventually the test is proven to unacceptable slow down the Travis CI then I can add a guard to skip it.
[1] https://github.com/avocado-framework/avocado/pull/3389
Wainer dos Santos Moschetta (2):
tests/acceptance: Add PVH boot test .travis.yml: Add kernel build deps for acceptance tests
.travis.yml | 2 ++ tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+) create mode 100644 tests/acceptance/pvh.py
Signature · 1 line
2.21.0
PATCH 1/2 tests/acceptance: Add PVH boot test
QEMU 4.0 onward is able to boot an uncompressed kernel image by using the x86/HVM direct boot ABI. It needs Linux >= 4.21 built with CONFIG_PVH=y.
This introduces an acceptance test which checks an uncompressed Linux kernel image boots properly.
--- tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 tests/acceptance/pvh.py
tests/acceptance/pvh.py new file +48 -0
@@ -0,0 +1,48 @@ +# Copyright (c) 2019 Red Hat, Inc. +# +# Author: +# Wainer dos Santos Moschetta <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +""" +x86/HVM direct boot acceptance tests. +""" + +from avocado.utils.kernel import KernelBuild +from avocado_qemu import Test +from avocado_qemu import wait_for_console_pattern + + +class Pvh(Test): + """ + Test suite for x86/HVM direct boot feature. + + :avocado: tags=slow,arch=x86_64,machine=q35 + """ + def test_boot_vmlinux(self): + """ + Boot uncompressed kernel image. + """ + # QEMU can boot a vmlinux image for kernel >= 4.21 built + # with CONFIG_PVH=y + kernel_version = '5.4.1' + kbuild = KernelBuild(kernel_version, work_dir=self.workdir) + try: + kbuild.download() + kbuild.uncompress() + kbuild.configure(targets=['defconfig', 'kvmconfig'], + extra_configs=['CONFIG_PVH=y']) + kbuild.build() + except: + self.cancel("Unable to build vanilla kernel %s" % kernel_version) + + self.vm.set_machine('q35') + self.vm.set_console() + kernel_command_line = 'printk.time=0 console=ttyS0' + self.vm.add_args('-kernel', kbuild.vmlinux, + '-append', kernel_command_line) + self.vm.launch() + wait_for_console_pattern(self, 'Kernel command line: %s' % + kernel_command_line)
Signature · 1 line
2.21.0
Re: [PATCH 1/2] tests/acceptance: Add PVH boot test
Quoted 69 lines · Wainer dos Santos Moschetta
QEMU 4.0 onward is able to boot an uncompressed kernel image by using the x86/HVM direct boot ABI. It needs Linux >= 4.21 built with
CONFIG_PVH=y.This introduces an acceptance test which checks an uncompressed Linux kernel image boots properly.
Signed-off-by Wainer dos Santos Moschetta <[email protected]>--- tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 tests/acceptance/pvh.pytests/acceptance/pvh.py new file +48
-1@@ -0,0 +1,48 @@ +# Copyright (c) 2019 Red Hat, Inc. +# +# Author: +# Wainer dos Santos Moschetta <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +""" +x86/HVM direct boot acceptance tests. +""" + +from avocado.utils.kernel import KernelBuild +from avocado_qemu import Test +from avocado_qemu import wait_for_console_pattern + + +class Pvh(Test): + """ + Test suite for x86/HVM direct boot feature. + + :avocado: tags=slow,arch=x86_64,machine=q35 + """ + def test_boot_vmlinux(self): + """ + Boot uncompressed kernel image. + """ + # QEMU can boot a vmlinux image for kernel >= 4.21 built + # with CONFIG_PVH=y + kernel_version = '5.4.1' + kbuild = KernelBuild(kernel_version, work_dir=self.workdir) + try: + kbuild.download() + kbuild.uncompress() + kbuild.configure(targets=['defconfig', 'kvmconfig'], + extra_configs=['CONFIG_PVH=y']) + kbuild.build() + except: + self.cancel("Unable to build vanilla kernel %s" % kernel_version) + + self.vm.set_machine('q35') + self.vm.set_console() + kernel_command_line = 'printk.time=0 console=ttyS0' + self.vm.add_args('-kernel', kbuild.vmlinux, + '-append', kernel_command_line) + self.vm.launch() + wait_for_console_pattern(self, 'Kernel command line: %s' % + kernel_command_line) --
Quoted 2 lines
2.21.0
Re: [PATCH 1/2] tests/acceptance: Add PVH boot test
In reply to Wainer dos Santos MoschettaQuoted 45 lines · Wainer dos Santos Moschetta
QEMU 4.0 onward is able to boot an uncompressed kernel image by using the x86/HVM direct boot ABI. It needs Linux >= 4.21 built with
CONFIG_PVH=y.This introduces an acceptance test which checks an uncompressed Linux kernel image boots properly.
Signed-off-by Wainer dos Santos Moschetta <[email protected]>--- tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 tests/acceptance/pvh.pytests/acceptance/pvh.py new file +26
-0@@ -0,0 +1,48 @@ +# Copyright (c) 2019 Red Hat, Inc. +# +# Author: +# Wainer dos Santos Moschetta <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +""" +x86/HVM direct boot acceptance tests. +""" + +from avocado.utils.kernel import KernelBuild +from avocado_qemu import Test +from avocado_qemu import wait_for_console_pattern + + +class Pvh(Test): + """ + Test suite for x86/HVM direct boot feature. + + :avocado: tags=slow,arch=x86_64,machine=q35 + """ + def test_boot_vmlinux(self): + """ + Boot uncompressed kernel image.
tests/acceptance/pvh.py line 36
Quoted 10 lines
tests/acceptance/pvh.py new file +10
-0+ """ + # QEMU can boot a vmlinux image for kernel >= 4.21 built + # with CONFIG_PVH=y + kernel_version = '5.4.1' + kbuild = KernelBuild(kernel_version, work_dir=self.workdir) + try: + kbuild.download() + kbuild.uncompress() + kbuild.configure(targets=['defconfig', 'kvmconfig'], + extra_configs=['CONFIG_PVH=y'])
I'm aware of the reason why this uses APIs not fulfilled by tests/requirements.txt, but, for the general public reviewing/testing code with extra requirements, it's a good idea to bump the requirements to a version that fulfills the requirement, and comment out clearly on the temporary nature of the change (marking the patch).
For instance, for this requirement, we could have:
tests/requirements.txt +2 -1
@@ -1,4 +1,5 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -avocado-framework==72.0 +# [REMOVE ME] use post 73.0 Avocado containing the new kernel build APIs +-e git+https://github.com/avocado-framework/avocado@d6fb24edcf847f09c312b55df3c674c64c79793e#egg=avocado_framework
This will not only help people to test it, but should also make it work transparently on CI.
tests/acceptance/pvh.py line 37
Quoted 1 line
tests/acceptance/pvh.py new file +1
-0+ kbuild.build()
As stated in my response to the cover letter, I think we need to move this elsewhere. The *very* minimum is to have this in a setUp() method, but we should strongly consider other solutions.
tests/acceptance/pvh.py line 45
Quoted 8 lines
tests/acceptance/pvh.py new file +8
-0+ except: + self.cancel("Unable to build vanilla kernel %s" % kernel_version) + + self.vm.set_machine('q35') + self.vm.set_console() + kernel_command_line = 'printk.time=0 console=ttyS0' + self.vm.add_args('-kernel', kbuild.vmlinux, + '-append', kernel_command_line)
And just for being thorough (and purist? idealistic? Utopian? :), if we stop and think about it, the following two lines are really what this test is all about. Everything else should be the test's setup.
I'm not arguing in favor of being radical and reject anything that is not perfect, but just reminding ourselves (myself very much included) of this general direction.
Cheers,
- Cleber.
Quoted 6 lines
tests/acceptance/pvh.py new file +3
-0+ self.vm.launch() + wait_for_console_pattern(self, 'Kernel command line: %s' % + kernel_command_line)Signature · 1 line
2.21.0
Re: [PATCH 1/2] tests/acceptance: Add PVH boot test
Quoted 53 lines · Cleber Rosa
Quoted 52 lines · Wainer dos Santos Moschetta
QEMU 4.0 onward is able to boot an uncompressed kernel image by using the x86/HVM direct boot ABI. It needs Linux >= 4.21 built with
CONFIG_PVH=y.This introduces an acceptance test which checks an uncompressed Linux kernel image boots properly.
Signed-off-by Wainer dos Santos Moschetta <[email protected]>--- tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)create mode 100644
tests/acceptance/pvh.pytests/acceptance/pvh.py new file +33
-0@@ -0,0 +1,48 @@ +# Copyright (c) 2019 Red Hat, Inc. +# +# Author: +# Wainer dos Santos Moschetta <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +""" +x86/HVM direct boot acceptance tests. +""" + +from avocado.utils.kernel import KernelBuild +from avocado_qemu import Test +from avocado_qemu import wait_for_console_pattern + + +class Pvh(Test): + """ + Test suite for x86/HVM direct boot feature. + + :avocado: tags=slow,arch=x86_64,machine=q35 + """ + def test_boot_vmlinux(self): + """ + Boot uncompressed kernel image. + """ + # QEMU can boot a vmlinux image for kernel >= 4.21 built + # with CONFIG_PVH=y + kernel_version = '5.4.1' + kbuild = KernelBuild(kernel_version, work_dir=self.workdir) + try: + kbuild.download()
Quoted 8 lines
tests/acceptance/pvh.py line 36
Quoted 3 lines
tests/acceptance/pvh.py new file +3
-0+ kbuild.uncompress() + kbuild.configure(targets=['defconfig', 'kvmconfig'], + extra_configs=['CONFIG_PVH=y'])I'm aware of the reason why this uses APIs not fulfilled by
tests/requirements.txt, but, for the general public reviewing/testing code with extra requirements, it's a good idea to bump the requirements to a version that fulfills the requirement, and comment out clearly on the temporary nature of the change (marking the patch).
Good idea, thanks for the tip.
Quoted 15 lines
For instance, for this requirement, we could have:
tests/requirements.txt +2
-1@@ -1,4 +1,5 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -avocado-framework==72.0 +# [REMOVE ME] use post 73.0 Avocado containing the new kernel build APIs +-e git+https://github.com/avocado-framework/avocado@d6fb24edcf847f09c312b55df3c674c64c79793e#egg=avocado_framework
Quoted 2 lines
This will not only help people to test it, but should also make it work transparently on CI.
True. It could had helped me to check the missing packages on Travis to build the kernel. I'm ashamed to tell how I did it. :)
Quoted 5 lines
tests/acceptance/pvh.py line 37
Quoted 2 lines
tests/acceptance/pvh.py new file +1
-0+ kbuild.build()As stated in my response to the cover letter, I think we need to move this elsewhere. The *very* minimum is to have this in a
setUp()method, but we should strongly consider other solutions.
On the proposed implementation the kernel is built only once and only for this test case. If I move the code to setUp() it will attempt to build the vmlinux for every case even when not needed (suppose I add a 'boot not CONFIG_PVH vmlinux to check it properly handle error' case which uses distro's kernel). Unless I put a guard like 'do not build if already present' which IMHO is weird. In other words, IMHO setUp() should hold only code that is share-able across cases.
Quoted 9 lines
tests/acceptance/pvh.py line 45
Quoted 9 lines
tests/acceptance/pvh.py new file +8
-0+ except: + self.cancel("Unable to build vanilla kernel %s" % kernel_version) + + self.vm.set_machine('q35') + self.vm.set_console() + kernel_command_line = 'printk.time=0 console=ttyS0' + self.vm.add_args('-kernel', kbuild.vmlinux, + '-append', kernel_command_line)
Quoted 7 lines
And just for being thorough (and purist? idealistic? Utopian? :), if we stop and think about it, the following two lines are really what this test is all about. Everything else should be the test's setup.
I'm not arguing in favor of being radical and reject anything that is not perfect, but just reminding ourselves (myself very much included) of this general direction.
IMHO we should merge tests which are "good enough" then interactively improve them. At least they will run with some frequency and eventually catch regressions while infra bits are improved. Now, what's 'good enough' for an acceptance test? Perhaps a test that run consistently?
Quoted 10 lines
Cheers,
- Cleber.Quoted 6 lines
tests/acceptance/pvh.py new file +3
-0+ self.vm.launch() + wait_for_console_pattern(self, 'Kernel command line: %s' % + kernel_command_line)Signature · 1 line
2.21.0
Re: [PATCH 1/2] tests/acceptance: Add PVH boot test
Quoted 36 lines · Wainer dos Santos Moschetta
Quoted 34 lines · Cleber Rosa
Quoted 33 lines · Wainer dos Santos Moschetta
QEMU 4.0 onward is able to boot an uncompressed kernel image by using the x86/HVM direct boot ABI. It needs Linux >= 4.21 built with
CONFIG_PVH=y.This introduces an acceptance test which checks an uncompressed Linux kernel image boots properly.
Signed-off-by Wainer dos Santos Moschetta <[email protected]>--- tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)create mode 100644
tests/acceptance/pvh.pytests/acceptance/pvh.py new file +14
-0@@ -0,0 +1,48 @@ +# Copyright (c) 2019 Red Hat, Inc. +# +# Author: +# Wainer dos Santos Moschetta <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +""" +x86/HVM direct boot acceptance tests. +""" + +from avocado.utils.kernel import KernelBuild +from avocado_qemu import Test
Quoted 8 lines
Quoted 8 lines
tests/acceptance/pvh.py line 22
Quoted 8 lines
tests/acceptance/pvh.py new file +8
-0+from avocado_qemu import wait_for_console_pattern + + +class Pvh(Test): + """ + Test suite for x86/HVM direct boot feature. + + :avocado: tags=slow,arch=x86_64,machine=q35
This should be:
:avocado: tags=slow,arch:x86_64,machine:q35
That is, the separator of key/val is ':', because the equal sign is used to separate the docstring directive type (here it's "tags") from their content. avocado list -V should show you the tag keys with all their values inside a parenthesis. That is, for the following docstring directive:
:avocado: tags=slow,arch:x86_64,machine:q35,machine:pc
You'd get:
slow,arch(x86_64),machine(q35,pc)
Quoted 37 lines
Quoted 14 lines
Quoted 4 lines
tests/acceptance/pvh.py new file +4
-0+ """ + def test_boot_vmlinux(self): + """ + Boot uncompressed kernel image.tests/acceptance/pvh.py line 36
Quoted 10 lines
tests/acceptance/pvh.py new file +10
-0+ """ + # QEMU can boot a vmlinux image for kernel >= 4.21 built + # with CONFIG_PVH=y + kernel_version = '5.4.1' + kbuild = KernelBuild(kernel_version, work_dir=self.workdir) + try: + kbuild.download() + kbuild.uncompress() + kbuild.configure(targets=['defconfig', 'kvmconfig'], + extra_configs=['CONFIG_PVH=y'])Quoted 5 lines
I'm aware of the reason why this uses APIs not fulfilled by
tests/requirements.txt, but, for the general public reviewing/testing code with extra requirements, it's a good idea to bump the requirements to a version that fulfills the requirement, and comment out clearly on the temporary nature of the change (marking the patch).Good idea, thanks for the tip.
Quoted 15 lines
For instance, for this requirement, we could have:
tests/requirements.txt +2
-1@@ -1,4 +1,5 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -avocado-framework==72.0 +# [REMOVE ME] use post 73.0 Avocado containing the new kernel build APIs +-e git+https://github.com/avocado-framework/avocado@d6fb24edcf847f09c312b55df3c674c64c79793e#egg=avocado_framework
Quoted 6 lines
Quoted 2 lines
This will not only help people to test it, but should also make it work transparently on CI.
True. It could had helped me to check the missing packages on Travis to build the kernel. I'm ashamed to tell how I did it. :)
Don't be, because you did check it. :)
Quoted 6 lines
Quoted 5 lines
tests/acceptance/pvh.py line 37
Quoted 2 lines
tests/acceptance/pvh.py new file +1
-0+ kbuild.build()As stated in my response to the cover letter, I think we need to move this elsewhere. The *very* minimum is to have this in a
setUp()method, but we should strongly consider other solutions.
Quoted 8 lines
On the proposed implementation the kernel is built only once and only for this test case. If I move the code to
setUp()it will attempt to build the vmlinux for every case even when not needed (suppose I add a 'boot notCONFIG_PVHvmlinux to check it properly handle error' case which uses distro's kernel). Unless I put a guard like 'do not build if already present' which IMHO is weird. In other words, IMHOsetUp()should hold only code that is share-able across cases.
I was thinking of *this* test setUp(), not avocado_qemu.Test.setUp().
Anyway, looking at the other options we talked about, I was able to boot a vmlinux image from a "mainstream distro" kernel package[1] that already has CONFIG_PVH enabled[2] with recent QEMU (and also tested that I wasn't able to do so with older QEMU). Other distros also provide a vmlinux image, but as part of the debuginfo packages and they can be HUGE, so not recommended here.
If we go with this route, compilation would be a non-issue, and this test would be just like the other "boot_linux_console.py" tests.
Quoted 13 lines
Quoted 13 lines
tests/acceptance/pvh.py line 45
Quoted 9 lines
tests/acceptance/pvh.py new file +8
-0+ except: + self.cancel("Unable to build vanilla kernel %s" % kernel_version) + + self.vm.set_machine('q35') + self.vm.set_console() + kernel_command_line = 'printk.time=0 console=ttyS0' + self.vm.add_args('-kernel', kbuild.vmlinux, + '-append', kernel_command_line)And just for being thorough (and purist? idealistic? Utopian? :), if we stop and think about it, the following two lines are really what this test is all about. Everything else should be the test's setup.
Quoted 9 lines
Quoted 3 lines
I'm not arguing in favor of being radical and reject anything that is not perfect, but just reminding ourselves (myself very much included) of this general direction.
IMHO we should merge tests which are "good enough" then interactively improve them. At least they will run with some frequency and eventually catch regressions while infra bits are improved. Now, what's 'good enough' for an acceptance test? Perhaps a test that run consistently?
Right. But even though this test can be proven stable (I can't disprove it), we also have to watch for the overall user experience. Like I've said before, I don't think users running this test are interested in building a kernel, but asserting a QEMU feature, and that can be a source of "test distrust" IMO.
Quoted 10 lines
Quoted 10 lines
Cheers,
- Cleber.Quoted 6 lines
tests/acceptance/pvh.py new file +3
-0+ self.vm.launch() + wait_for_console_pattern(self, 'Kernel command line: %s' % + kernel_command_line)Signature · 1 line
2.21.0
Please let me know what you think of reusing an available kernel instead of building one.
Cheers,
- Cleber.
[1] https://download.opensuse.org/repositories/openSUSE:/Factory/standard/x86_64/kernel-vanilla-5.3.12-1.1.x86_64.rpm
[2] https://kernel.opensuse.org/cgit/kernel-source/tree/config/x86_64/vanilla?h=linux-next&id=03bbea2f5521b0fe7bae800297509e9ca4c23117#n331
[3] http://mirrors.syringanetworks.net/fedora/linux/releases/31/Everything/x86_64/debug/tree/Packages/k/
Re: [PATCH 1/2] tests/acceptance: Add PVH boot test
In reply to Wainer dos Santos MoschettaQuoted 31 lines · Wainer dos Santos Moschetta
QEMU 4.0 onward is able to boot an uncompressed kernel image by using the x86/HVM direct boot ABI. It needs Linux >= 4.21 built with
CONFIG_PVH=y.This introduces an acceptance test which checks an uncompressed Linux kernel image boots properly.
Signed-off-by Wainer dos Santos Moschetta <[email protected]>--- tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)create mode 100644
tests/acceptance/pvh.pytests/acceptance/pvh.py new file +12
-0@@ -0,0 +1,48 @@ +# Copyright (c) 2019 Red Hat, Inc. +# +# Author: +# Wainer dos Santos Moschetta <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +""" +x86/HVM direct boot acceptance tests. +""" +
tests/acceptance/pvh.py line 22
Quoted 10 lines
tests/acceptance/pvh.py new file +10
-0+from avocado.utils.kernel import KernelBuild +from avocado_qemu import Test +from avocado_qemu import wait_for_console_pattern + + +class Pvh(Test): + """ + Test suite for x86/HVM direct boot feature. + + :avocado: tags=slow,arch=x86_64,machine=q35
Why is it slow? Because of the time spent to build the kernel?
This should be split in "prepare test" (here: slow) VS "run test", like we do when downloading assets.
Quoted 5 lines
tests/acceptance/pvh.py new file +5
-0+ """ + def test_boot_vmlinux(self): + """ + Boot uncompressed kernel image. + """
tests/acceptance/pvh.py line 37
Quoted 10 lines
tests/acceptance/pvh.py new file +10
-0+ # QEMU can boot a vmlinux image for kernel >= 4.21 built + # with CONFIG_PVH=y + kernel_version = '5.4.1' + kbuild = KernelBuild(kernel_version, work_dir=self.workdir) + try: + kbuild.download() + kbuild.uncompress() + kbuild.configure(targets=['defconfig', 'kvmconfig'], + extra_configs=['CONFIG_PVH=y']) + kbuild.build()
I like this feature, but I don't think it should be default for all users, they might start complaining, such:
- test drained battery while using laptop on the move
- test filled $HOME
This should be configurable, like users suggested they don't want the "download required assets from internet" feature.
Is it possible to build this once (ideally on some CI) and add the result binary + SHA1 hash in the Avocado assets public repository?
So users can fetch the prebuilt image, and are free to rebuild it.
Quoted 12 lines
tests/acceptance/pvh.py new file +11
-0+ except: + self.cancel("Unable to build vanilla kernel %s" % kernel_version) + + self.vm.set_machine('q35') + self.vm.set_console() + kernel_command_line = 'printk.time=0 console=ttyS0' + self.vm.add_args('-kernel', kbuild.vmlinux, + '-append', kernel_command_line) + self.vm.launch() + wait_for_console_pattern(self, 'Kernel command line: %s' % + kernel_command_line)
PATCH 2/2 .travis.yml: Add kernel build deps for acceptance tests
In reply to Wainer dos Santos MoschettaThe tests/acceptance/pvh.py test builds the Linux kernel at runtime so it needs dependencies installed in the container. Current used container image misses only libelf-dev, which is then added with this change.
--- .travis.yml | 2 ++ 1 file changed, 2 insertions(+)
.travis.yml +2 -0
@@ -277,6 +277,8 @@ matrix: - python3.5-venv - tesseract-ocr - tesseract-ocr-eng + # Additional kernel build dependencies + - libelf-dev # Using newer GCC with sanitizers
Signature · 1 line
2.21.0
Re: [PATCH 2/2] .travis.yml: Add kernel build deps for acceptance tests
Quoted 4 lines · Wainer dos Santos Moschetta
The
tests/acceptance/pvh.pytest builds the Linux kernel at runtime so it needs dependencies installed in the container. Current used container image misses only libelf-dev, which is then added with this change.
Hmm I'm not sure about this. Do we really want to building a whole kernel inside Travis? There must be stable PVH aware distro kernels we could use for the CI?
Quoted 19 lines
Signed-off-by Wainer dos Santos Moschetta <[email protected]>--- .travis.yml | 2 ++ 1 file changed, 2 insertions(+).travis.yml +2
-0@@ -277,6 +277,8 @@ matrix: - python3.5-venv - tesseract-ocr - tesseract-ocr-eng + # Additional kernel build dependencies + - libelf-dev # Using newer GCC with sanitizers
Signature · 1 line
Alex Bennée
Quoted 8 lines · Wainer dos Santos Moschetta
This series add a new acceptance test: boot an uncompressed Linux kernel built with
CONFIG_PVH, so checking the PVH capability introduced in QEMU 4.0 works.The test implementation depends on [1] which is likely released on next Avocado. So that will need a version 2 of this series to bump Avocado version.
Right, the Avocado bits have been merged, so unless a major reversal comes (not expected at all), it will be on Avocado 74.0.
Quoted 7 lines
Also I want to use this as an example of a scenario that test assets could be better managed. As you see on patch 01 the kernel is built at test time on localhost. While Avocado provides an API to easily fetch and build it, the whole process takes reasonable time - besides the fact that localhost must have all build dependencies installed. How could it be done better?
This is clearly not a "kernel build" test, so we should avoid building the kernel as part of the "PVH boot" test. The problem you expose here is a very real, and each possible solution has its own problems, unfortunately.
The best solution IMO would be to find a "well known" distribution of such a kernel. Maybe something maintained by the Xen project or one of its commercial products?
The second best solution is to have a helper script (using the Avocado utils API is fine) that will build a kernel and create/populate the test's data directory (tests/acceptance/pvh.py.data/) with a vmlinux. The test can cancel itself if it doesn't find a kernel there.
The third best solution IMO is for this test to require a parameter telling where the CONFIG_PVH enabled vmlinux kernel image is.
Also notice that, with a custom built kernel image (a different one for each user), the result of the test as a general indicator to the QEMU codebase diminishes quite a bit.
Quoted 5 lines
Nonetheless I argue in favor of merging this as is, and gradually implement corrections to improve the tests assets management. Also if eventually the test is proven to unacceptable slow down the Travis CI then I can add a guard to skip it.
I'm pretty sure that building the kernel will cause a major slow down of the Travis CI results for the "acceptance" block (and thus for the overall results). But, it may also time it out (there's a 50 minutes deadline).
I hope this brings some ideas, and thanks for coming up with interesting problems! :)
- Cleber.
Quoted 14 lines
[1] https://github.com/avocado-framework/avocado/pull/3389
Wainer dos Santos Moschetta (2):
tests/acceptance: Add PVH boot test .travis.yml: Add kernel build deps for acceptance tests.travis.yml | 2 ++ tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+) create mode 100644 tests/acceptance/pvh.pySignature · 1 line
2.21.0
Quoted 35 lines · Cleber Rosa
Quoted 8 lines · Wainer dos Santos Moschetta
This series add a new acceptance test: boot an uncompressed Linux kernel built with
CONFIG_PVH, so checking the PVH capability introduced in QEMU 4.0 works.The test implementation depends on [1] which is likely released on next Avocado. So that will need a version 2 of this series to bump Avocado version.
Right, the Avocado bits have been merged, so unless a major reversal comes (not expected at all), it will be on Avocado 74.0.
Quoted 7 lines
Also I want to use this as an example of a scenario that test assets could be better managed. As you see on patch 01 the kernel is built at test time on localhost. While Avocado provides an API to easily fetch and build it, the whole process takes reasonable time - besides the fact that localhost must have all build dependencies installed. How could it be done better?
This is clearly not a "kernel build" test, so we should avoid building the kernel as part of the "PVH boot" test. The problem you expose here is a very real, and each possible solution has its own problems, unfortunately.
The best solution IMO would be to find a "well known" distribution of such a kernel. Maybe something maintained by the Xen project or one of its commercial products?
The second best solution is to have a helper script (using the Avocado utils API is fine) that will build a kernel and create/populate the test's data directory (tests/acceptance/pvh.py.data/) with a vmlinux. The test can cancel itself if it doesn't find a kernel there.
Quoted 6 lines
The third best solution IMO is for this test to require a parameter telling where the
CONFIG_PVHenabled vmlinux kernel image is.Also notice that, with a custom built kernel image (a different one for each user), the result of the test as a general indicator to the QEMU codebase diminishes quite a bit.
I can see a use case for making it easier for developers to build test cases otherwise everyone has their own particular variant of the kernel and buildroot/busybox which they have in their own farm. However as Cleber has noted they make poor standardised tests given the variation in kernel builds you can get.
That said I think there are better targets. kvm-unit-tests can be built against a range of architectures and are fashioned as individual unit tests for specific functionality. It would be useful to make it easy for a developer to regenerate the test assets to re-run a test someone else has found to fail.
Quoted 30 lines
Quoted 5 lines
Nonetheless I argue in favor of merging this as is, and gradually implement corrections to improve the tests assets management. Also if eventually the test is proven to unacceptable slow down the Travis CI then I can add a guard to skip it.
I'm pretty sure that building the kernel will cause a major slow down of the Travis CI results for the "acceptance" block (and thus for the overall results). But, it may also time it out (there's a 50 minutes deadline).
I hope this brings some ideas, and thanks for coming up with interesting problems! :)
- Cleber.
Quoted 14 lines
[1] https://github.com/avocado-framework/avocado/pull/3389
Wainer dos Santos Moschetta (2):
tests/acceptance: Add PVH boot test .travis.yml: Add kernel build deps for acceptance tests.travis.yml | 2 ++ tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+) create mode 100644 tests/acceptance/pvh.pySignature · 1 line
2.21.0
Signature · 1 line
Alex Bennée