Acceptance test: Add "boot_linux" acceptance test
v8 cover Acceptance test: Add "boot_linux" acceptance test
This acceptance test, validates that a full blown Linux guest can successfully boot in QEMU. In this specific case, the guest chosen is Fedora version 31.
* x86_64, pc and q35 machine types, with and without kvm as an accellerator
* aarch64 and virt machine type, with and without kvm as an accellerator
* ppc64 and pseries machine type
* s390x and s390-ccw-virtio machine type
This has been tested on x86_64 and ppc64le hosts and has been running reliably (in my experience) on Travis CI.
On s390x hosts, it needs a pycdlib fix that has been merged on the upstream project, but it's still pending being part of a release.
Git:
- URI: https://github.com/clebergnu/qemu/tree/test_boot_linux_v8
- Remote: https://github.com/clebergnu/qemu
- Branch: test_boot_linux_v8
Travis CI:
- Build: https://travis-ci.org/clebergnu/qemu/builds/626935191
Previous version:
- v7: https://lists.gnu.org/archive/html/qemu-devel/2019-11/msg00220.html
- v6: https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg01202.html
- v5: https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04652.html
- v4: https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg02032.html
- v3: https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg01677.html
- v2: https://lists.gnu.org/archive/html/qemu-devel/2018-11/msg04318.html
- v1: http://lists.nongnu.org/archive/html/qemu-devel/2018-09/msg02530.html
Changes from v7:
================
This version drops a number of commits that had been already reviewed and have been merged:
* Dropped commit "Acceptance tests: use relative location for tests", already present in the latest master.
* Dropped commit "Acceptance tests: use avocado tags for machine type", already present in the latest master.
* Dropped commit: "Acceptance tests: introduce utility method for tags unique vals", already present in the latest master.
With regards to the handling of the build directory, and the usage of a qemu-img binary from the build tree, the following changed:
* Dropped commit "Acceptance tests: add the build directory to the system PATH", because the qemu-img binary to be used is now explicitly defined, instead of relying on the modification of the PATH environment variable.
* Dropped commit "Acceptance tests: depend on qemu-img", replaced by explicitly setting the qemu-img binary to be used for snapshot generation. Also, the newly added "--enable-tools" configure line on Travis CI makes sure that a matching qemu-img binary is available on CI.
* Dropped commit "Acceptance tests: keep a stable reference to the QEMU build dir", replaced by a different approach that introduces variables tracking the build dir, source dir and link (from build to source) dir.
* New commit "Acceptance tests: introduce BLD_DIR, SRC_DIR and LNK_DIR".
* New commit "Acceptance tests: add make targets to download images", that downloads the cloud images, aka vmimages, before the test execution itself.
* New commit "[TO BE REMOVED] Use Avocado master branch + vmimage fix" to facilitate the review/test of this version.
Additionally:
* The check for the availability of kvm now makes use of the strengthened qemu.accel.kvm_available() and passes the QEMU binary as an argument to make sure KVM support is compiled into that binary.
* The timeout was increased to 900 seconds. This is just one extra step to avoid false negatives on very slow systems. As a comparison, on Travis CI, on a x86_64 host, the slowest test takes around 250 seconds (boot_linux.py:BootLinuxAarch64.test_virt). On x86_64 systems with KVM enabled, my experience is that a test will take around 15 seconds.
Changes from v6:
================
* Bumped Fedora to most recently released version (31).
* Included new architectures (ppc64 and s390x), consolidating all tests into the same commit.
* New commit: "Acceptance tests: use avocado tags for machine type"
* New commit: "Acceptance tests: introduce utility method for tags unique vals"
* New commit: "Acceptance test x86_cpu_model_versions: use default vm", needed to normalize the use of the machine type tags
* Added a lot of leniency to the test setup (and reliability to the test/job), canceling the test if there are any failures while downloading/preparing the boot images.
* Made use of Avocado's data drainer a regular feature (dropped the commit with RFC) and squashed it.
* Bumped pycdlib version to 1.8.0
* Dropped explicit "--enable-slirp=git" (added on v5) to Travis CI configure line, as the default configuration on Travis CI now results in user networking capabilities.
Changes from v5:
================
* Added explicit "--enable-slirp=git" to Travis CI configure line, as these tests depend on "-netdev user" like networking.
* Bumped Fedora to most recently released version (30).
* Changed "checksum" parameter to 'sha256' and use the same hashes as provided by the Fedora project (instead of using Avocado's default sha1 and compute and use a different hash value).
* New commit: Add "boot_linux" test for aarch64 and virt machine type
* New commit: [RFC]: use Avocado data drainer for console logging
Changes from v4:
================
* New commit "Acceptance tests: use relative location for tests"
* New commit "Acceptance tests: keep a stable reference to the QEMU build dir"
* Pinned the Fedora 29 image by adding a checksum. The goal is to never allow more than one component to change at a time (the one allowed to change is QEMU itself). Updates to the image should be manual. (Based on comments from Cornelia)
* Moved the downloading of the Fedora 29 cloud image to the test setUp() method, canceling the test if the image can not be downloaded.
* Removed the ":avocado: enable" tag, given that Avocado versions 68.0 and later operate on a "recursive by default" manner, that is able to correctly identify this as an Avocado test.
Changes from v3:
================
* New patch "Acceptance tests: depend on qemu-img"
Known Issues on v3 (no longer applicable):
==========================================
* A recent TCG performance regression[1] affects this test in a number of ways:
- The test execution may timeout by itself
- The generation of SSH host keys in the guest's first boot is also affected (possibly also a timeout)
- The cloud-init "phone home" feature attempts to read the host keys and fails, causing the test to timeout and fail
These are not observed anymore once the fix[2] is applied.
[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg00338.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg01129.html
Changes from v2:
================
* Updated the tag to include the "arch:" key, in a similar fashion as to the tests in the "Acceptance Tests: target architecture support":
- https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg00369.html
* Renamed the test method name to test_x86_64_pc, again, similarly to the boot_linux_console.py tests in the series mentioned before.
* Set the machine type explicitly, again similarly to the boot_linux_console.py tests in the series mentioned before.
* Added messages after the launch of the VM, to let test runners know the test know waits for a boot confirmation from the the guest (Eduardo).
* Updated commit message to reflect the fact that this version does not allow for parameterization of the guest OS, version, etc.
* Dropped the RFC prefix on patch "RFC: Acceptance tests: add the build directory to the system PATH"
* Changed the comments on "RFC: Acceptance tests: add the build directory to the system PATH" to make it clear the addition of a the build directory to the PATH may influence other utility code.
Changes from v1:
================
* The commit message was adjusted, removing the reference to the avocado.utils.vmimage encoding issue on previous Avocado versions (<= 64.0) and the fix that would (and was) included in Avocado version 65.0.
* Effectively added pycdlib==1.6.0 to the requirements.txt file, added on a56931eef3, and adjusted the commit message was also to reflect that.
* Updated the default version of the guest OS, from Fedora 28 to 29. Besides possible improvements in the (virtual) hardware coverage, it brings a performance improvement in the order of 20% to the test.
* Removed all direct parameters usage. Because some parameters and its default values implemented in the test would prevent it from running on some environments. Example: the "accel" parameter had a default value of "kvm", which would prevent this test, that boots a x86_64 OS, from running on a host arch different than x86_64. I recognize that it's desirable to make tests reusable and parameterized (that was the reason for the first version doing so), but the mechanism to be used to define the architectures that a given test should support is still an open issue, and has been discussed in other threads. I'll follow up those discussions with a proposal, and until then, removing those aspects from this test implementation seemed to be the best option. A caveat: this test currently adds the same tag (x86_64) and follows other assumptions made on "boot_linux_console.py", that is, that a x86_64 target binary will be used to run it. If a user is in an environment that does not have a x86_64 target binary, it could filter those tests out with: "avocado run --filter-by-tags='-x86_64' tests/acceptance".
* Removed most arguments to the QEMU command line for pretty much the same reasons described above, and by following the general perception that I could grasp from other discussions that QEMU defaults should preferrably be used. This test, as well as others, can and should be extended later to allow for different test scenarios by passing well documented parameter values. That is, they should respect well-known parameters such as "accel" mentioned above, so that the same test can run with KVM or TCG.
* Changed the value of the memory argument to 1024, which based on my experimentations and observations is the minimum amount of RAM for the Fedora 29 cloud image to sucessfully boot on QEMU. I know there's no such thing as a "one size fits all", specially for QEMU, but this makes me wonder wether a x86_64 machine type shouldn't have its default_ram_size bumped to a number practical enough to run modern operating systems.
* Added a new patch "RFC: Acceptance tests: add the build directory to the system PATH", which is supposed to gather feedback on how to enable the use of built binaries, such as qemu-img, to code used by the test code. The specific situation here is that the vmimage, part of the avocado.utils libraries, makes use of qemu-img to create snapshot files. Even though we could require qemu-img to be installed as a dependency of tests, system wide, it actually goes against the goal of testing all QEMU things from the source/build tree. This became aparent with tests running on environments such as Travis CI, which don't necessarily have qemu-img available elsewhere.
Cleber Rosa (4):
Acceptance tests: introduce BLD_DIR, SRC_DIR and LNK_DIR Acceptance test: add "boot_linux" tests Acceptance tests: add make targets to download images [TO BE REMOVED] Use Avocado master branch + vmimage fix
.travis.yml | 2 +- tests/Makefile.include | 17 +- tests/acceptance/avocado_qemu/__init__.py | 27 +++- tests/acceptance/boot_linux.py | 180 ++++++++++++++++++++++ tests/requirements.txt | 3 +- 5 files changed, 219 insertions(+), 10 deletions(-) create mode 100644 tests/acceptance/boot_linux.py
Signature · 1 line
2.21.0
PATCH v8 1/4 Acceptance tests: introduce BLD_DIR, SRC_DIR and LNK_DIR
Some tests may benefit from using resources from a build directory. This introduces three variables that can help tests find resources in those directories.
First, a BLD_DIR is assumed to exist, given that the primary form of running the acceptance tests is from a build directory (which may or may not be the same as the source tree, that is, the SRC_DIR).
If the directory containing the acceptance tests happens to be a link to a directory (kept as LNK_DIR), it's assumed to it points to the source tree (SRC_DIR), which is the behavior defined on the QEMU Makefiles. If the directory containing the acceptance tests is not a link, then a in-tree build is assumed, and the BLD_DIR and SRC_DIR are the same and LNK_DIR is set None.
--- tests/acceptance/avocado_qemu/__init__.py | 27 ++++++++++++++++++----- 1 file changed, 21 insertions(+), 6 deletions(-)
tests/acceptance/avocado_qemu/__init__.py +21 -6
@@ -16,8 +16,23 @@ import tempfile import avocado -SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..') -sys.path.append(os.path.join(SRC_ROOT_DIR, 'python')) +#: The QEMU build root directory. It may also be the source directory +#: if building from the source dir, but it's safer to use BLD_DIR for +#: that purpose. Be aware that if this code is moved outside of a source +#: and build tree, it will not be accurate. +BLD_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__)))) + +if os.path.islink(os.path.dirname(os.path.dirname(__file__))): + #: The link to the acceptance tests dir in the source code directory. If + #: build dir is the same as the source dir, this is set to None + LNK_DIR = os.path.dirname(os.path.dirname(__file__)) + #: The QEMU root source directory + SRC_DIR = os.path.dirname(os.path.dirname(os.readlink(LNK_DIR))) +else: + LNK_DIR = None + SRC_DIR = BLD_DIR + +sys.path.append(os.path.join(SRC_DIR, 'python')) from qemu.machine import QEMUMachine
@@ -49,10 +64,10 @@ def pick_default_qemu_bin(arch=None): if is_readable_executable_file(qemu_bin_relative_path): return qemu_bin_relative_path - qemu_bin_from_src_dir_path = os.path.join(SRC_ROOT_DIR, + qemu_bin_from_bld_dir_path = os.path.join(BLD_DIR, qemu_bin_relative_path) - if is_readable_executable_file(qemu_bin_from_src_dir_path): - return qemu_bin_from_src_dir_path + if is_readable_executable_file(qemu_bin_from_bld_dir_path): + return qemu_bin_from_bld_dir_path def wait_for_console_pattern(test, success_message, failure_message=None):
@@ -122,7 +137,7 @@ class Test(avocado.Test): self.qemu_bin = self.params.get('qemu_bin', default=default_qemu_bin) if self.qemu_bin is None: - self.cancel("No QEMU binary defined or found in the source tree") + self.cancel("No QEMU binary defined or found in the build tree") def _new_vm(self, *args): vm = QEMUMachine(self.qemu_bin, sock_dir=tempfile.mkdtemp())
Signature · 1 line
2.21.0
Re: [PATCH v8 1/4] Acceptance tests: introduce BLD_DIR, SRC_DIR and LNK_DIR
Quoted 7 lines · Cleber Rosa
Some tests may benefit from using resources from a build directory. This introduces three variables that can help tests find resources in those directories.
First, a
BLD_DIRis assumed to exist, given that the primary form of running the acceptance tests is from a build directory (which may or may not be the same as the source tree, that is, theSRC_DIR).
Can we name this BUILD_DIR?
This would be more in line with the other buildsys files (configure/make).
Quoted 6 lines
If the directory containing the acceptance tests happens to be a link to a directory (kept as
LNK_DIR), it's assumed to it points to the source tree (SRC_DIR), which is the behavior defined on the QEMU Makefiles. If the directory containing the acceptance tests is not a link, then a in-tree build is assumed, and theBLD_DIRandSRC_DIRare the same andLNK_DIRis set None.
Similarly, can we name this CURRENT_DIR instead of LNK_DIR?
Quoted 60 lines
Signed-off-by Cleber Rosa <[email protected]>--- tests/acceptance/avocado_qemu/__init__.py | 27 ++++++++++++++++++----- 1 file changed, 21 insertions(+), 6 deletions(-)tests/acceptance/avocado_qemu/__init__.py +21
-6@@ -16,8 +16,23 @@ import tempfile import avocado -SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..') -sys.path.append(os.path.join(SRC_ROOT_DIR, 'python')) +#: The QEMU build root directory. It may also be the source directory +#: if building from the source dir, but it's safer to use BLD_DIR for +#: that purpose. Be aware that if this code is moved outside of a source +#: and build tree, it will not be accurate. +BLD_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__)))) + +if os.path.islink(os.path.dirname(os.path.dirname(__file__))): + #: The link to the acceptance tests dir in the source code directory. If + #: build dir is the same as the source dir, this is set to None + LNK_DIR = os.path.dirname(os.path.dirname(__file__)) + #: The QEMU root source directory + SRC_DIR = os.path.dirname(os.path.dirname(os.readlink(LNK_DIR))) +else: + LNK_DIR = None + SRC_DIR = BLD_DIR + +sys.path.append(os.path.join(SRC_DIR, 'python')) from qemu.machine import QEMUMachine@@ -49,10 +64,10 @@ def pick_default_qemu_bin(arch=None): if is_readable_executable_file(qemu_bin_relative_path): return qemu_bin_relative_path - qemu_bin_from_src_dir_path = os.path.join(SRC_ROOT_DIR, + qemu_bin_from_bld_dir_path = os.path.join(BLD_DIR, qemu_bin_relative_path) - if is_readable_executable_file(qemu_bin_from_src_dir_path): - return qemu_bin_from_src_dir_path + if is_readable_executable_file(qemu_bin_from_bld_dir_path): + return qemu_bin_from_bld_dir_path def wait_for_console_pattern(test, success_message, failure_message=None):@@ -122,7 +137,7 @@ class Test(avocado.Test): self.qemu_bin = self.params.get('qemu_bin', default=default_qemu_bin) if self.qemu_bin is None: - self.cancel("No QEMU binary defined or found in the source tree") + self.cancel("No QEMU binary defined or found in the build tree") def _new_vm(self, *args): vm = QEMUMachine(self.qemu_bin, sock_dir=tempfile.mkdtemp())
Re: [PATCH v8 1/4] Acceptance tests: introduce BLD_DIR, SRC_DIR and LNK_DIR
Quoted 11 lines · Philippe Mathieu-Daudé
Quoted 7 lines · Cleber Rosa
Some tests may benefit from using resources from a build directory. This introduces three variables that can help tests find resources in those directories.
First, a
BLD_DIRis assumed to exist, given that the primary form of running the acceptance tests is from a build directory (which may or may not be the same as the source tree, that is, theSRC_DIR).Can we name this
BUILD_DIR?
Yes, of course.
Quoted 2 lines
This would be more in line with the other buildsys files (configure/make).
That's a good point.
Quoted 9 lines
Quoted 6 lines
If the directory containing the acceptance tests happens to be a link to a directory (kept as
LNK_DIR), it's assumed to it points to the source tree (SRC_DIR), which is the behavior defined on the QEMU Makefiles. If the directory containing the acceptance tests is not a link, then a in-tree build is assumed, and theBLD_DIRandSRC_DIRare the same andLNK_DIRis set None.Similarly, can we name this
CURRENT_DIRinstead ofLNK_DIR?
Yes, or maybe even drop it? TBH, I can only see use cases for build and source dirs. So, I assume you'd propose SRC_DIR would be SOURCE_DIR?
Cheers,
- Cleber.
Re: [PATCH v8 1/4] Acceptance tests: introduce BLD_DIR, SRC_DIR and LNK_DIR
Quoted 27 lines · Cleber Rosa
Quoted 11 lines · Philippe Mathieu-Daudé
Quoted 7 lines · Cleber Rosa
Some tests may benefit from using resources from a build directory. This introduces three variables that can help tests find resources in those directories.
First, a
BLD_DIRis assumed to exist, given that the primary form of running the acceptance tests is from a build directory (which may or may not be the same as the source tree, that is, theSRC_DIR).Can we name this
BUILD_DIR?Yes, of course.
Quoted 2 lines
This would be more in line with the other buildsys files (configure/make).
That's a good point.
Quoted 7 lines
Quoted 6 lines
If the directory containing the acceptance tests happens to be a link to a directory (kept as
LNK_DIR), it's assumed to it points to the source tree (SRC_DIR), which is the behavior defined on the QEMU Makefiles. If the directory containing the acceptance tests is not a link, then a in-tree build is assumed, and theBLD_DIRandSRC_DIRare the same andLNK_DIRis set None.
Quoted 4 lines
Quoted 2 lines
Similarly, can we name this
CURRENT_DIRinstead ofLNK_DIR?Yes, or maybe even drop it? TBH, I can only see use cases for build
I haven't checked why you needed to add it, so if we don't need it, let's drop it :)
Quoted 2 lines
and source dirs. So, I assume you'd propose
SRC_DIRwould beSOURCE_DIR?
This one is understandable as it, but SOURCE_DIR is cleaner indeed.
Thanks,
Phil.
Re: [PATCH v8 1/4] Acceptance tests: introduce BLD_DIR, SRC_DIR and LNK_DIR
Quoted 28 lines · Philippe Mathieu-Daudé
Quoted 27 lines · Cleber Rosa
Quoted 11 lines · Philippe Mathieu-Daudé
Quoted 7 lines · Cleber Rosa
Some tests may benefit from using resources from a build directory. This introduces three variables that can help tests find resources in those directories.
First, a
BLD_DIRis assumed to exist, given that the primary form of running the acceptance tests is from a build directory (which may or may not be the same as the source tree, that is, theSRC_DIR).Can we name this
BUILD_DIR?Yes, of course.
Quoted 2 lines
This would be more in line with the other buildsys files (configure/make).
That's a good point.
Quoted 7 lines
Quoted 6 lines
If the directory containing the acceptance tests happens to be a link to a directory (kept as
LNK_DIR), it's assumed to it points to the source tree (SRC_DIR), which is the behavior defined on the QEMU Makefiles. If the directory containing the acceptance tests is not a link, then a in-tree build is assumed, and theBLD_DIRandSRC_DIRare the same andLNK_DIRis set None.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Similarly, can we name this
CURRENT_DIRinstead ofLNK_DIR?Yes, or maybe even drop it? TBH, I can only see use cases for build
I haven't checked why you needed to add it, so if we don't need it, let's drop it :)
1+ for dropping LNK_DIR variable.
Thanks,
Wainer
Quoted 12 lines
Quoted 4 lines
and source dirs. So, I assume you'd propose
SRC_DIRwould beSOURCE_DIR?This one is understandable as it, but
SOURCE_DIRis cleaner indeed.Thanks,
Phil.
PATCH v8 2/4 Acceptance test: add "boot_linux" tests
In reply to Cleber RosaThis acceptance test, validates that a full blown Linux guest can successfully boot in QEMU. In this specific case, the guest chosen is Fedora version 31.
* x86_64, pc and q35 machine types, with and without kvm as an accelerator
* aarch64 and virt machine type, with and without kvm as an accelerator
* ppc64 and pseries machine type
* s390x and s390-ccw-virtio machine type
The Avocado vmimage utils library is used to download and cache the Linux guest images, and from those images a snapshot image is created and given to QEMU. If a qemu-img binary is available in the build directory, it's used to create the snapshot image, so that matching qemu-system-* and qemu-img are used in the same test run. If qemu-img is not available in the build tree, one is attempted to be found installed system-wide (in the $PATH). If qemu-img is not found in the build dir or in the $PATH, the test is canceled.
The method for checking the successful boot is based on "cloudinit" and its "phone home" feature. The guest is given an ISO image with the location of the phone home server, and the information to post (the instance ID). Upon receiving the correct information, from the guest, the test is considered to have PASSed.
This test is currently limited to user mode networking only, and instructs the guest to connect to the "router" address that is hard coded in QEMU.
To create the cloudinit ISO image that will be used to configure the guest, the pycdlib library is also required and has been added as requirement to the virtual environment created by "check-venv".
The console output is read by a separate thread, by means of the Avocado datadrainer utility module.
--- .travis.yml | 2 +- tests/acceptance/boot_linux.py | 180 +++++++++++++++++++++++++++++++++ tests/requirements.txt | 3 +- 3 files changed, 183 insertions(+), 2 deletions(-) create mode 100644 tests/acceptance/boot_linux.py
.travis.yml +1 -1
@@ -264,7 +264,7 @@ matrix: # Acceptance (Functional) tests - env: - - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu" + - CONFIG="--python=/usr/bin/python3 --enable-tools --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu" - TEST_CMD="make check-acceptance" after_failure: - cat tests/results/latest/job.log
tests/acceptance/boot_linux.py new file +180 -0
180 lines not on this page · raw message
tests/requirements.txt +2 -1
@@ -1,4 +1,5 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -avocado-framework==72.0 +avocado-framework==73.0 +pycdlib==1.8.0
Signature · 1 line
2.21.0
Re: [PATCH v8 2/4] Acceptance test: add "boot_linux" tests
Hi Cleber,
Few minor questions...
Quoted 94 lines · Cleber Rosa
This acceptance test, validates that a full blown Linux guest can successfully boot in QEMU. In this specific case, the guest chosen is Fedora version 31.
*
x86_64, pc and q35 machine types, with and without kvm as an accelerator* aarch64 and virt machine type, with and without kvm as an accelerator
* ppc64 and pseries machine type
* s390x and s390-ccw-virtio machine type
The Avocado vmimage utils library is used to download and cache the Linux guest images, and from those images a snapshot image is created and given to QEMU. If a qemu-img binary is available in the build directory, it's used to create the snapshot image, so that matching qemu-system-* and qemu-img are used in the same test run. If qemu-img is not available in the build tree, one is attempted to be found installed system-wide (in the $PATH). If qemu-img is not found in the build dir or in the $PATH, the test is canceled.
The method for checking the successful boot is based on "cloudinit" and its "phone home" feature. The guest is given an ISO image with the location of the phone home server, and the information to post (the instance ID). Upon receiving the correct information, from the guest, the test is considered to have PASSed.
This test is currently limited to user mode networking only, and instructs the guest to connect to the "router" address that is hard coded in QEMU.
To create the cloudinit ISO image that will be used to configure the guest, the pycdlib library is also required and has been added as requirement to the virtual environment created by "check-venv".
The console output is read by a separate thread, by means of the Avocado datadrainer utility module.
Signed-off-by Cleber Rosa <[email protected]>--- .travis.yml | 2 +- tests/acceptance/boot_linux.py | 180 +++++++++++++++++++++++++++++++++ tests/requirements.txt | 3 +- 3 files changed, 183 insertions(+), 2 deletions(-)create mode 100644
tests/acceptance/boot_linux.py.travis.yml +1
-1@@ -264,7 +264,7 @@ matrix: # Acceptance (Functional) tests - env: - - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu" + - CONFIG="--python=/usr/bin/python3 --enable-tools --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu" - TEST_CMD="make check-acceptance" after_failure: - cat tests/results/latest/job.logtests/acceptance/boot_linux.py new file +27
-0@@ -0,0 +1,180 @@ +# Functional test that boots a complete Linux system via a cloud image +# +# Copyright (c) 2018-2019 Red Hat, Inc. +# +# Author: +# Cleber Rosa <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import os + +from avocado_qemu import Test, BLD_DIR + +from qemu.accel import kvm_available + +from avocado.utils import cloudinit +from avocado.utils import network +from avocado.utils import vmimage +from avocado.utils import datadrainer + + +KVM_NOT_AVAILABLE = "KVM accelerator does not seem to be available" + + +class BootLinux(Test): + """
tests/acceptance/boot_linux.py line 37
Quoted 10 lines
tests/acceptance/boot_linux.py new file +10
-0+ Boots a Linux system, checking for a successful initialization + """ + + timeout = 900 + chksum = None + + def setUp(self): + super(BootLinux, self).setUp() + self.prepare_boot() + self.vm.add_args('-smp', '2')
Hmmm are we assuming everybody has multicore systems?
tests/acceptance/boot_linux.py line 38
Quoted 1 line
tests/acceptance/boot_linux.py new file +1
-0+ self.vm.add_args('-m', '2048')
We should not fail the test if this condition is not possible.
Quoted 47 lines
tests/acceptance/boot_linux.py new file +47
-0+ self.vm.add_args('-drive', 'file=%s' % self.boot.path) + self.prepare_cloudinit() + + def prepare_boot(self): + self.log.info('Downloading/preparing boot image') + # Fedora 31 only provides ppc64le images + image_arch = self.arch + if image_arch == 'ppc64': + image_arch = 'ppc64le' + # If qemu-img has been built, use it, otherwise the system wide one + # will be used. If none is available, the test will cancel. + qemu_img = os.path.join(BLD_DIR, 'qemu-img') + if os.path.exists(qemu_img): + vmimage.QEMU_IMG = qemu_img + try: + self.boot = vmimage.get( + 'fedora', arch=image_arch, version='31', + checksum=self.chksum, + algorithm='sha256', + cache_dir=self.cache_dirs[0], + snapshot_dir=self.workdir) + except: + self.cancel('Failed to download/prepare boot image') + + def prepare_cloudinit(self): + self.log.info('Preparing cloudinit image') + try: + cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso') + self.phone_home_port = network.find_free_port() + cloudinit.iso(cloudinit_iso, self.name, + username='root', + password='password', + # QEMU's hard coded usermode router address + phone_home_host='10.0.2.2', + phone_home_port=self.phone_home_port) + self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso) + except Exception: + self.cancel('Failed to prepared cloudinit image') + + def launch_and_wait(self): + self.vm.set_console() + self.vm.launch() + console_drainer = datadrainer.LineLogger(self.vm.console_socket.fileno(), + logger=self.log.getChild('console')) + console_drainer.start() + self.log.info('VM launched, waiting for boot confirmation from guest') + cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), self.name)
tests/acceptance/boot_linux.py line 95
Quoted 10 lines
tests/acceptance/boot_linux.py new file +10
-0+ + +class BootLinuxX8664(BootLinux): + """ + :avocado: tags=arch:x86_64 + """ + + chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0' + + def test_pc(self):
I'd name this test_pc_i440fx_tcg, but are you sure the default is tcg?
tests/acceptance/boot_linux.py line 101
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ """ + :avocado: tags=machine:pc + """ + self.launch_and_wait() + + def test_kvm_pc(self):
This test_pc_i440fx_kvm
tests/acceptance/boot_linux.py line 111
Quoted 10 lines
tests/acceptance/boot_linux.py new file +10
-0+ """ + :avocado: tags=machine:pc + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm") + self.launch_and_wait() + + def test_q35(self):
This one test_pc_q35_tcg
tests/acceptance/boot_linux.py line 117
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ """ + :avocado: tags=machine:q35 + """ + self.launch_and_wait() + + def test_kvm_q35(self):
Here test_pc_q35_kvm.
Quoted 9 lines
tests/acceptance/boot_linux.py new file +9
-0+ """ + :avocado: tags=machine:q35 + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm") + self.launch_and_wait() +
tests/acceptance/boot_linux.py line 136
Quoted 10 lines
tests/acceptance/boot_linux.py new file +10
-0+ +class BootLinuxAarch64(BootLinux): + """ + :avocado: tags=arch:aarch64 + :avocado: tags=machine:virt + """ + + chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49' + + def test_virt(self):
We have other 'virt' machines:
$ git grep '"virt"'
hw/arm/virt.c:83: mc->alias = "virt"; \
hw/riscv/virt.c:613: .name = MACHINE_TYPE_NAME("virt"),
hw/xtensa/virt.c:135:DEFINE_MACHINE("virt", xtensa_virt_machine_init)Maybe rename test_aarch64_virt_tcg?
Quoted 16 lines
tests/acceptance/boot_linux.py new file +16
-0+ self.vm.add_args('-cpu', 'cortex-a53') + self.vm.add_args('-bios', + os.path.join(BLD_DIR, 'pc-bios', + 'edk2-aarch64-code.fd')) + self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') + self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom') + self.launch_and_wait() + + def test_kvm_virt(self): + """ + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm") + self.test_virt()
tests/acceptance/boot_linux.py line 162
Quoted 10 lines
tests/acceptance/boot_linux.py new file +10
-0+ + +class BootLinuxPPC64(BootLinux): + """ + :avocado: tags=arch:ppc64 + """ + + chksum = '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58' + + def test_pseries(self):
Rename as test_ppc64el_pseries_tcg?
Quoted 30 lines
tests/acceptance/boot_linux.py new file +18
-0+ """ + :avocado: tags=machine:pseries + """ + self.launch_and_wait() + + +class BootLinuxS390X(BootLinux): + """ + :avocado: tags=arch:s390x + """ + + chksum = '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d' + + def test_s390_ccw_virtio(self): + """ + :avocado: tags=machine:s390-ccw-virtio + """ + self.launch_and_wait()tests/requirements.txt +2
-1@@ -1,4 +1,5 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -avocado-framework==72.0 +avocado-framework==73.0 +pycdlib==1.8.0
Re: [PATCH v8 2/4] Acceptance test: add "boot_linux" tests
Quoted 102 lines · Philippe Mathieu-Daudé
Hi Cleber,
Few minor questions...
Quoted 97 lines · Cleber Rosa
This acceptance test, validates that a full blown Linux guest can successfully boot in QEMU. In this specific case, the guest chosen is Fedora version 31.
*
x86_64, pc and q35 machine types, with and without kvm as an accelerator* aarch64 and virt machine type, with and without kvm as an accelerator
* ppc64 and pseries machine type
* s390x and s390-ccw-virtio machine type
The Avocado vmimage utils library is used to download and cache the Linux guest images, and from those images a snapshot image is created and given to QEMU. If a qemu-img binary is available in the build directory, it's used to create the snapshot image, so that matching qemu-system-* and qemu-img are used in the same test run. If qemu-img is not available in the build tree, one is attempted to be found installed system-wide (in the $PATH). If qemu-img is not found in the build dir or in the $PATH, the test is canceled.
The method for checking the successful boot is based on "cloudinit" and its "phone home" feature. The guest is given an ISO image with the location of the phone home server, and the information to post (the instance ID). Upon receiving the correct information, from the guest, the test is considered to have PASSed.
This test is currently limited to user mode networking only, and instructs the guest to connect to the "router" address that is hard coded in QEMU.
To create the cloudinit ISO image that will be used to configure the guest, the pycdlib library is also required and has been added as requirement to the virtual environment created by "check-venv".
The console output is read by a separate thread, by means of the Avocado datadrainer utility module.
Signed-off-by Cleber Rosa <[email protected]>--- .travis.yml | 2 +- tests/acceptance/boot_linux.py | 180 +++++++++++++++++++++++++++++++++ tests/requirements.txt | 3 +- 3 files changed, 183 insertions(+), 2 deletions(-)create mode 100644
tests/acceptance/boot_linux.py.travis.yml +1
-1@@ -264,7 +264,7 @@ matrix: # Acceptance (Functional) tests - env: - - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu" + - CONFIG="--python=/usr/bin/python3 --enable-tools --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu" - TEST_CMD="make check-acceptance" after_failure: - cat tests/results/latest/job.logtests/acceptance/boot_linux.py new file +31
-0@@ -0,0 +1,180 @@ +# Functional test that boots a complete Linux system via a cloud image +# +# Copyright (c) 2018-2019 Red Hat, Inc. +# +# Author: +# Cleber Rosa <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import os + +from avocado_qemu import Test, BLD_DIR + +from qemu.accel import kvm_available + +from avocado.utils import cloudinit +from avocado.utils import network +from avocado.utils import vmimage +from avocado.utils import datadrainer + + +KVM_NOT_AVAILABLE = "KVM accelerator does not seem to be available" + + +class BootLinux(Test): + """ + Boots a Linux system, checking for a successful initialization + """ + + timeout = 900
Quoted 9 lines
tests/acceptance/boot_linux.py line 37
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ chksum = None + + def setUp(self): + super(BootLinux, self).setUp() + self.prepare_boot() + self.vm.add_args('-smp', '2')Hmmm are we assuming everybody has multicore systems?
Not really, but isn't it possible to have virtual CPUs > actual CPUs? IMO testing with smp > 1 is a better test than with smp == 1.
Quoted 4 lines
tests/acceptance/boot_linux.py line 38
Quoted 1 line
tests/acceptance/boot_linux.py new file +1
-0+ self.vm.add_args('-m', '2048')We should not fail the test if this condition is not possible.
You mean from the host side, right? I have doubts about what to do here, in the sense that we can't easily and reliably set aside memory in the system to run qemu. We could of course check the amount of physical or free memory in the system at the test start time, but there would still be somewhat of a race condition.
Also, with tests running in parallel (ie, avocado nrun tests/acceptance/), this would be even trickier...
Quoted 51 lines
Quoted 51 lines
tests/acceptance/boot_linux.py new file +51
-0+ self.vm.add_args('-drive', 'file=%s' % self.boot.path) + self.prepare_cloudinit() + + def prepare_boot(self): + self.log.info('Downloading/preparing boot image') + # Fedora 31 only provides ppc64le images + image_arch = self.arch + if image_arch == 'ppc64': + image_arch = 'ppc64le' + # If qemu-img has been built, use it, otherwise the system wide one + # will be used. If none is available, the test will cancel. + qemu_img = os.path.join(BLD_DIR, 'qemu-img') + if os.path.exists(qemu_img): + vmimage.QEMU_IMG = qemu_img + try: + self.boot = vmimage.get( + 'fedora', arch=image_arch, version='31', + checksum=self.chksum, + algorithm='sha256', + cache_dir=self.cache_dirs[0], + snapshot_dir=self.workdir) + except: + self.cancel('Failed to download/prepare boot image') + + def prepare_cloudinit(self): + self.log.info('Preparing cloudinit image') + try: + cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso') + self.phone_home_port = network.find_free_port() + cloudinit.iso(cloudinit_iso, self.name, + username='root', + password='password', + # QEMU's hard coded usermode router address + phone_home_host='10.0.2.2', + phone_home_port=self.phone_home_port) + self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso) + except Exception: + self.cancel('Failed to prepared cloudinit image') + + def launch_and_wait(self): + self.vm.set_console() + self.vm.launch() + console_drainer = datadrainer.LineLogger(self.vm.console_socket.fileno(), + logger=self.log.getChild('console')) + console_drainer.start() + self.log.info('VM launched, waiting for boot confirmation from guest') + cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), self.name) + + +class BootLinuxX8664(BootLinux): + """
Quoted 9 lines
tests/acceptance/boot_linux.py line 95
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ :avocado: tags=arch:x86_64 + """ + + chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0' + + def test_pc(self):I'd name this
test_pc_i440fx_tcg, but are you sure the default is tcg?
Even if there's no matching machine type named "pc-i440fx"?
Quoted 9 lines
tests/acceptance/boot_linux.py line 101
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ """ + :avocado: tags=machine:pc + """ + self.launch_and_wait() + + def test_kvm_pc(self):This
test_pc_i440fx_kvm
Ditto.
Quoted 11 lines
tests/acceptance/boot_linux.py line 111
Quoted 10 lines
tests/acceptance/boot_linux.py new file +10
-0+ """ + :avocado: tags=machine:pc + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm") + self.launch_and_wait() + + def test_q35(self):
Quoted 2 lines
This one
test_pc_q35_tcg
Both *pc* and *q35*? I'm missing something here. +1 for the explicit "tcg" along with an also explicit check for the availability of the tcg accel.
Quoted 9 lines
tests/acceptance/boot_linux.py line 117
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ """ + :avocado: tags=machine:q35 + """ + self.launch_and_wait() + + def test_kvm_q35(self):Here
test_pc_q35_kvm.
I don't get the "pc" part here.
Quoted 22 lines
Quoted 9 lines
tests/acceptance/boot_linux.py new file +9
-0+ """ + :avocado: tags=machine:q35 + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm") + self.launch_and_wait() +tests/acceptance/boot_linux.py line 136
Quoted 10 lines
tests/acceptance/boot_linux.py new file +10
-0+ +class BootLinuxAarch64(BootLinux): + """ + :avocado: tags=arch:aarch64 + :avocado: tags=machine:virt + """ + + chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49' + + def test_virt(self):We have other 'virt' machines:
Quoted 7 lines
$ git grep '"virt"' hw/arm/virt.c:83: mc->alias = "virt"; \ hw/riscv/virt.c:613: .name = MACHINE_TYPE_NAME("virt"), hw/xtensa/virt.c:135:DEFINE_MACHINE("virt", xtensa_virt_machine_init)Maybe rename
test_aarch64_virt_tcg?
The "test name", or "test ID" includes the class name, so that already makes it clear (IMO) that this test is about the "virt" machine type for the "aarch64" arch.
Quoted 20 lines
Quoted 20 lines
tests/acceptance/boot_linux.py new file +20
-0+ self.vm.add_args('-cpu', 'cortex-a53') + self.vm.add_args('-bios', + os.path.join(BLD_DIR, 'pc-bios', + 'edk2-aarch64-code.fd')) + self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') + self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom') + self.launch_and_wait() + + def test_kvm_virt(self): + """ + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm") + self.test_virt() + + +class BootLinuxPPC64(BootLinux): + """
Quoted 9 lines
tests/acceptance/boot_linux.py line 162
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ :avocado: tags=arch:ppc64 + """ + + chksum = '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58' + + def test_pseries(self):Rename as
test_ppc64el_pseries_tcg?
Same here (class name contains the arch).
- Cleber.
Quoted 32 lines
Quoted 30 lines
tests/acceptance/boot_linux.py new file +18
-0+ """ + :avocado: tags=machine:pseries + """ + self.launch_and_wait() + + +class BootLinuxS390X(BootLinux): + """ + :avocado: tags=arch:s390x + """ + + chksum = '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d' + + def test_s390_ccw_virtio(self): + """ + :avocado: tags=machine:s390-ccw-virtio + """ + self.launch_and_wait()tests/requirements.txt +2
-1@@ -1,4 +1,5 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -avocado-framework==72.0 +avocado-framework==73.0 +pycdlib==1.8.0
Re: [PATCH v8 2/4] Acceptance test: add "boot_linux" tests
Quoted 113 lines · Cleber Rosa
Quoted 102 lines · Philippe Mathieu-Daudé
Hi Cleber,
Few minor questions...
Quoted 97 lines · Cleber Rosa
This acceptance test, validates that a full blown Linux guest can successfully boot in QEMU. In this specific case, the guest chosen is Fedora version 31.
*
x86_64, pc and q35 machine types, with and without kvm as an accelerator* aarch64 and virt machine type, with and without kvm as an accelerator
* ppc64 and pseries machine type
* s390x and s390-ccw-virtio machine type
The Avocado vmimage utils library is used to download and cache the Linux guest images, and from those images a snapshot image is created and given to QEMU. If a qemu-img binary is available in the build directory, it's used to create the snapshot image, so that matching qemu-system-* and qemu-img are used in the same test run. If qemu-img is not available in the build tree, one is attempted to be found installed system-wide (in the $PATH). If qemu-img is not found in the build dir or in the $PATH, the test is canceled.
The method for checking the successful boot is based on "cloudinit" and its "phone home" feature. The guest is given an ISO image with the location of the phone home server, and the information to post (the instance ID). Upon receiving the correct information, from the guest, the test is considered to have PASSed.
This test is currently limited to user mode networking only, and instructs the guest to connect to the "router" address that is hard coded in QEMU.
To create the cloudinit ISO image that will be used to configure the guest, the pycdlib library is also required and has been added as requirement to the virtual environment created by "check-venv".
The console output is read by a separate thread, by means of the Avocado datadrainer utility module.
Signed-off-by Cleber Rosa <[email protected]>--- .travis.yml | 2 +- tests/acceptance/boot_linux.py | 180 +++++++++++++++++++++++++++++++++ tests/requirements.txt | 3 +- 3 files changed, 183 insertions(+), 2 deletions(-)create mode 100644
tests/acceptance/boot_linux.py.travis.yml +1
-1@@ -264,7 +264,7 @@ matrix: # Acceptance (Functional) tests - env: - - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu" + - CONFIG="--python=/usr/bin/python3 --enable-tools --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu" - TEST_CMD="make check-acceptance" after_failure: - cat tests/results/latest/job.logtests/acceptance/boot_linux.py new file +31
-0@@ -0,0 +1,180 @@ +# Functional test that boots a complete Linux system via a cloud image +# +# Copyright (c) 2018-2019 Red Hat, Inc. +# +# Author: +# Cleber Rosa <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import os + +from avocado_qemu import Test, BLD_DIR + +from qemu.accel import kvm_available + +from avocado.utils import cloudinit +from avocado.utils import network +from avocado.utils import vmimage +from avocado.utils import datadrainer + + +KVM_NOT_AVAILABLE = "KVM accelerator does not seem to be available" + + +class BootLinux(Test): + """ + Boots a Linux system, checking for a successful initialization + """ + + timeout = 900Quoted 9 lines
tests/acceptance/boot_linux.py line 37
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ chksum = None + + def setUp(self): + super(BootLinux, self).setUp() + self.prepare_boot() + self.vm.add_args('-smp', '2')Hmmm are we assuming everybody has multicore systems?
Quoted 2 lines
Not really, but isn't it possible to have virtual CPUs > actual CPUs? IMO testing with smp > 1 is a better test than with smp == 1.
I guess it is unusual to run acceptance tests on non-multicore systems, so this is probably not an issue.
Quoted 10 lines
Quoted 4 lines
tests/acceptance/boot_linux.py line 38
Quoted 1 line
tests/acceptance/boot_linux.py new file +1
-0+ self.vm.add_args('-m', '2048')We should not fail the test if this condition is not possible.
You mean from the host side, right? I have doubts about what to do here, in the sense that we can't easily and reliably set aside memory in the system to run qemu. We could of course check the amount of physical or free memory in the system at the test start time, but there would still be somewhat of a race condition.
See:
https://www.mail-archive.com/[email protected]/msg652535.html
Quoted 7 lines · Peter Maydell
Hi. I just discovered that this makes 'make check' fail on 32-bit systems, because you can't default to 2GB of RAM for a board:
(armhf)pmaydell@mustang-maydell:~/qemu$ ./build/all-a32/arm-softmmu/qemu-system-arm -M ast2600-evb
qemu-system-arm: at most 2047 MB RAM can be simulated
Looking at:
https://docs.python.org/2/library/platform.html#cross-platform
Maybe we can use:
@skipUnless(sys.maxsize > 2**32, 'Require 64-bit host')
Quoted 3 lines
Also, with tests running in parallel (ie, avocado nrun tests/acceptance/), this would be even trickier...
Yeah this is a complex case. Maybe for now we can add a tag for the minimum of memory required for a test. Then later it might be easier figure something.
This class can help:
https://psutil.readthedocs.io/en/latest/#psutil.virtual_memory
Quoted 61 lines
Quoted 51 lines
Quoted 51 lines
tests/acceptance/boot_linux.py new file +51
-0+ self.vm.add_args('-drive', 'file=%s' % self.boot.path) + self.prepare_cloudinit() + + def prepare_boot(self): + self.log.info('Downloading/preparing boot image') + # Fedora 31 only provides ppc64le images + image_arch = self.arch + if image_arch == 'ppc64': + image_arch = 'ppc64le' + # If qemu-img has been built, use it, otherwise the system wide one + # will be used. If none is available, the test will cancel. + qemu_img = os.path.join(BLD_DIR, 'qemu-img') + if os.path.exists(qemu_img): + vmimage.QEMU_IMG = qemu_img + try: + self.boot = vmimage.get( + 'fedora', arch=image_arch, version='31', + checksum=self.chksum, + algorithm='sha256', + cache_dir=self.cache_dirs[0], + snapshot_dir=self.workdir) + except: + self.cancel('Failed to download/prepare boot image') + + def prepare_cloudinit(self): + self.log.info('Preparing cloudinit image') + try: + cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso') + self.phone_home_port = network.find_free_port() + cloudinit.iso(cloudinit_iso, self.name, + username='root', + password='password', + # QEMU's hard coded usermode router address + phone_home_host='10.0.2.2', + phone_home_port=self.phone_home_port) + self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso) + except Exception: + self.cancel('Failed to prepared cloudinit image') + + def launch_and_wait(self): + self.vm.set_console() + self.vm.launch() + console_drainer = datadrainer.LineLogger(self.vm.console_socket.fileno(), + logger=self.log.getChild('console')) + console_drainer.start() + self.log.info('VM launched, waiting for boot confirmation from guest') + cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), self.name) + + +class BootLinuxX8664(BootLinux): + """Quoted 9 lines
tests/acceptance/boot_linux.py line 95
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ :avocado: tags=arch:x86_64 + """ + + chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0' + + def test_pc(self):I'd name this
test_pc_i440fx_tcg, but are you sure the default is tcg?
Quoted 1 line
Even if there's no matching machine type named "pc-i440fx"?
Hmm 'pc' is a confusing alias, kept for backward compatibility... 'q35' is also an alias, see:
$ qemu-system-x86_64 -M help Supported machines are: pc Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-3.1) pc-i440fx-3.1 Standard PC (i440FX + PIIX, 1996) (default) pc-i440fx-3.0 Standard PC (i440FX + PIIX, 1996) pc-i440fx-2.9 Standard PC (i440FX + PIIX, 1996) ... q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-3.1) pc-q35-3.1 Standard PC (Q35 + ICH9, 2009) pc-q35-3.0 Standard PC (Q35 + ICH9, 2009) pc-q35-2.9 Standard PC (Q35 + ICH9, 2009) pc-q35-2.8 Standard PC (Q35 + ICH9, 2009) ... isapc ISA-only PC none empty machine xenfv Xen Fully-virtualized PC xenpv Xen Para-virtualized PC
Historically the 'pc' machine was aiming at modeling a 'PC compatible' machine, as described here:
https://en.wikipedia.org/wiki/IBM_PC_compatible
We currently have 3 emulated PC machines (excluding the virtual machines such MicroVM, Xen).
Some history:
- commit 0824d6fc6
The first pc machine is introduced, inside main(). It only use an ISA bus. Today we know this as 'isapc'.
- commit 80cabfad1
Refactor, the pc machine is extracted to hw/pc.c
- commit 69b910399
PCI support is added.
- commit bb0c6722b
PCI is used by default. The machine becomes what we today calls 'pc'.
- commit b5ff2d6e2
The machine is officially called 'pc'.
- commit 3dbbdc255
The 'isapc' machine is added back: it is the 'pc' machine with PCI disabled.
- commit df2d8b3ed
The 'q35' machine is added.
Instead of a i440FX northbridge and PIIX3 southbridge chipsets, it uses a Q35 northbridge and ICH9 southbridge.
See: https://wiki.qemu.org/Features/Q35
So 'pc' is more a 'family' of different machines.
You can verify the codebase, the abstract parent is in hw/i386/pc.c:
static const TypeInfo pc_machine_info = {
.name = TYPE_PC_MACHINE,
.parent = TYPE_X86_MACHINE,
.abstract = true,
.instance_size = sizeof(PCMachineState),
While the children implementations are hw/i386/pc_piix.c and hw/i386/pc_q35.c (pc_piix.c implements both isapc + i440fx).
I'm not sure why one is named by its southbridge chipset, and the other one by its northbridge. I suppose because for a long time the i440fx+piix were mixed in the same file. I cleaned that recently in commits 0fd61a2d1..0f25d865a1:
$ git log --oneline --reverse 0fd61a2d1~..0f25d865a1 0fd61a2d1c hw/pci-host/piix: Move i440FX declarations to hw/pci-host/i440fx.h 553b4559dc hw/pci-host/piix: Fix code style issues 14a026dd58 hw/pci-host/piix: Extract PIIX3 functions to hw/isa/piix3.c 0f25d865a1 hw/pci-host: Rename incorrectly named 'piix' as 'i440fx'
Quoted 69 lines
Quoted 9 lines
tests/acceptance/boot_linux.py line 101
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ """ + :avocado: tags=machine:pc + """ + self.launch_and_wait() + + def test_kvm_pc(self):This
test_pc_i440fx_kvmDitto.
Quoted 11 lines
tests/acceptance/boot_linux.py line 111
Quoted 10 lines
tests/acceptance/boot_linux.py new file +10
-0+ """ + :avocado: tags=machine:pc + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm") + self.launch_and_wait() + + def test_q35(self):Quoted 2 lines
This one
test_pc_q35_tcgBoth *pc* and *q35*? I'm missing something here. +1 for the explicit "tcg" along with an also explicit check for the availability of the tcg accel.
Quoted 9 lines
tests/acceptance/boot_linux.py line 117
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ """ + :avocado: tags=machine:q35 + """ + self.launch_and_wait() + + def test_kvm_q35(self):Here
test_pc_q35_kvm.I don't get the "pc" part here.
Quoted 27 lines
Quoted 9 lines
tests/acceptance/boot_linux.py new file +9
-0+ """ + :avocado: tags=machine:q35 + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm") + self.launch_and_wait() +tests/acceptance/boot_linux.py line 136
Quoted 10 lines
tests/acceptance/boot_linux.py new file +10
-0+ +class BootLinuxAarch64(BootLinux): + """ + :avocado: tags=arch:aarch64 + :avocado: tags=machine:virt + """ + + chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49' + + def test_virt(self):We have other 'virt' machines:
$ git grep '"virt"' hw/arm/virt.c:83: mc->alias = "virt"; \ hw/riscv/virt.c:613: .name = MACHINE_TYPE_NAME("virt"), hw/xtensa/virt.c:135:DEFINE_MACHINE("virt", xtensa_virt_machine_init)
Quoted 6 lines
Quoted 2 lines
Maybe rename
test_aarch64_virt_tcg?The "test name", or "test ID" includes the class name, so that already makes it clear (IMO) that this test is about the "virt" machine type for the "aarch64" arch.
OK, good point. 'tcg' suffix still?
Quoted 30 lines
Quoted 20 lines
Quoted 20 lines
tests/acceptance/boot_linux.py new file +20
-0+ self.vm.add_args('-cpu', 'cortex-a53') + self.vm.add_args('-bios', + os.path.join(BLD_DIR, 'pc-bios', + 'edk2-aarch64-code.fd')) + self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') + self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom') + self.launch_and_wait() + + def test_kvm_virt(self): + """ + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm") + self.test_virt() + + +class BootLinuxPPC64(BootLinux): + """Quoted 9 lines
tests/acceptance/boot_linux.py line 162
Quoted 6 lines
tests/acceptance/boot_linux.py new file +6
-0+ :avocado: tags=arch:ppc64 + """ + + chksum = '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58' + + def test_pseries(self):Rename as
test_ppc64el_pseries_tcg?
Quoted 1 line
Same here (class name contains the arch).
Yes. 'tcg' suffix still?
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
tests/acceptance/boot_linux.py new file +18
-0+ """ + :avocado: tags=machine:pseries + """ + self.launch_and_wait() + + +class BootLinuxS390X(BootLinux): + """ + :avocado: tags=arch:s390x + """ + + chksum = '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d' + + def test_s390_ccw_virtio(self): + """ + :avocado: tags=machine:s390-ccw-virtio + """ + self.launch_and_wait()tests/requirements.txt +2
-1@@ -1,4 +1,5 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -avocado-framework==72.0 +avocado-framework==73.0 +pycdlib==1.8.0
Re: [PATCH v8 2/4] Acceptance test: add "boot_linux" tests
In reply to Cleber RosaHi Cleber,
just few comments below.
Quoted 96 lines · Cleber Rosa
This acceptance test, validates that a full blown Linux guest can successfully boot in QEMU. In this specific case, the guest chosen is Fedora version 31.
*
x86_64, pc and q35 machine types, with and without kvm as an accelerator* aarch64 and virt machine type, with and without kvm as an accelerator
* ppc64 and pseries machine type
* s390x and s390-ccw-virtio machine type
The Avocado vmimage utils library is used to download and cache the Linux guest images, and from those images a snapshot image is created and given to QEMU. If a qemu-img binary is available in the build directory, it's used to create the snapshot image, so that matching qemu-system-* and qemu-img are used in the same test run. If qemu-img is not available in the build tree, one is attempted to be found installed system-wide (in the $PATH). If qemu-img is not found in the build dir or in the $PATH, the test is canceled.
The method for checking the successful boot is based on "cloudinit" and its "phone home" feature. The guest is given an ISO image with the location of the phone home server, and the information to post (the instance ID). Upon receiving the correct information, from the guest, the test is considered to have PASSed.
This test is currently limited to user mode networking only, and instructs the guest to connect to the "router" address that is hard coded in QEMU.
To create the cloudinit ISO image that will be used to configure the guest, the pycdlib library is also required and has been added as requirement to the virtual environment created by "check-venv".
The console output is read by a separate thread, by means of the Avocado datadrainer utility module.
Signed-off-by Cleber Rosa <[email protected]>--- .travis.yml | 2 +- tests/acceptance/boot_linux.py | 180 +++++++++++++++++++++++++++++++++ tests/requirements.txt | 3 +- 3 files changed, 183 insertions(+), 2 deletions(-)create mode 100644
tests/acceptance/boot_linux.py.travis.yml +1
-1@@ -264,7 +264,7 @@ matrix: # Acceptance (Functional) tests - env: - - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu" + - CONFIG="--python=/usr/bin/python3 --enable-tools --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu" - TEST_CMD="make check-acceptance" after_failure: - cat tests/results/latest/job.logtests/acceptance/boot_linux.py new file +29
-0@@ -0,0 +1,180 @@ +# Functional test that boots a complete Linux system via a cloud image +# +# Copyright (c) 2018-2019 Red Hat, Inc. +# +# Author: +# Cleber Rosa <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import os + +from avocado_qemu import Test, BLD_DIR + +from qemu.accel import kvm_available + +from avocado.utils import cloudinit +from avocado.utils import network +from avocado.utils import vmimage +from avocado.utils import datadrainer + + +KVM_NOT_AVAILABLE = "KVM accelerator does not seem to be available" + + +class BootLinux(Test): + """ + Boots a Linux system, checking for a successful initialization + """
tests/acceptance/boot_linux.py line 39
Quoted 10 lines
tests/acceptance/boot_linux.py new file +10
-0+ + timeout = 900 + chksum = None + + def setUp(self): + super(BootLinux, self).setUp() + self.prepare_boot() + self.vm.add_args('-smp', '2') + self.vm.add_args('-m', '2048') + self.vm.add_args('-drive', 'file=%s' % self.boot.path)
Perhaps move above line to prepare_boot() then following the same logic as in prepare_cloudinit() - which sets the -drive path to the ISO file.
Quoted 75 lines
tests/acceptance/boot_linux.py new file +75
-0+ self.prepare_cloudinit() + + def prepare_boot(self): + self.log.info('Downloading/preparing boot image') + # Fedora 31 only provides ppc64le images + image_arch = self.arch + if image_arch == 'ppc64': + image_arch = 'ppc64le' + # If qemu-img has been built, use it, otherwise the system wide one + # will be used. If none is available, the test will cancel. + qemu_img = os.path.join(BLD_DIR, 'qemu-img') + if os.path.exists(qemu_img): + vmimage.QEMU_IMG = qemu_img + try: + self.boot = vmimage.get( + 'fedora', arch=image_arch, version='31', + checksum=self.chksum, + algorithm='sha256', + cache_dir=self.cache_dirs[0], + snapshot_dir=self.workdir) + except: + self.cancel('Failed to download/prepare boot image') + + def prepare_cloudinit(self): + self.log.info('Preparing cloudinit image') + try: + cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso') + self.phone_home_port = network.find_free_port() + cloudinit.iso(cloudinit_iso, self.name, + username='root', + password='password', + # QEMU's hard coded usermode router address + phone_home_host='10.0.2.2', + phone_home_port=self.phone_home_port) + self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso) + except Exception: + self.cancel('Failed to prepared cloudinit image') + + def launch_and_wait(self): + self.vm.set_console() + self.vm.launch() + console_drainer = datadrainer.LineLogger(self.vm.console_socket.fileno(), + logger=self.log.getChild('console')) + console_drainer.start() + self.log.info('VM launched, waiting for boot confirmation from guest') + cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), self.name) + + +class BootLinuxX8664(BootLinux): + """ + :avocado: tags=arch:x86_64 + """ + + chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0' + + def test_pc(self): + """ + :avocado: tags=machine:pc + """ + self.launch_and_wait() + + def test_kvm_pc(self): + """ + :avocado: tags=machine:pc + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm") + self.launch_and_wait() + + def test_q35(self): + """ + :avocado: tags=machine:q35 + """
tests/acceptance/boot_linux.py line 124
Quoted 10 lines
tests/acceptance/boot_linux.py new file +10
-0+ self.launch_and_wait() + + def test_kvm_q35(self): + """ + :avocado: tags=machine:q35 + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm")
Following patch will allow to reduce the above boilerplate:
https://www.mail-archive.com/[email protected]/msg666239.html
Quoted 67 lines
tests/acceptance/boot_linux.py new file +56
-0+ self.launch_and_wait() + + +class BootLinuxAarch64(BootLinux): + """ + :avocado: tags=arch:aarch64 + :avocado: tags=machine:virt + """ + + chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49' + + def test_virt(self): + self.vm.add_args('-cpu', 'cortex-a53') + self.vm.add_args('-bios', + os.path.join(BLD_DIR, 'pc-bios', + 'edk2-aarch64-code.fd')) + self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0') + self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom') + self.launch_and_wait() + + def test_kvm_virt(self): + """ + :avocado: tags=accel:kvm + """ + if not kvm_available(self.arch, self.qemu_bin): + self.cancel(KVM_NOT_AVAILABLE) + self.vm.add_args("-accel", "kvm") + self.test_virt() + + +class BootLinuxPPC64(BootLinux): + """ + :avocado: tags=arch:ppc64 + """ + + chksum = '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58' + + def test_pseries(self): + """ + :avocado: tags=machine:pseries + """ + self.launch_and_wait() + + +class BootLinuxS390X(BootLinux): + """ + :avocado: tags=arch:s390x + """ + + chksum = '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d' + + def test_s390_ccw_virtio(self): + """ + :avocado: tags=machine:s390-ccw-virtio + """ + self.launch_and_wait()tests/requirements.txt +2
-1@@ -1,4 +1,5 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -avocado-framework==72.0 +avocado-framework==73.0 +pycdlib==1.8.0
PATCH v8 3/4 Acceptance tests: add make targets to download images
In reply to Cleber RosaThe newly introduced "boot linux" tests make use of Linux images that are larger than usual, and fall into what Avocado calls "vmimages", and can be referred to by name, version and architecture.
The images can be downloaded automatically during the test. But, to make for more reliable test results, this introduces a target that will download the vmimages for the architectures that have been configured and are available for the currently used distro (Fedora 31).
--- tests/Makefile.include | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
tests/Makefile.include +15 -2
@@ -1177,7 +1177,20 @@ $(TESTS_RESULTS_DIR): check-venv: $(TESTS_VENV_DIR) -check-acceptance: check-venv $(TESTS_RESULTS_DIR) +FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(TARGETS)) +FEDORA_31_ARCHES := x86_64 aarch64 ppc64le s390x +FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES)) + +# download one specific Fedora 31 image +get-vmimage-fedora-31-%: $(check-venv) + $(call quiet-command, \ + $(TESTS_VENV_DIR)/bin/python -m avocado vmimage get \ + --distro=fedora --distro-version=31 --arch=$*) + +# download all vm images, according to defined targets +get-vmimage: $(patsubst %,get-vmimage-fedora-31-%, $(FEDORA_31_DOWNLOAD)) + +check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vmimage $(call quiet-command, \ $(TESTS_VENV_DIR)/bin/python -m avocado \ --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
@@ -1188,7 +1201,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) # Consolidated targets -.PHONY: check-block check-qapi-schema check-qtest check-unit check check-clean +.PHONY: check-block check-qapi-schema check-qtest check-unit check check-clean get-vmimage check-qapi-schema: check-tests/qapi-schema/frontend check-tests/qapi-schema/doc-good.texi check-qtest: $(patsubst %,check-qtest-%, $(QTEST_TARGETS)) check-block: $(patsubst %,check-%, $(check-block-y))
Signature · 1 line
2.21.0
Re: [PATCH v8 3/4] Acceptance tests: add make targets to download images
Quoted 43 lines · Cleber Rosa
The newly introduced "boot linux" tests make use of Linux images that are larger than usual, and fall into what Avocado calls "vmimages", and can be referred to by name, version and architecture.
The images can be downloaded automatically during the test. But, to make for more reliable test results, this introduces a target that will download the vmimages for the architectures that have been configured and are available for the currently used distro (Fedora 31).
Signed-off-by Cleber Rosa <[email protected]>--- tests/Makefile.include | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)tests/Makefile.include +14
-1@@ -1177,7 +1177,20 @@ $(TESTS_RESULTS_DIR): check-venv: $(TESTS_VENV_DIR) -check-acceptance: check-venv $(TESTS_RESULTS_DIR) +FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(TARGETS)) +FEDORA_31_ARCHES := x86_64 aarch64 ppc64le s390x +FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES)) + +# download one specific Fedora 31 image +get-vmimage-fedora-31-%: $(check-venv) + $(call quiet-command, \ + $(TESTS_VENV_DIR)/bin/python -m avocado vmimage get \ + --distro=fedora --distro-version=31 --arch=$*) + +# download all vm images, according to defined targets +get-vmimage: $(patsubst %,get-vmimage-fedora-31-%, $(FEDORA_31_DOWNLOAD)) + +check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vmimage $(call quiet-command, \ $(TESTS_VENV_DIR)/bin/python -m avocado \ --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
tests/Makefile.include check-acceptance: check-venv $(TESTS_RESULTS_DIR) · line 1207
Quoted 8 lines
tests/Makefile.include +1
-1@@ -1188,7 +1201,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) # Consolidated targets -.PHONY: check-block check-qapi-schema check-qtest check-unit check check-clean +.PHONY: check-block check-qapi-schema check-qtest check-unit check check-clean get-vmimage check-qapi-schema: check-tests/qapi-schema/frontend check-tests/qapi-schema/doc-good.texi check-qtest: $(patsubst %,check-qtest-%, $(QTEST_TARGETS)) check-block: $(patsubst %,check-%, $(check-block-y))
We have both 'make vm-help' and 'make check-help'. The check-acceptance target is in check-help. We get vm image... confusing.
Anyway, can you list this new target, with a hint about the storage size required? Can you add an entry in the 'make
Re: [PATCH v8 3/4] Acceptance tests: add make targets to download images
Quoted 43 lines · Philippe Mathieu-Daudé
Quoted 42 lines · Cleber Rosa
The newly introduced "boot linux" tests make use of Linux images that are larger than usual, and fall into what Avocado calls "vmimages", and can be referred to by name, version and architecture.
The images can be downloaded automatically during the test. But, to make for more reliable test results, this introduces a target that will download the vmimages for the architectures that have been configured and are available for the currently used distro (Fedora 31).
Signed-off-by Cleber Rosa <[email protected]>--- tests/Makefile.include | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)tests/Makefile.include +14
-2@@ -1177,7 +1177,20 @@ $(TESTS_RESULTS_DIR): check-venv: $(TESTS_VENV_DIR) -check-acceptance: check-venv $(TESTS_RESULTS_DIR) +FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(TARGETS)) +FEDORA_31_ARCHES := x86_64 aarch64 ppc64le s390x +FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES)) + +# download one specific Fedora 31 image +get-vmimage-fedora-31-%: $(check-venv) + $(call quiet-command, \ + $(TESTS_VENV_DIR)/bin/python -m avocado vmimage get \ + --distro=fedora --distro-version=31 --arch=$*) + +# download all vm images, according to defined targets +get-vmimage: $(patsubst %,get-vmimage-fedora-31-%, $(FEDORA_31_DOWNLOAD)) + +check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vmimage $(call quiet-command, \ $(TESTS_VENV_DIR)/bin/python -m avocado \ --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \@@ -1188,7 +1201,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) # Consolidated targets -.PHONY: check-block check-qapi-schema check-qtest check-unit check check-clean
Quoted 9 lines
tests/Makefile.include check-acceptance: check-venv $(TESTS_RESULTS_DIR)
Quoted 5 lines
tests/Makefile.include +1
-0+.PHONY: check-block check-qapi-schema check-qtest check-unit check check-clean get-vmimage check-qapi-schema: check-tests/qapi-schema/frontend check-tests/qapi-schema/doc-good.texi check-qtest: $(patsubst %,check-qtest-%, $(QTEST_TARGETS)) check-block: $(patsubst %,check-%, $(check-block-y))We have both 'make vm-help' and 'make check-help'. The check-acceptance target is in check-help. We get vm image... confusing.
I know... I had a hard time coming up with a name, and I'm aware it's not a very good one.
Quoted 2 lines
Anyway, can you list this new target, with a hint about the storage size required?
Sure thing, good point.
Quoted 2 lines
Can you add an entry in the 'make
I suspect you mean adding an entry in the 'make check-help' output, right?
- Cleber.
Re: [PATCH v8 3/4] Acceptance tests: add make targets to download images
Quoted 17 lines · Cleber Rosa
Quoted 16 lines · Philippe Mathieu-Daudé
Quoted 15 lines · Cleber Rosa
The newly introduced "boot linux" tests make use of Linux images that are larger than usual, and fall into what Avocado calls "vmimages", and can be referred to by name, version and architecture.
The images can be downloaded automatically during the test. But, to make for more reliable test results, this introduces a target that will download the vmimages for the architectures that have been configured and are available for the currently used distro (Fedora 31).
Signed-off-by Cleber Rosa <[email protected]>--- tests/Makefile.include | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
tests/Makefile.include +1
-1@@ -1177,7 +1177,20 @@ $(TESTS_RESULTS_DIR): check-venv: $(TESTS_VENV_DIR) -check-acceptance: check-venv $(TESTS_RESULTS_DIR) +FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(TARGETS))Quoted 8 lines
tests/Makefile.include $(TESTS_RESULTS_DIR): · line 1188
Quoted 8 lines
tests/Makefile.include +8
-0+FEDORA_31_ARCHES := x86_64 aarch64 ppc64le s390x +FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES)) + +# download one specific Fedora 31 image +get-vmimage-fedora-31-%: $(check-venv) + $(call quiet-command, \ + $(TESTS_VENV_DIR)/bin/python -m avocado vmimage get \ + --distro=fedora --distro-version=31 --arch=$*)
Another thing we can do here is check the host has sufficient storage.
Quoted 27 lines
Quoted 11 lines
Quoted 11 lines
tests/Makefile.include +5
-1+ +# download all vm images, according to defined targets +get-vmimage: $(patsubst %,get-vmimage-fedora-31-%, $(FEDORA_31_DOWNLOAD)) + +check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vmimage $(call quiet-command, \ $(TESTS_VENV_DIR)/bin/python -m avocado \ --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \@@ -1188,7 +1201,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) # Consolidated targets -.PHONY: check-block check-qapi-schema check-qtest check-unit check check-cleanQuoted 9 lines
tests/Makefile.include check-acceptance: check-venv $(TESTS_RESULTS_DIR)
Quoted 5 lines
tests/Makefile.include +1
-0+.PHONY: check-block check-qapi-schema check-qtest check-unit check check-clean get-vmimage check-qapi-schema: check-tests/qapi-schema/frontend check-tests/qapi-schema/doc-good.texi check-qtest: $(patsubst %,check-qtest-%, $(QTEST_TARGETS)) check-block: $(patsubst %,check-%, $(check-block-y))We have both 'make vm-help' and 'make check-help'. The check-acceptance target is in check-help. We get vm image... confusing.
I know... I had a hard time coming up with a name, and I'm aware it's not a very good one.
Quoted 2 lines
Anyway, can you list this new target, with a hint about the storage size required?
Quoted 6 lines
Sure thing, good point.
Quoted 2 lines
Can you add an entry in the 'make
I suspect you mean adding an entry in the 'make check-help' output, right?
Hehe I'm not sure what happened here. I probably fell asleep on the keyboard. Since 'check-acceptance' is listed in 'check-help', this seems the best place.
Thanks,
Phil.
PATCH v8 4/4 Use Avocado master branch + vmimage fix
In reply to Cleber RosaThis uses the Avocado from a custom branch that contains a fix, and is proposed on the upstream Avocado project as pull request #3406.
Upon inclusion and a new release, this should be dropped and the Avocado version bumped to 74.0.
Reference: https://github.com/avocado-framework/avocado/pull/3406
--- tests/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/requirements.txt +1 -1
@@ -1,5 +1,5 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -avocado-framework==73.0 +-e git+https://github.com/clebergnu/avocado@vmimage_lazy_no_snapshot#egg=avocado_framework pycdlib==1.8.0
Signature · 1 line
2.21.0