Improve reliability of VM tests
v4 cover Improve reliability of VM tests
Note: patches 10-12 are included for testing simplicity, they shouldn't be merged. They will be included in a forthcoming block PR.
V4:
- Addressed concern by Marc-Andre in patch 01.
- Squashed Ubuntu patches (rth)
This patch series attempts to improve the reliability of several of the VM test targets. In particular, both CentOS 8 tests are non-functional because CentOS 8 was EOL at the beginning of this calendar year, with repositories and mirrors going offline.
I also remove the ubuntu.i386 test because we no longer support Ubuntu 18.04 nor do we have explicit need of an i386 build test.
After this series, I am able to successfully run every VM target on an x86_64 host, except:
- ubuntu.aarch64: Hangs often during testing, see below.
- centos.aarch64: Hangs often during testing, see below.
- haiku.x86_64: Build failures not addressed by this series, see
https://lists.gnu.org/archive/html/qemu-devel/2022-06/msg02103.html
The unit tests that I see fail most often under aarch64 are:
- virtio-net-failover: Seems to like to hang on openbsd
- migration-test: Tends to hang under aarch64 tcg
Future work (next version? next series?);
- Try to get centos.aarch64 working reliably under TCG
- Upgrade ubuntu.aarch64 to 20.04 after fixing centos.aarch64
- Fix the Haiku build test, if possible.
- Ensure I can reliably run and pass "make vm-build-all". (Remove VMs from this recipe if necessary.)
John Snow (11):
qga: treat get-guest-fsinfo as "best effort" tests/vm: use 'cp' instead of 'ln' for temporary vm images tests/vm: switch CentOS 8 to CentOS 8 Stream tests/vm: switch centos.aarch64 to CentOS 8 Stream tests/vm: upgrade Ubuntu 18.04 VM to 20.04 tests/vm: remove ubuntu.i386 VM test
tests/vm: remove duplicate 'centos' VM test tests/vm: add 1GB extra memory per core tests/vm: Remove docker cross-compile test from CentOS VM tests/qemu-iotests: hotfix for 307, 223 output tests/qemu-iotests: skip 108 when FUSE is not loaded
Vladimir Sementsov-Ogievskiy (1):
iotests: fix copy-before-write for macOS and FreeBSD
qga/commands-posix.c | 10 +- tests/qemu-iotests/108 | 5 + tests/qemu-iotests/223.out | 4 +- tests/qemu-iotests/307.out | 4 +- tests/qemu-iotests/tests/copy-before-write | 5 + tests/vm/Makefile.include | 5 +- tests/vm/basevm.py | 5 + tests/vm/centos | 9 +- tests/vm/centos.aarch64 | 174 +++------------------ tests/vm/ubuntu.aarch64 | 10 +- tests/vm/ubuntu.i386 | 40 ----- 11 files changed, 65 insertions(+), 206 deletions(-) delete mode 100755 tests/vm/ubuntu.i386
Signature · 1 line
2.34.3
PATCH v4 1/12 qga: treat get-guest-fsinfo as "best effort"
In some container environments, there may be references to block devices witnessable from a container through /proc/self/mountinfo that reference devices we simply don't have access to in the container, and cannot provide information about.
Instead of failing the entire fsinfo command, return stub information for these failed lookups.
This allows test-qga to pass under docker tests, which are in turn used by the CentOS VM tests.
--- qga/commands-posix.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
qga/commands-posix.c +9 -1
@@ -1207,7 +1207,15 @@ static void build_guest_fsinfo_for_device(char const *devpath, syspath =realpath (devpath,NULL );if (!syspath) { -error_setg_errno (errp, errno,"realpath(\"%s\")" , devpath); +if (errno !=ENOENT ) { +error_setg_errno (errp, errno,"realpath(\"%s\")" , devpath); +return ; + } + +/* ENOENT: This devpath may not exist because of container config */ +if (!fs->name) { + fs->name =g_path_get_basename (devpath); + }return ; }
Signature · 1 line
2.34.3
Re: [PATCH v4 01/12] qga: treat get-guest-fsinfo as "best effort"
Quoted 5 lines · John Snow
In some container environments, there may be references to block devices witnessable from a container through
/proc/self/mountinfothat reference devices we simply don't have access to in the container, and cannot provide information about.
Quoted 8 lines
Instead of failing the entire fsinfo command, return stub information for these failed lookups.
This allows test-qga to pass under docker tests, which are in turn used by the CentOS VM tests.
Signed-off-by John Snow <[email protected]>
Quoted 32 lines
--- qga/commands-posix.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)qga/commands-posix.c +9
-1@@ -1207,7 +1207,15 @@ static void build_guest_fsinfo_for_device(char const *devpath, syspath =realpath (devpath,NULL );if (!syspath) { -error_setg_errno(errp, errno, "realpath(\"%s\")", devpath); +if (errno !=ENOENT ) { +error_setg_errno (errp, errno,"realpath(\"%s\")" , devpath); +return ; + } + +/* ENOENT: This devpath may not exist because of container config */ +if (!fs->name) { + fs->name =g_path_get_basename (devpath); + }return ;}--
2.34.3
Signature · 1 line
Marc-André Lureau
PATCH v4 2/12 tests/vm: use 'cp' instead of 'ln' for temporary vm images
In reply to John SnowIf the initial setup fails, you've permanently altered the state of the downloaded image in an unknowable way. Use 'cp' like our other test setup scripts do.
--- tests/vm/centos | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/vm/centos +1 -1
@@ -34,7 +34,7 @@ class CentosVM(basevm.BaseVM): def build_image(self, img): cimg = self._download_with_cache("https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.3.2011-20201204.2.x86_64.qcow2") img_tmp = img + ".tmp" - subprocess.check_call(["ln", "-f", cimg, img_tmp]) + subprocess.check_call(['cp', '-f', cimg, img_tmp]) self.exec_qemu_img("resize", img_tmp, "50G") self.boot(img_tmp, extra_args = ["-cdrom", self.gen_cloud_init_iso()]) self.wait_ssh()
Signature · 1 line
2.34.3
Re: [PATCH v4 02/12] tests/vm: use 'cp' instead of 'ln' for temporary vm images
Quoted 10 lines · John Snow
If the initial setup fails, you've permanently altered the state of the downloaded image in an unknowable way. Use 'cp' like our other test setup scripts do.
Signed-off-by John Snow<[email protected]>Reviewed-by Thomas Huth<[email protected]>Reviewed-by Daniel P. Berrangé<[email protected]>--- tests/vm/centos | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
r~
PATCH v4 3/12 tests/vm: switch CentOS 8 to CentOS 8 Stream
In reply to John SnowThe old CentOS image didn't work anymore because it was already EOL at the beginning of 2022.
--- tests/vm/centos | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tests/vm/centos +3 -3
@@ -1,8 +1,8 @@ #!/usr/bin/env python3 # -# CentOS image +# CentOS 8 Stream image # -# Copyright 2018 Red Hat Inc. +# Copyright 2018, 2022 Red Hat Inc. # # Authors: # Fam Zheng <[email protected]>
@@ -32,7 +32,7 @@ class CentosVM(basevm.BaseVM): """ def build_image(self, img): - cimg = self._download_with_cache("https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.3.2011-20201204.2.x86_64.qcow2") + cimg = self._download_with_cache("https://cloud.centos.org/centos/8-stream/x86_64/images/CentOS-Stream-GenericCloud-8-20220125.1.x86_64.qcow2") img_tmp = img + ".tmp" subprocess.check_call(['cp', '-f', cimg, img_tmp]) self.exec_qemu_img("resize", img_tmp, "50G")
Signature · 1 line
2.34.3
Re: [PATCH v4 03/12] tests/vm: switch CentOS 8 to CentOS 8 Stream
Quoted 9 lines · John Snow
The old CentOS image didn't work anymore because it was already EOL at the beginning of 2022.
Signed-off-by John Snow<[email protected]>Reviewed-by Thomas Huth<[email protected]>Reviewed-by Daniel P. Berrangé<[email protected]>--- tests/vm/centos | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
r~
PATCH v4 4/12 tests/vm: switch centos.aarch64 to CentOS 8 Stream
In reply to John SnowSwitch this test over to using a cloud image like the base CentOS8 VM test, which helps make this script a bit simpler too.
Note: At time of writing, this test seems pretty flaky when run without KVM support for aarch64. Certain unit tests like migration-test, virtio-net-failover, test-hmp and qom-test seem quite prone to fail under TCG. Still, this is an improvement in that at least pure build tests are functional.
--- tests/vm/centos.aarch64 | 174 ++++++---------------------------------- 1 file changed, 24 insertions(+), 150 deletions(-)
tests/vm/centos.aarch64 +24 -150
@@ -20,150 +20,38 @@ import time import traceback import aarch64vm + DEFAULT_CONFIG = { 'cpu' : "max", 'machine' : "virt,gic-version=max", - 'install_cmds' : "yum install -y make ninja-build git python3 gcc gcc-c++ flex bison, "\ - "yum install -y glib2-devel perl pixman-devel zlib-devel, "\ - "alternatives --set python /usr/bin/python3, "\ - "sudo dnf config-manager "\ - "--add-repo=https://download.docker.com/linux/centos/docker-ce.repo,"\ - "sudo dnf install --nobest -y docker-ce.aarch64,"\ - "systemctl enable docker", + 'install_cmds' : ( + "dnf config-manager --set-enabled powertools, " + "dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo, " + "dnf install -y make ninja-build git python38 gcc gcc-c++ flex bison "\ + "glib2-devel perl pixman-devel zlib-devel docker-ce.aarch64, " + "systemctl enable docker, " + ), # We increase beyond the default time since during boot # it can take some time (many seconds) to log into the VM. 'ssh_timeout' : 60, } + class CentosAarch64VM(basevm.BaseVM): - name = "centos.aarch64" + name = "centos8.aarch64" arch = "aarch64" - login_prompt = "localhost login:" - prompt = '[root@localhost ~]#' - image_name = "CentOS-8-aarch64-1905-dvd1.iso" - image_link = "http://mirrors.usc.edu/pub/linux/distributions/centos/8.0.1905/isos/aarch64/" + image_name = "CentOS-Stream-GenericCloud-8-20220125.1.aarch64.qcow2" + image_link = "https://cloud.centos.org/centos/8-stream/aarch64/images/" image_link += image_name BUILD_SCRIPT = """ set -e; cd $(mktemp -d); - sudo chmod a+r /dev/vdb; - tar --checkpoint=.10 -xf /dev/vdb; + export SRC_ARCHIVE=/dev/vdb; + sudo chmod a+r $SRC_ARCHIVE; + tar -xf $SRC_ARCHIVE; ./configure {configure_opts}; make --output-sync {target} -j{jobs} {verbose}; """ - def set_key_perm(self): - """Set permissions properly on certain files to allow - ssh access.""" - self.console_wait_send(self.prompt, - "/usr/sbin/restorecon -R -v /root/.ssh\n") - self.console_wait_send(self.prompt, - "/usr/sbin/restorecon -R -v "\ - "/home/{}/.ssh\n".format(self._config["guest_user"])) - - def create_kickstart(self): - """Generate the kickstart file used to generate the centos image.""" - # Start with the template for the kickstart. - ks_file = self._source_path + "/tests/vm/centos-8-aarch64.ks" - subprocess.check_call("cp {} ./ks.cfg".format(ks_file), shell=True) - # Append the ssh keys to the kickstart file - # as the post processing phase of installation. - with open("ks.cfg", "a") as f: - # Add in the root pw and guest user. - rootpw = "rootpw --plaintext {}\n" - f.write(rootpw.format(self._config["root_pass"])) - add_user = "user --groups=wheel --name={} "\ - "--password={} --plaintext\n" - f.write(add_user.format(self._config["guest_user"], - self._config["guest_pass"])) - # Add the ssh keys. - f.write("%post --log=/root/ks-post.log\n") - f.write("mkdir -p /root/.ssh\n") - addkey = 'echo "{}" >> /root/.ssh/authorized_keys\n' - addkey_cmd = addkey.format(self._config["ssh_pub_key"]) - f.write(addkey_cmd) - f.write('mkdir -p /home/{}/.ssh\n'.format(self._config["guest_user"])) - addkey = 'echo "{}" >> /home/{}/.ssh/authorized_keys\n' - addkey_cmd = addkey.format(self._config["ssh_pub_key"], - self._config["guest_user"]) - f.write(addkey_cmd) - f.write("%end\n") - # Take our kickstart file and create an .iso from it. - # The .iso will be provided to qemu as we boot - # from the install dvd. - # Anaconda will recognize the label "OEMDRV" and will - # start the automated installation. - gen_iso_img = 'genisoimage -output ks.iso -volid "OEMDRV" ks.cfg' - subprocess.check_call(gen_iso_img, shell=True) - - def wait_for_shutdown(self): - """We wait for qemu to shutdown the VM and exit. - While this happens we display the console view - for easier debugging.""" - # The image creation is essentially done, - # so whether or not the wait is successful we want to - # wait for qemu to exit (the self.wait()) before we return. - try: - self.console_wait("reboot: Power down") - except Exception as e: - sys.stderr.write("Exception hit\n") - if isinstance(e, SystemExit) and e.code == 0: - return 0 - traceback.print_exc() - finally: - self.wait() - - def build_base_image(self, dest_img): - """Run through the centos installer to create - a base image with name dest_img.""" - # We create the temp image, and only rename - # to destination when we are done. - img = dest_img + ".tmp" - # Create an empty image. - # We will provide this as the install destination. - qemu_img_create = "qemu-img create {} 50G".format(img) - subprocess.check_call(qemu_img_create, shell=True) - - # Create our kickstart file to be fed to the installer. - self.create_kickstart() - # Boot the install dvd with the params as our ks.iso - os_img = self._download_with_cache(self.image_link) - dvd_iso = "centos-8-dvd.iso" - subprocess.check_call(["cp", "-f", os_img, dvd_iso]) - extra_args = "-cdrom ks.iso" - extra_args += " -drive file={},if=none,id=drive1,cache=writeback" - extra_args += " -device virtio-blk,drive=drive1,bootindex=1" - extra_args = extra_args.format(dvd_iso).split(" ") - self.boot(img, extra_args=extra_args) - self.console_wait_send("change the selection", "\n") - # We seem to need to hit esc (chr(27)) twice to abort the - # media check, which takes a long time. - # Waiting a bit seems to be more reliable before hitting esc. - self.console_wait("Checking") - time.sleep(5) - self.console_wait_send("Checking", chr(27)) - time.sleep(5) - self.console_wait_send("Checking", chr(27)) - print("Found Checking") - # Give sufficient time for the installer to create the image. - self.console_init(timeout=7200) - self.wait_for_shutdown() - os.rename(img, dest_img) - print("Done with base image build: {}".format(dest_img)) - - def check_create_base_img(self, img_base, img_dest): - """Create a base image using the installer. - We will use the base image if it exists. - This helps cut down on install time in case we - need to restart image creation, - since the base image creation can take a long time.""" - if not os.path.exists(img_base): - print("Generate new base image: {}".format(img_base)) - self.build_base_image(img_base); - else: - print("Use existing base image: {}".format(img_base)) - # Save a copy of the base image and copy it to dest. - # which we will use going forward. - subprocess.check_call(["cp", img_base, img_dest]) def boot(self, img, extra_args=None): aarch64vm.create_flash_images(self._tmpdir, self._efi_aarch64)
@@ -185,42 +73,28 @@ class CentosAarch64VM(basevm.BaseVM): super(CentosAarch64VM, self).boot(img, extra_args=extra_args) def build_image(self, img): + cimg = self._download_with_cache(self.image_link) img_tmp = img + ".tmp" - self.check_create_base_img(img + ".base", img_tmp) - - # Boot the new image for the first time to finish installation. - self.boot(img_tmp) - self.console_init() - self.console_wait_send(self.login_prompt, "root\n") - self.console_wait_send("Password:", - "{}\n".format(self._config["root_pass"])) - - self.set_key_perm() - self.console_wait_send(self.prompt, "rpm -q centos-release\n") - enable_adapter = "sed -i 's/ONBOOT=no/ONBOOT=yes/g'" \ - " /etc/sysconfig/network-scripts/ifcfg-enp0s1\n" - self.console_wait_send(self.prompt, enable_adapter) - self.console_wait_send(self.prompt, "ifup enp0s1\n") - self.console_wait_send(self.prompt, - 'echo "qemu ALL=(ALL) NOPASSWD:ALL" | '\ - 'sudo tee /etc/sudoers.d/qemu\n') - self.console_wait(self.prompt) - - # Rest of the commands we issue through ssh. + subprocess.run(['cp', '-f', cimg, img_tmp]) + self.exec_qemu_img("resize", img_tmp, "50G") + self.boot(img_tmp, extra_args = ["-cdrom", self.gen_cloud_init_iso()]) self.wait_ssh(wait_root=True) + self.ssh_root_check("touch /etc/cloud/cloud-init.disabled") # If the user chooses *not* to do the second phase, # then we will jump right to the graceful shutdown if self._config['install_cmds'] != "": install_cmds = self._config['install_cmds'].split(',') for cmd in install_cmds: - self.ssh_root(cmd) + self.ssh_root_check(cmd) + self.ssh_root("poweroff") - self.wait_for_shutdown() + self.wait() os.rename(img_tmp, img) print("image creation complete: {}".format(img)) return 0 + if __name__ == "__main__": defaults = aarch64vm.get_config_defaults(CentosAarch64VM, DEFAULT_CONFIG) sys.exit(basevm.main(CentosAarch64VM, defaults))
Signature · 1 line
2.34.3
Re: [PATCH v4 04/12] tests/vm: switch centos.aarch64 to CentOS 8 Stream
Quoted 9 lines · John Snow
Switch this test over to using a cloud image like the base CentOS8 VM test, which helps make this script a bit simpler too.
Note: At time of writing, this test seems pretty flaky when run without KVM support for aarch64. Certain unit tests like migration-test, virtio-net-failover, test-hmp and qom-test seem quite prone to fail under TCG. Still, this is an improvement in that at least pure build tests are functional.
Quoted 5 lines
Signed-off-by John Snow<[email protected]>Reviewed-by Daniel P. Berrangé<[email protected]>--- tests/vm/centos.aarch64 | 174 ++++++---------------------------------- 1 file changed, 24 insertions(+), 150 deletions(-)
r~
PATCH v4 5/12 tests/vm: upgrade Ubuntu 18.04 VM to 20.04
In reply to John Snow18.04 has fallen out of our support window, so move ubuntu.aarch64 forward to ubuntu 20.04, which is now our oldest supported Ubuntu release.
Notes:
This checksum changes periodically; use a fixed point image with a known checksum so that the image isn't re-downloaded on every single invocation. (The checksum for the 18.04 image was already incorrect at the time of writing.)
Just like the centos.aarch64 test, this test currently seems very flaky when run as a TCG test.
--- tests/vm/ubuntu.aarch64 | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
tests/vm/ubuntu.aarch64 +7 -3
@@ -32,9 +32,13 @@ DEFAULT_CONFIG = { class UbuntuAarch64VM(ubuntuvm.UbuntuVM): name = "ubuntu.aarch64" arch = "aarch64" - image_name = "ubuntu-18.04-server-cloudimg-arm64.img" - image_link = "https://cloud-images.ubuntu.com/releases/18.04/release/" + image_name - image_sha256="0fdcba761965735a8a903d8b88df8e47f156f48715c00508e4315c506d7d3cb1" + # NOTE: The Ubuntu 20.04 cloud images are periodically updated. The + # fixed image chosen below is the latest release at time of + # writing. Using a rolling latest instead would mean that the SHA + # would be incorrect at an indeterminate point in the future. + image_name = "focal-server-cloudimg-arm64.img" + image_link = "https://cloud-images.ubuntu.com/focal/20220615/" + image_name + image_sha256="95a027336e197debe88c92ff2e554598e23c409139e1e750b71b3b820b514832" BUILD_SCRIPT = """ set -e; cd $(mktemp -d);
Signature · 1 line
2.34.3
Re: [PATCH v4 05/12] tests/vm: upgrade Ubuntu 18.04 VM to 20.04
Quoted 11 lines · John Snow
18.04 has fallen out of our support window, so move ubuntu.aarch64 forward to ubuntu 20.04, which is now our oldest supported Ubuntu release.
Notes:
This checksum changes periodically; use a fixed point image with a known checksum so that the image isn't re-downloaded on every single invocation. (The checksum for the 18.04 image was already incorrect at the time of writing.)
Quoted 8 lines
Just like the centos.aarch64 test, this test currently seems very flaky when run as a TCG test.
Signed-off-by John Snow<[email protected]>Reviewed-by Daniel P. Berrangé<[email protected]>--- tests/vm/ubuntu.aarch64 | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
r~
PATCH v4 6/12 tests/vm: remove ubuntu.i386 VM test
In reply to John SnowUbuntu 18.04 is out of our support window, and Ubuntu 20.04 does not support i386 anymore. The debian project does, but they do not provide any cloud images for it, a new expect-style script would have to be written.
Since we have i386 cross-compiler tests hosted on GitLab CI, we don't need to support this VM test anymore.
--- tests/vm/Makefile.include | 3 +-- tests/vm/ubuntu.i386 | 40 --------------------------------------- 2 files changed, 1 insertion(+), 42 deletions(-) delete mode 100755 tests/vm/ubuntu.i386
tests/vm/Makefile.include +1 -2
@@ -17,7 +17,7 @@ EFI_AARCH64 = $(wildcard $(BUILD_DIR)/pc-bios/edk2-aarch64-code.fd) X86_IMAGES := freebsd netbsd openbsd centos fedora haiku.x86_64 ifneq ($(GENISOIMAGE),) -X86_IMAGES += ubuntu.i386 centos +X86_IMAGES += centos ifneq ($(EFI_AARCH64),) ARM64_IMAGES += ubuntu.aarch64 centos.aarch64 endif
@@ -48,7 +48,6 @@ vm-help vm-test: @echo " vm-build-fedora - Build QEMU in Fedora VM" ifneq ($(GENISOIMAGE),) @echo " vm-build-centos - Build QEMU in CentOS VM, with Docker" - @echo " vm-build-ubuntu.i386 - Build QEMU in ubuntu i386 VM" ifneq ($(EFI_AARCH64),) @echo " vm-build-ubuntu.aarch64 - Build QEMU in ubuntu aarch64 VM" @echo " vm-build-centos.aarch64 - Build QEMU in CentOS aarch64 VM"
tests/vm/ubuntu.i386 deleted +0 -40
@@ -1,40 +0,0 @@ -#!/usr/bin/env python3 -# -# Ubuntu i386 image -# -# Copyright 2017 Red Hat Inc. -# -# Authors: -# Fam Zheng <[email protected]> -# -# This code is licensed under the GPL version 2 or later. See -# the COPYING file in the top-level directory. -# - -import sys -import basevm -import ubuntuvm - -DEFAULT_CONFIG = { - 'install_cmds' : "apt-get update,"\ - "apt-get build-dep -y qemu,"\ - "apt-get install -y libfdt-dev language-pack-en ninja-build", -} - -class UbuntuX86VM(ubuntuvm.UbuntuVM): - name = "ubuntu.i386" - arch = "i386" - image_link="https://cloud-images.ubuntu.com/releases/bionic/"\ - "release-20191114/ubuntu-18.04-server-cloudimg-i386.img" - image_sha256="28969840626d1ea80bb249c08eef1a4533e8904aa51a327b40f37ac4b4ff04ef" - BUILD_SCRIPT = """ - set -e; - cd $(mktemp -d); - sudo chmod a+r /dev/vdb; - tar -xf /dev/vdb; - ./configure {configure_opts}; - make --output-sync {target} -j{jobs} {verbose}; - """ - -if __name__ == "__main__": - sys.exit(basevm.main(UbuntuX86VM, DEFAULT_CONFIG))
Signature · 1 line
2.34.3
Re: [PATCH v4 06/12] tests/vm: remove ubuntu.i386 VM test
Quoted 8 lines · John Snow
Ubuntu 18.04 is out of our support window, and Ubuntu 20.04 does not support i386 anymore. The debian project does, but they do not provide any cloud images for it, a new expect-style script would have to be written.
Since we have i386 cross-compiler tests hosted on GitLab CI, we don't need to support this VM test anymore.
Quoted 8 lines
Signed-off-by John Snow<[email protected]>Reviewed-by Thomas Huth<[email protected]>Reviewed-by Daniel P. Berrangé<[email protected]>--- tests/vm/Makefile.include | 3 +-- tests/vm/ubuntu.i386 | 40 --------------------------------------- 2 files changed, 1 insertion(+), 42 deletions(-)delete mode 100755 tests/vm/ubuntu.i386
r~
PATCH v4 7/12 tests/vm: remove duplicate 'centos' VM test
In reply to John SnowThis is listed twice by accident; we require genisoimage to run the test, so remove the unconditional entry.
--- tests/vm/Makefile.include | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/vm/Makefile.include +1 -1
@@ -15,7 +15,7 @@ endif EFI_AARCH64 = $(wildcard $(BUILD_DIR)/pc-bios/edk2-aarch64-code.fd) -X86_IMAGES := freebsd netbsd openbsd centos fedora haiku.x86_64 +X86_IMAGES := freebsd netbsd openbsd fedora haiku.x86_64 ifneq ($(GENISOIMAGE),) X86_IMAGES += centos ifneq ($(EFI_AARCH64),)
Signature · 1 line
2.34.3
Re: [PATCH v4 07/12] tests/vm: remove duplicate 'centos' VM test
Quoted 9 lines · John Snow
This is listed twice by accident; we require genisoimage to run the test, so remove the unconditional entry.
Signed-off-by John Snow<[email protected]>Reviewed-by Thomas Huth<[email protected]>Reviewed-by Daniel P. Berrangé<[email protected]>--- tests/vm/Makefile.include | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
r~
PATCH v4 8/12 tests/vm: add 1GB extra memory per core
In reply to John SnowIf you try to run a 16 or 32 threaded test, you're going to run out of memory very quickly with qom-test and a few others. Bump the memory limit to try to scale with larger-core machines.
Granted, this means that a 16 core processor is going to ask for 16GB, but you *probably* meet that requirement if you have such a machine.
512MB per core didn't seem to be enough to avoid ENOMEM and SIGABRTs in the test cases in practice on a six core machine; so I bumped it up to 1GB which seemed to help.
Add this magic in early to the configuration process so that the config file, if provided, can still override it.
--- tests/vm/basevm.py | 5 +++++ 1 file changed, 5 insertions(+)
tests/vm/basevm.py +5 -0
@@ -99,6 +99,11 @@ def __init__(self, args, config=None): self._source_path = args.source_path # Allow input config to override defaults. self._config = DEFAULT_CONFIG.copy() + + # 1GB per core, minimum of 4. This is only a default. + mem = max(4, args.jobs) + self._config['memory'] = f"{mem}G" + if config != None: self._config.update(config) self.validate_ssh_keys()
Signature · 1 line
2.34.3
Re: [PATCH v4 08/12] tests/vm: add 1GB extra memory per core
Quoted 11 lines · John Snow
If you try to run a 16 or 32 threaded test, you're going to run out of memory very quickly with qom-test and a few others. Bump the memory limit to try to scale with larger-core machines.
Granted, this means that a 16 core processor is going to ask for 16GB, but you*probably* meet that requirement if you have such a machine.
512MB per core didn't seem to be enough to avoid ENOMEM and SIGABRTs in the test cases in practice on a six core machine; so I bumped it up to 1GB which seemed to help.
Quoted 8 lines
Add this magic in early to the configuration process so that the config file, if provided, can still override it.
Signed-off-by John Snow<[email protected]>Reviewed-by Daniel P. Berrangé<[email protected]>--- tests/vm/basevm.py | 5 +++++ 1 file changed, 5 insertions(+)
PATCH v4 9/12 tests/vm: Remove docker cross-compile test from CentOS VM
In reply to John SnowThe fedora container has since been split apart, so there's no suitable nearby target that would support "test-mingw" as it requires both x32 and x64 support -- so either fedora-cross-win32 nor fedora-cross-win64 would be truly suitable.
Just remove this test as superfluous with our current CI infrastructure.
--- tests/vm/centos | 1 - 1 file changed, 1 deletion(-)
tests/vm/centos +0 -1
@@ -28,7 +28,6 @@ class CentosVM(basevm.BaseVM): tar -xf $SRC_ARCHIVE; make docker-test-block@centos8 {verbose} J={jobs} NETWORK=1; make docker-test-quick@centos8 {verbose} J={jobs} NETWORK=1; - make docker-test-mingw@fedora {verbose} J={jobs} NETWORK=1; """ def build_image(self, img):
Signature · 1 line
2.34.3
Re: [PATCH v4 09/12] tests/vm: Remove docker cross-compile test from CentOS VM
Quoted 5 lines · John Snow
The fedora container has since been split apart, so there's no suitable nearby target that would support "test-mingw" as it requires both x32 and x64 support -- so either fedora-cross-win32 nor fedora-cross-win64 would be truly suitable.
Quoted 7 lines
Just remove this test as superfluous with our current CI infrastructure.
Signed-off-by John Snow<[email protected]>Reviewed-by Daniel P. Berrangé<[email protected]>--- tests/vm/centos | 1 - 1 file changed, 1 deletion(-)
r~
PATCH v4 10/12 tests/qemu-iotests: hotfix for 307, 223 output
In reply to John SnowDo not merge; staged in Hanna's branch.
--- tests/qemu-iotests/223.out | 4 ++-- tests/qemu-iotests/307.out | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
tests/qemu-iotests/223.out +2 -2
@@ -93,7 +93,7 @@ exports available: 3 export: 'n2' description: some text size: 4194304 - flags: 0xced ( flush fua trim zeroes df cache fast-zero ) + flags: 0xded ( flush fua trim zeroes df multi cache fast-zero ) min block: 1 opt block: 4096 max block: 33554432
@@ -212,7 +212,7 @@ exports available: 3 export: 'n2' description: some text size: 4194304 - flags: 0xced ( flush fua trim zeroes df cache fast-zero ) + flags: 0xded ( flush fua trim zeroes df multi cache fast-zero ) min block: 1 opt block: 4096 max block: 33554432
tests/qemu-iotests/307.out +2 -2
@@ -83,7 +83,7 @@ exports available: 2 export: 'export1' description: This is the writable second export size: 67108864 - flags: 0xced ( flush fua trim zeroes df cache fast-zero ) + flags: 0xded ( flush fua trim zeroes df multi cache fast-zero ) min block: XXX opt block: XXX max block: XXX
@@ -109,7 +109,7 @@ exports available: 1 export: 'export1' description: This is the writable second export size: 67108864 - flags: 0xced ( flush fua trim zeroes df cache fast-zero ) + flags: 0xded ( flush fua trim zeroes df multi cache fast-zero ) min block: XXX opt block: XXX max block: XXX
Signature · 1 line
2.34.3
PATCH v4 11/12 tests/qemu-iotests: skip 108 when FUSE is not loaded
In reply to John SnowDo not merge: Staged in Hanna's branch.
--- tests/qemu-iotests/108 | 5 +++++ 1 file changed, 5 insertions(+)
tests/qemu-iotests/108 +5 -0
@@ -60,6 +60,11 @@ if sudo -n losetup &>/dev/null; then else loopdev=false + # Check for usable FUSE in the host environment: + if test ! -c "/dev/fuse"; then + _notrun 'No passwordless sudo nor usable /dev/fuse' + fi + # QSD --export fuse will either yield "Parameter 'id' is missing" # or "Invalid parameter 'fuse'", depending on whether there is # FUSE support or not.
Signature · 1 line
2.34.3
PATCH v4 12/12 iotests: fix copy-before-write for macOS and FreeBSD
In reply to John SnowFrom: Vladimir Sementsov-Ogievskiy <[email protected]>
Do not merge: this is a copy of Vladimir's fix that will be taken in through the iotests tree.
--- tests/qemu-iotests/tests/copy-before-write | 5 +++++ 1 file changed, 5 insertions(+)
tests/qemu-iotests/tests/copy-before-write +5 -0
@@ -192,6 +192,11 @@ read 1048576/1048576 bytes at offset 0 def test_timeout_break_guest(self): log = self.do_cbw_timeout('break-guest-write') + # macOS and FreeBSD tend to represent ETIMEDOUT as + # "Operation timed out", when Linux prefer + # "Connection timed out" + log = log.replace('Operation timed out', + 'Connection timed out') self.assertEqual(log, """\ wrote 524288/524288 bytes at offset 0 512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
Signature · 1 line
2.34.3
Quoted 3 lines · John Snow
Note: patches 10-12 are included for testing simplicity, they shouldn't be merged. They will be included in a forthcoming block PR.
Patches 1-9 are fully reviewed. Whose tree should this go in?
Quoted 69 lines
V4:
- Addressed concern by Marc-Andre in patch 01.
- Squashed Ubuntu patches (rth)This patch series attempts to improve the reliability of several of the VM test targets. In particular, both CentOS 8 tests are non-functional because CentOS 8 was EOL at the beginning of this calendar year, with repositories and mirrors going offline.
I also remove the ubuntu.i386 test because we no longer support Ubuntu 18.04 nor do we have explicit need of an i386 build test.
After this series, I am able to successfully run every VM target on an
x86_64host, except:- ubuntu.aarch64: Hangs often during testing, see below.
- centos.aarch64: Hangs often during testing, see below.
- haiku.x86_64: Build failures not addressed by this series, see
https://lists.gnu.org/archive/html/qemu-devel/2022-06/msg02103.htmlThe unit tests that I see fail most often under aarch64 are:
- virtio-net-failover: Seems to like to hang on openbsd
- migration-test: Tends to hang under aarch64 tcgFuture work (next version? next series?);
- Try to get centos.aarch64 working reliably under TCG
- Upgrade ubuntu.aarch64 to 20.04 after fixing centos.aarch64
- Fix the Haiku build test, if possible.
- Ensure I can reliably run and pass "make vm-build-all". (Remove VMs from this recipe if necessary.)John Snow (11):
qga: treat get-guest-fsinfo as "best effort" tests/vm: use 'cp' instead of 'ln' for temporary vm images tests/vm: switch CentOS 8 to CentOS 8 Stream tests/vm: switch centos.aarch64 to CentOS 8 Stream tests/vm: upgrade Ubuntu 18.04 VM to 20.04 tests/vm: remove ubuntu.i386 VM test
tests/vm: remove duplicate 'centos' VM test tests/vm: add 1GB extra memory per core tests/vm: Remove docker cross-compile test from CentOS VM tests/qemu-iotests: hotfix for 307, 223 output tests/qemu-iotests: skip 108 when FUSE is not loadedVladimir Sementsov-Ogievskiy (1):
iotests: fix copy-before-write for macOS and FreeBSDqga/commands-posix.c | 10 +- tests/qemu-iotests/108 | 5 + tests/qemu-iotests/223.out | 4 +- tests/qemu-iotests/307.out | 4 +- tests/qemu-iotests/tests/copy-before-write | 5 + tests/vm/Makefile.include | 5 +- tests/vm/basevm.py | 5 + tests/vm/centos | 9 +- tests/vm/centos.aarch64 | 174 +++------------------ tests/vm/ubuntu.aarch64 | 10 +- tests/vm/ubuntu.i386 | 40 ----- 11 files changed, 65 insertions(+), 206 deletions(-) delete mode 100755 tests/vm/ubuntu.i386--
2.34.3
Quoted 6 lines · John Snow
Quoted 3 lines · John Snow
Note: patches 10-12 are included for testing simplicity, they shouldn't be merged. They will be included in a forthcoming block PR.
Patches 1-9 are fully reviewed. Whose tree should this go in?
I can take them - unless Alex beats me with his next testing pull request ;-)
Queued to my testing-next now:
https://gitlab.com/thuth/qemu/-/commits/testing-next
Thomas
Quoted 29 lines · John Snow
Note: patches 10-12 are included for testing simplicity, they shouldn't be merged. They will be included in a forthcoming block PR.
V4:
- Addressed concern by Marc-Andre in patch 01.
- Squashed Ubuntu patches (rth)This patch series attempts to improve the reliability of several of the VM test targets. In particular, both CentOS 8 tests are non-functional because CentOS 8 was EOL at the beginning of this calendar year, with repositories and mirrors going offline.
I also remove the ubuntu.i386 test because we no longer support Ubuntu 18.04 nor do we have explicit need of an i386 build test.
After this series, I am able to successfully run every VM target on an
x86_64host, except:- ubuntu.aarch64: Hangs often during testing, see below.
- centos.aarch64: Hangs often during testing, see below.
- haiku.x86_64: Build failures not addressed by this series, see
https://lists.gnu.org/archive/html/qemu-devel/2022-06/msg02103.htmlThe unit tests that I see fail most often under aarch64 are:
- virtio-net-failover: Seems to like to hang on openbsd
- migration-test: Tends to hang under aarch64 tcg
Quoted 7 lines
Future work (next version? next series?);
- Try to get centos.aarch64 working reliably under TCG
- Upgrade ubuntu.aarch64 to 20.04 after fixing centos.aarch64
- Fix the Haiku build test, if possible.
- Ensure I can reliably run and pass "make vm-build-all". (Remove VMs from this recipe if necessary.)
Not sure whether it's related to your patches, but when testing these I just got an error while running 'vm-build-openbsd' :
VM-BUILD openbsd
fatal: not a valid object name:
failed to archive qemu
Failed to prepare guest environment
Traceback (most recent call last):
File "/home/thuth/devel/qemu/tests/vm/basevm.py", line 641, in main
vm.add_source_dir(args.build_qemu)
File "/home/thuth/devel/qemu/tests/vm/basevm.py", line 277, in add_source_dir
stdout=self._stdout, stderr=self._stderr)
File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./scripts/archive-source.sh', '/home/thuth/tmp/qemu-build/vm-test-rm_z92hq.tmp/data-44e42.tar']' returned non-zero exit status 1.
The error did not occur again when running the command again, though.
Thomas
Quoted 57 lines · Thomas Huth
Quoted 29 lines · John Snow
Note: patches 10-12 are included for testing simplicity, they shouldn't be merged. They will be included in a forthcoming block PR.
V4:
- Addressed concern by Marc-Andre in patch 01.
- Squashed Ubuntu patches (rth)This patch series attempts to improve the reliability of several of the VM test targets. In particular, both CentOS 8 tests are non-functional because CentOS 8 was EOL at the beginning of this calendar year, with repositories and mirrors going offline.
I also remove the ubuntu.i386 test because we no longer support Ubuntu 18.04 nor do we have explicit need of an i386 build test.
After this series, I am able to successfully run every VM target on an
x86_64host, except:- ubuntu.aarch64: Hangs often during testing, see below.
- centos.aarch64: Hangs often during testing, see below.
- haiku.x86_64: Build failures not addressed by this series, see
https://lists.gnu.org/archive/html/qemu-devel/2022-06/msg02103.htmlThe unit tests that I see fail most often under aarch64 are:
- virtio-net-failover: Seems to like to hang on openbsd
- migration-test: Tends to hang under aarch64 tcgQuoted 7 lines
Future work (next version? next series?);
- Try to get centos.aarch64 working reliably under TCG
- Upgrade ubuntu.aarch64 to 20.04 after fixing centos.aarch64
- Fix the Haiku build test, if possible.
- Ensure I can reliably run and pass "make vm-build-all". (Remove VMs from this recipe if necessary.)Not sure whether it's related to your patches, but when testing these I just got an error while running 'vm-build-openbsd' :
VM-BUILD openbsd
fatal: not a valid object name:
failed to archive qemu
Failed to prepare guest environment
Traceback (most recent call last):
File "/home/thuth/devel/qemu/tests/vm/basevm.py", line 641, in main
vm.add_source_dir(args.build_qemu)
File "/home/thuth/devel/qemu/tests/vm/basevm.py", line 277, inadd_source_dir
stdout=self._stdout, stderr=self._stderr)
File "/usr/lib64/python3.6/subprocess.py", line 311, incheck_call
raiseCalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./scripts/archive-source.sh', '/home/thuth/tmp/qemu-build/vm-test-rm_z92hq.tmp/data-44e42.tar']' returned non-zero exit status 1.
Quoted 4 lines
The error did not occur again when running the command again, though.
Thomas
I haven't seen this one before, admittedly. Looks like scripts/archive-source.sh L52 choked?
git archive --format tar "$(tree_ish)" > "$tar_file"
I'm not sure what "fatal: not a valid object name:" might be referring to. Maybe tree_ish picked up something that tasted bad? I really don't know.