tests/vm/freebsd: Get up-to-date package list from lcitool
v2 cover tests/vm/freebsd: Get up-to-date package list from lcitool
Inspired by this patch from Thomas:
https://lore.kernel.org/qemu-devel/[email protected]/
Instead of updating the package list manually, use lcitool vars file.
Since v1:
- Addressed Erik & Daniel comments (generate in JSON)
Philippe Mathieu-Daudé (3):
tests/lcitool: Generate distribution packages list in JSON format tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper tests/vm/freebsd: Get up-to-date package list from lcitool vars file
tests/lcitool/refresh | 11 ++++++++++ tests/vm/Makefile.include | 6 ++++++ tests/vm/basevm.py | 9 +++++++++ tests/vm/freebsd | 42 ++------------------------------------- 4 files changed, 28 insertions(+), 40 deletions(-)
Signature · 1 line
2.38.1
PATCH v2 1/3 tests/lcitool: Generate distribution packages list in JSON format
Add the generate_pkglist() helper to generate a list of packages required by a distribution to build QEMU.
Generate the FreeBSD JSON file (based on FreeBSD 13).
--- tests/lcitool/refresh | 11 +++++++++++ 1 file changed, 11 insertions(+)
tests/lcitool/refresh +11 -0
@@ -84,6 +84,12 @@ def generate_cirrus(target, trailer=None): generate(filename, cmd, trailer) +def generate_pkglist(vm, target): + filename = Path(src_dir, "tests", "vm", vm + ".json") + cmd = lcitool_cmd + ["variables", "--format", "json", target, "qemu"] + generate(filename, cmd, None) + + # Netmap still needs to be manually built as it is yet to be packaged # into a distro. We also add cscope and gtags which are used in the CI # test
@@ -191,6 +197,11 @@ try: generate_cirrus("freebsd-13") generate_cirrus("macos-12") + # + # VM packages lists + # + generate_pkglist("freebsd", "freebsd-13") + sys.exit(0) except Exception as ex: print(str(ex), file=sys.stderr)
Signature · 1 line
2.38.1
Re: [PATCH v2 1/3] tests/lcitool: Generate distribution packages list in JSON format
Quoted 10 lines · Philippe Mathieu-Daudé
Add the
generate_pkglist()helper to generate a list of packages required by a distribution to build QEMU.Generate the FreeBSD JSON file (based on FreeBSD 13).
Suggested-by Erik Skultety <[email protected]>Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- tests/lcitool/refresh | 11 +++++++++++ 1 file changed, 11 insertions(+)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v2 1/3] tests/lcitool: Generate distribution packages list in JSON format
In reply to Philippe Mathieu-DaudéQuoted 16 lines · Philippe Mathieu-Daudé
Add the
generate_pkglist()helper to generate a list of packages required by a distribution to build QEMU.Generate the FreeBSD JSON file (based on FreeBSD 13).
Suggested-by Erik Skultety <[email protected]>Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- tests/lcitool/refresh | 11 +++++++++++ 1 file changed, 11 insertions(+)tests/lcitool/refresh +0
-0
tests/lcitool/refresh def generate_cirrus(target, trailer=None): · line 90
Quoted 7 lines
tests/lcitool/refresh +4
-0@@ -84,6 +84,12 @@ def generate_cirrus(target, trailer=None): generate(filename, cmd, trailer) +def generate_pkglist(vm, target): + filename = Path(src_dir, "tests", "vm", vm + ".json") + cmd = lcitool_cmd + ["variables", "--format", "json", target, "qemu"] + generate(filename, cmd, None)
Note since json files don't have comments, we can't have the "THIS IS GENERATED BY A SCRIPT" in header.
PATCH v2 2/3 tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper
In reply to Philippe Mathieu-DaudéAdd the get_qemu_packages_from_lcitool_json() helper which return such package list from a lcitool env var file in JSON format.
--- tests/vm/Makefile.include | 4 ++++ tests/vm/basevm.py | 9 +++++++++ 2 files changed, 13 insertions(+)
tests/vm/Makefile.include +4 -0
@@ -106,6 +106,10 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ --build-image $@, \ " VM-IMAGE $*") +$(SRC_PATH)/tests/vm/%.json: + $(call quiet-command, \ + make lcitool-refresh) + # Build in VM $(IMAGE) vm-build-%: $(IMAGES_DIR)/%.img $(VM_VENV) $(call quiet-command, \
tests/vm/basevm.py +9 -0
@@ -27,6 +27,7 @@ import multiprocessing import traceback import shlex +import json from qemu.machine import QEMUMachine from qemu.utils import get_info_usernet_hostfwd_port, kvm_available
@@ -501,6 +502,14 @@ def gen_cloud_init_iso(self): stderr=self._stdout) return os.path.join(cidir, "cloud-init.iso") + def get_qemu_packages_from_lcitool_json(self, json_path=None): + """Parse a lcitool variables json file and return the PKGS list.""" + if json_path is None: + json_path = os.path.join(os.path.dirname(__file__), self.name + ".json") + with open(json_path, "r") as fh: + return json.load(fh)["pkgs"] + + def get_qemu_path(arch, build_path=None): """Fetch the path to the qemu binary.""" # If QEMU environment variable set, it takes precedence
Signature · 1 line
2.38.1
Re: [PATCH v2 2/3] tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper
Quoted 15 lines · Philippe Mathieu-Daudé
Add the
get_qemu_packages_from_lcitool_json()helper which return such package list from a lcitool env var file in JSON format.Suggested-by Daniel P. Berrangé <[email protected]>Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- tests/vm/Makefile.include | 4 ++++ tests/vm/basevm.py | 9 +++++++++ 2 files changed, 13 insertions(+)tests/vm/Makefile.include +0
-0
tests/vm/Makefile.include $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ · line 111
Quoted 6 lines
tests/vm/Makefile.include +3
-0@@ -106,6 +106,10 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ --build-image $@, \ " VM-IMAGE $*") +$(SRC_PATH)/tests/vm/%.json: + $(call quiet-command, \ + make lcitool-refresh)
'lcitool-refresh' is something we only run explicitly when we change something about the lcitool, and also isn't provided to end users running from tarballs IIRC. So we shouldn't need to wire it up to build targets, just commit its output JSON file.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v2 2/3] tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper
Quoted 19 lines · Daniel P. Berrangé
Quoted 18 lines · Philippe Mathieu-Daudé
Add the
get_qemu_packages_from_lcitool_json()helper which return such package list from a lcitool env var file in JSON format.Suggested-by Daniel P. Berrangé <[email protected]>Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- tests/vm/Makefile.include | 4 ++++ tests/vm/basevm.py | 9 +++++++++ 2 files changed, 13 insertions(+)tests/vm/Makefile.include +0
-0@@ -106,6 +106,10 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ --build-image $@, \ " VM-IMAGE $*")
Quoted 9 lines
tests/vm/Makefile.include $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ · line 111
Quoted 3 lines
tests/vm/Makefile.include +3
-0+$(SRC_PATH)/tests/vm/%.json: + $(call quiet-command, \ + make lcitool-refresh)'lcitool-refresh' is something we only run explicitly when we change something about the lcitool, and also isn't provided to end users running from tarballs IIRC. So we shouldn't need to wire it up to build targets, just commit its output JSON file.
Right, simpler. Thanks!
PATCH v2 3/3 tests/vm/freebsd: Get up-to-date package list from lcitool vars file
In reply to Philippe Mathieu-DaudéGet an up-to-date package list from lcitool, that way we don't need to manually keep this array in sync.
--- tests/vm/Makefile.include | 2 ++ tests/vm/freebsd | 42 ++------------------------------------- 2 files changed, 4 insertions(+), 40 deletions(-)
tests/vm/Makefile.include +2 -0
@@ -110,6 +110,8 @@ $(SRC_PATH)/tests/vm/%.json: $(call quiet-command, \ make lcitool-refresh) +vm-build-freebsd: $(SRC_PATH)/tests/vm/freebsd.json + # Build in VM $(IMAGE) vm-build-%: $(IMAGES_DIR)/%.img $(VM_VENV) $(call quiet-command, \
tests/vm/freebsd +2 -40
55 lines not on this page · raw message
Signature · 1 line
2.38.1
Re: [PATCH v2 3/3] tests/vm/freebsd: Get up-to-date package list from lcitool vars file
Quoted 15 lines · Philippe Mathieu-Daudé
Get an up-to-date package list from lcitool, that way we don't need to manually keep this array in sync.
Inspired-by Thomas Huth <[email protected]>Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- tests/vm/Makefile.include | 2 ++ tests/vm/freebsd | 42 ++------------------------------------- 2 files changed, 4 insertions(+), 40 deletions(-)tests/vm/Makefile.include +0
-0
tests/vm/Makefile.include $(SRC_PATH)/tests/vm/%.json: · line 113
Quoted 4 lines
tests/vm/Makefile.include +1
-0@@ -110,6 +110,8 @@ $(SRC_PATH)/tests/vm/%.json: $(call quiet-command, \ make lcitool-refresh) +vm-build-freebsd: $(SRC_PATH)/tests/vm/freebsd.json
I don't think we need this, as we just commit the result of lcitool as needed.
Quoted 68 lines
tests/vm/Makefile.include +1
-0+ # Build in VM $(IMAGE) vm-build-%: $(IMAGES_DIR)/%.img $(VM_VENV) $(call quiet-command, \tests/vm/freebsd +2
-40@@ -31,45 +31,6 @@ class FreeBSDVM(basevm.BaseVM): link = "https://download.freebsd.org/releases/CI-IMAGES/13.2-RELEASE/amd64/Latest/FreeBSD-13.2-RELEASE-amd64-BASIC-CI.raw.xz" csum = "a4fb3b6c7b75dd4d58fb0d75e4caf72844bffe0ca00e66459c028b198ffb3c0e" size = "20G" - pkgs = [ - # build tools - "git", - "pkgconf", - "bzip2", - "python39", - "ninja", - - # gnu tools - "bash", - "gmake", - "gsed", - "gettext", - - # libs: crypto - "gnutls", - - # libs: images - "jpeg-turbo", - "png", - - # libs: ui - "sdl2", - "gtk3", - "libxkbcommon", - - # libs: opengl - "libepoxy", - "mesa-libs", - - # libs: migration - "zstd", - - # libs: networking - "libslirp", - - # libs: sndio - "sndio", - ] BUILD_SCRIPT = """ set -e;@@ -151,8 +112,9 @@ class FreeBSDVM(basevm.BaseVM): self.console_wait(prompt) self.console_send("echo 'chmod 666 /dev/vtbd1' >> /etc/rc.local\n") + pkgs = self.get_qemu_packages_from_lcitool_json() self.print_step("Installing packages") - self.ssh_root_check("pkg install -y %s\n" % " ".join(self.pkgs)) + self.ssh_root_check("pkg install -y %s\n" % " ".join(pkgs)) # shutdown self.ssh_root(self.poweroff)Signature · 1 line
2.38.1
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|