Skip to contentSkip to conversation

tests/vm/freebsd: Get up-to-date package list from lcitool

PATCH Superseded Philippe Mathieu-Daudé 2 participants 9 messages Jul 11, 2023

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover tests/vm/freebsd: Get up-to-date package list from lcitool

Inspired by this patch from Thomas:
https://lore.kernel.org/qemu-devel/[email protected]/

Instead of updating the package list manually, use lcitool vars file.

Since v1:
- Addressed Erik & Daniel comments (generate in JSON)

Philippe Mathieu-Daudé (3):
tests/lcitool: Generate distribution packages list in JSON format tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper tests/vm/freebsd: Get up-to-date package list from lcitool vars file

 tests/lcitool/refresh     | 11 ++++++++++
 tests/vm/Makefile.include |  6 ++++++
 tests/vm/basevm.py        |  9 +++++++++
 tests/vm/freebsd          | 42 ++-------------------------------------
 4 files changed, 28 insertions(+), 40 deletions(-)
Signature · 1 line
2.38.1
#1 lore raw

PATCH v2 1/3 tests/lcitool: Generate distribution packages list in JSON format

Add the generate_pkglist() helper to generate a list of packages required by a distribution to build QEMU.

Generate the FreeBSD JSON file (based on FreeBSD 13).

---
 tests/lcitool/refresh | 11 +++++++++++
 1 file changed, 11 insertions(+)
tests/lcitool/refresh +11 -0
index b54566edcc..a5df096074 100755
@@ -84,6 +84,12 @@ def generate_cirrus(target, trailer=None):
     generate(filename, cmd, trailer)
 
 
+def generate_pkglist(vm, target):
+    filename = Path(src_dir, "tests", "vm", vm + ".json")
+    cmd = lcitool_cmd + ["variables", "--format", "json", target, "qemu"]
+    generate(filename, cmd, None)
+
+
 # Netmap still needs to be manually built as it is yet to be packaged
 # into a distro. We also add cscope and gtags which are used in the CI
 # test
@@ -191,6 +197,11 @@ try:
     generate_cirrus("freebsd-13")
     generate_cirrus("macos-12")
 
+    #
+    # VM packages lists
+    #
+    generate_pkglist("freebsd", "freebsd-13")
+
     sys.exit(0)
 except Exception as ex:
     print(str(ex), file=sys.stderr)
Signature · 1 line
2.38.1
Reviewed-by on v2 1/3
#2 lore raw

Re: [PATCH v2 1/3] tests/lcitool: Generate distribution packages list in JSON format

Quoted 10 lines · Philippe Mathieu-Daudé

Add the generate_pkglist() helper to generate a list of packages required by a distribution to build QEMU.

Generate the FreeBSD JSON file (based on FreeBSD 13).

---
 tests/lcitool/refresh | 11 +++++++++++
 1 file changed, 11 insertions(+)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#3 lore raw

Re: [PATCH v2 1/3] tests/lcitool: Generate distribution packages list in JSON format

In reply to Philippe Mathieu-Daudé
Quoted 16 lines · Philippe Mathieu-Daudé

Add the generate_pkglist() helper to generate a list of packages required by a distribution to build QEMU.

Generate the FreeBSD JSON file (based on FreeBSD 13).

---
  tests/lcitool/refresh | 11 +++++++++++
  1 file changed, 11 insertions(+)
tests/lcitool/refresh +0 -0
index b54566edcc..a5df096074 100755

tests/lcitool/refresh def generate_cirrus(target, trailer=None): · line 90

Quoted 7 lines
tests/lcitool/refresh +4 -0
@@ -84,6 +84,12 @@ def generate_cirrus(target, trailer=None):
      generate(filename, cmd, trailer)
 
 
+def generate_pkglist(vm, target):
+    filename = Path(src_dir, "tests", "vm", vm + ".json")
+    cmd = lcitool_cmd + ["variables", "--format", "json", target, "qemu"]
+    generate(filename, cmd, None)

Note since json files don't have comments, we can't have the "THIS IS GENERATED BY A SCRIPT" in header.

#4 lore raw

PATCH v2 2/3 tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper

In reply to Philippe Mathieu-Daudé

Add the get_qemu_packages_from_lcitool_json() helper which return such package list from a lcitool env var file in JSON format.

---
 tests/vm/Makefile.include | 4 ++++
 tests/vm/basevm.py        | 9 +++++++++
 2 files changed, 13 insertions(+)
tests/vm/Makefile.include +4 -0
index c2a8ca1c17..b021b344b5 100644
@@ -106,6 +106,10 @@ $(IMAGES_DIR)/%.img:	$(SRC_PATH)/tests/vm/% \
 		--build-image $@, \
 		"  VM-IMAGE $*")
 
+$(SRC_PATH)/tests/vm/%.json:
+	$(call quiet-command, \
+		make lcitool-refresh)
+
 # Build in VM $(IMAGE)
 vm-build-%: $(IMAGES_DIR)/%.img $(VM_VENV)
 	$(call quiet-command, \
tests/vm/basevm.py +9 -0
index 23229e23d1..cc4cc0a973 100644
@@ -27,6 +27,7 @@
 import multiprocessing
 import traceback
 import shlex
+import json
 
 from qemu.machine import QEMUMachine
 from qemu.utils import get_info_usernet_hostfwd_port, kvm_available
@@ -501,6 +502,14 @@ def gen_cloud_init_iso(self):
                               stderr=self._stdout)
         return os.path.join(cidir, "cloud-init.iso")
 
+    def get_qemu_packages_from_lcitool_json(self, json_path=None):
+        """Parse a lcitool variables json file and return the PKGS list."""
+        if json_path is None:
+            json_path = os.path.join(os.path.dirname(__file__), self.name + ".json")
+        with open(json_path, "r") as fh:
+            return json.load(fh)["pkgs"]
+
+
 def get_qemu_path(arch, build_path=None):
     """Fetch the path to the qemu binary."""
     # If QEMU environment variable set, it takes precedence
Signature · 1 line
2.38.1
#5 lore raw

Re: [PATCH v2 2/3] tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper

Quoted 15 lines · Philippe Mathieu-Daudé

Add the get_qemu_packages_from_lcitool_json() helper which return such package list from a lcitool env var file in JSON format.

---
 tests/vm/Makefile.include | 4 ++++
 tests/vm/basevm.py        | 9 +++++++++
 2 files changed, 13 insertions(+)
tests/vm/Makefile.include +0 -0
index c2a8ca1c17..b021b344b5 100644

tests/vm/Makefile.include $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ · line 111

Quoted 6 lines
tests/vm/Makefile.include +3 -0
@@ -106,6 +106,10 @@ $(IMAGES_DIR)/%.img:	$(SRC_PATH)/tests/vm/% \
 		--build-image $@, \
 		"  VM-IMAGE $*")
 
+$(SRC_PATH)/tests/vm/%.json:
+	$(call quiet-command, \
+		make lcitool-refresh)

'lcitool-refresh' is something we only run explicitly when we change something about the lcitool, and also isn't provided to end users running from tarballs IIRC. So we shouldn't need to wire it up to build targets, just commit its output JSON file.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#6 lore raw

Re: [PATCH v2 2/3] tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper

Quoted 19 lines · Daniel P. Berrangé
Quoted 18 lines · Philippe Mathieu-Daudé

Add the get_qemu_packages_from_lcitool_json() helper which return such package list from a lcitool env var file in JSON format.

---
  tests/vm/Makefile.include | 4 ++++
  tests/vm/basevm.py        | 9 +++++++++
  2 files changed, 13 insertions(+)
tests/vm/Makefile.include +0 -0
index c2a8ca1c17..b021b344b5 100644
@@ -106,6 +106,10 @@ $(IMAGES_DIR)/%.img:	$(SRC_PATH)/tests/vm/% \
  		--build-image $@, \
  		"  VM-IMAGE $*")
 
Quoted 9 lines

tests/vm/Makefile.include $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ · line 111

Quoted 3 lines
tests/vm/Makefile.include +3 -0
+$(SRC_PATH)/tests/vm/%.json:
+	$(call quiet-command, \
+		make lcitool-refresh)

'lcitool-refresh' is something we only run explicitly when we change something about the lcitool, and also isn't provided to end users running from tarballs IIRC. So we shouldn't need to wire it up to build targets, just commit its output JSON file.

Right, simpler. Thanks!

#7 lore raw

PATCH v2 3/3 tests/vm/freebsd: Get up-to-date package list from lcitool vars file

In reply to Philippe Mathieu-Daudé

Get an up-to-date package list from lcitool, that way we don't need to manually keep this array in sync.

---
 tests/vm/Makefile.include |  2 ++
 tests/vm/freebsd          | 42 ++-------------------------------------
 2 files changed, 4 insertions(+), 40 deletions(-)
tests/vm/Makefile.include +2 -0
index b021b344b5..84b8ad5222 100644
@@ -110,6 +110,8 @@ $(SRC_PATH)/tests/vm/%.json:
 	$(call quiet-command, \
 		make lcitool-refresh)
 
+vm-build-freebsd: $(SRC_PATH)/tests/vm/freebsd.json
+
 # Build in VM $(IMAGE)
 vm-build-%: $(IMAGES_DIR)/%.img $(VM_VENV)
 	$(call quiet-command, \
tests/vm/freebsd +2 -40

55 lines not on this page · raw message

Signature · 1 line
2.38.1
#8 lore raw

Re: [PATCH v2 3/3] tests/vm/freebsd: Get up-to-date package list from lcitool vars file

Quoted 15 lines · Philippe Mathieu-Daudé

Get an up-to-date package list from lcitool, that way we don't need to manually keep this array in sync.

---
 tests/vm/Makefile.include |  2 ++
 tests/vm/freebsd          | 42 ++-------------------------------------
 2 files changed, 4 insertions(+), 40 deletions(-)
tests/vm/Makefile.include +0 -0
index b021b344b5..84b8ad5222 100644

tests/vm/Makefile.include $(SRC_PATH)/tests/vm/%.json: · line 113

Quoted 4 lines
tests/vm/Makefile.include +1 -0
@@ -110,6 +110,8 @@ $(SRC_PATH)/tests/vm/%.json:
 	$(call quiet-command, \
 		make lcitool-refresh)
 
+vm-build-freebsd: $(SRC_PATH)/tests/vm/freebsd.json

I don't think we need this, as we just commit the result of lcitool as needed.

Quoted 68 lines
tests/vm/Makefile.include +1 -0
+
 # Build in VM $(IMAGE)
 vm-build-%: $(IMAGES_DIR)/%.img $(VM_VENV)
 	$(call quiet-command, \
tests/vm/freebsd +2 -40
index 11de6473f4..ac51376c82 100755
@@ -31,45 +31,6 @@ class FreeBSDVM(basevm.BaseVM):
     link = "https://download.freebsd.org/releases/CI-IMAGES/13.2-RELEASE/amd64/Latest/FreeBSD-13.2-RELEASE-amd64-BASIC-CI.raw.xz"
     csum = "a4fb3b6c7b75dd4d58fb0d75e4caf72844bffe0ca00e66459c028b198ffb3c0e"
     size = "20G"
-    pkgs = [
-        # build tools
-        "git",
-        "pkgconf",
-        "bzip2",
-        "python39",
-        "ninja",
-
-        # gnu tools
-        "bash",
-        "gmake",
-        "gsed",
-        "gettext",
-
-        # libs: crypto
-        "gnutls",
-
-        # libs: images
-        "jpeg-turbo",
-        "png",
-
-        # libs: ui
-        "sdl2",
-        "gtk3",
-        "libxkbcommon",
-
-        # libs: opengl
-        "libepoxy",
-        "mesa-libs",
-
-        # libs: migration
-        "zstd",
-
-        # libs: networking
-        "libslirp",
-
-        # libs: sndio
-        "sndio",
-    ]
 
     BUILD_SCRIPT = """
         set -e;
@@ -151,8 +112,9 @@ class FreeBSDVM(basevm.BaseVM):
         self.console_wait(prompt)
         self.console_send("echo 'chmod 666 /dev/vtbd1' >> /etc/rc.local\n")
 
+        pkgs = self.get_qemu_packages_from_lcitool_json()
         self.print_step("Installing packages")
-        self.ssh_root_check("pkg install -y %s\n" % " ".join(self.pkgs))
+        self.ssh_root_check("pkg install -y %s\n" % " ".join(pkgs))
 
         # shutdown
         self.ssh_root(self.poweroff)
Signature · 1 line
2.38.1

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|