Skip to contentSkip to conversation

tests/vm: netbsd autoinstall

PATCH In review Gerd Hoffmann 4 participants 10 messages Oct 31, 2019 – Nov 11, 2019

lore
Share & tools
lore
#0 lore raw

v6 cover tests/vm: netbsd autoinstall

Hopefully the last revision on this one ...

Patch #1 is the latest patch sent by Eduardo, unmodified. Patch #2+#3 fix the installer boot hang.
Patch #4 finally updates to netbsd version 8.1

Gerd Hoffmann (4):
tests/vm: netbsd autoinstall, using serial console tests/vm: add console_consume helper tests/vm: use console_consume for netbsd tests/vm: update netbsd to version 8.1

 tests/vm/basevm.py |  19 +++++
 tests/vm/netbsd    | 189 ++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 198 insertions(+), 10 deletions(-)
Signature · 1 line
2.18.1
#1 lore raw

PATCH v6 1/4 tests/vm: netbsd autoinstall, using serial console

Instead of fetching the prebuilt image from patchew download the install iso and prepare the image locally. Install to disk, using the serial console. Create qemu user, configure ssh login. Install packages needed for qemu builds.

[ehabkost: rebased to latest qemu.git master]
---
 tests/vm/netbsd | 189 +++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 179 insertions(+), 10 deletions(-)
tests/vm/netbsd +179 -10
index 18aa56ae82..5e04dcd9b1 100755
@@ -2,10 +2,11 @@
 #
 # NetBSD VM image
 #
-# Copyright 2017 Red Hat Inc.
+# Copyright 2017-2019 Red Hat Inc.
 #
 # Authors:
 #  Fam Zheng <[email protected]>
+#  Gerd Hoffmann <[email protected]>
 #
 # This code is licensed under the GPL version 2 or later.  See
 # the COPYING file in the top-level directory.
@@ -13,20 +14,53 @@
 
 import os
 import sys
+import time
 import subprocess
 import basevm
 
 class NetBSDVM(basevm.BaseVM):
     name = "netbsd"
     arch = "x86_64"
+
+    link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso"
+    size = "20G"
+    pkgs = [
+        # tools
+        "git-base",
+        "pkgconf",
+        "xz",
+        "python37",
+
+        # gnu tools
+        "bash",
+        "gmake",
+        "gsed",
+        "flex", "bison",
+
+        # libs: crypto
+        "gnutls",
+
+        # libs: images
+        "jpeg",
+        "png",
+
+	# libs: ui
+        "SDL2",
+        "gtk3+",
+        "libxkbcommon",
+    ]
+
     BUILD_SCRIPT = """
         set -e;
-        rm -rf /var/tmp/qemu-test.*
-        cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+        rm -rf /home/qemu/qemu-test.*
+        cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
+        mkdir src build; cd src;
         tar -xf /dev/rld1a;
-        ./configure --python=python2.7 {configure_opts};
+        cd ../build
+        ../src/configure --python=python3.7 --disable-opengl {configure_opts};
         gmake --output-sync -j{jobs} {target} {verbose};
     """
+    poweroff = "/sbin/poweroff"
 
     # Workaround for NetBSD + IPv6 + slirp issues.
     # NetBSD seems to ignore the ICMPv6 Destination Unreachable
@@ -36,14 +70,149 @@ class NetBSDVM(basevm.BaseVM):
     ipv6 = False
 
     def build_image(self, img):
-        cimg = self._download_with_cache("http://download.patchew.org/netbsd-7.1-amd64.img.xz",
-                                         sha256sum='b633d565b0eac3d02015cd0c81440bd8a7a8df8512615ac1ee05d318be015732')
-        img_tmp_xz = img + ".tmp.xz"
+        cimg = self._download_with_cache(self.link)
         img_tmp = img + ".tmp"
-        sys.stderr.write("Extracting the image...\n")
-        subprocess.check_call(["ln", "-f", cimg, img_tmp_xz])
-        subprocess.check_call(["xz", "--keep", "-dvf", img_tmp_xz])
+        iso = img + ".install.iso"
+
+        self.print_step("Preparing iso and disk image")
+        subprocess.check_call(["ln", "-f", cimg, iso])
+        subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
+                               img_tmp, self.size])
+
+        self.print_step("Booting installer")
+        self.boot(img_tmp, extra_args = [
+            "-bios", "pc-bios/bios-256k.bin",
+            "-machine", "graphics=off",
+            "-cdrom", iso
+        ])
+        self.console_init()
+        self.console_wait("Primary Bootstrap")
+
+        # serial console boot menu output doesn't work for some
+        # reason, so we have to fly blind ...
+        for char in list("5consdev com0\n"):
+            time.sleep(0.2)
+            self.console_send(char)
+            self.console_wait("")
+        self.console_wait_send("> ", "boot\n")
+
+        self.console_wait_send("Terminal type",            "xterm\n")
+        self.console_wait_send("a: Installation messages", "a\n")
+        self.console_wait_send("b: US-English",            "b\n")
+        self.console_wait_send("a: Install NetBSD",        "a\n")
+        self.console_wait("Shall we continue?")
+        self.console_wait_send("b: Yes",                   "b\n")
+
+        self.console_wait_send("a: ld0",                   "a\n")
+        self.console_wait_send("a: This is the correct",   "a\n")
+        self.console_wait_send("b: Use the entire disk",   "b\n")
+        self.console_wait("NetBSD bootcode")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait_send("b: Use existing part",     "b\n")
+        self.console_wait_send("x: Partition sizes ok",    "x\n")
+        self.console_wait_send("for your NetBSD disk",     "\n")
+        self.console_wait("Shall we continue?")
+        self.console_wait_send("b: Yes",                   "b\n")
+
+        self.console_wait_send("b: Use serial port com0",  "b\n")
+        self.console_wait_send("f: Set serial baud rate",  "f\n")
+        self.console_wait_send("a: 9600",                  "a\n")
+        self.console_wait_send("x: Exit",                  "x\n")
+
+        self.console_wait_send("a: Full installation",     "a\n")
+        self.console_wait_send("a: CD-ROM",                "a\n")
+
+        self.print_step("Installation started now, this will take a while")
+        self.console_wait_send("Hit enter to continue",    "\n")
+
+        self.console_wait_send("d: Change root password",  "d\n")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+        self.console_wait("Retype new password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+
+        self.console_wait_send("o: Add a user",            "o\n")
+        self.console_wait("username")
+        self.console_send("%s\n" % self.GUEST_USER)
+        self.console_wait("to group wheel")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait_send("a: /bin/sh",               "a\n")
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+        self.console_wait("Retype new password:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+
+        self.console_wait_send("a: Configure network",     "a\n")
+        self.console_wait_send("a: vioif0",                "a\n")
+        self.console_wait_send("Network media type",       "\n")
+        self.console_wait("autoconfiguration")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait_send("DNS domain",               "localnet\n")
+        self.console_wait("Are they OK?")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait("installed in /etc")
+        self.console_wait_send("a: Yes",                   "a\n")
+
+        self.console_wait_send("e: Enable install",        "e\n")
+        proxy = os.environ.get("http_proxy")
+        if not proxy is None:
+            self.console_wait_send("f: Proxy",             "f\n")
+            self.console_wait("Proxy")
+            self.console_send("%s\n" % proxy)
+        self.console_wait_send("x: Install pkgin",         "x\n")
+        self.console_init(1200)
+        self.console_wait_send("Hit enter to continue", "\n")
+        self.console_init()
+
+        self.console_wait_send("g: Enable sshd",           "g\n")
+        self.console_wait_send("x: Finished conf",         "x\n")
+        self.console_wait_send("Hit enter to continue",    "\n")
+
+        self.print_step("Installation finished, rebooting")
+        self.console_wait_send("d: Reboot the computer",   "d\n")
+
+        # setup qemu user
+        prompt = "localhost$"
+        self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
+        self.console_wait_send(prompt, "exit\n")
+
+        # setup root user
+        prompt = "localhost#"
+        self.console_ssh_init(prompt, "root", self.ROOT_PASS)
+        self.console_sshd_config(prompt)
+
+        # setup virtio-blk #1 (tarfile)
+        self.console_wait(prompt)
+        self.console_send("echo 'chmod 666 /dev/rld1a' >> /etc/rc.local\n")
+
+        # turn off mprotect (conflicts with tcg)
+        self.console_wait(prompt)
+        self.console_send("echo security.pax.mprotect.enabled=0 >> /etc/sysctl.conf\n")
+
+        self.print_step("Configuration finished, rebooting")
+        self.console_wait_send(prompt, "reboot\n")
+        self.console_wait("login:")
+        self.wait_ssh()
+
+        self.print_step("Installing packages")
+        self.ssh_root_check("pkgin update\n")
+        self.ssh_root_check("pkgin -y install %s\n" % " ".join(self.pkgs))
+
+        # shutdown
+        self.ssh_root(self.poweroff)
+        self.console_wait("entering state S5")
+        self.wait()
+
+        if os.path.exists(img):
+            os.remove(img)
         os.rename(img_tmp, img)
+        os.remove(iso)
+        self.print_step("All done")
 
 if __name__ == "__main__":
     sys.exit(basevm.main(NetBSDVM))
Signature · 1 line
2.18.1
#2 lore raw

Re: [PATCH v6 1/4] tests/vm: netbsd autoinstall, using serial console

Quoted 13 lines · Gerd Hoffmann

Instead of fetching the prebuilt image from patchew download the install iso and prepare the image locally. Install to disk, using the serial console. Create qemu user, configure ssh login. Install packages needed for qemu builds.

[ehabkost: rebased to latest qemu.git master]
---
 tests/vm/netbsd | 189 +++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 179 insertions(+), 10 deletions(-)

[...]

tests/vm/netbsd class NetBSDVM(basevm.BaseVM): · line 212

Quoted 7 lines
tests/vm/netbsd +7 -0
+        # shutdown
+        self.ssh_root(self.poweroff)
+        self.console_wait("entering state S5")
+        self.wait()
+
+        if os.path.exists(img):
+            os.remove(img)

What about removing the above two lines again?
(see commit fcd2060e8efff83b7bde)

Thomas

#3 lore raw

PATCH v6 2/4 tests/vm: add console_consume helper

In reply to Gerd Hoffmann

Helper function to read all console output.

---
 tests/vm/basevm.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
tests/vm/basevm.py +19 -0
index 2929de23aa..086bfb2c66 100755
@@ -242,6 +242,25 @@ class BaseVM(object):
             return False
         return True
 
+    def console_consume(self):
+        vm = self._guest
+        output = ""
+        vm.console_socket.setblocking(0)
+        while True:
+            try:
+                chars = vm.console_socket.recv(1)
+            except:
+                break
+            output += chars.decode("latin1")
+            if "\r" in output or "\n" in output:
+                lines = re.split("[\r\n]", output)
+                output = lines.pop()
+                if self.debug:
+                    self.console_log("\n".join(lines))
+        if self.debug:
+            self.console_log(output)
+        vm.console_socket.setblocking(1)
+
     def console_send(self, command):
         vm = self._guest
         if self.debug:
Signature · 1 line
2.18.1
Reviewed-by on v6 2/4
#4 lore raw

Re: [PATCH v6 2/4] tests/vm: add console_consume helper

Quoted 15 lines · Gerd Hoffmann

Helper function to read all console output.

---
  tests/vm/basevm.py | 19 +++++++++++++++++++
  1 file changed, 19 insertions(+)
tests/vm/basevm.py +0 -0
index 2929de23aa..086bfb2c66 100755
@@ -242,6 +242,25 @@ class BaseVM(object):
              return False
          return True
 

tests/vm/basevm.py class BaseVM(object): · line 254

Quoted 10 lines
tests/vm/basevm.py +10 -0
+    def console_consume(self):
+        vm = self._guest
+        output = ""
+        vm.console_socket.setblocking(0)
+        while True:
+            try:
+                chars = vm.console_socket.recv(1)
+            except:
+                break
+            output += chars.decode("latin1")

I don't know enough decode(), but I'm very happy with the rest of the patch, thanks a lot for fixing this long standing issue Gerd!

Quoted 13 lines
tests/vm/basevm.py +9 -0
+            if "\r" in output or "\n" in output:
+                lines = re.split("[\r\n]", output)
+                output = lines.pop()
+                if self.debug:
+                    self.console_log("\n".join(lines))
+        if self.debug:
+            self.console_log(output)
+        vm.console_socket.setblocking(1)
+
      def console_send(self, command):
          vm = self._guest
          if self.debug:
#5 lore raw

PATCH v6 3/4 tests/vm: use console_consume for netbsd

In reply to Gerd Hoffmann

Use new helper to read all pending console output, not just a single char. Unblocks installer boot.

---
 tests/vm/netbsd | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/vm/netbsd +1 -1
index 5e04dcd9b1..d1bccccfd0 100755
@@ -93,7 +93,7 @@ class NetBSDVM(basevm.BaseVM):
         for char in list("5consdev com0\n"):
             time.sleep(0.2)
             self.console_send(char)
-            self.console_wait("")
+            self.console_consume()
         self.console_wait_send("> ", "boot\n")
 
         self.console_wait_send("Terminal type",            "xterm\n")
Signature · 1 line
2.18.1
#6 lore raw

Re: [PATCH v6 3/4] tests/vm: use console_consume for netbsd

Thank you for this work. I hope it will be fine now.

Quoted 22 lines · Gerd Hoffmann

Use new helper to read all pending console output, not just a single char. Unblocks installer boot.

---
 tests/vm/netbsd | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/vm/netbsd +1 -1
index 5e04dcd9b1..d1bccccfd0 100755
@@ -93,7 +93,7 @@ class NetBSDVM(basevm.BaseVM):
         for char in list("5consdev com0\n"):
             time.sleep(0.2)
             self.console_send(char)
-            self.console_wait("")
+            self.console_consume()
         self.console_wait_send("> ", "boot\n")
 
         self.console_wait_send("Terminal type",            "xterm\n")
Reviewed-by on v6 3/4
#7 lore raw

Re: [PATCH v6 3/4] tests/vm: use console_consume for netbsd

In reply to Gerd Hoffmann

tests/vm/netbsd class NetBSDVM(basevm.BaseVM):

Quoted 13 lines · Gerd Hoffmann

Use new helper to read all pending console output, not just a single char. Unblocks installer boot.

---
  tests/vm/netbsd | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/vm/netbsd +0 -0
index 5e04dcd9b1..d1bccccfd0 100755
@@ -93,7 +93,7 @@ class NetBSDVM(basevm.BaseVM):

Why not use this by default for everything?

Anyway,

Quoted 9 lines
tests/vm/netbsd +1 -1
          for char in list("5consdev com0\n"):
              time.sleep(0.2)
              self.console_send(char)
-            self.console_wait("")
+            self.console_consume()
          self.console_wait_send("> ", "boot\n")
 
          self.console_wait_send("Terminal type",            "xterm\n")
#8 lore raw

PATCH v6 4/4 tests/vm: update netbsd to version 8.1

In reply to Gerd Hoffmann
---
 tests/vm/netbsd | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/vm/netbsd +1 -1
index d1bccccfd0..33779402dd 100755
@@ -22,7 +22,7 @@ class NetBSDVM(basevm.BaseVM):
     name = "netbsd"
     arch = "x86_64"
 
-    link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso"
+    link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.1/images/NetBSD-8.1-amd64.iso"
     size = "20G"
     pkgs = [
         # tools
Signature · 1 line
2.18.1
Reviewed-by on v6 4/4
#9 lore raw

Re: [PATCH v6 4/4] tests/vm: update netbsd to version 8.1

Quoted 11 lines · Gerd Hoffmann
---
  tests/vm/netbsd | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/vm/netbsd +0 -0
index d1bccccfd0..33779402dd 100755

tests/vm/netbsd class NetBSDVM(basevm.BaseVM): · line 28

Quoted 8 lines
tests/vm/netbsd +1 -1
@@ -22,7 +22,7 @@ class NetBSDVM(basevm.BaseVM):
      name = "netbsd"
      arch = "x86_64"
 
-    link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso"
+    link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.1/images/NetBSD-8.1-amd64.iso"
      size = "20G"
      pkgs = [
          # tools