Skip to contentSkip to conversation

tests/vm: netbsd autoinstall, with IPv6 disabled

PATCH In review Eduardo Habkost 5 participants 11 messages Oct 18, 2019 – Oct 29, 2019

lore
Share & tools
lore
#0 lore raw

v5 cover tests/vm: netbsd autoinstall, with IPv6 disabled

I'm numbering this series v5 because it's a new version of the patch sent by Gerd, at:

Date: Mon, 17 Jun 2019 06:38:56 +0200
Message-Id: <[email protected]>
Subject: [PATCH v4 09/11] tests/vm: netbsd autoinstall, using serial console

Changes v4 -> v5:
* Rebase to latest qemu.git master
* Disable IPv6 by default (see
https://lore.kernel.org/qemu-devel/[email protected]/ for context)

Eduardo Habkost (2):
tests/vm: Let subclasses disable IPv6
tests/vm/netbsd: Disable IPv6

Gerd Hoffmann (1):
tests/vm: netbsd autoinstall, using serial console

 tests/vm/basevm.py |   5 +-
 tests/vm/netbsd    | 196 ++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 190 insertions(+), 11 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

PATCH v5 1/3 tests/vm: netbsd autoinstall, using serial console

From: Gerd Hoffmann <[email protected]>

Instead of fetching the prebuilt image from patchew download the install iso and prepare the image locally. Install to disk, using the serial console. Create qemu user, configure ssh login. Install packages needed for qemu builds.

[ehabkost: rebased to latest qemu.git master]
---
 tests/vm/netbsd | 189 +++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 179 insertions(+), 10 deletions(-)
tests/vm/netbsd +179 -10
index ee9eaeab50..49a99477f4 100755
@@ -2,10 +2,11 @@
 #
 # NetBSD VM image
 #
-# Copyright 2017 Red Hat Inc.
+# Copyright 2017-2019 Red Hat Inc.
 #
 # Authors:
 #  Fam Zheng <[email protected]>
+#  Gerd Hoffmann <[email protected]>
 #
 # This code is licensed under the GPL version 2 or later.  See
 # the COPYING file in the top-level directory.
@@ -13,30 +14,198 @@
 
 import os
 import sys
+import time
 import subprocess
 import basevm
 
 class NetBSDVM(basevm.BaseVM):
     name = "netbsd"
     arch = "x86_64"
+
+    link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso"
+    size = "20G"
+    pkgs = [
+        # tools
+        "git-base",
+        "pkgconf",
+        "xz",
+        "python37",
+
+        # gnu tools
+        "bash",
+        "gmake",
+        "gsed",
+        "flex", "bison",
+
+        # libs: crypto
+        "gnutls",
+
+        # libs: images
+        "jpeg",
+        "png",
+
+	# libs: ui
+        "SDL2",
+        "gtk3+",
+        "libxkbcommon",
+    ]
+
     BUILD_SCRIPT = """
         set -e;
-        rm -rf /var/tmp/qemu-test.*
-        cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+        rm -rf /home/qemu/qemu-test.*
+        cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
+        mkdir src build; cd src;
         tar -xf /dev/rld1a;
-        ./configure --python=python2.7 {configure_opts};
+        cd ../build
+        ../src/configure --python=python3.7 --disable-opengl {configure_opts};
         gmake --output-sync -j{jobs} {target} {verbose};
     """
+    poweroff = "/sbin/poweroff"
 
     def build_image(self, img):
-        cimg = self._download_with_cache("http://download.patchew.org/netbsd-7.1-amd64.img.xz",
-                                         sha256sum='b633d565b0eac3d02015cd0c81440bd8a7a8df8512615ac1ee05d318be015732')
-        img_tmp_xz = img + ".tmp.xz"
+        cimg = self._download_with_cache(self.link)
         img_tmp = img + ".tmp"
-        sys.stderr.write("Extracting the image...\n")
-        subprocess.check_call(["ln", "-f", cimg, img_tmp_xz])
-        subprocess.check_call(["xz", "--keep", "-dvf", img_tmp_xz])
+        iso = img + ".install.iso"
+
+        self.print_step("Preparing iso and disk image")
+        subprocess.check_call(["ln", "-f", cimg, iso])
+        subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
+                               img_tmp, self.size])
+
+        self.print_step("Booting installer")
+        self.boot(img_tmp, extra_args = [
+            "-bios", "pc-bios/bios-256k.bin",
+            "-machine", "graphics=off",
+            "-cdrom", iso
+        ])
+        self.console_init()
+        self.console_wait("Primary Bootstrap")
+
+        # serial console boot menu output doesn't work for some
+        # reason, so we have to fly blind ...
+        for char in list("5consdev com0\n"):
+            time.sleep(0.2)
+            self.console_send(char)
+            self.console_wait("")
+        self.console_wait_send("> ", "boot\n")
+
+        self.console_wait_send("Terminal type",            "xterm\n")
+        self.console_wait_send("a: Installation messages", "a\n")
+        self.console_wait_send("b: US-English",            "b\n")
+        self.console_wait_send("a: Install NetBSD",        "a\n")
+        self.console_wait("Shall we continue?")
+        self.console_wait_send("b: Yes",                   "b\n")
+
+        self.console_wait_send("a: ld0",                   "a\n")
+        self.console_wait_send("a: This is the correct",   "a\n")
+        self.console_wait_send("b: Use the entire disk",   "b\n")
+        self.console_wait("NetBSD bootcode")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait_send("b: Use existing part",     "b\n")
+        self.console_wait_send("x: Partition sizes ok",    "x\n")
+        self.console_wait_send("for your NetBSD disk",     "\n")
+        self.console_wait("Shall we continue?")
+        self.console_wait_send("b: Yes",                   "b\n")
+
+        self.console_wait_send("b: Use serial port com0",  "b\n")
+        self.console_wait_send("f: Set serial baud rate",  "f\n")
+        self.console_wait_send("a: 9600",                  "a\n")
+        self.console_wait_send("x: Exit",                  "x\n")
+
+        self.console_wait_send("a: Full installation",     "a\n")
+        self.console_wait_send("a: CD-ROM",                "a\n")
+
+        self.print_step("Installation started now, this will take a while")
+        self.console_wait_send("Hit enter to continue",    "\n")
+
+        self.console_wait_send("d: Change root password",  "d\n")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+        self.console_wait("Retype new password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+
+        self.console_wait_send("o: Add a user",            "o\n")
+        self.console_wait("username")
+        self.console_send("%s\n" % self.GUEST_USER)
+        self.console_wait("to group wheel")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait_send("a: /bin/sh",               "a\n")
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+        self.console_wait("Retype new password:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+
+        self.console_wait_send("a: Configure network",     "a\n")
+        self.console_wait_send("a: vioif0",                "a\n")
+        self.console_wait_send("Network media type",       "\n")
+        self.console_wait("autoconfiguration")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait_send("DNS domain",               "localnet\n")
+        self.console_wait("Are they OK?")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait("installed in /etc")
+        self.console_wait_send("a: Yes",                   "a\n")
+
+        self.console_wait_send("e: Enable install",        "e\n")
+        proxy = os.environ.get("http_proxy")
+        if not proxy is None:
+            self.console_wait_send("f: Proxy",             "f\n")
+            self.console_wait("Proxy")
+            self.console_send("%s\n" % proxy)
+        self.console_wait_send("x: Install pkgin",         "x\n")
+        self.console_init(1200)
+        self.console_wait_send("Hit enter to continue", "\n")
+        self.console_init()
+
+        self.console_wait_send("g: Enable sshd",           "g\n")
+        self.console_wait_send("x: Finished conf",         "x\n")
+        self.console_wait_send("Hit enter to continue",    "\n")
+
+        self.print_step("Installation finished, rebooting")
+        self.console_wait_send("d: Reboot the computer",   "d\n")
+
+        # setup qemu user
+        prompt = "localhost$"
+        self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
+        self.console_wait_send(prompt, "exit\n")
+
+        # setup root user
+        prompt = "localhost#"
+        self.console_ssh_init(prompt, "root", self.ROOT_PASS)
+        self.console_sshd_config(prompt)
+
+        # setup virtio-blk #1 (tarfile)
+        self.console_wait(prompt)
+        self.console_send("echo 'chmod 666 /dev/rld1a' >> /etc/rc.local\n")
+
+        # turn off mprotect (conflicts with tcg)
+        self.console_wait(prompt)
+        self.console_send("echo security.pax.mprotect.enabled=0 >> /etc/sysctl.conf\n")
+
+        self.print_step("Configuration finished, rebooting")
+        self.console_wait_send(prompt, "reboot\n")
+        self.console_wait("login:")
+        self.wait_ssh()
+
+        self.print_step("Installing packages")
+        self.ssh_root_check("pkgin update\n")
+        self.ssh_root_check("pkgin -y install %s\n" % " ".join(self.pkgs))
+
+        # shutdown
+        self.ssh_root(self.poweroff)
+        self.console_wait("entering state S5")
+        self.wait()
+
+        if os.path.exists(img):
+            os.remove(img)
         os.rename(img_tmp, img)
+        os.remove(iso)
+        self.print_step("All done")
 
 if __name__ == "__main__":
     sys.exit(basevm.main(NetBSDVM))
Signature · 1 line
2.21.0
#2 lore raw

Re: [PATCH v5 1/3] tests/vm: netbsd autoinstall, using serial console

Quoted 36 lines · Eduardo Habkost

From: Gerd Hoffmann <[email protected]>

Instead of fetching the prebuilt image from patchew download the install iso and prepare the image locally. Install to disk, using the serial console. Create qemu user, configure ssh login. Install packages needed for qemu builds.

[ehabkost: rebased to latest qemu.git master]
---
 tests/vm/netbsd | 189 +++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 179 insertions(+), 10 deletions(-)
tests/vm/netbsd +2 -1
index ee9eaeab50..49a99477f4 100755
@@ -2,10 +2,11 @@
 #
 # NetBSD VM image
 #
-# Copyright 2017 Red Hat Inc.
+# Copyright 2017-2019 Red Hat Inc.
 #
 # Authors:
 #  Fam Zheng <[email protected]>
+#  Gerd Hoffmann <[email protected]>
 #
 # This code is licensed under the GPL version 2 or later.  See
 # the COPYING file in the top-level directory.
@@ -13,30 +14,198 @@
 
 import os

tests/vm/netbsd line 25

Quoted 10 lines
tests/vm/netbsd +3 -0
 import sys
+import time
 import subprocess
 import basevm
 
 class NetBSDVM(basevm.BaseVM):
     name = "netbsd"
     arch = "x86_64"
+
+    link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso"

I'd like to suggest to go immediately with 8.1 instead of 8.0. I tested it and it worked for me out-of-the-box, without further modifications.

tests/vm/netbsd line 205

Quoted 3 lines
tests/vm/netbsd +3 -0
+
+        if os.path.exists(img):
+            os.remove(img)

These two lines have been removed recently with commit fcd2060e8efff83b7bdef04323077f87e011fdc4 ... please drop them from the patch.

Thanks,
Thomas

#3 lore raw

Re: [PATCH v5 1/3] tests/vm: netbsd autoinstall, using serial console

Quoted 42 lines · Thomas Huth
Quoted 41 lines · Eduardo Habkost

From: Gerd Hoffmann <[email protected]>

Instead of fetching the prebuilt image from patchew download the install iso and prepare the image locally. Install to disk, using the serial console. Create qemu user, configure ssh login. Install packages needed for qemu builds.

[ehabkost: rebased to latest qemu.git master]
---
 tests/vm/netbsd | 189 +++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 179 insertions(+), 10 deletions(-)
tests/vm/netbsd +3 -1
index ee9eaeab50..49a99477f4 100755
@@ -2,10 +2,11 @@
 #
 # NetBSD VM image
 #
-# Copyright 2017 Red Hat Inc.
+# Copyright 2017-2019 Red Hat Inc.
 #
 # Authors:
 #  Fam Zheng <[email protected]>
+#  Gerd Hoffmann <[email protected]>
 #
 # This code is licensed under the GPL version 2 or later.  See
 # the COPYING file in the top-level directory.
@@ -13,30 +14,198 @@
 
 import os
 import sys
+import time
 import subprocess
 import basevm
 
Quoted 8 lines

tests/vm/netbsd line 25

Quoted 5 lines
tests/vm/netbsd +2 -0
 class NetBSDVM(basevm.BaseVM):
     name = "netbsd"
     arch = "x86_64"
+
+    link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso"

I'd like to suggest to go immediately with 8.1 instead of 8.0. I tested it and it worked for me out-of-the-box, without further modifications.

*Sigh* I had to drop this from my PR after it broke again. It looked
like it was getting stuck on the installer waiting for something. I had it working at one point so I'm not sure why it's not working again.

The rest of the patches are merged now though.

Quoted 11 lines

tests/vm/netbsd line 205

Quoted 4 lines
tests/vm/netbsd +3 -0
+
+        if os.path.exists(img):
+            os.remove(img)

These two lines have been removed recently with commit fcd2060e8efff83b7bdef04323077f87e011fdc4 ... please drop them from the patch.

Thanks,
Thomas

--
Alex Bennée

#4 lore raw

PATCH v5 2/3 tests/vm: Let subclasses disable IPv6

In reply to Eduardo Habkost

The mechanism will be used to work around issues related to IPv6 on the netbsd image builder.

---
 tests/vm/basevm.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
tests/vm/basevm.py +4 -1
index b5d1479bee..2929de23aa 100755
@@ -57,6 +57,8 @@ class BaseVM(object):
     arch = "#arch"
     # command to halt the guest, can be overridden by subclasses
     poweroff = "poweroff"
+    # enable IPv6 networking
+    ipv6 = True
     def __init__(self, debug=False, vcpus=None):
         self._guest = None
         self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-",
@@ -81,7 +83,8 @@ class BaseVM(object):
         self._args = [ \
             "-nodefaults", "-m", "4G",
             "-cpu", "max",
-            "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22",
+            "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22" +
+                       (",ipv6=no" if not self.ipv6 else ""),
             "-device", "virtio-net-pci,netdev=vnet",
             "-vnc", "127.0.0.1:0,to=20"]
         if vcpus and vcpus > 1:
Signature · 1 line
2.21.0
Reviewed-by on v5 2/3
#5 lore raw

Re: [PATCH v5 2/3] tests/vm: Let subclasses disable IPv6

Quoted 23 lines · Eduardo Habkost

The mechanism will be used to work around issues related to IPv6 on the netbsd image builder.

---
  tests/vm/basevm.py | 5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)
tests/vm/basevm.py +2 -0
index b5d1479bee..2929de23aa 100755
@@ -57,6 +57,8 @@ class BaseVM(object):
      arch = "#arch"
      # command to halt the guest, can be overridden by subclasses
      poweroff = "poweroff"
+    # enable IPv6 networking
+    ipv6 = True
      def __init__(self, debug=False, vcpus=None):
          self._guest = None
          self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-",

tests/vm/basevm.py class BaseVM(object): · line 90

Quoted 9 lines
tests/vm/basevm.py +2 -1
@@ -81,7 +83,8 @@ class BaseVM(object):
          self._args = [ \
              "-nodefaults", "-m", "4G",
              "-cpu", "max",
-            "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22",
+            "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22" +
+                       (",ipv6=no" if not self.ipv6 else ""),
              "-device", "virtio-net-pci,netdev=vnet",
              "-vnc", "127.0.0.1:0,to=20"]
          if vcpus and vcpus > 1:
Reviewed-by on v5 2/3
#6 lore raw

Re: [PATCH v5 2/3] tests/vm: Let subclasses disable IPv6

In reply to Eduardo Habkost
Quoted 23 lines · Eduardo Habkost

The mechanism will be used to work around issues related to IPv6 on the netbsd image builder.

---
 tests/vm/basevm.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
tests/vm/basevm.py +2 -0
index b5d1479bee..2929de23aa 100755
@@ -57,6 +57,8 @@ class BaseVM(object):
     arch = "#arch"
     # command to halt the guest, can be overridden by subclasses
     poweroff = "poweroff"
+    # enable IPv6 networking
+    ipv6 = True
     def __init__(self, debug=False, vcpus=None):
         self._guest = None
         self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-",

tests/vm/basevm.py class BaseVM(object): · line 90

Quoted 9 lines
tests/vm/basevm.py +2 -1
@@ -81,7 +83,8 @@ class BaseVM(object):
         self._args = [ \
             "-nodefaults", "-m", "4G",
             "-cpu", "max",
-            "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22",
+            "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22" +
+                       (",ipv6=no" if not self.ipv6 else ""),
             "-device", "virtio-net-pci,netdev=vnet",
             "-vnc", "127.0.0.1:0,to=20"]
         if vcpus and vcpus > 1:
#7 lore raw

PATCH v5 3/3 tests/vm/netbsd: Disable IPv6

In reply to Eduardo Habkost

Workaround for issues when the host has no IPv6 connectivity.

---
 tests/vm/netbsd | 7 +++++++
 1 file changed, 7 insertions(+)
tests/vm/netbsd +7 -0
index 49a99477f4..5e04dcd9b1 100755
@@ -62,6 +62,13 @@ class NetBSDVM(basevm.BaseVM):
     """
     poweroff = "/sbin/poweroff"
 
+    # Workaround for NetBSD + IPv6 + slirp issues.
+    # NetBSD seems to ignore the ICMPv6 Destination Unreachable
+    # messages generated by slirp.  When the host has no IPv6
+    # connectivity, this causes every connection to ftp.NetBSD.org
+    # take more than a minute to be established.
+    ipv6 = False
+
     def build_image(self, img):
         cimg = self._download_with_cache(self.link)
         img_tmp = img + ".tmp"
Signature · 1 line
2.21.0
Reviewed-by on v5 3/3
#8 lore raw

Re: [PATCH v5 3/3] tests/vm/netbsd: Disable IPv6

Quoted 16 lines · Eduardo Habkost

Workaround for issues when the host has no IPv6 connectivity.

---
 tests/vm/netbsd | 7 +++++++
 1 file changed, 7 insertions(+)
tests/vm/netbsd +0 -0
index 49a99477f4..5e04dcd9b1 100755
@@ -62,6 +62,13 @@ class NetBSDVM(basevm.BaseVM):
     """
     poweroff = "/sbin/poweroff"
 

tests/vm/netbsd class NetBSDVM(basevm.BaseVM): · line 74

Quoted 10 lines
tests/vm/netbsd +7 -0
+    # Workaround for NetBSD + IPv6 + slirp issues.
+    # NetBSD seems to ignore the ICMPv6 Destination Unreachable
+    # messages generated by slirp.  When the host has no IPv6
+    # connectivity, this causes every connection to ftp.NetBSD.org
+    # take more than a minute to be established.
+    ipv6 = False
+
     def build_image(self, img):
         cimg = self._download_with_cache(self.link)
         img_tmp = img + ".tmp"
#9 lore raw
In reply to Eduardo Habkost
Quoted 7 lines · Eduardo Habkost

I'm numbering this series v5 because it's a new version of the patch sent by Gerd, at:

Date: Mon, 17 Jun 2019 06:38:56 +0200
Message-Id: <[email protected]>
Subject: [PATCH v4 09/11] tests/vm: netbsd autoinstall, using serial console

Queued to testing/next, thanks. I've made the changes Thomas suggested in his review.

Quoted 10 lines

Changes v4 -> v5:
* Rebase to latest qemu.git master
* Disable IPv6 by default (see
https://lore.kernel.org/qemu-devel/[email protected]/ for context)

Eduardo Habkost (2):
tests/vm: Let subclasses disable IPv6
tests/vm/netbsd: Disable IPv6

Quoted 6 lines

Gerd Hoffmann (1):
tests/vm: netbsd autoinstall, using serial console

 tests/vm/basevm.py |   5 +-
 tests/vm/netbsd    | 196 ++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 190 insertions(+), 11 deletions(-)

--
Alex Bennée

#10 lore raw
Quoted 6 lines · Alex Bennée
Quoted 3 lines · Eduardo Habkost

I'm numbering this series v5 because it's a new version of the patch sent by Gerd, at:

Quoted 18 lines
Quoted 4 lines

Date: Mon, 17 Jun 2019 06:38:56 +0200
Message-Id: <[email protected]>
Subject: [PATCH v4 09/11] tests/vm: netbsd autoinstall, using serial console

Queued to testing/next, thanks. I've made the changes Thomas suggested in his review.

Quoted 10 lines

Changes v4 -> v5:
* Rebase to latest qemu.git master
* Disable IPv6 by default (see
https://lore.kernel.org/qemu-devel/[email protected]/ for context)

Eduardo Habkost (2):
tests/vm: Let subclasses disable IPv6
tests/vm/netbsd: Disable IPv6

I'm going to report this one to other NetBSD developers and hopefully the problem will go away in future.

I have faced such problems in the past myself but didn't know what and who is culprit.

Quoted 11 lines
Quoted 6 lines

Gerd Hoffmann (1):
tests/vm: netbsd autoinstall, using serial console

 tests/vm/basevm.py |   5 +-
 tests/vm/netbsd    | 196 ++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 190 insertions(+), 11 deletions(-)

--
Alex Bennée