runqemu: fixes for cpio.gz, nfs and slirp

PATCH Superseded Robert Yang 2 participants 10 messages Nov 29, 2016

lore

Share & tools

lore

Robert Yang

author

2016-11-29 07:35 UTC #0 lore raw

cover runqemu: fixes for cpio.gz, nfs and slirp

The following changes since commit af280990bff4a484fd8a18e3442b56a0a39b7611:

Revert "classes/populate_sdk_ext: require uninative" (2016-11-28 14:49:37 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/qemu
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/qemu

Robert Yang (5):
runqemu: fix checking for <file>.cpio.gz qemuboot.bbclass: use IMGDEPLOYDIR
runqemu-export-rootfs: fix inconsistent var names
runqemu: support mutiple qemus running when nfs
runqemu: set default device and hostfwd for slirp

 meta/classes/qemuboot.bbclass              |   5 +-
 meta/conf/machine/include/qemuboot-x86.inc |   1 -
 meta/conf/machine/qemuarm64.conf           |   1 -
 scripts/runqemu                            | 111 ++++++++++++++++++++++-------
 scripts/runqemu-export-rootfs              |  21 +++---
 5 files changed, 96 insertions(+), 43 deletions(-)

Signature · 1 line

2.9.0

Robert Yang

author

2016-11-29 07:35 UTC #1 lore raw

PATCH 1/5 runqemu: fix checking for <file>.cpio.gz

When "runqemu /path/to/<file>.cpio.gz", it used the last suffix "gz" as the fstype which was wrong. Check filename against self.fstypes firstly can fix the problem.

Signed-off-by Robert Yang <[email protected]>

---
 scripts/runqemu | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

scripts/runqemu +16 -6

index af25423..c737eb2 100755
@@ -269,19 +269,29 @@ class BaseConfig(object):
             self.kernel =  p
         elif os.path.exists(p) and (not os.path.isdir(p)) and re.search('-image-', os.path.basename(p)):
             self.rootfs = p
-            dirpath = os.path.dirname(p)
-            m = re.search('(.*)\.(.*)$', p)
-            if m:
-                qb = '%s%s' % (re.sub('\.rootfs$', '', m.group(1)), '.qemuboot.conf')
+            # Check filename against self.fstypes can hanlde <file>.cpio.gz,
+            # otherwise, its type would be "gz", which is incorrect.
+            fst = ""
+            for t in self.fstypes:
+                if p.endswith(t):
+                    fst = t
+                    break
+            if not fst:
+                m = re.search('.*\.(.*)$', self.rootfs)
+                if m:
+                    fst =  m.group(1)
+            if fst:
+                self.check_arg_fstype(fst)
+                qb = re.sub('\.' + fst + "$", '', self.rootfs)
+                qb = '%s%s' % (re.sub('\.rootfs$', '', qb), '.qemuboot.conf')
                 if os.path.exists(qb):
                     self.qemuboot = qb
                     self.qbconfload = True
                 else:
                     logger.warn("%s doesn't exist" % qb)
-                fst = m.group(2)
-                self.check_arg_fstype(fst)
             else:
                 raise Exception("Can't find FSTYPE from: %s" % p)
+
         elif os.path.isdir(p) or re.search(':', arg) and re.search('/', arg):
             if self.is_deploy_dir_image(p):
                 logger.info('DEPLOY_DIR_IMAGE: %s' % p)

Signature · 1 line

2.9.0

Robert Yang

author

2016-11-29 07:35 UTC #2 lore raw

PATCH 2/5 qemuboot.bbclass: use IMGDEPLOYDIR

In reply to Robert Yang

So that "bitbake <image> -ccleansstate" can remove qemuboot.conf

Signed-off-by Robert Yang <[email protected]>

---
 meta/classes/qemuboot.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

meta/classes/qemuboot.bbclass +3 -2

index 39df3ad..eb0ee57 100644
@@ -43,6 +43,7 @@ QB_OPT_APPEND ?= "-show-cursor"
 
 # Create qemuboot.conf
 ROOTFS_POSTPROCESS_COMMAND += "write_qemuboot_conf; "
+IMGDEPLOYDIR ?= "${WORKDIR}/deploy-${PN}-image-complete"
 
 def qemuboot_vars(d):
     build_vars = ['MACHINE', 'TUNE_ARCH', 'DEPLOY_DIR_IMAGE',

@@ -55,8 +56,8 @@ write_qemuboot_conf[vardeps] += "${@' '.join(qemuboot_vars(d))}"
 python write_qemuboot_conf() {
     import configparser
 
-    qemuboot = "%s/%s.qemuboot.conf" % (d.getVar('DEPLOY_DIR_IMAGE', True), d.getVar('IMAGE_NAME', True))
-    qemuboot_link = "%s/%s.qemuboot.conf" % (d.getVar('DEPLOY_DIR_IMAGE', True), d.getVar('IMAGE_LINK_NAME', True))
+    qemuboot = "%s/%s.qemuboot.conf" % (d.getVar('IMGDEPLOYDIR', True), d.getVar('IMAGE_NAME', True))
+    qemuboot_link = "%s/%s.qemuboot.conf" % (d.getVar('IMGDEPLOYDIR', True), d.getVar('IMAGE_LINK_NAME', True))
     cf = configparser.ConfigParser()
     cf.add_section('config_bsp')
     for k in qemuboot_vars(d):

Signature · 1 line

2.9.0

Robert Yang

author

2016-11-29 07:35 UTC #3 lore raw

PATCH 3/5 runqemu-export-rootfs: fix inconsistent var names

In reply to Robert Yang

Fixed:

$ runqemu nfs qemux86-64
[snip]
On your target please remember to add the following options for NFS
nfsroot=IP_ADDRESS:/path/to/nfsroot,nfsvers=3,port=,mountprog=,nfsprog=,udp,mountport=
[snip]

Note that the values are null, this is because their var names are inconsistent.

[YOCTO #10519]

Signed-off-by Robert Yang <[email protected]>

---
 scripts/runqemu-export-rootfs | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)

scripts/runqemu-export-rootfs +9 ~~-12~~

index 3dee131..0dd3eba 100755
@@ -78,23 +78,17 @@ if [ ! -d "$PSEUDO_LOCALSTATEDIR" ]; then
 fi
 
 # rpc.mountd RPC port
-NFS_MOUNTPROG=$[ 21111 + $NFS_INSTANCE ]
+MOUNTD_RPCPORT=$[ 21111 + $NFS_INSTANCE ]
 # rpc.nfsd RPC port
-NFS_NFSPROG=$[ 11111 + $NFS_INSTANCE ]
-# NFS port number
-NFS_PORT=$[ 3049 + 2 * $NFS_INSTANCE ]
+NFSD_RPCPORT=$[ 11111 + $NFS_INSTANCE ]
+# NFS server port number
+NFSD_PORT=$[ 3049 + 2 * $NFS_INSTANCE ]
 # mountd port number
-MOUNT_PORT=$[ 3048 + 2 * $NFS_INSTANCE ]
+MOUNTD_PORT=$[ 3048 + 2 * $NFS_INSTANCE ]
 
 ## For debugging you would additionally add
 ## --debug all
-UNFSD_OPTS="-p -N -i $NFSPID -e $EXPORTS -x $NFS_NFSPROG -n $NFS_PORT -y $NFS_MOUNTPROG -m $MOUNT_PORT"
-
-# Setup the exports file
-if [ "$1" = "start" ]; then
-	echo "Creating exports file..."
-	echo "$NFS_EXPORT_DIR (rw,async,no_root_squash,no_all_squash,insecure)" > $EXPORTS
-fi
+UNFSD_OPTS="-p -N -i $NFSPID -e $EXPORTS -x $NFSD_RPCPORT -n $NFSD_PORT -y $MOUNTD_RPCPORT -m $MOUNTD_PORT"
 
 # See how we were called.
 case "$1" in

@@ -114,6 +108,9 @@ case "$1" in
 		exit 1
 	fi
 
+	echo "Creating exports file..."
+	echo "$NFS_EXPORT_DIR (rw,async,no_root_squash,no_all_squash,insecure)" > $EXPORTS
+
 	echo "Starting User Mode nfsd"
 	echo "  $PSEUDO $PSEUDO_OPTS $OECORE_NATIVE_SYSROOT/usr/bin/unfsd $UNFSD_OPTS"
 	$PSEUDO $PSEUDO_OPTS $OECORE_NATIVE_SYSROOT/usr/bin/unfsd $UNFSD_OPTS

Signature · 1 line

2.9.0

Robert Yang

author

2016-11-29 07:36 UTC #4 lore raw

PATCH 4/5 runqemu: support mutiple qemus running when nfs

In reply to Robert Yang

Fixed:
* In build1:

  $ runqemu nfs qemux86-64
  In build2:
  $ runqemu nfs qemux86-64

It would fail before since the port numerbs and conf files are conflicted, now make runqemu-export-rootfs work together with runqemu to fix the problem.

* And we don't need export PSEUDO_LOCALSTATEDIR in runqemu, the runqemu-export-rootfs can handle it well based on NFS_EXPORT_DIR.

* Remove "async" option from unfsd to fix warning in syslog:
Warning: unknown exports option `async' ignored

* Fixed typos

Both slirp and tap can work.

Signed-off-by Robert Yang <[email protected]>

---
 scripts/runqemu               | 55 ++++++++++++++++++++++++++++---------------
 scripts/runqemu-export-rootfs | 10 ++++----
 2 files changed, 41 insertions(+), 24 deletions(-)

scripts/runqemu +36 ~~-19~~

index c737eb2..a10270c 100755
@@ -690,17 +690,33 @@ class BaseConfig(object):
             else:
                 self.nfs_server = '192.168.7.1'
 
-        nfs_instance = int(self.nfs_instance)
-
-        mountd_rpcport = 21111 + nfs_instance
-        nfsd_rpcport = 11111 + nfs_instance
-        nfsd_port = 3049 + 2 * nfs_instance
-        mountd_port = 3048 + 2 * nfs_instance
-        unfs_opts="nfsvers=3,port=%s,mountprog=%s,nfsprog=%s,udp,mountport=%s" % (nfsd_port, mountd_rpcport, nfsd_rpcport, mountd_port)
-        self.unfs_opts = unfs_opts
+        # Figure out a new nfs_instance to allow multiple qemus running.
+        cmd = "ps aux"
+        ps = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
+        pattern = '/bin/unfsd .* -i .*\.pid -e .*/exports([0-9]+) '
+        all_instances = re.findall(pattern, ps, re.M)
+        if all_instances:
+            all_instances.sort(key=int)
+            self.nfs_instance = int(all_instances.pop()) + 1
+
+        mountd_rpcport = 21111 + self.nfs_instance
+        nfsd_rpcport = 11111 + self.nfs_instance
+        nfsd_port = 3049 + 2 * self.nfs_instance
+        mountd_port = 3048 + 2 * self.nfs_instance
+
+        # Export vars for runqemu-export-rootfs
+        export_dict = {
+            'NFS_INSTANCE': self.nfs_instance,
+            'MOUNTD_RPCPORT': mountd_rpcport,
+            'NFSD_RPCPORT': nfsd_rpcport,
+            'NFSD_PORT': nfsd_port,
+            'MOUNTD_PORT': mountd_port,
+        }
+        for k, v in export_dict.items():
+            # Use '%s' since they are integers
+            os.putenv(k, '%s' % v)
 
-        p = '%s/.runqemu-sdk/pseudo' % os.getenv('HOME')
-        os.putenv('PSEUDO_LOCALSTATEDIR', p)
+        self.unfs_opts="nfsvers=3,port=%s,mountprog=%s,nfsprog=%s,udp,mountport=%s" % (nfsd_port, mountd_rpcport, nfsd_rpcport, mountd_port)
 
         # Extract .tar.bz2 or .tar.bz if no self.nfs_dir
         if not self.nfs_dir:

@@ -728,7 +744,7 @@ class BaseConfig(object):
                 self.nfs_dir = dest
 
         # Start the userspace NFS server
-        cmd = 'runqemu-export-rootfs restart %s' % self.nfs_dir
+        cmd = 'runqemu-export-rootfs start %s' % self.nfs_dir
         logger.info('Running %s...' % cmd)
         if subprocess.call(cmd, shell=True) != 0:
             raise Exception('Failed to run %s' % cmd)

@@ -737,6 +753,8 @@ class BaseConfig(object):
 
 
     def setup_slirp(self):
+        """Setup user networking"""
+
         if self.fstype == 'nfs':
             self.setup_nfs()
         self.kernel_cmdline_script += ' ip=dhcp'

@@ -804,14 +822,13 @@ class BaseConfig(object):
             logger.error("Failed to setup tap device. Run runqemu-gen-tapdevs to manually create.")
             return 1
         self.tap = tap
-        n0 = tap[3:]
-        n1 = int(n0) * 2 + 1
-        n2 = n1 + 1
-        self.nfs_instance = n0
+        tapnum = int(tap[3:])
+        gateway = tapnum * 2 + 1
+        client = gateway + 1
         if self.fstype == 'nfs':
             self.setup_nfs()
-        self.kernel_cmdline_script += " ip=192.168.7.%s::192.168.7.%s:255.255.255.0" % (n2, n1)
-        mac = "52:54:00:12:34:%02x" % n2
+        self.kernel_cmdline_script += " ip=192.168.7.%s::192.168.7.%s:255.255.255.0" % (client, gateway)
+        mac = "52:54:00:12:34:%02x" % client
         qb_tap_opt = self.get('QB_TAP_OPT')
         if qb_tap_opt:
             qemu_tap_opt = qb_tap_opt.replace('@TAP@', tap).replace('@MAC@', mac)

@@ -854,11 +871,11 @@ class BaseConfig(object):
                         vm_drive = '-drive if=none,id=hd,file=%s,format=%s -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd' \
                                        % (self.rootfs, rootfs_format)
                     elif subprocess.call(cmd2, shell=True) == 0:
-                        logger.info('Using scsi drive')
+                        logger.info('Using ide drive')
                         vm_drive = "%s,format=%s" % (self.rootfs, rootfs_format)
                     else:
                         logger.warn("Can't detect drive type %s" % self.rootfs)
-                        logger.warn('Tring to use virtio block drive')
+                        logger.warn('Trying to use virtio block drive')
                         vm_drive = '-drive if=virtio,file=%s,format=%s' % (self.rootfs, rootfs_format)
                 self.rootfs_options = '%s -no-reboot' % vm_drive
             self.kernel_cmdline = 'root=%s rw highres=off' % (self.get('QB_KERNEL_ROOT'))

scripts/runqemu-export-rootfs +5 -5

index 0dd3eba..7ebc071 100755
@@ -78,13 +78,13 @@ if [ ! -d "$PSEUDO_LOCALSTATEDIR" ]; then
 fi
 
 # rpc.mountd RPC port
-MOUNTD_RPCPORT=$[ 21111 + $NFS_INSTANCE ]
+MOUNTD_RPCPORT=${MOUNTD_RPCPORT:=$[ 21111 + $NFS_INSTANCE ]}
 # rpc.nfsd RPC port
-NFSD_RPCPORT=$[ 11111 + $NFS_INSTANCE ]
+NFSD_RPCPORT=${NFSD_RPCPORT:=$[ 11111 + $NFS_INSTANCE ]}
 # NFS server port number
-NFSD_PORT=$[ 3049 + 2 * $NFS_INSTANCE ]
+NFSD_PORT=${NFSD_PORT:=$[ 3049 + 2 * $NFS_INSTANCE ]}
 # mountd port number
-MOUNTD_PORT=$[ 3048 + 2 * $NFS_INSTANCE ]
+MOUNTD_PORT=${MOUNTD_PORT:=$[ 3048 + 2 * $NFS_INSTANCE ]}
 
 ## For debugging you would additionally add
 ## --debug all

@@ -109,7 +109,7 @@ case "$1" in
 	fi
 
 	echo "Creating exports file..."
-	echo "$NFS_EXPORT_DIR (rw,async,no_root_squash,no_all_squash,insecure)" > $EXPORTS
+	echo "$NFS_EXPORT_DIR (rw,no_root_squash,no_all_squash,insecure)" > $EXPORTS
 
 	echo "Starting User Mode nfsd"
 	echo "  $PSEUDO $PSEUDO_OPTS $OECORE_NATIVE_SYSROOT/usr/bin/unfsd $UNFSD_OPTS"

Signature · 1 line

2.9.0

Robert Yang

author

2016-11-29 07:35 UTC #5 lore raw

PATCH 5/5 runqemu: set default device and hostfwd for slirp

In reply to Robert Yang

There is no network rather than lo whitout set device, so set it by default. And set hostfwd: 2222 -> 22, 2323 -> 23

[YOCTO #7887]

Signed-off-by Robert Yang <[email protected]>

---
 meta/conf/machine/include/qemuboot-x86.inc |  1 -
 meta/conf/machine/qemuarm64.conf           |  1 -
 scripts/runqemu                            | 34 ++++++++++++++++++++++++++++--
 3 files changed, 32 insertions(+), 4 deletions(-)

meta/conf/machine/include/qemuboot-x86.inc +0 -1

index 0870294..06ac983 100644
@@ -13,4 +13,3 @@ QB_AUDIO_OPT = "-soundhw ac97,es1370"
 QB_KERNEL_CMDLINE_APPEND = "vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1 uvesafb.task_timeout=-1"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci"
-QB_SLIRP_OPT = "-net nic,model=e1000 -net user,hostfwd=tcp::2222-:22"

meta/conf/machine/qemuarm64.conf +0 -1

index df2010c..7a5570c 100644
@@ -18,7 +18,6 @@ QB_KERNEL_CMDLINE_APPEND = "console=ttyAMA0,38400"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-show-cursor -device virtio-rng-pci -monitor null"
 QB_TAP_OPT = "-netdev tap,id=net0,ifname=@TAP@,script=no,downscript=no -device virtio-net-device,netdev=net0,mac=@MAC@"
-QB_SLIRP_OPT = "-netdev user,id=net0 -device virtio-net-device,netdev=net0"
 QB_ROOTFS_OPT = "-drive id=disk0,file=@ROOTFS@,if=none,format=raw -device virtio-blk-device,drive=disk0"
 QB_SERIAL_OPT = "-device virtio-serial-device -chardev null,id=virtcon -device virtconsole,chardev=virtcon"
 QB_TCPSERIAL_OPT = " -device virtio-serial-device -chardev socket,id=virtcon,port=@PORT@,host=127.0.0.1 -device virtconsole,chardev=virtcon"

scripts/runqemu +32 -2

index a10270c..7123b8f 100755
@@ -147,6 +147,19 @@ def get_first_file(cmds):
                     return f
     return ''
 
+def check_free_port(host, port):
+    """ Check whether the port is free or not """
+    import socket
+    from contextlib import closing
+
+    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
+        if sock.connect_ex((host, port)) == 0:
+            # Port is open, so not free
+            return False
+        else:
+            # Port is not open, so free
+            return True
+
 class BaseConfig(object):
     def __init__(self):
         # Vars can be merged with .qemuboot.conf, use a dict to manage them.

@@ -751,14 +764,31 @@ class BaseConfig(object):
 
         self.nfs_running = True
 
-
     def setup_slirp(self):
         """Setup user networking"""
 
         if self.fstype == 'nfs':
             self.setup_nfs()
         self.kernel_cmdline_script += ' ip=dhcp'
-        self.set('NETWORK_CMD', self.get('QB_SLIRP_OPT'))
+        # Port mapping
+        hostfwd = ",hostfwd=tcp::2222-:22,hostfwd=tcp::2323-:23"
+        qb_slirp_opt_default = "-netdev user,id=net0%s -device virtio-net-pci,netdev=net0" % hostfwd
+        qb_slirp_opt = self.get('QB_SLIRP_OPT') or qb_slirp_opt_default
+        # Figure out the port
+        ports = re.findall('hostfwd=[^-]*:([0-9]+)-[^,-]*', qb_slirp_opt)
+        ports = [int(i) for i in ports]
+        # Find a free port to avoid conflicts
+        for p in ports[:]:
+            p_new = p
+            while not check_free_port('127.0.0.1', p_new):
+                p_new += 1
+                while p_new in ports:
+                        p_new += 1
+            if p != p_new:
+                ports.append(p_new)
+                qb_slirp_opt = re.sub(':%s-' % p, ':%s-' % p_new, qb_slirp_opt)
+                logger.info("Port forward changed: %s -> %s" % (p, p_new))
+        self.set('NETWORK_CMD', qb_slirp_opt)
 
     def setup_tap(self):
         """Setup tap"""

Signature · 1 line

2.9.0

Nathan Rossi

2016-11-29 09:38 UTC #6 lore raw

Re: [PATCH 5/5] runqemu: set default device and hostfwd for slirp

Quoted 66 lines · Robert Yang

There is no network rather than lo whitout set device, so set it by default. And set hostfwd: 2222 -> 22, 2323 -> 23

[YOCTO #7887]

Signed-off-by Robert Yang <[email protected]>

---
 meta/conf/machine/include/qemuboot-x86.inc |  1 -
 meta/conf/machine/qemuarm64.conf           |  1 -
 scripts/runqemu                            | 34 ++++++++++++++++++++++++++++--
 3 files changed, 32 insertions(+), 4 deletions(-)

meta/conf/machine/include/qemuboot-x86.inc +0 -1

index 0870294..06ac983 100644
@@ -13,4 +13,3 @@ QB_AUDIO_OPT = "-soundhw ac97,es1370"
 QB_KERNEL_CMDLINE_APPEND = "vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1 uvesafb.task_timeout=-1"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci"
-QB_SLIRP_OPT = "-net nic,model=e1000 -net user,hostfwd=tcp::2222-:22"

meta/conf/machine/qemuarm64.conf +0 -1

index df2010c..7a5570c 100644
@@ -18,7 +18,6 @@ QB_KERNEL_CMDLINE_APPEND = "console=ttyAMA0,38400"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-show-cursor -device virtio-rng-pci -monitor null"
 QB_TAP_OPT = "-netdev tap,id=net0,ifname=@TAP@,script=no,downscript=no -device virtio-net-device,netdev=net0,mac=@MAC@"
-QB_SLIRP_OPT = "-netdev user,id=net0 -device virtio-net-device,netdev=net0"
 QB_ROOTFS_OPT = "-drive id=disk0,file=@ROOTFS@,if=none,format=raw -device virtio-blk-device,drive=disk0"
 QB_SERIAL_OPT = "-device virtio-serial-device -chardev null,id=virtcon -device virtconsole,chardev=virtcon"
 QB_TCPSERIAL_OPT = " -device virtio-serial-device -chardev socket,id=virtcon,port=@PORT@,host=127.0.0.1 -device virtconsole,chardev=virtcon"

scripts/runqemu +13 -1

index a10270c..7123b8f 100755
@@ -147,6 +147,19 @@ def get_first_file(cmds):
                     return f
     return ''
 
+def check_free_port(host, port):
+    """ Check whether the port is free or not """
+    import socket
+    from contextlib import closing
+
+    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
+        if sock.connect_ex((host, port)) == 0:
+            # Port is open, so not free
+            return False
+        else:
+            # Port is not open, so free
+            return True
+
 class BaseConfig(object):
     def __init__(self):
         # Vars can be merged with .qemuboot.conf, use a dict to manage them.

@@ -751,14 +764,31 @@ class BaseConfig(object):
 
         self.nfs_running = True
 
-
     def setup_slirp(self):
         """Setup user networking"""
 
         if self.fstype == 'nfs':

scripts/runqemu class BaseConfig(object): · line 779

Quoted 10 lines

scripts/runqemu +7 -1

             self.setup_nfs()
         self.kernel_cmdline_script += ' ip=dhcp'
-        self.set('NETWORK_CMD', self.get('QB_SLIRP_OPT'))
+        # Port mapping
+        hostfwd = ",hostfwd=tcp::2222-:22,hostfwd=tcp::2323-:23"
+        qb_slirp_opt_default = "-netdev user,id=net0%s -device virtio-net-pci,netdev=net0" % hostfwd
+        qb_slirp_opt = self.get('QB_SLIRP_OPT') or qb_slirp_opt_default
+        # Figure out the port
+        ports = re.findall('hostfwd=[^-]*:([0-9]+)-[^,-]*', qb_slirp_opt)
+        ports = [int(i) for i in ports]

Should runqemu not just always supply "-netdev user,id=net0..." itself for slirp. And qemuboot should supply only the devices, e.g. the "-dev virtio-net-pci,netdev=net0" or "-net nic,netdev=net0". This could even allow for consolidation of the "QB_SLIRP_OPT" and "QB_TAP_OPT" into "QB_NETWORK".

This way runqemu would not need to do any searching/substitution and could just build the user netdev arg itself. This would allow for easy extension of custom forwards (e.g gdbserver, etc), and also even setting up the in qemu user/slirp 'tftp=<dir>' server.

scripts/runqemu class BaseConfig(object): · line 783

Quoted 4 lines

scripts/runqemu +4 -0

+        # Find a free port to avoid conflicts
+        for p in ports[:]:
+            p_new = p
+            while not check_free_port('127.0.0.1', p_new):

Should this be 'localhost', since 127.0.0.1 will only check for ipv4 listeners.

Regards,
Nathan

Quoted 19 lines

scripts/runqemu +8 -1

+                p_new += 1
+                while p_new in ports:
+                        p_new += 1
+            if p != p_new:
+                ports.append(p_new)
+                qb_slirp_opt = re.sub(':%s-' % p, ':%s-' % p_new, qb_slirp_opt)
+                logger.info("Port forward changed: %s -> %s" % (p, p_new))
+        self.set('NETWORK_CMD', qb_slirp_opt)
 
     def setup_tap(self):
         """Setup tap"""
--

2.9.0

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Nathan Rossi

2016-11-29 09:54 UTC #7 lore raw

Re: [PATCH 5/5] runqemu: set default device and hostfwd for slirp

Quoted 75 lines · Nathan Rossi

Quoted 74 lines · Robert Yang

There is no network rather than lo whitout set device, so set it by default. And set hostfwd: 2222 -> 22, 2323 -> 23

[YOCTO #7887]

Signed-off-by Robert Yang <[email protected]>

---
 meta/conf/machine/include/qemuboot-x86.inc |  1 -
 meta/conf/machine/qemuarm64.conf           |  1 -
 scripts/runqemu                            | 34 ++++++++++++++++++++++++++++--
 3 files changed, 32 insertions(+), 4 deletions(-)

meta/conf/machine/include/qemuboot-x86.inc +0 -1

index 0870294..06ac983 100644
@@ -13,4 +13,3 @@ QB_AUDIO_OPT = "-soundhw ac97,es1370"
 QB_KERNEL_CMDLINE_APPEND = "vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1 uvesafb.task_timeout=-1"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci"
-QB_SLIRP_OPT = "-net nic,model=e1000 -net user,hostfwd=tcp::2222-:22"

meta/conf/machine/qemuarm64.conf +0 -1

index df2010c..7a5570c 100644
@@ -18,7 +18,6 @@ QB_KERNEL_CMDLINE_APPEND = "console=ttyAMA0,38400"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-show-cursor -device virtio-rng-pci -monitor null"
 QB_TAP_OPT = "-netdev tap,id=net0,ifname=@TAP@,script=no,downscript=no -device virtio-net-device,netdev=net0,mac=@MAC@"
-QB_SLIRP_OPT = "-netdev user,id=net0 -device virtio-net-device,netdev=net0"
 QB_ROOTFS_OPT = "-drive id=disk0,file=@ROOTFS@,if=none,format=raw -device virtio-blk-device,drive=disk0"
 QB_SERIAL_OPT = "-device virtio-serial-device -chardev null,id=virtcon -device virtconsole,chardev=virtcon"
 QB_TCPSERIAL_OPT = " -device virtio-serial-device -chardev socket,id=virtcon,port=@PORT@,host=127.0.0.1 -device virtconsole,chardev=virtcon"

scripts/runqemu +18 -2

index a10270c..7123b8f 100755
@@ -147,6 +147,19 @@ def get_first_file(cmds):
                     return f
     return ''
 
+def check_free_port(host, port):
+    """ Check whether the port is free or not """
+    import socket
+    from contextlib import closing
+
+    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
+        if sock.connect_ex((host, port)) == 0:
+            # Port is open, so not free
+            return False
+        else:
+            # Port is not open, so free
+            return True
+
 class BaseConfig(object):
     def __init__(self):
         # Vars can be merged with .qemuboot.conf, use a dict to manage them.

@@ -751,14 +764,31 @@ class BaseConfig(object):
 
         self.nfs_running = True
 
-
     def setup_slirp(self):
         """Setup user networking"""
 
         if self.fstype == 'nfs':
             self.setup_nfs()
         self.kernel_cmdline_script += ' ip=dhcp'
-        self.set('NETWORK_CMD', self.get('QB_SLIRP_OPT'))
+        # Port mapping
+        hostfwd = ",hostfwd=tcp::2222-:22,hostfwd=tcp::2323-:23"
+        qb_slirp_opt_default = "-netdev user,id=net0%s -device virtio-net-pci,netdev=net0" % hostfwd
+        qb_slirp_opt = self.get('QB_SLIRP_OPT') or qb_slirp_opt_default
+        # Figure out the port

Quoted 8 lines

scripts/runqemu class BaseConfig(object): · line 779
Quoted 2 lines
scripts/runqemu +2 -0
+        ports = re.findall('hostfwd=[^-]*:([0-9]+)-[^,-]*', qb_slirp_opt)
+        ports = [int(i) for i in ports]
Should runqemu not just always supply "-netdev user,id=net0..." itself for slirp. And qemuboot should supply only the devices, e.g. the "-dev virtio-net-pci,netdev=net0" or "-net nic,netdev=net0". This could even allow for consolidation of the "QB_SLIRP_OPT" and "QB_TAP_OPT" into "QB_NETWORK".

Just saw this patch was an updated version of
https://patchwork.openembedded.org/patch/134399/ which was doing similar to what I suggested above. (I read this patch before the other one linked)

Maybe QB_SLIRP_OPT and QB_TAP_OPT should provide overrides for the -netdev user/tap, and if a BSP/etc needs to overwrite them it can. But the "-net"/"-device" is provided as a separate QB_NETWORK variable. This would allow for BSPs to take advantage of the default netdev user/tap args that runqemu provides without needing to specify any of the forwards/etc. And would also allow for backwards compatibility with current BSP QB_* setups.

Regards,
Nathan

Quoted 35 lines

This way runqemu would not need to do any searching/substitution and could just build the user netdev arg itself. This would allow for easy extension of custom forwards (e.g gdbserver, etc), and also even setting up the in qemu user/slirp 'tftp=<dir>' server.
scripts/runqemu class BaseConfig(object): · line 783
Quoted 4 lines
scripts/runqemu +4 -0
+        # Find a free port to avoid conflicts
+        for p in ports[:]:
+            p_new = p
+            while not check_free_port('127.0.0.1', p_new):
Should this be 'localhost', since 127.0.0.1 will only check for ipv4 listeners.
Regards,
Nathan
Quoted 19 lines
scripts/runqemu +8 -1
+                p_new += 1
+                while p_new in ports:
+                        p_new += 1
+            if p != p_new:
+                ports.append(p_new)
+                qb_slirp_opt = re.sub(':%s-' % p, ':%s-' % p_new, qb_slirp_opt)
+                logger.info("Port forward changed: %s -> %s" % (p, p_new))
+        self.set('NETWORK_CMD', qb_slirp_opt)
 
     def setup_tap(self):
         """Setup tap"""
--
2.9.0
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Robert Yang

author

2016-11-29 10:12 UTC #8 lore raw

Re: [PATCH 5/5] runqemu: set default device and hostfwd for slirp

Quoted 90 lines · Nathan Rossi

Quoted 75 lines · Nathan Rossi

Quoted 74 lines · Robert Yang

There is no network rather than lo whitout set device, so set it by default. And set hostfwd: 2222 -> 22, 2323 -> 23

[YOCTO #7887]

Signed-off-by Robert Yang <[email protected]>

---
 meta/conf/machine/include/qemuboot-x86.inc |  1 -
 meta/conf/machine/qemuarm64.conf           |  1 -
 scripts/runqemu                            | 34 ++++++++++++++++++++++++++++--
 3 files changed, 32 insertions(+), 4 deletions(-)

meta/conf/machine/include/qemuboot-x86.inc +0 -1

index 0870294..06ac983 100644
@@ -13,4 +13,3 @@ QB_AUDIO_OPT = "-soundhw ac97,es1370"
 QB_KERNEL_CMDLINE_APPEND = "vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1 uvesafb.task_timeout=-1"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci"
-QB_SLIRP_OPT = "-net nic,model=e1000 -net user,hostfwd=tcp::2222-:22"

meta/conf/machine/qemuarm64.conf +0 -1

index df2010c..7a5570c 100644
@@ -18,7 +18,6 @@ QB_KERNEL_CMDLINE_APPEND = "console=ttyAMA0,38400"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-show-cursor -device virtio-rng-pci -monitor null"
 QB_TAP_OPT = "-netdev tap,id=net0,ifname=@TAP@,script=no,downscript=no -device virtio-net-device,netdev=net0,mac=@MAC@"
-QB_SLIRP_OPT = "-netdev user,id=net0 -device virtio-net-device,netdev=net0"
 QB_ROOTFS_OPT = "-drive id=disk0,file=@ROOTFS@,if=none,format=raw -device virtio-blk-device,drive=disk0"
 QB_SERIAL_OPT = "-device virtio-serial-device -chardev null,id=virtcon -device virtconsole,chardev=virtcon"
 QB_TCPSERIAL_OPT = " -device virtio-serial-device -chardev socket,id=virtcon,port=@PORT@,host=127.0.0.1 -device virtconsole,chardev=virtcon"

scripts/runqemu +18 -2

index a10270c..7123b8f 100755
@@ -147,6 +147,19 @@ def get_first_file(cmds):
                     return f
     return ''
 
+def check_free_port(host, port):
+    """ Check whether the port is free or not """
+    import socket
+    from contextlib import closing
+
+    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
+        if sock.connect_ex((host, port)) == 0:
+            # Port is open, so not free
+            return False
+        else:
+            # Port is not open, so free
+            return True
+
 class BaseConfig(object):
     def __init__(self):
         # Vars can be merged with .qemuboot.conf, use a dict to manage them.

@@ -751,14 +764,31 @@ class BaseConfig(object):
 
         self.nfs_running = True
 
-
     def setup_slirp(self):
         """Setup user networking"""
 
         if self.fstype == 'nfs':
             self.setup_nfs()
         self.kernel_cmdline_script += ' ip=dhcp'
-        self.set('NETWORK_CMD', self.get('QB_SLIRP_OPT'))
+        # Port mapping
+        hostfwd = ",hostfwd=tcp::2222-:22,hostfwd=tcp::2323-:23"
+        qb_slirp_opt_default = "-netdev user,id=net0%s -device virtio-net-pci,netdev=net0" % hostfwd
+        qb_slirp_opt = self.get('QB_SLIRP_OPT') or qb_slirp_opt_default
+        # Figure out the port

Quoted 8 lines

scripts/runqemu class BaseConfig(object): · line 779
Quoted 2 lines
scripts/runqemu +2 -0
+        ports = re.findall('hostfwd=[^-]*:([0-9]+)-[^,-]*', qb_slirp_opt)
+        ports = [int(i) for i in ports]
Should runqemu not just always supply "-netdev user,id=net0..." itself for slirp. And qemuboot should supply only the devices, e.g. the "-dev virtio-net-pci,netdev=net0" or "-net nic,netdev=net0". This could even allow for consolidation of the "QB_SLIRP_OPT" and "QB_TAP_OPT" into "QB_NETWORK".

Just saw this patch was an updated version of
https://patchwork.openembedded.org/patch/134399/ which was doing similar to what I suggested above. (I read this patch before the other one linked)

Quoted 7 lines

Maybe QB_SLIRP_OPT and QB_TAP_OPT should provide overrides for the -netdev user/tap, and if a BSP/etc needs to overwrite them it can. But the "-net"/"-device" is provided as a separate QB_NETWORK variable. This would allow for BSPs to take advantage of the default netdev user/tap args that runqemu provides without needing to specify any of the forwards/etc. And would also allow for backwards compatibility with current BSP QB_* setups.

Hi Nathan,

I have to go now, will reply later after more thinking.

// Robert

Quoted 40 lines

Regards,
Nathan
Quoted 35 lines
This way runqemu would not need to do any searching/substitution and could just build the user netdev arg itself. This would allow for easy extension of custom forwards (e.g gdbserver, etc), and also even setting up the in qemu user/slirp 'tftp=<dir>' server.
scripts/runqemu class BaseConfig(object): · line 783
Quoted 4 lines
scripts/runqemu +4 -0
+        # Find a free port to avoid conflicts
+        for p in ports[:]:
+            p_new = p
+            while not check_free_port('127.0.0.1', p_new):
Should this be 'localhost', since 127.0.0.1 will only check for ipv4 listeners.
Regards,
Nathan
Quoted 19 lines
scripts/runqemu +8 -1
+                p_new += 1
+                while p_new in ports:
+                        p_new += 1
+            if p != p_new:
+                ports.append(p_new)
+                qb_slirp_opt = re.sub(':%s-' % p, ':%s-' % p_new, qb_slirp_opt)
+                logger.info("Port forward changed: %s -> %s" % (p, p_new))
+        self.set('NETWORK_CMD', qb_slirp_opt)
 
     def setup_tap(self):
         """Setup tap"""
--
2.9.0
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Robert Yang

author

2016-11-29 09:59 UTC #9 lore raw

Re: [PATCH 5/5] runqemu: set default device and hostfwd for slirp

In reply to Nathan Rossi

Quoted 66 lines · Nathan Rossi

Quoted 65 lines · Robert Yang

There is no network rather than lo whitout set device, so set it by default. And set hostfwd: 2222 -> 22, 2323 -> 23

[YOCTO #7887]

Signed-off-by Robert Yang <[email protected]>

---
 meta/conf/machine/include/qemuboot-x86.inc |  1 -
 meta/conf/machine/qemuarm64.conf           |  1 -
 scripts/runqemu                            | 34 ++++++++++++++++++++++++++++--
 3 files changed, 32 insertions(+), 4 deletions(-)

meta/conf/machine/include/qemuboot-x86.inc +0 -1

index 0870294..06ac983 100644
@@ -13,4 +13,3 @@ QB_AUDIO_OPT = "-soundhw ac97,es1370"
 QB_KERNEL_CMDLINE_APPEND = "vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1 uvesafb.task_timeout=-1"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci"
-QB_SLIRP_OPT = "-net nic,model=e1000 -net user,hostfwd=tcp::2222-:22"

meta/conf/machine/qemuarm64.conf +0 -1

index df2010c..7a5570c 100644
@@ -18,7 +18,6 @@ QB_KERNEL_CMDLINE_APPEND = "console=ttyAMA0,38400"
 # Add the 'virtio-rng-pci' device otherwise the guest may run out of entropy
 QB_OPT_APPEND = "-show-cursor -device virtio-rng-pci -monitor null"
 QB_TAP_OPT = "-netdev tap,id=net0,ifname=@TAP@,script=no,downscript=no -device virtio-net-device,netdev=net0,mac=@MAC@"
-QB_SLIRP_OPT = "-netdev user,id=net0 -device virtio-net-device,netdev=net0"
 QB_ROOTFS_OPT = "-drive id=disk0,file=@ROOTFS@,if=none,format=raw -device virtio-blk-device,drive=disk0"
 QB_SERIAL_OPT = "-device virtio-serial-device -chardev null,id=virtcon -device virtconsole,chardev=virtcon"
 QB_TCPSERIAL_OPT = " -device virtio-serial-device -chardev socket,id=virtcon,port=@PORT@,host=127.0.0.1 -device virtconsole,chardev=virtcon"

scripts/runqemu +13 -1

index a10270c..7123b8f 100755
@@ -147,6 +147,19 @@ def get_first_file(cmds):
                     return f
     return ''
 
+def check_free_port(host, port):
+    """ Check whether the port is free or not """
+    import socket
+    from contextlib import closing
+
+    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
+        if sock.connect_ex((host, port)) == 0:
+            # Port is open, so not free
+            return False
+        else:
+            # Port is not open, so free
+            return True
+
 class BaseConfig(object):
     def __init__(self):
         # Vars can be merged with .qemuboot.conf, use a dict to manage them.

@@ -751,14 +764,31 @@ class BaseConfig(object):
 
         self.nfs_running = True
 
-
     def setup_slirp(self):
         """Setup user networking"""

Quoted 15 lines

scripts/runqemu class BaseConfig(object): · line 779
Quoted 11 lines
scripts/runqemu +7 -1
         if self.fstype == 'nfs':
             self.setup_nfs()
         self.kernel_cmdline_script += ' ip=dhcp'
-        self.set('NETWORK_CMD', self.get('QB_SLIRP_OPT'))
+        # Port mapping
+        hostfwd = ",hostfwd=tcp::2222-:22,hostfwd=tcp::2323-:23"
+        qb_slirp_opt_default = "-netdev user,id=net0%s -device virtio-net-pci,netdev=net0" % hostfwd
+        qb_slirp_opt = self.get('QB_SLIRP_OPT') or qb_slirp_opt_default
+        # Figure out the port
+        ports = re.findall('hostfwd=[^-]*:([0-9]+)-[^,-]*', qb_slirp_opt)
+        ports = [int(i) for i in ports]
Should runqemu not just always supply "-netdev user,id=net0..." itself for slirp. And qemuboot should supply only the devices, e.g. the "-dev virtio-net-pci,netdev=net0" or "-net nic,netdev=net0". This could even

The problem is if runqemu only uses the part "-dev virtio-net-pci,netdev=net0", and QB_SLIRP_OPT doesn't replace it, then the bsp,conf can't custom freely, for example, its bsp may not support "-dev virtio-net-pci", then it would cause troubles. So if bsp.conf wants to redfine the options, manage the full options is better than part of them.

Quoted 4 lines

allow for consolidation of the "QB_SLIRP_OPT" and "QB_TAP_OPT" into "QB_NETWORK".
This way runqemu would not need to do any searching/substitution and

The searching/substitution is still needed, suppose you defined something
like: hostfwd=tcp::2222-:22 in bsp.conf, then they will go into qemuboot.conf, and it would fail when you run more than one qemu targets since the port is used by the first running qemu.

Quoted 10 lines

could just build the user netdev arg itself. This would allow for easy extension of custom forwards (e.g gdbserver, etc), and also even setting up the in qemu user/slirp 'tftp=<dir>' server.
scripts/runqemu class BaseConfig(object): · line 783
Quoted 4 lines
scripts/runqemu +4 -0
+        # Find a free port to avoid conflicts
+        for p in ports[:]:
+            p_new = p
+            while not check_free_port('127.0.0.1', p_new):
Should this be 'localhost', since 127.0.0.1 will only check for ipv4 listeners.

Sounds good, updated in the repo.

// Robert

Quoted 24 lines

Regards,
Nathan
Quoted 19 lines
scripts/runqemu +8 -1
+                p_new += 1
+                while p_new in ports:
+                        p_new += 1
+            if p != p_new:
+                ports.append(p_new)
+                qb_slirp_opt = re.sub(':%s-' % p, ':%s-' % p_new, qb_slirp_opt)
+                logger.info("Port forward changed: %s -> %s" % (p, p_new))
+        self.set('NETWORK_CMD', qb_slirp_opt)
 
     def setup_tap(self):
         """Setup tap"""
--
2.9.0
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core