Python 3 runqemu fixes

PATCH New Joshua Lock 1 participant 3 messages Sep 7, 2016

Share & tools

Joshua Lock

author

2016-09-07 22:04 UTC #0 lore raw

cover Python 3 runqemu fixes

Note: This patch is against master-next, as the Python 3 runqemu hasn't made it
to master yet.
This supercedes my earlier one patch series "Python3 runqemu: remove use of subprocess.run()" (both versions)

Two fixes for the Python3 runqemu:
* Use of subprocess.run() in recent runqemu changes (by me) elevates the Python 3 requirement from 3.4+ to 3.5+. Replace subprocess.run() with subprocess.check_output(), as that is available in Python 3.1+
* testimage calls runqemu from within a bitbake run, which prevents us from using bitbake -e to determine bitbake environment variables. The test code exports an OE_TMPDIR variable which was used by the shell runqemu to infer/guess paths into the sysroot. This patch does the equivalent.

Changes since "Python3 runqemu: remove use of subprocess.run()" v2:
* Fix typo in exception handler log message.
* Add extra fix to support testimage

Changes since "Python3 runqemu: remove use of subprocess.run()" v1:
* Fix printing of error message in exception handler

The following changes since commit 3df3462d81216d8b26a87d9915abd9de1b2c2faa:

bitbake: cooker: record events on cooker exit (2016-09-06 23:18:06 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib joshuagl/runqemu
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=joshuagl/runqemu

Joshua Lock (2):
runqemu: remove use of subprocess.run()
runqemu: fixes for when invoked during a bitbake run

 scripts/runqemu | 37 +++++++++++++++++++++++++++----------
 1 file changed, 27 insertions(+), 10 deletions(-)

Signature · 1 line

2.7.4

Joshua Lock

author

2016-09-07 22:04 UTC #1 lore raw

PATCH 1/2 runqemu: remove use of subprocess.run()

We aim to support Python 3.4+ whereas subprocess.run() was added in Python 3.5.
Replace subprocess.run() with subprocess.check_output().

Signed-off-by Joshua Lock <[email protected]>

---
 scripts/runqemu | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

scripts/runqemu +5 -4

index 0a56c60..cbc5cc6 100755
@@ -858,10 +858,11 @@ class BaseConfig(object):
             cmd = 'bitbake -e'
 
         logger.info('Running %s...' % cmd)
-        proc = subprocess.run(cmd, shell=True, stdout=subprocess.PIPE)
-        if proc.returncode != 0:
-            logger.warn("Couldn't run 'bitbake -e' to gather environment information")
-        self.bitbake_e = proc.stdout.decode('utf-8')
+        try:
+            self.bitbake_e = subprocess.check_output(cmd, shell=True).decode('utf-8')
+        except subprocess.CalledProcessError as err:
+            self.bitbake_e = ''
+            logger.warn("Couldn't run 'bitbake -e' to gather environment information:\n%s" % err.output.decode('utf-8'))
 
 def main():
     if len(sys.argv) == 1 or "help" in sys.argv:

Signature · 1 line

2.7.4

Joshua Lock

author

2016-09-07 22:04 UTC #2 lore raw

PATCH 2/2 runqemu: fixes for when invoked during a bitbake run

In reply to Joshua Lock

When runqemu is invoked from a running bitbake instance it will be unable to call bitbake -e due to the lock held by the calling bitbake instance.

Our test code sets an OE_TMPDIR environment variable from which we can infer/guess paths. Add code to do so when self.bitbake_e can't be set, much as the sh version of runqemu did.

[YOCTO #10240]

Signed-off-by Joshua Lock <[email protected]>

---
 scripts/runqemu | 28 ++++++++++++++++++++++------
 1 file changed, 22 insertions(+), 6 deletions(-)

scripts/runqemu +22 -6

index cbc5cc6..7919564 100755
@@ -570,12 +570,28 @@ class BaseConfig(object):
         if not havenative:
             if not self.bitbake_e:
                 self.load_bitbake_env()
-            native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE']
-            for nv in native_vars:
-                s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
-                if s and s.group(1) != self.get(nv):
-                    logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
-                    self.set(nv, s.group(1))
+
+            if self.bitbake_e:
+                native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE']
+                for nv in native_vars:
+                    s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M)
+                    if s and s.group(1) != self.get(nv):
+                        logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1)))
+                        self.set(nv, s.group(1))
+            else:
+                # when we're invoked from a running bitbake instance we won't
+                # be able to call `bitbake -e` but should have OE_TMPDIR set in
+                # the environment and can guess paths based on it
+                tmpdir = os.environ.get('OE_TMPDIR', None)
+                if tmpdir:
+                    logger.info('Setting STAGING_DIR_NATIVE and STAGING_BINDIR_NATIVE relative to OE_TMPDIR (%s)' % tmpdir)
+                    hostos, _, _, _, machine = os.uname()
+                    buildsys = '%s-%s' % (machine, hostos.lower())
+                    staging_dir_native = '%s/sysroots/%s' % (tmpdir, buildsys)
+                    self.set('STAGING_DIR_NATIVE', staging_dir_native)
+                    # we have to assume that STAGING_BINDIR_NATIVE is at usr/bin
+                    staging_bindir_native = '%s/usr/bin' % staging_dir_native
+                    self.set('STAGING_BINDIR_NATIVE', staging_bindir_native)
 
     def print_config(self):
         logger.info('Continuing with the following parameters:\n')

Signature · 1 line

2.7.4