Python3 runqemu
cover Python3 runqemu
This small series against master-next (as the python3 runqemu hasn't made it to master yet) fixes the new runqemu for a workflow of mine where I build on a headless machine and copy images locally to test them with runqemu.
The following changes since commit e0f246cefb8b6642e0df98067ade06833e83c1f8:
gobject-introspection: set GI_SCANNER_DISABLE_CACHE for native (2016-09-05 11:59:01 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib joshuagl/runqemu
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=joshuagl/runqemu
Joshua Lock (4):
runqemu: fix typos
runqemu: use glob rather than calling out to ls
runqemu: assume artefacts are relative to *.qemuboot.conf
runqemu: get NATIVE dirs from env when in source OE build dir
scripts/runqemu | 44 ++++++++++++++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 6 deletions(-)
Signature · 1 line
2.7.4
PATCH 1/4 runqemu: fix typos
Remove a stray whitespace when accessing a member variable and fix a spelling mistake in an Exception message.
--- scripts/runqemu | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/runqemu +2 -2
@@ -160,7 +160,7 @@ class BaseConfig(object): self.custombiosdir = '' self.lock = '' self.lock_descriptor = '' - self. bitbake_e = '' + self.bitbake_e = '' self.fstypes = ('ext2', 'ext3', 'ext4', 'jffs2', 'nfs', 'btrfs', 'cpio.gz', 'cpio', 'ramfs') self.vmtypes = ('hddimg', 'hdddirect', 'wic', 'vmdk', 'qcow2', 'vdi', 'iso')
@@ -289,7 +289,7 @@ class BaseConfig(object): self.set_machine_deploy_dir(arg, deploy_dir_image) else: logger.error("%s not a directory valid DEPLOY_DIR_IMAGE" % deploy_dir_image) - raise Exception("Falied to set MACHINE to %s. Unknown arg: %s" % (arg, arg)) + raise Exception("Failed to set MACHINE to %s. Unknown arg: %s" % (arg, arg)) def check_args(self): unknown_arg = ""
Signature · 1 line
2.7.4
PATCH 2/4 runqemu: use glob rather than calling out to ls
In reply to Joshua LockUse the Python glob module to find files matching a pattern, rather than calling out to ls and parsing the output.
--- scripts/runqemu | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
scripts/runqemu +3 -4
@@ -528,11 +528,10 @@ class BaseConfig(object): self.qemuboot = "%s/%s-%s.qemuboot.conf" % (deploy_dir_image, self.rootfs, machine) else: - cmd = 'ls -t %s/*.qemuboot.conf' % deploy_dir_image - logger.info('Running %s...' % cmd) - qbs = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8') + import glob + qbs = glob.glob('%s/*.qemuboot.conf' % deploy_dir_image) if qbs: - self.qemuboot = qbs.split()[0] + self.qemuboot = qbs.split[0] if not os.path.exists(self.qemuboot): raise Exception("Failed to find <image>.qemuboot.conf!")
Signature · 1 line
2.7.4
Re: [PATCH 2/4] runqemu: use glob rather than calling out to ls
Quoted 13 lines · Joshua Lock
Use the Python glob module to find files matching a pattern, rather than calling out to ls and parsing the output.
Signed-off-by Joshua Lock <[email protected]>--- scripts/runqemu | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)scripts/runqemu +0
-0
scripts/runqemu class BaseConfig(object): · line 532
Quoted 8 lines
scripts/runqemu +2
-3@@ -528,11 +528,10 @@ class BaseConfig(object): self.qemuboot = "%s/%s-%s.qemuboot.conf" % (deploy_dir_image, self.rootfs, machine) else: - cmd = 'ls -t %s/*.qemuboot.conf' % deploy_dir_image - logger.info('Running %s...' % cmd) - qbs = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8') + import glob + qbs = glob.glob('%s/*.qemuboot.conf' % deploy_dir_image)
I'm not sure about this, "ls -lt" will sort by time and we will use the latest one when the user run "runqemu core-image-minimal qemux86", but glob.glob() seems can't do this.
I have to go to sleep now, have a good day, talk to you tomorrow.
// Robert
Quoted 7 lines
scripts/runqemu +1
-1if qbs: - self.qemuboot = qbs.split()[0] + self.qemuboot = qbs.split[0] if not os.path.exists(self.qemuboot): raise Exception("Failed to find <image>.qemuboot.conf!")
Re: [PATCH 2/4] runqemu: use glob rather than calling out to ls
Quoted 24 lines · Robert Yang
Quoted 20 lines · Joshua Lock
Use the Python glob module to find files matching a pattern, rather than calling out to ls and parsing the output.
Signed-off-by Joshua Lock <[email protected]>--- scripts/runqemu | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)scripts/runqemu +0
-3@@ -528,11 +528,10 @@ class BaseConfig(object): self.qemuboot = "%s/%s-%s.qemuboot.conf" % (deploy_dir_image, self.rootfs, machine) else: - cmd = 'ls -t %s/*.qemuboot.conf' % deploy_dir_image - logger.info('Running %s...' % cmd) - qbs = subprocess.Popen(cmd, shell=True,
Quoted 9 lines
scripts/runqemu class BaseConfig(object): · line 535
Quoted 4 lines
scripts/runqemu +2
-0stdout=subprocess.PIPE).stdout.read().decode('utf-8') + import glob + qbs = glob.glob('%s/*.qemuboot.conf' % deploy_dir_image)I'm not sure about this, "ls -lt" will sort by time and we will use the latest one when the user run "runqemu core-image-minimal qemux86", but
glob.glob()seems can't do this.
We could ensure the list returned by glob is sorted, whilst I prefer to avoid calling out to the shell unnecessarily I'm not really attached to this change and can drop it for v2 of the series.
Quoted 2 lines
I have to go to sleep now, have a good day, talk to you tomorrow.
Thanks for the review!
Quoted 11 lines
// Robert
if qbs:
Quoted 7 lines
scripts/runqemu +1
-1- self.qemuboot = qbs.split()[0] + self.qemuboot = qbs.split[0]if not
os.path.exists(self.qemuboot):
raiseException("Failed to find <image>.qemuboot.conf!")
Signature · 1 line
Joshua Lock <[email protected]>
PATCH 3/4 runqemu: assume artefacts are relative to *.qemuboot.conf
In reply to Joshua LockWhen runqemu is started with a *.qemuboot.conf arg assume that image artefacts are relative to that file, rather than in whatever directory the DEPLOY_DIR_IMAGE variable in the conf file points to.
--- scripts/runqemu | 11 +++++++++++ 1 file changed, 11 insertions(+)
scripts/runqemu +11 -0
@@ -143,6 +143,7 @@ class BaseConfig(object): self.nfs_server = '' self.rootfs = '' self.qemuboot = '' + self.qbconfload = False self.kernel = '' self.kernel_cmdline = '' self.kernel_cmdline_script = ''
@@ -239,6 +240,7 @@ class BaseConfig(object): """ if p.endswith('.qemuboot.conf'): self.qemuboot = p + self.qbconfload = True elif re.search('\.bin$', p) or re.search('bzImage', p) or \ re.search('zImage', p) or re.search('vmlinux', p) or \ re.search('fitImage', p) or re.search('uImage', p):
@@ -544,6 +546,15 @@ class BaseConfig(object): k_upper = k.upper() self.set(k_upper, v) + # When we're started with a *.qemuboot.conf arg assume that image + # artefacts are relative to that file, rather than in whatever + # directory DEPLOY_DIR_IMAGE in the conf file points to. + if self.qbconfload: + imgdir = os.path.dirname(self.qemuboot) + if imgdir != self.get('DEPLOY_DIR_IMAGE'): + logger.info('Setting DEPLOY_DIR_IMAGE to folder containing %s (%s)' % (self.qemuboot, imgdir)) + self.set('DEPLOY_DIR_IMAGE', imgdir) + def print_config(self): logger.info('Continuing with the following parameters:\n') if not self.fstype in self.vmtypes:
Signature · 1 line
2.7.4
PATCH 4/4 runqemu: get NATIVE dirs from env when in source OE build dir
In reply to Joshua LockWhen we're running from a sourced OE build directory parse the STAGING_* directories from the bitbake environment, rather than using those hard-coded in the conf file.
--- scripts/runqemu | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
scripts/runqemu +22 -0
@@ -555,6 +555,16 @@ class BaseConfig(object): logger.info('Setting DEPLOY_DIR_IMAGE to folder containing %s (%s)' % (self.qemuboot, imgdir)) self.set('DEPLOY_DIR_IMAGE', imgdir) + # When we're running from a sourced OE environment use the STAGING_* + # directories from the environment. + if self.bitbake_e: + native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE'] + for nv in native_vars: + s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M) + if s and s.group(1) != self.get(nv): + logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1))) + self.set(nv, s.group(1)) + def print_config(self): logger.info('Continuing with the following parameters:\n') if not self.fstype in self.vmtypes:
@@ -815,6 +825,17 @@ class BaseConfig(object): shutil.rmtree(self.nfs_dir) shutil.rmtree('%s.pseudo_state' % self.nfs_dir) + def check_sourced_env(self): + bitbake = shutil.which('bitbake') + if bitbake and not self.bitbake_e: + mach = self.get('MACHINE') + if mach: + cmd = 'MACHINE=%s bitbake -e' % mach + else: + cmd = 'bitbake -e' + logger.info('Running %s...' % cmd) + self.bitbake_e = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8') + def main(): if len(sys.argv) == 1 or "help" in sys.argv: print_usage()
@@ -826,6 +847,7 @@ def main(): logger.error(esc) logger.error("Try 'runqemu help' on how to use it") return 1 + config.check_sourced_env() config.read_qemuboot() config.check_and_set() config.print_config()
Signature · 1 line
2.7.4
Re: [PATCH 4/4] runqemu: get NATIVE dirs from env when in source OE build dir
Hi Josh,
Thanks for fixing this, I got errors when test this patch:
$ runqemu tmp/deploy/images/qemux86/
runqemu - INFO - Running bitbake -e...
Traceback (most recent call last):
File "/buildarea/lyang1/poky/scripts/runqemu", line 864, in <module>
ret = main()
File "/buildarea/lyang1/poky/scripts/runqemu", line 851, in main
config.read_qemuboot()
File "/buildarea/lyang1/poky/scripts/runqemu", line 536, in read_qemuboot
self.qemuboot = qbs.split[0]
AttributeError: 'list' object has no attribute 'split'And after your patches, it always need run "bitbake -e", this makes us can't use runqemu when another build is running:
ERROR: Only one copy of bitbake should be run against a build directory
Maybe we check STAGING_DIR_NATIVE, STAGING_BINDIR_NATIVE
and STAGING_DIR_HOST, if they are existed, then we don't need run bitbake -e.
And please see my comments below.
Quoted 34 lines · Joshua Lock
When we're running from a sourced OE build directory parse the STAGING_* directories from the bitbake environment, rather than using those hard-coded in the conf file.
Signed-off-by Joshua Lock <[email protected]>--- scripts/runqemu | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)scripts/runqemu +10
-0@@ -555,6 +555,16 @@ class BaseConfig(object): logger.info('Setting DEPLOY_DIR_IMAGE to folder containing %s (%s)' % (self.qemuboot, imgdir)) self.set('DEPLOY_DIR_IMAGE', imgdir) + # When we're running from a sourced OE environment use the STAGING_* + # directories from the environment. + if self.bitbake_e: + native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE'] + for nv in native_vars: + s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M) + if s and s.group(1) != self.get(nv): + logger.info('Overriding conf file setting of %s to %s from Bitbake environment' % (nv, s.group(1))) + self.set(nv, s.group(1)) + def print_config(self): logger.info('Continuing with the following parameters:\n') if not self.fstype in self.vmtypes:@@ -815,6 +825,17 @@ class BaseConfig(object): shutil.rmtree(self.nfs_dir) shutil.rmtree('%s.pseudo_state' % self.nfs_dir)
scripts/runqemu class BaseConfig(object): · line 837
Quoted 10 lines
scripts/runqemu +10
-0+ def check_sourced_env(self): + bitbake = shutil.which('bitbake') + if bitbake and not self.bitbake_e: + mach = self.get('MACHINE') + if mach: + cmd = 'MACHINE=%s bitbake -e' % mach + else: + cmd = 'bitbake -e' + logger.info('Running %s...' % cmd) + self.bitbake_e = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
I think that we need check whether the command is successed or not, and print errors if it is failed to run.
// Robert
Quoted 13 lines
scripts/runqemu +2
-0+ def main(): if len(sys.argv) == 1 or "help" in sys.argv: print_usage()@@ -826,6 +847,7 @@ def main(): logger.error(esc) logger.error("Try 'runqemu help' on how to use it") return 1 + config.check_sourced_env() config.read_qemuboot() config.check_and_set() config.print_config()
Re: [PATCH 4/4] runqemu: get NATIVE dirs from env when in source OE build dir
Quoted 4 lines · Robert Yang
Hi Josh,
Thanks for fixing this, I got errors when test this patch:
Quoted 12 lines
$ runqemu tmp/deploy/images/qemux86/
runqemu - INFO - Running bitbake -e...
Traceback (most recent call last):
File "/buildarea/lyang1/poky/scripts/runqemu", line 864, in <module>
ret = main()
File "/buildarea/lyang1/poky/scripts/runqemu", line 851, in main
config.read_qemuboot()
File "/buildarea/lyang1/poky/scripts/runqemu", line 536, in
read_qemuboot
self.qemuboot = qbs.split[0]
AttributeError: 'list' object has no attribute 'split'Thanks for reviewing and catching this.
Quoted 8 lines
And after your patches, it always need run "bitbake -e", this makes us can't use runqemu when another build is running:
ERROR: Only one copy of bitbake should be run against a build directoryMaybe we check
STAGING_DIR_NATIVE,STAGING_BINDIR_NATIVE
andSTAGING_DIR_HOST, if they are existed, then we don't need run bitbake -e.
That's a nice idea, I'll implement that for v2 of the series.
Quoted 42 lines
And please see my comments below.
Quoted 37 lines · Joshua Lock
When we're running from a sourced OE build directory parse the STAGING_* directories from the bitbake environment, rather than using those hard-coded in the conf file.
Signed-off-by Joshua Lock <[email protected]>--- scripts/runqemu | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)scripts/runqemu +1
-0@@ -555,6 +555,16 @@ class BaseConfig(object): logger.info('Setting DEPLOY_DIR_IMAGE to folder containing %s (%s)' % (self.qemuboot, imgdir)) self.set('DEPLOY_DIR_IMAGE', imgdir) + # When we're running from a sourced OE environment use theSTAGING_*
scripts/runqemu +7
-0+ # directories from the environment. + if self.bitbake_e: + native_vars = ['STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE'] + for nv in native_vars: + s = re.search('^%s="(.*)"' % nv, self.bitbake_e, re.M) + if s and s.group(1) != self.get(nv): + logger.info('Overriding conf file setting of %s to%s from Bitbake environment' % (nv,
s.group(1)))scripts/runqemu +2
-0+ self.set(nv, s.group(1)) + def print_config(self): logger.info('Continuing with the following parameters:\n') if not self.fstype in self.vmtypes:@@ -815,6 +825,17 @@ class BaseConfig(object): shutil.rmtree(self.nfs_dir) shutil.rmtree('%s.pseudo_state' % self.nfs_dir)
Quoted 16 lines
scripts/runqemu class BaseConfig(object):
Quoted 12 lines
scripts/runqemu +10
-0+ def check_sourced_env(self): + bitbake = shutil.which('bitbake') + if bitbake and not self.bitbake_e: + mach = self.get('MACHINE') + if mach: + cmd = 'MACHINE=%s bitbake -e' % mach + else: + cmd = 'bitbake -e' + logger.info('Running %s...' % cmd) + self.bitbake_e = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')I think that we need check whether the command is successed or not, and print errors if it is failed to run.
Sure, I'll implement that.
Joshua <[email protected]>