Skip to contentSkip to conversation

Python 3 support

PATCH New Ed Bartosh 1 participant 15 messages May 4, 2016

lore
Share & tools
lore

Ed Bartosh

author
#0 lore raw

cover Python 3 support

Hi,

This is a first set of changes implementing Python 3 support for wic. As bitbake codebase is not ported to Python 3 yet I tried to avoid using bitbake APIs as much as possible.
With this set of changes wic can be run on both Python 2 and Python 3.

These commands work and produce rootfs of the same size:
python3.4 ../scripts/wic create my-kickstart-file.wks -e core-image-minimal -v ./tmp/sysroots/qemux86-64/imgdata/ -o img/ python2.7 ../scripts/wic create my-kickstart-file.wks -e core-image-minimal -v ./tmp/sysroots/qemux86-64/imgdata/ -o img/

Functionality that requires bb/lib such es wic -e <image> doesn't work.

The following changes since commit 7587285a21cd1e3465640bf9871f3fd65cca4e07:

scripts/lib/bsp/kernel.py: force patching when branch is machine branch is re-use (2016-04-29 12:11:20 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/wic/python3-9412
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/python3-9412

Ed Bartosh (14):
wic: replace print statements with print function
wic: use items instead of iteritems
wic: don't use dict.keys and dict.has_key
wic: don't inherit classes from object
wic: remove with_statement imports
wic: use new syntax of 'except' statement
wic: remove unused functions
wic: refactor pluginbase
wic: don't use L suffix for integers
wic: use find_executable in favor of bb.utils.which
wic: use // operator instead of /
wic: decode output of subprocess.communicate
wic: don't encode unicode strings
wic: use next builtin instead of .next method

 scripts/lib/wic/conf.py                |  2 +-
 scripts/lib/wic/creator.py             |  4 +-
 scripts/lib/wic/engine.py              | 20 ++++-----
 scripts/lib/wic/filemap.py             | 16 +++----
 scripts/lib/wic/help.py                |  4 +-
 scripts/lib/wic/imager/baseimager.py   |  5 +--
 scripts/lib/wic/imager/direct.py       |  2 +-
 scripts/lib/wic/ksparser.py            | 10 ++---
 scripts/lib/wic/msger.py               | 78 +---------------------------------
 scripts/lib/wic/partition.py           |  2 +-
 scripts/lib/wic/plugin.py              | 10 ++---
 scripts/lib/wic/pluginbase.py          | 42 ++++++------------
 scripts/lib/wic/utils/oe/misc.py       | 18 +++-----
 scripts/lib/wic/utils/partitionedfs.py | 14 +++---
 scripts/lib/wic/utils/runner.py        |  6 +--
 scripts/wic                            | 43 ++++++++++---------
 16 files changed, 91 insertions(+), 185 deletions(-)
Signature · 1 line
2.1.4
author
#1 lore raw

PATCH 1/14 wic: replace print statements with print function

Print statements have been replaced with print function in Python 3. Replaced them in wic code to be able to run it under both Python 2 and Python 3.

[YOCTO #9412]

---
 scripts/lib/wic/engine.py        | 20 +++++++++----------
 scripts/lib/wic/help.py          |  2 +-
 scripts/lib/wic/utils/oe/misc.py |  8 ++++----
 scripts/wic                      | 43 ++++++++++++++++++++--------------------
 4 files changed, 37 insertions(+), 36 deletions(-)
scripts/lib/wic/engine.py +10 -10
index 76b93e8..5d35c46 100644
@@ -44,7 +44,7 @@ def verify_build_env():
     Returns True if it is, false otherwise
     """
     if not os.environ.get("BUILDDIR"):
-        print "BUILDDIR not found, exiting. (Did you forget to source oe-init-build-env?)"
+        print("BUILDDIR not found, exiting. (Did you forget to source oe-init-build-env?)")
         sys.exit(1)
 
     return True
@@ -107,7 +107,7 @@ def list_canned_images(scripts_path):
                                 desc = line[idx + len("short-description:"):].strip()
                                 break
                     basename = os.path.splitext(fname)[0]
-                    print "  %s\t\t%s" % (basename.ljust(30), desc)
+                    print("  %s\t\t%s" % (basename.ljust(30), desc))
 
 
 def list_canned_image_help(scripts_path, fullpath):
@@ -120,15 +120,15 @@ def list_canned_image_help(scripts_path, fullpath):
             if not found:
                 idx = line.find("long-description:")
                 if idx != -1:
-                    print
-                    print line[idx + len("long-description:"):].strip()
+                    print()
+                    print(line[idx + len("long-description:"):].strip())
                     found = True
                 continue
             if not line.strip():
                 break
             idx = line.find("#")
             if idx != -1:
-                print line[idx + len("#:"):].rstrip()
+                print(line[idx + len("#:"):].rstrip())
             else:
                 break
 
@@ -140,7 +140,7 @@ def list_source_plugins():
     plugins = pluginmgr.get_source_plugins()
 
     for plugin in plugins:
-        print "  %s" % plugin
+        print("  %s" % plugin)
 
 
 def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
@@ -178,7 +178,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
     try:
         oe_builddir = os.environ["BUILDDIR"]
     except KeyError:
-        print "BUILDDIR not found, exiting. (Did you forget to source oe-init-build-env?)"
+        print("BUILDDIR not found, exiting. (Did you forget to source oe-init-build-env?)")
         sys.exit(1)
 
     if debug:
@@ -189,7 +189,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
     crobj.main(["direct", native_sysroot, kernel_dir, bootimg_dir, rootfs_dir,
                 wks_file, image_output_dir, oe_builddir, compressor or ""])
 
-    print "\nThe image(s) were created using OE kickstart file:\n  %s" % wks_file
+    print("\nThe image(s) were created using OE kickstart file:\n  %s" % wks_file)
 
 
 def wic_list(args, scripts_path):
@@ -209,10 +209,10 @@ def wic_list(args, scripts_path):
         wks_file = args[0]
         fullpath = find_canned_image(scripts_path, wks_file)
         if not fullpath:
-            print "No image named %s found, exiting. "\
+            print("No image named %s found, exiting. "\
                   "(Use 'wic list images' to list available images, or "\
                   "specify a fully-qualified OE kickstart (.wks) "\
-                  "filename)\n" % wks_file
+                  "filename)\n" % wks_file)
             sys.exit(1)
         list_canned_image_help(scripts_path, fullpath)
         return True
scripts/lib/wic/help.py +1 -1
index 394e3fd..158b6c1 100644
@@ -55,7 +55,7 @@ def wic_help(args, usage_str, subcommands):
     Subcommand help dispatcher.
     """
     if len(args) == 1 or not display_help(args[1], subcommands):
-        print usage_str
+        print(usage_str)
 
 
 def get_wic_plugins_help():
scripts/lib/wic/utils/oe/misc.py +4 -4
index a3cbe5f..11209be 100644
@@ -187,8 +187,8 @@ class BitbakeVars(defaultdict):
                         for line in varsfile:
                             self._parse_line(line, image)
                 else:
-                    print "Couldn't get bitbake variable from %s." % fname
-                    print "File %s doesn't exist." % fname
+                    print("Couldn't get bitbake variable from %s." % fname)
+                    print("File %s doesn't exist." % fname)
                     return
             else:
                 # Get bitbake -e output
@@ -202,8 +202,8 @@ class BitbakeVars(defaultdict):
                 msger.set_loglevel(log_level)
 
                 if ret:
-                    print "Couldn't get '%s' output." % cmd
-                    print "Bitbake failed with error:\n%s\n" % lines
+                    print("Couldn't get '%s' output." % cmd)
+                    print("Bitbake failed with error:\n%s\n" % lines)
                     return
 
                 # Parse bitbake -e output
scripts/wic +22 -21

112 lines not on this page · raw message

Signature · 1 line
2.1.4
author
#2 lore raw

PATCH 2/14 wic: use items instead of iteritems

In reply to Ed Bartosh

Dictionary method 'iteritems' doesn't exist in Python 3. Replaced 'iteritems' with 'items' to be able to run the code under both Python 3 and Python 2.

[YOCTO #9412]

---
 scripts/lib/wic/conf.py    | 2 +-
 scripts/lib/wic/creator.py | 2 +-
 scripts/lib/wic/help.py    | 2 +-
 scripts/lib/wic/plugin.py  | 4 ++--
 4 files changed, 5 insertions(+), 5 deletions(-)
scripts/lib/wic/conf.py +1 -1
index f7d56d0..070ec30 100644
@@ -70,7 +70,7 @@ class ConfigMgr(object):
         self.create = {}
 
         # initialize the values with defaults
-        for sec, vals in self.DEFAULTS.iteritems():
+        for sec, vals in self.DEFAULTS.items():
             setattr(self, sec, vals)
 
     def __set_ksconf(self, ksconf):
scripts/lib/wic/creator.py +1 -1
index 5231297..d0158fb 100644
@@ -41,7 +41,7 @@ class Creator(object):
 
         # get cmds from pluginmgr
         # mix-in do_subcmd interface
-        for subcmd, klass in pluginmgr.get_plugins('imager').iteritems():
+        for subcmd, klass in pluginmgr.get_plugins('imager').items():
             if not hasattr(klass, 'do_create'):
                 msger.warning("Unsupported subcmd: %s" % subcmd)
                 continue
scripts/lib/wic/help.py +1 -1
index 158b6c1..7dcc717 100644
@@ -66,7 +66,7 @@ def get_wic_plugins_help():
     result = wic_plugins_help
     for plugin_type in PLUGIN_TYPES:
         result += '\n\n%s PLUGINS\n\n' % plugin_type.upper()
-        for name, plugin in pluginmgr.get_plugins(plugin_type).iteritems():
+        for name, plugin in pluginmgr.get_plugins(plugin_type).items():
             result += "\n %s plugin:\n" % name
             if plugin.__doc__:
                 result += plugin.__doc__
scripts/lib/wic/plugin.py +2 -2
index ccfdfcb..80c609c 100644
@@ -81,7 +81,7 @@ class PluginMgr(object):
             # the value True/False means "loaded"
 
     def _load_all(self):
-        for (pdir, loaded) in self.plugin_dirs.iteritems():
+        for (pdir, loaded) in self.plugin_dirs.items():
             if loaded:
                 continue
 
@@ -135,7 +135,7 @@ class PluginMgr(object):
         None is returned.
         """
         return_methods = None
-        for _source_name, klass in self.get_plugins('source').iteritems():
+        for _source_name, klass in self.get_plugins('source').items():
             if _source_name == source_name:
                 for _method_name in methods.keys():
                     if not hasattr(klass, _method_name):
Signature · 1 line
2.1.4
author
#3 lore raw

PATCH 3/14 wic: don't use dict.keys and dict.has_key

In reply to Ed Bartosh

Replaced calls of dict.keys and dict.has_key methods with the 'key in dict' statement. 'key in dict' is more pythonic, faster and readable. dict.has_key doesn't exist in Python 3.

[YOCTO #9412]

---
 scripts/lib/wic/imager/baseimager.py   | 2 +-
 scripts/lib/wic/plugin.py              | 2 +-
 scripts/lib/wic/utils/partitionedfs.py | 6 +++---
 3 files changed, 5 insertions(+), 5 deletions(-)
scripts/lib/wic/imager/baseimager.py +1 -1
index 760cf8a..557f536 100644
@@ -68,7 +68,7 @@ class BaseImageCreator(object):
                      }
 
             # update setting from createopts
-            for key in createopts.keys():
+            for key in createopts:
                 if key in optmap:
                     option = optmap[key]
                 else:
scripts/lib/wic/plugin.py +1 -1
index 80c609c..933647d 100644
@@ -137,7 +137,7 @@ class PluginMgr(object):
         return_methods = None
         for _source_name, klass in self.get_plugins('source').items():
             if _source_name == source_name:
-                for _method_name in methods.keys():
+                for _method_name in methods:
                     if not hasattr(klass, _method_name):
                         msger.warning("Unimplemented %s source interface for: %s"\
                                       % (_method_name, _source_name))
scripts/lib/wic/utils/partitionedfs.py +3 -3
index 534635b..89a7c13 100644
@@ -132,7 +132,7 @@ class Image(object):
         for num in range(len(self.partitions)):
             part = self.partitions[num]
 
-            if not self.disks.has_key(part['disk_name']):
+            if part['disk_name'] not in self.disks:
                 raise ImageError("No disk %s for partition %s" \
                                  % (part['disk_name'], part['mountpoint']))
 
@@ -236,7 +236,7 @@ class Image(object):
     def __format_disks(self):
         self.layout_partitions()
 
-        for dev in self.disks.keys():
+        for dev in self.disks:
             disk = self.disks[dev]
             msger.debug("Initializing partition table for %s" % \
                         (disk['disk'].device))
@@ -354,7 +354,7 @@ class Image(object):
                 os.rename(source, image_file + '.p%d' % part['num'])
 
     def create(self):
-        for dev in self.disks.keys():
+        for dev in self.disks:
             disk = self.disks[dev]
             disk['disk'].create()
 
Signature · 1 line
2.1.4
author
#4 lore raw

PATCH 4/14 wic: don't inherit classes from object

In reply to Ed Bartosh

All classes in Python3 are new style classes and don't need to be inherited from object. Wic code works fine without this inheritance even with Python2, so it's harmless to remove it.

[YOCTO #9412]

---
 scripts/lib/wic/creator.py             | 2 +-
 scripts/lib/wic/imager/baseimager.py   | 2 +-
 scripts/lib/wic/ksparser.py            | 2 +-
 scripts/lib/wic/partition.py           | 2 +-
 scripts/lib/wic/plugin.py              | 2 +-
 scripts/lib/wic/utils/partitionedfs.py | 2 +-
 6 files changed, 6 insertions(+), 6 deletions(-)
scripts/lib/wic/creator.py +1 -1
index d0158fb..d4972e8 100644
@@ -24,7 +24,7 @@ from wic.conf import configmgr
 from wic.plugin import pluginmgr
 
 
-class Creator(object):
+class Creator():
     """${name}: create an image
 
     Usage:
scripts/lib/wic/imager/baseimager.py +1 -1
index 557f536..463ae93 100644
@@ -25,7 +25,7 @@ from wic import msger
 from wic.utils.errors import CreatorError
 from wic.utils import runner
 
-class BaseImageCreator(object):
+class BaseImageCreator():
     """Base class for image creation.
 
     BaseImageCreator is the simplest creator class available; it will
scripts/lib/wic/ksparser.py +1 -1
index 10d0d7c..27bc1c7 100644
@@ -110,7 +110,7 @@ def systemidtype(arg):
 
     return arg
 
-class KickStart(object):
+class KickStart():
     """"Kickstart parser implementation."""
 
     def __init__(self, confpath):
scripts/lib/wic/partition.py +1 -1
index 6d21193..34fd7d6 100644
@@ -38,7 +38,7 @@ partition_methods = {
     "do_configure_partition":None,
 }
 
-class Partition(object):
+class Partition():
 
     def __init__(self, args, lineno):
         self.args = args
scripts/lib/wic/plugin.py +1 -1
index 933647d..83af9b0 100644
@@ -29,7 +29,7 @@ PLUGIN_TYPES = ["imager", "source"]
 PLUGIN_DIR = "/lib/wic/plugins" # relative to scripts
 SCRIPTS_PLUGIN_DIR = "scripts" + PLUGIN_DIR
 
-class PluginMgr(object):
+class PluginMgr():
     plugin_dirs = {}
 
     # make the manager class as singleton
scripts/lib/wic/utils/partitionedfs.py +1 -1
index 89a7c13..8f4db4e 100644
@@ -33,7 +33,7 @@ GPT_OVERHEAD = 34
 # Size of a sector in bytes
 SECTOR_SIZE = 512
 
-class Image(object):
+class Image():
     """
     Generic base object for an image.
 
Signature · 1 line
2.1.4
author
#6 lore raw

PATCH 6/14 wic: use new syntax of 'except' statement

In reply to Ed Bartosh

New syntax 'except Exception as err' is supported by Python >= 2.7. Old syntax 'except Exception, err' is not supported by Python 3.

Used new syntax to be able to run wic on Python 3.

[YOCTO #9412]

---
 scripts/lib/wic/imager/direct.py | 2 +-
 scripts/lib/wic/plugin.py        | 2 +-
 scripts/lib/wic/utils/runner.py  | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
scripts/lib/wic/imager/direct.py +1 -1
index 319ec16..5a3b655 100644
@@ -395,6 +395,6 @@ class DirectImageCreator(BaseImageCreator):
         if not self.__image is None:
             try:
                 self.__image.cleanup()
-            except ImageError, err:
+            except ImageError as err:
                 msger.warning("%s" % err)
 
scripts/lib/wic/plugin.py +1 -1
index 83af9b0..dcbaa08 100644
@@ -97,7 +97,7 @@ class PluginMgr():
                             self.plugin_dirs[pdir] = True
                             msger.debug("Plugin module %s:%s imported"\
                                         % (mod, pymod.__file__))
-                        except ImportError, err:
+                        except ImportError as err:
                             msg = 'Failed to load plugin %s/%s: %s' \
                                 % (os.path.basename(pdir), mod, err)
                             msger.warning(msg)
scripts/lib/wic/utils/runner.py +1 -1
index 7431917..737751b 100644
@@ -67,7 +67,7 @@ def runtool(cmdln_or_args, catch=1):
         (sout, serr) = process.communicate()
         # combine stdout and stderr, filter None out
         out = ''.join(filter(None, [sout, serr]))
-    except OSError, err:
+    except OSError as err:
         if err.errno == 2:
             # [Errno 2] No such file or directory
             msger.error('Cannot run command: %s, lost dependency?' % cmd)
Signature · 1 line
2.1.4
author
#14 lore raw

PATCH 14/14 wic: use next builtin instead of .next method

In reply to Ed Bartosh

Generators in Python 3 don't have .next method. It's recommended to use 'next' builtin instead. As it also present in Python >= 2.6 it should make wic code compatible with both Python 2 and Python 3.

[YOCTO #9412]

---
 scripts/lib/wic/filemap.py | 2 +-
 scripts/lib/wic/msger.py   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
scripts/lib/wic/filemap.py +1 -1
index 2778be5..f3240ba 100644
@@ -482,7 +482,7 @@ class FilemapFiemap(_FilemapBase):
         self._log.debug("FilemapFiemap: get_mapped_ranges(%d,  %d(%d))"
                         % (start, count, start + count - 1))
         iterator = self._do_get_mapped_ranges(start, count)
-        first_prev, last_prev = iterator.next()
+        first_prev, last_prev = next(iterator)
 
         for first, last in iterator:
             if last_prev == first - 1:
scripts/lib/wic/msger.py +1 -1
index 4d8e704..fb8336d 100644
@@ -148,7 +148,7 @@ def _split_msg(head, msg):
     return head, msg
 
 def get_loglevel():
-    return (k for k, v in LOG_LEVELS.items() if v == LOG_LEVEL).next()
+    return next((k for k, v in LOG_LEVELS.items() if v == LOG_LEVEL))
 
 def set_loglevel(level):
     global LOG_LEVEL
Signature · 1 line
2.1.4
author
#5 lore raw

PATCH 5/14 wic: remove with_statement imports

In reply to Ed Bartosh

'with' statement is not used in baseimager.py
It's supported by Python 2.7, which is included into all target distros. Other wic modules use this statement.

Removed useless 'from __future__ import with_statement' from wic code.

---
 scripts/lib/wic/imager/baseimager.py | 1 -
 1 file changed, 1 deletion(-)
scripts/lib/wic/imager/baseimager.py +0 -1
index 463ae93..1a52dd8 100644
@@ -16,7 +16,6 @@
 # with this program; if not, write to the Free Software Foundation, Inc., 59
 # Temple Place - Suite 330, Boston, MA 02111-1307, USA.
 
-from __future__ import with_statement
 import os
 import tempfile
 import shutil
Signature · 1 line
2.1.4
author
#7 lore raw

PATCH 7/14 wic: remove unused functions

In reply to Ed Bartosh

Removed 'raw', 'ask', 'choice' and 'pause' functions from msger.py as they're not used in wic code and some of them use raw_input, which is not present in Python 3.

[YOCTO #9412]

---
 scripts/lib/wic/msger.py | 69 +-----------------------------------------------
 1 file changed, 1 insertion(+), 68 deletions(-)
scripts/lib/wic/msger.py +1 -68
index b737554..2340ac2 100644
@@ -21,18 +21,14 @@ import sys
 import re
 import time
 
-__ALL__ = ['set_mode',
-           'get_loglevel',
+__ALL__ = ['get_loglevel',
            'set_loglevel',
            'set_logfile',
-           'raw',
            'debug',
            'verbose',
            'info',
            'warning',
            'error',
-           'ask',
-           'pause',
           ]
 
 # COLORs in ANSI
@@ -182,9 +178,6 @@ def log(msg=''):
     if msg:
         LOG_CONTENT += msg
 
-def raw(msg=''):
-    _general_print('', NO_COLOR, msg)
-
 def info(msg):
     head, msg = _split_msg('Info', msg)
     _general_print(head, INFO_COLOR, msg)
@@ -206,66 +199,6 @@ def error(msg):
     _color_perror(head, ERR_COLOR, msg)
     sys.exit(1)
 
-def ask(msg, default=True):
-    _general_print('\rQ', ASK_COLOR, '')
-    try:
-        if default:
-            msg += '(Y/n) '
-        else:
-            msg += '(y/N) '
-        if INTERACTIVE:
-            while True:
-                repl = raw_input(msg)
-                if repl.lower() == 'y':
-                    return True
-                elif repl.lower() == 'n':
-                    return False
-                elif not repl.strip():
-                    # <Enter>
-                    return default
-
-                # else loop
-        else:
-            if default:
-                msg += ' Y'
-            else:
-                msg += ' N'
-            _general_print('', NO_COLOR, msg)
-
-            return default
-    except KeyboardInterrupt:
-        sys.stdout.write('\n')
-        sys.exit(2)
-
-def choice(msg, choices, default=0):
-    if default >= len(choices):
-        return None
-    _general_print('\rQ', ASK_COLOR, '')
-    try:
-        msg += " [%s] " % '/'.join(choices)
-        if INTERACTIVE:
-            while True:
-                repl = raw_input(msg)
-                if repl in choices:
-                    return repl
-                elif not repl.strip():
-                    return choices[default]
-        else:
-            msg += choices[default]
-            _general_print('', NO_COLOR, msg)
-
-            return choices[default]
-    except KeyboardInterrupt:
-        sys.stdout.write('\n')
-        sys.exit(2)
-
-def pause(msg=None):
-    if INTERACTIVE:
-        _general_print('\rQ', ASK_COLOR, '')
-        if msg is None:
-            msg = 'press <ENTER> to continue ...'
-        raw_input(msg)
-
 def set_logfile(fpath):
     global LOG_FILE_FP
 
Signature · 1 line
2.1.4
author
#8 lore raw

PATCH 8/14 wic: refactor pluginbase

In reply to Ed Bartosh

Wic plugin machinery implemented using metaclasses.

Reimplemented plugin machinery using this advice from
https://wiki.python.org/moin/PortingToPy3k/BilingualQuickRef
Syntax for creating instances with different metaclasses is very different between Python 2 and 3. Use the ability to call type instances as a way to portably create such instances.

Now it should work under both Python 2 and Python 3.

[YOCTO #9412]

---
 scripts/lib/wic/pluginbase.py | 42 ++++++++++++++----------------------------
 1 file changed, 14 insertions(+), 28 deletions(-)
scripts/lib/wic/pluginbase.py +14 -28
index ee8fe95..e737dee 100644
@@ -15,34 +15,26 @@
 # with this program; if not, write to the Free Software Foundation, Inc., 59
 # Temple Place - Suite 330, Boston, MA 02111-1307, USA.
 
-from wic import msger
-
-class _Plugin(object):
-    class __metaclass__(type):
-        def __init__(cls, name, bases, attrs):
-            if not hasattr(cls, 'plugins'):
-                cls.plugins = {}
-
-            elif 'wic_plugin_type' in attrs:
-                if attrs['wic_plugin_type'] not in cls.plugins:
-                    cls.plugins[attrs['wic_plugin_type']] = {}
+__all__ = ['ImagerPlugin', 'SourcePlugin', 'get_plugins']
 
-            elif hasattr(cls, 'wic_plugin_type') and 'name' in attrs:
-                cls.plugins[cls.wic_plugin_type][attrs['name']] = cls
+import sys
+from collections import defaultdict
 
-        def show_plugins(cls):
-            for cls in cls.plugins[cls.wic_plugin_type]:
-                print cls
+from wic import msger
 
-        def get_plugins(cls):
-            return cls.plugins
+class PluginMeta(type):
+    plugins = defaultdict(dict)
+    def __new__(cls, name, bases, attrs):
+        class_type = type.__new__(cls, name, bases, attrs)
+        if 'name' in attrs:
+            cls.plugins[class_type.wic_plugin_type][attrs['name']] = class_type
 
+        return class_type
 
-class ImagerPlugin(_Plugin):
+class ImagerPlugin(PluginMeta("Plugin", (), {})):
     wic_plugin_type = "imager"
 
-
-class SourcePlugin(_Plugin):
+class SourcePlugin(PluginMeta("Plugin", (), {})):
     wic_plugin_type = "source"
     """
     The methods that can be implemented by --source plugins.
@@ -99,10 +91,4 @@ class SourcePlugin(_Plugin):
         msger.debug("SourcePlugin: do_prepare_partition: part: %s" % part)
 
 def get_plugins(typen):
-    plugins = ImagerPlugin.get_plugins()
-    if typen in plugins:
-        return plugins[typen]
-    else:
-        return None
-
-__all__ = ['ImagerPlugin', 'SourcePlugin', 'get_plugins']
+    return PluginMeta.plugins.get(typen)
Signature · 1 line
2.1.4
author
#10 lore raw

PATCH 10/14 wic: use find_executable in favor of bb.utils.which

In reply to Ed Bartosh

As bitbake is not ported to Python 3 yet it's better to avoid using its APIs as much as possible to be able to test wic under Python 3 at least partially.

Used distutils.spawn.find_executable API in favor of bb.utils.which to get path of the command to run.

---
 scripts/lib/wic/utils/oe/misc.py | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)
scripts/lib/wic/utils/oe/misc.py +2 -8
index 11209be..0854dfb 100644
@@ -28,6 +28,7 @@
 
 import os
 from collections import defaultdict
+from distutils import spawn
 
 from wic import msger
 from wic.utils import runner
@@ -83,13 +84,6 @@ def exec_cmd(cmd_and_args, as_shell=False, catch=3):
 
     return out
 
-def cmd_in_path(cmd, path):
-    import scriptpath
-
-    scriptpath.add_bitbake_lib_path()
-
-    return bb.utils.which(path, cmd) != "" or False
-
 def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""):
     """
     Execute native command, catching stderr, stdout
@@ -112,7 +106,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""):
     msger.debug("exec_native_cmd: %s" % cmd_and_args)
 
     # If the command isn't in the native sysroot say we failed.
-    if cmd_in_path(args[0], native_paths):
+    if spawn.find_executable(args[0], native_paths):
         ret, out = _exec_cmd(native_cmd_and_args, True, catch)
     else:
         ret = 127
Signature · 1 line
2.1.4
author
#13 lore raw

PATCH 13/14 wic: don't encode unicode strings

In reply to Ed Bartosh

Removed check for unicode type as it doesn't work in Python 3. This check is not needed for wic as all its output seem to be strings. This allows to run code under both pythons.

[YOCTO #9412]

---
 scripts/lib/wic/msger.py | 7 -------
 1 file changed, 7 deletions(-)
scripts/lib/wic/msger.py +0 -7
index 2340ac2..4d8e704 100644
@@ -66,10 +66,6 @@ def _general_print(head, color, msg=None, stream=None, level='normal'):
         # skip
         return
 
-    # encode raw 'unicode' str to utf8 encoded str
-    if msg and isinstance(msg, unicode):
-        msg = msg.encode('utf-8', 'ignore')
-
     errormsg = ''
     if CATCHERR_BUFFILE_FD > 0:
         size = os.lseek(CATCHERR_BUFFILE_FD, 0, os.SEEK_END)
@@ -118,9 +114,6 @@ def _color_print(head, color, msg, stream, level):
                 newline = True
 
     if msg is not None:
-        if isinstance(msg, unicode):
-            msg = msg.encode('utf8', 'ignore')
-
         stream.write('%s%s' % (head, msg))
         if newline:
             stream.write('\n')
Signature · 1 line
2.1.4
author
#9 lore raw

PATCH 9/14 wic: don't use L suffix for integers

In reply to Ed Bartosh

This suffix is not supported by Python 3. Wic code works without it on Python 2 too, so it's safe to remove it.

[YOCTO #9412]

---
 scripts/lib/wic/ksparser.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/lib/wic/ksparser.py +4 -4
index 27bc1c7..6887a7d 100644
@@ -51,7 +51,7 @@ def sizetype(arg):
     Converts size string in <num>[K|k|M|G] format into the integer value
     """
     if arg.isdigit():
-        return int(arg) * 1024L
+        return int(arg) * 1024
 
     if not arg[:-1].isdigit():
         raise ArgumentTypeError("Invalid size: %r" % arg)
@@ -60,9 +60,9 @@ def sizetype(arg):
     if arg.endswith("k") or arg.endswith("K"):
         return size
     if arg.endswith("M"):
-        return size * 1024L
+        return size * 1024
     if arg.endswith("G"):
-        return size * 1024L * 1024L
+        return size * 1024 * 1024
 
     raise ArgumentTypeError("Invalid size: %r" % arg)
 
@@ -127,7 +127,7 @@ class KickStart():
         part.add_argument('mountpoint')
         part.add_argument('--active', action='store_true')
         part.add_argument('--align', type=int)
-        part.add_argument("--extra-space", type=sizetype, default=10*1024L)
+        part.add_argument("--extra-space", type=sizetype, default=10*1024)
         part.add_argument('--fsoptions', dest='fsopts')
         part.add_argument('--fstype')
         part.add_argument('--label')
Signature · 1 line
2.1.4
author
#11 lore raw

PATCH 11/14 wic: use // operator instead of /

In reply to Ed Bartosh

Division operator works differently in Python 3. It results in float unlike in Python 2, where it results in int. Explicitly used "floor division" operator instead of 'division' operator. This should make the code to result in integer under both pythons.

[YOCTO #9412]

---
 scripts/lib/wic/filemap.py             | 14 +++++++-------
 scripts/lib/wic/utils/partitionedfs.py |  6 +++---
 2 files changed, 10 insertions(+), 10 deletions(-)
scripts/lib/wic/filemap.py +7 -7
index f8b2ba7..2778be5 100644
@@ -95,7 +95,7 @@ class _FilemapBase(object):
                         % (self._image_path, err))
 
         self.blocks_cnt = self.image_size + self.block_size - 1
-        self.blocks_cnt /= self.block_size
+        self.blocks_cnt //= self.block_size
 
         try:
             self._f_image.flush()
@@ -254,7 +254,7 @@ class FilemapSeek(_FilemapBase):
         if offs == -1:
             result = False
         else:
-            result = (offs / self.block_size == block)
+            result = (offs // self.block_size == block)
 
         self._log.debug("FilemapSeek: block_is_mapped(%d) returns %s"
                         % (block, result))
@@ -286,8 +286,8 @@ class FilemapSeek(_FilemapBase):
             if end > limit:
                 end = limit
 
-            start_blk = start / self.block_size
-            end_blk = end / self.block_size - 1
+            start_blk = start // self.block_size
+            end_blk = end // self.block_size - 1
             self._log.debug("FilemapSeek: yielding range (%d, %d)"
                             % (start_blk, end_blk))
             yield (start_blk, end_blk)
@@ -351,7 +351,7 @@ class FilemapFiemap(_FilemapBase):
 
         # Calculate how many 'struct fiemap_extent' elements fit the buffer
         self._buf_size -= _FIEMAP_SIZE
-        self._fiemap_extent_cnt = self._buf_size / _FIEMAP_EXTENT_SIZE
+        self._fiemap_extent_cnt = self._buf_size // _FIEMAP_EXTENT_SIZE
         assert self._fiemap_extent_cnt > 0
         self._buf_size = self._fiemap_extent_cnt * _FIEMAP_EXTENT_SIZE
         self._buf_size += _FIEMAP_SIZE
@@ -456,11 +456,11 @@ class FilemapFiemap(_FilemapBase):
                 # Start of the extent
                 extent_start = fiemap_extent[0]
                 # Starting block number of the extent
-                extent_block = extent_start / self.block_size
+                extent_block = extent_start // self.block_size
                 # Length of the extent
                 extent_len = fiemap_extent[2]
                 # Count of blocks in the extent
-                extent_count = extent_len / self.block_size
+                extent_count = extent_len // self.block_size
 
                 # Extent length and offset have to be block-aligned
                 assert extent_start % self.block_size == 0
scripts/lib/wic/utils/partitionedfs.py +3 -3
index 8f4db4e..46b5d34 100644
@@ -95,7 +95,7 @@ class Image():
         ks_pnum = len(self.partitions)
 
         # Converting kB to sectors for parted
-        size = size * 1024 / self.sector_size
+        size = size * 1024 // self.sector_size
 
         part = {'ks_pnum': ks_pnum, # Partition number in the KS file
                 'size': size, # In sectors
@@ -173,12 +173,12 @@ class Image():
                 # gaps we could enlargea the previous partition?
 
                 # Calc how much the alignment is off.
-                align_sectors = disk['offset'] % (part['align'] * 1024 / self.sector_size)
+                align_sectors = disk['offset'] % (part['align'] * 1024 // self.sector_size)
 
                 if align_sectors:
                     # If partition is not aligned as required, we need
                     # to move forward to the next alignment point
-                    align_sectors = (part['align'] * 1024 / self.sector_size) - align_sectors
+                    align_sectors = (part['align'] * 1024 // self.sector_size) - align_sectors
 
                     msger.debug("Realignment for %s%s with %s sectors, original"
                                 " offset %s, target alignment is %sK." %
Signature · 1 line
2.1.4
author
#12 lore raw

PATCH 12/14 wic: decode output of subprocess.communicate

In reply to Ed Bartosh

stdeout and stderr content returned by communicate API has different types in Python 3(bytes) and Python 2(string). Decoding it to 'utf-8' makes it unicode on both pythons.

Decoded stdout and stderr output to utf-8 to make the code working under both Python 2 and Python 3.

---
 scripts/lib/wic/utils/runner.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/lib/wic/utils/runner.py +2 -2
index 737751b..db536ba 100644
@@ -65,8 +65,8 @@ def runtool(cmdln_or_args, catch=1):
         process = subprocess.Popen(cmdln_or_args, stdout=sout,
                                    stderr=serr, shell=shell)
         (sout, serr) = process.communicate()
-        # combine stdout and stderr, filter None out
-        out = ''.join(filter(None, [sout, serr]))
+        # combine stdout and stderr, filter None out and decode
+        out = ''.join([out.decode('utf-8') for out in [sout, serr] if out])
     except OSError as err:
         if err.errno == 2:
             # [Errno 2] No such file or directory
Signature · 1 line
2.1.4