Skip to contentSkip to conversation

porting scripts to python3

PATCH In review Ed Bartosh 3 participants 20 messages Jun 2, 2016 – Jun 6, 2016

lore
Share & tools
lore

Ed Bartosh

author
#0 lore raw

cover porting scripts to python3

Hi,

Please review set of python3 fixes for oe scripts.

As most of the scripts are not covered by tests it was quite hard to test them. I tested only those which I know how to work with. It would be greate to have any help with testing.

The following changes since commit dcbfc72303d6782d9108538c830cf2ee7849bf59:

poky: Add Ubuntu 16.04 to tested/supported list since I run it myself (2016-06-02 10:37:05 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/oe/python3
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe/python3

Ed Bartosh (16):
scripts: python3: convert iterables to lists
scripts: python3: use new style except statement
scripts: python3: rename raw_input to input
scripts: python3: fix urllib imports ksize.py: python3: get rid of strings.join
scripts: python3: get rid of __future__ imports
scripts: python3: use new metaclass syntax
combo-layer: python3: import reduce engine.py: python3: rename sys.maxint to sys.maxsize
scripts: python3: use explicit relative imports
scripts: python3: replace exec statement with builtin
engine: python3: replace iteritems() -> items()
dirsize: python3: fix TypeError: unorderable types
scripts: python3: decode subprocess output
combo-layer: python3: use tempfile.TemporaryFile
scripts: python3: change python to python3 in shebang

 scripts/bitbake-whatchanged                      |  9 +++----
 scripts/buildhistory-collect-srcrevs             |  2 +-
 scripts/cleanup-workdir                          |  4 +--
 scripts/combo-layer                              | 15 ++++++-----
 scripts/contrib/bbvars.py                        | 16 +++++------
 scripts/contrib/devtool-stress.py                |  2 +-
 scripts/contrib/list-packageconfig-flags.py      |  2 +-
 scripts/contrib/verify-homepage.py               |  6 ++---
 scripts/cp-noerror                               | 10 +++----
 scripts/gen-lockedsig-cache                      |  2 +-
 scripts/lib/bsp/engine.py                        | 34 +++++++++++-------------
 scripts/lib/bsp/kernel.py                        | 25 +++++++++--------
 scripts/lib/devtool/__init__.py                  |  2 +-
 scripts/lib/devtool/build.py                     |  2 +-
 scripts/lib/devtool/standard.py                  |  4 +--
 scripts/lib/recipetool/create_buildsys.py        |  2 +-
 scripts/lib/recipetool/create_buildsys_python.py |  6 ++---
 scripts/oe-pkgdata-util                          |  2 +-
 scripts/oe-publish-sdk                           |  2 +-
 scripts/oe-selftest                              |  2 +-
 scripts/oe-trim-schemas                          |  2 +-
 scripts/oepydevshell-internal.py                 |  4 +--
 scripts/opkg-query-helper.py                     |  2 +-
 scripts/pybootchartgui/pybootchartgui/main.py.in |  2 --
 scripts/pybootchartgui/pybootchartgui/parsing.py |  3 ---
 scripts/pythondeps                               |  2 +-
 scripts/relocate_sdk.py                          |  2 +-
 scripts/send-error-report                        | 22 +++++++--------
 scripts/swabber-strace-attach                    |  2 +-
 scripts/sysroot-relativelinks.py                 |  2 +-
 scripts/test-remote-image                        | 10 +++----
 scripts/tiny/dirsize.py                          | 16 +++++------
 scripts/tiny/ksize.py                            |  8 +++---
 scripts/wic                                      |  2 --
 scripts/yocto-bsp                                |  2 +-
 scripts/yocto-kernel                             |  2 +-
 scripts/yocto-layer                              |  2 +-
 37 files changed, 107 insertions(+), 127 deletions(-)

--
Regards,
Ed

author
#1 lore raw

PATCH 1/16 scripts: python3: convert iterables to lists

Converted return value of items() keys() and values() to lists when dictionary is modified in the loop and when the result is added to the list.

---
 scripts/bitbake-whatchanged                      | 6 +++---
 scripts/combo-layer                              | 2 +-
 scripts/lib/devtool/build.py                     | 2 +-
 scripts/lib/devtool/standard.py                  | 4 ++--
 scripts/lib/recipetool/create_buildsys.py        | 2 +-
 scripts/lib/recipetool/create_buildsys_python.py | 6 +++---
 scripts/oe-pkgdata-util                          | 2 +-
 7 files changed, 12 insertions(+), 12 deletions(-)
scripts/bitbake-whatchanged +3 -3
index a20adb2..b05aead 100755
@@ -120,7 +120,7 @@ def print_added(d_new = None, d_old = None):
     Print the newly added tasks
     """
     added = {}
-    for k in d_new.keys():
+    for k in list(d_new.keys()):
         if k not in d_old:
             # Add the new one to added dict, and remove it from
             # d_new, so the remaining ones are the changed ones
@@ -155,7 +155,7 @@ def print_vrchanged(d_new = None, d_old = None, vr = None):
     """
     pvchanged = {}
     counter = 0
-    for k in d_new.keys():
+    for k in list(d_new.keys()):
         if d_new.get(k).get(vr) != d_old.get(k).get(vr):
             counter += 1
             pn, task = split_pntask(k)
@@ -279,7 +279,7 @@ Note:
 
         # Remove the same one from both stamps.
         cnt_unchanged = 0
-        for k in new_dict.keys():
+        for k in list(new_dict.keys()):
             if k in old_dict:
                 cnt_unchanged += 1
                 del(new_dict[k])
scripts/combo-layer +1 -1
index 234d9e4..7c41f92 100755
@@ -1186,7 +1186,7 @@ def update_with_history(conf, components, revisions, repos):
         msg = conf_commit_msg(conf, components)
         new_tree = runcmd("git write-tree", **wargs).strip()
         new_rev = runcmd("git commit-tree".split() +
-                         add_p([head] + additional_heads.keys()) +
+                         add_p([head] + list(additional_heads.keys())) +
                          ["-m", msg, new_tree],
                          **wargs).strip()
         # And done! This is the first time we change the HEAD in the actual work tree.
scripts/lib/devtool/build.py +1 -1
index 48f6fe1..6be549d 100644
@@ -27,7 +27,7 @@ logger = logging.getLogger('devtool')
 
 
 def _set_file_values(fn, values):
-    remaining = values.keys()
+    remaining = list(values.keys())
 
     def varfunc(varname, origvalue, op, newlines):
         newvalue = values.get(varname, origvalue)
scripts/lib/devtool/standard.py +2 -2
index 08153c6..a2516d6 100644
@@ -998,7 +998,7 @@ def _export_local_files(srctree, rd, destdir):
         bb.process.run(['git', 'checkout', tree, '--', '.'], cwd=srctree,
                         env=dict(os.environ, GIT_WORK_TREE=destdir,
                                  GIT_INDEX_FILE=tmp_index))
-        new_set = _git_ls_tree(srctree, tree, True).keys()
+        new_set = list(_git_ls_tree(srctree, tree, True).keys())
     elif os.path.isdir(local_files_dir):
         # If not tracked by Git, just copy from working copy
         new_set = _ls_tree(os.path.join(srctree, 'oe-local-files'))
@@ -1309,7 +1309,7 @@ def reset(args, config, basepath, workspace):
         raise DevtoolError("Recipe must be specified, or specify -a/--all to "
                            "reset all recipes")
     if args.all:
-        recipes = workspace.keys()
+        recipes = list(workspace.keys())
     else:
         recipes = [args.recipename]
 
scripts/lib/recipetool/create_buildsys.py +1 -1
index de3d9ae..78ae4bc 100644
@@ -682,7 +682,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
                 process_macro(in_keyword, partial)
 
         if extravalues:
-            for k,v in extravalues.items():
+            for k,v in list(extravalues.items()):
                 if v:
                     if v.startswith('$') or v.startswith('@') or v.startswith('%'):
                         del extravalues[k]
scripts/lib/recipetool/create_buildsys_python.py +3 -3
index 55cce0e..aff13cf 100644
@@ -361,7 +361,7 @@ class PythonRecipeHandler(RecipeHandler):
 
         # Naive mapping of setup() arguments to PKG-INFO field names
         for d in [info, non_literals]:
-            for key, value in d.items():
+            for key, value in list(d.items()):
                 new_key = _map(key)
                 if new_key != key:
                     del d[key]
@@ -443,7 +443,7 @@ class PythonRecipeHandler(RecipeHandler):
                 elif new_value != value:
                     info[variable] = new_value
             elif hasattr(value, 'items'):
-                for dkey, dvalue in value.items():
+                for dkey, dvalue in list(value.items()):
                     new_list = []
                     for pos, a_value in enumerate(dvalue):
                         new_value = replace_value(search, replace, a_value)
@@ -608,7 +608,7 @@ def gather_setup_info(fileobj):
     visitor.visit(parsed)
 
     non_literals, extensions = {}, []
-    for key, value in visitor.keywords.items():
+    for key, value in list(visitor.keywords.items()):
         if key == 'ext_modules':
             if isinstance(value, list):
                 for ext in value:
scripts/oe-pkgdata-util +1 -1
index b39d9b5..b16ecc9 100755
@@ -240,7 +240,7 @@ def lookup_pkg(args):
         sys.exit(1)
 
     if args.reverse:
-        items = mappings.values()
+        items = list(mappings.values())
     else:
         items = []
         for pkg in pkgs:
Signature · 1 line
2.1.4
author
#4 lore raw

PATCH 2/16 scripts: python3: use new style except statement

In reply to Ed Bartosh

Changed old syle except statements 'except <exception>, var' to new style 'except <exception> as var' as old style is not supported in python3.

---
 scripts/combo-layer       |  4 ++--
 scripts/contrib/bbvars.py | 14 +++++++-------
 scripts/cp-noerror        |  8 ++++----
 scripts/tiny/ksize.py     |  2 +-
 4 files changed, 14 insertions(+), 14 deletions(-)
scripts/combo-layer +2 -2
index 7c41f92..0644cdc 100755
@@ -190,7 +190,7 @@ def runcmd(cmd,destdir=None,printerr=True,out=None,env=None):
         err = os.tmpfile()
     try:
         subprocess.check_call(cmd, stdout=out, stderr=err, cwd=destdir, shell=isinstance(cmd, str), env=env or os.environ)
-    except subprocess.CalledProcessError,e:
+    except subprocess.CalledProcessError as e:
         err.seek(0)
         if printerr:
             logger.error("%s" % err.read())
@@ -429,7 +429,7 @@ file_exclude = %s''' % (name, file_filter or '<empty>', repo.get('file_exclude',
                     runcmd('git replace --graft %s %s' % (start, startrev))
             try:
                 runcmd(merge)
-            except Exception, error:
+            except Exception as error:
                 logger.info('''Merging component repository history failed, perhaps because of merge conflicts.
 It may be possible to commit anyway after resolving these conflicts.
 
scripts/contrib/bbvars.py +7 -7
index 6f47935..b865dd1 100755
@@ -37,9 +37,9 @@ def recipe_bbvars(recipe):
     vset = set()
     try:
         r = open(recipe)
-    except IOError as (errno, strerror):
+    except IOError as err:
         print('WARNING: Failed to open recipe ', recipe)
-        print(strerror)
+        print(err.args[1])
 
     for line in r:
         # Strip any comments from the line
@@ -71,9 +71,9 @@ def bbvar_is_documented(var, docfiles):
     for doc in docfiles:
         try:
             f = open(doc)
-        except IOError as (errno, strerror):
+        except IOError as err:
             print('WARNING: Failed to open doc ', doc)
-            print(strerror)
+            print(err.args[1])
         for line in f:
             if prog.match(line):
                 return True
@@ -87,8 +87,8 @@ def bbvar_doctag(var, docconf):
 
     try:
         f = open(docconf)
-    except IOError as (errno, strerror):
-        return strerror
+    except IOError as err:
+        return err.args[1]
 
     for line in f:
         m = prog.search(line)
@@ -109,7 +109,7 @@ def main():
     # Collect and validate input
     try:
         opts, args = getopt.getopt(sys.argv[1:], "d:hm:t:T", ["help"])
-    except getopt.GetoptError, err:
+    except getopt.GetoptError as err:
         print('%s' % str(err))
         usage()
         sys.exit(2)
scripts/cp-noerror +4 -4
index 28eb90d..d8be677 100755
@@ -33,16 +33,16 @@ def copytree(src, dst, symlinks=False, ignore=None):
                 shutil.copy2(srcname, dstname)
         # catch the Error from the recursive copytree so that we can
         # continue with other files
-        except shutil.Error, err:
+        except shutil.Error as err:
             errors.extend(err.args[0])
-        except EnvironmentError, why:
+        except EnvironmentError as why:
             errors.append((srcname, dstname, str(why)))
     try:
         shutil.copystat(src, dst)
-    except OSError, why:
+    except OSError as why:
         errors.extend((src, dst, str(why)))
     if errors:
-        raise shutil.Error, errors
+        raise shutil.Error(errors)
 
 try:
     copytree(sys.argv[1], sys.argv[2])
scripts/tiny/ksize.py +1 -1
index 275c983..54b71f8 100755
@@ -133,7 +133,7 @@ class Report:
 def main():
     try:
         opts, args = getopt.getopt(sys.argv[1:], "dh", ["help"])
-    except getopt.GetoptError, err:
+    except getopt.GetoptError as err:
         print('%s' % str(err))
         usage()
         sys.exit(2)
Signature · 1 line
2.1.4
author
#5 lore raw

PATCH 3/16 scripts: python3: rename raw_input to input

In reply to Ed Bartosh

Renamed raw_input to input as raw_input does not
exist in python 3.

---
 scripts/lib/bsp/engine.py        | 12 ++++++------
 scripts/lib/bsp/kernel.py        | 12 ++++++------
 scripts/oepydevshell-internal.py |  2 +-
 scripts/send-error-report        |  6 +++---
 4 files changed, 16 insertions(+), 16 deletions(-)
scripts/lib/bsp/engine.py +6 -6
index 3ab4295..f75454d 100644
@@ -464,9 +464,9 @@ class ListInputLine(InputLine):
         choices_str = self.gen_choices_str(choicepairs)
         choices_val_list = self.gen_choices_val_list(choicepairs)
         if checklist:
-            choiceval = default(find_choicevals(raw_input(msg + "\n" + choices_str), choices_val_list), default_choice)
+            choiceval = default(find_choicevals(input(msg + "\n" + choices_str), choices_val_list), default_choice)
         else:
-            choiceval = default(find_choiceval(raw_input(msg + "\n" + choices_str), choices_val_list), default_choice)
+            choiceval = default(find_choiceval(input(msg + "\n" + choices_str), choices_val_list), default_choice)
 
         return choiceval
 
@@ -540,12 +540,12 @@ def get_verified_git_repo(input_str, name):
     """
     msg = input_str.strip() + " "
 
-    giturl = default(raw_input(msg), name)
+    giturl = default(input(msg), name)
 
     while True:
         if verify_git_repo(giturl):
             return giturl
-        giturl = default(raw_input(msg), name)
+        giturl = default(input(msg), name)
 
 
 def get_verified_file(input_str, name, filename_can_be_null):
@@ -555,14 +555,14 @@ def get_verified_file(input_str, name, filename_can_be_null):
     """
     msg = input_str.strip() + " "
 
-    filename = default(raw_input(msg), name)
+    filename = default(input(msg), name)
 
     while True:
         if not filename and filename_can_be_null:
             return filename
         if os.path.isfile(filename):
             return filename
-        filename = default(raw_input(msg), name)
+        filename = default(input(msg), name)
 
 
 def replace_file(replace_this, with_this):
scripts/lib/bsp/kernel.py +6 -6
index 5bfa663..3c4b798 100644
@@ -202,8 +202,8 @@ def yocto_kernel_config_rm(scripts_path, machine):
     config_items = read_config_items(scripts_path, machine)
 
     print("Specify the kernel config items to remove:")
-    input = raw_input(gen_choices_str(config_items))
-    rm_choices = input.split()
+    inp = input(gen_choices_str(config_items))
+    rm_choices = inp.split()
     rm_choices.sort()
 
     removed = []
@@ -359,8 +359,8 @@ def yocto_kernel_patch_rm(scripts_path, machine):
     patches = read_patch_items(scripts_path, machine)
 
     print("Specify the patches to remove:")
-    input = raw_input(gen_choices_str(patches))
-    rm_choices = input.split()
+    inp = input(gen_choices_str(patches))
+    rm_choices = inp.split()
     rm_choices.sort()
 
     removed = []
@@ -610,8 +610,8 @@ def yocto_kernel_feature_rm(scripts_path, machine):
     features = read_features(scripts_path, machine)
 
     print("Specify the features to remove:")
-    input = raw_input(gen_choices_str(features))
-    rm_choices = input.split()
+    inp = input(gen_choices_str(features))
+    rm_choices = inp.split()
     rm_choices.sort()
 
     removed = []
scripts/oepydevshell-internal.py +1 -1
index f7b2e4e..2566606 100755
@@ -67,7 +67,7 @@ try:
                     i = i[4096:]
                 if sys.stdin in ready:
                     echonocbreak(sys.stdin.fileno())
-                    o = raw_input()
+                    o = input()
                     cbreaknoecho(sys.stdin.fileno())
                     pty.write(o + "\n")
             except (IOError, OSError) as e:
scripts/send-error-report +3 -3
index ed78bd6..7251e27 100755
@@ -44,12 +44,12 @@ def getPayloadLimit(url):
 
 def ask_for_contactdetails():
     print("Please enter your name and your email (optionally), they'll be saved in the file you send.")
-    username = raw_input("Name (required): ")
-    email = raw_input("E-mail (not required): ")
+    username = input("Name (required): ")
+    email = input("E-mail (not required): ")
     return username, email
 
 def edit_content(json_file_path):
-    edit = raw_input("Review information before sending? (y/n): ")
+    edit = input("Review information before sending? (y/n): ")
     if 'y' in edit or 'Y' in edit:
         editor = os.environ.get('EDITOR', None)
         if editor:
Signature · 1 line
2.1.4
author
#6 lore raw

PATCH 4/16 scripts: python3: fix urllib imports

In reply to Ed Bartosh

Some functions and classes have been moved from urllib[2] to urllib.request and urllib.error in python 3.

Used new imports to make the code working in python 3.

---
 scripts/contrib/verify-homepage.py |  4 ++--
 scripts/send-error-report          | 14 +++++++-------
 2 files changed, 9 insertions(+), 9 deletions(-)
scripts/contrib/verify-homepage.py +2 -2
index 265ff65..18bb15b 100755
@@ -7,7 +7,7 @@
 import sys
 import os
 import subprocess
-import urllib2
+import urllib.request
 
 
 # Allow importing scripts/lib modules
@@ -47,7 +47,7 @@ def verifyHomepage(bbhandler):
             homepage = data.getVar("HOMEPAGE", True)
             if homepage:
                 try:
-                    urllib2.urlopen(homepage, timeout=5)
+                    urllib.request.urlopen(homepage, timeout=5)
                 except Exception:
                     count = count + wgetHomepage(os.path.basename(realfn), homepage)
             checked.append(realfn)
scripts/send-error-report +7 -7
index 7251e27..122ce32 100755
@@ -7,7 +7,7 @@
 # Author: Andreea Proca <[email protected]>
 # Author: Michael Wood <[email protected]>
 
-import urllib2
+import urllib.request, urllib.error
 import sys
 import json
 import os
@@ -25,10 +25,10 @@ log = logging.getLogger("send-error-report")
 logging.basicConfig(format='%(levelname)s: %(message)s')
 
 def getPayloadLimit(url):
-    req = urllib2.Request(url, None)
+    req = urllib.request.Request(url, None)
     try:
-        response = urllib2.urlopen(req)
-    except urllib2.URLError as e:
+        response = urllib.request.urlopen(req)
+    except urllib.error.URLError as e:
         # Use this opportunity to bail out if we can't even contact the server
         log.error("Could not contact server: " + url)
         log.error(e.reason)
@@ -136,10 +136,10 @@ def send_data(data, args):
     else:
         url = "http://"+args.server+"/ClientPost/"
 
-    req = urllib2.Request(url, data=data, headers=headers)
+    req = urllib.request.Request(url, data=data, headers=headers)
     try:
-        response = urllib2.urlopen(req)
-    except urllib2.HTTPError, e:
+        response = urllib.request.urlopen(req)
+    except urllib.error.HTTPError as e:
         logging.error(e.reason)
         sys.exit(1)
 
Signature · 1 line
2.1.4
author
#7 lore raw

PATCH 5/16 ksize.py: python3: get rid of strings.join

In reply to Ed Bartosh

Used join method instead of strings.join as stings.join doesn't exist in python 3.

---
 scripts/tiny/ksize.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/tiny/ksize.py +1 -3
index 54b71f8..587c930 100755
@@ -28,8 +28,6 @@ import sys
 import getopt
 import os
 from subprocess import *
-from string import join
-
 
 def usage():
     prog = os.path.basename(sys.argv[0])
@@ -66,7 +64,7 @@ class Report:
 
         p = Popen("ls " + path + "/*.o | grep -v built-in.o",
                   shell=True, stdout=PIPE, stderr=PIPE)
-        glob = join(p.communicate()[0].splitlines())
+        glob = ' '.join(p.communicate()[0].splitlines())
         oreport = Report(glob, path + "/*.o")
         oreport.sizes.title = path + "/*.o"
         r.parts.append(oreport)
Signature · 1 line
2.1.4
author
#8 lore raw

PATCH 6/16 scripts: python3: get rid of __future__ imports

In reply to Ed Bartosh

Removed print_function and with_statement imports from __future__ as they're supported by python 3 by default.

---
 scripts/bitbake-whatchanged                      | 1 -
 scripts/pybootchartgui/pybootchartgui/main.py.in | 2 --
 scripts/pybootchartgui/pybootchartgui/parsing.py | 3 ---
 scripts/wic                                      | 2 --
 4 files changed, 8 deletions(-)
scripts/bitbake-whatchanged +0 -1
index b05aead..55b61d0 100755
@@ -17,7 +17,6 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 
-from __future__ import print_function
 import os
 import sys
 import getopt
scripts/pybootchartgui/pybootchartgui/main.py.in +0 -2
index 21bb0be..af26bd2 100644
@@ -16,8 +16,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with pybootchartgui. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import print_function
-
 import sys
 import os
 import optparse
scripts/pybootchartgui/pybootchartgui/parsing.py +0 -3
index d423b9f..a3a0b0b 100644
@@ -13,9 +13,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with pybootchartgui. If not, see <http://www.gnu.org/licenses/>.
 
-
-from __future__ import with_statement
-
 import os
 import string
 import re
scripts/wic +0 -2
index 5cc06f4..9023755 100755
@@ -28,8 +28,6 @@
 # AUTHORS
 # Tom Zanussi <tom.zanussi (at] linux.intel.com>
 #
-from __future__ import print_function
-
 __version__ = "0.2.0"
 
 # Python Standard Library modules
Signature · 1 line
2.1.4
author
#9 lore raw

PATCH 7/16 scripts: python3: use new metaclass syntax

In reply to Ed Bartosh

Used metaclass=<metaclass> syntax instead old __metaclass__ = <metaclass> as only new one is supported in python 3.

---
 scripts/lib/bsp/engine.py | 6 ++----
 scripts/test-remote-image | 8 ++------
 2 files changed, 4 insertions(+), 10 deletions(-)
scripts/lib/bsp/engine.py +2 -4
index f75454d..5b49486 100644
@@ -40,12 +40,11 @@ import json
 import subprocess
 import shutil
 
-class Line():
+class Line(metaclass=ABCMeta):
     """
     Generic (abstract) container representing a line that will appear
     in the BSP-generating program.
     """
-    __metaclass__ = ABCMeta
 
     def __init__(self, line):
         self.line = line
@@ -319,11 +318,10 @@ class BooleanInputLine(InputLine):
         return line
 
 
-class ListInputLine(InputLine):
+class ListInputLine(InputLine, metaclass=ABCMeta):
     """
     Base class for List-based Input lines. e.g. Choicelist, Checklist.
     """
-    __metaclass__ = ABCMeta
 
     def __init__(self, props, tag, lineno):
         InputLine.__init__(self, props, tag, lineno)
scripts/test-remote-image +2 -6
index 7a00db9..698948b 100755
@@ -92,13 +92,11 @@ def get_args_parser():
     parser.add_argument('--skip-download', required=False, action="store_true", dest="skip_download", default=False, help='Skip downloading the images completely. This needs the correct files to be present in the directory specified by the target profile.')
     return parser
 
-class BaseTargetProfile(object):
+class BaseTargetProfile(object, metaclass=ABCMeta):
     """
     This class defines the meta profile for a specific target (MACHINE type + image type).
     """
 
-    __metaclass__ = ABCMeta
-
     def __init__(self, image_type):
         self.image_type = image_type
 
@@ -191,13 +189,11 @@ class AutoTargetProfile(BaseTargetProfile):
         return controller.get_extra_files()
 
 
-class BaseRepoProfile(object):
+class BaseRepoProfile(object, metaclass=ABCMeta):
     """
     This class defines the meta profile for an images repository.
     """
 
-    __metaclass__ = ABCMeta
-
     def __init__(self, repolink, localdir):
         self.localdir = localdir
         self.repolink = repolink
Signature · 1 line
2.1.4
author
#10 lore raw

PATCH 8/16 combo-layer: python3: import reduce

In reply to Ed Bartosh

Reduce is not a builtin function in python3.
It has to be imported from functools.

---
 scripts/combo-layer | 1 +
 1 file changed, 1 insertion(+)
scripts/combo-layer +1 -0
index 0644cdc..0954bb6 100755
@@ -33,6 +33,7 @@ import pipes
 import shutil
 from collections import OrderedDict
 from string import Template
+from functools import reduce
 
 __version__ = "0.2.1"
 
Signature · 1 line
2.1.4
author
#11 lore raw

PATCH 9/16 engine.py: python3: rename sys.maxint to sys.maxsize

In reply to Ed Bartosh

Renamed sys.maxint -> sys.maxsize as sys.maxint doesn't exist in python 3.

---
 scripts/lib/bsp/engine.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
scripts/lib/bsp/engine.py +3 -3
index 5b49486..03bd66e 100644
@@ -49,7 +49,7 @@ class Line(metaclass=ABCMeta):
     def __init__(self, line):
         self.line = line
         self.generated_line = ""
-        self.prio = sys.maxint
+        self.prio = sys.maxsize
         self.discard = False
 
     @abstractmethod
@@ -154,7 +154,7 @@ class InputLine(Line):
         try:
             self.prio = int(props["prio"])
         except KeyError:
-            self.prio = sys.maxint
+            self.prio = sys.maxsize
 
     def gen(self, context = None):
         try:
@@ -1284,7 +1284,7 @@ class InputLineGroup(InputLine):
     def __init__(self, codeline):
         InputLine.__init__(self, {}, "", 0)
         self.group = []
-        self.prio = sys.maxint
+        self.prio = sys.maxsize
         self.group.append(codeline)
 
     def append(self, line):
Signature · 1 line
2.1.4
author
#12 lore raw

PATCH 10/16 scripts: python3: use explicit relative imports

In reply to Ed Bartosh

Implicit relative imports within packages are not supported in python 3. They have to be converted to explicit imports.

Used 'from .module import' syntax for relative imports.

---
 scripts/lib/bsp/engine.py | 2 +-
 scripts/lib/bsp/kernel.py | 3 +--
 2 files changed, 2 insertions(+), 3 deletions(-)
scripts/lib/bsp/engine.py +1 -1
index 03bd66e..5ff6c98 100644
@@ -34,7 +34,7 @@
 import os
 import sys
 from abc import ABCMeta, abstractmethod
-from tags import *
+from .tags import *
 import shlex
 import json
 import subprocess
scripts/lib/bsp/kernel.py +1 -2
index 3c4b798..fe7133d 100644
@@ -32,8 +32,7 @@ import shutil
 from tags import *
 import glob
 import subprocess
-from engine import create_context
-
+from .engine import create_context
 
 def find_bblayers():
     """
Signature · 1 line
2.1.4
author
#13 lore raw

PATCH 11/16 scripts: python3: replace exec statement with builtin

In reply to Ed Bartosh

Used exec() builtin instead of 'exec' statement as this statement doesn't exist in python 3.

---
 scripts/lib/bsp/engine.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/bsp/engine.py +1 -1
index 5ff6c98..2b87d33 100644
@@ -1362,7 +1362,7 @@ def run_program_lines(linelist, codedump):
         of = open("bspgen.out", "w")
         of.write(buf)
         of.close()
-    exec buf
+    exec(buf)
 
 
 def gen_target(files, context = None):
Signature · 1 line
2.1.4
author
#14 lore raw

PATCH 12/16 engine: python3: replace iteritems() -> items()

In reply to Ed Bartosh

Used items() as iteritems() doesn't exist in python 3.

---
 scripts/lib/bsp/engine.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
scripts/lib/bsp/engine.py +3 -3
index 2b87d33..3ed90fa 100644
@@ -1385,7 +1385,7 @@ def gen_supplied_property_vals(properties, program_lines):
     Generate user-specified entries for input values instead of
     generating input prompts.
     """
-    for name, val in properties.iteritems():
+    for name, val in properties.items():
         program_line = name + " = \"" + val + "\""
         program_lines.append(program_line)
 
@@ -1621,7 +1621,7 @@ def print_dict(items, indent = 0):
     """
     Print the values in a possibly nested dictionary.
     """
-    for key, val in items.iteritems():
+    for key, val in items.items():
         print("    "*indent + "\"%s\" :" % key)
         if type(val) == dict:
             print("{")
@@ -1654,7 +1654,7 @@ def get_properties(input_lines):
         props = line.props
         item = {}
         name = props["name"]
-        for key, val in props.items():
+        for key, val props.items():
             if not key == "name":
                 item[key] = val
         properties[name] = item
Signature · 1 line
2.1.4
author
#16 lore raw

PATCH 13/16 dirsize: python3: fix TypeError: unorderable types

In reply to Ed Bartosh

Python 3 ignores the __cmp__() method and doesn't have cmp() builtin function. This caused sorted() call to raise
TypeError: unorderable types: Record() < Record()

Removing __cmp__ method and implementing __lt__ should solve the problem as __lt__ is the only method needed for sort[ed] to work.

---
 scripts/tiny/dirsize.py | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)
scripts/tiny/dirsize.py +5 -9
index 5329b86..0b4fbd1 100755
@@ -52,20 +52,16 @@ class Record:
         self.size = 0
         self.records = []
 
-    def __cmp__(this, that):
+    def __lt__(this, that):
         if that is None:
-            return 1
+            return False
         if not isinstance(that, Record):
             raise TypeError
         if len(this.records) > 0 and len(that.records) == 0:
-            return -1
-        if len(this.records) == 0 and len(that.records) > 0:
-            return 1
-        if this.size < that.size:
-            return -1
+            return False
         if this.size > that.size:
-            return 1
-        return 0
+            return False
+        return True
 
     def show(self, minsize):
         total = 0
Signature · 1 line
2.1.4
author
#17 lore raw

PATCH 14/16 scripts: python3: decode subprocess output

In reply to Ed Bartosh

stdeout and stderr content returned by subprocess API has different types in Python 3(bytes) and Python 2(string). Decoding it to 'utf-8' makes it unicode on both pythons.

---
 scripts/cleanup-workdir   |  2 +-
 scripts/combo-layer       |  2 +-
 scripts/lib/bsp/kernel.py | 10 +++++-----
 scripts/oe-selftest       |  2 +-
 4 files changed, 8 insertions(+), 8 deletions(-)
scripts/cleanup-workdir +1 -1
index fee464c..0b2cf99 100755
@@ -45,7 +45,7 @@ def run_command(cmd):
     if pipe.returncode != 0:
         print("Execute command '%s' failed." % cmd)
         sys.exit(1)
-    return output
+    return output.decode('utf-8')
 
 def get_cur_arch_dirs(workdir, arch_dirs):
     pattern = workdir + '/(.*?)/'
scripts/combo-layer +1 -1
index 0954bb6..7f0fa48 100755
@@ -198,7 +198,7 @@ def runcmd(cmd,destdir=None,printerr=True,out=None,env=None):
         raise e
 
     err.seek(0)
-    output = err.read()
+    output = err.read().decode('utf-8')
     logger.debug("output: %s" % output.replace(chr(0), '\\0'))
     return output
 
scripts/lib/bsp/kernel.py +5 -5
index fe7133d..91cc79c 100644
@@ -49,7 +49,7 @@ def find_bblayers():
 
     bitbake_env_cmd = "bitbake -e"
     bitbake_env_lines = subprocess.Popen(bitbake_env_cmd, shell=True,
-                                         stdout=subprocess.PIPE).stdout.read()
+                                         stdout=subprocess.PIPE).stdout.read().decode('utf-8')
 
     if not bitbake_env_lines:
         print("Couldn't get '%s' output, exiting." % bitbake_env_cmd)
@@ -734,7 +734,7 @@ def yocto_kernel_available_features_list(scripts_path, machine):
     feature_url = find_feature_url(giturl)
 
     feature_cmd = "wget -q -O - " + feature_url
-    tmp = subprocess.Popen(feature_cmd, shell=True, stdout=subprocess.PIPE).stdout.read()
+    tmp = subprocess.Popen(feature_cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
 
     print("The current set of kernel features available to %s is:\n" % machine)
 
@@ -785,7 +785,7 @@ def get_feature_desc(git_url, feature):
     """
     feature_desc_url = find_feature_desc_url(git_url, feature)
     feature_desc_cmd = "wget -q -O - " + feature_desc_url
-    tmp = subprocess.Popen(feature_desc_cmd, shell=True, stdout=subprocess.PIPE).stdout.read()
+    tmp = subprocess.Popen(feature_desc_cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
 
     return find_feature_desc(tmp.split("\n"))
 
@@ -1001,7 +1001,7 @@ def base_branches(context):
     print("Getting branches from remote repo %s..." % giturl)
 
     gitcmd = "git ls-remote %s *heads* 2>&1" % (giturl)
-    tmp = subprocess.Popen(gitcmd, shell=True, stdout=subprocess.PIPE).stdout.read()
+    tmp = subprocess.Popen(gitcmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
 
     branches = []
 
@@ -1031,7 +1031,7 @@ def all_branches(context):
     print("Getting branches from remote repo %s..." % giturl)
 
     gitcmd = "git ls-remote %s *heads* 2>&1" % (giturl)
-    tmp = subprocess.Popen(gitcmd, shell=True, stdout=subprocess.PIPE).stdout.read()
+    tmp = subprocess.Popen(gitcmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
 
     branches = []
 
scripts/oe-selftest +1 -1
index b1ecf7f..df76f94 100755
@@ -371,7 +371,7 @@ def coverage_setup(coverage_source, coverage_include, coverage_omit):
     import subprocess
     builddir = os.environ.get("BUILDDIR")
     pokydir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
-    curcommit= subprocess.check_output(["git", "--git-dir", os.path.join(pokydir, ".git"), "rev-parse", "HEAD"])
+    curcommit= subprocess.check_output(["git", "--git-dir", os.path.join(pokydir, ".git"), "rev-parse", "HEAD"]).decode('utf-8')
     coveragerc = "%s/.coveragerc" % builddir
     data_file = "%s/.coverage." % builddir
     data_file += datetime.datetime.now().strftime('%Y%m%dT%H%M%S')
Signature · 1 line
2.1.4
author
#18 lore raw

PATCH 15/16 combo-layer: python3: use tempfile.TemporaryFile

In reply to Ed Bartosh

Used tempfile.TemporaryFile() API instead of deprecated os.tmpfile().

---
 scripts/combo-layer | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/combo-layer +2 -2
index 7f0fa48..eaa7f5a 100755
@@ -185,10 +185,10 @@ def runcmd(cmd,destdir=None,printerr=True,out=None,env=None):
     """
     logger.debug("run cmd '%s' in %s" % (cmd, os.getcwd() if destdir is None else destdir))
     if not out:
-        out = os.tmpfile()
+        out = tempfile.TemporaryFile()
         err = out
     else:
-        err = os.tmpfile()
+        err = tempfile.TemporaryFile()
     try:
         subprocess.check_call(cmd, stdout=out, stderr=err, cwd=destdir, shell=isinstance(cmd, str), env=env or os.environ)
     except subprocess.CalledProcessError as e:
Signature · 1 line
2.1.4
author
#19 lore raw

PATCH 16/16 scripts: python3: change python to python3 in shebang

In reply to Ed Bartosh
---
 scripts/bitbake-whatchanged                 | 2 +-
 scripts/buildhistory-collect-srcrevs        | 2 +-
 scripts/cleanup-workdir                     | 2 +-
 scripts/combo-layer                         | 2 +-
 scripts/contrib/bbvars.py                   | 2 +-
 scripts/contrib/devtool-stress.py           | 2 +-
 scripts/contrib/list-packageconfig-flags.py | 2 +-
 scripts/contrib/verify-homepage.py          | 2 +-
 scripts/cp-noerror                          | 2 +-
 scripts/gen-lockedsig-cache                 | 2 +-
 scripts/lib/devtool/__init__.py             | 2 +-
 scripts/oe-publish-sdk                      | 2 +-
 scripts/oe-trim-schemas                     | 2 +-
 scripts/oepydevshell-internal.py            | 2 +-
 scripts/opkg-query-helper.py                | 2 +-
 scripts/pythondeps                          | 2 +-
 scripts/relocate_sdk.py                     | 2 +-
 scripts/send-error-report                   | 2 +-
 scripts/swabber-strace-attach               | 2 +-
 scripts/sysroot-relativelinks.py            | 2 +-
 scripts/test-remote-image                   | 2 +-
 scripts/tiny/dirsize.py                     | 2 +-
 scripts/tiny/ksize.py                       | 2 +-
 scripts/yocto-bsp                           | 2 +-
 scripts/yocto-kernel                        | 2 +-
 scripts/yocto-layer                         | 2 +-
 26 files changed, 26 insertions(+), 26 deletions(-)
scripts/bitbake-whatchanged +1 -1
index 55b61d0..0207777 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # ex:ts=4:sw=4:sts=4:et
 # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 
scripts/buildhistory-collect-srcrevs +1 -1
index 79d2657..8a03580 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Collects the recorded SRCREV values from buildhistory and reports on them
 #
scripts/cleanup-workdir +1 -1
index 0b2cf99..86eae37 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Copyright (c) 2012 Wind River Systems, Inc.
 #
scripts/combo-layer +1 -1
index eaa7f5a..8f57ba5 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # ex:ts=4:sw=4:sts=4:et
 # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
scripts/contrib/bbvars.py +1 -1
index b865dd1..d8d0594 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
scripts/contrib/devtool-stress.py +1 -1
index 8cf92ca..2723491 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # devtool stress tester
 #
scripts/contrib/list-packageconfig-flags.py +1 -1
index 22d0c49..b8327e4 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
scripts/contrib/verify-homepage.py +1 -1
index 18bb15b..61a047c4 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # This script can be used to verify HOMEPAGE values for all recipes in
 # the current configuration.
scripts/cp-noerror +1 -1
index d8be677..35eb211 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Allow copying of $1 to $2 but if files in $1 disappear during the copy operation,
 # don't error.
scripts/gen-lockedsig-cache +1 -1
index 0986a21..26e9b63 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 import os
 import sys
scripts/lib/devtool/__init__.py +1 -1
index 7005363..77b1fd9 100644
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Development tool - utility functions for plugins
 #
scripts/oe-publish-sdk +1 -1
index 55872f2..4fe8974 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # OpenEmbedded SDK publishing tool
 
scripts/oe-trim-schemas +1 -1
index 29fb3a1..66a1b8d 100755
@@ -1,4 +1,4 @@
-#! /usr/bin/env python
+#! /usr/bin/env python3
 
 import sys
 try:
scripts/oepydevshell-internal.py +1 -1
index 2566606..7761f66 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 import os
 import sys
scripts/opkg-query-helper.py +1 -1
index 2fb1a78..ce89491 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # OpenEmbedded opkg query helper utility
 #
scripts/pythondeps +1 -1
index f1e6452..590b976 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Determine dependencies of python scripts or available python modules in a search path.
 #
scripts/relocate_sdk.py +1 -1
index 99fca86..e47b4d9 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Copyright (c) 2012 Intel Corporation
 #
scripts/send-error-report +1 -1
index 122ce32..ff23552 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Sends an error report (if the report-error class was enabled) to a
 # remote server.
scripts/swabber-strace-attach +1 -1
index f258987..e8f3258 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 import os
 import sys
 import subprocess
scripts/sysroot-relativelinks.py +1 -1
index ac26367..e44eba2 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 import sys
 import os
 
scripts/test-remote-image +1 -1
index 698948b..27b1cae 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Copyright (c) 2014 Intel Corporation
 #
scripts/tiny/dirsize.py +1 -1
index 0b4fbd1..ddccc5a 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Copyright (c) 2011, Intel Corporation.
 # All rights reserved.
scripts/tiny/ksize.py +1 -1
index 587c930..b9d2b19 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Copyright (c) 2011, Intel Corporation.
 # All rights reserved.
scripts/yocto-bsp +1 -1
index ce30e55..6a422c3 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # ex:ts=4:sw=4:sts=4:et
 # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
scripts/yocto-kernel +1 -1
index daaad07..fd97cb7 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # ex:ts=4:sw=4:sts=4:et
 # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
scripts/yocto-layer +1 -1
index 356972e..a5267f2 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # ex:ts=4:sw=4:sts=4:et
 # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
Signature · 1 line
2.1.4
#2 lore raw

Re: [PATCH 01/16] scripts: python3: convert iterables to lists

In reply to Ed Bartosh

Ed,

what is the reason for this change?

Quoted 151 lines · Ed Bartosh

Converted return value of items() keys() and values() to lists when dictionary is modified in the loop and when the result is added to the list.

---
  scripts/bitbake-whatchanged                      | 6 +++---
  scripts/combo-layer                              | 2 +-
  scripts/lib/devtool/build.py                     | 2 +-
  scripts/lib/devtool/standard.py                  | 4 ++--
  scripts/lib/recipetool/create_buildsys.py        | 2 +-
  scripts/lib/recipetool/create_buildsys_python.py | 6 +++---
  scripts/oe-pkgdata-util                          | 2 +-
  7 files changed, 12 insertions(+), 12 deletions(-)
scripts/bitbake-whatchanged +3 -3
index a20adb2..b05aead 100755
@@ -120,7 +120,7 @@ def print_added(d_new = None, d_old = None):
      Print the newly added tasks
      """
      added = {}
-    for k in d_new.keys():
+    for k in list(d_new.keys()):
          if k not in d_old:
              # Add the new one to added dict, and remove it from
              # d_new, so the remaining ones are the changed ones
@@ -155,7 +155,7 @@ def print_vrchanged(d_new = None, d_old = None, vr = None):
      """
      pvchanged = {}
      counter = 0
-    for k in d_new.keys():
+    for k in list(d_new.keys()):
          if d_new.get(k).get(vr) != d_old.get(k).get(vr):
              counter += 1
              pn, task = split_pntask(k)
@@ -279,7 +279,7 @@ Note:
 
          # Remove the same one from both stamps.
          cnt_unchanged = 0
-        for k in new_dict.keys():
+        for k in list(new_dict.keys()):
              if k in old_dict:
                  cnt_unchanged += 1
                  del(new_dict[k])
scripts/combo-layer +1 -1
index 234d9e4..7c41f92 100755
@@ -1186,7 +1186,7 @@ def update_with_history(conf, components, revisions, repos):
          msg = conf_commit_msg(conf, components)
          new_tree = runcmd("git write-tree", **wargs).strip()
          new_rev = runcmd("git commit-tree".split() +
-                         add_p([head] + additional_heads.keys()) +
+                         add_p([head] + list(additional_heads.keys())) +
                           ["-m", msg, new_tree],
                           **wargs).strip()
          # And done! This is the first time we change the HEAD in the actual work tree.
scripts/lib/devtool/build.py +1 -1
index 48f6fe1..6be549d 100644
@@ -27,7 +27,7 @@ logger = logging.getLogger('devtool')
 
 
  def _set_file_values(fn, values):
-    remaining = values.keys()
+    remaining = list(values.keys())
 
      def varfunc(varname, origvalue, op, newlines):
          newvalue = values.get(varname, origvalue)
scripts/lib/devtool/standard.py +2 -2
index 08153c6..a2516d6 100644
@@ -998,7 +998,7 @@ def _export_local_files(srctree, rd, destdir):
          bb.process.run(['git', 'checkout', tree, '--', '.'], cwd=srctree,
                          env=dict(os.environ, GIT_WORK_TREE=destdir,
                                   GIT_INDEX_FILE=tmp_index))
-        new_set = _git_ls_tree(srctree, tree, True).keys()
+        new_set = list(_git_ls_tree(srctree, tree, True).keys())
      elif os.path.isdir(local_files_dir):
          # If not tracked by Git, just copy from working copy
          new_set = _ls_tree(os.path.join(srctree, 'oe-local-files'))
@@ -1309,7 +1309,7 @@ def reset(args, config, basepath, workspace):
          raise DevtoolError("Recipe must be specified, or specify -a/--all to "
                             "reset all recipes")
      if args.all:
-        recipes = workspace.keys()
+        recipes = list(workspace.keys())
      else:
          recipes = [args.recipename]
 
scripts/lib/recipetool/create_buildsys.py +1 -1
index de3d9ae..78ae4bc 100644
@@ -682,7 +682,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
                  process_macro(in_keyword, partial)
 
          if extravalues:
-            for k,v in extravalues.items():
+            for k,v in list(extravalues.items()):
                  if v:
                      if v.startswith('$') or v.startswith('@') or v.startswith('%'):
                          del extravalues[k]
scripts/lib/recipetool/create_buildsys_python.py +3 -3
index 55cce0e..aff13cf 100644
@@ -361,7 +361,7 @@ class PythonRecipeHandler(RecipeHandler):
 
          # Naive mapping of setup() arguments to PKG-INFO field names
          for d in [info, non_literals]:
-            for key, value in d.items():
+            for key, value in list(d.items()):
                  new_key = _map(key)
                  if new_key != key:
                      del d[key]
@@ -443,7 +443,7 @@ class PythonRecipeHandler(RecipeHandler):
                  elif new_value != value:
                      info[variable] = new_value
              elif hasattr(value, 'items'):
-                for dkey, dvalue in value.items():
+                for dkey, dvalue in list(value.items()):
                      new_list = []
                      for pos, a_value in enumerate(dvalue):
                          new_value = replace_value(search, replace, a_value)
@@ -608,7 +608,7 @@ def gather_setup_info(fileobj):
      visitor.visit(parsed)
 
      non_literals, extensions = {}, []
-    for key, value in visitor.keywords.items():
+    for key, value in list(visitor.keywords.items()):
          if key == 'ext_modules':
              if isinstance(value, list):
                  for ext in value:
scripts/oe-pkgdata-util +1 -1
index b39d9b5..b16ecc9 100755
@@ -240,7 +240,7 @@ def lookup_pkg(args):
          sys.exit(1)
 
      if args.reverse:
-        items = mappings.values()
+        items = list(mappings.values())
      else:
          items = []
          for pkg in pkgs:
author
#3 lore raw

Re: [PATCH 01/16] scripts: python3: convert iterables to lists

Quoted 4 lines · Leonardo Sandoval

Ed,

what is the reason for this change?

The reason is that in python 3 dict.keys() returns iterator unlike in
pyton 2 where it returns list. If dictionary is changed inside the loop
(which is the case here) "RuntimeError: dictionary changed size during iteration" will be raised.

Another case is when result of dict.keys() is added to the list. This kind of operations would raise TypeError: unsupported operand type(s) for +: 'dict_keys' and 'list'

Quoted 145 lines · Ed Bartosh
Quoted 142 lines

Converted return value of items() keys() and values() to lists when dictionary is modified in the loop and when the result is added to the list.

---
 scripts/bitbake-whatchanged                      | 6 +++---
 scripts/combo-layer                              | 2 +-
 scripts/lib/devtool/build.py                     | 2 +-
 scripts/lib/devtool/standard.py                  | 4 ++--
 scripts/lib/recipetool/create_buildsys.py        | 2 +-
 scripts/lib/recipetool/create_buildsys_python.py | 6 +++---
 scripts/oe-pkgdata-util                          | 2 +-
 7 files changed, 12 insertions(+), 12 deletions(-)
scripts/bitbake-whatchanged +3 -3
index a20adb2..b05aead 100755
@@ -120,7 +120,7 @@ def print_added(d_new = None, d_old = None):
     Print the newly added tasks
     """
     added = {}
-    for k in d_new.keys():
+    for k in list(d_new.keys()):
         if k not in d_old:
             # Add the new one to added dict, and remove it from
             # d_new, so the remaining ones are the changed ones
@@ -155,7 +155,7 @@ def print_vrchanged(d_new = None, d_old = None, vr = None):
     """
     pvchanged = {}
     counter = 0
-    for k in d_new.keys():
+    for k in list(d_new.keys()):
         if d_new.get(k).get(vr) != d_old.get(k).get(vr):
             counter += 1
             pn, task = split_pntask(k)
@@ -279,7 +279,7 @@ Note:
         # Remove the same one from both stamps.
         cnt_unchanged = 0
-        for k in new_dict.keys():
+        for k in list(new_dict.keys()):
             if k in old_dict:
                 cnt_unchanged += 1
                 del(new_dict[k])
scripts/combo-layer +1 -1
index 234d9e4..7c41f92 100755
@@ -1186,7 +1186,7 @@ def update_with_history(conf, components, revisions, repos):
         msg = conf_commit_msg(conf, components)
         new_tree = runcmd("git write-tree", **wargs).strip()
         new_rev = runcmd("git commit-tree".split() +
-                         add_p([head] + additional_heads.keys()) +
+                         add_p([head] + list(additional_heads.keys())) +
                          ["-m", msg, new_tree],
                          **wargs).strip()
         # And done! This is the first time we change the HEAD in the actual work tree.
scripts/lib/devtool/build.py +1 -1
index 48f6fe1..6be549d 100644
@@ -27,7 +27,7 @@ logger = logging.getLogger('devtool')
 def _set_file_values(fn, values):
-    remaining = values.keys()
+    remaining = list(values.keys())
     def varfunc(varname, origvalue, op, newlines):
         newvalue = values.get(varname, origvalue)
scripts/lib/devtool/standard.py +2 -2
index 08153c6..a2516d6 100644
@@ -998,7 +998,7 @@ def _export_local_files(srctree, rd, destdir):
         bb.process.run(['git', 'checkout', tree, '--', '.'], cwd=srctree,
                         env=dict(os.environ, GIT_WORK_TREE=destdir,
                                  GIT_INDEX_FILE=tmp_index))
-        new_set = _git_ls_tree(srctree, tree, True).keys()
+        new_set = list(_git_ls_tree(srctree, tree, True).keys())
     elif os.path.isdir(local_files_dir):
         # If not tracked by Git, just copy from working copy
         new_set = _ls_tree(os.path.join(srctree, 'oe-local-files'))
@@ -1309,7 +1309,7 @@ def reset(args, config, basepath, workspace):
         raise DevtoolError("Recipe must be specified, or specify -a/--all to "
                            "reset all recipes")
     if args.all:
-        recipes = workspace.keys()
+        recipes = list(workspace.keys())
     else:
         recipes = [args.recipename]
scripts/lib/recipetool/create_buildsys.py +1 -1
index de3d9ae..78ae4bc 100644
@@ -682,7 +682,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
                 process_macro(in_keyword, partial)
         if extravalues:
-            for k,v in extravalues.items():
+            for k,v in list(extravalues.items()):
                 if v:
                     if v.startswith('$') or v.startswith('@') or v.startswith('%'):
                         del extravalues[k]
scripts/lib/recipetool/create_buildsys_python.py +3 -3
index 55cce0e..aff13cf 100644
@@ -361,7 +361,7 @@ class PythonRecipeHandler(RecipeHandler):
         # Naive mapping of setup() arguments to PKG-INFO field names
         for d in [info, non_literals]:
-            for key, value in d.items():
+            for key, value in list(d.items()):
                 new_key = _map(key)
                 if new_key != key:
                     del d[key]
@@ -443,7 +443,7 @@ class PythonRecipeHandler(RecipeHandler):
                 elif new_value != value:
                     info[variable] = new_value
             elif hasattr(value, 'items'):
-                for dkey, dvalue in value.items():
+                for dkey, dvalue in list(value.items()):
                     new_list = []
                     for pos, a_value in enumerate(dvalue):
                         new_value = replace_value(search, replace, a_value)
@@ -608,7 +608,7 @@ def gather_setup_info(fileobj):
     visitor.visit(parsed)
     non_literals, extensions = {}, []
-    for key, value in visitor.keywords.items():
+    for key, value in list(visitor.keywords.items()):
         if key == 'ext_modules':
             if isinstance(value, list):
                 for ext in value:
scripts/oe-pkgdata-util +1 -1
index b39d9b5..b16ecc9 100755
@@ -240,7 +240,7 @@ def lookup_pkg(args):
         sys.exit(1)
     if args.reverse:
-        items = mappings.values()
+        items = list(mappings.values())
     else:
         items = []
         for pkg in pkgs:
Signature · 3 lines
--
Regards,
Ed
#15 lore raw

Re: [PATCH 12/16] engine: python3: replace iteritems() -> items()

In reply to Ed Bartosh

This patch introduced a syntax error. Sent a patch to fix it http://lists.openembedded.org/pipermail/openembedded-core/2016-June/122522.html

Humberto

Quoted 56 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ed Bartosh
Sent: Thursday, June 2, 2016 5:13 AM
To: [email protected]
Subject: [OE-core] [PATCH 12/16] engine: python3: replace iteritems() -> items()

Used items() as iteritems() doesn't exist in python 3.

---
 scripts/lib/bsp/engine.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
scripts/lib/bsp/engine.py index renamed from scripts/lib/bsp/engine.py +0 -0

2b87d33..3ed90fa 100644

scripts/lib/bsp/engine.py +0 -0
@@ -1385,7 +1385,7 @@ def gen_supplied_property_vals(properties,

program_lines):
Generate user-specified entries for input values instead of generating input prompts.
"""

scripts/lib/bsp/engine.py +3 -4
-    for name, val in properties.iteritems():
+    for name, val in properties.items():
         program_line = name + " = \"" + val + "\""
         program_lines.append(program_line)
@@ -1621,7 +1621,7 @@ def print_dict(items, indent = 0):
     """
     Print the values in a possibly nested dictionary.
     """
-    for key, val in items.iteritems():
+    for key, val in items.items():
         print("    "*indent + "\"%s\" :" % key)
         if type(val) == dict:
             print("{")
@@ -1654,7 +1654,7 @@ def get_properties(input_lines):
         props = line.props
         item = {}
         name = props["name"]
-        for key, val in props.items():
+        for key, val props.items():
             if not key == "name":
                 item[key] = val
         properties[name] = item
--

2.1.4

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core