Python 3 support (the beginning)
cover Python 3 support (the beginning)
Hi,
This is first series of patches to make Toaster working in Python 3. The series should not break Python 2 support. The code should be able to work in both Python 2 and Python 3.
This patchset doesn't change build code as it uses bitbake from master. This means that Toaster is not able to run builds in python 3 environment yet. Build machinery will be addressed in the next series.
Test instructions:
1. fetch the branch and checkout to it: git fetch contrib ed/toaster/python3 && git checkout -b python3 contrib/ed/toaster/python3
2. Test if Toaster still works in current Python 2 setup.
3. Setup python 3 virtual environment: mkvirtualenv --python=/usr/bin/python3 toaster3
4. deactivate it as next step can't be done in python 3 environment: deactivate
5. initialize build environment: . oe-oe-init-build-env
6. switch to python3 virtual env: workon toaster3
7. move toaster database: mv toaster.sqlite toaster.sqlite.backup
7. run toaster: . ../bitbake/bin/toaster start
8. Check if toaster script starts properly, fetches the layers etc
9. Check if UI is accessible, that project can be created and basic functionality(excluding building)
is not broken.
The following changes since commit 0b717a9551d23579f5293e83655bc21e139bdaa2:
toaster: tests browser Add test for creating a project (2016-05-10 12:03:44 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/toaster/python3
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/python3
Ed Bartosh (14):
toaster: use force_text instead of force_bytes
toaster: use print function in toaster script
toaster: get rid of using reduce
toaster: fix imports to work for python 3
toaster: fix local imports
toaster: modified list of dependencies
toaster: use decode('utf-8') for binary data
toaster: use items and range instead of old APIs
toaster: replace map with list comprehensions
toaster: use 'in' instead of has_key
toaster: use re.sub() instead of translate()
toaster: use new syntax of except statement
toaster: read timezone files in binary mode
toaster: moved import bb.server.xmlrpc
Michael Wood (1):
toaster: Remove DATABASE_URL being passed around as an environment var
Richard Purdie (1):
bin, toaster: Fix print and exception syntax
bitbake/bin/bitbake-diffsigs | 6 +- bitbake/bin/bitbake-dumpsig | 4 +- bitbake/bin/toaster | 6 +- bitbake/lib/bb/ui/buildinfohelper.py | 20 +-- bitbake/lib/bb/utils.py | 6 +- bitbake/lib/toaster/bldcontrol/bbcontroller.py | 4 +- .../toaster/bldcontrol/localhostbecontroller.py | 4 +- .../management/commands/checksettings.py | 20 +-- .../bldcontrol/management/commands/loadconf.py | 4 +- .../bldcontrol/management/commands/runbuilds.py | 4 +- bitbake/lib/toaster/bldcontrol/models.py | 4 +- bitbake/lib/toaster/bldcontrol/tests.py | 4 +- bitbake/lib/toaster/contrib/tts/config.py | 2 +- .../contrib/tts/toasteruitest/run_toastertests.py | 4 +- .../tts/toasteruitest/toaster_automation_test.py | 186 ++++++++++----------- bitbake/lib/toaster/orm/models.py | 22 ++- bitbake/lib/toaster/toastergui/api.py | 3 +- bitbake/lib/toaster/toastergui/tablefilter.py | 2 +- bitbake/lib/toaster/toastergui/tables.py | 3 +- bitbake/lib/toaster/toastergui/views.py | 119 +++++++------ bitbake/lib/toaster/toastergui/widgets.py | 25 +-- .../toastermain/management/commands/builddelete.py | 2 +- .../toastermain/management/commands/buildslist.py | 2 +- .../toastermain/management/commands/get-dburl.py | 9 - .../toastermain/management/commands/perf.py | 6 +- bitbake/lib/toaster/toastermain/settings.py | 78 ++------- bitbake/lib/toaster/toastermain/urls.py | 2 +- bitbake/toaster-requirements.txt | 4 +- 28 files changed, 264 insertions(+), 291 deletions(-) delete mode 100644 bitbake/lib/toaster/toastermain/management/commands/get-dburl.py
--
Regards,
Ed
PATCH 1/16 toaster: Remove DATABASE_URL being passed around as an environment var
From: Michael Wood <[email protected]>
We don't need to pass the DATABASE_URL around and read it back if we setup the django framework in the correct way.
We make the default sqlite database path a full path so that the database isn't being assumed to be in CWD.
Also add some more useful comments on the database settings.
This is preparation work to migrate the build tests and be able to trigger builds on differently configured databases.
--- bitbake/bin/toaster | 2 - bitbake/lib/bb/ui/buildinfohelper.py | 20 +++---- .../toastermain/management/commands/get-dburl.py | 9 --- bitbake/lib/toaster/toastermain/settings.py | 67 ++++------------------ 4 files changed, 20 insertions(+), 78 deletions(-) delete mode 100644 bitbake/lib/toaster/toastermain/management/commands/get-dburl.py
bitbake/bin/toaster +0 -2
@@ -100,7 +100,6 @@ stop_system() fi webserverKillAll # unset exported variables - unset DATABASE_URL unset TOASTER_CONF unset TOASTER_DIR unset BITBAKE_UI
@@ -278,7 +277,6 @@ case $CMD in return 4 fi export BITBAKE_UI='toasterui' - export DATABASE_URL=`$MANAGE get-dburl` $MANAGE runbuilds & echo $! >${BUILDDIR}/.runbuilds.pid # set fail safe stop system on terminal exit trap stop_system SIGHUP
bitbake/lib/bb/ui/buildinfohelper.py +10 -10
@@ -21,19 +21,19 @@ import bb import re import os -os.environ["DJANGO_SETTINGS_MODULE"] = "toaster.toastermain.settings" - - import django from django.utils import timezone +import toaster +# Add toaster module to the search path to help django.setup() find the right +# modules +sys.path.insert(0, os.path.dirname(toaster.__file__)) -def _configure_toaster(): - """ Add toaster to sys path for importing modules - """ - sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'toaster')) -_configure_toaster() - +#Set the DJANGO_SETTINGS_MODULE if it's not already set +os.environ["DJANGO_SETTINGS_MODULE"] =\ + os.environ.get("DJANGO_SETTINGS_MODULE", + "toaster.toastermain.settings") +# Setup django framework (needs to be done before importing modules) django.setup() from orm.models import Build, Task, Recipe, Layer_Version, Layer, Target, LogMessage, HelpText
@@ -54,11 +54,11 @@ from datetime import datetime, timedelta from django.db import transaction, connection + # pylint: disable=invalid-name # the logger name is standard throughout BitBake logger = logging.getLogger("ToasterLogger") - class NotExisting(Exception): pass
bitbake/lib/toaster/toastermain/management/commands/get-dburl.py deleted +0 -9
@@ -1,9 +0,0 @@ -from toastermain.settings import getDATABASE_URL -from django.core.management.base import NoArgsCommand - -class Command(NoArgsCommand): - args = "" - help = "get database url" - - def handle_noargs(self,**options): - print getDATABASE_URL()
bitbake/lib/toaster/toastermain/settings.py +10 -57
89 lines not on this page · raw message
Signature · 1 line
2.1.4
PATCH 2/16 bin, toaster: Fix print and exception syntax
In reply to Ed BartoshFrom: Richard Purdie <[email protected]>
This updates the print "" syntax to print() and fixes some exception handling syntax such that its compatible with python v2 and v3.
--- bitbake/bin/bitbake-diffsigs | 6 +- bitbake/bin/bitbake-dumpsig | 4 +- .../management/commands/checksettings.py | 18 +- bitbake/lib/toaster/bldcontrol/tests.py | 4 +- .../contrib/tts/toasteruitest/run_toastertests.py | 4 +- .../tts/toasteruitest/toaster_automation_test.py | 186 ++++++++++----------- bitbake/lib/toaster/toastergui/views.py | 2 +- .../toastermain/management/commands/builddelete.py | 2 +- .../toastermain/management/commands/buildslist.py | 2 +- .../toastermain/management/commands/perf.py | 6 +- bitbake/lib/toaster/toastermain/settings.py | 2 +- 11 files changed, 118 insertions(+), 118 deletions(-)
bitbake/bin/bitbake-diffsigs +3 -3
@@ -95,7 +95,7 @@ def find_compare_task(bbhandler, pn, taskname): # Recurse into signature comparison output = bb.siggen.compare_sigfiles(latestfiles[0], latestfiles[1], recursecb) if output: - print '\n'.join(output) + print('\n'.join(output)) sys.exit(0)
@@ -130,9 +130,9 @@ else: except IOError as e: logger.error(str(e)) sys.exit(1) - except cPickle.UnpicklingError, EOFError: + except (pickle.UnpicklingError, EOFError): logger.error('Invalid signature data - ensure you are specifying sigdata/siginfo files') sys.exit(1) if output: - print '\n'.join(output) + print('\n'.join(output))
bitbake/bin/bitbake-dumpsig +2 -2
@@ -57,9 +57,9 @@ else: except IOError as e: logger.error(str(e)) sys.exit(1) - except cPickle.UnpicklingError, EOFError: + except (pickle.UnpicklingError, EOFError): logger.error('Invalid signature data - ensure you are specifying a sigdata/siginfo file') sys.exit(1) if output: - print '\n'.join(output) + print('\n'.join(output))
bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py +9 -9
@@ -70,11 +70,11 @@ class Command(NoArgsCommand): return True if len(be.sourcedir) == 0: - print "\n -- Validation: The layers checkout directory must be set." + print("\n -- Validation: The layers checkout directory must be set.") is_changed = _update_sourcedir() if not be.sourcedir.startswith("/"): - print "\n -- Validation: The layers checkout directory must be set to an absolute path." + print("\n -- Validation: The layers checkout directory must be set to an absolute path.") is_changed = _update_sourcedir() if is_changed:
@@ -87,16 +87,16 @@ class Command(NoArgsCommand): return True if len(be.builddir) == 0: - print "\n -- Validation: The build directory must be set." + print("\n -- Validation: The build directory must be set.") is_changed = _update_builddir() if not be.builddir.startswith("/"): - print "\n -- Validation: The build directory must to be set to an absolute path." + print("\n -- Validation: The build directory must to be set to an absolute path.") is_changed = _update_builddir() if is_changed: - print "\nBuild configuration saved" + print("\nBuild configuration saved") be.save() return True
@@ -104,20 +104,20 @@ class Command(NoArgsCommand): if be.needs_import: try: config_file = os.environ.get('TOASTER_CONF') - print "\nImporting file: %s" % config_file + print("\nImporting file: %s" % config_file) from loadconf import Command as LoadConfigCommand LoadConfigCommand()._import_layer_config(config_file) # we run lsupdates after config update - print "\nLayer configuration imported. Updating information from the layer sources, please wait.\nYou can re-update any time later by running bitbake/lib/toaster/manage.py lsupdates" + print("\nLayer configuration imported. Updating information from the layer sources, please wait.\nYou can re-update any time later by running bitbake/lib/toaster/manage.py lsupdates") from django.core.management import call_command call_command("lsupdates") # we don't look for any other config files return is_changed except Exception as e: - print "Failure while trying to import the toaster config file %s: %s" %\ - (config_file, e) + print("Failure while trying to import the toaster config file %s: %s" %\ + (config_file, e)) traceback.print_exc(e) return is_changed
bitbake/lib/toaster/bldcontrol/tests.py +2 -2
@@ -53,7 +53,7 @@ class BEControllerTests(object): # setting layers, skip any layer info bc.setLayers(BITBAKE_LAYER, POKY_LAYERS) except NotImplementedError: - print "Test skipped due to command not implemented yet" + print("Test skipped due to command not implemented yet") return True # We are ok with the exception as we're handling the git already exists except BuildSetupException:
@@ -79,7 +79,7 @@ class BEControllerTests(object): # setting layers, skip any layer info layerSet = bc.setLayers(BITBAKE_LAYER, POKY_LAYERS) except NotImplementedError: - print "Test skipped due to command not implemented yet" + print("Test skipped due to command not implemented yet") return True # We are ok with the exception as we're handling the git already exists except BuildSetupException:
bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py +2 -2
@@ -74,7 +74,7 @@ def get_tests_from_cfg(suite=None): try: tests_from_cfg = eval(config.get('toaster_test_' + target_suite, 'test_cases')) except: - print 'Failed to get test cases from cfg file. Make sure the format is correct.' + print('Failed to get test cases from cfg file. Make sure the format is correct.') return None prefix = 'toaster_automation_test.toaster_cases.test_'
@@ -100,7 +100,7 @@ def main(): testslist = get_tests_from_cfg() if not testslist: - print 'Failed to get test cases.' + print('Failed to get test cases.') exit(1) suite = unittest.TestSuite()
bitbake/lib/toaster/contrib/tts/toasteruitest/toaster_automation_test.py +93 -93
573 lines not on this page · raw message
bitbake/lib/toaster/toastergui/views.py +1 -1
@@ -910,7 +910,7 @@ def _get_dir_entries(build_id, target_id, start): response.append(entry) except Exception as e: - print "Exception ", e + print("Exception ", e) traceback.print_exc(e) # sort by directories first, then by name
bitbake/lib/toaster/toastermain/management/commands/builddelete.py +1 -1
@@ -15,7 +15,7 @@ class Command(BaseCommand): try: b = Build.objects.get(pk = bid) except ObjectDoesNotExist: - print 'build %s does not exist, skipping...' %(bid) + print('build %s does not exist, skipping...' %(bid)) continue # theoretically, just b.delete() would suffice # however SQLite runs into problems when you try to
bitbake/lib/toaster/toastermain/management/commands/buildslist.py +1 -1
@@ -10,4 +10,4 @@ class Command(NoArgsCommand): def handle_noargs(self,**options): for b in Build.objects.all(): - print "%d: %s %s %s" % (b.pk, b.machine, b.distro, ",".join([x.target for x in b.target_set.all()])) + print("%d: %s %s %s" % (b.pk, b.machine, b.distro, ",".join([x.target for x in b.target_set.all()])))
bitbake/lib/toaster/toastermain/management/commands/perf.py +3 -3
@@ -25,7 +25,7 @@ class Command(BaseCommand): info = self.url_info(full_url) status_code = info[0] load_time = info[1] - print 'Trying \'' + full_url + '\', ' + str(status_code) + ', ' + str(load_time) + print('Trying \'' + full_url + '\', ' + str(status_code) + ', ' + str(load_time)) def get_full_url(self, url_patt, url_root_res): full_url = str(url_patt).split('^')[1].replace('$>', '').replace('(?P<file_path>(?:/[', '/bin/busybox').replace('.*', '')
@@ -54,5 +54,5 @@ class Command(BaseCommand): def error(self, *args): for arg in args: - print >>sys.stderr, arg, - print >>sys.stderr + print(arg, end=' ', file=sys.stderr) + print(file=sys.stderr)
bitbake/lib/toaster/toastermain/settings.py +1 -1
@@ -91,7 +91,7 @@ else: try: if pytz.timezone(zonename) is not None: zonefilelist[hashlib.md5(open(filepath).read()).hexdigest()] = zonename - except UnknownTimeZoneError, ValueError: + except UnknownTimeZoneError as ValueError: # we expect timezone failures here, just move over pass except ImportError:
Signature · 1 line
2.1.4
PATCH 3/16 toaster: use force_text instead of force_bytes
In reply to Ed BartoshUsage of force_bytes in BuildRequest.__str__ method caused python 3 to throw "__str__ returned non-string (type bytes)" error.
Replaced force_bytes with force_text to make the code working on both python 2 and python 3.
[YOCTO #9584]
--- bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py | 4 ++-- bitbake/lib/toaster/bldcontrol/models.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py +2 -2
@@ -54,8 +54,8 @@ class Command(NoArgsCommand): logger.debug("runbuilds: No build env") return - logger.debug("runbuilds: starting build %s, environment %s" % \ - (str(br).decode('utf-8'), bec.be)) + logger.info("runbuilds: starting build %s, environment %s" % \ + (br, bec.be)) # let the build request know where it is being executed br.environment = bec.be
bitbake/lib/toaster/bldcontrol/models.py +2 -2
@@ -1,7 +1,7 @@ from __future__ import unicode_literals from django.db import models from django.core.validators import MaxValueValidator, MinValueValidator -from django.utils.encoding import force_bytes +from django.utils.encoding import force_text from orm.models import Project, ProjectLayer, ProjectVariable, ProjectTarget, Build, Layer_Version import logging
@@ -121,7 +121,7 @@ class BuildRequest(models.Model): return self.brvariable_set.get(name="MACHINE").value def __str__(self): - return force_bytes('%s %s' % (self.project, self.get_state_display())) + return force_text('%s %s' % (self.project, self.get_state_display())) # These tables specify the settings for running an actual build. # They MUST be kept in sync with the tables in orm.models.Project*
Signature · 1 line
2.1.4
PATCH 4/16 toaster: use print function in toaster script
In reply to Ed BartoshUsed print() function instead of print statement
to make toaster script to work with both python 2 and python 3
[YOCTO #9584]
--- bitbake/bin/toaster | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
bitbake/bin/toaster +2 -2
@@ -111,7 +111,7 @@ stop_system() verify_prereq() { # Verify Django version - reqfile=$(python -c "import os; print os.path.realpath('$BBBASEDIR/toaster-requirements.txt')") + reqfile=$(python -c "import os; print(os.path.realpath('$BBBASEDIR/toaster-requirements.txt'))") exp='s/Django\([><=]\+\)\([^,]\+\),\([><=]\+\)\(.\+\)/' exp=$exp'import sys,django;version=django.get_version().split(".");' exp=$exp'sys.exit(not (version \1 "\2".split(".") and version \3 "\4".split(".")))/p'
@@ -162,7 +162,7 @@ fi if [ "$TOASTER_CONF" = "" ]; then TOASTER_CONF="$TEMPLATECONF/toasterconf.json" - export TOASTER_CONF=$(python -c "import os; print os.path.realpath('$TOASTER_CONF')") + export TOASTER_CONF=$(python -c "import os; print(os.path.realpath('$TOASTER_CONF'))") fi if [ ! -f $TOASTER_CONF ]; then
Signature · 1 line
2.1.4
PATCH 6/16 toaster: fix imports to work for python 3
In reply to Ed BartoshSome APIs have been moved to other modules in python 3:
getstatusoutput: moved from commands to subproces
urlopen: moved from urllib2 to urllib.request
urlparse: moved from urlparse to urllib.parse
Made the imports work for both python versions by catching ImportError and importing APIs from different modules.
[YOCTO #9584]
--- bitbake/lib/bb/utils.py | 6 +++++- bitbake/lib/toaster/orm/models.py | 16 ++++++++++++---- 2 files changed, 17 insertions(+), 5 deletions(-)
bitbake/lib/bb/utils.py +5 -1
@@ -36,10 +36,14 @@ import traceback import errno import signal import ast -from commands import getstatusoutput from contextlib import contextmanager from ctypes import cdll +try: + from subprocess import getstatusoutput +except ImportError: + from commands import getstatusoutput + logger = logging.getLogger("BitBake.Util") python_extensions = [e for e, _, _ in imp.get_suffixes()]
bitbake/lib/toaster/orm/models.py +12 -4
@@ -1146,18 +1146,26 @@ class LayerIndexLayerSource(LayerSource): assert self.apiurl is not None from django.db import transaction, connection - import urllib2, urlparse, json + import json import os + + try: + from urllib.request import urlopen, URLError + from urllib.parse import urlparse + except ImportError: + from urllib2 import urlopen, URLError + from urlparse import urlparse + proxy_settings = os.environ.get("http_proxy", None) oe_core_layer = 'openembedded-core' def _get_json_response(apiurl = self.apiurl): - _parsedurl = urlparse.urlparse(apiurl) + _parsedurl = urlparse(apiurl) path = _parsedurl.path try: - res = urllib2.urlopen(apiurl) - except urllib2.URLError as e: + res = urlopen(apiurl) + except URLError as e: raise Exception("Failed to read %s: %s" % (path, e.reason)) return json.loads(res.read())
Signature · 1 line
2.1.4
PATCH 7/16 toaster: fix local imports
In reply to Ed BartoshReplaced local imports with absolute imports.
Used .module for local imports.
This should make the code to work on python 2 and python 3.
[YOCTO #9584]
--- bitbake/lib/toaster/bldcontrol/bbcontroller.py | 2 +- bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 2 +- bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py | 2 +- bitbake/lib/toaster/bldcontrol/management/commands/loadconf.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-)
bitbake/lib/toaster/bldcontrol/bbcontroller.py +1 -1
@@ -79,7 +79,7 @@ def getBuildEnvironmentController(**kwargs): The return object MUST always be a BuildEnvironmentController. """ - from localhostbecontroller import LocalhostBEController + from bldcontrol.localhostbecontroller import LocalhostBEController be = BuildEnvironment.objects.filter(Q(**kwargs))[0] if be.betype == BuildEnvironment.TYPE_LOCAL:
bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +1 -1
@@ -32,7 +32,7 @@ import subprocess from toastermain import settings -from bbcontroller import BuildEnvironmentController, ShellCmdException, BuildSetupException, BitbakeController +from bldcontrol.bbcontroller import BuildEnvironmentController, ShellCmdException, BuildSetupException, BitbakeController import logging logger = logging.getLogger("toaster")
bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py +1 -1
@@ -105,7 +105,7 @@ class Command(NoArgsCommand): try: config_file = os.environ.get('TOASTER_CONF') print("\nImporting file: %s" % config_file) - from loadconf import Command as LoadConfigCommand + from .loadconf import Command as LoadConfigCommand LoadConfigCommand()._import_layer_config(config_file) # we run lsupdates after config update
bitbake/lib/toaster/bldcontrol/management/commands/loadconf.py +1 -1
@@ -4,7 +4,7 @@ from orm.models import BitbakeVersion, Release, ReleaseDefaultLayer, ReleaseLaye from django.db import IntegrityError import os -from checksettings import DN +from .checksettings import DN import logging logger = logging.getLogger("toaster")
Signature · 1 line
2.1.4
PATCH 9/16 toaster: use decode('utf-8') for binary data
In reply to Ed BartoshDecoded binary data to utf-8 to fix the following error when runnign on python 3:
TypeError: Type str doesn't support the buffer API
[YOCTO #9584]
--- bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 2 +- bitbake/lib/toaster/bldcontrol/management/commands/loadconf.py | 2 +- bitbake/lib/toaster/orm/models.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +1 -1
@@ -70,7 +70,7 @@ class LocalhostBEController(BuildEnvironmentController): raise ShellCmdException(err) else: logger.debug("localhostbecontroller: shellcmd success") - return out + return out.decode('utf-8') def getGitCloneDirectory(self, url, branch): """Construct unique clone directory name out of url and branch."""
bitbake/lib/toaster/bldcontrol/management/commands/loadconf.py +1 -1
@@ -54,7 +54,7 @@ class Command(BaseCommand): (out,err) = cmd.communicate() if cmd.returncode != 0: logging.warning("Error while importing layer vcs_url: git error: %s" % err) - for line in out.split("\n"): + for line in out.decode('utf-8').split("\n"): try: (name, path) = line.split("\t", 1) if name == remote_name:
bitbake/lib/toaster/orm/models.py +1 -1
@@ -1168,7 +1168,7 @@ class LayerIndexLayerSource(LayerSource): except URLError as e: raise Exception("Failed to read %s: %s" % (path, e.reason)) - return json.loads(res.read()) + return json.loads(res.read().decode('utf-8')) # verify we can get the basic api try:
Signature · 1 line
2.1.4
PATCH 10/16 toaster: use items and range instead of old APIs
In reply to Ed BartoshUsed items() and range() APIs instead of iteritems() and xrange() as latter don't exist in python 3
[YOCTO #9584]
--- bitbake/lib/toaster/contrib/tts/config.py | 2 +- bitbake/lib/toaster/orm/models.py | 2 +- bitbake/lib/toaster/toastergui/tablefilter.py | 2 +- bitbake/lib/toaster/toastergui/views.py | 2 +- bitbake/lib/toaster/toastergui/widgets.py | 4 ++-- 5 files changed, 6 insertions(+), 6 deletions(-)
bitbake/lib/toaster/contrib/tts/config.py +1 -1
@@ -74,7 +74,7 @@ BACKLOGFILE = os.path.join(os.path.dirname(__file__), "backlog.txt") # task states def enum(*sequential, **named): enums = dict(zip(sequential, range(len(sequential))), **named) - reverse = dict((value, key) for key, value in enums.iteritems()) + reverse = dict((value, key) for key, value in enums.items()) enums['reverse_mapping'] = reverse return type('Enum', (), enums)
bitbake/lib/toaster/orm/models.py +1 -1
@@ -103,7 +103,7 @@ class GitURLValidator(validators.URLValidator): def GitURLField(**kwargs): r = models.URLField(**kwargs) - for i in xrange(len(r.validators)): + for i in range(len(r.validators)): if isinstance(r.validators[i], validators.URLValidator): r.validators[i] = GitURLValidator() return r
bitbake/lib/toaster/toastergui/tablefilter.py +1 -1
@@ -286,7 +286,7 @@ class TableFilterMap(object): def to_json(self, queryset): data = {} - for filter_name, table_filter in self.__filters.iteritems(): + for filter_name, table_filter in self.__filters.items(): data[filter_name] = table_filter.to_json() return data
bitbake/lib/toaster/toastergui/views.py +1 -1
@@ -2279,7 +2279,7 @@ if True: # Strip trailing/leading whitespace from all values # put into a new dict because POST one is immutable post_data = dict() - for key,val in request.POST.iteritems(): + for key,val in request.POST.items(): post_data[key] = val.strip()
bitbake/lib/toaster/toastergui/widgets.py +2 -2
@@ -281,12 +281,12 @@ class ToasterTable(TemplateView): # Make a unique cache name cache_name = self.__class__.__name__ - for key, val in request.GET.iteritems(): + for key, val in request.GET.items(): if key == 'nocache': continue cache_name = cache_name + str(key) + str(val) - for key, val in kwargs.iteritems(): + for key, val in kwargs.items(): cache_name = cache_name + str(key) + str(val) # No special chars allowed in the cache name apart from dash
Signature · 1 line
2.1.4
PATCH 13/16 toaster: use re.sub() instead of translate()
In reply to Ed Bartoshtranslate has different set of parameters in python 3 and can't be used the way it's used in toaster api module.
Replacing it with re.sub() should make the code work in both python 2 and python 3.
[YOCTO #9584]
--- bitbake/lib/toaster/toastergui/api.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
bitbake/lib/toaster/toastergui/api.py +2 -1
@@ -18,6 +18,7 @@ # Temporary home for the UI's misc API +import re from orm.models import Project, ProjectTarget, Build from bldcontrol.models import BuildRequest
@@ -92,7 +93,7 @@ class XhrBuildRequest(View): if 'targets' in request.POST: ProjectTarget.objects.filter(project = project).delete() s = str(request.POST['targets']) - for t in s.translate(None, ";%|\"").split(" "): + for t in re.sub(r'[;%|"]', '', s).split(" "): if ":" in t: target, task = t.split(":") else:
Signature · 1 line
2.1.4
PATCH 11/16 toaster: replace map with list comprehensions
In reply to Ed Bartoshmap returns map object in python 3. Replaced calls of map to equivalent list comprehensions in the code which requires lists.
[YOCTO #9584]
--- bitbake/lib/toaster/orm/models.py | 2 +- bitbake/lib/toaster/toastergui/tables.py | 3 +-- bitbake/lib/toaster/toastergui/views.py | 45 +++++++++++++------------------- 3 files changed, 20 insertions(+), 30 deletions(-)
bitbake/lib/toaster/orm/models.py +1 -1
@@ -1182,7 +1182,7 @@ class LayerIndexLayerSource(LayerSource): # update branches; only those that we already have names listed in the # Releases table - whitelist_branch_names = map(lambda x: x.branch_name, Release.objects.all()) + whitelist_branch_names = [rel.branch_name for rel in Release.objects.all()] if len(whitelist_branch_names) == 0: raise Exception("Failed to make list of branches to fetch")
bitbake/lib/toaster/toastergui/tables.py +1 -2
@@ -344,8 +344,7 @@ class RecipesTable(ToasterTable): context = super(RecipesTable, self).get_context_data(**kwargs) context['project'] = project - - context['projectlayers'] = map(lambda prjlayer: prjlayer.layercommit.id, ProjectLayer.objects.filter(project=context['project'])) + context['projectlayers'] = [player.layercommit.id for player in ProjectLayer.objects.filter(project=context['project'])] return context
bitbake/lib/toaster/toastergui/views.py +18 -27
86 lines not on this page · raw message
Signature · 1 line
2.1.4
PATCH 16/16 toaster: moved import bb.server.xmlrpc
In reply to Ed BartoshMoved import xmlrpc module to the place where it's used to avoid toaster crashes when importing bitbake code.
NOTE: This patch is made to be able to partly test toaster with bitbake from master. It can be removed as soon as bb.server.xmlrpc is ported to python 3.
--- bitbake/lib/toaster/bldcontrol/bbcontroller.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
bitbake/lib/toaster/bldcontrol/bbcontroller.py +1 -1
@@ -30,7 +30,6 @@ from bldcontrol.models import BuildEnvironment, BRLayer, BRVariable, BRTarget, B # load Bitbake components path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))) sys.path.insert(0, path) -import bb.server.xmlrpc class BitbakeController(object): """ This is the basic class that controlls a bitbake server.
@@ -38,6 +37,7 @@ class BitbakeController(object): """ def __init__(self, be): + import bb.server.xmlrpc self.connection = bb.server.xmlrpc._create_server(be.bbaddress, int(be.bbport))[0]
Signature · 1 line
2.1.4
PATCH 15/16 toaster: read timezone files in binary mode
In reply to Ed BartoshUsed 'rb' mode to open files to avoid unicode error when code runs on python 3:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80
in position 44: invalid start byte
[YOCTO #9584]
--- bitbake/lib/toaster/toastermain/settings.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
bitbake/lib/toaster/toastermain/settings.py +3 -4
@@ -87,17 +87,16 @@ else: try: import pytz from pytz.exceptions import UnknownTimeZoneError - pass try: if pytz.timezone(zonename) is not None: - zonefilelist[hashlib.md5(open(filepath).read()).hexdigest()] = zonename + zonefilelist[hashlib.md5(open(filepath, 'rb').read()).hexdigest()] = zonename except UnknownTimeZoneError as ValueError: # we expect timezone failures here, just move over pass except ImportError: - zonefilelist[hashlib.md5(open(filepath).read()).hexdigest()] = zonename + zonefilelist[hashlib.md5(open(filepath, 'rb').read()).hexdigest()] = zonename - TIME_ZONE = zonefilelist[hashlib.md5(open('/etc/localtime').read()).hexdigest()] + TIME_ZONE = zonefilelist[hashlib.md5(open('/etc/localtime', 'rb').read()).hexdigest()] # Language code for this installation. All choices can be found here: # http://www.i18nguy.com/unicode/language-identifiers.html
Signature · 1 line
2.1.4
PATCH 5/16 toaster: get rid of using reduce
In reply to Ed BartoshReplaced compicated calls of reduce with more clear code. As reduce was removed from python 3 this change is mandatory for the code to work on both pythons.
Here is an example change for illustration purposes:
original code:
querydict = dict(zip(or_keys, or_values))
query = reduce(operator.or_, map(lambda x: __get_q_for_val(x, querydict[x]), [k for k in querydict])))
replaced with:
query = None
for key, val in zip(or_keys, or_values):
x = __get_q_for_val(k, val)
query = query | x if query else x
[YOCTO #9584]
--- bitbake/lib/toaster/toastergui/views.py | 64 +++++++++++++++++++++---------- bitbake/lib/toaster/toastergui/widgets.py | 21 ++++++---- 2 files changed, 57 insertions(+), 28 deletions(-)
bitbake/lib/toaster/toastergui/views.py +44 -20
@@ -229,10 +229,18 @@ OR_VALUE_SEPARATOR = "|" DESCENDING = "-" def __get_q_for_val(name, value): - if "OR" in value: - return reduce(operator.or_, map(lambda x: __get_q_for_val(name, x), [ x for x in value.split("OR") ])) + if "OR" in value or "AND" in value: + result = None + for x in value.split("OR"): + x = __get_q_for_val(name, x) + result = result | x if result else x + return result if "AND" in value: - return reduce(operator.and_, map(lambda x: __get_q_for_val(name, x), [ x for x in value.split("AND") ])) + result = None + for x in value.split("AND"): + x = __get_q_for_val(name, x) + result = result & x if result else x + return result if value.startswith("NOT"): value = value[3:] if value == 'None':
@@ -251,14 +259,18 @@ def _get_filtering_query(filter_string): and_keys = search_terms[0].split(AND_VALUE_SEPARATOR) and_values = search_terms[1].split(AND_VALUE_SEPARATOR) - and_query = [] + and_query = None for kv in zip(and_keys, and_values): or_keys = kv[0].split(OR_VALUE_SEPARATOR) or_values = kv[1].split(OR_VALUE_SEPARATOR) - querydict = dict(zip(or_keys, or_values)) - and_query.append(reduce(operator.or_, map(lambda x: __get_q_for_val(x, querydict[x]), [k for k in querydict]))) + query = None + for key, val in zip(or_keys, or_values): + x = __get_q_for_val(k, val) + query = query | x if query else x - return reduce(operator.and_, [k for k in and_query]) + and_query = and_query & query if and_query else query + + return and_query def _get_toggle_order(request, orderkey, toggle_reverse = False): if toggle_reverse:
@@ -295,21 +307,24 @@ def _validate_input(field_input, model): # Check we are looking for a valid field valid_fields = model._meta.get_all_field_names() for field in field_input_list[0].split(AND_VALUE_SEPARATOR): - if not reduce(lambda x, y: x or y, [ field.startswith(x) for x in valid_fields ]): - return None, (field, [ x for x in valid_fields ]) + if True in [field.startswith(x) for x in valid_fields]: + break + else: + return None, (field, valid_fields) return field_input, invalid # uses search_allowed_fields in orm/models.py to create a search query # for these fields with the supplied input text def _get_search_results(search_term, queryset, model): - search_objects = [] + search_object = None for st in search_term.split(" "): - q_map = map(lambda x: Q(**{x+'__icontains': st}), - model.search_allowed_fields) + queries = None + for field in model.search_allowed_fields: + query = Q(**{x+'__icontains': st}) + queries = queries | query if queries else query - search_objects.append(reduce(operator.or_, q_map)) - search_object = reduce(operator.and_, search_objects) + search_object = search_object & queries if search_object else queries queryset = queryset.filter(search_object) return queryset
@@ -1938,10 +1953,10 @@ if True: if ptype == "build": mandatory_fields.append('projectversion') # make sure we have values for all mandatory_fields - if reduce( lambda x, y: x or y, map(lambda x: len(request.POST.get(x, '')) == 0, mandatory_fields)): - # set alert for missing fields - raise BadParameterException("Fields missing: " + - ", ".join([x for x in mandatory_fields if len(request.POST.get(x, '')) == 0 ])) + missing = [field for field in mandatory_fields if len(request.POST.get(field, '')) == 0] + if missing: + # set alert for missing fields + raise BadParameterException("Fields missing: %s" % ", ".join(missing)) if not request.user.is_authenticated(): user = authenticate(username = request.POST.get('username', '_anonuser'), password = 'nopass')
@@ -2035,13 +2050,22 @@ if True: from collections import Counter freqtargets = [] try: - freqtargets += map(lambda x: x.target, reduce(lambda x, y: x + y, map(lambda x: list(x.target_set.all()), Build.objects.filter(project = prj, outcome__lt = Build.IN_PROGRESS)))) - freqtargets += map(lambda x: x.target, reduce(lambda x, y: x + y, map(lambda x: list(x.brtarget_set.all()), BuildRequest.objects.filter(project = prj, state = BuildRequest.REQ_FAILED)))) + btargets = sum(build.target_set.all() for build in Build.objects.filter(project=prj, outcome__lt=Build.IN_PROGRESS)) + brtargets = sum(br.brtarget_set.all() for br in BuildRequest.objects.filter(project = prj, state = BuildRequest.REQ_FAILED)) + freqtargets = [x.target for x in btargets] + [x.target for x in brtargets] except TypeError: pass freqtargets = Counter(freqtargets) freqtargets = sorted(freqtargets, key = lambda x: freqtargets[x], reverse=True) + layers = [{"id": x.layercommit.pk, "orderid": x.pk, "name" : x.layercommit.layer.name, + "vcs_url": x.layercommit.layer.vcs_url, "vcs_reference" : x.layercommit.get_vcs_reference(), + "url": x.layercommit.layer.layer_index_url, "layerdetailurl": x.layercommit.get_detailspage_url(prj.pk), + # This branch name is actually the release + "branch" : {"name" : x.layercommit.get_vcs_reference(), + "layersource" : x.layercommit.up_branch.layer_source.name if x.layercommit.up_branch != None else None} + } for x in prj.projectlayer_set.all().order_by("id")] + context = { "project" : prj, "lvs_nos" : Layer_Version.objects.all().count(),
bitbake/lib/toaster/toastergui/widgets.py +13 -8
@@ -243,18 +243,23 @@ class ToasterTable(TemplateView): raise Exception("Search fields aren't defined in the model %s" % self.queryset.model) - search_queries = [] + search_queries = None for st in search_term.split(" "): - q_map = [Q(**{field + '__icontains': st}) - for field in self.queryset.model.search_allowed_fields] - - search_queries.append(reduce(operator.or_, q_map)) - - search_queries = reduce(operator.and_, search_queries) + queries = None + for field in self.queryset.model.search_allowed_fields: + query = Q(**{field + '__icontains': st}) + if queries: + queries |= query + else: + queries = query + + if search_queries: + search_queries &= queries + else: + search_queries = queries self.queryset = self.queryset.filter(search_queries) - def get_data(self, request, **kwargs): """ Returns the data for the page requested with the specified
Signature · 1 line
2.1.4
PATCH 8/16 toaster: modified list of dependencies
In reply to Ed BartoshRemoved dependency to argparse and wsgiref as they're in standard python library.
wsgiref crashes during installation with python 3 as it's not compatible with python 3.
Added dependency to pytz as it's used in the toaster code.
--- bitbake/toaster-requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
bitbake/toaster-requirements.txt +2 -2
@@ -1,4 +1,4 @@ Django>1.8,<1.9 -argparse==1.2.1 -wsgiref==0.1.2 beautifulsoup4>=4.4.0 +wsgiref==0.1.2 +pytz
Signature · 1 line
2.1.4
PATCH 12/16 toaster: use 'in' instead of has_key
In reply to Ed BartoshDictionary method has_key is deprecated in python 2 and absent in python 3.
Used '<key> in <dict>' statement to make the code working on both python 2 and python 3.
[YOCTO #9584]
--- bitbake/lib/toaster/toastermain/settings.py | 2 +- bitbake/lib/toaster/toastermain/urls.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
bitbake/lib/toaster/toastermain/settings.py +1 -1
@@ -274,7 +274,7 @@ currentdir = os.path.dirname(__file__) for t in os.walk(os.path.dirname(currentdir)): modulename = os.path.basename(t[0]) #if we have a virtualenv skip it to avoid incorrect imports - if os.environ.has_key('VIRTUAL_ENV') and os.environ['VIRTUAL_ENV'] in t[0]: + if 'VIRTUAL_ENV' in os.environ and os.environ['VIRTUAL_ENV'] in t[0]: continue if ("views.py" in t[2] or "models.py" in t[2]) and not modulename in INSTALLED_APPS:
bitbake/lib/toaster/toastermain/urls.py +1 -1
@@ -71,7 +71,7 @@ import os currentdir = os.path.dirname(__file__) for t in os.walk(os.path.dirname(currentdir)): #if we have a virtualenv skip it to avoid incorrect imports - if os.environ.has_key('VIRTUAL_ENV') and os.environ['VIRTUAL_ENV'] in t[0]: + if 'VIRTUAL_ENV' in os.environ and os.environ['VIRTUAL_ENV'] in t[0]: continue if "urls.py" in t[2] and t[0] != currentdir:
Signature · 1 line
2.1.4
PATCH 14/16 toaster: use new syntax of except statement
In reply to Ed BartoshUsed except 'except (<exception1>, <exception2>):' syntax as it's supported by python 2 and pythone 3.
Old syntax 'except <exception1>, <exception2>:' is not supported by python 3.
[YOCTO #9584]
--- bitbake/lib/toaster/toastergui/views.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
bitbake/lib/toaster/toastergui/views.py +3 -3
@@ -2902,7 +2902,7 @@ if True: else: context['dl_dir'] = ProjectVariable.objects.get(project = prj, name = "DL_DIR").value context['dl_dir_defined'] = "1" - except ProjectVariable.DoesNotExist,BuildEnvironment.DoesNotExist: + except (ProjectVariable.DoesNotExist, BuildEnvironment.DoesNotExist): pass try: context['fstypes'] = ProjectVariable.objects.get(project = prj, name = "IMAGE_FSTYPES").value
@@ -2930,7 +2930,7 @@ if True: else: context['sstate_dir'] = ProjectVariable.objects.get(project = prj, name = "SSTATE_DIR").value context['sstate_dir_defined'] = "1" - except ProjectVariable.DoesNotExist, BuildEnvironment.DoesNotExist: + except (ProjectVariable.DoesNotExist, BuildEnvironment.DoesNotExist): pass return context
@@ -2993,5 +2993,5 @@ if True: return response else: return render(request, "unavailable_artifact.html") - except ObjectDoesNotExist, IOError: + except (ObjectDoesNotExist, IOError): return render(request, "unavailable_artifact.html")
Signature · 1 line
2.1.4