Skip to contentSkip to conversation

toaster: upgrade to Django 1.8

PATCH New brian avery 1 participant 18 messages Dec 10, 2015

lore
Share & tools
lore
#0 lore raw

cover toaster: upgrade to Django 1.8

This rather large patchset upgrades toaster to work with Django 1.8. It was tested with Django 1.8.6 and 1.8.7.

yay for LTS!
-b

The following changes since commit 2cb81fb08395ec9cd68b02bc48391c9610eac9d9:

toasterui: process SetBRBE event (2015-12-09 19:45:13 -0800)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib bavery/submit/ed/toaster/django_upgrade-8364
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/submit/ed/toaster/django_upgrade-8364

Ed Bartosh (9):
toaster: Remove South migrations
toaster: Fix references to app paths
toaster: rename get_query_set -> get_queryset
toaster: use OneToOneField instead of ForeignKey
toaster: fixed format strings
toaster: work around 'database is locked' error
toaster: removed extra calls of migrate
toaster: monkey patch Queryset
toaster: rework checking of Django version

Elliot Smith (9):
toaster: Upgrade to Django 1.8.6 and remove South
toaster: Create default project with get_or_create* method
toaster: Start Django machinery for database access
toaster: Update API used to make runbuilds methods run in transactions
toaster: Prevent deprecation warnings for RedirectView
toaster: Update deprecated manage.py command
toaster: Check Django version against toaster-requirements.txt
toaster: Remove compatible_layerversions() method
toaster: Amend regex for MySQL database URLs

 bin/toaster                                        |  44 +-
 lib/bb/ui/buildinfohelper.py                       |  26 +-
 lib/toaster/bldcollector/urls.py                   |   1 -
 lib/toaster/bldcontrol/bbcontroller.py             |   4 +-
 lib/toaster/bldcontrol/localhostbecontroller.py    |  13 +-
 .../bldcontrol/management/commands/runbuilds.py    |   6 +-
 lib/toaster/bldcontrol/migrations/0001_initial.py  | 265 +++---
 ...onment_sourcedir__add_field_buildenvironment.py | 106 ---
 .../0003_auto__add_field_brlayer_dirpath.py        |  99 ---
 .../bldcontrol/migrations/0004_loadinitialdata.py  | 104 ---
 .../migrations/0005_auto__add_brerror.py           | 112 ---
 .../migrations/0006_auto__add_brbitbake.py         | 128 ---
 ...st_environment__chg_field_buildrequest_build.py | 145 ----
 .../bldcontrol/migrations/0008_brarchive.py        | 138 ---
 .../0009_auto__add_field_brlayer_layer_version.py  | 180 ----
 lib/toaster/bldcontrol/models.py                   |   2 +-
 lib/toaster/bldcontrol/tests.py                    |   6 +-
 lib/toaster/orm/migrations/0001_initial.py         | 928 ++++++++++++---------
 .../0002_auto__add_field_build_timespent.py        | 180 ----
 lib/toaster/orm/migrations/0003_timespent.py       | 182 ----
 .../0004_auto__add_field_package_installed_name.py | 181 ----
 ...e__add_target_file__add_field_variablehistor.py | 281 -------
 ...ge_size__add_field_target_license_manifest_p.py | 235 ------
 .../orm/migrations/0007_auto__add_helptext.py      | 214 -----
 ...story_operation__chg_field_recipe_descriptio.py | 225 -----
 ..._add_projectlayer__add_projecttarget__add_pr.py | 286 -------
 ...anch__add_field_project_short_description__a.py | 257 ------
 .../0011_auto__add_field_projectlayer_dirpath.py   | 242 ------
 ...layer_optional__add_field_projecttarget_task.py | 252 ------
 ...erversiondependency__add_unique_layerversion.py | 710 ----------------
 ...mmary__chg_field_layer_summary__chg_field_re.py | 336 --------
 ...web_url__add_field_layer_vcs_web_tree_base_u.py | 336 --------
 ...lptext__chg_field_release_branch__add_index_.py | 359 --------
 ...faultlayer__add_releaselayersourcepriority__.py | 396 ---------
 .../0018_auto__add_field_layer_version_project.py  | 331 --------
 .../orm/migrations/0019_auto__add_buildartifact.py | 342 --------
 ...ion_local_path__add_field_recipe_pathflags__.py | 361 --------
 ...ect__chg_field_project_bitbake_version__chg_.py | 371 --------
 ...k__add_field_layer_version_local_path__del_f.py | 343 --------
 ...ings_no__del_field_build_errors_no__del_fiel.py | 353 --------
 .../0024_auto__add_field_recipe_is_image.py        | 338 --------
 .../0025_auto__add_field_project_is_default.py     | 346 --------
 .../orm/migrations/0026_set_default_project.py     | 374 ---------
 ...e__add_unique_customimagerecipe_name_project.py | 375 ---------
 .../0028_auto__chg_field_logmessage_message.py     | 345 --------
 lib/toaster/orm/models.py                          | 108 ++-
 lib/toaster/toastergui/urls.py                     |   2 +-
 lib/toaster/toastergui/views.py                    |  24 +-
 lib/toaster/toastermain/settings.py                |   4 +-
 lib/toaster/toastermain/urls.py                    |   2 +-
 toaster-requirements.txt                           |   3 +-
 51 files changed, 779 insertions(+), 10222 deletions(-)
 delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto__add_field_buildenvironment_sourcedir__add_field_buildenvironment.py
 delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0003_auto__add_field_brlayer_dirpath.py
 delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0004_loadinitialdata.py
 delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0005_auto__add_brerror.py
 delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0006_auto__add_brbitbake.py
 delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0007_auto__add_field_buildrequest_environment__chg_field_buildrequest_build.py
 delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0008_brarchive.py
 delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0009_auto__add_field_brlayer_layer_version.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0002_auto__add_field_build_timespent.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0003_timespent.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0004_auto__add_field_package_installed_name.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0005_auto__add_target_image_file__add_target_file__add_field_variablehistor.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0006_auto__add_field_target_image_size__add_field_target_license_manifest_p.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0007_auto__add_helptext.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0008_auto__chg_field_variablehistory_operation__chg_field_recipe_descriptio.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0009_auto__add_projectvariable__add_projectlayer__add_projecttarget__add_pr.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0010_auto__add_field_project_branch__add_field_project_short_description__a.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0011_auto__add_field_projectlayer_dirpath.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0012_auto__add_field_projectlayer_optional__add_field_projecttarget_task.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0013_auto__add_release__add_layerversiondependency__add_unique_layerversion.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0014_auto__chg_field_package_summary__chg_field_layer_summary__chg_field_re.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0015_auto__add_field_layer_vcs_web_url__add_field_layer_vcs_web_tree_base_u.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0016_auto__add_field_release_helptext__chg_field_release_branch__add_index_.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0017_auto__del_toastersettingdefaultlayer__add_releaselayersourcepriority__.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0018_auto__add_field_layer_version_project.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0019_auto__add_buildartifact.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0020_auto__add_field_layer_version_local_path__add_field_recipe_pathflags__.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0021_auto__chg_field_build_project__chg_field_project_bitbake_version__chg_.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0022_auto__add_field_target_task__add_field_layer_version_local_path__del_f.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0023_auto__del_field_build_warnings_no__del_field_build_errors_no__del_fiel.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0024_auto__add_field_recipe_is_image.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0025_auto__add_field_project_is_default.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0026_set_default_project.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0027_auto__add_customimagerecipe__add_unique_customimagerecipe_name_project.py
 delete mode 100644 bitbake/lib/toaster/orm/migrations/0028_auto__chg_field_logmessage_message.py

--
1.9.1

#1 lore raw

PATCH 1/18 toaster: Upgrade to Django 1.8.6 and remove South

From: Elliot Smith <[email protected]>

Upgrade Django to long-term support version.

Django now provides its own migration framework, so remove requirement for South.

[YOCTO #8364]

---
 bin/toaster                         | 8 ++------
 lib/toaster/toastermain/settings.py | 1 -
 toaster-requirements.txt            | 3 +--
 3 files changed, 3 insertions(+), 9 deletions(-)
bin/toaster +2 -6
index 6d1ec05..528e17e 100755
@@ -176,15 +176,11 @@ notify_chldexit() {
 verify_prereq() {
     # Verify prerequisites
 
-    if ! echo "import django; print (1,) == django.VERSION[0:1] and django.VERSION[1:2][0] in (6,)" | python 2>/dev/null | grep True >/dev/null; then
-        printf "This program needs Django 1.6. Please install with\n\npip install django==1.6\n"
+    if ! echo "import django; print (1,) == django.VERSION[0:1] and django.VERSION[1:2][0] in (8,)" | python 2>/dev/null | grep True >/dev/null; then
+        printf "This program needs Django 1.8. Please install with\n\npip install django<=1.8.7\n"
         return 2
     fi
 
-    if ! echo "import south; print reduce(lambda x, y: 2 if x==2 else 0 if x == 0 else y, map(lambda x: 1+cmp(x[1]-x[0],0), zip([0,8,4], map(int,south.__version__.split(\".\"))))) > 0" | python 2>/dev/null | grep True >/dev/null; then
-        printf "This program needs South 0.8.4. Please install with\n\npip install south==0.8.4\n"
-        return 2
-    fi
     return 0
 }
 
lib/toaster/toastermain/settings.py +0 -1
index de686b2..7c559bc 100644
@@ -276,7 +276,6 @@ INSTALLED_APPS = (
     'django.contrib.humanize',
     'bldcollector',
     'toastermain',
-    'south',
 )
 
 
toaster-requirements.txt +1 -2
index 1d7d21b..5d34494 100644
@@ -1,5 +1,4 @@
-Django==1.6
-South==0.8.4
+Django==1.8.6
 argparse==1.2.1
 wsgiref==0.1.2
 beautifulsoup4>=4.4.0
Signature · 1 line
1.9.1
#2 lore raw

PATCH 3/18 toaster: Fix references to app paths

From: Ed Bartosh <[email protected]>

The buildinfohelper runs outside of Django, but still needs access to our Django app classes.

Previously, the imports referenced the toaster.* app, which worked fine. But in Django 1.8, this causes an error about the same module being loaded multiple times from different paths.

Change the paths to our Django modules so they don't cause this error to be thrown. We can do this as we've added our application libraries to sys.path in the buildinfohelper anyway.

[YOCTO #8364]

---
 lib/bb/ui/buildinfohelper.py | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
lib/bb/ui/buildinfohelper.py +8 -8
index 1d41bba..c099551 100644
@@ -33,14 +33,14 @@ def _configure_toaster():
     sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'toaster'))
 _configure_toaster()
 
-from toaster.orm.models import Build, Task, Recipe, Layer_Version, Layer, Target, LogMessage, HelpText
-from toaster.orm.models import Target_Image_File, BuildArtifact
-from toaster.orm.models import Variable, VariableHistory
-from toaster.orm.models import Package, Package_File, Target_Installed_Package, Target_File
-from toaster.orm.models import Task_Dependency, Package_Dependency
-from toaster.orm.models import Recipe_Dependency
-
-from toaster.orm.models import Project
+from orm.models import Build, Task, Recipe, Layer_Version, Layer, Target, LogMessage, HelpText
+from orm.models import Target_Image_File, BuildArtifact
+from orm.models import Variable, VariableHistory
+from orm.models import Package, Package_File, Target_Installed_Package, Target_File
+from orm.models import Task_Dependency, Package_Dependency
+from orm.models import Recipe_Dependency
+
+from orm.models import Project
 from bldcontrol.models import BuildEnvironment, BuildRequest
 
 from bb.msg import BBLogFormatter as formatter
Signature · 1 line
1.9.1
#3 lore raw

PATCH 4/18 toaster: Create default project with get_or_create* method

In reply to brian avery

From: Elliot Smith <[email protected]>

Rather than maintain data as part of the migrations (as was done for the default project previously), create the default (cli builds) project on demand as a by-product of getting it from the database.

[YOCTO #8364]

---
 lib/bb/ui/buildinfohelper.py    |  9 +++++++--
 lib/toaster/orm/models.py       | 23 +++++++++++++++--------
 lib/toaster/toastergui/views.py |  2 +-
 3 files changed, 23 insertions(+), 11 deletions(-)
lib/bb/ui/buildinfohelper.py +7 -2
index c099551..87e03e7 100644
@@ -146,7 +146,7 @@ class ORMWrapper(object):
             prj = Project.objects.get(pk = project_id)
 
         else:                           # this build was triggered by a legacy system, or command line interactive mode
-            prj = Project.objects.get_default_project()
+            prj = Project.objects.get_or_create_default_project()
             logger.debug(1, "buildinfohelper: project is not specified, defaulting to %s" % prj)
 
 
@@ -308,6 +308,11 @@ class ORMWrapper(object):
         # then we are wholly responsible for the data
         # and therefore we return the 'real' recipe rather than the build
         # history copy of the recipe.
+        if  recipe_information['layer_version'].build is not None and \
+            recipe_information['layer_version'].build.project == \
+                Project.objects.get_or_create_default_project():
+            return recipe
+
         if built_recipe is None:
             return recipe
 
@@ -345,7 +350,7 @@ class ORMWrapper(object):
         # If we're doing a command line build then associate this new layer with the
         # project to avoid it 'contaminating' toaster data
         project = None
-        if build_obj.project == Project.objects.get_default_project():
+        if build_obj.project == Project.objects.get_or_create_default_project():
             project = build_obj.project
 
         layer_version_object, _ = Layer_Version.objects.get_or_create(
lib/toaster/orm/models.py +15 -8
index 0174233..4a868e7 100644
@@ -91,18 +91,25 @@ class ProjectManager(models.Manager):
 
         return prj
 
-    def create(self, *args, **kwargs):
-        raise Exception("Invalid call to Project.objects.create. Use Project.objects.create_project() to create a project")
-
     # return single object with is_default = True
-    def get_default_project(self):
+    def get_or_create_default_project(self):
         projects = super(ProjectManager, self).filter(is_default = True)
+
         if len(projects) > 1:
-            raise Exception("Inconsistent project data: multiple " +
-                            "default projects (i.e. with is_default=True)")
+            raise Exception('Inconsistent project data: multiple ' +
+                            'default projects (i.e. with is_default=True)')
         elif len(projects) < 1:
-            raise Exception("Inconsistent project data: no default project found")
-        return projects[0]
+            options = {
+                'name': 'Command line builds',
+                'short_description': 'Project for builds started outside Toaster',
+                'is_default': True
+            }
+            project = Project.objects.create(**options)
+            project.save()
+
+            return project
+        else:
+            return projects[0]
 
 class Project(models.Model):
     search_allowed_fields = ['name', 'short_description', 'release__name', 'release__branch_name']
lib/toaster/toastergui/views.py +1 -1
index 243bb09..16f27b8 100755
@@ -73,7 +73,7 @@ class MimeTypeFinder(object):
 def landing(request):
     # in build mode, we redirect to the command-line builds page
     # if there are any builds for the default (cli builds) project
-    default_project = Project.objects.get_default_project()
+    default_project = Project.objects.get_or_create_default_project()
     default_project_builds = Build.objects.filter(project = default_project)
 
     # we only redirect to projects page if there is a user-generated project
Signature · 1 line
1.9.1
#4 lore raw

PATCH 5/18 toaster: Start Django machinery for database access

In reply to brian avery

From: Elliot Smith <[email protected]>

buildinfohelper, with the new import paths for our Django models and Django 1.8, was not getting an active connection to the database.

In buildinfohelper, call django.setup() explicitly to make sure that the database connection is ready and models can be queried and saved.

[YOCTO #8364]

---
 lib/bb/ui/buildinfohelper.py | 3 +++
 1 file changed, 3 insertions(+)
lib/bb/ui/buildinfohelper.py +3 -0
index 87e03e7..04ce23a 100644
@@ -24,6 +24,7 @@ import os
 os.environ["DJANGO_SETTINGS_MODULE"] = "toaster.toastermain.settings"
 
 
+import django
 from django.utils import timezone
 
 
@@ -33,6 +34,8 @@ def _configure_toaster():
     sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'toaster'))
 _configure_toaster()
 
+django.setup()
+
 from orm.models import Build, Task, Recipe, Layer_Version, Layer, Target, LogMessage, HelpText
 from orm.models import Target_Image_File, BuildArtifact
 from orm.models import Variable, VariableHistory
Signature · 1 line
1.9.1
#5 lore raw

PATCH 6/18 toaster: rename get_query_set -> get_queryset

In reply to brian avery

From: Ed Bartosh <[email protected]>

Renamed get_query_set to get_queryset as get_queryset causes deprecation warning in Django 1.7 and is deprecated in Django 1.8

---
 lib/toaster/orm/models.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
lib/toaster/orm/models.py +4 -4
index 4a868e7..b25910a 100644
@@ -598,8 +598,8 @@ class Package(models.Model):
 class Package_DependencyManager(models.Manager):
     use_for_related_fields = True
 
-    def get_query_set(self):
-        return super(Package_DependencyManager, self).get_query_set().exclude(package_id = F('depends_on__id'))
+    def get_queryset(self):
+        return super(Package_DependencyManager, self).get_queryset().exclude(package_id = F('depends_on__id'))
 
 class Package_Dependency(models.Model):
     TYPE_RDEPENDS = 0
@@ -699,8 +699,8 @@ class Recipe(models.Model):
 class Recipe_DependencyManager(models.Manager):
     use_for_related_fields = True
 
-    def get_query_set(self):
-        return super(Recipe_DependencyManager, self).get_query_set().exclude(recipe_id = F('depends_on__id'))
+    def get_queryset(self):
+        return super(Recipe_DependencyManager, self).get_queryset().exclude(recipe_id = F('depends_on__id'))
 
 class Recipe_Dependency(models.Model):
     TYPE_DEPENDS = 0
Signature · 1 line
1.9.1
#6 lore raw

PATCH 7/18 toaster: Update API used to make runbuilds methods run in transactions

In reply to brian avery

From: Elliot Smith <[email protected]>

runbuilds previously had its methods decorated with transaction.commit_on_success, which is now deprecated. transaction.atomic is an (almost) drop-in replacement for this, so use this instead.

[YOCTO #8364]

---
 lib/toaster/bldcontrol/management/commands/runbuilds.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
lib/toaster/bldcontrol/management/commands/runbuilds.py +2 -2
index 48dc618..d40dedb 100644
@@ -14,14 +14,14 @@ class Command(NoArgsCommand):
     help    = "Schedules and executes build requests as possible. Does not return (interrupt with Ctrl-C)"
 
 
-    @transaction.commit_on_success
+    @transaction.atomic
     def _selectBuildEnvironment(self):
         bec = getBuildEnvironmentController(lock = BuildEnvironment.LOCK_FREE)
         bec.be.lock = BuildEnvironment.LOCK_LOCK
         bec.be.save()
         return bec
 
-    @transaction.commit_on_success
+    @transaction.atomic
     def _selectBuildRequest(self):
         br = BuildRequest.objects.filter(state = BuildRequest.REQ_QUEUED).order_by('pk')[0]
         br.state = BuildRequest.REQ_INPROGRESS
Signature · 1 line
1.9.1
#7 lore raw

PATCH 8/18 toaster: Prevent deprecation warnings for RedirectView

In reply to brian avery

From: Elliot Smith <[email protected]>

The API for RedirectView is due to change in Django 1.9, which means that Toaster generates deprecation warnings.

Set the "permanent" flag when constructing RedirectView instances to prevent this warning.

[YOCTO #8364]

---
 lib/toaster/bldcollector/urls.py | 1 -
 lib/toaster/toastergui/urls.py   | 2 +-
 lib/toaster/toastermain/urls.py  | 2 +-
 3 files changed, 2 insertions(+), 3 deletions(-)
lib/toaster/bldcollector/urls.py +0 -1
index 144387b..64722f2 100644
@@ -18,7 +18,6 @@
 
 
 from django.conf.urls import patterns, include, url
-from django.views.generic import RedirectView
 
 urlpatterns = patterns('bldcollector.views',
         # landing point for pushing a bitbake_eventlog.json file to this toaster instace
lib/toaster/toastergui/urls.py +1 -1
index a1adbb7..2bf2d99 100644
@@ -168,5 +168,5 @@ urlpatterns = patterns('toastergui.views',
             name='xhr_customrecipe'),
 
           # default redirection
-        url(r'^$', RedirectView.as_view( url= 'landing')),
+        url(r'^$', RedirectView.as_view(url='landing', permanent=True)),
 )
lib/toaster/toastermain/urls.py +1 -1
index 5ce5a92..534679d 100644
@@ -45,7 +45,7 @@ urlpatterns = patterns('',
     url(r'^orm/eventfile$', 'bldcollector.views.eventfile'),
 
     # if no application is selected, we have the magic toastergui app here
-    url(r'^$', never_cache(RedirectView.as_view(url='/toastergui/'))),
+    url(r'^$', never_cache(RedirectView.as_view(url='/toastergui/', permanent=True))),
 )
 
 import toastermain.settings
Signature · 1 line
1.9.1
#8 lore raw

PATCH 9/18 toaster: Update deprecated manage.py command

In reply to brian avery

From: Elliot Smith <[email protected]>

syncdb is deprecated in favour of migrate in Django 1.7:
https://docs.djangoproject.com/en/1.8/releases/1.7/#schema-migrations

Update to the "migrate" command in Toaster's start script.

[YOCTO #8364]

---
 bin/toaster | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bin/toaster +1 -1
index 528e17e..e3c7867 100755
@@ -56,7 +56,7 @@ webserverStartAll()
     retval=0
     # you can always add a superuser later via
     # python bitbake/lib/toaster/manage.py python manage.py createsuperuser --username=<ME>
-    python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1
+    python $BBBASEDIR/lib/toaster/manage.py migrate --noinput || retval=1
 
     python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2
 
Signature · 1 line
1.9.1
#9 lore raw

PATCH 10/18 toaster: Check Django version against toaster-requirements.txt

In reply to brian avery

From: Elliot Smith <[email protected]>

Instead of hard-coding the required Django version in the start script, look it up from the toaster-requirements.txt file.

[YOCTO #8364]

---
 bin/toaster | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
bin/toaster +8 -2
index e3c7867..0862a35 100755
@@ -175,9 +175,15 @@ notify_chldexit() {
 
 verify_prereq() {
     # Verify prerequisites
+    regex="([0-9])\.([0-9])"
+    for line in `cat $BBBASEDIR/toaster-requirements.txt | grep Django` ; do
+        [[ `echo $line` =~ $regex ]]
+        required_major_version="${BASH_REMATCH[1]}"
+        required_minor_version="${BASH_REMATCH[2]}"
+    done
 
-    if ! echo "import django; print (1,) == django.VERSION[0:1] and django.VERSION[1:2][0] in (8,)" | python 2>/dev/null | grep True >/dev/null; then
-        printf "This program needs Django 1.8. Please install with\n\npip install django<=1.8.7\n"
+    if ! echo "import django; print ($required_major_version,) == django.VERSION[0:1] and django.VERSION[1:2][0] in ($required_minor_version,)" | python 2>/dev/null | grep True >/dev/null; then
+        printf "This program needs Django $required_major_version.$required_minor_version. Please install with\n\npip install -r $BBBASEDIR/toaster-requirements.txt\n"
         return 2
     fi
 
Signature · 1 line
1.9.1
#10 lore raw

PATCH 11/18 toaster: Remove compatible_layerversions() method

In reply to brian avery

From: Elliot Smith <[email protected]>

Remove compatible_layerversions() and replace calls to it with calls to get_all_compatible_layer_versions().

The sorting done in compatible_layerversions() is not relevant, and the code can be simplified by using get_all_compatible_layer_versions().

[YOCTO #8364]

---
 lib/toaster/orm/models.py       | 21 +++------------------
 lib/toaster/toastergui/views.py | 22 ++++++++++++++--------
 2 files changed, 17 insertions(+), 26 deletions(-)
lib/toaster/orm/models.py +3 -18
index b25910a..f07feda 100644
@@ -195,23 +195,6 @@ class Project(models.Model):
         except (Variable.DoesNotExist,IndexError):
             return( "not_found" )
 
-    # returns a queryset of compatible layers for a project
-    def compatible_layerversions(self, release = None, layer_name = None):
-        logger.warning("This function is deprecated")
-        if release == None:
-            release = self.release
-        # layers on the same branch or layers specifically set for this project
-        queryset = Layer_Version.objects.filter(((Q(up_branch__name = release.branch_name) & Q(project = None)) | Q(project = self)) & Q(build__isnull=True))
-
-        if layer_name is not None:
-            # we select only a layer name
-            queryset = queryset.filter(layer__name = layer_name)
-
-        # order by layer version priority
-        queryset = queryset.filter(Q(layer_source=None) | Q(layer_source__releaselayersourcepriority__release = release)).select_related('layer_source', 'layer', 'up_branch', "layer_source__releaselayersourcepriority__priority").order_by("-layer_source__releaselayersourcepriority__priority")
-
-        return queryset
-
     def get_all_compatible_layer_versions(self):
         """ Returns Queryset of all Layer_Versions which are compatible with
         this project"""
@@ -1184,7 +1167,9 @@ class Layer_Version(models.Model):
         return self._handle_url_path(self.layer.vcs_web_tree_base_url, '')
 
     def get_equivalents_wpriority(self, project):
-        return project.compatible_layerversions(layer_name = self.layer.name)
+        layer_versions = project.get_all_compatible_layer_versions()
+        filtered = layer_versions.filter(layer__name = self.layer.name)
+        return filtered.order_by("-layer_source__releaselayersourcepriority__priority")
 
     def get_vcs_reference(self):
         if self.branch is not None and len(self.branch) > 0:
lib/toaster/toastergui/views.py +14 -8
index 16f27b8..bd334b9 100755
@@ -2264,16 +2264,17 @@ if True:
                 prj.bitbake_version = prj.release.bitbake_version
                 prj.save()
                 # we need to change the layers
-                for i in prj.projectlayer_set.all():
+                for project in prj.projectlayer_set.all():
                     # find and add a similarly-named layer on the new branch
                     try:
-                        lv = prj.compatible_layerversions(layer_name = i.layercommit.layer.name)[0]
-                        ProjectLayer.objects.get_or_create(project = prj, layercommit = lv)
+                        layer_versions = prj.get_all_compatible_layer_versions()
+                        layer_versions = layer_versions.filter(layer__name = project.layercommit.layer.name)
+                        ProjectLayer.objects.get_or_create(project = prj, layercommit = layer_versions.first())
                     except IndexError:
                         pass
                     finally:
                         # get rid of the old entry
-                        i.delete()
+                        project.delete()
 
             if 'machineName' in request.POST:
                 machinevar = prj.projectvariable_set.get(name="MACHINE")
@@ -2383,12 +2384,17 @@ if True:
 
             retval = []
 
-            for i in prj.projectlayer_set.all():
-                lv = prj.compatible_layerversions(release = Release.objects.get(pk=new_release_id)).filter(layer__name = i.layercommit.layer.name)
+            for project in prj.projectlayer_set.all():
+                release = Release.objects.get(pk = new_release_id)
+
+                layer_versions = prj.get_all_compatible_layer_versions()
+                layer_versions = layer_versions.filter(release = release)
+                layer_versions = layer_versions.filter(layer__name = project.layercommit.layer.name)
+
                 # there is no layer_version with the new release id,
                 # and the same name
-                if lv.count() < 1:
-                    retval.append(i)
+                if layer_versions.count() < 1:
+                    retval.append(project)
 
             return response({"error":"ok",
                              "rows" : map( _lv_to_dict(prj),
Signature · 1 line
1.9.1
#11 lore raw

PATCH 12/18 toaster: Amend regex for MySQL database URLs

In reply to brian avery

From: Elliot Smith <[email protected]>

The database URL detection doesn't admit a MySQL URL without a port.

As this is a common case (e.g. you would set the HOST to a mysql.sock path if pointing at a local MySQL server, with no port setting), amend the regex so it will correctly recognise paths, as well as HOST:PORT URLs.

---
 lib/toaster/toastermain/settings.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
lib/toaster/toastermain/settings.py +2 -1
index 7c559bc..b6bdcd8 100644
@@ -64,6 +64,7 @@ if 'sqlite' in DATABASES['default']['ENGINE']:
 
 if 'DATABASE_URL' in os.environ:
     dburl = os.environ['DATABASE_URL']
+
     if dburl.startswith('sqlite3://'):
         result = re.match('sqlite3://(.*)', dburl)
         if result is None:
@@ -78,7 +79,7 @@ if 'DATABASE_URL' in os.environ:
         }
     elif dburl.startswith('mysql://'):
         # URL must be in this form: mysql://user:pass@host:port/name
-        result = re.match(r"mysql://([^:]*):([^@]*)@([^:]*):(\d+)/([^/]*)", dburl)
+        result = re.match(r"mysql://([^:]*):([^@]*)@([^:]*):(\d*)/([^/]*)", dburl)
         if result is None:
             raise Exception("ERROR: Could not read mysql database url: %s" % dburl)
         DATABASES['default'] = {
Signature · 1 line
1.9.1
#12 lore raw

PATCH 13/18 toaster: use OneToOneField instead of ForeignKey

In reply to brian avery

From: Ed Bartosh <[email protected]>

Used OneToOneField to reference BuildRequest in BRBitbake model.

Fixed django warning:
WARNINGS: Setting unique=True on a ForeignKey has the same effect
as using a OneToOneField.

---
 lib/toaster/bldcontrol/bbcontroller.py                  |  4 ++--
 lib/toaster/bldcontrol/localhostbecontroller.py         | 13 ++++++-------
 lib/toaster/bldcontrol/management/commands/runbuilds.py |  2 +-
 lib/toaster/bldcontrol/models.py                        |  2 +-
 lib/toaster/bldcontrol/tests.py                         |  6 +++---
 5 files changed, 13 insertions(+), 14 deletions(-)
lib/toaster/bldcontrol/bbcontroller.py +2 -2
index 781ff73..1387bda 100644
@@ -141,10 +141,10 @@ class BuildEnvironmentController(object):
         raise Exception("FIXME: Must override in order to actually start the BB server")
 
 
-    def setLayers(self, bbs, ls):
+    def setLayers(self, bitbake, ls):
         """ Checks-out bitbake executor and layers from git repositories.
             Sets the layer variables in the config file, after validating local layer paths.
-            The bitbakes must be a 1-length list of BRBitbake
+            bitbake must be a single BRBitbake instance
             The layer paths must be in a list of BRLayer object
 
             a word of attention: by convention, the first layer for any build will be poky!
lib/toaster/bldcontrol/localhostbecontroller.py +6 -7
index 00228e9..4f6f15c 100644
@@ -115,18 +115,17 @@ class LocalhostBEController(BuildEnvironmentController):
         return local_checkout_path
 
 
-    def setLayers(self, bitbakes, layers, targets):
+    def setLayers(self, bitbake, layers, targets):
         """ a word of attention: by convention, the first layer for any build will be poky! """
 
         assert self.be.sourcedir is not None
-        assert len(bitbakes) == 1
         # set layers in the layersource
 
         # 1. get a list of repos with branches, and map dirpaths for each layer
         gitrepos = {}
 
-        gitrepos[(bitbakes[0].giturl, bitbakes[0].commit)] = []
-        gitrepos[(bitbakes[0].giturl, bitbakes[0].commit)].append( ("bitbake", bitbakes[0].dirpath) )
+        gitrepos[(bitbake.giturl, bitbake.commit)] = []
+        gitrepos[(bitbake.giturl, bitbake.commit)].append( ("bitbake", bitbake.dirpath) )
 
         for layer in layers:
             # we don't process local URLs
@@ -198,7 +197,7 @@ class LocalhostBEController(BuildEnvironmentController):
                 # make sure we have a working bitbake
                 if not os.path.exists(os.path.join(self.pokydirname, 'bitbake')):
                     logger.debug("localhostbecontroller: checking bitbake into the poky dirname %s " % self.pokydirname)
-                    self._shellcmd("git clone -b \"%s\" \"%s\" \"%s\" " % (bitbakes[0].commit, bitbakes[0].giturl, os.path.join(self.pokydirname, 'bitbake')))
+                    self._shellcmd("git clone -b \"%s\" \"%s\" \"%s\" " % (bitbake.commit, bitbake.giturl, os.path.join(self.pokydirname, 'bitbake')))
 
             # verify our repositories
             for name, dirpath in gitrepos[(giturl, commit)]:
@@ -224,7 +223,7 @@ class LocalhostBEController(BuildEnvironmentController):
         for target in targets:
             try:
                 customrecipe = CustomImageRecipe.objects.get(name=target.target,
-                                                             project=bitbakes[0].req.project)
+                                                             project=bitbake.req.project)
             except CustomImageRecipe.DoesNotExist:
                 continue # not a custom recipe, skip
 
@@ -278,7 +277,7 @@ class LocalhostBEController(BuildEnvironmentController):
 
 
     def triggerBuild(self, bitbake, layers, variables, targets):
-        # set up the buid environment with the needed layers
+        # set up the build environment with the needed layers
         self.setLayers(bitbake, layers, targets)
 
         # get the bb server running with the build req id and build env id
lib/toaster/bldcontrol/management/commands/runbuilds.py +1 -1
index d40dedb..edf71a7 100644
@@ -57,7 +57,7 @@ class Command(NoArgsCommand):
             br.save()
 
             # this triggers an async build
-            bec.triggerBuild(br.brbitbake_set.all(), br.brlayer_set.all(), br.brvariable_set.all(), br.brtarget_set.all())
+            bec.triggerBuild(br.brbitbake, br.brlayer_set.all(), br.brvariable_set.all(), br.brtarget_set.all())
 
         except Exception as e:
             logger.error("runbuilds: Error launching build %s" % e)
lib/toaster/bldcontrol/models.py +1 -1
index ab41105..a3a49ce 100644
@@ -106,7 +106,7 @@ class BRLayer(models.Model):
     layer_version = models.ForeignKey(Layer_Version, null=True)
 
 class BRBitbake(models.Model):
-    req         = models.ForeignKey(BuildRequest, unique = True)    # only one bitbake for a request
+    req         = models.OneToOneField(BuildRequest)    # only one bitbake for a request
     giturl      = models.CharField(max_length =254)
     commit      = models.CharField(max_length = 254)
     dirpath     = models.CharField(max_length = 254)
lib/toaster/bldcontrol/tests.py +3 -3
index f54cf7f..141b42a 100644
@@ -18,7 +18,7 @@ import subprocess
 import os
 
 # standard poky data hardcoded for testing
-BITBAKE_LAYERS = [type('bitbake_info', (object,), { "giturl": "git://git.yoctoproject.org/poky.git", "dirpath": "", "commit": "HEAD"})]
+BITBAKE_LAYER = type('bitbake_info', (object,), { "giturl": "git://git.yoctoproject.org/poky.git", "dirpath": "", "commit": "HEAD"})
 POKY_LAYERS = [
     type('poky_info', (object,), { "name": "meta", "giturl": "git://git.yoctoproject.org/poky.git", "dirpath": "meta", "commit": "HEAD"}),
     type('poky_info', (object,), { "name": "meta-yocto", "giturl": "git://git.yoctoproject.org/poky.git", "dirpath": "meta-yocto", "commit": "HEAD"}),
@@ -53,7 +53,7 @@ class BEControllerTests(object):
         bc = self._getBEController(obe)
         try:
             # setting layers, skip any layer info
-            bc.setLayers(BITBAKE_LAYERS, POKY_LAYERS)
+            bc.setLayers(BITBAKE_LAYER, POKY_LAYERS)
         except NotImplementedException,  e:
             print "Test skipped due to command not implemented yet"
             return True
@@ -80,7 +80,7 @@ class BEControllerTests(object):
         layerSet = False
         try:
             # setting layers, skip any layer info
-            layerSet = bc.setLayers(BITBAKE_LAYERS, POKY_LAYERS)
+            layerSet = bc.setLayers(BITBAKE_LAYER, POKY_LAYERS)
         except NotImplementedException:
             print "Test skipped due to command not implemented yet"
             return True
Signature · 1 line
1.9.1
#13 lore raw

PATCH 14/18 toaster: fixed format strings

In reply to brian avery

From: Ed Bartosh <[email protected]>

Printing id as a number breaks toaster if object is not saved yet as id/pk is None in this case.

Changed format spec to print id as a string in __str__ and __unicode__ methods of Build, Task and LayerVersion models.

---
 lib/toaster/orm/models.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
lib/toaster/orm/models.py +3 -3
index f07feda..3c51c6f 100644
@@ -370,7 +370,7 @@ class Build(models.Model):
             return self.get_outcome_text()
 
     def __str__(self):
-        return "%d %s %s" % (self.id, self.project, ",".join([t.target for t in self.target_set.all()]))
+        return "%s %s %s" % (self.id, self.project, ",".join([t.target for t in self.target_set.all()]))
 
 
 # an Artifact is anything that results from a Build, and may be of interest to the user, and is not stored elsewhere
@@ -552,7 +552,7 @@ class Task(models.Model):
     sstate_text  = property(get_sstate_text)
 
     def __unicode__(self):
-        return "%d(%d) %s:%s" % (self.pk, self.build.pk, self.recipe.name, self.task_name)
+        return "%s(%s) %s:%s" % (self.pk, self.build.pk, self.recipe.name, self.task_name)
 
     class Meta:
         ordering = ('order', 'recipe' ,)
@@ -1203,7 +1203,7 @@ class Layer_Version(models.Model):
         return sorted(result, key=lambda x: x.layer.name)
 
     def __unicode__(self):
-        return "%d %s (VCS %s, Project %s)" % (self.pk, str(self.layer), self.get_vcs_reference(), self.build.project if self.build is not None else "No project")
+        return "%s %s (VCS %s, Project %s)" % (self.pk, str(self.layer), self.get_vcs_reference(), self.build.project if self.build is not None else "No project")
 
     class Meta:
         unique_together = ("layer_source", "up_id")
Signature · 1 line
1.9.1
#14 lore raw

PATCH 15/18 toaster: work around 'database is locked' error

In reply to brian avery

From: Ed Bartosh <[email protected]>

When sqlite can not cope with a stream of 'insert' queries it throws 'database is locked' exception.

Wrapping model.save in transaction.atomic context and repeating the call should solve this issue.

---
 lib/toaster/orm/models.py | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
lib/toaster/orm/models.py +19 -1
index 3c51c6f..0d598ac 100644
@@ -29,10 +29,28 @@ from django.core import validators
 from django.conf import settings
 import django.db.models.signals
 
-
 import logging
 logger = logging.getLogger("toaster")
 
+if 'sqlite' in settings.DATABASES['default']['ENGINE']:
+    from django.db import transaction, OperationalError
+    from time import sleep
+
+    _base_save = models.Model.save
+    def save(self, *args, **kwargs):
+        while True:
+            try:
+                with transaction.atomic():
+                    return _base_save(self, *args, **kwargs)
+            except OperationalError as err:
+                if 'database is locked' in str(err):
+                    logger.warning("%s, model: %s, args: %s, kwargs: %s",
+                                   err, self.__class__, args, kwargs)
+                    sleep(0.5)
+                    continue
+                raise
+
+    models.Model.save = save
 
 class GitURLValidator(validators.URLValidator):
     import re
Signature · 1 line
1.9.1
#15 lore raw

PATCH 16/18 toaster: removed extra calls of migrate

In reply to brian avery

From: Ed Bartosh <[email protected]>

One manage.py migrate should be enough to apply migrations for all applications. Extra calls just slow toaster script down.

Removed calls of migrate for orm and bldcontrol apps.

---
 bin/toaster | 20 +-------------------
 1 file changed, 1 insertion(+), 19 deletions(-)
bin/toaster +1 -19
index 0862a35..7645c56 100755
@@ -58,26 +58,8 @@ webserverStartAll()
     # python bitbake/lib/toaster/manage.py python manage.py createsuperuser --username=<ME>
     python $BBBASEDIR/lib/toaster/manage.py migrate --noinput || retval=1
 
-    python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2
-
-    if [ $retval -eq 1 ]; then
-        echo "Failed db sync, aborting system start" 1>&2
-        return $retval
-    fi
-
-    python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1
-
-    if [ $retval -eq 1 ]; then
-        printf "\nError on orm migration, rolling back...\n"
-        python $BBBASEDIR/lib/toaster/manage.py migrate orm 0001_initial --fake
-        return $retval
-    fi
-
-    python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
-
     if [ $retval -eq 1 ]; then
-        printf "\nError on bldcontrol migration, rolling back...\n"
-        python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol 0001_initial --fake
+        echo "Failed migrations, aborting system start" 1>&2
         return $retval
     fi
 
Signature · 1 line
1.9.1
#16 lore raw

PATCH 17/18 toaster: monkey patch Queryset

In reply to brian avery

From: Ed Bartosh <[email protected]>

Fixed 'database is locked' issue by monkey patching django QuerySet methods.

The actual patching places were found by bisecting Django codebase.

This commit should be removed after Django is fixed if it's fixed at all.

---
 lib/toaster/orm/models.py | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
lib/toaster/orm/models.py +30 -0
index 0d598ac..6e87c54 100644
@@ -52,6 +52,36 @@ if 'sqlite' in settings.DATABASES['default']['ENGINE']:
 
     models.Model.save = save
 
+    # HACK: Monkey patch Django to fix 'database is locked' issue
+
+    from django.db.models.query import QuerySet
+    _base_insert = QuerySet._insert
+    def _insert(self,  *args, **kwargs):
+        with transaction.atomic(using=self.db, savepoint=False):
+            return _base_insert(self, *args, **kwargs)
+    QuerySet._insert = _insert
+
+    from django.utils import six
+    def _create_object_from_params(self, lookup, params):
+        """
+        Tries to create an object using passed params.
+        Used by get_or_create and update_or_create
+        """
+        try:
+            obj = self.create(**params)
+            return obj, True
+        except IntegrityError:
+            exc_info = sys.exc_info()
+            try:
+                return self.get(**lookup), False
+            except self.model.DoesNotExist:
+                pass
+            six.reraise(*exc_info)
+
+    QuerySet._create_object_from_params = _create_object_from_params
+
+    # end of HACK
+
 class GitURLValidator(validators.URLValidator):
     import re
     regex = re.compile(
Signature · 1 line
1.9.1
#17 lore raw

PATCH 18/18 toaster: rework checking of Django version

In reply to brian avery

From: Ed Bartosh <[email protected]>

Checked django version considering information from toaster-requirements.txt, e.g. if requirements file contains line "Django>1.8,<1.9" toaster should be able to check that requirement correctly.

---
 bin/toaster              | 24 ++++++++++--------------
 toaster-requirements.txt |  2 +-
 2 files changed, 11 insertions(+), 15 deletions(-)
bin/toaster +10 -14
index 7645c56..1c1e029 100755
@@ -156,16 +156,15 @@ notify_chldexit() {
 
 
 verify_prereq() {
-    # Verify prerequisites
-    regex="([0-9])\.([0-9])"
-    for line in `cat $BBBASEDIR/toaster-requirements.txt | grep Django` ; do
-        [[ `echo $line` =~ $regex ]]
-        required_major_version="${BASH_REMATCH[1]}"
-        required_minor_version="${BASH_REMATCH[2]}"
-    done
-
-    if ! echo "import django; print ($required_major_version,) == django.VERSION[0:1] and django.VERSION[1:2][0] in ($required_minor_version,)" | python 2>/dev/null | grep True >/dev/null; then
-        printf "This program needs Django $required_major_version.$required_minor_version. Please install with\n\npip install -r $BBBASEDIR/toaster-requirements.txt\n"
+    # Verify Django version
+    reqfile=$(python -c "import os; print os.path.realpath('$BBBASEDIR/toaster-requirements.txt')")
+    exp='s/Django\([><=]\+\)\([^,]\+\),\([><=]\+\)\(.\+\)/'
+    exp=$exp'import sys,django;version=django.get_version().split(".");'
+    exp=$exp'sys.exit(not (version \1 "\2".split(".") and version \3 "\4".split(".")))/p'
+    if ! sed -n "$exp" $reqfile | python - ; then
+        req=`grep ^Django $reqfile`
+        echo "This program needs $req"
+        echo "Please install with pip install -r $reqfile"
         return 2
     fi
 
@@ -265,10 +264,7 @@ if [ "$1" = 'restart-bitbake' ] ; then
     return $rc
 fi
 
-if ! verify_prereq; then
-    echo "Error: Could not verify that the needed dependencies are installed. Please use virtualenv and pip to install dependencies listed in toaster-requirements.txt" 1>&2
-    return 1
-fi
+verify_prereq || return 1
 
 # We make sure we're running in the current shell and in a good environment
 if [ -z "$BUILDDIR" ] ||  ! which bitbake >/dev/null 2>&1 ; then
toaster-requirements.txt +1 -1
index 5d34494..0e8c742 100644
@@ -1,4 +1,4 @@
-Django==1.8.6
+Django>1.8,<1.9
 argparse==1.2.1
 wsgiref==0.1.2
 beautifulsoup4>=4.4.0
Signature · 1 line
1.9.1