toaster: upgrade to Django 1.8
cover toaster: upgrade to Django 1.8
This rather large patchset upgrades toaster to work with Django 1.8. It was tested with Django 1.8.6 and 1.8.7.
yay for LTS!
-b
The following changes since commit 2cb81fb08395ec9cd68b02bc48391c9610eac9d9:
toasterui: process SetBRBE event (2015-12-09 19:45:13 -0800)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib bavery/submit/ed/toaster/django_upgrade-8364
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/submit/ed/toaster/django_upgrade-8364
Ed Bartosh (9):
toaster: Remove South migrations
toaster: Fix references to app paths
toaster: rename get_query_set -> get_queryset
toaster: use OneToOneField instead of ForeignKey
toaster: fixed format strings
toaster: work around 'database is locked' error
toaster: removed extra calls of migrate
toaster: monkey patch Queryset
toaster: rework checking of Django version
Elliot Smith (9):
toaster: Upgrade to Django 1.8.6 and remove South
toaster: Create default project with get_or_create* method
toaster: Start Django machinery for database access
toaster: Update API used to make runbuilds methods run in transactions
toaster: Prevent deprecation warnings for RedirectView
toaster: Update deprecated manage.py command
toaster: Check Django version against toaster-requirements.txt
toaster: Remove compatible_layerversions() method
toaster: Amend regex for MySQL database URLs
bin/toaster | 44 +- lib/bb/ui/buildinfohelper.py | 26 +- lib/toaster/bldcollector/urls.py | 1 - lib/toaster/bldcontrol/bbcontroller.py | 4 +- lib/toaster/bldcontrol/localhostbecontroller.py | 13 +- .../bldcontrol/management/commands/runbuilds.py | 6 +- lib/toaster/bldcontrol/migrations/0001_initial.py | 265 +++--- ...onment_sourcedir__add_field_buildenvironment.py | 106 --- .../0003_auto__add_field_brlayer_dirpath.py | 99 --- .../bldcontrol/migrations/0004_loadinitialdata.py | 104 --- .../migrations/0005_auto__add_brerror.py | 112 --- .../migrations/0006_auto__add_brbitbake.py | 128 --- ...st_environment__chg_field_buildrequest_build.py | 145 ---- .../bldcontrol/migrations/0008_brarchive.py | 138 --- .../0009_auto__add_field_brlayer_layer_version.py | 180 ---- lib/toaster/bldcontrol/models.py | 2 +- lib/toaster/bldcontrol/tests.py | 6 +- lib/toaster/orm/migrations/0001_initial.py | 928 ++++++++++++--------- .../0002_auto__add_field_build_timespent.py | 180 ---- lib/toaster/orm/migrations/0003_timespent.py | 182 ---- .../0004_auto__add_field_package_installed_name.py | 181 ---- ...e__add_target_file__add_field_variablehistor.py | 281 ------- ...ge_size__add_field_target_license_manifest_p.py | 235 ------ .../orm/migrations/0007_auto__add_helptext.py | 214 ----- ...story_operation__chg_field_recipe_descriptio.py | 225 ----- ..._add_projectlayer__add_projecttarget__add_pr.py | 286 ------- ...anch__add_field_project_short_description__a.py | 257 ------ .../0011_auto__add_field_projectlayer_dirpath.py | 242 ------ ...layer_optional__add_field_projecttarget_task.py | 252 ------ ...erversiondependency__add_unique_layerversion.py | 710 ---------------- ...mmary__chg_field_layer_summary__chg_field_re.py | 336 -------- ...web_url__add_field_layer_vcs_web_tree_base_u.py | 336 -------- ...lptext__chg_field_release_branch__add_index_.py | 359 -------- ...faultlayer__add_releaselayersourcepriority__.py | 396 --------- .../0018_auto__add_field_layer_version_project.py | 331 -------- .../orm/migrations/0019_auto__add_buildartifact.py | 342 -------- ...ion_local_path__add_field_recipe_pathflags__.py | 361 -------- ...ect__chg_field_project_bitbake_version__chg_.py | 371 -------- ...k__add_field_layer_version_local_path__del_f.py | 343 -------- ...ings_no__del_field_build_errors_no__del_fiel.py | 353 -------- .../0024_auto__add_field_recipe_is_image.py | 338 -------- .../0025_auto__add_field_project_is_default.py | 346 -------- .../orm/migrations/0026_set_default_project.py | 374 --------- ...e__add_unique_customimagerecipe_name_project.py | 375 --------- .../0028_auto__chg_field_logmessage_message.py | 345 -------- lib/toaster/orm/models.py | 108 ++- lib/toaster/toastergui/urls.py | 2 +- lib/toaster/toastergui/views.py | 24 +- lib/toaster/toastermain/settings.py | 4 +- lib/toaster/toastermain/urls.py | 2 +- toaster-requirements.txt | 3 +- 51 files changed, 779 insertions(+), 10222 deletions(-) delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto__add_field_buildenvironment_sourcedir__add_field_buildenvironment.py delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0003_auto__add_field_brlayer_dirpath.py delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0004_loadinitialdata.py delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0005_auto__add_brerror.py delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0006_auto__add_brbitbake.py delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0007_auto__add_field_buildrequest_environment__chg_field_buildrequest_build.py delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0008_brarchive.py delete mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0009_auto__add_field_brlayer_layer_version.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0002_auto__add_field_build_timespent.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0003_timespent.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0004_auto__add_field_package_installed_name.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0005_auto__add_target_image_file__add_target_file__add_field_variablehistor.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0006_auto__add_field_target_image_size__add_field_target_license_manifest_p.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0007_auto__add_helptext.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0008_auto__chg_field_variablehistory_operation__chg_field_recipe_descriptio.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0009_auto__add_projectvariable__add_projectlayer__add_projecttarget__add_pr.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0010_auto__add_field_project_branch__add_field_project_short_description__a.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0011_auto__add_field_projectlayer_dirpath.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0012_auto__add_field_projectlayer_optional__add_field_projecttarget_task.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0013_auto__add_release__add_layerversiondependency__add_unique_layerversion.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0014_auto__chg_field_package_summary__chg_field_layer_summary__chg_field_re.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0015_auto__add_field_layer_vcs_web_url__add_field_layer_vcs_web_tree_base_u.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0016_auto__add_field_release_helptext__chg_field_release_branch__add_index_.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0017_auto__del_toastersettingdefaultlayer__add_releaselayersourcepriority__.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0018_auto__add_field_layer_version_project.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0019_auto__add_buildartifact.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0020_auto__add_field_layer_version_local_path__add_field_recipe_pathflags__.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0021_auto__chg_field_build_project__chg_field_project_bitbake_version__chg_.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0022_auto__add_field_target_task__add_field_layer_version_local_path__del_f.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0023_auto__del_field_build_warnings_no__del_field_build_errors_no__del_fiel.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0024_auto__add_field_recipe_is_image.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0025_auto__add_field_project_is_default.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0026_set_default_project.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0027_auto__add_customimagerecipe__add_unique_customimagerecipe_name_project.py delete mode 100644 bitbake/lib/toaster/orm/migrations/0028_auto__chg_field_logmessage_message.py
--
1.9.1
PATCH 1/18 toaster: Upgrade to Django 1.8.6 and remove South
From: Elliot Smith <[email protected]>
Upgrade Django to long-term support version.
Django now provides its own migration framework, so remove requirement for South.
[YOCTO #8364]
--- bin/toaster | 8 ++------ lib/toaster/toastermain/settings.py | 1 - toaster-requirements.txt | 3 +-- 3 files changed, 3 insertions(+), 9 deletions(-)
bin/toaster +2 -6
@@ -176,15 +176,11 @@ notify_chldexit() { verify_prereq() { # Verify prerequisites - if ! echo "import django; print (1,) == django.VERSION[0:1] and django.VERSION[1:2][0] in (6,)" | python 2>/dev/null | grep True >/dev/null; then - printf "This program needs Django 1.6. Please install with\n\npip install django==1.6\n" + if ! echo "import django; print (1,) == django.VERSION[0:1] and django.VERSION[1:2][0] in (8,)" | python 2>/dev/null | grep True >/dev/null; then + printf "This program needs Django 1.8. Please install with\n\npip install django<=1.8.7\n" return 2 fi - if ! echo "import south; print reduce(lambda x, y: 2 if x==2 else 0 if x == 0 else y, map(lambda x: 1+cmp(x[1]-x[0],0), zip([0,8,4], map(int,south.__version__.split(\".\"))))) > 0" | python 2>/dev/null | grep True >/dev/null; then - printf "This program needs South 0.8.4. Please install with\n\npip install south==0.8.4\n" - return 2 - fi return 0 }
lib/toaster/toastermain/settings.py +0 -1
@@ -276,7 +276,6 @@ INSTALLED_APPS = ( 'django.contrib.humanize', 'bldcollector', 'toastermain', - 'south', )
toaster-requirements.txt +1 -2
@@ -1,5 +1,4 @@ -Django==1.6 -South==0.8.4 +Django==1.8.6 argparse==1.2.1 wsgiref==0.1.2 beautifulsoup4>=4.4.0
Signature · 1 line
1.9.1
PATCH 3/18 toaster: Fix references to app paths
From: Ed Bartosh <[email protected]>
The buildinfohelper runs outside of Django, but still needs access to our Django app classes.
Previously, the imports referenced the toaster.* app, which worked fine. But in Django 1.8, this causes an error about the same module being loaded multiple times from different paths.
Change the paths to our Django modules so they don't cause this error to be thrown. We can do this as we've added our application libraries to sys.path in the buildinfohelper anyway.
[YOCTO #8364]
--- lib/bb/ui/buildinfohelper.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
lib/bb/ui/buildinfohelper.py +8 -8
@@ -33,14 +33,14 @@ def _configure_toaster(): sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'toaster')) _configure_toaster() -from toaster.orm.models import Build, Task, Recipe, Layer_Version, Layer, Target, LogMessage, HelpText -from toaster.orm.models import Target_Image_File, BuildArtifact -from toaster.orm.models import Variable, VariableHistory -from toaster.orm.models import Package, Package_File, Target_Installed_Package, Target_File -from toaster.orm.models import Task_Dependency, Package_Dependency -from toaster.orm.models import Recipe_Dependency - -from toaster.orm.models import Project +from orm.models import Build, Task, Recipe, Layer_Version, Layer, Target, LogMessage, HelpText +from orm.models import Target_Image_File, BuildArtifact +from orm.models import Variable, VariableHistory +from orm.models import Package, Package_File, Target_Installed_Package, Target_File +from orm.models import Task_Dependency, Package_Dependency +from orm.models import Recipe_Dependency + +from orm.models import Project from bldcontrol.models import BuildEnvironment, BuildRequest from bb.msg import BBLogFormatter as formatter
Signature · 1 line
1.9.1
PATCH 4/18 toaster: Create default project with get_or_create* method
In reply to brian averyFrom: Elliot Smith <[email protected]>
Rather than maintain data as part of the migrations (as was done for the default project previously), create the default (cli builds) project on demand as a by-product of getting it from the database.
[YOCTO #8364]
--- lib/bb/ui/buildinfohelper.py | 9 +++++++-- lib/toaster/orm/models.py | 23 +++++++++++++++-------- lib/toaster/toastergui/views.py | 2 +- 3 files changed, 23 insertions(+), 11 deletions(-)
lib/bb/ui/buildinfohelper.py +7 -2
@@ -146,7 +146,7 @@ class ORMWrapper(object): prj = Project.objects.get(pk = project_id) else: # this build was triggered by a legacy system, or command line interactive mode - prj = Project.objects.get_default_project() + prj = Project.objects.get_or_create_default_project() logger.debug(1, "buildinfohelper: project is not specified, defaulting to %s" % prj)
@@ -308,6 +308,11 @@ class ORMWrapper(object): # then we are wholly responsible for the data # and therefore we return the 'real' recipe rather than the build # history copy of the recipe. + if recipe_information['layer_version'].build is not None and \ + recipe_information['layer_version'].build.project == \ + Project.objects.get_or_create_default_project(): + return recipe + if built_recipe is None: return recipe
@@ -345,7 +350,7 @@ class ORMWrapper(object): # If we're doing a command line build then associate this new layer with the # project to avoid it 'contaminating' toaster data project = None - if build_obj.project == Project.objects.get_default_project(): + if build_obj.project == Project.objects.get_or_create_default_project(): project = build_obj.project layer_version_object, _ = Layer_Version.objects.get_or_create(
lib/toaster/orm/models.py +15 -8
@@ -91,18 +91,25 @@ class ProjectManager(models.Manager): return prj - def create(self, *args, **kwargs): - raise Exception("Invalid call to Project.objects.create. Use Project.objects.create_project() to create a project") - # return single object with is_default = True - def get_default_project(self): + def get_or_create_default_project(self): projects = super(ProjectManager, self).filter(is_default = True) + if len(projects) > 1: - raise Exception("Inconsistent project data: multiple " + - "default projects (i.e. with is_default=True)") + raise Exception('Inconsistent project data: multiple ' + + 'default projects (i.e. with is_default=True)') elif len(projects) < 1: - raise Exception("Inconsistent project data: no default project found") - return projects[0] + options = { + 'name': 'Command line builds', + 'short_description': 'Project for builds started outside Toaster', + 'is_default': True + } + project = Project.objects.create(**options) + project.save() + + return project + else: + return projects[0] class Project(models.Model): search_allowed_fields = ['name', 'short_description', 'release__name', 'release__branch_name']
lib/toaster/toastergui/views.py +1 -1
@@ -73,7 +73,7 @@ class MimeTypeFinder(object): def landing(request): # in build mode, we redirect to the command-line builds page # if there are any builds for the default (cli builds) project - default_project = Project.objects.get_default_project() + default_project = Project.objects.get_or_create_default_project() default_project_builds = Build.objects.filter(project = default_project) # we only redirect to projects page if there is a user-generated project
Signature · 1 line
1.9.1
PATCH 5/18 toaster: Start Django machinery for database access
In reply to brian averyFrom: Elliot Smith <[email protected]>
buildinfohelper, with the new import paths for our Django models and Django 1.8, was not getting an active connection to the database.
In buildinfohelper, call django.setup() explicitly to make sure that the database connection is ready and models can be queried and saved.
[YOCTO #8364]
--- lib/bb/ui/buildinfohelper.py | 3 +++ 1 file changed, 3 insertions(+)
lib/bb/ui/buildinfohelper.py +3 -0
@@ -24,6 +24,7 @@ import os os.environ["DJANGO_SETTINGS_MODULE"] = "toaster.toastermain.settings" +import django from django.utils import timezone
@@ -33,6 +34,8 @@ def _configure_toaster(): sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'toaster')) _configure_toaster() +django.setup() + from orm.models import Build, Task, Recipe, Layer_Version, Layer, Target, LogMessage, HelpText from orm.models import Target_Image_File, BuildArtifact from orm.models import Variable, VariableHistory
Signature · 1 line
1.9.1
PATCH 6/18 toaster: rename get_query_set -> get_queryset
In reply to brian averyFrom: Ed Bartosh <[email protected]>
Renamed get_query_set to get_queryset as get_queryset causes deprecation warning in Django 1.7 and is deprecated in Django 1.8
--- lib/toaster/orm/models.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
lib/toaster/orm/models.py +4 -4
@@ -598,8 +598,8 @@ class Package(models.Model): class Package_DependencyManager(models.Manager): use_for_related_fields = True - def get_query_set(self): - return super(Package_DependencyManager, self).get_query_set().exclude(package_id = F('depends_on__id')) + def get_queryset(self): + return super(Package_DependencyManager, self).get_queryset().exclude(package_id = F('depends_on__id')) class Package_Dependency(models.Model): TYPE_RDEPENDS = 0
@@ -699,8 +699,8 @@ class Recipe(models.Model): class Recipe_DependencyManager(models.Manager): use_for_related_fields = True - def get_query_set(self): - return super(Recipe_DependencyManager, self).get_query_set().exclude(recipe_id = F('depends_on__id')) + def get_queryset(self): + return super(Recipe_DependencyManager, self).get_queryset().exclude(recipe_id = F('depends_on__id')) class Recipe_Dependency(models.Model): TYPE_DEPENDS = 0
Signature · 1 line
1.9.1
PATCH 7/18 toaster: Update API used to make runbuilds methods run in transactions
In reply to brian averyFrom: Elliot Smith <[email protected]>
runbuilds previously had its methods decorated with transaction.commit_on_success, which is now deprecated. transaction.atomic is an (almost) drop-in replacement for this, so use this instead.
[YOCTO #8364]
--- lib/toaster/bldcontrol/management/commands/runbuilds.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
lib/toaster/bldcontrol/management/commands/runbuilds.py +2 -2
@@ -14,14 +14,14 @@ class Command(NoArgsCommand): help = "Schedules and executes build requests as possible. Does not return (interrupt with Ctrl-C)" - @transaction.commit_on_success + @transaction.atomic def _selectBuildEnvironment(self): bec = getBuildEnvironmentController(lock = BuildEnvironment.LOCK_FREE) bec.be.lock = BuildEnvironment.LOCK_LOCK bec.be.save() return bec - @transaction.commit_on_success + @transaction.atomic def _selectBuildRequest(self): br = BuildRequest.objects.filter(state = BuildRequest.REQ_QUEUED).order_by('pk')[0] br.state = BuildRequest.REQ_INPROGRESS
Signature · 1 line
1.9.1
PATCH 8/18 toaster: Prevent deprecation warnings for RedirectView
In reply to brian averyFrom: Elliot Smith <[email protected]>
The API for RedirectView is due to change in Django 1.9, which means that Toaster generates deprecation warnings.
Set the "permanent" flag when constructing RedirectView instances to prevent this warning.
[YOCTO #8364]
--- lib/toaster/bldcollector/urls.py | 1 - lib/toaster/toastergui/urls.py | 2 +- lib/toaster/toastermain/urls.py | 2 +- 3 files changed, 2 insertions(+), 3 deletions(-)
lib/toaster/bldcollector/urls.py +0 -1
@@ -18,7 +18,6 @@ from django.conf.urls import patterns, include, url -from django.views.generic import RedirectView urlpatterns = patterns('bldcollector.views', # landing point for pushing a bitbake_eventlog.json file to this toaster instace
lib/toaster/toastergui/urls.py +1 -1
@@ -168,5 +168,5 @@ urlpatterns = patterns('toastergui.views', name='xhr_customrecipe'), # default redirection - url(r'^$', RedirectView.as_view( url= 'landing')), + url(r'^$', RedirectView.as_view(url='landing', permanent=True)), )
lib/toaster/toastermain/urls.py +1 -1
@@ -45,7 +45,7 @@ urlpatterns = patterns('', url(r'^orm/eventfile$', 'bldcollector.views.eventfile'), # if no application is selected, we have the magic toastergui app here - url(r'^$', never_cache(RedirectView.as_view(url='/toastergui/'))), + url(r'^$', never_cache(RedirectView.as_view(url='/toastergui/', permanent=True))), ) import toastermain.settings
Signature · 1 line
1.9.1
PATCH 9/18 toaster: Update deprecated manage.py command
In reply to brian averyFrom: Elliot Smith <[email protected]>
syncdb is deprecated in favour of migrate in Django 1.7:
https://docs.djangoproject.com/en/1.8/releases/1.7/#schema-migrations
Update to the "migrate" command in Toaster's start script.
[YOCTO #8364]
--- bin/toaster | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
bin/toaster +1 -1
@@ -56,7 +56,7 @@ webserverStartAll() retval=0 # you can always add a superuser later via # python bitbake/lib/toaster/manage.py python manage.py createsuperuser --username=<ME> - python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1 + python $BBBASEDIR/lib/toaster/manage.py migrate --noinput || retval=1 python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2
Signature · 1 line
1.9.1
PATCH 10/18 toaster: Check Django version against toaster-requirements.txt
In reply to brian averyFrom: Elliot Smith <[email protected]>
Instead of hard-coding the required Django version in the start script, look it up from the toaster-requirements.txt file.
[YOCTO #8364]
--- bin/toaster | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
bin/toaster +8 -2
@@ -175,9 +175,15 @@ notify_chldexit() { verify_prereq() { # Verify prerequisites + regex="([0-9])\.([0-9])" + for line in `cat $BBBASEDIR/toaster-requirements.txt | grep Django` ; do + [[ `echo $line` =~ $regex ]] + required_major_version="${BASH_REMATCH[1]}" + required_minor_version="${BASH_REMATCH[2]}" + done - if ! echo "import django; print (1,) == django.VERSION[0:1] and django.VERSION[1:2][0] in (8,)" | python 2>/dev/null | grep True >/dev/null; then - printf "This program needs Django 1.8. Please install with\n\npip install django<=1.8.7\n" + if ! echo "import django; print ($required_major_version,) == django.VERSION[0:1] and django.VERSION[1:2][0] in ($required_minor_version,)" | python 2>/dev/null | grep True >/dev/null; then + printf "This program needs Django $required_major_version.$required_minor_version. Please install with\n\npip install -r $BBBASEDIR/toaster-requirements.txt\n" return 2 fi
Signature · 1 line
1.9.1
PATCH 11/18 toaster: Remove compatible_layerversions() method
In reply to brian averyFrom: Elliot Smith <[email protected]>
Remove compatible_layerversions() and replace calls to it with calls to get_all_compatible_layer_versions().
The sorting done in compatible_layerversions() is not relevant, and the code can be simplified by using get_all_compatible_layer_versions().
[YOCTO #8364]
--- lib/toaster/orm/models.py | 21 +++------------------ lib/toaster/toastergui/views.py | 22 ++++++++++++++-------- 2 files changed, 17 insertions(+), 26 deletions(-)
lib/toaster/orm/models.py +3 -18
@@ -195,23 +195,6 @@ class Project(models.Model): except (Variable.DoesNotExist,IndexError): return( "not_found" ) - # returns a queryset of compatible layers for a project - def compatible_layerversions(self, release = None, layer_name = None): - logger.warning("This function is deprecated") - if release == None: - release = self.release - # layers on the same branch or layers specifically set for this project - queryset = Layer_Version.objects.filter(((Q(up_branch__name = release.branch_name) & Q(project = None)) | Q(project = self)) & Q(build__isnull=True)) - - if layer_name is not None: - # we select only a layer name - queryset = queryset.filter(layer__name = layer_name) - - # order by layer version priority - queryset = queryset.filter(Q(layer_source=None) | Q(layer_source__releaselayersourcepriority__release = release)).select_related('layer_source', 'layer', 'up_branch', "layer_source__releaselayersourcepriority__priority").order_by("-layer_source__releaselayersourcepriority__priority") - - return queryset - def get_all_compatible_layer_versions(self): """ Returns Queryset of all Layer_Versions which are compatible with this project"""
@@ -1184,7 +1167,9 @@ class Layer_Version(models.Model): return self._handle_url_path(self.layer.vcs_web_tree_base_url, '') def get_equivalents_wpriority(self, project): - return project.compatible_layerversions(layer_name = self.layer.name) + layer_versions = project.get_all_compatible_layer_versions() + filtered = layer_versions.filter(layer__name = self.layer.name) + return filtered.order_by("-layer_source__releaselayersourcepriority__priority") def get_vcs_reference(self): if self.branch is not None and len(self.branch) > 0:
lib/toaster/toastergui/views.py +14 -8
@@ -2264,16 +2264,17 @@ if True: prj.bitbake_version = prj.release.bitbake_version prj.save() # we need to change the layers - for i in prj.projectlayer_set.all(): + for project in prj.projectlayer_set.all(): # find and add a similarly-named layer on the new branch try: - lv = prj.compatible_layerversions(layer_name = i.layercommit.layer.name)[0] - ProjectLayer.objects.get_or_create(project = prj, layercommit = lv) + layer_versions = prj.get_all_compatible_layer_versions() + layer_versions = layer_versions.filter(layer__name = project.layercommit.layer.name) + ProjectLayer.objects.get_or_create(project = prj, layercommit = layer_versions.first()) except IndexError: pass finally: # get rid of the old entry - i.delete() + project.delete() if 'machineName' in request.POST: machinevar = prj.projectvariable_set.get(name="MACHINE")
@@ -2383,12 +2384,17 @@ if True: retval = [] - for i in prj.projectlayer_set.all(): - lv = prj.compatible_layerversions(release = Release.objects.get(pk=new_release_id)).filter(layer__name = i.layercommit.layer.name) + for project in prj.projectlayer_set.all(): + release = Release.objects.get(pk = new_release_id) + + layer_versions = prj.get_all_compatible_layer_versions() + layer_versions = layer_versions.filter(release = release) + layer_versions = layer_versions.filter(layer__name = project.layercommit.layer.name) + # there is no layer_version with the new release id, # and the same name - if lv.count() < 1: - retval.append(i) + if layer_versions.count() < 1: + retval.append(project) return response({"error":"ok", "rows" : map( _lv_to_dict(prj),
Signature · 1 line
1.9.1
PATCH 12/18 toaster: Amend regex for MySQL database URLs
In reply to brian averyFrom: Elliot Smith <[email protected]>
The database URL detection doesn't admit a MySQL URL without a port.
As this is a common case (e.g. you would set the HOST to a mysql.sock path if pointing at a local MySQL server, with no port setting), amend the regex so it will correctly recognise paths, as well as HOST:PORT URLs.
--- lib/toaster/toastermain/settings.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
lib/toaster/toastermain/settings.py +2 -1
@@ -64,6 +64,7 @@ if 'sqlite' in DATABASES['default']['ENGINE']: if 'DATABASE_URL' in os.environ: dburl = os.environ['DATABASE_URL'] + if dburl.startswith('sqlite3://'): result = re.match('sqlite3://(.*)', dburl) if result is None:
@@ -78,7 +79,7 @@ if 'DATABASE_URL' in os.environ: } elif dburl.startswith('mysql://'): # URL must be in this form: mysql://user:pass@host:port/name - result = re.match(r"mysql://([^:]*):([^@]*)@([^:]*):(\d+)/([^/]*)", dburl) + result = re.match(r"mysql://([^:]*):([^@]*)@([^:]*):(\d*)/([^/]*)", dburl) if result is None: raise Exception("ERROR: Could not read mysql database url: %s" % dburl) DATABASES['default'] = {
Signature · 1 line
1.9.1
PATCH 13/18 toaster: use OneToOneField instead of ForeignKey
In reply to brian averyFrom: Ed Bartosh <[email protected]>
Used OneToOneField to reference BuildRequest in BRBitbake model.
Fixed django warning:
WARNINGS: Setting unique=True on a ForeignKey has the same effect
as using a OneToOneField.
--- lib/toaster/bldcontrol/bbcontroller.py | 4 ++-- lib/toaster/bldcontrol/localhostbecontroller.py | 13 ++++++------- lib/toaster/bldcontrol/management/commands/runbuilds.py | 2 +- lib/toaster/bldcontrol/models.py | 2 +- lib/toaster/bldcontrol/tests.py | 6 +++--- 5 files changed, 13 insertions(+), 14 deletions(-)
lib/toaster/bldcontrol/bbcontroller.py +2 -2
@@ -141,10 +141,10 @@ class BuildEnvironmentController(object): raise Exception("FIXME: Must override in order to actually start the BB server") - def setLayers(self, bbs, ls): + def setLayers(self, bitbake, ls): """ Checks-out bitbake executor and layers from git repositories. Sets the layer variables in the config file, after validating local layer paths. - The bitbakes must be a 1-length list of BRBitbake + bitbake must be a single BRBitbake instance The layer paths must be in a list of BRLayer object a word of attention: by convention, the first layer for any build will be poky!
lib/toaster/bldcontrol/localhostbecontroller.py +6 -7
@@ -115,18 +115,17 @@ class LocalhostBEController(BuildEnvironmentController): return local_checkout_path - def setLayers(self, bitbakes, layers, targets): + def setLayers(self, bitbake, layers, targets): """ a word of attention: by convention, the first layer for any build will be poky! """ assert self.be.sourcedir is not None - assert len(bitbakes) == 1 # set layers in the layersource # 1. get a list of repos with branches, and map dirpaths for each layer gitrepos = {} - gitrepos[(bitbakes[0].giturl, bitbakes[0].commit)] = [] - gitrepos[(bitbakes[0].giturl, bitbakes[0].commit)].append( ("bitbake", bitbakes[0].dirpath) ) + gitrepos[(bitbake.giturl, bitbake.commit)] = [] + gitrepos[(bitbake.giturl, bitbake.commit)].append( ("bitbake", bitbake.dirpath) ) for layer in layers: # we don't process local URLs
@@ -198,7 +197,7 @@ class LocalhostBEController(BuildEnvironmentController): # make sure we have a working bitbake if not os.path.exists(os.path.join(self.pokydirname, 'bitbake')): logger.debug("localhostbecontroller: checking bitbake into the poky dirname %s " % self.pokydirname) - self._shellcmd("git clone -b \"%s\" \"%s\" \"%s\" " % (bitbakes[0].commit, bitbakes[0].giturl, os.path.join(self.pokydirname, 'bitbake'))) + self._shellcmd("git clone -b \"%s\" \"%s\" \"%s\" " % (bitbake.commit, bitbake.giturl, os.path.join(self.pokydirname, 'bitbake'))) # verify our repositories for name, dirpath in gitrepos[(giturl, commit)]:
@@ -224,7 +223,7 @@ class LocalhostBEController(BuildEnvironmentController): for target in targets: try: customrecipe = CustomImageRecipe.objects.get(name=target.target, - project=bitbakes[0].req.project) + project=bitbake.req.project) except CustomImageRecipe.DoesNotExist: continue # not a custom recipe, skip
@@ -278,7 +277,7 @@ class LocalhostBEController(BuildEnvironmentController): def triggerBuild(self, bitbake, layers, variables, targets): - # set up the buid environment with the needed layers + # set up the build environment with the needed layers self.setLayers(bitbake, layers, targets) # get the bb server running with the build req id and build env id
lib/toaster/bldcontrol/management/commands/runbuilds.py +1 -1
@@ -57,7 +57,7 @@ class Command(NoArgsCommand): br.save() # this triggers an async build - bec.triggerBuild(br.brbitbake_set.all(), br.brlayer_set.all(), br.brvariable_set.all(), br.brtarget_set.all()) + bec.triggerBuild(br.brbitbake, br.brlayer_set.all(), br.brvariable_set.all(), br.brtarget_set.all()) except Exception as e: logger.error("runbuilds: Error launching build %s" % e)
lib/toaster/bldcontrol/models.py +1 -1
@@ -106,7 +106,7 @@ class BRLayer(models.Model): layer_version = models.ForeignKey(Layer_Version, null=True) class BRBitbake(models.Model): - req = models.ForeignKey(BuildRequest, unique = True) # only one bitbake for a request + req = models.OneToOneField(BuildRequest) # only one bitbake for a request giturl = models.CharField(max_length =254) commit = models.CharField(max_length = 254) dirpath = models.CharField(max_length = 254)
lib/toaster/bldcontrol/tests.py +3 -3
@@ -18,7 +18,7 @@ import subprocess import os # standard poky data hardcoded for testing -BITBAKE_LAYERS = [type('bitbake_info', (object,), { "giturl": "git://git.yoctoproject.org/poky.git", "dirpath": "", "commit": "HEAD"})] +BITBAKE_LAYER = type('bitbake_info', (object,), { "giturl": "git://git.yoctoproject.org/poky.git", "dirpath": "", "commit": "HEAD"}) POKY_LAYERS = [ type('poky_info', (object,), { "name": "meta", "giturl": "git://git.yoctoproject.org/poky.git", "dirpath": "meta", "commit": "HEAD"}), type('poky_info', (object,), { "name": "meta-yocto", "giturl": "git://git.yoctoproject.org/poky.git", "dirpath": "meta-yocto", "commit": "HEAD"}),
@@ -53,7 +53,7 @@ class BEControllerTests(object): bc = self._getBEController(obe) try: # setting layers, skip any layer info - bc.setLayers(BITBAKE_LAYERS, POKY_LAYERS) + bc.setLayers(BITBAKE_LAYER, POKY_LAYERS) except NotImplementedException, e: print "Test skipped due to command not implemented yet" return True
@@ -80,7 +80,7 @@ class BEControllerTests(object): layerSet = False try: # setting layers, skip any layer info - layerSet = bc.setLayers(BITBAKE_LAYERS, POKY_LAYERS) + layerSet = bc.setLayers(BITBAKE_LAYER, POKY_LAYERS) except NotImplementedException: print "Test skipped due to command not implemented yet" return True
Signature · 1 line
1.9.1
PATCH 14/18 toaster: fixed format strings
In reply to brian averyFrom: Ed Bartosh <[email protected]>
Printing id as a number breaks toaster if object is not saved yet as id/pk is None in this case.
Changed format spec to print id as a string in __str__ and __unicode__ methods of Build, Task and LayerVersion models.
--- lib/toaster/orm/models.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
lib/toaster/orm/models.py +3 -3
@@ -370,7 +370,7 @@ class Build(models.Model): return self.get_outcome_text() def __str__(self): - return "%d %s %s" % (self.id, self.project, ",".join([t.target for t in self.target_set.all()])) + return "%s %s %s" % (self.id, self.project, ",".join([t.target for t in self.target_set.all()])) # an Artifact is anything that results from a Build, and may be of interest to the user, and is not stored elsewhere
@@ -552,7 +552,7 @@ class Task(models.Model): sstate_text = property(get_sstate_text) def __unicode__(self): - return "%d(%d) %s:%s" % (self.pk, self.build.pk, self.recipe.name, self.task_name) + return "%s(%s) %s:%s" % (self.pk, self.build.pk, self.recipe.name, self.task_name) class Meta: ordering = ('order', 'recipe' ,)
@@ -1203,7 +1203,7 @@ class Layer_Version(models.Model): return sorted(result, key=lambda x: x.layer.name) def __unicode__(self): - return "%d %s (VCS %s, Project %s)" % (self.pk, str(self.layer), self.get_vcs_reference(), self.build.project if self.build is not None else "No project") + return "%s %s (VCS %s, Project %s)" % (self.pk, str(self.layer), self.get_vcs_reference(), self.build.project if self.build is not None else "No project") class Meta: unique_together = ("layer_source", "up_id")
Signature · 1 line
1.9.1
PATCH 15/18 toaster: work around 'database is locked' error
In reply to brian averyFrom: Ed Bartosh <[email protected]>
When sqlite can not cope with a stream of 'insert' queries it throws 'database is locked' exception.
Wrapping model.save in transaction.atomic context and repeating the call should solve this issue.
--- lib/toaster/orm/models.py | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
lib/toaster/orm/models.py +19 -1
@@ -29,10 +29,28 @@ from django.core import validators from django.conf import settings import django.db.models.signals - import logging logger = logging.getLogger("toaster") +if 'sqlite' in settings.DATABASES['default']['ENGINE']: + from django.db import transaction, OperationalError + from time import sleep + + _base_save = models.Model.save + def save(self, *args, **kwargs): + while True: + try: + with transaction.atomic(): + return _base_save(self, *args, **kwargs) + except OperationalError as err: + if 'database is locked' in str(err): + logger.warning("%s, model: %s, args: %s, kwargs: %s", + err, self.__class__, args, kwargs) + sleep(0.5) + continue + raise + + models.Model.save = save class GitURLValidator(validators.URLValidator): import re
Signature · 1 line
1.9.1
PATCH 16/18 toaster: removed extra calls of migrate
In reply to brian averyFrom: Ed Bartosh <[email protected]>
One manage.py migrate should be enough to apply migrations for all applications. Extra calls just slow toaster script down.
Removed calls of migrate for orm and bldcontrol apps.
--- bin/toaster | 20 +------------------- 1 file changed, 1 insertion(+), 19 deletions(-)
bin/toaster +1 -19
@@ -58,26 +58,8 @@ webserverStartAll() # python bitbake/lib/toaster/manage.py python manage.py createsuperuser --username=<ME> python $BBBASEDIR/lib/toaster/manage.py migrate --noinput || retval=1 - python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2 - - if [ $retval -eq 1 ]; then - echo "Failed db sync, aborting system start" 1>&2 - return $retval - fi - - python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1 - - if [ $retval -eq 1 ]; then - printf "\nError on orm migration, rolling back...\n" - python $BBBASEDIR/lib/toaster/manage.py migrate orm 0001_initial --fake - return $retval - fi - - python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1 - if [ $retval -eq 1 ]; then - printf "\nError on bldcontrol migration, rolling back...\n" - python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol 0001_initial --fake + echo "Failed migrations, aborting system start" 1>&2 return $retval fi
Signature · 1 line
1.9.1
PATCH 17/18 toaster: monkey patch Queryset
In reply to brian averyFrom: Ed Bartosh <[email protected]>
Fixed 'database is locked' issue by monkey patching django QuerySet methods.
The actual patching places were found by bisecting Django codebase.
This commit should be removed after Django is fixed if it's fixed at all.
--- lib/toaster/orm/models.py | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
lib/toaster/orm/models.py +30 -0
@@ -52,6 +52,36 @@ if 'sqlite' in settings.DATABASES['default']['ENGINE']: models.Model.save = save + # HACK: Monkey patch Django to fix 'database is locked' issue + + from django.db.models.query import QuerySet + _base_insert = QuerySet._insert + def _insert(self, *args, **kwargs): + with transaction.atomic(using=self.db, savepoint=False): + return _base_insert(self, *args, **kwargs) + QuerySet._insert = _insert + + from django.utils import six + def _create_object_from_params(self, lookup, params): + """ + Tries to create an object using passed params. + Used by get_or_create and update_or_create + """ + try: + obj = self.create(**params) + return obj, True + except IntegrityError: + exc_info = sys.exc_info() + try: + return self.get(**lookup), False + except self.model.DoesNotExist: + pass + six.reraise(*exc_info) + + QuerySet._create_object_from_params = _create_object_from_params + + # end of HACK + class GitURLValidator(validators.URLValidator): import re regex = re.compile(
Signature · 1 line
1.9.1
PATCH 18/18 toaster: rework checking of Django version
In reply to brian averyFrom: Ed Bartosh <[email protected]>
Checked django version considering information from toaster-requirements.txt, e.g. if requirements file contains line "Django>1.8,<1.9" toaster should be able to check that requirement correctly.
--- bin/toaster | 24 ++++++++++-------------- toaster-requirements.txt | 2 +- 2 files changed, 11 insertions(+), 15 deletions(-)
bin/toaster +10 -14
@@ -156,16 +156,15 @@ notify_chldexit() { verify_prereq() { - # Verify prerequisites - regex="([0-9])\.([0-9])" - for line in `cat $BBBASEDIR/toaster-requirements.txt | grep Django` ; do - [[ `echo $line` =~ $regex ]] - required_major_version="${BASH_REMATCH[1]}" - required_minor_version="${BASH_REMATCH[2]}" - done - - if ! echo "import django; print ($required_major_version,) == django.VERSION[0:1] and django.VERSION[1:2][0] in ($required_minor_version,)" | python 2>/dev/null | grep True >/dev/null; then - printf "This program needs Django $required_major_version.$required_minor_version. Please install with\n\npip install -r $BBBASEDIR/toaster-requirements.txt\n" + # Verify Django version + reqfile=$(python -c "import os; print os.path.realpath('$BBBASEDIR/toaster-requirements.txt')") + exp='s/Django\([><=]\+\)\([^,]\+\),\([><=]\+\)\(.\+\)/' + exp=$exp'import sys,django;version=django.get_version().split(".");' + exp=$exp'sys.exit(not (version \1 "\2".split(".") and version \3 "\4".split(".")))/p' + if ! sed -n "$exp" $reqfile | python - ; then + req=`grep ^Django $reqfile` + echo "This program needs $req" + echo "Please install with pip install -r $reqfile" return 2 fi
@@ -265,10 +264,7 @@ if [ "$1" = 'restart-bitbake' ] ; then return $rc fi -if ! verify_prereq; then - echo "Error: Could not verify that the needed dependencies are installed. Please use virtualenv and pip to install dependencies listed in toaster-requirements.txt" 1>&2 - return 1 -fi +verify_prereq || return 1 # We make sure we're running in the current shell and in a good environment if [ -z "$BUILDDIR" ] || ! which bitbake >/dev/null 2>&1 ; then
toaster-requirements.txt +1 -1
@@ -1,4 +1,4 @@ -Django==1.8.6 +Django>1.8,<1.9 argparse==1.2.1 wsgiref==0.1.2 beautifulsoup4>=4.4.0
Signature · 1 line
1.9.1