Skip to contentSkip to conversation

Fix handling of "default" project

PATCH Superseded Elliot Smith 1 participant 7 messages Aug 28, 2015

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Fix handling of "default" project

This fixes how we handle the default project used to house command line builds. It originally started from a bug encountered when using MySQL for the db back-end, but fixing that revealed some other issues which needed to be resolved on top.

Instead of relying on an ID of 0 as being the identifier for the default project, use an explicit is_default flag in the project table.

Because the default project is now added when the db for the app is initialised, this required some changes to how we manage project visibility and landing page redirects based on the state of projects and builds.

Changes since poky master 393bd7496d71eb101f21234c1233a2d18fd2c73e are in
git://git.yoctoproject.org/poky-contrib, elliot/toaster/mysql-migrations
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/mysql-migrations

Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=7932

Elliot Smith (6):
toaster: Make 0021 migration compatible with MySQL
toaster: Improve how default project is identified and fetched
toaster: Only redirect to projects page if user has added projects
toaster: Exclude default project unless it has builds
toaster: Fix test for projects page JSON
toaster: Add tests for pages which show the default project

 bitbake/lib/bb/ui/buildinfohelper.py               |   3 +-
 ...ect__chg_field_project_bitbake_version__chg_.py |   4 +-
 .../0025_auto__add_field_project_is_default.py     | 339 +++++++++++++++++++
 .../orm/migrations/0026_add_default_project.py     | 357 +++++++++++++++++++++
 bitbake/lib/toaster/orm/models.py                  |  18 +-
 bitbake/lib/toaster/toastergui/tests.py            | 124 ++++++-
 bitbake/lib/toaster/toastergui/views.py            |  14 +-
 7 files changed, 846 insertions(+), 13 deletions(-)
 create mode 100644 bitbake/lib/toaster/orm/migrations/0025_auto__add_field_project_is_default.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0026_add_default_project.py
Signature · 14 lines
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#1 lore raw

PATCH 1/6 toaster: Make 0021 migration compatible with MySQL

Two issues prevent this migration from working correctly with a MySQL back-end:

1. MySQL won't allow a default value to be set for an AutoField, which is what the migration tries to do
for project_id ("ValueError: The database backend does not accept 0 as a value for AutoField.")

2. When migrations are applied to a MySQL back-end, Django (via South) attempts a dry run of the migration first: it applies the forward migration then rolls it back. However, this migration raises an exception on roll back, which causes the whole series of migrations to fail.

This commit fixes both issues.

[YOCTO #7932]

---
 ...hg_field_build_project__chg_field_project_bitbake_version__chg_.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/toaster/orm/migrations/0021_auto__chg_field_build_project__chg_field_project_bitbake_version__chg_.py +3 -1
index 924a5c4..a62ddb7 100644
@@ -7,10 +7,12 @@ from django.db import models
 
 class Migration(SchemaMigration):
 
+    no_dry_run = True
+
     def forwards(self, orm):
 
         # Changing field 'Build.project'
-        db.alter_column(u'orm_build', 'project_id', self.gf('django.db.models.fields.related.ForeignKey')(default=0, to=orm['orm.Project']))
+        db.alter_column(u'orm_build', 'project_id', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['orm.Project']))
 
         # Changing field 'Project.bitbake_version'
         db.alter_column(u'orm_project', 'bitbake_version_id', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['orm.BitbakeVersion'], null=True))
Signature · 14 lines
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#2 lore raw

PATCH 2/6 toaster: Improve how default project is identified and fetched

In reply to Elliot Smith

Command line builds are associated with a "default project" (as we currently require a build to have a project). This acts as a container for builds initiated outside Toaster. Currently, this project is marked as the default by its ID being 0. However, this doesn't work with MySQL, as MySQL won't allow 0 in a foreign key which references an autoincrement field.

Instead, use an is_default field to track the default Project for builds initiated outside Toaster.

Add a method to fetch this default project, rather than fetching a project with a magic ID.

Add this default project in a migration, rather than as a side effect of a get_or_create() style method.

Also ensure that builds always have a project explicitly assigned to avoid any magic with a build's project foreign key defaulting to 0 (as it no longer does).

[YOCTO #7932]

---
 bitbake/lib/bb/ui/buildinfohelper.py               |   3 +-
 .../0025_auto__add_field_project_is_default.py     | 339 +++++++++++++++++++
 .../orm/migrations/0026_add_default_project.py     | 357 +++++++++++++++++++++
 bitbake/lib/toaster/orm/models.py                  |  18 +-
 4 files changed, 711 insertions(+), 6 deletions(-)
 create mode 100644 bitbake/lib/toaster/orm/migrations/0025_auto__add_field_project_is_default.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0026_add_default_project.py
bitbake/lib/bb/ui/buildinfohelper.py +2 -1
index 09e8853..2d1ed51 100644
@@ -145,13 +145,14 @@ class ORMWrapper(object):
             prj = Project.objects.get(pk = project_id)
 
         else:                           # this build was triggered by a legacy system, or command line interactive mode
-            prj, _ = Project.objects.get_or_create(pk=0, name="Default Project")
+            prj = Project.objects.get_default_project()
             logger.debug(1, "buildinfohelper: project is not specified, defaulting to %s" % prj)
 
 
         if buildrequest is not None:
             build = buildrequest.build
             logger.info("Updating existing build, with %s", build_info)
+            build.project = prj
             build.machine=build_info['machine']
             build.distro=build_info['distro']
             build.distro_version=build_info['distro_version']
bitbake/lib/toaster/orm/migrations/0025_auto__add_field_project_is_default.py new file +339 -0

340 lines not on this page · raw message

bitbake/lib/toaster/orm/migrations/0026_add_default_project.py new file +357 -0

357 lines not on this page · raw message

bitbake/lib/toaster/orm/models.py +13 -5
index 92fcaa7..c5d06f4 100644
@@ -94,11 +94,15 @@ class ProjectManager(models.Manager):
     def create(self, *args, **kwargs):
         raise Exception("Invalid call to Project.objects.create. Use Project.objects.create_project() to create a project")
 
-    def get_or_create(self, **kwargs):
-        # allow project creation for default data
-        if 'pk' in kwargs and kwargs['pk'] == 0:
-            return super(ProjectManager, self).get_or_create(**kwargs)
-        raise Exception("Invalid call to Project.objects.get_or_create. Use Project.objects.create_project() to create a project")
+    # return single object with is_default = True
+    def get_default_project(self):
+        projects = super(ProjectManager, self).filter(is_default = True)
+        if len(projects) > 1:
+            raise Exception("Inconsistent project data: multiple " +
+                            "default projects (i.e. with is_default=True)")
+        elif len(projects) < 1:
+            raise Exception("Inconsistent project data: no default project found")
+        return projects[0]
 
 class Project(models.Model):
     search_allowed_fields = ['name', 'short_description', 'release__name', 'release__branch_name']
@@ -115,6 +119,10 @@ class Project(models.Model):
     user_id     = models.IntegerField(null = True)
     objects     = ProjectManager()
 
+    # set to True for the project which is the default container
+    # for builds initiated by the command line etc.
+    is_default  = models.BooleanField(default = False)
+
     def __unicode__(self):
         return "%s (Release %s, BBV %s)" % (self.name, self.release, self.bitbake_version)
 
Signature · 14 lines
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#3 lore raw

PATCH 3/6 toaster: Only redirect to projects page if user has added projects

In reply to Elliot Smith

The landing page currently redirects the user if there are any projects in the db. Because we now always have at least one (the default one added by a migration), we always get the redirect.

Change this so that when the user hits the landing page, we only redirect them to the projects page if there is at least one user-added project and there are no builds.

[YOCTO #7932]

---
 bitbake/lib/toaster/toastergui/views.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
bitbake/lib/toaster/toastergui/views.py +5 -1
index 98d21f4..6b1f9d4 100755
@@ -53,7 +53,11 @@ logger = logging.getLogger("toaster")
 # all new sessions should come through the landing page;
 # determine in which mode we are running in, and redirect appropriately
 def landing(request):
-    if Build.objects.count() == 0 and Project.objects.count() > 0:
+    # we only redirect to projects page if there is a user-generated project
+    user_projects = Project.objects.filter(is_default = False)
+    has_user_project = user_projects.count() > 0
+
+    if Build.objects.count() == 0 and has_user_project:
         return redirect(reverse('all-projects'), permanent = False)
 
     if Build.objects.all().count() > 0:
Signature · 14 lines
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#4 lore raw

PATCH 4/6 toaster: Exclude default project unless it has builds

In reply to Elliot Smith

Don't include the default "command line builds" project in the projects view unless it has builds associated with it.

[YOCTO #7932]

---
 bitbake/lib/toaster/toastergui/views.py | 8 ++++++++
 1 file changed, 8 insertions(+)
bitbake/lib/toaster/toastergui/views.py +8 -0
index 6b1f9d4..8c8a4bb 100755
@@ -2796,6 +2796,14 @@ if True:
 
         queryset_all = Project.objects.all()
 
+        # annotate each project with its number of builds
+        queryset_all = queryset_all.annotate(num_builds=Count('build'))
+
+        # exclude the command line builds project if it has no builds
+        q_default_with_builds = Q(is_default=True) & Q(num_builds__gt=0)
+        queryset_all = queryset_all.filter(Q(is_default=False) |
+                                           q_default_with_builds)
+
         # boilerplate code that takes a request for an object type and returns a queryset
         # for that object type. copypasta for all needed table searches
         (filter_string, search_term, ordering_string) = _search_tuple(request, Project)
Signature · 14 lines
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#5 lore raw

PATCH 5/6 toaster: Fix test for projects page JSON

In reply to Elliot Smith

The tests expect data to be returned in a particular format, which doesn't include is_default or the num_builds fields added to support default project improvements.

Add the extra fields so the tests pass again.

[YOCTO #7932]

---
 bitbake/lib/toaster/toastergui/tests.py | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
bitbake/lib/toaster/toastergui/tests.py +5 -4
index 85e27fe..7acf4bc 100644
@@ -40,6 +40,7 @@ class ViewTests(TestCase):
                                          bitbake_version=bbv)
         self.project = Project.objects.create_project(name=PROJECT_NAME,
                                                       release=release)
+
         layersrc = LayerSource.objects.create(sourcetype=LayerSource.TYPE_IMPORTED)
         self.priority = ReleaseLayerSourcePriority.objects.create(release=release,
                                                                   layer_source=layersrc)
@@ -86,10 +87,10 @@ class ViewTests(TestCase):
 
         self.assertEqual(sorted(data["rows"][0]),
                          ['bitbake_version_id', 'created', 'id',
-                          'layersTypeAheadUrl', 'name', 'projectBuildsUrl',
-                          'projectPageUrl', 'recipesTypeAheadUrl',
-                          'release_id', 'short_description', 'updated',
-                          'user_id'])
+                          'is_default', 'layersTypeAheadUrl', 'name',
+                          'num_builds', 'projectBuildsUrl', 'projectPageUrl',
+                          'recipesTypeAheadUrl', 'release_id',
+                          'short_description', 'updated', 'user_id'])
 
     def test_typeaheads(self):
         """Test typeahead ReST API"""
Signature · 14 lines
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#6 lore raw

PATCH 6/6 toaster: Add tests for pages which show the default project

In reply to Elliot Smith

Test that the correct landing page redirect is applied, depending on the state of the default project.

Test that the default project is only shown on the /projects page if it has at least one build.

[YOCTO #7932]

---
 bitbake/lib/toaster/toastergui/tests.py | 115 +++++++++++++++++++++++++++++++-
 1 file changed, 114 insertions(+), 1 deletion(-)
bitbake/lib/toaster/toastergui/tests.py +114 -1
index 7acf4bc..1a8b478 100644
@@ -23,7 +23,8 @@
 
 from django.test import TestCase
 from django.core.urlresolvers import reverse
-from orm.models import Project, Release, BitbakeVersion
+from django.utils import timezone
+from orm.models import Project, Release, BitbakeVersion, Build
 from orm.models import ReleaseLayerSourcePriority, LayerSource, Layer
 from orm.models import Layer_Version, Recipe, Machine, ProjectLayer
 import json
@@ -179,3 +180,115 @@ class ViewTests(TestCase):
         response = self.client.post(reverse('xhr_importlayer'), args)
         data = json.loads(response.content)
         self.assertNotEqual(data["error"], "ok")
+
+class LandingPageTests(TestCase):
+    """ Tests for redirects on the landing page """
+    # disable bogus pylint message error:
+    # "Instance of 'WSGIRequest' has no 'url' member (no-member)"
+    # (see https://github.com/landscapeio/pylint-django/issues/42)
+    # pylint: disable=E1103
+
+    LANDING_PAGE_TITLE = 'This is Toaster'
+
+    def setUp(self):
+        """ Add default project manually """
+        self.project = Project.objects.create_project('foo', None)
+        self.project.is_default = True
+        self.project.save()
+
+    def test_only_default_project(self):
+        """
+        No projects except default
+        => get the landing page
+        """
+        response = self.client.get(reverse('landing'))
+        self.assertTrue(self.LANDING_PAGE_TITLE in response.content)
+
+    def test_default_project_has_build(self):
+        """
+        Default project has a build, no other projects
+        => get the builds page
+        """
+        now = timezone.now()
+        build = Build.objects.create(project=self.project,
+                                     started_on=now,
+                                     completed_on=now)
+        build.save()
+
+        response = self.client.get(reverse('landing'))
+        self.assertEqual(response.status_code, 302,
+                         'response should be a redirect')
+        self.assertTrue('/builds' in response.url,
+                        'should redirect to builds')
+
+    def test_user_project_exists(self):
+        """
+        User has added a project (without builds)
+        => get the projects page
+        """
+        user_project = Project.objects.create_project('foo', None)
+        user_project.save()
+
+        response = self.client.get(reverse('landing'))
+        self.assertEqual(response.status_code, 302,
+                         'response should be a redirect')
+        self.assertTrue('/projects' in response.url,
+                        'should redirect to projects')
+
+    def test_user_project_has_build(self):
+        """
+        User has added a project (with builds)
+        => get the builds page
+        """
+        user_project = Project.objects.create_project('foo', None)
+        user_project.save()
+
+        now = timezone.now()
+        build = Build.objects.create(project=user_project,
+                                     started_on=now,
+                                     completed_on=now)
+        build.save()
+
+        response = self.client.get(reverse('landing'))
+        self.assertEqual(response.status_code, 302,
+                         'response should be a redirect')
+        self.assertTrue('/builds' in response.url,
+                        'should redirect to builds')
+
+class ProjectsPageTests(TestCase):
+    """ Tests for projects page """
+
+    PROJECT_NAME = 'cli builds'
+
+    def setUp(self):
+        """ Add default project manually """
+        project = Project.objects.create_project(self.PROJECT_NAME, None)
+        self.default_project = project
+        self.default_project.is_default = True
+        self.default_project.save()
+
+    def test_default_project_hidden(self):
+        """ The default project should be hidden if it has no builds """
+        params = {"count": 10, "orderby": "updated:-", "page": 1}
+        response = self.client.get(reverse('all-projects'), params)
+
+        self.assertTrue(not('tr class="data"' in response.content),
+                        'should be no project rows in the page')
+        self.assertTrue(not(self.PROJECT_NAME in response.content),
+                        'default project "cli builds" should not be in page')
+
+    def test_default_project_has_build(self):
+        """ The default project should be shown if it has builds """
+        now = timezone.now()
+        build = Build.objects.create(project=self.default_project,
+                                     started_on=now,
+                                     completed_on=now)
+        build.save()
+
+        params = {"count": 10, "orderby": "updated:-", "page": 1}
+        response = self.client.get(reverse('all-projects'), params)
+
+        self.assertTrue('tr class="data"' in response.content,
+                        'should be a project row in the page')
+        self.assertTrue(self.PROJECT_NAME in response.content,
+                        'default project "cli builds" should be in page')
Signature · 14 lines
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.