Skip to contentSkip to conversation

Build cancellation

PATCH In review Ed Bartosh 4 participants 27 messages Mar 17, 2016 – Mar 22, 2016

lore
Share & tools
lore

Ed Bartosh

author
#0 lore raw

cover Build cancellation

This is build cancelation functionality implemented by Sujith and Michael and modified by me to work on top of 'per project build directory' patchset.

NOTE: This patchset depends on 'per project build directory' patchset' v5:

      https://lists.yoctoproject.org/pipermail/toaster/2016-March/004170.html

The following changes since commit e4ba8a50641b6a2ad8ccda5fc62c61fdf7cef733:

toasterui: shutdown on BuildCompleted event (2016-03-17 17:31:00 +0200)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/toaster/project-build-dir-7880-cancel
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/project-build-dir-7880-cancel

Ed Bartosh (2):
toaster: fix conflicting migrations
toaster: use empty token

Michael Wood (9):
toaster: Move xhr calls for starting and stopping builds
toaster: xhr Update the implementation of the build cancellation
request
toaster: libtoaster Update implementation of startABuild and
cancelABuild
toaster: bldcontrol models Add a cancelling state the BuildRequest
buildinfohelper: Add handler for cancelling a build
toaster: mrb_section template Add build cancel button
toaster: models Exclude the CANCELLED builds from get_number_of_builds
toaster: runbuilds Make runbuilds aware of the build CANCELLED state
toaster: runbuilds Clean up runbuilds

Sujith H (4):
toaster: bldcontrol Add forceShutDown function to BitbakeController
toaster: update BuildEnvironmentController and BitbakeController
toaster: models Add cancelled state to build outcome
toaster: tables BuildsTable exclude cancelled builds

 bitbake/lib/bb/ui/buildinfohelper.py               |  19 ++-
 bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  29 +---
 .../toaster/bldcontrol/localhostbecontroller.py    |   4 +-
 .../bldcontrol/management/commands/runbuilds.py    | 107 ++++++++++-----
 .../migrations/0002_add_cancelling_state.py        |  19 +++
 .../toaster/bldcontrol/migrations/0003_merge.py    |  15 +++
 bitbake/lib/toaster/bldcontrol/models.py           |  27 +++-
 .../orm/migrations/0006_add_cancelled_state.py     |  19 +++
 bitbake/lib/toaster/orm/migrations/0007_merge.py   |  15 +++
 bitbake/lib/toaster/orm/models.py                  |  12 +-
 bitbake/lib/toaster/toastergui/api.py              | 110 +++++++++++++++
 .../toaster/toastergui/static/js/customrecipe.js   |   4 +-
 .../lib/toaster/toastergui/static/js/layerBtn.js   |   3 +-
 .../lib/toaster/toastergui/static/js/libtoaster.js |  41 +++---
 .../lib/toaster/toastergui/static/js/mrbsection.js |  95 +++++++++++++
 .../toaster/toastergui/static/js/projectpage.js    |   4 +-
 .../toaster/toastergui/static/js/projecttopbar.js  |   6 +-
 .../toaster/toastergui/static/js/recipedetails.js  |   4 +-
 bitbake/lib/toaster/toastergui/tables.py           |  48 +------
 bitbake/lib/toaster/toastergui/templates/base.html |   1 +
 .../toaster/toastergui/templates/mrb_section.html  | 148 ++++++++-------------
 bitbake/lib/toaster/toastergui/urls.py             |   5 +
 22 files changed, 507 insertions(+), 228 deletions(-)
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_add_cancelling_state.py
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0003_merge.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0006_add_cancelled_state.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0007_merge.py
 create mode 100644 bitbake/lib/toaster/toastergui/api.py
 create mode 100644 bitbake/lib/toaster/toastergui/static/js/mrbsection.js

--
Regards,
Ed

author
#1 lore raw

PATCH 1/15 toaster: bldcontrol Add forceShutDown function to BitbakeController

From: Sujith H <[email protected]>

Add forceShutDown function to BitbakeController class. This function provides a mechanism to cancel the build from toaster. An API which can be used safely to cancel build and hence shutdown running bitbake server.

[YOCTO #6787]

---
 bitbake/lib/toaster/bldcontrol/bbcontroller.py | 3 +++
 1 file changed, 3 insertions(+)
bitbake/lib/toaster/bldcontrol/bbcontroller.py +3 -0
index 058e490..0f7b66d 100644
@@ -63,6 +63,9 @@ class BitbakeController(object):
             task = "build"
         return self._runCommand(["buildTargets", targets, task])
 
+    def forceShutDown(self):
+        return self._runCommand(["stateForceShutdown"])
+
 
 
 def getBuildEnvironmentController(**kwargs):
Signature · 1 line
2.1.4
author
#2 lore raw

PATCH 2/15 toaster: Move xhr calls for starting and stopping builds

In reply to Ed Bartosh

From: Michael Wood <[email protected]>

Move the backend xhr implementation of the build request changes into it's own file and out of the ToasterTable definition.
It used to live in the views.py but in a hope of starting to collate logical groups of views move this to a new file called api.

---
 bitbake/lib/toaster/toastergui/api.py    | 74 ++++++++++++++++++++++++++++++++
 bitbake/lib/toaster/toastergui/tables.py | 43 +------------------
 bitbake/lib/toaster/toastergui/urls.py   |  5 +++
 3 files changed, 80 insertions(+), 42 deletions(-)
 create mode 100644 bitbake/lib/toaster/toastergui/api.py
bitbake/lib/toaster/toastergui/api.py new file +74 -0
new file mode 100644
index 0000000..1b1f598
@@ -0,0 +1,74 @@
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2016        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+
+# Temporary home for the UI's misc API
+
+from orm.models import Project, ProjectTarget
+from bldcontrol.models import BuildRequest
+from bldcontrol import bbcontroller
+from django.http import HttpResponse, JsonResponse
+from django.views.generic import View
+
+
+class XhrBuildRequest(View):
+
+    def get(self, request, *args, **kwargs):
+        return HttpResponse()
+
+    def post(self, request, *args, **kwargs):
+        """ Process HTTP POSTs which make build requests """
+
+        project = Project.objects.get(pk=kwargs['pid'])
+
+        if 'buildCancel' in request.POST:
+            for i in request.POST['buildCancel'].strip().split(" "):
+                try:
+                    br = BuildRequest.objects.select_for_update().get(project = project, pk = i, state__lte = BuildRequest.REQ_QUEUED)
+                    br.state = BuildRequest.REQ_DELETED
+                    br.save()
+                except BuildRequest.DoesNotExist:
+                    pass
+
+        if 'buildDelete' in request.POST:
+            for i in request.POST['buildDelete'].strip().split(" "):
+                try:
+                    BuildRequest.objects.select_for_update().get(project = project, pk = i, state__lte = BuildRequest.REQ_DELETED).delete()
+                except BuildRequest.DoesNotExist:
+                    pass
+
+        if 'targets' in request.POST:
+            ProjectTarget.objects.filter(project = project).delete()
+            s = str(request.POST['targets'])
+            for t in s.translate(None, ";%|\"").split(" "):
+                if ":" in t:
+                    target, task = t.split(":")
+                else:
+                    target = t
+                    task = ""
+                ProjectTarget.objects.create(project = project,
+                                             target = target,
+                                             task = task)
+            project.schedule_build()
+
+        # redirect back to builds page so any new builds in progress etc.
+        # are visible
+        response = HttpResponse()
+        response.status_code = 302
+        response['Location'] = request.build_absolute_uri()
+        return response
bitbake/lib/toaster/toastergui/tables.py +1 -42

55 lines not on this page · raw message

bitbake/lib/toaster/toastergui/urls.py +5 -0
index 400580a..27b0baa 100644
@@ -22,6 +22,7 @@ from django.views.generic import RedirectView, TemplateView
 from django.http import HttpResponseBadRequest
 from toastergui import tables
 from toastergui import typeaheads
+from toastergui import api
 
 urlpatterns = patterns('toastergui.views',
         # landing page
@@ -180,6 +181,10 @@ urlpatterns = patterns('toastergui.views',
         url(r'^xhr_customrecipe/', 'xhr_customrecipe',
             name='xhr_customrecipe'),
 
+        url(r'^xhr_buildrequest/project/(?P<pid>\d+)$',
+           api.XhrBuildRequest.as_view(),
+            name='xhr_buildrequest'),
+
           # default redirection
         url(r'^$', RedirectView.as_view(url='landing', permanent=True)),
 )
Signature · 1 line
2.1.4
author
#3 lore raw

PATCH 3/15 toaster: xhr Update the implementation of the build cancellation request

In reply to Ed Bartosh

From: Michael Wood <[email protected]>

Update the implementation of the backend api for cancelling builds with the new cancelling BuildRequest state and cancelled Build state. Also added some docstring about general usage.

Co-Author: Sujith H <[email protected]>

---
 bitbake/lib/toaster/toastergui/api.py | 54 +++++++++++++++++++++++++++++------
 1 file changed, 45 insertions(+), 9 deletions(-)
bitbake/lib/toaster/toastergui/api.py +45 -9
index 1b1f598..42901f75 100644
@@ -19,7 +19,7 @@
 
 # Temporary home for the UI's misc API
 
-from orm.models import Project, ProjectTarget
+from orm.models import Project, ProjectTarget, Build
 from bldcontrol.models import BuildRequest
 from bldcontrol import bbcontroller
 from django.http import HttpResponse, JsonResponse
@@ -32,18 +32,54 @@ class XhrBuildRequest(View):
         return HttpResponse()
 
     def post(self, request, *args, **kwargs):
-        """ Process HTTP POSTs which make build requests """
+        """
+          Build control
+
+          Entry point: /xhr_buildrequest/<project_id>
+          Method: POST
+
+          Args:
+              id: id of build to change
+              buildCancel = build_request_id ...
+              buildDelete = id ...
+              targets = recipe_name ...
+
+          Returns:
+              {"error": "ok"}
+            or
+              {"error": <error message>}
+        """
 
         project = Project.objects.get(pk=kwargs['pid'])
 
         if 'buildCancel' in request.POST:
             for i in request.POST['buildCancel'].strip().split(" "):
                 try:
-                    br = BuildRequest.objects.select_for_update().get(project = project, pk = i, state__lte = BuildRequest.REQ_QUEUED)
-                    br.state = BuildRequest.REQ_DELETED
+                    br = BuildRequest.objects.get(project=project, pk=i)
+
+                    try:
+                        bbctrl = bbcontroller.BitbakeController(br.environment)
+                        bbctrl.forceShutDown()
+                    except:
+                        # We catch a bunch of exceptions here because
+                        # this is where the server has not had time to start up
+                        # and the build request or build is in transit between
+                        # processes.
+                        # We can safely just set the build as cancelled
+                        # already as it never got started
+                        build = br.build
+                        build.outcome = Build.CANCELLED
+                        build.save()
+
+                    # We now hand over to the buildinfohelper to update the
+                    # build state once we've finished cancelling
+                    br.state = BuildRequest.REQ_CANCELLING
                     br.save()
+
                 except BuildRequest.DoesNotExist:
-                    pass
+                    return JsonResponse({'error':'No such build id %s' % i})
+
+            return JsonResponse({'error': 'ok'})
 
         if 'buildDelete' in request.POST:
             for i in request.POST['buildDelete'].strip().split(" "):
@@ -51,6 +87,7 @@ class XhrBuildRequest(View):
                     BuildRequest.objects.select_for_update().get(project = project, pk = i, state__lte = BuildRequest.REQ_DELETED).delete()
                 except BuildRequest.DoesNotExist:
                     pass
+            return JsonResponse({'error': 'ok' })
 
         if 'targets' in request.POST:
             ProjectTarget.objects.filter(project = project).delete()
@@ -66,9 +103,8 @@ class XhrBuildRequest(View):
                                              task = task)
             project.schedule_build()
 
-        # redirect back to builds page so any new builds in progress etc.
-        # are visible
+            return JsonResponse({'error': 'ok' })
+
         response = HttpResponse()
-        response.status_code = 302
-        response['Location'] = request.build_absolute_uri()
+        response.status_code = 500
         return response
Signature · 1 line
2.1.4
author
#4 lore raw

PATCH 4/15 toaster: libtoaster Update implementation of startABuild and cancelABuild

In reply to Ed Bartosh

From: Michael Wood <[email protected]>

Update the implementation of startABuild and cancelAbuild to reflect changes to the backend api. We now have a dedicated endpoint to make calls into so add this url to libtoaster.ctx and allow passing null in as a url value to indicate that we want to use the current project

Also:
- Fix some documentation comments
- Add the convenience of passing in an array of targets to startABuild

---
 .../toaster/toastergui/static/js/customrecipe.js   |  4 +--
 .../lib/toaster/toastergui/static/js/layerBtn.js   |  3 +-
 .../lib/toaster/toastergui/static/js/libtoaster.js | 41 ++++++++++++++--------
 .../toaster/toastergui/static/js/projectpage.js    |  4 +--
 .../toaster/toastergui/static/js/projecttopbar.js  |  6 ++--
 .../toaster/toastergui/static/js/recipedetails.js  |  4 +--
 bitbake/lib/toaster/toastergui/templates/base.html |  1 +
 7 files changed, 34 insertions(+), 29 deletions(-)
bitbake/lib/toaster/toastergui/static/js/customrecipe.js +1 -3
index 3c57899..131ac2a 100644
@@ -207,9 +207,7 @@ function customRecipePageInit(ctx) {
 
   /* Trigger a build of your custom image */
   $(".build-custom-image").click(function(){
-    libtoaster.startABuild(libtoaster.ctx.projectBuildsUrl,
-      libtoaster.ctx.projectId,
-      ctx.recipe.name,
+    libtoaster.startABuild(null, ctx.recipe.name,
       function(){
         window.location.replace(libtoaster.ctx.projectBuildsUrl);
     });
bitbake/lib/toaster/toastergui/static/js/layerBtn.js +1 -2
index b2666ab..aa43284 100644
@@ -60,8 +60,7 @@ function layerBtnsInit() {
     e.preventDefault();
     var recipe = $(this).data('recipe-name');
 
-    libtoaster.startABuild(libtoaster.ctx.projectBuildsUrl,
-      libtoaster.ctx.projectId, recipe,
+    libtoaster.startABuild(null, recipe,
       function(){
         /* Success */
         window.location.replace(libtoaster.ctx.projectBuildsUrl);
bitbake/lib/toaster/toastergui/static/js/libtoaster.js +26 -15

75 lines not on this page · raw message

bitbake/lib/toaster/toastergui/static/js/projectpage.js +1 -3
index 6655a18..3013416 100644
@@ -232,9 +232,7 @@ function projectPageInit(ctx) {
 
     toBuild = toBuild.trim();
 
-    libtoaster.startABuild(libtoaster.ctx.projectBuildsUrl,
-      libtoaster.ctx.projectId,
-      toBuild,
+    libtoaster.startABuild(null, toBuild,
       function(){
         /* Build request started */
         window.location.replace(libtoaster.ctx.projectBuildsUrl);
bitbake/lib/toaster/toastergui/static/js/projecttopbar.js +3 -3
index 58a32a0..b09f974 100644
@@ -82,9 +82,9 @@ function projectTopBarInit(ctx) {
     selectedTarget = { name: newBuildTargetInput.val() };
 
     /* Fire off the build */
-    libtoaster.startABuild(libtoaster.ctx.projectBuildsUrl,
-      null, selectedTarget.name, function(){
-      window.location.replace(libtoaster.ctx.projectBuildsUrl);
+    libtoaster.startABuild(null, selectedTarget.name,
+      function(){
+        window.location.replace(libtoaster.ctx.projectBuildsUrl);
     }, null);
   });
 }
bitbake/lib/toaster/toastergui/static/js/recipedetails.js +1 -3
index 2bfd0a4..d5f9eac 100644
@@ -42,9 +42,7 @@ function recipeDetailsPageInit(ctx){
 
   /* Trigger a build of your custom image */
   $(".build-recipe-btn").click(function(){
-    libtoaster.startABuild(libtoaster.ctx.projectBuildsUrl,
-      libtoaster.ctx.projectId,
-      ctx.recipe.name,
+    libtoaster.startABuild(null, ctx.recipe.name,
       function(){
         window.location.replace(libtoaster.ctx.projectBuildsUrl);
     });
bitbake/lib/toaster/toastergui/templates/base.html +1 -0
index 121a75c..192f9fb 100644
@@ -47,6 +47,7 @@
         projectBuildsUrl: {% url 'projectbuilds' project.id as pburl %}{{pburl|json}},
         xhrCustomRecipeUrl : "{% url 'xhr_customrecipe' %}",
         projectId : {{project.id}},
+        xhrBuildRequestUrl: "{% url 'xhr_buildrequest' project.id %}",
         {% else %}
         projectId : undefined,
         projectPageUrl : undefined,
Signature · 1 line
2.1.4
author
#5 lore raw

PATCH 5/15 toaster: update BuildEnvironmentController and BitbakeController

In reply to Ed Bartosh

From: Sujith H <[email protected]>

Remove getBBController function from BuildEnvironmentController. The constructor of BitbakeController is updated appropriately so that call can be made to connect to running server.
The call to startBBServer is removed from bbcontroller and handledin localhostbecontroller.

[YOCTO #6787]

---
 bitbake/lib/toaster/bldcontrol/bbcontroller.py     | 26 ++++------------------
 .../toaster/bldcontrol/localhostbecontroller.py    |  2 +-
 2 files changed, 5 insertions(+), 23 deletions(-)
bitbake/lib/toaster/bldcontrol/bbcontroller.py +4 -22
index 0f7b66d..d09ac17 100644
@@ -37,11 +37,12 @@ class BitbakeController(object):
         It is outside the scope of this class on how the server is started and aquired
     """
 
-    def __init__(self, connection):
-        self.connection = connection
+    def __init__(self, be):
+        self.connection = bb.server.xmlrpc._create_server(be.bbaddress,
+                                                          int(be.bbport))[0]
 
     def _runCommand(self, command):
-        result, error = self.connection.connection.runCommand(command)
+        result, error = self.connection.runCommand(command)
         if error:
             raise Exception(error)
         return result
@@ -122,25 +123,6 @@ class BuildEnvironmentController(object):
         """
         raise NotImplementedError("FIXME: Must override setLayers")
 
-
-    def getBBController(self):
-        """ returns a BitbakeController to an already started server; this is the point where the server
-            starts if needed; or reconnects to the server if we can
-        """
-        if not self.connection:
-            self.be.lock = BuildEnvironment.LOCK_RUNNING
-            self.be.save()
-
-        server = bb.server.xmlrpc.BitBakeXMLRPCClient()
-        server.initServer()
-        server.saveConnectionDetails("%s:%s" % (self.be.bbaddress, self.be.bbport))
-        self.connection = server.establishConnection([])
-
-        self.be.bbtoken = self.connection.transport.connection_token
-        self.be.save()
-
-        return BitbakeController(self.connection)
-
     def getArtifact(self, path):
         """ This call returns an artifact identified by the 'path'. How 'path' is interpreted as
             up to the implementing BEC. The return MUST be a REST URL where a GET will actually return
bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +1 -1
index 480f945..6a6cff8 100644
@@ -32,7 +32,7 @@ import subprocess
 
 from toastermain import settings
 
-from bbcontroller import BuildEnvironmentController, ShellCmdException, BuildSetupException
+from bbcontroller import BuildEnvironmentController, ShellCmdException, BuildSetupException, BitbakeController
 
 import logging
 logger = logging.getLogger("toaster")
Signature · 1 line
2.1.4
author
#6 lore raw

PATCH 6/15 toaster: models Add cancelled state to build outcome

In reply to Ed Bartosh

From: Sujith H <[email protected]>

A new state CANCELLED is introduced to, distinguish the state of build.

[YOCTO #6787]

---
 .../orm/migrations/0006_add_cancelled_state.py        | 19 +++++++++++++++++++
 bitbake/lib/toaster/orm/models.py                     |  7 ++++++-
 2 files changed, 25 insertions(+), 1 deletion(-)
 create mode 100644 bitbake/lib/toaster/orm/migrations/0006_add_cancelled_state.py
bitbake/lib/toaster/orm/migrations/0006_add_cancelled_state.py new file +19 -0
new file mode 100644
index 0000000..f8ec65a
@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('orm', '0004_provides'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='build',
+            name='outcome',
+            field=models.IntegerField(default=2, choices=[(0, b'Succeeded'), (1, b'Failed'), (2, b'In Progress'), (3, b'Cancelled')]),
+        ),
+    ]
bitbake/lib/toaster/orm/models.py +6 -1
index 3c98c0b..e0c2096 100644
@@ -358,11 +358,13 @@ class Build(models.Model):
     SUCCEEDED = 0
     FAILED = 1
     IN_PROGRESS = 2
+    CANCELLED = 3
 
     BUILD_OUTCOME = (
         (SUCCEEDED, 'Succeeded'),
         (FAILED, 'Failed'),
         (IN_PROGRESS, 'In Progress'),
+        (CANCELLED, 'Cancelled'),
     )
 
     search_allowed_fields = ['machine', 'cooker_log_path', "target__target", "target__target_image_file__file_name"]
@@ -389,7 +391,10 @@ class Build(models.Model):
         if project:
             builds = builds.filter(project=project)
 
-        finished_criteria = Q(outcome=Build.SUCCEEDED) | Q(outcome=Build.FAILED)
+        finished_criteria = \
+                Q(outcome=Build.SUCCEEDED) | \
+                Q(outcome=Build.FAILED) | \
+                Q(outcome=Build.CANCELLED)
 
         recent_builds = list(itertools.chain(
             builds.filter(outcome=Build.IN_PROGRESS).order_by("-started_on"),
Signature · 1 line
2.1.4
author
#7 lore raw

PATCH 7/15 toaster: bldcontrol models Add a cancelling state the BuildRequest

In reply to Ed Bartosh

From: Michael Wood <[email protected]>

To accurately reflect the state of a build request we also need a cancelling state. This is set when we've started a build and then for whatever reason cancel it, cancelling is not instantaneous so we have this state to indicate that a cancel is in progress.

Also add a state transition guard. As the state of a BuildRequest can currently be modified by three processes; Toastergui, Runbuilds/bldcontrol and the buildinofhelper we cannot say for sure which process will be running at the time of cancellation so in order to avoid one of these processes making an incorrect transition only allow transitions of state to increase.

e.g. CREATED -> QUEUED -> INPROGRESS
And to ignore such requested changes such as
INPROGRESS -> CREATED

---
 .../migrations/0002_add_cancelling_state.py        | 19 +++++++++++++++
 bitbake/lib/toaster/bldcontrol/models.py           | 27 +++++++++++++++++++++-
 2 files changed, 45 insertions(+), 1 deletion(-)
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_add_cancelling_state.py
bitbake/lib/toaster/bldcontrol/migrations/0002_add_cancelling_state.py new file +19 -0
new file mode 100644
index 0000000..efcdf49
@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('bldcontrol', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='buildrequest',
+            name='state',
+            field=models.IntegerField(default=0, choices=[(0, b'created'), (1, b'queued'), (2, b'in progress'), (3, b'completed'), (4, b'failed'), (5, b'deleted'), (6, b'cancelling'), (7, b'archive')]),
+        ),
+    ]
bitbake/lib/toaster/bldcontrol/models.py +26 -1
index 9b2d0d0..cb49a58 100644
@@ -4,6 +4,8 @@ from django.core.validators import MaxValueValidator, MinValueValidator
 from django.utils.encoding import force_bytes
 from orm.models import Project, ProjectLayer, ProjectVariable, ProjectTarget, Build, Layer_Version
 
+import logging
+logger = logging.getLogger("toaster")
 # a BuildEnvironment is the equivalent of the "build/" directory on the localhost
 class BuildEnvironment(models.Model):
     SERVER_STOPPED = 0
@@ -64,7 +66,8 @@ class BuildRequest(models.Model):
     REQ_COMPLETED = 3
     REQ_FAILED = 4
     REQ_DELETED = 5
-    REQ_ARCHIVE = 6
+    REQ_CANCELLING = 6
+    REQ_ARCHIVE = 7
 
     REQUEST_STATE = (
         (REQ_CREATED, "created"),
@@ -73,6 +76,7 @@ class BuildRequest(models.Model):
         (REQ_COMPLETED, "completed"),
         (REQ_FAILED, "failed"),
         (REQ_DELETED, "deleted"),
+        (REQ_CANCELLING, "cancelling"),
         (REQ_ARCHIVE, "archive"),
     )
 
@@ -85,6 +89,27 @@ class BuildRequest(models.Model):
     created     = models.DateTimeField(auto_now_add = True)
     updated     = models.DateTimeField(auto_now = True)
 
+    def __init__(self, *args, **kwargs):
+        super(BuildRequest, self).__init__(*args, **kwargs)
+        # Save the old state incase it's about to be modified
+        self.old_state = self.state
+
+    def save(self, *args, **kwargs):
+        # Check that the state we're trying to set is not going backwards
+        # e.g. from REQ_FAILED to REQ_INPROGRESS
+        if self.old_state != self.state and self.old_state > self.state:
+            logger.warn("Invalid state change requested: "
+                        "Cannot go from %s to %s - ignoring request" %
+                        (BuildRequest.REQUEST_STATE[self.old_state][1],
+                         BuildRequest.REQUEST_STATE[self.state][1])
+                       )
+            # Set property back to the old value
+            self.state = self.old_state
+            return
+
+        super(BuildRequest, self).save(*args, **kwargs)
+
+
     def get_duration(self):
         return (self.updated - self.created).total_seconds()
 
Signature · 1 line
2.1.4
author
#8 lore raw

PATCH 8/15 buildinfohelper: Add handler for cancelling a build

In reply to Ed Bartosh

From: Michael Wood <[email protected]>

When a build is cancelled the build (action) is complete if it has been caused the request being cancelled then update the build outcome accordingly.

---
 bitbake/lib/bb/ui/buildinfohelper.py | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
bitbake/lib/bb/ui/buildinfohelper.py +18 -1
index 79fd37d..dfbce47 100644
@@ -215,6 +215,15 @@ class ORMWrapper(object):
         assert isinstance(errors, int)
         assert isinstance(warnings, int)
 
+        if build.outcome == Build.CANCELLED:
+            return
+        try:
+            if build.buildrequest.state == BuildRequest.REQ_CANCELLING:
+                return
+        except AttributeError:
+            # We may not have a buildrequest if this is a command line build
+            pass
+
         outcome = Build.SUCCEEDED
         if errors or taskfailures:
             outcome = Build.FAILED
@@ -1390,9 +1399,17 @@ class BuildInfoHelper(object):
         be.lock = BuildEnvironment.LOCK_LOCK
         be.save()
         br = BuildRequest.objects.get(pk = br_id)
+
+        # if we're 'done' because we got cancelled update the build outcome
+        if br.state == BuildRequest.REQ_CANCELLING:
+            logger.info("Build cancelled")
+            br.build.outcome = Build.CANCELLED
+            br.build.save()
+            errorcode = 0
+
         if errorcode == 0:
             # request archival of the project artifacts
-            br.state = BuildRequest.REQ_ARCHIVE
+            br.state = BuildRequest.REQ_COMPLETED
         else:
             br.state = BuildRequest.REQ_FAILED
         br.save()
Signature · 1 line
2.1.4
author
#9 lore raw

PATCH 9/15 toaster: tables BuildsTable exclude cancelled builds

In reply to Ed Bartosh

From: Sujith H <[email protected]>

Exclude cancelled builds from showing in the builds table

[YOCTO #6787]

---
 bitbake/lib/toaster/toastergui/tables.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
bitbake/lib/toaster/toastergui/tables.py +3 -2
index 822c7e5..385d658 100644
@@ -1080,8 +1080,9 @@ class BuildsTable(ToasterTable):
         """
         queryset = self.get_builds()
 
-        # don't include in progress builds
-        queryset = queryset.exclude(outcome=Build.IN_PROGRESS)
+        # Don't include in progress builds pr cancelled builds
+        queryset = queryset.exclude(Q(outcome=Build.IN_PROGRESS) |
+                                    Q(outcome=Build.CANCELLED))
 
         # sort
         queryset = queryset.order_by(self.default_orderby)
Signature · 1 line
2.1.4
author
#10 lore raw

PATCH 10/15 toaster: mrb_section template Add build cancel button

In reply to Ed Bartosh

From: Michael Wood <[email protected]>

Add the cancel build button to the mrb section template and add the event handlers to cancelABuild.
Also clean up the calls to startABuild to use the updated libtoaster methods and to make the code consistent with it's cancelABuild counterpart.

Co-Author: Sujith H <[email protected]>

---
 .../lib/toaster/toastergui/static/js/mrbsection.js |  95 +++++++++++++
 .../toaster/toastergui/templates/mrb_section.html  | 148 ++++++++-------------
 2 files changed, 149 insertions(+), 94 deletions(-)
 create mode 100644 bitbake/lib/toaster/toastergui/static/js/mrbsection.js
bitbake/lib/toaster/toastergui/static/js/mrbsection.js new file +95 -0
new file mode 100644
index 0000000..09117e1
@@ -0,0 +1,95 @@
+
+function mrbSectionInit(ctx){
+
+  var projectBuilds;
+
+  if (ctx.mrbType === 'project')
+    projectBuilds = true;
+
+  $(".cancel-build-btn").click(function(e){
+    e.preventDefault();
+
+    var url = $(this).data('request-url');
+    var buildReqIds = $(this).data('buildrequest-id');
+    var banner = $(this).parents(".alert");
+
+    banner.find(".progress-info").fadeOut().promise().done(function(){
+      $("#cancelling-msg-" + buildReqIds).show();
+      console.log("cancel build");
+      libtoaster.cancelABuild(url, buildReqIds, function(){
+        if (projectBuilds == false){
+          /* the all builds page is not 'self updating' like thei
+           * project Builds
+           */
+          window.location.reload();
+        }
+      }, null);
+    });
+  });
+
+  $(".run-again-btn").click(function(e){
+    e.preventDefault();
+
+    var url = $(this).data('request-url');
+    var target = $(this).data('target');
+
+    libtoaster.startABuild(url, target, function(){
+      window.location.reload();
+    }, null);
+  });
+
+
+  var progressTimer;
+
+  if (projectBuilds === true){
+    progressTimer = window.setInterval(function() {
+      libtoaster.getProjectInfo(libtoaster.ctx.projectPageUrl,
+        function(prjInfo){
+          /* These two are needed because a build can be 100% and still
+           * in progress due to the fact that the % done is updated at the
+           * start of a task so it can be doing the last task at 100%
+           */
+          var inProgress = 0;
+          var allPercentDone = 0;
+          if (prjInfo.builds.length === 0)
+            return
+
+          for (var i in prjInfo.builds){
+            var build = prjInfo.builds[i];
+
+            if (build.outcome === "In Progress" ||
+               $(".progress .bar").length > 0){
+              /* Update the build progress */
+              var percentDone;
+
+              if (build.outcome !== "In Progress"){
+                /* We have to ignore the value when it's Succeeded because it
+                *   goes back to 0
+                */
+                percentDone = 100;
+              } else {
+                percentDone = build.percentDone;
+                inProgress++;
+              }
+
+              $("#build-pc-done-" + build.id).text(percentDone);
+              $("#build-pc-done-title-" + build.id).attr("title", percentDone);
+              $("#build-pc-done-bar-" + build.id).css("width",
+                String(percentDone) + "%");
+
+              allPercentDone += percentDone;
+            }
+          }
+
+          if (allPercentDone === (100 * prjInfo.builds.length) && !inProgress)
+            window.location.reload();
+
+          /* Our progress bar is not still showing so shutdown the polling. */
+          if ($(".progress .bar").length === 0)
+            window.clearInterval(progressTimer);
+
+      });
+    }, 1500);
+  }
+}
+
bitbake/lib/toaster/toastergui/templates/mrb_section.html +54 -94

177 lines not on this page · raw message

Signature · 1 line
2.1.4
author
#11 lore raw

PATCH 11/15 toaster: models Exclude the CANCELLED builds from get_number_of_builds

In reply to Ed Bartosh

From: Michael Wood <[email protected]>

Don't count CANCELLED builds when returning the number of builds.

---
 bitbake/lib/toaster/orm/models.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/toaster/orm/models.py +4 -1
index e0c2096..d837912 100644
@@ -197,7 +197,10 @@ class Project(models.Model):
     def get_number_of_builds(self):
         """Return the number of builds which have ended"""
 
-        return self.build_set.filter(~Q(outcome=Build.IN_PROGRESS)).count()
+        return self.build_set.exclude(
+            Q(outcome=Build.IN_PROGRESS) |
+            Q(outcome=Build.CANCELLED)
+        ).count()
 
     def get_last_build_id(self):
         try:
Signature · 1 line
2.1.4
author
#12 lore raw

PATCH 12/15 toaster: runbuilds Make runbuilds aware of the build CANCELLED state

In reply to Ed Bartosh

From: Michael Wood <[email protected]>

Add handlers to make sure we remove the BuildEnvironment LOCK when we have cancelled a build.

---
 .../bldcontrol/management/commands/runbuilds.py    | 26 ++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)
bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py +22 -4
index 8ba836e..4c1c6a7 100644
@@ -1,5 +1,7 @@
 from django.core.management.base import NoArgsCommand, CommandError
 from django.db import transaction
+from django.db.models import Q
+
 from orm.models import Build, ToasterSetting, LogMessage, Target
 from bldcontrol.bbcontroller import getBuildEnvironmentController, ShellCmdException, BuildSetupException
 from bldcontrol.models import BuildRequest, BuildEnvironment, BRError, BRVariable
@@ -86,13 +88,20 @@ class Command(NoArgsCommand):
     def cleanup(self):
         from django.utils import timezone
         from datetime import timedelta
-        # environments locked for more than 30 seconds - they should be unlocked
-        BuildEnvironment.objects.filter(buildrequest__state__in=[BuildRequest.REQ_FAILED, BuildRequest.REQ_COMPLETED]).filter(lock=BuildEnvironment.LOCK_LOCK).filter(updated__lt = timezone.now() - timedelta(seconds = 30)).update(lock = BuildEnvironment.LOCK_FREE)
-
-
         # update all Builds that failed to start
 
         for br in BuildRequest.objects.filter(state = BuildRequest.REQ_FAILED, build__outcome = Build.IN_PROGRESS):
+        # environments locked for more than 30 seconds
+        # they should be unlocked
+        BuildEnvironment.objects.filter(
+            Q(buildrequest__state__in=[BuildRequest.REQ_FAILED,
+                                       BuildRequest.REQ_COMPLETED,
+                                       BuildRequest.REQ_CANCELLING]) &
+            Q(lock=BuildEnvironment.LOCK_LOCK) &
+            Q(updated__lt=timezone.now() - timedelta(seconds = 30))
+        ).update(lock=BuildEnvironment.LOCK_FREE)
+
+
             # transpose the launch errors in ToasterExceptions
             br.build.outcome = Build.FAILED
             for brerror in br.brerror_set.all():
@@ -125,6 +134,15 @@ class Command(NoArgsCommand):
             br.build.save()
         pass
 
+        # Make sure the LOCK is removed for builds which have been fully
+        # cancelled
+        for br in BuildRequest.objects.filter(
+            Q(build__outcome=Build.CANCELLED) &
+            Q(state=BuildRequest.REQ_CANCELLING) &
+            ~Q(environment=None)):
+            br.environment.lock = BuildEnvironment.LOCK_FREE
+            br.environment.save()
+
 
     def handle_noargs(self, **options):
         while True:
Signature · 1 line
2.1.4
author
#13 lore raw

PATCH 13/15 toaster: runbuilds Clean up runbuilds

In reply to Ed Bartosh

From: Michael Wood <[email protected]>

- Organise the imports into logical groups
- Fix 80 col wrapping
- Remove catch all exceptions
- Log to the toaster log
- Use QuerySet functions such as .first() and Q()

---
 .../bldcontrol/management/commands/runbuilds.py    | 81 ++++++++++++++--------
 1 file changed, 52 insertions(+), 29 deletions(-)
bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py +52 -29
index 4c1c6a7..27289be 100644
@@ -2,18 +2,25 @@ from django.core.management.base import NoArgsCommand, CommandError
 from django.db import transaction
 from django.db.models import Q
 
+from bldcontrol.bbcontroller import getBuildEnvironmentController
+from bldcontrol.bbcontroller import ShellCmdException, BuildSetupException
+from bldcontrol.models import BuildRequest, BuildEnvironment
+from bldcontrol.models import BRError, BRVariable
+
 from orm.models import Build, ToasterSetting, LogMessage, Target
-from bldcontrol.bbcontroller import getBuildEnvironmentController, ShellCmdException, BuildSetupException
-from bldcontrol.models import BuildRequest, BuildEnvironment, BRError, BRVariable
+
 import os
 import logging
 import time
+import sys
+import traceback
 
-logger = logging.getLogger("ToasterScheduler")
+logger = logging.getLogger("toaster")
 
 class Command(NoArgsCommand):
     args    = ""
-    help    = "Schedules and executes build requests as possible. Does not return (interrupt with Ctrl-C)"
+    help    = "Schedules and executes build requests as possible."
+    "Does not return (interrupt with Ctrl-C)"
 
 
     @transaction.atomic
@@ -25,21 +32,19 @@ class Command(NoArgsCommand):
 
     @transaction.atomic
     def _selectBuildRequest(self):
-        br = BuildRequest.objects.filter(state = BuildRequest.REQ_QUEUED).order_by('pk')[0]
-        br.state = BuildRequest.REQ_INPROGRESS
-        br.save()
+        br = BuildRequest.objects.filter(state=BuildRequest.REQ_QUEUED).first()
         return br
 
     def schedule(self):
-        import traceback
         try:
-            br = None
-            try:
-                # select the build environment and the request to build
-                br = self._selectBuildRequest()
-            except IndexError as e:
-                #logger.debug("runbuilds: No build request")
+            # select the build environment and the request to build
+            br = self._selectBuildRequest()
+            if br:
+                br.state = BuildRequest.REQ_INPROGRESS
+                br.save()
+            else:
                 return
+
             try:
                 bec = self._selectBuildEnvironment()
             except IndexError as e:
@@ -49,15 +54,17 @@ class Command(NoArgsCommand):
                 logger.debug("runbuilds: No build env")
                 return
 
-            logger.debug("runbuilds: starting build %s, environment %s" % (str(br).decode('utf-8'), bec.be))
+            logger.debug("runbuilds: starting build %s, environment %s" % \
+                         (str(br).decode('utf-8'), bec.be))
 
             # let the build request know where it is being executed
             br.environment = bec.be
             br.save()
 
             # this triggers an async build
-            bec.triggerBuild(br.brbitbake, br.brlayer_set.all(), br.brvariable_set.all(),
-                             br.brtarget_set.all(), "%d:%d" % (br.pk, bec.be.pk))
+            bec.triggerBuild(br.brbitbake, br.brlayer_set.all(),
+                             br.brvariable_set.all(), br.brtarget_set.all(),
+                             "%d:%d" % (br.pk, bec.be.pk))
 
         except Exception as e:
             logger.error("runbuilds: Error launching build %s" % e)
@@ -88,9 +95,6 @@ class Command(NoArgsCommand):
     def cleanup(self):
         from django.utils import timezone
         from datetime import timedelta
-        # update all Builds that failed to start
-
-        for br in BuildRequest.objects.filter(state = BuildRequest.REQ_FAILED, build__outcome = Build.IN_PROGRESS):
         # environments locked for more than 30 seconds
         # they should be unlocked
         BuildEnvironment.objects.filter(
@@ -102,22 +106,30 @@ class Command(NoArgsCommand):
         ).update(lock=BuildEnvironment.LOCK_FREE)
 
 
+        # update all Builds that were in progress and failed to start
+        for br in BuildRequest.objects.filter(
+            state=BuildRequest.REQ_FAILED,
+            build__outcome=Build.IN_PROGRESS):
             # transpose the launch errors in ToasterExceptions
             br.build.outcome = Build.FAILED
             for brerror in br.brerror_set.all():
                 logger.debug("Saving error %s" % brerror)
-                LogMessage.objects.create(build = br.build, level = LogMessage.EXCEPTION, message = brerror.errmsg)
+                LogMessage.objects.create(build=br.build,
+                                          level=LogMessage.EXCEPTION,
+                                          message=brerror.errmsg)
             br.build.save()
 
-            # we don't have a true build object here; hence, toasterui didn't have a change to release the BE lock
+            # we don't have a true build object here; hence, toasterui
+            # didn't have a change to release the BE lock
             br.environment.lock = BuildEnvironment.LOCK_FREE
             br.environment.save()
 
 
-
         # update all BuildRequests without a build created
         for br in BuildRequest.objects.filter(build = None):
-            br.build = Build.objects.create(project = br.project, completed_on = br.updated, started_on = br.created)
+            br.build = Build.objects.create(project=br.project,
+                                            completed_on=br.updated,
+                                            started_on=br.created)
             br.build.outcome = Build.FAILED
             try:
                 br.build.machine = br.brvariable_set.get(name='MACHINE').value
@@ -126,13 +138,16 @@ class Command(NoArgsCommand):
             br.save()
             # transpose target information
             for brtarget in br.brtarget_set.all():
-                Target.objects.create(build=br.build, target=brtarget.target, task=brtarget.task)
+                Target.objects.create(build=br.build,
+                                      target=brtarget.target,
+                                      task=brtarget.task)
             # transpose the launch errors in ToasterExceptions
             for brerror in br.brerror_set.all():
-                LogMessage.objects.create(build = br.build, level = LogMessage.EXCEPTION, message = brerror.errmsg)
+                LogMessage.objects.create(build=br.build,
+                                          level=LogMessage.EXCEPTION,
+                                          message=brerror.errmsg)
 
             br.build.save()
-        pass
 
         # Make sure the LOCK is removed for builds which have been fully
         # cancelled
@@ -148,9 +163,17 @@ class Command(NoArgsCommand):
         while True:
             try:
                 self.cleanup()
+            except Exception as e:
+                logger.warn("runbuilds: cleanup exception %s" % str(e))
+
+            try:
                 self.archive()
+            except Exception as e:
+                logger.warn("runbuilds: archive exception %s" % str(e))
+
+            try:
                 self.schedule()
-            except:
-                pass
+            except Exception as e:
+                logger.warn("runbuilds: schedule exception %s" % str(e))
 
             time.sleep(1)
Signature · 1 line
2.1.4
author
#14 lore raw

PATCH 14/15 toaster: fix conflicting migrations

In reply to Ed Bartosh

Fixed by running manage.py makemigrations --merge

---
 bitbake/lib/toaster/bldcontrol/migrations/0003_merge.py | 15 +++++++++++++++
 bitbake/lib/toaster/orm/migrations/0007_merge.py        | 15 +++++++++++++++
 2 files changed, 30 insertions(+)
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0003_merge.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0007_merge.py
bitbake/lib/toaster/bldcontrol/migrations/0003_merge.py new file +15 -0
new file mode 100644
index 0000000..b0f6b16
@@ -0,0 +1,15 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('bldcontrol', '0002_add_cancelling_state'),
+        ('bldcontrol', '0002_auto_20160120_1250'),
+    ]
+
+    operations = [
+    ]
bitbake/lib/toaster/orm/migrations/0007_merge.py new file +15 -0
new file mode 100644
index 0000000..5d92f336
@@ -0,0 +1,15 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('orm', '0006_add_cancelled_state'),
+        ('orm', '0006_remove_build_build_name'),
+    ]
+
+    operations = [
+    ]
Signature · 1 line
2.1.4
author
#15 lore raw

PATCH 15/15 toaster: use empty token

In reply to Ed Bartosh

If client xmlrpc token is not provided in the command line, bitbake generates random token. Server token in --server-only mode is always empty. This prevents clients with non-empty tokens to communicate with the server. Specifying empty token should prevent generation of random token and makes it possible to communicate with the server.

---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +1 -1
index 6a6cff8..2c67f1b 100644
@@ -319,7 +319,7 @@ class LocalhostBEController(BuildEnvironmentController):
         # run build with local bitbake. stop the server after the build.
         log = os.path.join(builddir, 'toaster_ui.log')
         self._shellcmd(['bash -c \"(TOASTER_BRBE="%s" BBSERVER="0.0.0.0:-1" '
-                        '../bitbake/bin/bitbake %s -u toasterui >>%s 2>&1;'
+                        '../bitbake/bin/bitbake %s -u toasterui --token="" >>%s 2>&1;'
                         'BITBAKE_UI="" BBSERVER=0.0.0.0:-1 %s -m)&\"' \
                         % (brbe, bbtargets, log, bitbake)], builddir,
                         nowait=True)
Signature · 1 line
2.1.4
#16 lore raw
In reply to Ed Bartosh

Hi,

I finally had a chance to try out the 'poky-contrib/ed/toaster/project-build-dir-7880-cancel-dldir_sstatedir' branch and I had comments:

1) use project name for build dirs? This would make it clearer which build dir has which projects in it.

2) we cannot assume I am building in ~/poky/build. There are env variables to let us find where poky is $OEROOT for instance. My bin/sh is bash but I still got the following when building from $HOME/cow/test/ when poky was in $HOME/src/poky :

: less toaster_ui.log

bash: ../bitbake/bin/bitbake: No such file or directory

3) if a build fails (as above) and then I cancel it, my state stays in Cancelling the build ...

4) when I cancel a build it says “Failed” it should probably say “Canceled” and still be red.

5) If I change one of my base project settings (like my download folder) the next (and following) projects should probably use my last setting as the default rather than the hardcoded default.

For instance I change my downloads from /home/…/downloads to /home/…./DL in cowMaster, when I make cowJethro, the default DL_DIR should be /home/…./DL

The following were done based on Jethro:

6) built rpi-hwup-image - then made custom image based on it. CI interface says: “Toaster has no package information for rpime. To generate package information, build rpime”

7) same true for customizing off off core-image-minimal. for both 6/7 machine = raspberrypi2.

8) trying to download the recipe file gives me the yellow error screen [Errno 2] No such file or directory:
u'/recipes-core/images/rpi-hwup-image.bb'

9) if I switch back to qemux86 as the machine, build core-image-minimal, and then customize it I also get the Toaster has no package information for cimx86me. To generate package information, build cimx86me

so it is not just due to it being a rpi image. this also gives [Errno 2] No such file or directory:
u'meta/recipes-core/images/core-image-minimal.bb'

if it try to download it.

10) after building the cimx86me, I can now download the recipe but I still do not see any packages.

11) I’m not personally fond of having the sstate and dl_dir being set on the project creation page. they are available in the bitbake var page for each project which is nice. This is more a Belén thing. just putting my opinion down.

12) if I cancel a command line build from the command line (as the little question mark suggests) the cancellation is not reflected in the UI. Mine got stuck at 46% when I cancelled my dropbear cli build.

——

So as far as upstreaming, I think we need to have the following 2 things fixed first:

1) being able to run/build from an arbitrary dir and not assuming we are in the poky tree. (should be ez)

2) fixing the custom image stuff. Since the CI is the main selling point of the 2.1 release we need to make sure we don’t break it. (hopefully not too hard?)

Other issues can be made as bugs on top of the upstream, i think.

Hope this helps,
Brian
an Intel employee

Quoted 76 lines · Ed Bartosh

This is build cancelation functionality implemented by Sujith and Michael and modified by me to work on top of 'per project build directory' patchset.

NOTE: This patchset depends on 'per project build directory' patchset' v5:

      https://lists.yoctoproject.org/pipermail/toaster/2016-March/004170.html

The following changes since commit e4ba8a50641b6a2ad8ccda5fc62c61fdf7cef733:

toasterui: shutdown on BuildCompleted event (2016-03-17 17:31:00 +0200)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/toaster/project-build-dir-7880-cancel
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/project-build-dir-7880-cancel

Ed Bartosh (2):
toaster: fix conflicting migrations
toaster: use empty token

Michael Wood (9):
toaster: Move xhr calls for starting and stopping builds
toaster: xhr Update the implementation of the build cancellation
request
toaster: libtoaster Update implementation of startABuild and
cancelABuild
toaster: bldcontrol models Add a cancelling state the BuildRequest
buildinfohelper: Add handler for cancelling a build
toaster: mrb_section template Add build cancel button
toaster: models Exclude the CANCELLED builds from get_number_of_builds
toaster: runbuilds Make runbuilds aware of the build CANCELLED state
toaster: runbuilds Clean up runbuilds

Sujith H (4):
toaster: bldcontrol Add forceShutDown function to BitbakeController
toaster: update BuildEnvironmentController and BitbakeController
toaster: models Add cancelled state to build outcome
toaster: tables BuildsTable exclude cancelled builds

 bitbake/lib/bb/ui/buildinfohelper.py               |  19 ++-
 bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  29 +---
 .../toaster/bldcontrol/localhostbecontroller.py    |   4 +-
 .../bldcontrol/management/commands/runbuilds.py    | 107 ++++++++++-----
 .../migrations/0002_add_cancelling_state.py        |  19 +++
 .../toaster/bldcontrol/migrations/0003_merge.py    |  15 +++
 bitbake/lib/toaster/bldcontrol/models.py           |  27 +++-
 .../orm/migrations/0006_add_cancelled_state.py     |  19 +++
 bitbake/lib/toaster/orm/migrations/0007_merge.py   |  15 +++
 bitbake/lib/toaster/orm/models.py                  |  12 +-
 bitbake/lib/toaster/toastergui/api.py              | 110 +++++++++++++++
 .../toaster/toastergui/static/js/customrecipe.js   |   4 +-
 .../lib/toaster/toastergui/static/js/layerBtn.js   |   3 +-
 .../lib/toaster/toastergui/static/js/libtoaster.js |  41 +++---
 .../lib/toaster/toastergui/static/js/mrbsection.js |  95 +++++++++++++
 .../toaster/toastergui/static/js/projectpage.js    |   4 +-
 .../toaster/toastergui/static/js/projecttopbar.js  |   6 +-
 .../toaster/toastergui/static/js/recipedetails.js  |   4 +-
 bitbake/lib/toaster/toastergui/tables.py           |  48 +------
 bitbake/lib/toaster/toastergui/templates/base.html |   1 +
 .../toaster/toastergui/templates/mrb_section.html  | 148 ++++++++-------------
 bitbake/lib/toaster/toastergui/urls.py             |   5 +
 22 files changed, 507 insertions(+), 228 deletions(-)
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_add_cancelling_state.py
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0003_merge.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0006_add_cancelled_state.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0007_merge.py
 create mode 100644 bitbake/lib/toaster/toastergui/api.py
 create mode 100644 bitbake/lib/toaster/toastergui/static/js/mrbsection.js

--
Regards,
Ed
--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster

#17 lore raw

a couple of other thoughts after looking at it a little more today:

1) pid of bitbake started in proj builds dir for cleanup? do we want to save these?
bitbake.lock with lsof might mean we don't need these. however, having them might negate the lsof requirement...)

2)should poky oe-init be a #!/bin/bash if dash fails?

3) toasterui event loop should probably check for liveness of server connection…. or do we get an error on waitEvent if the server has died (segv’d, kill -9 by user, etc)?

4) add tests for start, cancel builds... this needs to be a bugzilla entry so we have some tests for it. We don't need to do a build,just make sure we can start/cancel them.
- we may want to add a success.bbclass inteo meta-oe-selftest like
the error.bbclass i added awhile ago. the error class doesn’t build anything so it’s very fast...

-b
an intel employee

Quoted 179 lines · Brian Avery

Hi,

I finally had a chance to try out the 'poky-contrib/ed/toaster/project-build-dir-7880-cancel-dldir_sstatedir' branch and I had comments:

1) use project name for build dirs? This would make it clearer which build dir has which projects in it.

2) we cannot assume I am building in ~/poky/build. There are env variables to let us find where poky is $OEROOT for instance. My bin/sh is bash but I still got the following when building from $HOME/cow/test/ when poky was in $HOME/src/poky :

: less toaster_ui.log

bash: ../bitbake/bin/bitbake: No such file or directory

3) if a build fails (as above) and then I cancel it, my state stays in Cancelling the build ...

4) when I cancel a build it says “Failed” it should probably say “Canceled” and still be red.

5) If I change one of my base project settings (like my download folder) the next (and following) projects should probably use my last setting as the default rather than the hardcoded default.

For instance I change my downloads from /home/…/downloads to /home/…./DL in cowMaster, when I make cowJethro, the default DL_DIR should be /home/…./DL

The following were done based on Jethro:

6) built rpi-hwup-image - then made custom image based on it. CI interface says: “Toaster has no package information for rpime. To generate package information, build rpime”

7) same true for customizing off off core-image-minimal. for both 6/7 machine = raspberrypi2.

8) trying to download the recipe file gives me the yellow error screen [Errno 2] No such file or directory:
u'/recipes-core/images/rpi-hwup-image.bb'

9) if I switch back to qemux86 as the machine, build core-image-minimal, and then customize it I also get the Toaster has no package information for cimx86me. To generate package information, build cimx86me

so it is not just due to it being a rpi image. this also gives [Errno 2] No such file or directory:
u'meta/recipes-core/images/core-image-minimal.bb'

if it try to download it.

10) after building the cimx86me, I can now download the recipe but I still do not see any packages.

11) I’m not personally fond of having the sstate and dl_dir being set on the project creation page. they are available in the bitbake var page for each project which is nice. This is more a Belén thing. just putting my opinion down.

12) if I cancel a command line build from the command line (as the little question mark suggests) the cancellation is not reflected in the UI. Mine got stuck at 46% when I cancelled my dropbear cli build.

——

So as far as upstreaming, I think we need to have the following 2 things fixed first:

1) being able to run/build from an arbitrary dir and not assuming we are in the poky tree. (should be ez)

2) fixing the custom image stuff. Since the CI is the main selling point of the 2.1 release we need to make sure we don’t break it. (hopefully not too hard?)

Other issues can be made as bugs on top of the upstream, i think.

Hope this helps,
Brian
an Intel employee

Quoted 76 lines · Ed Bartosh

This is build cancelation functionality implemented by Sujith and Michael and modified by me to work on top of 'per project build directory' patchset.

NOTE: This patchset depends on 'per project build directory' patchset' v5:

      https://lists.yoctoproject.org/pipermail/toaster/2016-March/004170.html

The following changes since commit e4ba8a50641b6a2ad8ccda5fc62c61fdf7cef733:

toasterui: shutdown on BuildCompleted event (2016-03-17 17:31:00 +0200)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/toaster/project-build-dir-7880-cancel
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/project-build-dir-7880-cancel

Ed Bartosh (2):
toaster: fix conflicting migrations
toaster: use empty token

Michael Wood (9):
toaster: Move xhr calls for starting and stopping builds
toaster: xhr Update the implementation of the build cancellation
request
toaster: libtoaster Update implementation of startABuild and
cancelABuild
toaster: bldcontrol models Add a cancelling state the BuildRequest
buildinfohelper: Add handler for cancelling a build
toaster: mrb_section template Add build cancel button
toaster: models Exclude the CANCELLED builds from get_number_of_builds
toaster: runbuilds Make runbuilds aware of the build CANCELLED state
toaster: runbuilds Clean up runbuilds

Sujith H (4):
toaster: bldcontrol Add forceShutDown function to BitbakeController
toaster: update BuildEnvironmentController and BitbakeController
toaster: models Add cancelled state to build outcome
toaster: tables BuildsTable exclude cancelled builds

 bitbake/lib/bb/ui/buildinfohelper.py               |  19 ++-
 bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  29 +---
 .../toaster/bldcontrol/localhostbecontroller.py    |   4 +-
 .../bldcontrol/management/commands/runbuilds.py    | 107 ++++++++++-----
 .../migrations/0002_add_cancelling_state.py        |  19 +++
 .../toaster/bldcontrol/migrations/0003_merge.py    |  15 +++
 bitbake/lib/toaster/bldcontrol/models.py           |  27 +++-
 .../orm/migrations/0006_add_cancelled_state.py     |  19 +++
 bitbake/lib/toaster/orm/migrations/0007_merge.py   |  15 +++
 bitbake/lib/toaster/orm/models.py                  |  12 +-
 bitbake/lib/toaster/toastergui/api.py              | 110 +++++++++++++++
 .../toaster/toastergui/static/js/customrecipe.js   |   4 +-
 .../lib/toaster/toastergui/static/js/layerBtn.js   |   3 +-
 .../lib/toaster/toastergui/static/js/libtoaster.js |  41 +++---
 .../lib/toaster/toastergui/static/js/mrbsection.js |  95 +++++++++++++
 .../toaster/toastergui/static/js/projectpage.js    |   4 +-
 .../toaster/toastergui/static/js/projecttopbar.js  |   6 +-
 .../toaster/toastergui/static/js/recipedetails.js  |   4 +-
 bitbake/lib/toaster/toastergui/tables.py           |  48 +------
 bitbake/lib/toaster/toastergui/templates/base.html |   1 +
 .../toaster/toastergui/templates/mrb_section.html  | 148 ++++++++-------------
 bitbake/lib/toaster/toastergui/urls.py             |   5 +
 22 files changed, 507 insertions(+), 228 deletions(-)
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_add_cancelling_state.py
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0003_merge.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0006_add_cancelled_state.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0007_merge.py
 create mode 100644 bitbake/lib/toaster/toastergui/api.py
 create mode 100644 bitbake/lib/toaster/toastergui/static/js/mrbsection.js

--
Regards,
Ed
--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster

author
#18 lore raw
Quoted 4 lines · Brian Avery

a couple of other thoughts after looking at it a little more today:

1) pid of bitbake started in proj builds dir for cleanup? do we want to save these?

not until we use them.

Quoted 2 lines

bitbake.lock with lsof might mean we don't need these. however, having them might negate the lsof requirement...)

I'll look what I can do. I was planning to reuse what I did in my old ed/toaster/bec branch: to generate shell script for cloning all repos and running the build. It would be much more easy to deal with pid files in the shell script than in current python code. However, I think it's better not to do right now as we already have quite a bit of changes and very close to release.

Quoted 1 line

2)should poky oe-init be a #!/bin/bash if dash fails?

No, we shouldn't as it works with zsh.

Quoted 3 lines

3) toasterui event loop should probably check for liveness of server connection…. or do we get an error on waitEvent if the server has died (segv’d, kill -9 by user, etc)?

We run sever only during the build. We probably need to set build state to 'Failed' if server is killed or crashed.

Regards,
Ed

#19 lore raw

Hi!

Quoted 7 lines · Ed Bartosh
Quoted 4 lines · Brian Avery

a couple of other thoughts after looking at it a little more today:

1) pid of bitbake started in proj builds dir for cleanup? do we want to save these?

not until we use them.

Quoted 9 lines
Quoted 2 lines

bitbake.lock with lsof might mean we don't need these. however, having them might negate the lsof requirement...)

I'll look what I can do. I was planning to reuse what I did in my old ed/toaster/bec branch: to generate shell script for cloning all repos and running the build. It would be much more easy to deal with pid files in the shell script than in current python code. However, I think it's better not to do right now as we already have quite a bit of changes and very close to release.

k. was more of a me musing about it than anything else...

Quoted 3 lines
Quoted 1 line

2)should poky oe-init be a #!/bin/bash if dash fails?

No, we shouldn't as it works with zsh.

k. btw, how did it fail? I just ran oe-init and bitbake core-image-minimal in an ubuntu:14.04 container and it seemed fine. /bin/sh was liked to dash...

Quoted 5 lines
Quoted 3 lines

3) toasterui event loop should probably check for liveness of server connection…. or do we get an error on waitEvent if the server has died (segv’d, kill -9 by user, etc)?

We run sever only during the build. We probably need to set build state to 'Failed' if server is killed or crashed.

yep. though it can easily be a follow on bug.

Quoted 3 lines

Regards,
Ed

My thoughts:
since Ed has fixed the ../bitbake/bin issue and if
https://lists.yoctoproject.org/pipermail/toaster/2016-March/004162.html
fixes the CI not working I'm good with upstreaming and filing additional bugs for the corner cases since this fixes alot.

TY!
-brian
-an intel employee

#20 lore raw
Quoted 5 lines · On 21/03/2016 18:07, "[email protected] on behalf of Brian…

My thoughts:
since Ed has fixed the ../bitbake/bin issue and if
https://lists.yoctoproject.org/pipermail/toaster/2016-March/004162.html
fixes the CI not working I'm good with upstreaming and filing additional bugs for the corner cases since this fixes alot.

I finally managed to look into

poky-contrib/ed/toaster/project-build-dir-7880-cancel-dldir_sstatedir

but my looking was brief. I can't get any builds started from the web UI:
none of the 'build' buttons seem to work. I click and nothing happens.

I can build from the command line though, and the builds do show up in Toaster.

I wonder what I am doing wrong this time ...

Cheers

Belén

#21 lore raw

what happens when starting builds from toaster?

If you are seeing :

Quoted 3 lines

: less toaster_ui.log

bash: ../bitbake/bin/bitbake: No such file or directory

Then Ed says ->
Should be fixed by this commit:
https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=ed/toaster/project-build-dir-7880-cancel-dldir_sstatedir&id=a3566432d7d42939a482322abb7ead6026008d49

-b
...

Quoted 28 lines · Belen
Quoted 5 lines · On 21/03/2016 18:07, "[email protected] on behalf of Brian…

My thoughts:
since Ed has fixed the ../bitbake/bin issue and if
https://lists.yoctoproject.org/pipermail/toaster/2016-March/004162.html
fixes the CI not working I'm good with upstreaming and filing additional bugs for the corner cases since this fixes alot.

I finally managed to look into

poky-contrib/ed/toaster/project-build-dir-7880-cancel-dldir_sstatedir

but my looking was brief. I can't get any builds started from the web UI:
none of the 'build' buttons seem to work. I click and nothing happens.

I can build from the command line though, and the builds do show up in Toaster.

I wonder what I am doing wrong this time ...

Cheers

Belén

#22 lore raw
Quoted 11 lines · Brian Avery

what happens when starting builds from toaster?

If you are seeing :

Quoted 3 lines

: less toaster_ui.log

bash: ../bitbake/bin/bitbake: No such file or directory

Then Ed says ->
Should be fixed by this commit:
https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=ed/toast
er/project-build-dir-7880-cancel-dldir_sstatedir&id=a3566432d7d42939a48232 2abb7ead6026008d49

Builds are now working in the branch: I can start them and cancel them. Unfortunately, something has gone funny when applying Michael's patch for the image customisation problem, so that is not fixed yet. Ed is looking into it.

Thanks!

Belén

Quoted 36 lines

-b
...

Quoted 30 lines · Belen

On 21/03/2016 18:07, "[email protected] on behalf of Brian
Avery" <[email protected] on behalf of

Quoted 5 lines · [email protected]> wrote:

My thoughts:
since Ed has fixed the ../bitbake/bin issue and if
https://lists.yoctoproject.org/pipermail/toaster/2016-March/004162.html
fixes the CI not working I'm good with upstreaming and filing additional bugs for the corner cases since this fixes alot.

I finally managed to look into

poky-contrib/ed/toaster/project-build-dir-7880-cancel-dldir_sstatedir

but my looking was brief. I can't get any builds started from the web UI:
none of the 'build' buttons seem to work. I click and nothing happens.

I can build from the command line though, and the builds do show up in Toaster.

I wonder what I am doing wrong this time ...

Cheers

Belén

#23 lore raw
In reply to Brian Avery

Brian, a few comments on your comments...

Quoted 9 lines · Brian Avery

Hi,

I finally had a chance to try out the 'poky-contrib/ed/toaster/project-build-dir-7880-cancel-dldir_sstatedir' branch and I had comments:

1) use project name for build dirs? This would make it clearer which build dir has which projects in it.

This introduces the problem of what to do if a project's name changes, which could happen in the middle of a build; it would also need additional code to sanitise the project name to produce a valid directory name. It seems simpler to me to just use the build ID, as it's not a massive stretch to figure out which build goes with which numbered directory.

...

Quoted 6 lines

11) I’m not personally fond of having the sstate and dl_dir being set on the project creation page. they are available in the bitbake var page for each project which is nice. This is more a Belén thing. just putting my opinion down.

After discussion on the list, Sujith has provided a version of Ed's branch which removes these fields from the new project page:
sujith/set_dldir_sstatedirv2

Elliot

Signature · 3 lines
Elliot Smith
Software Engineer
Intel Open Source Technology Centre
#24 lore raw
In reply to Brian Avery
Quoted 8 lines · On 19/03/2016 23:13, "[email protected] on behalf of Brian…

Hi,

I finally had a chance to try out the 'poky-contrib/ed/toaster/project-build-dir-7880-cancel-dldir_sstatedir' branch and I had comments:

1) use project name for build dirs? This would make it clearer which build dir has which projects in it.

We had a discussion about this, part of which Elliot has summarised in his previous email. It is true that it might be easier to identify the build directory of a project if we use the project name, but Toaster does not enforce unique names, it does not validate for special characters or length limits, and it allows you change the project name at any time. The problems all these could cause made us go with the project id. To ease identification, we could show the project id and the path to the build directory in the project page, unless anybody has issues with displaying this information.

Quoted 7 lines

2) we cannot assume I am building in ~/poky/build. There are env variables to let us find where poky is $OEROOT for instance. My bin/sh is bash but I still got the following when building from $HOME/cow/test/ when poky was in $HOME/src/poky :

Quoted 6 lines

: less toaster_ui.log

bash: ../bitbake/bin/bitbake: No such file or directory

3) if a build fails (as above) and then I cancel it, my state stays in Cancelling the build ...

This sounds like a bug: you should never be able to cancel a failed build. The 'cancel' button should only be available while bitbake is running.

Quoted 4 lines

4) when I cancel a build it says ³Failed² it should probably say ³Canceled² and still be red.

This is also a bug. Cancelled builds have their own presentation (they say "cancelled"). See page 4 of this design document

https://bugzilla.yoctoproject.org/attachment.cgi?id=2706

Quoted 5 lines

5) If I change one of my base project settings (like my download folder) the next (and following) projects should probably use my last setting as the default rather than the hardcoded default.

I am not sure I agree. The UI right now provides only per project
configuration: I wouldn't make exceptions to that rule. If we did, you would be in a project context surrounded by variables that only apply to the selected project, and all of a sudden you would have 2 variables that look the same as all others but behave differently and override the Toaster defaults.

If you want to override the Toaster defaults, you should change the default value via the Django admin interface or in the toasterconf.json file. The UI only provides access to project configuration at this time ... consistently limited ;)

Quoted 8 lines

For instance I change my downloads from /home/Š/downloads to /home/Š./DL in cowMaster, when I make cowJethro, the default DL_DIR should be /home/Š./DL

The following were done based on Jethro:

Quoted 7 lines

6) built rpi-hwup-image - then made custom image based on it. CI interface says: ³Toaster has no package information for rpime. To generate package information, build rpime²

7) same true for customizing off off core-image-minimal. for both 6/7 machine = raspberrypi2.

I've seen this too, but I assumed is was caused by this:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=9266

Although I could be completely wrong. There is a patch under review to fix it

https://lists.yoctoproject.org/pipermail/toaster/2016-March/004162.html

But I haven't had a chance to try it on top of Ed's branch.

Quoted 19 lines

8) trying to download the recipe file gives me the yellow error screen [Errno 2] No such file or directory:
u'/recipes-core/images/rpi-hwup-image.bb'

9) if I switch back to qemux86 as the machine, build core-image-minimal, and then customize it I also get the Toaster has no package information for cimx86me. To generate package information, build cimx86me

so it is not just due to it being a rpi image. this also gives [Errno 2] No such file or directory:
u'meta/recipes-core/images/core-image-minimal.bb'

if it try to download it.

Quoted 8 lines

10) after building the cimx86me, I can now download the recipe but I still do not see any packages.

11) I¹m not personally fond of having the sstate and dl_dir being set on the project creation page. they are available in the bitbake var page for each project which is nice. This is more a Belén thing. just putting my opinion down.

We also discussed this (everybody complained) and decided to remove them.

Quoted 127 lines

12) if I cancel a command line build from the command line (as the little question mark suggests) the cancellation is not reflected in the UI. Mine got stuck at 46% when I cancelled my dropbear cli build.

‹‹

So as far as upstreaming, I think we need to have the following 2 things fixed first:

1) being able to run/build from an arbitrary dir and not assuming we are in the poky tree. (should be ez)

2) fixing the custom image stuff. Since the CI is the main selling point of the 2.1 release we need to make sure we don¹t break it. (hopefully not too hard?)

Other issues can be made as bugs on top of the upstream, i think.

Hope this helps,
Brian
an Intel employee

Quoted 92 lines · Ed Bartosh

This is build cancelation functionality implemented by Sujith and Michael
and modified by me to work on top of 'per project build directory' patchset.

NOTE: This patchset depends on 'per project build directory' patchset' v5:

https://lists.yoctoproject.org/pipermail/toaster/2016-March/004170.html

The following changes since commit
e4ba8a50641b6a2ad8ccda5fc62c61fdf7cef733:

toasterui: shutdown on BuildCompleted event (2016-03-17 17:31:00
+0200)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib
ed/toaster/project-build-dir-7880-cancel

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/proje
ct-build-dir-7880-cancel

Ed Bartosh (2):
toaster: fix conflicting migrations
toaster: use empty token

Michael Wood (9):
toaster: Move xhr calls for starting and stopping builds
toaster: xhr Update the implementation of the build cancellation
request
toaster: libtoaster Update implementation of startABuild and
cancelABuild
toaster: bldcontrol models Add a cancelling state the BuildRequest
buildinfohelper: Add handler for cancelling a build
toaster: mrb_section template Add build cancel button
toaster: models Exclude the CANCELLED builds from get_number_of_builds
toaster: runbuilds Make runbuilds aware of the build CANCELLED state
toaster: runbuilds Clean up runbuilds

Sujith H (4):
toaster: bldcontrol Add forceShutDown function to BitbakeController
toaster: update BuildEnvironmentController and BitbakeController
toaster: models Add cancelled state to build outcome
toaster: tables BuildsTable exclude cancelled builds

 bitbake/lib/bb/ui/buildinfohelper.py               |  19 ++-
 bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  29 +---
 .../toaster/bldcontrol/localhostbecontroller.py    |   4 +-
 .../bldcontrol/management/commands/runbuilds.py    | 107
++++++++++-----
 .../migrations/0002_add_cancelling_state.py        |  19 +++
 .../toaster/bldcontrol/migrations/0003_merge.py    |  15 +++
 bitbake/lib/toaster/bldcontrol/models.py           |  27 +++-
 .../orm/migrations/0006_add_cancelled_state.py     |  19 +++
 bitbake/lib/toaster/orm/migrations/0007_merge.py   |  15 +++
 bitbake/lib/toaster/orm/models.py                  |  12 +-
 bitbake/lib/toaster/toastergui/api.py              | 110
+++++++++++++++
 .../toaster/toastergui/static/js/customrecipe.js   |   4 +-
 .../lib/toaster/toastergui/static/js/layerBtn.js   |   3 +-
 .../lib/toaster/toastergui/static/js/libtoaster.js |  41 +++---
 .../lib/toaster/toastergui/static/js/mrbsection.js |  95 +++++++++++++
 .../toaster/toastergui/static/js/projectpage.js    |   4 +-
 .../toaster/toastergui/static/js/projecttopbar.js  |   6 +-
 .../toaster/toastergui/static/js/recipedetails.js  |   4 +-
 bitbake/lib/toaster/toastergui/tables.py           |  48 +------
 bitbake/lib/toaster/toastergui/templates/base.html |   1 +
 .../toaster/toastergui/templates/mrb_section.html  | 148
++++++++-------------
 bitbake/lib/toaster/toastergui/urls.py             |   5 +
 22 files changed, 507 insertions(+), 228 deletions(-)
 create mode 100644
bitbake/lib/toaster/bldcontrol/migrations/0002_add_cancelling_state.py
 create mode 100644
bitbake/lib/toaster/bldcontrol/migrations/0003_merge.py
 create mode 100644
bitbake/lib/toaster/orm/migrations/0006_add_cancelled_state.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0007_merge.py
 create mode 100644 bitbake/lib/toaster/toastergui/api.py
 create mode 100644

bitbake/lib/toaster/toastergui/static/js/mrbsection.js

--
Regards,
Ed
--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster

Signature · 4 lines
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster
#25 lore raw

Hi,
Responding to comments on my comments:

Quoted 15 lines · Belen
Quoted 8 lines · On 19/03/2016 23:13, "[email protected] on behalf of Brian…

Hi,

I finally had a chance to try out the 'poky-contrib/ed/toaster/project-build-dir-7880-cancel-dldir_sstatedir' branch and I had comments:

1) use project name for build dirs? This would make it clearer which build dir has which projects in it.

Quoted 10 lines

We had a discussion about this, part of which Elliot has summarised in his previous email. It is true that it might be easier to identify the build directory of a project if we use the project name, but Toaster does not enforce unique names, it does not validate for special characters or length limits, and it allows you change the project name at any time. The problems all these could cause made us go with the project id. To ease identification, we could show the project id and the path to the build directory in the project page, unless anybody has issues with displaying this information.

That would work for me. I just wanted an ez way to tie them together and i feared the proj_id would not be clear to users.

Quoted 40 lines
Quoted 7 lines

2) we cannot assume I am building in ~/poky/build. There are env variables to let us find where poky is $OEROOT for instance. My bin/sh is bash but I still got the following when building from $HOME/cow/test/ when poky was in $HOME/src/poky :

Quoted 6 lines

: less toaster_ui.log

bash: ../bitbake/bin/bitbake: No such file or directory

3) if a build fails (as above) and then I cancel it, my state stays in Cancelling the build ...

This sounds like a bug: you should never be able to cancel a failed build. The 'cancel' button should only be available while bitbake is running.

Quoted 4 lines

4) when I cancel a build it says ³Failed² it should probably say ³Canceled² and still be red.

This is also a bug. Cancelled builds have their own presentation (they say "cancelled"). See page 4 of this design document

https://bugzilla.yoctoproject.org/attachment.cgi?id=2706

Quoted 5 lines

5) If I change one of my base project settings (like my download folder) the next (and following) projects should probably use my last setting as the default rather than the hardcoded default.

I am not sure I agree. The UI right now provides only per project
configuration: I wouldn't make exceptions to that rule. If we did, you would be in a project context surrounded by variables that only apply to the selected project, and all of a sudden you would have 2 variables that look the same as all others but behave differently and override the Toaster defaults.

Quoted 5 lines

If you want to override the Toaster defaults, you should change the default value via the Django admin interface or in the toasterconf.json file. The UI only provides access to project configuration at this time ... consistently limited ;)

fair point. and since Elliot said they were moving off the main proj page i withdraw the suggestion.

Quoted 18 lines
Quoted 8 lines

For instance I change my downloads from /home/Š/downloads to /home/Š./DL in cowMaster, when I make cowJethro, the default DL_DIR should be /home/Š./DL

The following were done based on Jethro:

Quoted 7 lines

6) built rpi-hwup-image - then made custom image based on it. CI interface says: ³Toaster has no package information for rpime. To generate package information, build rpime²

7) same true for customizing off off core-image-minimal. for both 6/7 machine = raspberrypi2.

I've seen this too, but I assumed is was caused by this:

Quoted 8 lines

https://bugzilla.yoctoproject.org/show_bug.cgi?id=9266

Although I could be completely wrong. There is a patch under review to fix it

https://lists.yoctoproject.org/pipermail/toaster/2016-March/004162.html

But I haven't had a chance to try it on top of Ed's branch.

Could be true; i hope so :).

Quoted 24 lines
Quoted 24 lines

8) trying to download the recipe file gives me the yellow error screen [Errno 2] No such file or directory:
u'/recipes-core/images/rpi-hwup-image.bb'

9) if I switch back to qemux86 as the machine, build core-image-minimal, and then customize it I also get the Toaster has no package information for cimx86me. To generate package information, build cimx86me

so it is not just due to it being a rpi image. this also gives [Errno 2] No such file or directory:
u'meta/recipes-core/images/core-image-minimal.bb'

if it try to download it.

10) after building the cimx86me, I can now download the recipe but I still do not see any packages.

Quoted 135 lines
Quoted 4 lines

11) I¹m not personally fond of having the sstate and dl_dir being set on the project creation page. they are available in the bitbake var page for each project which is nice. This is more a Belén thing. just putting my opinion down.

We also discussed this (everybody complained) and decided to remove them.

Quoted 127 lines

12) if I cancel a command line build from the command line (as the little question mark suggests) the cancellation is not reflected in the UI. Mine got stuck at 46% when I cancelled my dropbear cli build.

‹‹

So as far as upstreaming, I think we need to have the following 2 things fixed first:

1) being able to run/build from an arbitrary dir and not assuming we are in the poky tree. (should be ez)

2) fixing the custom image stuff. Since the CI is the main selling point of the 2.1 release we need to make sure we don¹t break it. (hopefully not too hard?)

Other issues can be made as bugs on top of the upstream, i think.

Hope this helps,
Brian
an Intel employee

Quoted 84 lines · Ed Bartosh

This is build cancelation functionality implemented by Sujith and Michael
and modified by me to work on top of 'per project build directory' patchset.

NOTE: This patchset depends on 'per project build directory' patchset' v5:

https://lists.yoctoproject.org/pipermail/toaster/2016-March/004170.html

The following changes since commit
e4ba8a50641b6a2ad8ccda5fc62c61fdf7cef733:

toasterui: shutdown on BuildCompleted event (2016-03-17 17:31:00
+0200)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib
ed/toaster/project-build-dir-7880-cancel

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/proje
ct-build-dir-7880-cancel

Ed Bartosh (2):
toaster: fix conflicting migrations
toaster: use empty token

Michael Wood (9):
toaster: Move xhr calls for starting and stopping builds
toaster: xhr Update the implementation of the build cancellation
request
toaster: libtoaster Update implementation of startABuild and
cancelABuild
toaster: bldcontrol models Add a cancelling state the BuildRequest
buildinfohelper: Add handler for cancelling a build
toaster: mrb_section template Add build cancel button
toaster: models Exclude the CANCELLED builds from get_number_of_builds
toaster: runbuilds Make runbuilds aware of the build CANCELLED state
toaster: runbuilds Clean up runbuilds

Sujith H (4):
toaster: bldcontrol Add forceShutDown function to BitbakeController
toaster: update BuildEnvironmentController and BitbakeController
toaster: models Add cancelled state to build outcome
toaster: tables BuildsTable exclude cancelled builds

 bitbake/lib/bb/ui/buildinfohelper.py               |  19 ++-
 bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  29 +---
 .../toaster/bldcontrol/localhostbecontroller.py    |   4 +-
 .../bldcontrol/management/commands/runbuilds.py    | 107
++++++++++-----
 .../migrations/0002_add_cancelling_state.py        |  19 +++
 .../toaster/bldcontrol/migrations/0003_merge.py    |  15 +++
 bitbake/lib/toaster/bldcontrol/models.py           |  27 +++-
 .../orm/migrations/0006_add_cancelled_state.py     |  19 +++
 bitbake/lib/toaster/orm/migrations/0007_merge.py   |  15 +++
 bitbake/lib/toaster/orm/models.py                  |  12 +-
 bitbake/lib/toaster/toastergui/api.py              | 110
+++++++++++++++
 .../toaster/toastergui/static/js/customrecipe.js   |   4 +-
 .../lib/toaster/toastergui/static/js/layerBtn.js   |   3 +-
 .../lib/toaster/toastergui/static/js/libtoaster.js |  41 +++---
 .../lib/toaster/toastergui/static/js/mrbsection.js |  95 +++++++++++++
 .../toaster/toastergui/static/js/projectpage.js    |   4 +-
 .../toaster/toastergui/static/js/projecttopbar.js  |   6 +-
 .../toaster/toastergui/static/js/recipedetails.js  |   4 +-
 bitbake/lib/toaster/toastergui/tables.py           |  48 +------
 bitbake/lib/toaster/toastergui/templates/base.html |   1 +
 .../toaster/toastergui/templates/mrb_section.html  | 148
++++++++-------------
 bitbake/lib/toaster/toastergui/urls.py             |   5 +
 22 files changed, 507 insertions(+), 228 deletions(-)
 create mode 100644
bitbake/lib/toaster/bldcontrol/migrations/0002_add_cancelling_state.py
 create mode 100644
bitbake/lib/toaster/bldcontrol/migrations/0003_merge.py
 create mode 100644
bitbake/lib/toaster/orm/migrations/0006_add_cancelled_state.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0007_merge.py
 create mode 100644 bitbake/lib/toaster/toastergui/api.py
 create mode 100644

bitbake/lib/toaster/toastergui/static/js/mrbsection.js

Quoted 8 lines

--
Regards,
Ed
--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster

--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster

-brian
an intel employee

author
#26 lore raw
In reply to Brian Avery
Quoted 8 lines · Brian Avery

2) we cannot assume I am building in ~/poky/build. There are env variables to let us find where poky is $OEROOT for instance. My bin/sh is bash but I still got the following when building from $HOME/cow/test/ when poky was in $HOME/src/poky :

: less toaster_ui.log

bash: ../bitbake/bin/bitbake: No such file or directory

Should be fixed by this commit:
https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=ed/toaster/project-build-dir-7880-cancel-dldir_sstatedir&id=a3566432d7d42939a482322abb7ead6026008d49

...

Quoted 4 lines

11) I’m not personally fond of having the sstate and dl_dir being set on the project creation page. they are available in the bitbake var page for each project which is nice. This is more a Belén thing. just putting my opinion down.

This should be alread fixed by Sujith. The fix is also in ed/toaster/project-build-dir-7880-cancel-dldir_sstatedir

...

Quoted 5 lines

So as far as upstreaming, I think we need to have the following 2 things fixed first:

1) being able to run/build from an arbitrary dir and not assuming we are in the poky tree. (should be ez)

yes, it should be fixed already. see my comment above.

Quoted 3 lines

2) fixing the custom image stuff. Since the CI is the main selling point of the 2.1 release we need to make sure we don’t break it. (hopefully not too hard?)

OK, I'll try to look at it.

BTW, the full patchset has 42 patches and becomes hard to maintain. We may need to consider upstreaming it in 2 or 3 steps if the above fixes are hard to do.

Regards,
Ed