Skip to contentSkip to conversation

Toaster patchset

PATCH New Alex DAMIAN 1 participant 11 messages Feb 18, 2015

lore
Share & tools
lore
#0 lore raw

cover Toaster patchset

From: Alexandru DAMIAN <[email protected]>

Hello,

This is a toaster patchset with bug fixes and new features. The patches have been reviewed on the toaster mailing list.

Can you please pull ?

Cheers,
Alex

The following changes since commit 7cfe279bf77b59d5cbd20d8a93e1d33279bebc20:

toasterui: empty state for pages (2015-02-16 17:48:30 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib adamian/20150218-submission-bb
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=adamian/20150218-submission-bb

Alexandru DAMIAN (4):
toaster: bitbake cooker log saving and downloading
toaster: properly detect starting in rolling log file
toasterui: UI query performance improvements
toaster: get proper version string

Alexandru Damian (1):
toasterui: fix time estimation in project page

Belen Barros Pena (2):
toasterui: style the Toaster version information
toasterui: remove layer directory in managed mode

David Reyna (1):
toaster: landing page when no build module

Michael Wood (2):
toaster: layerdetails Don't show None type in description/summary
toaster: importlayer Remove description input field

 lib/bb/ui/buildinfohelper.py                       |   3 +-
 lib/bb/ui/toasterui.py                             |  29 +++-
 lib/toaster/bldcontrol/localhostbecontroller.py    |  12 +-
 .../management/commands/checksettings.py           |  17 +++
 .../bldcontrol/management/commands/runbuilds.py    |  29 +++-
 .../bldcontrol/migrations/0008_brarchive.py        | 138 +++++++++++++++++
 lib/toaster/bldcontrol/models.py                   |   2 +
 lib/toaster/orm/models.py                          |  13 +-
 lib/toaster/toastergui/static/css/default.css      |  22 +--
 lib/toaster/toastergui/static/js/importlayer.js    |   1 -
 lib/toaster/toastergui/static/js/projectapp.js     |  24 ++-
 lib/toaster/toastergui/templates/base.html         |  13 +-
 lib/toaster/toastergui/templates/bpackage.html     |   4 +-
 lib/toaster/toastergui/templates/build.html        |   9 +-
 .../toastergui/templates/builddashboard.html       |  11 +-
 lib/toaster/toastergui/templates/importlayer.html  |   6 -
 .../toastergui/templates/landing_not_managed.html  |  32 ++++
 lib/toaster/toastergui/templates/layerdetails.html |   4 +-
 .../toastergui/templates/managed_builds.html       |   9 +-
 .../toastergui/templates/managed_mrb_section.html  |   3 +-
 lib/toaster/toastergui/templates/project.html      |   2 +-
 .../toastergui/templates/projectbuilds.html        |  14 +-
 .../toastergui/templates/unavailable_artifact.html |  17 +++
 lib/toaster/toastergui/views.py                    | 168 ++++++++++++++-------
 lib/toaster/toastermain/settings.py                |   3 +-
 lib/toaster/toastermain/urls.py                    |   6 +-
 26 files changed, 465 insertions(+), 126 deletions(-)
 create mode 100644 lib/toaster/bldcontrol/migrations/0008_brarchive.py
 create mode 100644 lib/toaster/toastergui/templates/landing_not_managed.html
 create mode 100644 lib/toaster/toastergui/templates/unavailable_artifact.html
Signature · 1 line
1.9.1
#3 lore raw

PATCH 3/10 toasterui: fix time estimation in project page

In reply to Alex DAMIAN

From: Alexandru Damian <[email protected]>

This patch fixes the time estimation to build completion in project page. Additionally it fixes the Most Recent Builds section used in various pages in managed mode, and proper time to build estimation in all pages.

[YOCTO #7004]

---
 lib/toaster/orm/models.py                          |  2 +-
 lib/toaster/toastergui/static/js/projectapp.js     | 24 +++++++++++++++++++---
 .../toastergui/templates/managed_mrb_section.html  |  3 +--
 lib/toaster/toastergui/templates/project.html      |  2 +-
 4 files changed, 24 insertions(+), 7 deletions(-)
lib/toaster/orm/models.py +1 -1
index f70c006..0f85c8f 100644
@@ -194,7 +194,7 @@ class Build(models.Model):
         eta = timezone.now()
         completeper = self.completeper()
         if self.completeper() > 0:
-            eta = timezone.now() + ((timezone.now() - self.started_on)*(100-completeper)/completeper)
+            eta += ((eta - self.started_on)*100)/completeper
         return eta
 
 
lib/toaster/toastergui/static/js/projectapp.js +21 -3
index bee3c56..4d00f40 100644
@@ -136,6 +136,16 @@ projectApp.filter('timediff', function() {
     }
 });
 
+// add "time to future" eta that computes time from now to a point in the future
+projectApp.filter('toeta', function() {
+    return function(input) {
+       var crtmiliseconds = new Date().getTime();
+        diff = (parseInt(input) - crtmiliseconds ) / 1000;
+        console.log("Debug: future time ", input, "crt time", crtmiliseconds, ":", diff);
+        return diff < 0 ? 300 : diff;
+    }
+});
+
 /**
  * main controller for the project page
  */
@@ -259,7 +269,14 @@ projectApp.controller('prjCtrl', function($scope, $modal, $http, $interval, $loc
                         var i = 0;
                         for (i = 0 ; i < $scope.builds.length; i ++) {
                             if ($scope.builds[i].id > elem.id) continue;
-                            if ($scope.builds[i].id == elem.id) { found=true; break;}
+                            if ($scope.builds[i].id == elem.id) {
+                                found=true;
+                                // do deep data copy
+                                for (var attr in elem) {
+                                    $scope.builds[i][attr] = elem[attr];
+                                }
+                                break;
+                            }
                             if ($scope.builds[i].id < elem.id) break;
                         }
                         if (!found) {
@@ -272,8 +289,8 @@ projectApp.controller('prjCtrl', function($scope, $modal, $http, $interval, $loc
                         var found = false;
                         var i = 0;
                         for (i = 0; i < $scope.builds.length; i ++) {
-                           if ($scope.builds[i].id > elem.id) continue;
-                            if ($scope.builds[i].id == elem.id) { found=true; break;}
+                            if ($scope.builds[i].id > elem.id) continue;
+                            if ($scope.builds[i].id == elem.id) { found=true; break; }
                             if ($scope.builds[i].id < elem.id) break;
                         }
                         if (!found) {
@@ -281,6 +298,7 @@ projectApp.controller('prjCtrl', function($scope, $modal, $http, $interval, $loc
                         }
                     });
 
+
                     $scope.fetchLayersForTargets();
                 }
                 if (_data.targets !== undefined) {
lib/toaster/toastergui/templates/managed_mrb_section.html +1 -2
index 6f31ee0..b8d087a 100644
@@ -2,8 +2,7 @@
 {% load projecttags %}
 {% load humanize %}
 
-
-{%if len.mru > 0%}
+{%if mru|length > 0%}
   <div class="page-header top-air">
       <h1>
           Latest builds
lib/toaster/toastergui/templates/project.html +1 -1
index 7da2361..a1cce33 100644
@@ -196,7 +196,7 @@ vim: expandtab tabstop=2
                       <div style="width: {[b.build[0].completeper]}%;" class="bar"></div>
                   </div>
                 </div>
-                <div class="text-right lead">ETA: {[b.build[0].eta|date:"HH:mm:ss"]}</div>
+                <div class="text-right lead">ETA in {[b.build[0].eta|toeta|timediff]}</div>
               </case>
           </case>
 
Signature · 1 line
1.9.1
#2 lore raw

PATCH 2/10 toaster: properly detect starting in rolling log file

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

This patch brings in properly detection of system start in a rolling toaster_ui log file.

---
 lib/toaster/bldcontrol/localhostbecontroller.py | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
lib/toaster/bldcontrol/localhostbecontroller.py +10 -2
index 6bd019a..25ce052 100644
@@ -107,6 +107,13 @@ class LocalhostBEController(BuildEnvironmentController):
 
         logger.debug("localhostbecontroller: running the listener at %s" % own_bitbake)
 
+        toaster_ui_log_filepath = os.path.join(self.be.builddir, "toaster_ui.log")
+        # get the file length; we need to detect the _last_ start of the toaster UI, not the first
+        toaster_ui_log_filelength = 0
+        if os.path.exists(toaster_ui_log_filepath):
+            with open(toaster_ui_log_filepath, "r") as f:
+                f.seek(0, 2)    # jump to the end
+                toaster_ui_log_filelength = f.tell()
 
         cmd = "bash -c \"source %s/oe-init-build-env %s && bitbake --read conf/toaster-pre.conf --postread conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0 >toaster_server.log && DATABASE_URL=%s BBSERVER=0.0.0.0:-1 daemon -d -i -D %s -o toaster_ui.log -- %s --observe-only -u toasterui &\"" % (self.pokydirname, self.be.builddir,
                 self.dburl, self.be.builddir, own_bitbake)
@@ -117,10 +124,11 @@ class LocalhostBEController(BuildEnvironmentController):
                 port = i.split(" ")[-1]
                 logger.debug("localhostbecontroller: Found bitbake server port %s" % port)
 
-        def _toaster_ui_started(filepath):
+        def _toaster_ui_started(filepath, filepos = 0):
             if not os.path.exists(filepath):
                 return False
             with open(filepath, "r") as f:
+                f.seek(filepos)
                 for line in f:
                     if line.startswith("NOTE: ToasterUI waiting for events"):
                         return True
@@ -129,7 +137,7 @@ class LocalhostBEController(BuildEnvironmentController):
         retries = 0
         started = False
         while not started and retries < 10:
-            started = _toaster_ui_started(os.path.join(self.be.builddir, "toaster_ui.log"))
+            started = _toaster_ui_started(toaster_ui_log_filepath, toaster_ui_log_filelength)
             import time
             logger.debug("localhostbecontroller: Waiting bitbake server to start")
             time.sleep(0.5)
Signature · 1 line
1.9.1
#5 lore raw

PATCH 5/10 toaster: get proper version string

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

This patch brings in the toaster tree commit ID in the version string to allow users properly report their toaster version when submitting issues.

---
 lib/toaster/toastermain/settings.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
lib/toaster/toastermain/settings.py +2 -1
index b6a42c0..a7304fa 100644
@@ -232,7 +232,8 @@ from os.path import dirname as DN
 SITE_ROOT=DN(DN(os.path.abspath(__file__)))
 
 import subprocess
-TOASTER_VERSION = subprocess.Popen('cd %s; git branch | grep "^* " | tr -d "* "' % SITE_ROOT, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0]
+TOASTER_VERSION = subprocess.Popen('git branch | grep "^* " | tr -d "* "', cwd = SITE_ROOT, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0]
+TOASTER_VERSION += " " + subprocess.Popen('git rev-parse HEAD ', cwd = SITE_ROOT, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0]
 
 ROOT_URLCONF = 'toastermain.urls'
 
Signature · 1 line
1.9.1
#4 lore raw

PATCH 4/10 toasterui: UI query performance improvements

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

We reduce the number of queries by using "select_related" to bring in more data in a single query. Improvements in project page refresh, recipes table and tasks table.

---
 lib/toaster/orm/models.py       | 11 +++++++----
 lib/toaster/toastergui/views.py | 15 +++++++++------
 2 files changed, 16 insertions(+), 10 deletions(-)
lib/toaster/orm/models.py +7 -4
index 0f85c8f..54cd535 100644
@@ -362,11 +362,14 @@ class Task(models.Model):
         return "Not Executed"
 
     def get_description(self):
-        helptext = HelpText.objects.filter(key=self.task_name, area=HelpText.VARIABLE, build=self.build)
+        if '_helptext' in vars(self) and self._helptext != None:
+            return self._helptext
         try:
-            return helptext[0].text
-        except IndexError:
-            return ''
+            self._helptext = HelpText.objects.get(key=self.task_name, area=HelpText.VARIABLE, build=self.build).text
+        except HelpText.DoesNotExit:
+            self._helptext = None
+
+        return self._helptext
 
     build = models.ForeignKey(Build, related_name='task_build')
     order = models.IntegerField(null=True)
lib/toaster/toastergui/views.py +9 -6
index eb323ec..a2e9206 100755
@@ -75,8 +75,9 @@ def _project_recent_build_list(prj):
                         'build_time_page_url': reverse('buildtime', args=(y.pk,)),
                         "errors": y.errors_no,
                         "warnings": y.warnings_no,
-                        "completeper": y.completeper(),
-                        "eta": y.eta().strftime('%s')+"000"}, Build.objects.filter(buildrequest = x)),
+                        "completeper": y.completeper() if y.outcome == Build.IN_PROGRESS else "0",
+                        "eta": y.eta().strftime('%s')+"000" if y.outcome == Build.IN_PROGRESS else "0",
+                        }, Build.objects.filter(buildrequest = x)),
         }, list(prj.buildrequest_set.filter(Q(state__lt=BuildRequest.REQ_COMPLETED) or Q(state=BuildRequest.REQ_DELETED)).order_by("-pk")) +
             list(prj.buildrequest_set.filter(state__in=[BuildRequest.REQ_COMPLETED, BuildRequest.REQ_FAILED]).order_by("-pk")[:3]))
 
@@ -828,6 +829,8 @@ def tasks_common(request, build_id, variant, task_anchor):
     else:
         queryset = _get_queryset(Task, queryset_all, filter_string, search_term, ordering_string, 'order')
 
+    queryset = queryset.select_related("recipe", "build")
+
     # compute the anchor's page
     if anchor:
         request.GET = request.GET.copy()
@@ -1023,14 +1026,14 @@ def recipes(request, build_id):
     if retval:
         return _redirect_parameters( 'recipes', request.GET, mandatory_parameters, build_id = build_id)
     (filter_string, search_term, ordering_string) = _search_tuple(request, Recipe)
-    queryset = Recipe.objects.filter(layer_version__id__in=Layer_Version.objects.filter(build=build_id))
+    queryset = Recipe.objects.filter(layer_version__id__in=Layer_Version.objects.filter(build=build_id)).select_related("layer_version", "layer_version__layer")
     queryset = _get_queryset(Recipe, queryset, filter_string, search_term, ordering_string, 'name')
 
     recipes = _build_page_range(Paginator(queryset, pagesize),request.GET.get('page', 1))
 
     # prefetch the forward and reverse recipe dependencies
     deps = { }; revs = { }
-    queryset_dependency=Recipe_Dependency.objects.filter(recipe__layer_version__build_id = build_id)
+    queryset_dependency=Recipe_Dependency.objects.filter(recipe__layer_version__build_id = build_id).select_related("depends_on", "recipe")
     for recipe in recipes:
         deplist = [ ]
         for recipe_dep in [x for x in queryset_dependency if x.recipe_id == recipe.id]:
@@ -1804,7 +1807,7 @@ if toastermain.settings.MANAGED:
         (filter_string, search_term, ordering_string) = _search_tuple(request, BuildRequest)
         # we don't display in-progress or deleted builds
         queryset_all = buildrequests.exclude(state = BuildRequest.REQ_DELETED)
-        queryset_all = queryset_all.annotate(Count('brerror'))
+        queryset_all = queryset_all.select_related("build", "build__project").annotate(Count('brerror'))
         queryset_with_search = _get_queryset(BuildRequest, queryset_all, filter_string, search_term, ordering_string, '-updated')
 
 
@@ -2179,7 +2182,7 @@ if toastermain.settings.MANAGED:
             # return all project settings
             return HttpResponse(jsonfilter( {
                 "error": "ok",
-                "layers" :  map(lambda x: {"id": x.layercommit.pk, "orderid" : x.pk, "name" : x.layercommit.layer.name, "giturl" : x.layercommit.layer.vcs_url, "url": x.layercommit.layer.layer_index_url, "layerdetailurl": reverse("layerdetails", args=(x.layercommit.layer.pk,)), "branch" : { "name" : x.layercommit.up_branch.name, "layersource" : x.layercommit.up_branch.layer_source.name}}, prj.projectlayer_set.all().order_by("id")),
+                "layers" :  map(lambda x: {"id": x.layercommit.pk, "orderid" : x.pk, "name" : x.layercommit.layer.name, "giturl" : x.layercommit.layer.vcs_url, "url": x.layercommit.layer.layer_index_url, "layerdetailurl": reverse("layerdetails", args=(x.layercommit.layer.pk,)), "branch" : { "name" : x.layercommit.up_branch.name, "layersource" : x.layercommit.up_branch.layer_source.name}}, prj.projectlayer_set.all().select_related("layer").order_by("id")),
                 "builds" : _project_recent_build_list(prj),
                 "variables": map(lambda x: (x.name, x.value), prj.projectvariable_set.all()),
                 "machine": {"name": prj.projectvariable_set.get(name="MACHINE").value},
Signature · 1 line
1.9.1
#7 lore raw

PATCH 7/10 toaster: layerdetails Don't show None type in description/summary

In reply to Alex DAMIAN

From: Michael Wood <[email protected]>

When the result for the summary or description is None don't output the result as the string version of None, use an empty value so that the "Not set" mechanism works.

[YOCTO #7244]

---
 lib/toaster/toastergui/templates/layerdetails.html | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
lib/toaster/toastergui/templates/layerdetails.html +2 -2
index 465d633..c162c8b 100644
@@ -453,7 +453,7 @@
                 </dt>
                 <dd>
                   <span class="muted" style="display:none">Not set</span>
-                  <span class="current-value">{{layerversion.layer.summary}}</span>
+                  <span class="current-value">{{layerversion.layer.summary|default_if_none:''}}</span>
                   <form style="display:none; margin-bottom:20px">
                     <textarea class="span12" rows="2">{% if layerversion.layer.summary %}{{layerversion.layer.summary}}{% endif %}</textarea>
                     <button class="btn change-btn" data-layer-prop="summary" type="button">Save</button>
@@ -467,7 +467,7 @@
                 </dt>
                 <dd>
                   <span class="muted" style="display:none">Not set</span>
-                  <span class="current-value">{{layerversion.layer.description}}</span>
+                  <span class="current-value">{{layerversion.layer.description|default_if_none:''}}</span>
                   <form style="display:none; margin-bottom:20px">
                     <textarea class="span12" rows="6">{% if layerversion.layer.description %}{{layerversion.layer.description}}{% endif %}</textarea>
                     <button class="btn change-btn" data-layer-prop="description" type="button" >Save</button>
Signature · 1 line
1.9.1
#1 lore raw

PATCH 1/10 toaster: bitbake cooker log saving and downloading

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

This patch brings in cooker log saving and proper download links.

* toasterui will now write the cooker log file if running in managed mode
* the BuildRequest has a new state, REQ_ARCHIVE, indicating that the build is completed, and the artifacts are ready to be grabbed
* the runbuild test execution commands will gather needed artifacts, and save them to a storage directory selected during Toaster setup.
* the build dashboard, project builds and all builds pages have permanent links for the cooker log

[YOCTO #7220]
[YOCTO #7206]

---
 lib/bb/ui/buildinfohelper.py                       |   3 +-
 lib/bb/ui/toasterui.py                             |  29 ++++-
 .../management/commands/checksettings.py           |  17 +++
 .../bldcontrol/management/commands/runbuilds.py    |  29 ++++-
 .../bldcontrol/migrations/0008_brarchive.py        | 138 +++++++++++++++++++++
 lib/toaster/bldcontrol/models.py                   |   2 +
 lib/toaster/toastergui/templates/build.html        |   9 +-
 .../toastergui/templates/builddashboard.html       |  11 +-
 .../toastergui/templates/managed_builds.html       |   9 +-
 .../toastergui/templates/projectbuilds.html        |  14 ++-
 .../toastergui/templates/unavailable_artifact.html |  17 +++
 lib/toaster/toastergui/views.py                    |  94 +++++++++++---
 lib/toaster/toastermain/urls.py                    |   6 +-
 13 files changed, 335 insertions(+), 43 deletions(-)
 create mode 100644 lib/toaster/bldcontrol/migrations/0008_brarchive.py
 create mode 100644 lib/toaster/toastergui/templates/unavailable_artifact.html
lib/bb/ui/buildinfohelper.py +2 -1
index 491fd15..1096ccf 100644
@@ -1114,7 +1114,8 @@ class BuildInfoHelper(object):
         be.save()
         br = BuildRequest.objects.get(pk = br_id)
         if errorcode == 0:
-            br.state = BuildRequest.REQ_COMPLETED
+            # request archival of the project artifacts
+            br.state = BuildRequest.REQ_ARCHIVE
         else:
             br.state = BuildRequest.REQ_FAILED
         br.save()
lib/bb/ui/toasterui.py +27 -2
index a85ad5a..df9f362 100644
@@ -58,7 +58,12 @@ def _log_settings_from_server(server):
     if error:
         logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error)
         raise BaseException(error)
-    return includelogs, loglines
+    consolelogfile, error = server.runCommand(["getVariable", "BB_CONSOLELOG"])
+    if error:
+        logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
+        raise BaseException(error)
+    return includelogs, loglines, consolelogfile
+
 
 def main(server, eventHandler, params ):
 
@@ -71,7 +76,7 @@ def main(server, eventHandler, params ):
     console.setFormatter(format)
     logger.addHandler(console)
 
-    includelogs, loglines = _log_settings_from_server(server)
+    includelogs, loglines, consolelogfile = _log_settings_from_server(server)
 
     # verify and warn
     build_history_enabled = True
@@ -96,6 +101,16 @@ def main(server, eventHandler, params ):
 
     buildinfohelper = BuildInfoHelper(server, build_history_enabled)
 
+    if buildinfohelper.brbe is not None and consolelogfile:
+        # if we are under managed mode we have no other UI and we need to write our own file
+        bb.utils.mkdirhier(os.path.dirname(consolelogfile))
+        conlogformat = bb.msg.BBLogFormatter(format_str)
+        consolelog = logging.FileHandler(consolelogfile)
+        bb.msg.addDefaultlogFilter(consolelog)
+        consolelog.setFormatter(conlogformat)
+        logger.addHandler(consolelog)
+
+
     while True:
         try:
             event = eventHandler.waitEvent(0.25)
@@ -115,8 +130,12 @@ def main(server, eventHandler, params ):
 
             if isinstance(event, (bb.build.TaskStarted, bb.build.TaskSucceeded, bb.build.TaskFailedSilent)):
                 buildinfohelper.update_and_store_task(event)
+                logger.warn("Logfile for task %s" % event.logfile)
                 continue
 
+            if isinstance(event, bb.build.TaskBase):
+                logger.info(event._message)
+
             if isinstance(event, bb.event.LogExecTTY):
                 logger.warn(event.msg)
                 continue
@@ -162,7 +181,12 @@ def main(server, eventHandler, params ):
             if isinstance(event, bb.event.CacheLoadCompleted):
                 continue
             if isinstance(event, bb.event.MultipleProviders):
+                logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "",
+                            event._item,
+                            ", ".join(event._candidates))
+                logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item)
                 continue
+
             if isinstance(event, bb.event.NoProvider):
                 return_value = 1
                 errors = errors + 1
@@ -229,6 +253,7 @@ def main(server, eventHandler, params ):
                     buildinfohelper.store_log_event(event)
                     errors += 1
                     errorcode = 1
+                    logger.error("Command execution failed: %s", event.error)
 
                 buildinfohelper.update_build_information(event, errors, warnings, taskfailures)
                 buildinfohelper.close(errorcode)
lib/toaster/bldcontrol/management/commands/checksettings.py +17 -0
index 5d80bc7..1ff5c92 100644
@@ -62,6 +62,23 @@ class Command(NoArgsCommand):
 
 
     def handle(self, **options):
+        # verify that we have a settings for downloading artifacts
+        while ToasterSetting.objects.filter(name="ARTIFACTS_STORAGE_DIR").count() == 0:
+            guessedpath = os.getcwd() + "/toaster_build_artifacts/"
+            print("Toaster needs to know in which directory it can download build log files and other artifacts.\n Toaster suggests \"%s\"." % guessedpath)
+            artifacts_storage_dir = raw_input(" Press Enter to select \"%s\" or type the full path to a different directory: " % guessedpath)
+            if len(artifacts_storage_dir) == 0:
+                artifacts_storage_dir = guessedpath
+            if len(artifacts_storage_dir) > 0 and artifacts_storage_dir.startswith("/"):
+                try:
+                    os.makedirs(artifacts_storage_dir)
+                except OSError as ose:
+                    if "File exists" in str(ose):
+                        pass
+                    else:
+                        raise ose
+                ToasterSetting.objects.create(name="ARTIFACTS_STORAGE_DIR", value=artifacts_storage_dir)
+
         self.guesspath = DN(DN(DN(DN(DN(DN(DN(__file__)))))))
         # refuse to start if we have no build environments
         while BuildEnvironment.objects.count() == 0:
lib/toaster/bldcontrol/management/commands/runbuilds.py +28 -1
index 3de582c..808318f 100644
@@ -1,6 +1,6 @@
 from django.core.management.base import NoArgsCommand, CommandError
 from django.db import transaction
-from orm.models import Build
+from orm.models import Build, ToasterSetting
 from bldcontrol.bbcontroller import getBuildEnvironmentController, ShellCmdException, BuildSetupException
 from bldcontrol.models import BuildRequest, BuildEnvironment, BRError, BRVariable
 import os
@@ -93,7 +93,33 @@ class Command(NoArgsCommand):
             bec.be.lock = BuildEnvironment.LOCK_FREE
             bec.be.save()
 
+    def archive(self):
+        ''' archives data from the builds '''
+        artifact_storage_dir = ToasterSetting.objects.get(name="ARTIFACTS_STORAGE_DIR").value
+        for br in BuildRequest.objects.filter(state = BuildRequest.REQ_ARCHIVE):
+            # save cooker log
+            if br.build == None:
+                br.state = BuildRequest.REQ_FAILED
+                br.save()
+                continue
+            build_artifact_storage_dir = os.path.join(artifact_storage_dir, "%d" % br.build.pk)
+            try:
+                os.makedirs(build_artifact_storage_dir)
+            except OSError as ose:
+                if "File exists" in str(ose):
+                    pass
+                else:
+                    raise ose
+
+            file_name = os.path.join(build_artifact_storage_dir, "cooker_log.txt")
+            try:
+                with open(file_name, "w") as f:
+                    f.write(br.environment.get_artifact(br.build.cooker_log_path).read())
+            except IOError:
+                os.unlink(file_name)
 
+            br.state = BuildRequest.REQ_COMPLETED
+            br.save()
 
     def cleanup(self):
         from django.utils import timezone
@@ -104,4 +130,5 @@ class Command(NoArgsCommand):
 
     def handle_noargs(self, **options):
         self.cleanup()
+        self.archive()
         self.schedule()
lib/toaster/bldcontrol/migrations/0008_brarchive.py new file +138 -0

138 lines not on this page · raw message

lib/toaster/bldcontrol/models.py +2 -0
index dc4afca..25d94cd 100644
@@ -94,6 +94,7 @@ class BuildRequest(models.Model):
     REQ_COMPLETED = 3
     REQ_FAILED = 4
     REQ_DELETED = 5
+    REQ_ARCHIVE = 6
 
     REQUEST_STATE = (
         (REQ_CREATED, "created"),
@@ -102,6 +103,7 @@ class BuildRequest(models.Model):
         (REQ_COMPLETED, "completed"),
         (REQ_FAILED, "failed"),
         (REQ_DELETED, "deleted"),
+        (REQ_ARCHIVE, "archive"),
     )
 
     search_allowed_fields = ("brtarget__target",)
lib/toaster/toastergui/templates/build.html +3 -6
index e71e38f..684ec65 100644
@@ -40,7 +40,9 @@
         <!-- Table data rows; the order needs to match the order of "tablecols" definitions; and the <td class value needs to match the tablecols clclass value for show/hide buttons to work -->
         {% for build in objects %}
         <tr class="data">
-            <td class="outcome"><a href="{% url "builddashboard" build.id %}">{%if build.outcome == build.SUCCEEDED%}<i class="icon-ok-sign success"></i>{%elif build.outcome == build.FAILED%}<i class="icon-minus-sign error"></i>{%else%}{%endif%}</a></td>
+            <td class="outcome">
+	    <a href="{% url "builddashboard" build.id %}">{%if build.outcome == build.SUCCEEDED%}<i class="icon-ok-sign success"></i>{%elif build.outcome == build.FAILED%}<i class="icon-minus-sign error"></i>{%else%}{%endif%}</a> &nbsp;
+	    </td>
             <td class="target">{% for t in build.target_set.all %} <a href="{% url "builddashboard" build.id %}"> {{t.target}} </a> <br />{% endfor %}</td>
             <td class="machine"><a href="{% url "builddashboard" build.id %}">{{build.machine}}</a></td>
             <td class="started_on"><a href="{% url "builddashboard" build.id %}">{{build.started_on|date:"d/m/y H:i"}}</a></td>
@@ -61,11 +63,6 @@
             <td class="errors_no">
                 {% if  build.errors_no %}
                     <a class="errors_no error" href="{% url "builddashboard" build.id %}#errors">{{build.errors_no}} error{{build.errors_no|pluralize}}</a>
-                    {% if MANAGED and build.project %}
-                        <a href="{% url 'build_artifact' build.id "cookerlog" build.id %}">
-                            <i class="icon-download-alt" title="" data-original-title="Download build log"></i>
-                        </a>
-                    {% endif %}
                 {%endif%}
             </td>
             <td class="warnings_no">{% if  build.warnings_no %}<a class="warnings_no warning" href="{% url "builddashboard" build.id %}#warnings">{{build.warnings_no}} warning{{build.warnings_no|pluralize}}</a>{%endif%}</td>
lib/toaster/toastergui/templates/builddashboard.html +6 -5
index 2458cdb..c0898e2 100644
@@ -36,7 +36,11 @@
 {% endif %}
     <span > <i class="icon-warning-sign yellow"></i><strong><a href="#warnings" class="warning show-warnings"> {{build.warnings_no}} warning{{build.warnings_no|pluralize}}</a></strong></span>
 {% endif %}
-            <span class="pull-right">Build time: <a href="{% url 'buildtime' build.pk %}">{{ build.timespent|sectohms }}</a></span>
+            <span class="pull-right">Build time: <a href="{% url 'buildtime' build.pk %}">{{ build.timespent|sectohms }}</a> &nbsp;
+      {% if MANAGED and build.project %}
+            <a class="btn {%if build.outcome == build.SUCCEEDED%}btn-success{%else%}btn-danger{%endif%} pull-right" href="{% url 'build_artifact' build.id "cookerlog" build.id %}">Download build log</a>
+      {% endif %}
+            </span>
 {%endif%}
     </div>
     {% if build.toaster_exceptions.count > 0 %}
@@ -54,10 +58,7 @@
 <div class="accordion span10 pull-right" id="errors">
   <div class="accordion-group">
     <div class="accordion-heading">
-      {% if MANAGED and build.project %}
-          <a class="btn btn-large pull-right" href="{% url 'build_artifact' build.id "cookerlog" build.id %}" style="margin:15px;">Download build log</a>
-      {% endif %}
-      <a class="accordion-toggle error toggle-errors">
+            <a class="accordion-toggle error toggle-errors">
          <h2 id="error-toggle">
            <i class="icon-minus-sign"></i>
            {{build.errors_no}} error{{build.errors_no|pluralize}}
lib/toaster/toastergui/templates/managed_builds.html +8 -1
index 121ad07..a4db55b 100644
@@ -46,7 +46,14 @@
         <!-- Table data rows; the order needs to match the order of "tablecols" definitions; and the <td class value needs to match the tablecols clclass value for show/hide buttons to work -->
         {% for buildrequest in objects %}{% if buildrequest.build %}  {% with build=buildrequest.build %} {# if we have a build, just display it #}
         <tr class="data">
-            <td class="outcome"><a href="{% url "builddashboard" build.id %}">{%if build.outcome == build.SUCCEEDED%}<i class="icon-ok-sign success"></i>{%elif build.outcome == build.FAILED%}<i class="icon-minus-sign error"></i>{%else%}{%endif%}</a></td>
+            <td class="outcome"><a href="{% url "builddashboard" build.id %}">{%if build.outcome == build.SUCCEEDED%}<i class="icon-ok-sign success"></i>{%elif build.outcome == build.FAILED%}<i class="icon-minus-sign error"></i>{%else%}{%endif%}</a>
+	            {% if build.project %}
+                        &nbsp; <a href="{% url 'build_artifact' build.id "cookerlog" build.id %}">
+                            <i class="icon-download-alt" title="" data-original-title="Download build log"></i>
+                        </a>
+                    {% endif %}
+
+            </td>
             <td class="target">{% for t in build.target_set.all %} <a href="{% url "builddashboard" build.id %}"> {{t.target}} </a> <br />{% endfor %}</td>
             <td class="machine"><a href="{% url "builddashboard" build.id %}">{{build.machine}}</a></td>
             <td class="started_on"><a href="{% url "builddashboard" build.id %}">{{build.started_on|date:"d/m/y H:i"}}</a></td>
lib/toaster/toastergui/templates/projectbuilds.html +8 -6
index 02d1663..afcf519 100644
@@ -49,7 +49,14 @@
         <!-- Table data rows; the order needs to match the order of "tablecols" definitions; and the <td class value needs to match the tablecols clclass value for show/hide buttons to work -->
         {% for br in objects %}{% if br.build %}  {% with build=br.build %} {# if we have a build, just display it #}
         <tr class="data">
-            <td class="outcome"><a href="{% url "builddashboard" build.id %}">{%if build.outcome == build.SUCCEEDED%}<i class="icon-ok-sign success"></i>{%elif build.outcome == build.FAILED%}<i class="icon-minus-sign error"></i>{%else%}{%endif%}</a></td>
+            <td class="outcome"><a href="{% url "builddashboard" build.id %}">{%if build.outcome == build.SUCCEEDED%}<i class="icon-ok-sign success"></i>{%elif build.outcome == build.FAILED%}<i class="icon-minus-sign error"></i>{%else%}{%endif%}</a>
+                   {% if build.project %}
+                        &nbsp; <a href="{% url 'build_artifact' build.id "cookerlog" build.id %}">
+                            <i class="icon-download-alt" title="" data-original-title="Download build log"></i>
+                        </a>
+                    {% endif %}
+            </td>
+
             <td class="target">{% for t in build.target_set.all %} <a href="{% url "builddashboard" build.id %}"> {{t.target}} </a> <br />{% endfor %}</td>
             <td class="machine"><a href="{% url "builddashboard" build.id %}">{{build.machine}}</a></td>
             <td class="started_on"><a href="{% url "builddashboard" build.id %}">{{build.started_on|date:"d/m/y H:i"}}</a></td>
@@ -70,11 +77,6 @@
             <td class="errors_no">
                 {% if  build.errors_no %}
                     <a class="errors_no error" href="{% url "builddashboard" build.id %}#errors">{{build.errors_no}} error{{build.errors_no|pluralize}}</a>
-                    {% if MANAGED and build.project %}
-                        <a href="{% url 'build_artifact' build.id "cookerlog" build.id %}">
-                            <i class="icon-download-alt" title="" data-original-title="Download build log"></i>
-                        </a>
-                    {% endif %}
                 {%endif%}
             </td>
             <td class="warnings_no">{% if  build.warnings_no %}<a class="warnings_no warning" href="{% url "builddashboard" build.id %}#warnings">{{build.warnings_no}} warning{{build.warnings_no|pluralize}}</a>{%endif%}</td>
lib/toaster/toastergui/templates/unavailable_artifact.html new file +17 -0
new file mode 100644
index 0000000..c93d425
@@ -0,0 +1,17 @@
+{% extends "base.html" %}
+{% load projecttags %}
+{% load humanize %}
+{% load static %}
+
+{% block pagecontent %}
+<div class="section">
+</div>
+<div class="row-fluid">
+
+    <div class="alert alert-info">
+    <p class="lead"> The artifact you are seeking to download is not available. We are sorry. You can <a href="javascript:window.history.back()">go back</a>
+    </p>
+    </div>
+</div>
+{% endblock %}
+
lib/toaster/toastergui/views.py +76 -18

111 lines not on this page · raw message

lib/toaster/toastermain/urls.py +3 -3
index f66f11d..395c4e8 100644
@@ -50,12 +50,12 @@ import toastermain.settings
 
 if toastermain.settings.FRESH_ENABLED:
     urlpatterns.insert(1, url(r'', include('fresh.urls')))
-    logger.info("Enabled django-fresh extension")
+    #logger.info("Enabled django-fresh extension")
 
 if toastermain.settings.DEBUG_PANEL_ENABLED:
     import debug_toolbar
     urlpatterns.insert(1, url(r'', include(debug_toolbar.urls)))
-    logger.info("Enabled django_toolbar extension")
+    #logger.info("Enabled django_toolbar extension")
 
 
 if toastermain.settings.MANAGED:
@@ -86,4 +86,4 @@ for t in os.walk(os.path.dirname(currentdir)):
             logger.warn("Module \'%s\' has a regexp conflict, was not added to the urlpatterns" % modulename)
 
 from pprint import pformat
-logger.debug("urlpatterns list %s", pformat(urlpatterns))
+#logger.debug("urlpatterns list %s", pformat(urlpatterns))
Signature · 1 line
1.9.1
#6 lore raw

PATCH 6/10 toaster: landing page when no build module

From: David Reyna <[email protected]>

Add landing page for managed build pages when build module not installed.

Some wording and whitespace changed.

[YOCTO #6483]

---
 .../toastergui/templates/landing_not_managed.html  | 32 +++++++++++++++++
 lib/toaster/toastergui/views.py                    | 42 +++++++++++-----------
 2 files changed, 53 insertions(+), 21 deletions(-)
 create mode 100644 lib/toaster/toastergui/templates/landing_not_managed.html
lib/toaster/toastergui/templates/landing_not_managed.html new file +32 -0
new file mode 100644
index 0000000..fe56655
@@ -0,0 +1,32 @@
+{% extends "base.html" %}
+
+{% load static %}
+{% load projecttags %}
+{% load humanize %}
+
+{% block pagecontent %}
+
+  <div class="container-fluid">
+   <div class="row-fluid">
+    <!-- Empty - no build module -->
+    <div class="page-header top-air">
+     <h1>
+       This page only works with the Toaster 'Build' mode
+     </h1>
+    </div>
+    <div class="alert alert-info">
+     <p class="lead">
+     The 'Build' mode allows you to configure and run your Yocto Project builds from Toaster
+     <ul>
+       <li class="lead"><a href="https://wiki.yoctoproject.org/wiki/Toaster#Modes">
+       Read about the 'Build' mode
+       </a></li>
+       <li class="lead"><a href="/">
+       View your builds
+       </a></li>
+     </ul>
+     </p>
+    </div>
+   </div>
+
+{% endblock %}
lib/toaster/toastergui/views.py +21 -21

85 lines not on this page · raw message

Signature · 1 line
1.9.1
#8 lore raw

PATCH 8/10 toasterui: style the Toaster version information

In reply to Alex DAMIAN

From: Belen Barros Pena <[email protected]>

In debug mode, we show the Toaster version and mode in the top bar. Display them inside a tooltip that appears when you hover over a nice info icon to make them less conspicuous.

---
 lib/toaster/toastergui/static/css/default.css | 22 ++++------------------
 lib/toaster/toastergui/templates/base.html    | 13 ++++++-------
 2 files changed, 10 insertions(+), 25 deletions(-)
lib/toaster/toastergui/static/css/default.css +4 -18
index 41b73cc..86cf67a 100644
@@ -2,24 +2,10 @@
 .logo img { height: 30px; width: auto !important; }
 .logo { padding-top: 4px !important; padding-bottom:0px !important; }
 
-/* style the version text */
-.version-text {
-	color: #AAA;
-	font-size: 0.6em;
-	display: table-footer-group;
-	padding: 0px;
-	margin: 1px 1px 1px 1px;
-}
-
-.version-brand, .version-brand:hover, a.version-brand:focus  {
-    color: #777;
-    display: block;
-    float: left;
-    font-size: 20px;
-    font-weight: 200;
-    padding: 00px 0px;
-    text-shadow: 0 1px 0 #fff;
-}
+/* style the version information */
+.brand > a { color: #777; }
+.brand > a:hover { color: #999; text-decoration: none; }
+.icon-info-sign.get-help { color: #777; font-size: 16px; margin-left: 5px;}
 
 /* Style the breadcrumb */
 .breadcrumb { display: inline-block; background-color: transparent; }
lib/toaster/toastergui/templates/base.html +6 -7
index 6f249bc..695ca5d 100644
@@ -58,13 +58,12 @@
 <div class="navbar navbar-static-top">
     <div class="navbar-inner">
             <a class="brand logo" href="#"><img src="{% static 'img/logo.png' %}" class="" alt="Yocto logo project"/></a>
-            <a class="brand" href="/">Toaster</a>
-            {% if DEBUG %}
-            <div style="display: inline;" class="version-brand">
-              <p class="version-text">version {{TOASTER_VERSION}}</p>
-              <p class="version-text">{%if MANAGED%}managed{%else%}interactive{%endif%} mode</p>
-            </div>
-            {% endif %}
+            <span class="brand">
+                <a href="/">Toaster</a>
+                {% if DEBUG %}
+                    <i class="icon-info-sign get-help heading-help" data-placement="bottom" title="Version: {{TOASTER_VERSION}} <br /> Mode: {%if MANAGED%}build{%else%}analysis{%endif%}"></i>
+                {% endif %}
+            </span>
             <a class="pull-right manual" target="_blank" href="http://www.yoctoproject.org/documentation/toaster-manual">
                 <i class="icon-book"></i>
                 Toaster manual
Signature · 1 line
1.9.1
#9 lore raw

PATCH 9/10 toasterui: remove layer directory in managed mode

In reply to Alex DAMIAN

From: Belen Barros Pena <[email protected]>

Removes the layer directory information from the packages built page when in managed mode.

[YOCTO #7221]

---
 lib/toaster/toastergui/templates/bpackage.html |  4 +++-
 lib/toaster/toastergui/views.py                | 14 ++++++++++----
 2 files changed, 13 insertions(+), 5 deletions(-)
lib/toaster/toastergui/templates/bpackage.html +3 -1
index 2d4948b..80229f9 100644
@@ -88,7 +88,9 @@
                 </a>
             </td>
             <!-- Layer directory -->
-            <td class="recipe__layer_version__layer__local_path">{{package.recipe.layer_version.layer.local_path}}</td>
+            {% if not MANAGED or not build.project %}
+                <td class="recipe__layer_version__layer__local_path">{{package.recipe.layer_version.layer.local_path}}</td>
+            {% endif %}
         {%else%}
             <td class="recipe__name"></td>
             <td class="recipe__version"></td>
lib/toaster/toastergui/views.py +10 -4
index 70241dc..bf148e2 100755
@@ -1282,9 +1282,11 @@ def bpackage(request, build_id):
 
     packages = _build_page_range(Paginator(queryset, pagesize),request.GET.get('page', 1))
 
+    build = Build.objects.get( pk = build_id )
+
     context = {
         'objectname': 'packages built',
-        'build': Build.objects.get(pk=build_id),
+        'build': build,
         'objects' : packages,
         'default_orderby' : 'name:+',
         'tablecols':[
@@ -1349,16 +1351,20 @@ def bpackage(request, build_id):
                 'qhelp':'The Git commit of the layer providing the recipe that builds the package',
                 'clclass': 'recipe__layer_version__layer__commit', 'hidden': 1,
             },
-            {
+            ]
+        }
+
+    if not toastermain.settings.MANAGED or build.project is None:
+
+        tc_layerDir = {
                 'name':'Layer directory',
                 'qhelp':'Path to the layer providing the recipe that builds the package',
                 'orderfield': _get_toggle_order(request, "recipe__layer_version__layer__local_path"),
                 'ordericon':_get_toggle_order_icon(request, "recipe__layer_version__layer__local_path"),
                 'orderkey' : 'recipe__layer_version__layer__local_path',
                 'clclass': 'recipe__layer_version__layer__local_path', 'hidden': 1,
-            },
-            ]
         }
+        context['tablecols'].append(tc_layerDir)
 
     response = render(request, template, context)
     _save_parameters_cookies(response, pagesize, orderby, request)
Signature · 1 line
1.9.1
#10 lore raw

PATCH 10/10 toaster: importlayer Remove description input field

In reply to Alex DAMIAN

From: Michael Wood <[email protected]>

This wasn't required or working due to a typo and adds ambiguity between the summary and description. The correct method for changing the description or summary is via the layerdetails page.

[YOCTO #7190]

---
 lib/toaster/toastergui/static/js/importlayer.js   | 1 -
 lib/toaster/toastergui/templates/importlayer.html | 6 ------
 lib/toaster/toastergui/views.py                   | 3 ---
 3 files changed, 10 deletions(-)
lib/toaster/toastergui/static/js/importlayer.js +0 -1
index e782bda..9d54286 100644
@@ -136,7 +136,6 @@ function importLayerPageInit (ctx) {
         name: layerNameInput.val(),
         vcs_url: vcsURLInput.val(),
         git_ref: gitRefInput.val(),
-        summary:  $("#layer-summary").val(),
         dir_path: $("#layer-subdir").val(),
         project_id: ctx.projectId,
         layer_deps: layerDepsCsv,
lib/toaster/toastergui/templates/importlayer.html +0 -6
index a4d8ee1..1196e0c 100644
@@ -84,12 +84,6 @@
                             </div>
                           </div>
 
-                          <label class="project-form" for="layer-description">Layer description
-                            <span class="muted">(optional)</span>
-                            <span class="icon-question-sign get-help" title="A short layer explanation" />
-                          </label>
-                          <textarea id="layer-description" class="input-xxlarge"></textarea>
-
                     </fieldset>
                     <fieldset class="air">
                         <legend>
lib/toaster/toastergui/views.py +0 -3
index bf148e2..be74c10 100755
@@ -2422,9 +2422,6 @@ if toastermain.settings.MANAGED:
             if layer_created:
                 layer.layer_source = layer_source
                 layer.vcs_url = post_data['vcs_url']
-                if post_data.has_key('summary'):
-                    layer.summary = layer.description = post_data['summary']
-
                 layer.up_date = timezone.now()
                 layer.save()
             else:
Signature · 1 line
1.9.1