Skip to contentSkip to conversation

Toaster patchset

PATCH New Alex DAMIAN 1 participant 9 messages Feb 26, 2015

lore
Share & tools
lore
#0 lore raw

cover Toaster patchset

From: Alexandru DAMIAN <[email protected]>

Hello,

This patchset is targeted at bringing critical fixes for M3 rc2 release, including multiple show stoppers. It also adds fixes for M3 features.

The patches have been tested and reviewed on the toaster mailing list.

Can you please pull before M3 rc2 cut-off ?

Thank you,
Alex

The following changes since commit 7182cdd3c4a534a87147bb0aad7b360ffef6426b:

bitbake-layers: fix wildcard support in remove-layer (2015-02-25 23:28:15 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib adamian/20150226-submission-bb
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=adamian/20150226-submission-bb

Alexandru DAMIAN (5):
toastergui: stop covered task graph traversal early
toasterui: fix sstate task identification
toaster: order layers with same priority
toasterui: do not use transactions if the database autocommits
toastergui: add csrf tokens for AJAX calls

Belen Barros Pena (1):
toaster: fix typo in models.py

David Reyna (2):
toaster: all projects data and sorts
toaster: config var layout fixes

 lib/bb/ui/buildinfohelper.py                       |  24 +++--
 lib/toaster/bldcontrol/models.py                   |   2 +-
 lib/toaster/orm/models.py                          |  80 +++++++++++++---
 lib/toaster/toastergui/static/css/default.css      |   2 +-
 lib/toaster/toastergui/templates/layers.html       |   1 +
 .../toastergui/templates/managed_builds.html       |  22 +++--
 .../toastergui/templates/managed_mrb_section.html  |   1 +
 lib/toaster/toastergui/templates/projectconf.html  |  23 ++---
 lib/toaster/toastergui/templates/projects.html     |  60 +++++++++---
 lib/toaster/toastergui/templates/targets.html      |   1 +
 lib/toaster/toastergui/views.py                    | 104 ++++++++++++++-------
 11 files changed, 229 insertions(+), 91 deletions(-)
Signature · 1 line
1.9.1
#1 lore raw

PATCH 1/8 toaster: fix typo in models.py

From: Belen Barros Pena <[email protected]>

The typo caused the tasks table to break when a task has no help text.

---
 lib/toaster/orm/models.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/toaster/orm/models.py +1 -1
index 9e6089b..d2b579f 100644
@@ -366,7 +366,7 @@ class Task(models.Model):
             return self._helptext
         try:
             self._helptext = HelpText.objects.get(key=self.task_name, area=HelpText.VARIABLE, build=self.build).text
-        except HelpText.DoesNotExit:
+        except HelpText.DoesNotExist:
             self._helptext = None
 
         return self._helptext
Signature · 1 line
1.9.1
#2 lore raw

PATCH 2/8 toastergui: stop covered task graph traversal early

From: Alexandru DAMIAN <[email protected]>

This patch stops the breadth-frist graph traversal for finding covering tasks after the finding the first level of executed tasks.

The reasoning is that the first level of executed tasks is the actually relevent information on how a task was covered.

---
 lib/toaster/toastergui/views.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
lib/toaster/toastergui/views.py +3 -4
index aabc3d0..a206f80 100755
@@ -357,6 +357,8 @@ def generateCoveredList2( revlist = [] ):
     covered_list =  [ x for x in revlist if x.outcome == Task.OUTCOME_COVERED ]
     while len(covered_list):
         revlist =  [ x for x in revlist if x.outcome != Task.OUTCOME_COVERED ]
+        if len(revlist) > 0:
+            return revlist
 
         newlist = _find_task_revdep_list(covered_list)
 
@@ -379,10 +381,7 @@ def task( request, build_id, task_id ):
     coveredBy = '';
     if ( task.outcome == Task.OUTCOME_COVERED ):
 #        _list = generateCoveredList( task )
-        _list = generateCoveredList2( _find_task_revdep( task ) )
-        coveredBy = [ ]
-        for t in _list:
-            coveredBy.append( t )
+        coveredBy = sorted(generateCoveredList2( _find_task_revdep( task ) ), key = lambda x: x.recipe.name)
     log_head = ''
     log_body = ''
     if task.outcome == task.OUTCOME_FAILED:
Signature · 1 line
1.9.1
#3 lore raw

PATCH 3/8 toasterui: fix sstate task identification

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

This patch fixes a problem where set sstate scene tasks were not identified, causing cache attempt not being recorded.

[YOCTO #7223]

---
 lib/bb/ui/buildinfohelper.py |  4 ++--
 lib/toaster/orm/models.py    | 13 ++++---------
 2 files changed, 6 insertions(+), 11 deletions(-)
lib/bb/ui/buildinfohelper.py +2 -2
index 967e4bd..c0f42d2 100644
@@ -190,8 +190,8 @@ class ORMWrapper(object):
                     vars(task_object)[v] = task_information[v]
                     object_changed = True
 
-        # update setscene-related information if the task was just created
-        if created and task_object.outcome == Task.OUTCOME_COVERED and 1 == Task.objects.related_setscene(task_object).count():
+        # update setscene-related information if the task has a setscene
+        if task_object.outcome == Task.OUTCOME_COVERED and 1 == task_object.get_related_setscene().count():
             task_object.outcome = Task.OUTCOME_CACHED
             object_changed = True
 
lib/toaster/orm/models.py +4 -9
index d2b579f..03efac4 100644
@@ -281,10 +281,6 @@ class Target_File(models.Model):
     sym_target = models.ForeignKey('Target_File', related_name="symlink_set", null=True)
 
 
-class TaskManager(models.Manager):
-    def related_setscene(self, task_object):
-        return Task.objects.filter(task_executed=True, build = task_object.build, recipe = task_object.recipe, task_name=task_object.task_name+"_setscene")
-
 class Task(models.Model):
 
     SSTATE_NA = 0
@@ -339,10 +335,8 @@ class Task(models.Model):
 
     search_allowed_fields = [ "recipe__name", "recipe__version", "task_name", "logfile" ]
 
-    objects = TaskManager()
-
     def get_related_setscene(self):
-        return Task.objects.related_setscene(self)
+        return Task.objects.filter(task_executed=True, build = self.build, recipe = self.recipe, task_name=self.task_name+"_setscene")
 
     def get_outcome_text(self):
         return Task.TASK_OUTCOME[self.outcome + 1][1]
@@ -377,7 +371,7 @@ class Task(models.Model):
     outcome = models.IntegerField(choices=TASK_OUTCOME, default=OUTCOME_NA)
     sstate_checksum = models.CharField(max_length=100, blank=True)
     path_to_sstate_obj = models.FilePathField(max_length=500, blank=True)
-    recipe = models.ForeignKey('Recipe', related_name='build_recipe')
+    recipe = models.ForeignKey('Recipe', related_name='tasks')
     task_name = models.CharField(max_length=100)
     source_url = models.FilePathField(max_length=255, blank=True)
     work_directory = models.FilePathField(max_length=255, blank=True)
@@ -394,7 +388,8 @@ class Task(models.Model):
     sstate_text  = property(get_sstate_text)
 
     def __unicode__(self):
-        return "%d %s:%s" % (self.id, self.recipe.name, self.task_name)
+        return "%d(%d) %s:%s" % (self.pk, self.build.pk, self.recipe.name, self.task_name)
+
     class Meta:
         ordering = ('order', 'recipe' ,)
         unique_together = ('build', 'recipe', 'task_name', )
Signature · 1 line
1.9.1
#4 lore raw

PATCH 4/8 toaster: order layers with same priority

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

When we have multiple layers with the same priority, we need to order them by date, newest first. The layer version ID is a complete ordering by recent-ness, so it is suitable.

---
 lib/toaster/orm/models.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/toaster/orm/models.py +1 -1
index 03efac4..4fa9f81 100644
@@ -994,7 +994,7 @@ class Layer_Version(models.Model):
             query |= Q(projectlayer__project=project)
 
         return sorted(
-                Layer_Version.objects.filter(layer__name = self.layer.name).filter(query).select_related('layer_source', 'layer'),
+                Layer_Version.objects.filter(layer__name = self.layer.name).filter(query).select_related('layer_source', 'layer').order_by("-id"),
                 key = lambda x: _get_ls_priority(x.layer_source),
                 reverse = True)
 
Signature · 1 line
1.9.1
#5 lore raw

PATCH 5/8 toasterui: do not use transactions if the database autocommits

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Some databases, notably the SQLite3 adapter, force autocommits even if the autocommit is turned off. The behavious is tracked in this bug:
http://bugs.python.org/issue8145#msg109965

Django refuses to work with autocommit off in this case, so we have to take the same precautions when using manual transaction support.

[YOCTO #7363]
[YOCTO #7365]

---
 lib/bb/ui/buildinfohelper.py | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)
lib/bb/ui/buildinfohelper.py +12 -8
index c0f42d2..19ddd8e 100644
@@ -35,7 +35,7 @@ from bb.msg import BBLogFormatter as format
 from django.db import models
 import logging
 
-from django.db import transaction
+from django.db import transaction, connection
 
 logger = logging.getLogger("BitBake")
 
@@ -609,7 +609,9 @@ class BuildInfoHelper(object):
         self.task_order = 0
         self.autocommit_step = 1
         self.server = server
-        transaction.set_autocommit(False)
+        # we use manual transactions if the database doesn't autocommit on us
+        if not connection.features.autocommits_when_autocommit_is_off:
+            transaction.set_autocommit(False)
         self.orm_wrapper = ORMWrapper()
         self.has_build_history = has_build_history
         self.tmp_dir = self.server.runCommand(["getVariable", "TMPDIR"])[0]
@@ -929,11 +931,12 @@ class BuildInfoHelper(object):
                 task_information['outcome'] = Task.OUTCOME_FAILED
                 del self.internal_state['taskdata'][identifier]
 
-        # we force a sync point here, to get the progress bar to show
-        if self.autocommit_step % 3 == 0:
-            transaction.set_autocommit(True)
-            transaction.set_autocommit(False)
-        self.autocommit_step += 1
+        if not connection.features.autocommits_when_autocommit_is_off:
+            # we force a sync point here, to get the progress bar to show
+            if self.autocommit_step % 3 == 0:
+                transaction.set_autocommit(True)
+                transaction.set_autocommit(False)
+            self.autocommit_step += 1
 
         self.orm_wrapper.get_update_task_object(task_information, True) # must exist
 
@@ -1207,4 +1210,5 @@ class BuildInfoHelper(object):
                 for event in self.internal_state['backlog']:
                    logger.error("UNSAVED log: %s", event.msg)
 
-        transaction.set_autocommit(True)
+        if not connection.features.autocommits_when_autocommit_is_off:
+            transaction.set_autocommit(True)
Signature · 1 line
1.9.1
#6 lore raw

PATCH 6/8 toastergui: add csrf tokens for AJAX calls

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

This patch adds CSRF tokens in pages using AJAX calls in order to force Django to generate the CSRF cookie used to authenticate the call.

---
 lib/toaster/toastergui/templates/layers.html              | 1 +
 lib/toaster/toastergui/templates/managed_mrb_section.html | 1 +
 lib/toaster/toastergui/templates/projectconf.html         | 1 +
 lib/toaster/toastergui/templates/targets.html             | 1 +
 4 files changed, 4 insertions(+)
lib/toaster/toastergui/templates/layers.html +1 -0
index 5cfda5b..864e156 100644
@@ -117,6 +117,7 @@
 
 var tooltipUpdateText;
 
+/* ensure csrf cookie exists {% csrf_token %} */
 function _makeXHREditCall(data, onsuccess, onfail) {
     $.ajax( {
         type: "POST",
lib/toaster/toastergui/templates/managed_mrb_section.html +1 -0
index 08bb05d..d6e32f8 100644
@@ -156,6 +156,7 @@
 
 <script>
 
+/* ensure csrf cookie exists {% csrf_token %} */
 function _makeXHRBuildCall(url, data, onsuccess, onfail) {
     $.ajax( {
         type: "POST",
lib/toaster/toastergui/templates/projectconf.html +1 -0
index 9cacce4..dde8a11 100644
@@ -429,6 +429,7 @@
             // re-assert the event handlers
         }
 
+	/* ensure cookie exists {% csrf_token %} */
         function postEditAjaxRequest(reqdata) {
             var ajax = $.ajax({
                     type:"POST",
lib/toaster/toastergui/templates/targets.html +1 -0
index 234159a..690c6a5 100644
@@ -120,6 +120,7 @@
 
 var tooltipUpdateText;
 
+/* ensure csrf cookie exists {% csrf_token %} */
 function _makeXHREditCall(data, onsuccess, onfail) {
     $.ajax( {
         type: "POST",
Signature · 1 line
1.9.1
#7 lore raw

PATCH 7/8 toaster: all projects data and sorts

In reply to Alex DAMIAN

From: David Reyna <[email protected]>

Implement the 'last build' data methods, enhance variable display, add empty page and empty sort support.

[YOCTO #6682]

---
 lib/toaster/bldcontrol/models.py                   |  2 +-
 lib/toaster/orm/models.py                          | 63 ++++++++++++++
 .../toastergui/templates/managed_builds.html       | 22 ++---
 lib/toaster/toastergui/templates/projects.html     | 60 ++++++++++---
 lib/toaster/toastergui/views.py                    | 97 +++++++++++++++-------
 5 files changed, 190 insertions(+), 54 deletions(-)
lib/toaster/bldcontrol/models.py +1 -1
index 25d94cd..02cfaf7 100644
@@ -106,7 +106,7 @@ class BuildRequest(models.Model):
         (REQ_ARCHIVE, "archive"),
     )
 
-    search_allowed_fields = ("brtarget__target",)
+    search_allowed_fields = ("brtarget__target", "build__project__name")
 
     project     = models.ForeignKey(Project)
     build       = models.OneToOneField(Build, null = True)     # TODO: toasterui should set this when Build is created
lib/toaster/orm/models.py +63 -0

69 lines not on this page · raw message

lib/toaster/toastergui/templates/managed_builds.html +12 -10
index a4db55b..e23b832 100644
@@ -56,6 +56,13 @@
             </td>
             <td class="target">{% for t in build.target_set.all %} <a href="{% url "builddashboard" build.id %}"> {{t.target}} </a> <br />{% endfor %}</td>
             <td class="machine"><a href="{% url "builddashboard" build.id %}">{{build.machine}}</a></td>
+   {% if MANAGED %}
+            <td class="project">
+            {% if build.project %}
+                <a href="{% url 'project' build.project.id %}">{{build.project.name}}</a>
+            {% endif %}
+            </td>
+   {% endif %}
             <td class="started_on"><a href="{% url "builddashboard" build.id %}">{{build.started_on|date:"d/m/y H:i"}}</a></td>
             <td class="completed_on"><a href="{% url "builddashboard" build.id %}">{{build.completed_on|date:"d/m/y H:i"}}</a></td>
             <td class="failed_tasks error">
@@ -91,13 +98,6 @@
               <a href="{%url "builddashboard" build.id%}#images">{{fstypes|get_dict_value:build.id}}</a>
               {% endif %}
             </td>
-   {% if MANAGED %}
-            <td class="project">
-            {% if build.project %}
-                <a href="{% url 'project' build.project.id %}">{{build.project.name}}</a>
-            {% endif %}
-            </td>
-   {% endif %}
         </tr>
 
 
@@ -114,6 +114,11 @@
             <td class="machine">
                 <a href="{% url "buildrequestdetails" buildrequest.project.id buildrequest.id %}">{{buildrequest.machine}}</a>
             </td>
+   {% if MANAGED %}
+            <td class="project">
+                <a href="{% url 'project' buildrequest.project.id %}">{{buildrequest.project.name}}</a>
+            </td>
+   {% endif %}
             <td class="started_on">
                 <a href="{% url "buildrequestdetails" buildrequest.project.id buildrequest.id %}">{{buildrequest.created|date:"d/m/y H:i"}}</a>
             </td>
@@ -132,9 +137,6 @@
             </td>
             <td class="output"> {# we have no output here #}
             </td>
-            <td class="project">
-                <a href="{% url 'project' buildrequest.project.id %}">{{buildrequest.project.name}}</a>
-            </td>
         </tr>
           {%endif%}
         {% endfor %}
lib/toaster/toastergui/templates/projects.html +48 -12

73 lines not on this page · raw message

lib/toaster/toastergui/views.py +66 -31

190 lines not on this page · raw message

Signature · 1 line
1.9.1
#8 lore raw

PATCH 8/8 toaster: config var layout fixes

In reply to Alex DAMIAN

From: David Reyna <[email protected]>

Block duplicates of custom variables regardless of case, fix spacing of delete icon, no extra error msg on blank DISTRO name

[YOCTO #7359]

---
 lib/toaster/toastergui/static/css/default.css     |  2 +-
 lib/toaster/toastergui/templates/projectconf.html | 22 +++++++++++-----------
 2 files changed, 12 insertions(+), 12 deletions(-)
lib/toaster/toastergui/static/css/default.css +1 -1
index 0cf0f19..dc1b92a 100644
@@ -151,7 +151,7 @@ select { width: auto; }
 #dependencies_modal_form { margin-bottom: 0px; }
 
 /* Configuration styles */
-.icon-trash { color: #B94A48; font-size: 16px; padding-left: 2px; }
+.icon-trash { color: #B94A48; font-size: 16px; padding-left: 5px; }
 .icon-trash:hover { color: #943A38; text-decoration: none; cursor: pointer; }
 .icon-pencil, .icon-download-alt, .icon-refresh, .icon-star-empty, .icon-star, .icon-tasks { font-size: 16px; color: #0088CC; padding-left: 2px; }
 .icon-pencil:hover, .icon-download-alt:hover, .icon-refresh:hover, .icon-star-empty:hover, .icon-star:hover, .icon-tasks:hover { color: #005580; text-decoration: none; cursor: pointer; }
lib/toaster/toastergui/templates/projectconf.html +11 -11
index dde8a11..774e9ea 100644
@@ -23,12 +23,12 @@
                 <i class="icon-pencil" id="change-distro-icon"></i>
                 <form id="change-distro-form" style="display:none;">
                     <div class="input-append">
-                  <span  id="edit-distro-name-div" class="control-group">
-                        <input type="text" id="new-distro" value="{{distro}}">
-                        <button id="apply-change-distro" class="btn" type="button">Save</button>
-                        <button id="cancel-change-distro" type="button" class="btn btn-link">Cancel</button>
+                        <span  id="edit-distro-name-div" class="control-group">
+                            <input type="text" id="new-distro" value="{{distro}}">
+                            <button id="apply-change-distro" class="btn" type="button">Save</button>
+                            <button id="cancel-change-distro" type="button" class="btn btn-link">Cancel</button>
+                        </span>
                         <span class="help-block error" id="distro-error-message"></span>
-                  </span>
                     </div>
                 </form>
             </dd>
@@ -52,7 +52,7 @@
                 <i class="icon-question-sign get-help" title="Specifies additional packages to install into an image. If your build creates more than one image, the packages will be installed in <strong>all of them</strong> <br /><a href='http://www.yoctoproject.org/docs/1.6.1/ref-manual/ref-manual.html#var-IMAGE_INSTALL' target='_blank'>Read more in the manual</a>"></i>
             </dt>
             <dd class="lead">
-                <span id="image_install">{% if image_install_append %}{{image_install_append}}{%else%}<i>Not set</i>{%endif%}</span>
+                <span id="image_install"{% if image_install_append %}{%else%} class="muted"{%endif%}>{% if image_install_append %}{{image_install_append}}{%else%}Not set{%endif%}</span>
                 <i class="icon-pencil" id="change-image_install-icon"></i>
                 <i class="icon-trash" id="delete-image_install-icon" {% if image_install_append %}{%else%}style="display:none;"{%endif%}></i>
                 <form id="change-image_install-form" style="display:none;">
@@ -198,14 +198,14 @@
 
             var existing_configvars = document.getElementsByClassName('js-config-var-name');
             for (var i = 0, length = existing_configvars.length; i < length; i++) {
-                if (existing_configvars[i].innerHTML == variable) {
+                if (existing_configvars[i].innerHTML.toUpperCase() == variable.toUpperCase()) {
                     error_msg = "This variable is already set in this page, edit its value instead";
                 }
             }
 
             var blacklist_configvars = document.getElementsByClassName('js-config-blacklist-name');
             for (var i = 0, length = blacklist_configvars.length; i < length; i++) {
-                if (blacklist_configvars[i].value == variable.toUpperCase()) {
+                if (blacklist_configvars[i].value.toUpperCase() == variable.toUpperCase()) {
                     error_msg = "You cannot edit this variable in Toaster because it is set by the build servers";
                 }
             }
@@ -257,7 +257,7 @@
             if (has_spaces) {
                 error_msg = "A valid distro name cannot include spaces";
             } else if (0 == value.length) {
-                error_msg = "A valid distro name cannot be empty";
+                error_msg = " ";
             }
 
             if ("" != error_msg) {
@@ -609,7 +609,7 @@
                 var name = $('#new-image_install').val();
                 postEditAjaxRequest({"configvarChange" : 'IMAGE_INSTALL_append:'+name});
                 $('#image_install').text(name);
-                $('#image_install').parent().removeClass('muted');
+                $('#image_install').removeClass('muted');
                 $("#change-image_install-form").slideUp(function () {
                     $('#image_install, #change-image_install-icon').show();
                     if (name.length > -1) {
@@ -624,7 +624,7 @@
                 $(this).tooltip('hide');
                 postEditAjaxRequest({"configvarChange" : 'IMAGE_INSTALL_append:'+''});
                 $('#image_install').parent().fadeOut(1000, function(){
-                    $('#image_install').parent().addClass('muted');
+                    $('#image_install').addClass('muted');
                     $('#image_install').text('Not set');
                     $('#delete-image_install-icon').hide();
                     $('#image_install').parent().fadeIn(1000);
Signature · 1 line
1.9.1