Skip to contentSkip to conversation

toaster: SUMO cummulative patch Aug 28 2018

PATCH In review David Reyna 2 participants 11 messages Sep 6, 2018 – Sep 7, 2018

lore
Share & tools
lore
#0 lore raw

cover toaster: SUMO cummulative patch Aug 28 2018

From: David Reyna <[email protected]>

This cumulative enhancement patch includes the following, to support flexibility and usability for Toaster.

* 12891 - "Allow flexible value for TOASTERDIR plus custom fixture support"

This supports the flexibility of having the Toaster base directory (used got the database and project build directories) to be specified by the user. It also allows for flexibility in the location of bitbake to be within one the specified local layers instead of being in the default fixed location.

* bitbake: toaster: Fix comparison in recipe template

This helps prevent a page failure for certain circumstances, and is well worth porting back to Sumo.

- David

The following changes since commit 45ef387cc54a0584807e05a952e1e4681ec4c664:

bitbake: checksum: sanity check path when recursively checksumming (2018-08-29 15:23:51 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib dreyna/submit/dreyna/toaster/sumo_cummulative_082618
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=dreyna/submit/dreyna/toaster/sumo_cummulative_082618

Awais Belal (8):
toaster: allow pokydirname to be evaluated when all layers are local
toaster: use a more flexible way to find bitbake
bitbake: toaster: allow TOASTER_DIR to be overridden from cmdline toaster/widgets.py: avoid divide by zero issues toastergui/newproject.html: fix release divs toaster/checksettings: allow CUSTOM_XML_ONLY setting through env toaster/models.py: allow local paths for custom recipe's base toaster/layerdetails.js: don't hide local layer info

Karsten Strand (1):
bitbake: toaster: Fix comparison in recipe template

 bin/toaster                                        |  9 +++++--
 lib/toaster/bldcontrol/localhostbecontroller.py    | 31 +++++++++++++++++++---
 .../management/commands/checksettings.py           |  3 ++-
 lib/toaster/orm/models.py                          |  7 +++++
 lib/toaster/toastergui/static/js/layerdetails.js   |  3 ++-
 lib/toaster/toastergui/templates/newproject.html   |  4 +--
 lib/toaster/toastergui/templates/recipe.html       |  2 +-
 lib/toaster/toastergui/widgets.py                  | 19 ++++++++-----
 8 files changed, 61 insertions(+), 17 deletions(-)
Signature · 1 line
1.9.1
#1 lore raw

PATCH 1/9 toaster: allow pokydirname to be evaluated when all layers are local

From: Awais Belal <[email protected]>

Toaster depends on pokydirname for identifying the location of the oe-init-build-env script (and there might be other purposes in the future). The problem with current approach is that it only checks/sets the variable with git based repos, whereas toaster provides mechanisms to allow having layers that are all locally available. The evaluation of the variable fails in such scenarios, so use a more flexible mechanism in this case and try to locate poky in the local layers as well, if not already set.

[YOCTO #12891]

---
 lib/toaster/bldcontrol/localhostbecontroller.py | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
lib/toaster/bldcontrol/localhostbecontroller.py +15 -3
index 16c7c80..f960a38 100644
@@ -217,9 +217,21 @@ class LocalhostBEController(BuildEnvironmentController):
         self.setCloneStatus(bitbake,'complete',clone_total,clone_count)
         logger.debug("localhostbecontroller: current layer list %s " % pformat(layerlist))
 
-        if self.pokydirname is None and os.path.exists(os.path.join(self.be.sourcedir, "oe-init-build-env")):
-            logger.debug("localhostbecontroller: selected poky dir name %s" % self.be.sourcedir)
-            self.pokydirname = self.be.sourcedir
+        # Resolve self.pokydirname if not resolved yet, consider the scenario
+        # where all layers are local, that's the else clause
+        if self.pokydirname is None:
+            if os.path.exists(os.path.join(self.be.sourcedir, "oe-init-build-env")):
+                logger.debug("localhostbecontroller: selected poky dir name %s" % self.be.sourcedir)
+                self.pokydirname = self.be.sourcedir
+            else:
+                # Alternatively, scan local layers for relative "oe-init-build-env" location
+                for layer in layers:
+                    if os.path.exists(os.path.join(layer.layer_version.layer.local_source_dir,"..","oe-init-build-env")):
+                        logger.debug("localhostbecontroller, setting pokydirname to %s" % (layer.layer_version.layer.local_source_dir))
+                        self.pokydirname = os.path.join(layer.layer_version.layer.local_source_dir,"..")
+                        break
+                else:
+                    logger.error("pokydirname is not set, you will run into trouble!")
 
         # 5. create custom layer and add custom recipes to it
         for target in targets:
Signature · 1 line
1.9.1
#2 lore raw

PATCH 2/9 toaster: use a more flexible way to find bitbake

In reply to David Reyna

From: Awais Belal <[email protected]>

The current mechanism for finding the bitbake binary assumes a directory structure which is identical to poky, where oe-core's meta and bitbake directories are at the same level. There can be a case where bitbake is used from elsewhere and in such cases the above mentioned assumption fails to hold, whereas this is totally allowed by the oe-init-build-env script which can take bitbakedir as an argument.
The better approach is to allow bitbake to be derived from PATH, while keeping the older mechanism in place so it can be removed after tests are done in various environments. This makes more sense as toaster has also been launched from the same bitbake instance that is the one in PATH.

[YOCTO #12891]

---
 lib/toaster/bldcontrol/localhostbecontroller.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)
lib/toaster/bldcontrol/localhostbecontroller.py +13 -0
index f960a38..3850334 100644
@@ -351,6 +351,19 @@ class LocalhostBEController(BuildEnvironmentController):
         # clean the Toaster to build environment
         env_clean = 'unset BBPATH;' # clean BBPATH for <= YP-2.4.0
 
+        # run bitbake server from the clone if available
+        # otherwise pick it from the PATH
+        bitbake = os.path.join(self.pokydirname, 'bitbake', 'bin', 'bitbake')
+        if not os.path.exists(bitbake):
+            logger.info("Bitbake not available under %s, will try to use it from PATH" %
+                        self.pokydirname)
+            for path in os.environ["PATH"].split(os.pathsep):
+                if os.path.exists(os.path.join(path, 'bitbake')):
+                    bitbake = os.path.join(path, 'bitbake')
+                    break
+            else:
+                logger.error("Looks like Bitbake is not available, please fix your environment")
+
         # run bitbake server from the clone
         bitbake = os.path.join(self.pokydirname, 'bitbake', 'bin', 'bitbake')
         toasterlayers = os.path.join(builddir,"conf/toaster-bblayers.conf")
Signature · 1 line
1.9.1
#3 lore raw

PATCH 3/9 bitbake: toaster: allow TOASTER_DIR to be overridden from cmdline

In reply to David Reyna

From: Awais Belal <[email protected]>

TOASTER_DIR is used for higher level toaster artifacts such the SQL DB and creating toaster internal build directories for projects. Prior to this change it was evaluated as dirname $BUILDDIR and user had no control over it. This change allows to override this variable from the command line for more flexibility. The variable defaults to its original setting if the optional argument is not passed.

[YOCTO #12891]

(Bitbake rev: e073775d3b6980fc8004ae28a3ccc3c5bbf50fb2)

---
 bin/toaster | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
bin/toaster +7 -2
index ed365ee..9fffbc6 100755
@@ -18,11 +18,12 @@
 # along with this program. If not, see http://www.gnu.org/licenses/.
 
 HELP="
-Usage: source toaster start|stop [webport=<address:port>] [noweb] [nobuild]
+Usage: source toaster start|stop [webport=<address:port>] [noweb] [nobuild] [toasterdir]
     Optional arguments:
         [nobuild] Setup the environment for capturing builds with toaster but disable managed builds
         [noweb] Setup the environment for capturing builds with toaster but don't start the web server
         [webport] Set the development server (default: localhost:8000)
+        [toasterdir] Set absolute path to be used as TOASTER_DIR (default: BUILDDIR/../)
 "
 
 custom_extention()
@@ -186,6 +187,7 @@ unset OE_ROOT
 WEBSERVER=1
 export TOASTER_BUILDSERVER=1
 ADDR_PORT="localhost:8000"
+TOASTERDIR=`dirname $BUILDDIR`
 unset CMD
 for param in $*; do
     case $param in
@@ -211,6 +213,9 @@ for param in $*; do
                 ADDR_PORT="localhost:$PORT"
             fi
     ;;
+    toasterdir=*)
+            TOASTERDIR="${param#*=}"
+    ;;
     --help)
             echo "$HELP"
             return 0
@@ -241,7 +246,7 @@ fi
 # 2) the build dir (in build)
 # 3) the sqlite db if that is being used.
 # 4) pid's we need to clean up on exit/shutdown
-export TOASTER_DIR=`dirname $BUILDDIR`
+export TOASTER_DIR=$TOASTERDIR
 export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE TOASTER_DIR"
 
 # Determine the action. If specified by arguments, fine, if not, toggle it
Signature · 1 line
1.9.1
#4 lore raw

PATCH 4/9 toaster/widgets.py: avoid divide by zero issues

In reply to David Reyna

From: Awais Belal <[email protected]>

There can be cases where the variables being used to divide in build percentage expressions can be zero. For example, a setup consisting of only local repos will have repos_to_clone=0 and will generate a divide by zero scenario.
Fix this by checking the divisor in such cases.

[YOCTO #12891]

---
 lib/toaster/toastergui/widgets.py | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)
lib/toaster/toastergui/widgets.py +12 -7
index a1792d9..feef7c5 100644
@@ -511,13 +511,18 @@ class MostRecentBuildsView(View):
                 buildrequest_id = build_obj.buildrequest.pk
             build['buildrequest_id'] = buildrequest_id
 
-            build['recipes_parsed_percentage'] = \
-                int((build_obj.recipes_parsed /
-                     build_obj.recipes_to_parse) * 100)
-
-            build['repos_cloned_percentage'] = \
-                int((build_obj.repos_cloned /
-                     build_obj.repos_to_clone) * 100)
+            if build_obj.recipes_to_parse > 0:
+                build['recipes_parsed_percentage'] = \
+                    int((build_obj.recipes_parsed /
+                         build_obj.recipes_to_parse) * 100)
+            else:
+                build['recipes_parsed_percentage'] = 0
+            if build_obj.repos_to_clone > 0:
+                build['repos_cloned_percentage'] = \
+                    int((build_obj.repos_cloned /
+                         build_obj.repos_to_clone) * 100)
+            else:
+                build['repos_cloned_percentage'] = 0
 
             tasks_complete_percentage = 0
             if build_obj.outcome in (Build.SUCCEEDED, Build.FAILED):
Signature · 1 line
1.9.1
#5 lore raw

PATCH 5/9 toastergui/newproject.html: fix release divs

In reply to David Reyna

From: Awais Belal <[email protected]>

The release drop down divs are not being closed appropriately, which showed adverse reactions on the UI that aligned the "Create project" button with the left edge of the screen without any margins. This fixes these divs which in turn aligns the button appropriately.

[YOCTO #12891]

---
 lib/toaster/toastergui/templates/newproject.html | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
lib/toaster/toastergui/templates/newproject.html +2 -2
index acb614e..bd03bb5 100644
@@ -54,12 +54,12 @@
                     <span class="help-block">{{release.helptext|safe}}</span>
                   </div>
                 {% endfor %}
+                </div>
+              </div>
             {% else %}
               <input type="hidden" name="projectversion" value="{{releases.0.id}}"/>
             {% endif %}
                 </div>
-              </div>
-            </fieldset>
         {% endif %}
             <div class="top-air">
               <input type="submit" id="create-project-button" class="btn btn-primary btn-lg" value="Create project"/>
Signature · 1 line
1.9.1
#6 lore raw

PATCH 6/9 toaster/checksettings: allow CUSTOM_XML_ONLY setting through env

In reply to David Reyna

From: Awais Belal <[email protected]>

This change allows the CUSTOM_XML_ONLY toaster setting to be provided through the environment so the user can do this without mingling with the settings.xml, for scenarios where modifying settings.xml is not achievable.

[YOCTO #12891]

---
 lib/toaster/bldcontrol/management/commands/checksettings.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
lib/toaster/bldcontrol/management/commands/checksettings.py +2 -1
index 823c6f1..14298d9 100644
@@ -74,8 +74,9 @@ class Command(BaseCommand):
                         print("Loading default settings")
                         call_command("loaddata", "settings")
                         template_conf = os.environ.get("TEMPLATECONF", "")
+                        custom_xml_only = os.environ.get("CUSTOM_XML_ONLY")
 
-                        if ToasterSetting.objects.filter(name='CUSTOM_XML_ONLY').count() > 0:
+                        if ToasterSetting.objects.filter(name='CUSTOM_XML_ONLY').count() > 0 or (not custom_xml_only == None):
                             # only use the custom settings
                             pass
                         elif "poky" in template_conf:
Signature · 1 line
1.9.1
#7 lore raw

PATCH 7/9 toaster/models.py: allow local paths for custom recipe's base

In reply to David Reyna

From: Awais Belal <[email protected]>

In a case where the layer source is local only and the recipe is not yet built, we can search for the path with layer's local_source_dir, and if available that should be used rather than just skipping the scenario.

[YOCTO #12891]

---
 lib/toaster/orm/models.py | 7 +++++++
 1 file changed, 7 insertions(+)
lib/toaster/orm/models.py +7 -0
index 3a7dff8..4b77e8f 100644
@@ -1663,6 +1663,9 @@ class CustomImageRecipe(Recipe):
 
         path_schema_two = self.base_recipe.file_path
 
+        path_schema_three = "%s/%s" % (self.base_recipe.layer_version.layer.local_source_dir,
+                                     self.base_recipe.file_path)
+
         if os.path.exists(path_schema_one):
             return path_schema_one
 
@@ -1670,6 +1673,10 @@ class CustomImageRecipe(Recipe):
         if os.path.exists(path_schema_two):
             return path_schema_two
 
+        # Or a local path if all layers are local
+        if os.path.exists(path_schema_three):
+            return path_schema_three
+
         return None
 
     def generate_recipe_file_contents(self):
Signature · 1 line
1.9.1
#8 lore raw

PATCH 8/9 toaster/layerdetails.js: don't hide local layer info

In reply to David Reyna

From: Awais Belal <[email protected]>

The local layer info (provided through custom fixtures) should not be hidden. It is better to handle it in the same manner as an imported layer, otherwise the layer path and dependency info is not shown. The layer editing fields are handled in the html side of things appropriately so this does not harm that implementation.

[YOCTO #12891]

---
 lib/toaster/toastergui/static/js/layerdetails.js | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
lib/toaster/toastergui/static/js/layerdetails.js +2 -1
index 9ead393..933b65b 100644
@@ -359,7 +359,8 @@ function layerDetailsPageInit (ctx) {
     if ($(this).is("dt")) {
       var dd = $(this).next("dd");
       if (!dd.children("form:visible")|| !dd.find(".current-value").html()){
-        if (ctx.layerVersion.layer_source == ctx.layerSourceTypes.TYPE_IMPORTED){
+        if (ctx.layerVersion.layer_source == ctx.layerSourceTypes.TYPE_IMPORTED ||
+            ctx.layerVersion.layer_source == ctx.layerSourceTypes.TYPE_LOCAL) {
         /* There's no current value and the layer is editable
          * so show the "Not set" and hide the delete icon
          */
Signature · 1 line
1.9.1
#9 lore raw

PATCH 9/9 bitbake: toaster: Fix comparison in recipe template

In reply to David Reyna

From: Karsten Strand <[email protected]>

Use == instead of = when comparing task outcome to OUTCOME_FAILED.

Prior to this fix the recipe template would cause a TemplateSyntaxError exception.

(Bitbake rev: a53ffec4ed3d0f9221bca398e20e8f480fb2b325)

---
 lib/toaster/toastergui/templates/recipe.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/toaster/toastergui/templates/recipe.html +1 -1
index bf2cd71..3f76e65 100644
@@ -176,7 +176,7 @@
                     <td>{{task.get_executed_display}}</td>
 
                     <td>{{task.get_outcome_display}} 
-                        {% if task.outcome = task.OUTCOME_FAILED %}
+                        {% if task.outcome == task.OUTCOME_FAILED %}
                             <a href="{% url 'build_artifact' build.pk "tasklogfile" task.pk %}">
                                 <span class="glyphicon glyphicon-download-alt
                                     get-help" title="Download task log
Signature · 1 line
1.9.1
#10 lore raw
In reply to David Reyna
Quoted 21 lines · David Reyna

From: David Reyna <[email protected]>

This cumulative enhancement patch includes the following, to support flexibility and usability for Toaster.

* 12891 - "Allow flexible value for TOASTERDIR plus custom fixture support"

This supports the flexibility of having the Toaster base directory (used got the database and project build directories) to be specified by the user. It also allows for flexibility in the location of bitbake to be within one the specified local layers instead of being in the default fixed location.

* bitbake: toaster: Fix comparison in recipe template

This helps prevent a page failure for certain circumstances, and is well worth porting back to Sumo.

- David

Quoted 8 lines

The following changes since commit 45ef387cc54a0584807e05a952e1e4681ec4c664:

bitbake: checksum: sanity check path when recursively checksumming (2018-08-29 15:23:51 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib dreyna/submit/dreyna/toaster/sumo_cummulative_082618
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=dreyna/submit/dreyna/toaster/sumo_cummulative_082618

Series in sumo nmut.

thanks,
Armin

Quoted 24 lines

Awais Belal (8):
toaster: allow pokydirname to be evaluated when all layers are local
toaster: use a more flexible way to find bitbake
bitbake: toaster: allow TOASTER_DIR to be overridden from cmdline toaster/widgets.py: avoid divide by zero issues toastergui/newproject.html: fix release divs toaster/checksettings: allow CUSTOM_XML_ONLY setting through env toaster/models.py: allow local paths for custom recipe's base toaster/layerdetails.js: don't hide local layer info

Karsten Strand (1):
bitbake: toaster: Fix comparison in recipe template

 bin/toaster                                        |  9 +++++--
 lib/toaster/bldcontrol/localhostbecontroller.py    | 31 +++++++++++++++++++---
 .../management/commands/checksettings.py           |  3 ++-
 lib/toaster/orm/models.py                          |  7 +++++
 lib/toaster/toastergui/static/js/layerdetails.js   |  3 ++-
 lib/toaster/toastergui/templates/newproject.html   |  4 +--
 lib/toaster/toastergui/templates/recipe.html       |  2 +-
 lib/toaster/toastergui/widgets.py                  | 19 ++++++++-----
 8 files changed, 61 insertions(+), 17 deletions(-)