Skip to contentSkip to conversation

Add support for PROVIDES to Toaster

PATCH New Elliot Smith 1 participant 9 messages Jan 8, 2016

lore
Share & tools
lore
#0 lore raw

cover Add support for PROVIDES to Toaster

Add support of PROVIDES to Toaster backend and frontend.

Please see https://bugzilla.yoctoproject.org/show_bug.cgi?id=6169 for more details.

The following changes since commit 3e0c0fb8eaf9b764c19f2c25fe50c4eccbd4a9bd:

toaster-manual: describe the way to start Toaster (2016-01-08 11:09:50 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib elliot/submit/ed/toaster/provides-6169
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/submit/ed/toaster/provides-6169

Ed Bartosh (8):
taskdata: refactor get_providermap
cooker: add providermap to dep_tree
buildinfohelper: use providermap
toaster: add Provider model
buildinfohelper: add provides info to the db
toaster: show list of provides for the recipe
toaster: show 'satisfied via' text for build deps
toaster: show 'satisfied via' text for reverse deps

 lib/bb/cooker.py                                   |  7 +++++
 lib/bb/runqueue.py                                 |  2 +-
 lib/bb/taskdata.py                                 | 17 +++++-------
 lib/bb/ui/buildinfohelper.py                       | 31 +++++++++++++++-------
 .../orm/migrations/0002_auto_20151223_1528.py      | 27 +++++++++++++++++++
 lib/toaster/orm/models.py                          |  5 ++++
 lib/toaster/toastergui/templates/recipe.html       | 28 +++++++++++++++++--
 7 files changed, 95 insertions(+), 22 deletions(-)
 create mode 100644 bitbake/lib/toaster/orm/migrations/0002_auto_20151223_1528.py

--
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

#1 lore raw

PATCH 1/8 taskdata: refactor get_providermap

From: Ed Bartosh <[email protected]>

Added optional parameter 'prefix' to filter out names that don't start with specified prefix. Changed existing call of get_providermap according to changed API.

Optimized the code: got rid of extra loop and temporary list variable virts.

---
 lib/bb/runqueue.py |  2 +-
 lib/bb/taskdata.py | 17 +++++++----------
 2 files changed, 8 insertions(+), 11 deletions(-)
lib/bb/runqueue.py +1 -1
index ee06f0e..7fd0044 100644
@@ -808,7 +808,7 @@ class RunQueueData:
                     invalidate_task(fn, st, True)
 
         # Create and print to the logs a virtual/xxxx -> PN (fn) table
-        virtmap = taskData.get_providermap()
+        virtmap = taskData.get_providermap(prefix="virtual/")
         virtpnmap = {}
         for v in virtmap:
             virtpnmap[v] = self.dataCache.pkg_fn[virtmap[v]]
lib/bb/taskdata.py +7 -10
index 4d12b33..b083870 100644
@@ -612,17 +612,14 @@ class TaskData:
                 break
         # self.dump_data()
 
-    def get_providermap(self):
-        virts = []
-        virtmap = {}
-
+    def get_providermap(self, prefix=None):
+        provmap = {}
         for name in self.build_names_index:
-            if name.startswith("virtual/"):
-                virts.append(name)
-        for v in virts:
-            if self.have_build_target(v):
-                virtmap[v] = self.fn_index[self.get_provider(v)[0]]
-        return virtmap
+            if prefix and not name.startswith(prefix):
+                continue
+            if self.have_build_target(name):
+                provmap[name] = self.fn_index[self.get_provider(name)[0]]
+        return provmap
 
     def dump_data(self):
         """
Signature · 12 lines
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#2 lore raw

PATCH 2/8 cooker: add providermap to dep_tree

From: Ed Bartosh <[email protected]>

Added providermap information to the result of buildDependTree API.

This will be used by Toaster to map virtual dependencies to recipes.

---
 lib/bb/cooker.py | 7 +++++++
 1 file changed, 7 insertions(+)
lib/bb/cooker.py +7 -0
index edceca0..e957169 100644
@@ -725,8 +725,15 @@ class BBCooker:
         depend_tree["packages"] = {}
         depend_tree["rdepends-pkg"] = {}
         depend_tree["rrecs-pkg"] = {}
+        depend_tree['providermap'] = {}
         depend_tree["layer-priorities"] = self.recipecache.bbfile_config_priorities
 
+        for name, fn in taskdata.get_providermap().iteritems():
+            pn = self.recipecache.pkg_fn[fn]
+            if name != pn:
+                version = "%s:%s-%s" % self.recipecache.pkg_pepvpr[fn]
+                depend_tree['providermap'][name] = (pn, version)
+
         for task in xrange(len(rq.rqdata.runq_fnid)):
             taskname = rq.rqdata.runq_task[task]
             fnid = rq.rqdata.runq_fnid[task]
Signature · 12 lines
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#3 lore raw

PATCH 3/8 buildinfohelper: use providermap

In reply to Elliot Smith

From: Ed Bartosh <[email protected]>

Used providermap in store_dependency_information function to find virtual dependencies. This should fix annoying warnings "stpd: KeyError saving recipe dependency"

[YOCTO #6169]

---
 lib/bb/ui/buildinfohelper.py | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)
lib/bb/ui/buildinfohelper.py +16 -8
index 279c5c7..cba0165 100644
@@ -1186,6 +1186,7 @@ class BuildInfoHelper(object):
         assert 'layer-priorities' in event._depgraph
         assert 'pn' in event._depgraph
         assert 'tdepends' in event._depgraph
+        assert 'providermap' in event._depgraph
 
         errormsg = ""
 
@@ -1263,15 +1264,22 @@ class BuildInfoHelper(object):
         # buildtime
         recipedeps_objects = []
         for recipe in event._depgraph['depends']:
-            try:
-                target = self.internal_state['recipes'][recipe]
-                for dep in event._depgraph['depends'][recipe]:
+           target = self.internal_state['recipes'][recipe]
+           for dep in event._depgraph['depends'][recipe]:
+                if dep in assume_provided:
+                    continue
+                if dep in event._depgraph['providermap']:
+                    dep = event._depgraph['providermap'][dep][0]
+                if dep in self.internal_state['recipes']:
                     dependency = self.internal_state['recipes'][dep]
-                    recipedeps_objects.append(Recipe_Dependency( recipe = target,
-                            depends_on = dependency, dep_type = Recipe_Dependency.TYPE_DEPENDS))
-            except KeyError as e:
-                if e not in assume_provided and not str(e).startswith("virtual/"):
-                    errormsg += "  stpd: KeyError saving recipe dependency for %s, %s \n" % (recipe, e)
+                else:
+                    errormsg += "  stpd: KeyError saving recipe dependency for %s, %s \n" % (recipe, dep)
+                    continue
+                recipe_dep = Recipe_Dependency(recipe=target,
+                                               depends_on=dependency,
+                                               dep_type=Recipe_Dependency.TYPE_DEPENDS)
+                recipedeps_objects.append(recipe_dep)
+
         Recipe_Dependency.objects.bulk_create(recipedeps_objects)
 
         # save all task information
Signature · 12 lines
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#4 lore raw

PATCH 4/8 toaster: add Provider model

In reply to Elliot Smith

From: Ed Bartosh <[email protected]>

Added new model Provider and a foreign key 'via' to link Recipe_Dependency to it.

[YOCTO #6169]

---
 .../orm/migrations/0002_auto_20151223_1528.py      | 27 ++++++++++++++++++++++
 lib/toaster/orm/models.py                          |  5 ++++
 2 files changed, 32 insertions(+)
 create mode 100644 bitbake/lib/toaster/orm/migrations/0002_auto_20151223_1528.py
lib/toaster/orm/migrations/0002_auto_20151223_1528.py new file +27 -0
new file mode 100644
index 0000000..194c897
@@ -0,0 +1,27 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('orm', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='Provides',
+            fields=[
+                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
+                ('name', models.CharField(max_length=100)),
+                ('recipe', models.ForeignKey(to='orm.Recipe')),
+            ],
+        ),
+        migrations.AddField(
+            model_name='recipe_dependency',
+            name='via',
+            field=models.ForeignKey(null=True, default=None, to='orm.Provides'),
+        ),
+    ]
lib/toaster/orm/models.py +5 -0
index 6e87c54..e4ab0bb 100644
@@ -733,6 +733,10 @@ class Recipe_DependencyManager(models.Manager):
     def get_queryset(self):
         return super(Recipe_DependencyManager, self).get_queryset().exclude(recipe_id = F('depends_on__id'))
 
+class Provides(models.Model):
+    name = models.CharField(max_length=100)
+    recipe = models.ForeignKey(Recipe)
+
 class Recipe_Dependency(models.Model):
     TYPE_DEPENDS = 0
     TYPE_RDEPENDS = 1
@@ -743,6 +747,7 @@ class Recipe_Dependency(models.Model):
     )
     recipe = models.ForeignKey(Recipe, related_name='r_dependencies_recipe')
     depends_on = models.ForeignKey(Recipe, related_name='r_dependencies_depends')
+    via = models.ForeignKey(Provides, null=True, default=None)
     dep_type = models.IntegerField(choices=DEPENDS_TYPE)
     objects = Recipe_DependencyManager()
 
Signature · 12 lines
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#5 lore raw

PATCH 5/8 buildinfohelper: add provides info to the db

In reply to Elliot Smith

From: Ed Bartosh <[email protected]>

Added new entries to Provides model and link them to Recipe_Dependency using 'via' field.

This data will be used by Toaster UI to show 'Provides:' information for the recipes.

[YOCTO #6169]

---
 lib/bb/ui/buildinfohelper.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
lib/bb/ui/buildinfohelper.py +8 -3
index cba0165..0cb6f68 100644
@@ -41,7 +41,7 @@ from orm.models import Target_Image_File, BuildArtifact
 from orm.models import Variable, VariableHistory
 from orm.models import Package, Package_File, Target_Installed_Package, Target_File
 from orm.models import Task_Dependency, Package_Dependency
-from orm.models import Recipe_Dependency
+from orm.models import Recipe_Dependency, Provides
 
 from orm.models import Project
 from bldcontrol.models import BuildEnvironment, BuildRequest
@@ -1268,15 +1268,20 @@ class BuildInfoHelper(object):
            for dep in event._depgraph['depends'][recipe]:
                 if dep in assume_provided:
                     continue
+                via = None
                 if dep in event._depgraph['providermap']:
-                    dep = event._depgraph['providermap'][dep][0]
-                if dep in self.internal_state['recipes']:
+                    deprecipe = event._depgraph['providermap'][dep][0]
+                    dependency = self.internal_state['recipes'][deprecipe]
+                    via = Provides.objects.get_or_create(name=dep,
+                                                         recipe=dependency)[0]
+                elif dep in self.internal_state['recipes']:
                     dependency = self.internal_state['recipes'][dep]
                 else:
                     errormsg += "  stpd: KeyError saving recipe dependency for %s, %s \n" % (recipe, dep)
                     continue
                 recipe_dep = Recipe_Dependency(recipe=target,
                                                depends_on=dependency,
+                                               via=via,
                                                dep_type=Recipe_Dependency.TYPE_DEPENDS)
                 recipedeps_objects.append(recipe_dep)
 
Signature · 12 lines
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#6 lore raw

PATCH 6/8 toaster: show list of provides for the recipe

In reply to Elliot Smith

From: Ed Bartosh <[email protected]>

Showed list of names that recipe provides.

---
 lib/toaster/toastergui/templates/recipe.html | 10 ++++++++++
 1 file changed, 10 insertions(+)
lib/toaster/toastergui/templates/recipe.html +10 -0
index c6ae2f3..bbe7475 100644
@@ -71,6 +71,16 @@
                     Layer commit
                 </dt>
                 <dd class="iscommit">{{layer_version.commit}}</dd>
+                {% if object.provides_set.all %}
+                <dt>
+                    <i class="icon-question-sign get-help"
+                     title="A list of aliases by which a particular recipe can be known. The additional aliases are
+                           synonyms for the recipe and can be useful satisfying dependencies of other recipes during
+                           the build"></i>
+                    PROVIDES
+                </dt>
+                 <dd><code>{% for provider in object.provides_set.all %}{{ provider.name }}&nbsp;{% endfor %}</code></dd>
+                {% endif %}
             </dl>
 
             <h2 class="details">Tasks</h2>
Signature · 12 lines
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#7 lore raw

PATCH 7/8 toaster: show 'satisfied via' text for build deps

In reply to Elliot Smith

From: Ed Bartosh <[email protected]>

Showed '<dependency> satisfied via <provider>' text and help tooltip for the build dependencies provided through 'PROVIDES' in the 'Build dependencies' tab.

[YOCTO #6169]

---
 lib/toaster/toastergui/templates/recipe.html | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
lib/toaster/toastergui/templates/recipe.html +8 -1
index bbe7475..d6b464b 100644
@@ -172,7 +172,14 @@
 
                     {% for rr in object.r_dependencies_recipe.all|dictsort:"depends_on.name" %}
                     <tr>
-                        <td><a href="{% url "recipe" build.pk rr.depends_on.pk %}">{{rr.depends_on.name}}</a></td>
+                        <td><a href="{% url "recipe" build.pk rr.depends_on.pk %}">{{rr.depends_on.name}}</a>
+                            {% if rr.via %}
+                                <span class="muted">satisfied via {{rr.via.name}}</span>
+                                <i class="icon-question-sign get-help hover-help"
+                                 title="This dependency is satisfied by the PROVIDES value
+                                       {{rr.via.name}} in the {{rr.depends_on.name}} recipe"></i>
+                            {% endif %}
+                        </td>
                         <td><a href="{% url "recipe" build.pk rr.depends_on.pk %}">{{rr.depends_on.version}}</a></td>
                     </tr>
                     {% endfor %}
Signature · 12 lines
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#8 lore raw

PATCH 8/8 toaster: show 'satisfied via' text for reverse deps

In reply to Elliot Smith

From: Ed Bartosh <[email protected]>

Showed '<dependency> satisfied via <provider>' text and help tooltip for the reverse build dependencies provided through 'PROVIDES' in the 'Reverse build dependencies' tab.

[YOCTO #6169]

---
 lib/toaster/toastergui/templates/recipe.html | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
lib/toaster/toastergui/templates/recipe.html +8 -1
index d6b464b..1d6d64e 100644
@@ -211,7 +211,14 @@
 
                     {% for rr in object.r_dependencies_depends.all|dictsort:"recipe.name" %}
                     <tr>
-                        <td><a href="{% url "recipe" build.pk rr.recipe.pk %}">{{rr.recipe.name}}</a></td>
+                        <td><a href="{% url "recipe" build.pk rr.recipe.pk %}">{{rr.recipe.name}}</a>
+                        {% if rr.via %}
+                            <span class="muted"> satisfied via {{rr.via.name}}</span>
+                            <i class="icon-question-sign get-help hover-help"
+                             title="This dependency is satisfied by the PROVIDES value
+                            {{rr.via.name}} in the {{rr.depends_on.name}} recipe"></i>
+                        {% endif %}
+                        </td>
                         <td><a href="{% url "recipe" build.pk rr.recipe.pk %}">{{rr.recipe.version}}</a></td>
                     </tr>
                     {% endfor %}
Signature · 12 lines
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.