Add support for PROVIDES to Toaster
cover Add support for PROVIDES to Toaster
Add support of PROVIDES to Toaster backend and frontend.
Please see https://bugzilla.yoctoproject.org/show_bug.cgi?id=6169 for more details.
The following changes since commit 3e0c0fb8eaf9b764c19f2c25fe50c4eccbd4a9bd:
toaster-manual: describe the way to start Toaster (2016-01-08 11:09:50 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib elliot/submit/ed/toaster/provides-6169
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/submit/ed/toaster/provides-6169
Ed Bartosh (8):
taskdata: refactor get_providermap
cooker: add providermap to dep_tree
buildinfohelper: use providermap
toaster: add Provider model
buildinfohelper: add provides info to the db
toaster: show list of provides for the recipe
toaster: show 'satisfied via' text for build deps
toaster: show 'satisfied via' text for reverse deps
lib/bb/cooker.py | 7 +++++ lib/bb/runqueue.py | 2 +- lib/bb/taskdata.py | 17 +++++------- lib/bb/ui/buildinfohelper.py | 31 +++++++++++++++------- .../orm/migrations/0002_auto_20151223_1528.py | 27 +++++++++++++++++++ lib/toaster/orm/models.py | 5 ++++ lib/toaster/toastergui/templates/recipe.html | 28 +++++++++++++++++-- 7 files changed, 95 insertions(+), 22 deletions(-) create mode 100644 bitbake/lib/toaster/orm/migrations/0002_auto_20151223_1528.py
--
1.9.3
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
PATCH 1/8 taskdata: refactor get_providermap
From: Ed Bartosh <[email protected]>
Added optional parameter 'prefix' to filter out names that don't start with specified prefix. Changed existing call of get_providermap according to changed API.
Optimized the code: got rid of extra loop and temporary list variable virts.
--- lib/bb/runqueue.py | 2 +- lib/bb/taskdata.py | 17 +++++++---------- 2 files changed, 8 insertions(+), 11 deletions(-)
lib/bb/runqueue.py +1 -1
@@ -808,7 +808,7 @@ class RunQueueData: invalidate_task(fn, st, True) # Create and print to the logs a virtual/xxxx -> PN (fn) table - virtmap = taskData.get_providermap() + virtmap = taskData.get_providermap(prefix="virtual/") virtpnmap = {} for v in virtmap: virtpnmap[v] = self.dataCache.pkg_fn[virtmap[v]]
lib/bb/taskdata.py +7 -10
@@ -612,17 +612,14 @@ class TaskData: break # self.dump_data() - def get_providermap(self): - virts = [] - virtmap = {} - + def get_providermap(self, prefix=None): + provmap = {} for name in self.build_names_index: - if name.startswith("virtual/"): - virts.append(name) - for v in virts: - if self.have_build_target(v): - virtmap[v] = self.fn_index[self.get_provider(v)[0]] - return virtmap + if prefix and not name.startswith(prefix): + continue + if self.have_build_target(name): + provmap[name] = self.fn_index[self.get_provider(name)[0]] + return provmap def dump_data(self): """
Signature · 12 lines
1.9.3 --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
PATCH 2/8 cooker: add providermap to dep_tree
From: Ed Bartosh <[email protected]>
Added providermap information to the result of buildDependTree API.
This will be used by Toaster to map virtual dependencies to recipes.
--- lib/bb/cooker.py | 7 +++++++ 1 file changed, 7 insertions(+)
lib/bb/cooker.py +7 -0
@@ -725,8 +725,15 @@ class BBCooker: depend_tree["packages"] = {} depend_tree["rdepends-pkg"] = {} depend_tree["rrecs-pkg"] = {} + depend_tree['providermap'] = {} depend_tree["layer-priorities"] = self.recipecache.bbfile_config_priorities + for name, fn in taskdata.get_providermap().iteritems(): + pn = self.recipecache.pkg_fn[fn] + if name != pn: + version = "%s:%s-%s" % self.recipecache.pkg_pepvpr[fn] + depend_tree['providermap'][name] = (pn, version) + for task in xrange(len(rq.rqdata.runq_fnid)): taskname = rq.rqdata.runq_task[task] fnid = rq.rqdata.runq_fnid[task]
Signature · 12 lines
1.9.3 --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
PATCH 3/8 buildinfohelper: use providermap
In reply to Elliot SmithFrom: Ed Bartosh <[email protected]>
Used providermap in store_dependency_information function to find virtual dependencies. This should fix annoying warnings "stpd: KeyError saving recipe dependency"
[YOCTO #6169]
--- lib/bb/ui/buildinfohelper.py | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)
lib/bb/ui/buildinfohelper.py +16 -8
@@ -1186,6 +1186,7 @@ class BuildInfoHelper(object): assert 'layer-priorities' in event._depgraph assert 'pn' in event._depgraph assert 'tdepends' in event._depgraph + assert 'providermap' in event._depgraph errormsg = ""
@@ -1263,15 +1264,22 @@ class BuildInfoHelper(object): # buildtime recipedeps_objects = [] for recipe in event._depgraph['depends']: - try: - target = self.internal_state['recipes'][recipe] - for dep in event._depgraph['depends'][recipe]: + target = self.internal_state['recipes'][recipe] + for dep in event._depgraph['depends'][recipe]: + if dep in assume_provided: + continue + if dep in event._depgraph['providermap']: + dep = event._depgraph['providermap'][dep][0] + if dep in self.internal_state['recipes']: dependency = self.internal_state['recipes'][dep] - recipedeps_objects.append(Recipe_Dependency( recipe = target, - depends_on = dependency, dep_type = Recipe_Dependency.TYPE_DEPENDS)) - except KeyError as e: - if e not in assume_provided and not str(e).startswith("virtual/"): - errormsg += " stpd: KeyError saving recipe dependency for %s, %s \n" % (recipe, e) + else: + errormsg += " stpd: KeyError saving recipe dependency for %s, %s \n" % (recipe, dep) + continue + recipe_dep = Recipe_Dependency(recipe=target, + depends_on=dependency, + dep_type=Recipe_Dependency.TYPE_DEPENDS) + recipedeps_objects.append(recipe_dep) + Recipe_Dependency.objects.bulk_create(recipedeps_objects) # save all task information
Signature · 12 lines
1.9.3 --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
PATCH 4/8 toaster: add Provider model
In reply to Elliot SmithFrom: Ed Bartosh <[email protected]>
Added new model Provider and a foreign key 'via' to link Recipe_Dependency to it.
[YOCTO #6169]
--- .../orm/migrations/0002_auto_20151223_1528.py | 27 ++++++++++++++++++++++ lib/toaster/orm/models.py | 5 ++++ 2 files changed, 32 insertions(+) create mode 100644 bitbake/lib/toaster/orm/migrations/0002_auto_20151223_1528.py
lib/toaster/orm/migrations/0002_auto_20151223_1528.py new file +27 -0
@@ -0,0 +1,27 @@ +# -*- coding: utf-8 -*- +from __future__ import unicode_literals + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('orm', '0001_initial'), + ] + + operations = [ + migrations.CreateModel( + name='Provides', + fields=[ + ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)), + ('name', models.CharField(max_length=100)), + ('recipe', models.ForeignKey(to='orm.Recipe')), + ], + ), + migrations.AddField( + model_name='recipe_dependency', + name='via', + field=models.ForeignKey(null=True, default=None, to='orm.Provides'), + ), + ]
lib/toaster/orm/models.py +5 -0
@@ -733,6 +733,10 @@ class Recipe_DependencyManager(models.Manager): def get_queryset(self): return super(Recipe_DependencyManager, self).get_queryset().exclude(recipe_id = F('depends_on__id')) +class Provides(models.Model): + name = models.CharField(max_length=100) + recipe = models.ForeignKey(Recipe) + class Recipe_Dependency(models.Model): TYPE_DEPENDS = 0 TYPE_RDEPENDS = 1
@@ -743,6 +747,7 @@ class Recipe_Dependency(models.Model): ) recipe = models.ForeignKey(Recipe, related_name='r_dependencies_recipe') depends_on = models.ForeignKey(Recipe, related_name='r_dependencies_depends') + via = models.ForeignKey(Provides, null=True, default=None) dep_type = models.IntegerField(choices=DEPENDS_TYPE) objects = Recipe_DependencyManager()
Signature · 12 lines
1.9.3 --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
PATCH 5/8 buildinfohelper: add provides info to the db
In reply to Elliot SmithFrom: Ed Bartosh <[email protected]>
Added new entries to Provides model and link them to Recipe_Dependency using 'via' field.
This data will be used by Toaster UI to show 'Provides:' information for the recipes.
[YOCTO #6169]
--- lib/bb/ui/buildinfohelper.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
lib/bb/ui/buildinfohelper.py +8 -3
@@ -41,7 +41,7 @@ from orm.models import Target_Image_File, BuildArtifact from orm.models import Variable, VariableHistory from orm.models import Package, Package_File, Target_Installed_Package, Target_File from orm.models import Task_Dependency, Package_Dependency -from orm.models import Recipe_Dependency +from orm.models import Recipe_Dependency, Provides from orm.models import Project from bldcontrol.models import BuildEnvironment, BuildRequest
@@ -1268,15 +1268,20 @@ class BuildInfoHelper(object): for dep in event._depgraph['depends'][recipe]: if dep in assume_provided: continue + via = None if dep in event._depgraph['providermap']: - dep = event._depgraph['providermap'][dep][0] - if dep in self.internal_state['recipes']: + deprecipe = event._depgraph['providermap'][dep][0] + dependency = self.internal_state['recipes'][deprecipe] + via = Provides.objects.get_or_create(name=dep, + recipe=dependency)[0] + elif dep in self.internal_state['recipes']: dependency = self.internal_state['recipes'][dep] else: errormsg += " stpd: KeyError saving recipe dependency for %s, %s \n" % (recipe, dep) continue recipe_dep = Recipe_Dependency(recipe=target, depends_on=dependency, + via=via, dep_type=Recipe_Dependency.TYPE_DEPENDS) recipedeps_objects.append(recipe_dep)
Signature · 12 lines
1.9.3 --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
PATCH 6/8 toaster: show list of provides for the recipe
In reply to Elliot SmithFrom: Ed Bartosh <[email protected]>
Showed list of names that recipe provides.
--- lib/toaster/toastergui/templates/recipe.html | 10 ++++++++++ 1 file changed, 10 insertions(+)
lib/toaster/toastergui/templates/recipe.html +10 -0
@@ -71,6 +71,16 @@ Layer commit </dt> <dd class="iscommit">{{layer_version.commit}}</dd> + {% if object.provides_set.all %} + <dt> + <i class="icon-question-sign get-help" + title="A list of aliases by which a particular recipe can be known. The additional aliases are + synonyms for the recipe and can be useful satisfying dependencies of other recipes during + the build"></i> + PROVIDES + </dt> + <dd><code>{% for provider in object.provides_set.all %}{{ provider.name }} {% endfor %}</code></dd> + {% endif %} </dl> <h2 class="details">Tasks</h2>
Signature · 12 lines
1.9.3 --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
PATCH 7/8 toaster: show 'satisfied via' text for build deps
In reply to Elliot SmithFrom: Ed Bartosh <[email protected]>
Showed '<dependency> satisfied via <provider>' text and help tooltip for the build dependencies provided through 'PROVIDES' in the 'Build dependencies' tab.
[YOCTO #6169]
--- lib/toaster/toastergui/templates/recipe.html | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
lib/toaster/toastergui/templates/recipe.html +8 -1
@@ -172,7 +172,14 @@ {% for rr in object.r_dependencies_recipe.all|dictsort:"depends_on.name" %} <tr> - <td><a href="{% url "recipe" build.pk rr.depends_on.pk %}">{{rr.depends_on.name}}</a></td> + <td><a href="{% url "recipe" build.pk rr.depends_on.pk %}">{{rr.depends_on.name}}</a> + {% if rr.via %} + <span class="muted">satisfied via {{rr.via.name}}</span> + <i class="icon-question-sign get-help hover-help" + title="This dependency is satisfied by the PROVIDES value + {{rr.via.name}} in the {{rr.depends_on.name}} recipe"></i> + {% endif %} + </td> <td><a href="{% url "recipe" build.pk rr.depends_on.pk %}">{{rr.depends_on.version}}</a></td> </tr> {% endfor %}
Signature · 12 lines
1.9.3 --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
PATCH 8/8 toaster: show 'satisfied via' text for reverse deps
In reply to Elliot SmithFrom: Ed Bartosh <[email protected]>
Showed '<dependency> satisfied via <provider>' text and help tooltip for the reverse build dependencies provided through 'PROVIDES' in the 'Reverse build dependencies' tab.
[YOCTO #6169]
--- lib/toaster/toastergui/templates/recipe.html | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
lib/toaster/toastergui/templates/recipe.html +8 -1
@@ -211,7 +211,14 @@ {% for rr in object.r_dependencies_depends.all|dictsort:"recipe.name" %} <tr> - <td><a href="{% url "recipe" build.pk rr.recipe.pk %}">{{rr.recipe.name}}</a></td> + <td><a href="{% url "recipe" build.pk rr.recipe.pk %}">{{rr.recipe.name}}</a> + {% if rr.via %} + <span class="muted"> satisfied via {{rr.via.name}}</span> + <i class="icon-question-sign get-help hover-help" + title="This dependency is satisfied by the PROVIDES value + {{rr.via.name}} in the {{rr.depends_on.name}} recipe"></i> + {% endif %} + </td> <td><a href="{% url "recipe" build.pk rr.recipe.pk %}">{{rr.recipe.version}}</a></td> </tr> {% endfor %}
Signature · 12 lines
1.9.3 --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.