Add Distro, dependency and recommends detection, and import_project
v2 cover Add Distro, dependency and recommends detection, and import_project
This set of patches adds a number of features to the layerindex code:
- Adds the ability to set the actual-branch to import_layer. - Adds distro to the database and web interface. - Adds collection & version information to layerbranch. - Uses layer name and collection name to determine dependencies and
recommends.
- Adds import_project to simplify pulling all layers and openembedded-core into a layer index.
Liam R. Howlett (11):
import_layer: Add --actual-branch option layerindex/tools/import_layer.py: Sanitize layer name. layerindex/tools/import_layer.py: Avoid failing if there is any layer
to add.
layerindex/utils: Update runcmd to decode binary strings to strings.
layerindex: Add distro to web interface and model. layerindex/tools/import_project: Add import_project layerindex/recipeparse.py: refactor setup_tinfoil,
checkout_layer_branch, parse_layer_conf to utils.py
layerindex: Detect dependencies from layer.conf files
layerindex: Add collection and version to layerbranch
layerindexer: Add layer recommends support
recipeparse: remove unnecessary else statement.
Mark Hatle (1):
layerindex/update_layer.py: Preserve the recipedependency files
layerindex/admin.py | 11 +++ layerindex/bulkchange.py | 4 +- layerindex/layerconfparse.py | 50 +++++++++++ layerindex/models.py | 23 +++++ layerindex/recipeparse.py | 53 +++-------- layerindex/restviews.py | 10 ++- layerindex/tools/import_classic.py | 2 + layerindex/tools/import_layer.py | 47 +++++++++- layerindex/tools/import_project.py | 180 +++++++++++++++++++++++++++++++++++++ layerindex/update.py | 42 ++++++++- layerindex/update_layer.py | 87 ++++++++++++++++-- layerindex/urls.py | 3 + layerindex/urls_branch.py | 6 +- layerindex/utils.py | 159 +++++++++++++++++++++++++++++++- layerindex/views.py | 29 +++++- templates/layerindex/detail.html | 55 +++++++++--- templates/layerindex/distros.html | 76 ++++++++++++++++ templates/layerindex/layers.html | 1 + templates/layerindex/machines.html | 1 + templates/layerindex/recipes.html | 1 + 20 files changed, 767 insertions(+), 73 deletions(-) create mode 100644 layerindex/layerconfparse.py create mode 100755 layerindex/tools/import_project.py create mode 100644 templates/layerindex/distros.html
Signature · 1 line
1.9.1
PATCH v2 1/12 import_layer: Add --actual-branch option
Allow users to set actual-branch from the command line import of layers.
--- layerindex/tools/import_layer.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
layerindex/tools/import_layer.py +8 -2
@@ -200,6 +200,9 @@ def main(): parser.add_option("-q", "--quiet", help = "Hide all output except error messages", action="store_const", const=logging.ERROR, dest="loglevel") + parser.add_option("-a", "--actual-branch", + help = "Set actual branch", + action="store", dest="actual_branch") options, args = parser.parse_args(sys.argv)
@@ -273,10 +276,13 @@ def main(): logger.error("Fetch failed: %s" % str(e)) sys.exit(1) - actual_branch = '' + actual_branch = 'master' + if (options.actual_branch): + actual_branch = options.actual_branch try: - out = utils.runcmd("git checkout origin/master", repodir, logger=logger) + out = utils.runcmd("git checkout origin/%s" % actual_branch, repodir, logger=logger) except subprocess.CalledProcessError: + actual_branch = None branches = utils.runcmd("git branch -r", repodir, logger=logger) for line in branches.splitlines(): if 'origin/HEAD ->' in line:
Signature · 1 line
1.9.1
PATCH v2 2/12 layerindex/tools/import_layer.py: Sanitize layer name.
In reply to Liam R. HowlettDjango will produce a cryptic error message if layers are added with invalid names. Sanitize the layer names when trying to add them.
--- layerindex/tools/import_layer.py | 6 ++++++ 1 file changed, 6 insertions(+)
layerindex/tools/import_layer.py +6 -0
@@ -181,6 +181,8 @@ def get_github_layerinfo(layer_url, username = None, password = None): def main(): + valid_layer_name = re.compile('[-\w]+$') + parser = optparse.OptionParser( usage = """ %prog [options] <url> [name]""")
@@ -222,6 +224,10 @@ def main(): if layer_name.endswith('.git'): layer_name = layer_name[:-4] + if not valid_layer_name.match(layer_name): + logger.error('Invalid layer name "%s" - Layer name can only include letters, numbers and dashes.', layer_name) + sys.exit(1) + if options.github_auth: if not ':' in options.github_auth: logger.error('--github-auth value must be specified as username:password')
Signature · 1 line
1.9.1
PATCH v2 3/12 layerindex/tools/import_layer.py: Avoid failing if there is any layer to add.
In reply to Liam R. HowlettSubdirectories are scanned when adding layers. If any of the subdirectories or root directory layers already exist in the database, then the addition fails. This changes that behaviour to report the failure as a warning and remove it from the list. That way, if a repo has a new layer added it can be rescanned without issue. Layers being rescanned are checked against the vcs_url to ensure there is not a name collision. A name collision without the same vcs_url will still produce a hard failure.
Note that multiple layers with the same vcs_url are supported in the error reporting even though this should never happen.
--- layerindex/tools/import_layer.py | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
layerindex/tools/import_layer.py +18 -2
@@ -334,8 +334,20 @@ def main(): else: subdir = '' if LayerItem.objects.filter(name=layer.name).exists(): - logger.error('A layer named "%s" already exists in the database' % layer_name) - sys.exit(1) + if LayerItem.objects.filter(name=layer.name).exclude(vcs_url=layer.vcs_url).exists(): + conflict_list = LayerItem.objects.filter(name=layer.name).exclude(vcs_url=layer.vcs_url) + conflict_list_urls = [] + for conflict in conflict_list: + conflict_list_urls.append(conflict.vcs_url) + cln = ', '.join(conflict_list_urls) + logger.error('A layer named "%s" already exists in the database. Possible name collision with %s.vcs_url = %s' % (layer.name, layer.name, cln)) + sys.exit(1) + else: + logger.info('The layer named "%s" already exists in the database. Skipping this layer with same vcs_url' % layer.name) + layer_paths = [x for x in layer_paths if x != layerdir] + continue + + logger.info('Creating layer %s' % layer.name) # Guess layer type
@@ -411,6 +423,10 @@ def main(): layer.save() + if not layer_paths: + logger.error('No layers added.') + sys.exit(1); + if options.dryrun: raise DryRunRollbackException() except DryRunRollbackException:
Signature · 1 line
1.9.1
PATCH v2 4/12 layerindex/utils: Update runcmd to decode binary strings to strings.
In reply to Liam R. HowlettConvert binary strings to strings and strip leading/trailing whitespace prior to returning errors and output.
--- layerindex/utils.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
layerindex/utils.py +6 -2
@@ -32,7 +32,8 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None): execute command, raise CalledProcessError if fail return output if succeed """ - #logger.debug("run cmd '%s' in %s" % (cmd, os.getcwd() if destdir is None else destdir)) + if logger: + logger.debug("run cmd '%s' in %s" % (cmd, os.getcwd() if destdir is None else destdir)) out = tempfile.TemporaryFile() try: subprocess.check_call(cmd, stdout=out, stderr=out, cwd=destdir, shell=True)
@@ -40,6 +41,7 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None): out.seek(0) if printerr: output = out.read() + output = output.decode('ascii').strip() if logger: logger.error("%s" % output) else:
@@ -49,7 +51,9 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None): out.seek(0) output = out.read() - #logger.debug("output: %s" % output.rstrip() ) + output = output.decode('ascii').strip() + if logger: + logger.debug("output: %s" % output.rstrip() ) return output def setup_django():
Signature · 1 line
1.9.1
PATCH v2 5/12 layerindex: Add distro to web interface and model.
In reply to Liam R. HowlettAdd the distros to the index. This looks a lot like the machines and allows users to search for a particular distro.
--- layerindex/admin.py | 11 ++++++ layerindex/models.py | 14 +++++++ layerindex/recipeparse.py | 5 +++ layerindex/restviews.py | 10 ++++- layerindex/tools/import_classic.py | 2 + layerindex/update_layer.py | 46 ++++++++++++++++++++++- layerindex/urls.py | 3 ++ layerindex/urls_branch.py | 6 ++- layerindex/views.py | 29 ++++++++++++++- templates/layerindex/detail.html | 23 ++++++++++++ templates/layerindex/distros.html | 76 ++++++++++++++++++++++++++++++++++++++ templates/layerindex/layers.html | 1 + templates/layerindex/machines.html | 1 + templates/layerindex/recipes.html | 1 + 14 files changed, 224 insertions(+), 4 deletions(-) create mode 100644 templates/layerindex/distros.html
layerindex/admin.py +11 -0
@@ -75,6 +75,16 @@ class MachineAdmin(admin.ModelAdmin): def has_delete_permission(self, request, obj=None): return False +class DistroAdmin(admin.ModelAdmin): + search_fields = ['name'] + list_filter = ['layerbranch__layer__name', 'layerbranch__branch__name'] + readonly_fields = Distro._meta.get_all_field_names() + def has_add_permission(self, request, obj=None): + return False + def has_delete_permission(self, request, obj=None): + return False + + class BBAppendAdmin(admin.ModelAdmin): search_fields = ['filename'] list_filter = ['layerbranch__layer__name', 'layerbranch__branch__name']
@@ -111,6 +121,7 @@ admin.site.register(LayerNote, LayerNoteAdmin) admin.site.register(Recipe, RecipeAdmin) admin.site.register(RecipeFileDependency) admin.site.register(Machine, MachineAdmin) +admin.site.register(Distro, DistroAdmin) admin.site.register(BBAppend, BBAppendAdmin) admin.site.register(BBClass, BBClassAdmin) admin.site.register(RecipeChangeset, RecipeChangesetAdmin)
layerindex/models.py +14 -0
@@ -361,6 +361,20 @@ class Machine(models.Model): def __str__(self): return '%s (%s)' % (self.name, self.layerbranch.layer.name) +class Distro(models.Model): + layerbranch = models.ForeignKey(LayerBranch) + name = models.CharField(max_length=255) + description = models.CharField(max_length=255) + + updated = models.DateTimeField(auto_now=True) + + def vcs_web_url(self): + url = self.layerbranch.file_url(os.path.join('conf/distro/%s.conf' % self.name)) + return url or '' + + def __str__(self): + return '%s (%s)' % (self.name, self.layerbranch.layer.name) + class BBAppend(models.Model): layerbranch = models.ForeignKey(LayerBranch)
layerindex/recipeparse.py +5 -0
@@ -152,6 +152,7 @@ def get_var_files(fn, varlist, d): return varfiles machine_conf_re = re.compile(r'conf/machine/([^/.]*).conf$') +distro_conf_re = re.compile(r'conf/distro/([^/.]*).conf$') bbclass_re = re.compile(r'classes/([^/.]*).bbclass$') def detect_file_type(path, subdir_start): typename = None
@@ -171,6 +172,10 @@ def detect_file_type(path, subdir_start): if res: typename = 'bbclass' return (typename, None, res.group(1)) + res = distro_conf_re.match(subpath) + if res: + typename = 'distro' + return (typename, None, res.group(1)) if typename == 'recipe' or typename == 'bbappend': if subdir_start:
layerindex/restviews.py +9 -1
@@ -1,4 +1,4 @@ -from layerindex.models import Branch, LayerItem, LayerNote, LayerBranch, LayerDependency, Recipe, Machine +from layerindex.models import Branch, LayerItem, LayerNote, LayerBranch, LayerDependency, Recipe, Machine, Distro from rest_framework import viewsets, serializers from layerindex.querysethelper import params_to_queryset, get_search_tuple
@@ -56,3 +56,11 @@ class MachineSerializer(serializers.ModelSerializer): class MachineViewSet(ParametricSearchableModelViewSet): queryset = Machine.objects.all() serializer_class = MachineSerializer + +class DistroSerializer(serializers.ModelSerializer): + class Meta: + model = Distro + +class DistroViewSet(ParametricSearchableModelViewSet): + queryset = Distro.objects.all() + serializer_class = DistroSerializer
layerindex/tools/import_classic.py +2 -0
@@ -160,6 +160,7 @@ def main(): layerdir_start = os.path.normpath(oeclassicpath) + os.sep layerrecipes = Recipe.objects.filter(layerbranch=layerbranch) layermachines = Machine.objects.filter(layerbranch=layerbranch) + layerdistros = Distro.objects.filter(layerbranch=layerbranch) layerappends = BBAppend.objects.filter(layerbranch=layerbranch) layerclasses = BBClass.objects.filter(layerbranch=layerbranch)
@@ -172,6 +173,7 @@ def main(): layerrecipes.delete() layermachines.delete() + layerdistros.delete() layerappends.delete() layerclasses.delete() for root, dirs, files in os.walk(oeclassicpath):
layerindex/update_layer.py +45 -1
100 lines not on this page · raw message
layerindex/urls.py +3 -0
@@ -21,6 +21,7 @@ router.register(r'layerBranches', restviews.LayerBranchViewSet) router.register(r'layerDependencies', restviews.LayerDependencyViewSet) router.register(r'recipes', restviews.RecipeViewSet) router.register(r'machines', restviews.MachineViewSet) +router.register(r'distros', restviews.DistroViewSet) urlpatterns = patterns('', url(r'^$',
@@ -37,6 +38,8 @@ urlpatterns = patterns('', RedirectView.as_view(url=reverse_lazy('recipe_search', args=('master',)), permanent=False)), url(r'^machines/$', RedirectView.as_view(url=reverse_lazy('machine_search', args=('master',)), permanent=False)), + url(r'^distros/$', + RedirectView.as_view(url=reverse_lazy('distro_search', args=('master',)), permanent=False)), url(r'^submit/$', edit_layer_view, {'template_name': 'layerindex/submitlayer.html'}, name="submit_layer"), url(r'^submit/thanks$',
layerindex/urls_branch.py +5 -1
@@ -7,7 +7,7 @@ from django.conf.urls import * from django.views.defaults import page_not_found from django.core.urlresolvers import reverse_lazy -from layerindex.views import LayerListView, RecipeSearchView, MachineSearchView, PlainTextListView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, RedirectParamsView, DuplicatesView +from layerindex.views import LayerListView, RecipeSearchView, MachineSearchView, DistroSearchView, PlainTextListView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, RedirectParamsView, DuplicatesView urlpatterns = patterns('', url(r'^$',
@@ -28,6 +28,10 @@ urlpatterns = patterns('', MachineSearchView.as_view( template_name='layerindex/machines.html'), name='machine_search'), + url(r'^distros/$', + DistroSearchView.as_view( + template_name='layerindex/distros.html'), + name='distro_search'), url(r'^edit/(?P<slug>[-\w]+)/$', edit_layer_view, {'template_name': 'layerindex/editlayer.html'}, name="edit_layer"), url(r'^duplicates/$', DuplicatesView.as_view(
layerindex/views.py +28 -1
@@ -10,7 +10,7 @@ from django.http import HttpResponse, HttpResponseRedirect, HttpResponseForbidde from django.core.urlresolvers import reverse, reverse_lazy, resolve from django.core.exceptions import PermissionDenied from django.template import RequestContext -from layerindex.models import Branch, LayerItem, LayerMaintainer, LayerBranch, LayerDependency, LayerNote, Recipe, Machine, BBClass, BBAppend, RecipeChange, RecipeChangeset, ClassicRecipe +from layerindex.models import Branch, LayerItem, LayerMaintainer, LayerBranch, LayerDependency, LayerNote, Recipe, Machine, Distro, BBClass, BBAppend, RecipeChange, RecipeChangeset, ClassicRecipe from datetime import datetime from itertools import chain from django.views.generic import TemplateView, DetailView, ListView
@@ -326,6 +326,7 @@ class LayerDetailView(DetailView): if layerbranch: context['layerbranch'] = layerbranch context['machines'] = layerbranch.machine_set.order_by('name') + context['distros'] = layerbranch.distro_set.order_by('name') context['appends'] = layerbranch.bbappend_set.order_by('filename') context['classes'] = layerbranch.bbclass_set.order_by('name') context['url_branch'] = self.kwargs['branch']
@@ -596,6 +597,32 @@ class MachineSearchView(ListView): return context +class DistroSearchView(ListView): + context_object_name = 'distro_list' + paginate_by = 50 + + def get_queryset(self): + _check_url_branch(self.kwargs) + query_string = self.request.GET.get('q', '') + init_qs = Distro.objects.filter(layerbranch__branch__name=self.kwargs['branch']) + if query_string.strip(): + entry_query = simplesearch.get_query(query_string, ['name', 'description']) + return init_qs.filter(entry_query).order_by('name', 'layerbranch__layer') + + if 'q' in self.request.GET: + return init_qs.order_by('name', 'layerbranch__layer') + + # Be consistent with RecipeSearchView + return Distro.objects.none() + + def get_context_data(self, **kwargs): + context = super(DistroSearchView, self).get_context_data(**kwargs) + context['search_keyword'] = self.request.GET.get('q', '') + context['url_branch'] = self.kwargs['branch'] + context['this_url_name'] = resolve(self.request.path_info).url_name + return context + + class PlainTextListView(ListView): def render_to_response(self, context): "Returns a plain text response rendering of the template"
templates/layerindex/detail.html +23 -0
@@ -168,6 +168,9 @@ {% if classes.count > 0 %} <li><a href="#classes" data-toggle="tab">Classes</a></li> {% endif %} + {% if distros.count > 0 %} + <li><a href="#distros" data-toggle="tab">Distros</a></li> + {% endif %} </ul> <div class="tab-content">
@@ -265,6 +268,26 @@ </table> </div> {% endif %} + {% if distros.count > 0 %} + <div class="tab-pane" id="distros"> + <div class="navbar"> + <div class="navbar-inner"> + <a class="brand pull-left">{{ layeritem.name }} distros</a> + </div> + </div> + + <table class="table table-bordered"> + <tbody> + {% for distro in distros %} + <tr> + <td><a href="{{ distro.vcs_web_url }}">{{ distro.name }}</a></td> + <td>{{ distro.description }}</td> + </tr> + {% endfor %} + </tbody> + </table> + </div> + {% endif %} </div>
templates/layerindex/distros.html new file +76 -0
76 lines not on this page · raw message
templates/layerindex/layers.html +1 -0
@@ -21,6 +21,7 @@ <li class="active"><a href="{% url 'layer_list' url_branch %}">Layers</a></li> <li><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li> <li><a href="{% url 'machine_search' url_branch %}">Machines</a></li> + <li><a href="{% url 'distro_search' url_branch %}">Distros</a></li> {% endautoescape %} {% endblock %}
templates/layerindex/machines.html +1 -0
@@ -20,6 +20,7 @@ <li><a href="{% url 'layer_list' url_branch %}">Layers</a></li> <li><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li> <li class="active"><a href="{% url 'machine_search' url_branch %}">Machines</a></li> + <li><a href="{% url 'distro_search' url_branch %}">Distros</a></li> {% endautoescape %} {% endblock %}
templates/layerindex/recipes.html +1 -0
@@ -20,6 +20,7 @@ <li><a href="{% url 'layer_list' url_branch %}">Layers</a></li> <li class="active"><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li> <li><a href="{% url 'machine_search' url_branch %}">Machines</a></li> + <li><a href="{% url 'distro_search' url_branch %}">Distros</a></li> {% endautoescape %} {% endblock %}
Signature · 1 line
1.9.1
Re: [layerindex-web][PATCH v2 05/12] layerindex: Add distro to web interface and model.
This looks good, just a few notes about this bit:
layerindex/update_layer.py def update_machine_conf_file(path, machine): · line 127
Quoted 10 lines · On Fri, 07 Oct 2016 11:57:14 Liam R. Howlett wrote:
layerindex/update_layer.py +10
-0+def update_distro_conf_file(path, distro): + logger.debug('Updating distro %s' % path) + desc = "" + with open(path, 'r') as f: + for line in f: + if line.startswith('#@NAME:'): + desc = line[7:].strip() + if line.startswith('#@DESCRIPTION:'): + desc = line[14:].strip() + desc = re.sub(r'Distribtuion configuration for( running)*( an)*( the)*', '', desc)
Aside from the typo, that last line isn't needed for distros.
Also, unlike machines, the readable distro name actually gets set in a proper variable (DISTRO_NAME) and though some distro conf files do still use the meta-comments, a lot don't use them at all. Technically being a variable, DISTRO_NAME could be set in an inc file rather than the distro conf file, and often it is, so we should really parse to get it rather than scraping it.
(I'm happy if we sort this out as a later improvement, I just wanted to make a note of it now.)
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Re: [layerindex-web][PATCH v2 05/12] layerindex: Add distro to web interface and model.
Quoted 16 lines · Paul Eggleton
This looks good, just a few notes about this bit:
layerindex/update_layer.py def update_machine_conf_file(path, machine): · line 127
Quoted 10 lines · On Fri, 07 Oct 2016 11:57:14 Liam R. Howlett wrote:
layerindex/update_layer.py +10
-0+def update_distro_conf_file(path, distro): + logger.debug('Updating distro %s' % path) + desc = "" + with open(path, 'r') as f: + for line in f: + if line.startswith('#@NAME:'): + desc = line[7:].strip() + if line.startswith('#@DESCRIPTION:'): + desc = line[14:].strip() + desc = re.sub(r'Distribtuion configuration for( running)*( an)*( the)*', '', desc)Aside from the typo, that last line isn't needed for distros.
Quoted 8 lines
Also, unlike machines, the readable distro name actually gets set in a proper variable (
DISTRO_NAME) and though some distro conf files do still use the meta-comments, a lot don't use them at all. Technically being a variable,DISTRO_NAMEcould be set in an inc file rather than the distro conf file, and often it is, so we should really parse to get it rather than scraping it.(I'm happy if we sort this out as a later improvement, I just wanted to make a note of it now.)
I have fixed the typo in the current mhatle/django18 layer.
Quoted 3 lines
Cheers,
Paul
PATCH v2 6/12 layerindex/tools/import_project: Add import_project
In reply to Liam R. Howlettimport_project will scan through a project and find any layer and add it to the database by calling import_layer on each layer. This differs from import_layer as it tires to figure out the remote url and uses the subdirectory (if one exists) as the name.
--- layerindex/tools/import_project.py | 180 +++++++++++++++++++++++++++++++++++++ 1 file changed, 180 insertions(+) create mode 100755 layerindex/tools/import_project.py
layerindex/tools/import_project.py new file +180 -0
@@ -0,0 +1,180 @@ +#!/usr/bin/python3 + +# Import a project into the database. +# This will scan through the directories in a project and find any layer and +# call import_layer. +# +# +# Copyright (C) 2016 Wind River Systems +# Author: Liam R. Howlett <[email protected]> +# +# Licensed under the MIT license, see COPYING.MIT for details + +from git import Repo +from urllib.parse import urlparse +import logging +import optparse +import os, fnmatch +import sys + +sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__), '..'))) + +import import_layer +import update + +import utils + + + + +class ImportProject: + logger = utils.logger_create('ProjectIndexImport') + + def find_layers(self, path): + self.logger.debug("finding layer.."); + result = [] + for root, dirs, files in os.walk(path, followlinks=True): + for name in fnmatch.filter(files, 'layer.conf'): + if not root.endswith('conf'): + continue + + self.logger.debug("Found %s" % root) + result.append(root) + return result + + + def main(self): + parser = optparse.OptionParser( + usage = """ + %prog [options] [directory]""") + + parser.add_option("-d", "--debug", + help = "Enable debug output", + action="store_const", const=logging.DEBUG, dest="loglevel", default=logging.INFO) + parser.add_option("-n", "--dry-run", + help = "Don't write any data back to the database", + action="store_true", dest="dryrun") + + self.options, args = parser.parse_args(sys.argv) + + self.logger.setLevel(self.options.loglevel) + + if len(args) == 1: + print("Please provide a directory."); + sys.exit(1) + + install_dir = args[1] + lc_list = self.find_layers(install_dir) + core_layer = self.add_core(lc_list) + if core_layer: + lc_list.remove(core_layer) + + + + for layer in lc_list: + self.add_layer(layer) + + def add_layer(self, layer): + self.logger.debug("Processing layer %s" % layer); + try: + git_dir = utils.runcmd("git rev-parse --show-toplevel", destdir=layer, logger=self.logger) + except Exception as e: + self.logger.error("Cannot get root dir for layer %s: %s - Skipping." % (layer, str(e))) + return 1 + + repo = Repo(git_dir) + actual_branch = repo.active_branch.name + + + layer_name = layer.split('/')[-2] + + layer_subdir = None + if os.path.basename(git_dir) != layer_name: + layer_subdir = layer_name + + layer_name = self.get_layer_name(layer) + + for i in [1, 2, 3]: + try: + git_url = utils.runcmd("git config --get remote.origin.url", destdir=git_dir, logger=self.logger) + except Exception as e: + self.logger.info("Cannot get remote.origin.url for git dir %s: %s" % (git_dir, str(e))) + + if not os.path.exists(git_url): + # Assume this is remote. + self.logger.debug("Found git url = %s" % git_url) + break; + self.logger.debug("Iterating to find git url into %s" % git_dir) + git_dir = git_url + + cmd = ['import_layer.py'] + if self.options.loglevel == logging.DEBUG: + cmd.append("-d") + if layer_subdir: + cmd.append("-s") + cmd.append(layer_subdir) + + if actual_branch: + cmd.append("-a") + cmd.append(actual_branch) + cmd.append(git_url) + cmd.append(layer_name) + prefix = "Calling" + + if self.options.dryrun: + prefix = "Would Call" + + + self.logger.info("%s import_layer.main with %s for dir %s" % (prefix, str(cmd), layer)) + sys.argv = cmd + if not self.options.dryrun: + try: + import_layer.main(); + except SystemExit as see: + return see.code + return 0 + + def get_layer_name(self, layerconfdir): + layer_name = layerconfdir.split('/')[-2] + self.logger.debug('getting layer %s' % layerconfdir) + layer_conf = os.path.join(layerconfdir, 'layer.conf') + if os.path.isfile(layer_conf): + with open(layer_conf) as conf: + for line in conf: + if 'BBLAYERS_LAYERINDEX_NAME' in line: + layer_name = line.split('=')[1].strip(' "\n') + return layer_name + + def add_core(self, layers): + utils.setup_django() + core = None + import settings + for layer in layers: + layer_name = self.get_layer_name(layer) + if layer_name == settings.CORE_LAYER_NAME: + if self.add_layer(layer): + self.logger.info('Failed to add core layer\n') + core = layer + self.update() + break + return core + + def update(self): + update_py = os.path.realpath(os.path.join(os.path.dirname(__file__), '../update.py')) + cmd = [update_py] + if self.options.loglevel == logging.DEBUG: + cmd.append("-d") + sys.argv = cmd + self.logger.info("update") + if not self.options.dryrun: + try: + update.main(); + except SystemExit: + return 1 + + return 0 + + +if __name__ == "__main__": + x = ImportProject() + x.main()
Signature · 1 line
1.9.1
PATCH v2 7/12 layerindex/recipeparse.py: refactor setup_tinfoil, checkout_layer_branch, parse_layer_conf to utils.py
In reply to Liam R. HowlettMove functions to utils to be used by other classes. _
--- layerindex/bulkchange.py | 4 ++-- layerindex/recipeparse.py | 39 +++----------------------------------- layerindex/utils.py | 48 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 53 insertions(+), 38 deletions(-)
layerindex/bulkchange.py +2 -2
@@ -42,9 +42,9 @@ def generate_patches(tinfoil, fetchdir, changeset, outputdir): patchname = "%s.patch" % layer.name patches.append(patchname) layerfetchdir = os.path.join(fetchdir, layer.get_fetch_dir()) - recipeparse.checkout_layer_branch(layerbranch, layerfetchdir) + utils.checkout_layer_branch(layerbranch, layerfetchdir) layerdir = os.path.join(layerfetchdir, layerbranch.vcs_subdir) - config_data_copy = recipeparse.setup_layer(tinfoil.config_data, fetchdir, layerdir, layer, layerbranch) + config_data_copy = utils.setup_layer(tinfoil.config_data, fetchdir, layerdir, layer, layerbranch) if outfile: outfile.close() outfile = open(os.path.join(tmpoutdir, patchname), 'w')
layerindex/recipeparse.py +3 -36
@@ -20,31 +20,6 @@ class RecipeParseError(Exception): def __str__(self): return self.msg -def _setup_tinfoil(bitbakepath, enable_tracking): - sys.path.insert(0, bitbakepath + '/lib') - import bb.tinfoil - import bb.cooker - import bb.data - try: - tinfoil = bb.tinfoil.Tinfoil(tracking=enable_tracking) - except TypeError: - # old API - tinfoil = bb.tinfoil.Tinfoil() - if enable_tracking: - tinfoil.cooker.enableDataTracking() - tinfoil.prepare(config_only = True) - - return tinfoil - -def _parse_layer_conf(layerdir, data): - data.setVar('LAYERDIR', str(layerdir)) - if hasattr(bb, "cookerdata"): - # Newer BitBake - data = bb.cookerdata.parse_config_file(os.path.join(layerdir, "conf", "layer.conf"), data) - else: - # Older BitBake (1.18 and below) - data = bb.cooker._parse(os.path.join(layerdir, "conf", "layer.conf"), data) - data.expandVarref('LAYERDIR') def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout=False, classic=False, logger=None):
@@ -97,7 +72,7 @@ def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout tempdir = tempfile.mkdtemp(dir=settings.TEMP_BASE_DIR) os.chdir(tempdir) - tinfoil = _setup_tinfoil(bitbakepath, enable_tracking) + tinfoil = utils.setup_tinfoil(bitbakepath, enable_tracking) # Ensure TMPDIR exists (or insane.bbclass will blow up trying to write to the QA log) oe_tmpdir = tinfoil.config_data.getVar('TMPDIR', True)
@@ -110,14 +85,6 @@ def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout return (tinfoil, tempdir) -def checkout_layer_branch(layerbranch, repodir, logger=None): - if layerbranch.actual_branch: - branchname = layerbranch.actual_branch - else: - branchname = layerbranch.branch.name - out = utils.runcmd("git checkout origin/%s" % branchname, repodir, logger=logger) - out = utils.runcmd("git clean -f -x", repodir, logger=logger) - def setup_layer(config_data, fetchdir, layerdir, layer, layerbranch): # Parse layer.conf files for this layer and its dependencies # This is necessary not just because BBPATH needs to be set in order
@@ -125,7 +92,7 @@ def setup_layer(config_data, fetchdir, layerdir, layer, layerbranch): # or across layers, but also because custom variable values might be # set in layer.conf. config_data_copy = bb.data.createCopy(config_data) - _parse_layer_conf(layerdir, config_data_copy) + utils.parse_layer_conf(layerdir, config_data_copy) for dep in layerbranch.dependencies_set.all(): depurldir = dep.dependency.get_fetch_dir() deprepodir = os.path.join(fetchdir, depurldir)
@@ -133,7 +100,7 @@ def setup_layer(config_data, fetchdir, layerdir, layer, layerbranch): if not deplayerbranch: raise RecipeParseError('Dependency %s of layer %s does not have branch record for branch %s' % (dep.dependency.name, layer.name, layerbranch.branch.name)) deplayerdir = os.path.join(deprepodir, deplayerbranch.vcs_subdir) - _parse_layer_conf(deplayerdir, config_data_copy) + utils.parse_layer_conf(deplayerdir, config_data_copy) config_data_copy.delVar('LAYERDIR') return config_data_copy
layerindex/utils.py +48 -0
54 lines not on this page · raw message
Signature · 1 line
1.9.1
PATCH v2 8/12 layerindex: Detect dependencies from layer.conf files
In reply to Liam R. HowlettRead dependencies from layer.conf and try to create the LayerDependency entry by looking up the correct database object. Dependencies are found by layer name only - no collection support. layer.conf parsing is handled by the bitbake code.
Once all layers are added, the dependencies have to be rechecked in case the layers are not added in order.
--- layerindex/layerconfparse.py | 50 ++++++++++++++++++++++++ layerindex/tools/import_layer.py | 11 ++++++ layerindex/update.py | 41 +++++++++++++++++++- layerindex/update_layer.py | 20 ++++++---- layerindex/utils.py | 82 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 196 insertions(+), 8 deletions(-) create mode 100644 layerindex/layerconfparse.py
layerindex/layerconfparse.py new file +50 -0
@@ -0,0 +1,50 @@ +# Utility functions for parsing layer.conf using bitbake within layerindex-web +# +# Copyright (C) 2016 Wind River Systems +# Author: Liam R. Howlett <[email protected]> +# +# Licensed under the MIT license, see COPYING.MIT for details +# + +import sys +import os +import os.path +import utils +import tempfile +import re + +class LayerConfParse: + def __init__(self, enable_tracking=False, logger=None, bitbakepath=None, tinfoil=None): + import settings + self.logger = logger + + if not bitbakepath: + fetchdir = settings.LAYER_FETCH_DIR + bitbakepath = os.path.join(fetchdir, 'bitbake') + self.bbpath = bitbakepath + + # Set up BBPATH. + os.environ['BBPATH'] = str("%s" % self.bbpath) + self.tinfoil = tinfoil + + if not self.tinfoil: + self.tinfoil = utils.setup_tinfoil(self.bbpath, enable_tracking) + + self.config_data_copy = bb.data.createCopy(self.tinfoil.config_data) + + def parse_layer(self, layerbranch, layerdir): + + utils.checkout_layer_branch(layerbranch, layerdir, self.logger) + + + # This is not a valid layer, parsing will cause exception. + if not utils.is_layer_valid(layerdir): + return None + + utils.parse_layer_conf(layerdir, self.config_data_copy, logger=self.logger) + return self.config_data_copy + + def shutdown(self): + self.tinfoil.shutdown() + +
layerindex/tools/import_layer.py +11 -0
@@ -19,6 +19,7 @@ import glob import utils import logging import subprocess +from layerconfparse import LayerConfParse class DryRunRollbackException(Exception): pass
@@ -367,11 +368,21 @@ def main(): if layer.name != settings.CORE_LAYER_NAME: if not core_layer: core_layer = utils.get_layer(settings.CORE_LAYER_NAME) + if core_layer: + logger.debug('Adding dep %s to %s' % (core_layer.name, layer.name)) layerdep = LayerDependency() layerdep.layerbranch = layerbranch layerdep.dependency = core_layer layerdep.save() + try: + layerconfparser = LayerConfParse(logger=logger) + config_data = layerconfparser.parse_layer(layerbranch, layerdir) + finally: + layerconfparser.shutdown() + if config_data: + utils.add_dependencies(layerbranch, config_data, logger=logger) + # Get some extra meta-information readme_files = glob.glob(os.path.join(layerdir, 'README*'))
layerindex/update.py +40 -1
71 lines not on this page · raw message
layerindex/update_layer.py +13 -7
@@ -196,7 +196,6 @@ def main(): except recipeparse.RecipeParseError as e: logger.error(str(e)) sys.exit(1) - # Clear the default value of SUMMARY so that we can use DESCRIPTION instead if it hasn't been set tinfoil.config_data.setVar('SUMMARY', '') # Clear the default value of DESCRIPTION so that we can see where it's not set
@@ -244,7 +243,7 @@ def main(): layerbranch = LayerBranch() layerbranch.layer = layer layerbranch.branch = branch - layerbranch_source = layer.get_layerbranch('master') + layerbranch_source = layer.get_layerbranch(branch) if not layerbranch_source: layerbranch_source = layer.get_layerbranch(None) if layerbranch_source:
@@ -256,11 +255,6 @@ def main(): maintainer.id = None maintainer.layerbranch = layerbranch maintainer.save() - for dep in layerbranch_source.dependencies_set.all(): - dep.pk = None - dep.id = None - dep.layerbranch = layerbranch - dep.save() if layerbranch.vcs_subdir and not options.nocheckout: # Find latest commit in subdirectory
@@ -280,6 +274,17 @@ def main(): layerdir = os.path.join(repodir, layerbranch.vcs_subdir) layerdir_start = os.path.normpath(layerdir) + os.sep + + from layerconfparse import LayerConfParse + layerconfparser = LayerConfParse(logger=logger, tinfoil=tinfoil) + layer_config_data = layerconfparser.parse_layer(layerbranch, layerdir) + if not layer_config_data: + logger.info("Skipping update of layer %s for branch %s - conf/layer.conf may have parse issues" % (layer.name, branchdesc)) + layerconfparser.shutdown() + sys.exit(1) + utils.add_dependencies(layerbranch, layer_config_data, logger=logger) + layerbranch.save() + layerrecipes = Recipe.objects.filter(layerbranch=layerbranch) layermachines = Machine.objects.filter(layerbranch=layerbranch) layerdistros = Distro.objects.filter(layerbranch=layerbranch)
@@ -655,6 +660,7 @@ def main(): import traceback traceback.print_exc() + tinfoil.shutdown() shutil.rmtree(tempdir) sys.exit(0)
layerindex/utils.py +82 -0
88 lines not on this page · raw message
Signature · 1 line
1.9.1
PATCH v2 9/12 layerindex: Add collection and version to layerbranch
In reply to Liam R. HowlettCollection and version will be pulled from the layer.conf if it exists and dependencies will be resolved by first checking for layers with the dependency name and then checking for collections. It is necessary to shutdown tinfoil to avoid bitbake complaining about multiple instances.
--- layerindex/models.py | 2 ++ layerindex/update_layer.py | 1 + layerindex/utils.py | 20 ++++++++++++++++++-- 3 files changed, 21 insertions(+), 2 deletions(-)
layerindex/models.py +2 -0
@@ -130,6 +130,8 @@ class LayerItem(models.Model): class LayerBranch(models.Model): layer = models.ForeignKey(LayerItem) branch = models.ForeignKey(Branch) + collection = models.CharField('Layer Collection', max_length=40, null=True, help_text='Name of the layer that could be used in the list of dependencies - can only contain letters, numbers and dashes') + version = models.CharField('Layer Version', max_length=10, null=True, blank=True, help_text='The layer version for this particular branch.') vcs_subdir = models.CharField('Repository subdirectory', max_length=40, blank=True, help_text='Subdirectory within the repository where the layer is located, if not in the root (usually only used if the repository contains more than one layer)') vcs_last_fetch = models.DateTimeField('Last successful fetch', blank=True, null=True) vcs_last_rev = models.CharField('Last revision fetched', max_length=80, blank=True)
layerindex/update_layer.py +1 -0
@@ -283,6 +283,7 @@ def main(): layerconfparser.shutdown() sys.exit(1) utils.add_dependencies(layerbranch, layer_config_data, logger=logger) + utils.set_layerbranch_collection_version(layerbranch, layer_config_data, logger=logger) layerbranch.save() layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
layerindex/utils.py +18 -2
@@ -30,8 +30,10 @@ def get_layer(layername): def get_dependency_layer(depname, version_str=None, logger=None): from layerindex.models import LayerItem, LayerBranch - # Get any LayerBranch with a layer that has a name that matches the depname - res = list(LayerBranch.objects.filter(layer__name=depname)) + # Get any LayerBranch with a layer that has a name that matches depmod, or + # a LayerBranch that has the collection name depmod. + res = list(LayerBranch.objects.filter(layer__name=depname)) + \ + list(LayerBranch.objects.filter(collection=depname)) # Nothing found, return. if not res:
@@ -68,6 +70,10 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None): layer_name = layerbranch.layer.name var_name = layer_name + if layerbranch.collection: + var_name = layerbranch.collection + + dep_list = config_data.getVar("%s_%s" % (var, var_name), True) if not dep_list:
@@ -104,11 +110,21 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None): if logger: logger.debug('Adding %s %s to %s' % (name, dep_layer.name, layer_name)) + layerdep = LayerDependency() layerdep.layerbranch = layerbranch layerdep.dependency = dep_layer layerdep.save() +def set_layerbranch_collection_version(layerbranch, config_data, logger=None): + + layerbranch.collection = config_data.getVar('BBFILE_COLLECTIONS', True) + ver_str = "LAYERVERSION_" + if layerbranch.collection: + layerbranch.collection = layerbranch.collection.strip() + ver_str += layerbranch.collection + layerbranch.version = config_data.getVar(ver_str, True) + def setup_tinfoil(bitbakepath, enable_tracking): sys.path.insert(0, bitbakepath + '/lib') import bb.tinfoil
Signature · 1 line
1.9.1
Re: [layerindex-web][PATCH v2 09/12] layerindex: Add collection and version to layerbranch
Quoted 4 lines · On Fri, 07 Oct 2016 11:57:18 Liam R. Howlett wrote:
Collection and version will be pulled from the layer.conf if it exists and dependencies will be resolved by first checking for layers with the dependency name and then checking for collections. It is necessary to shutdown tinfoil to avoid bitbake complaining about multiple instances.
That last bit of the commit message is now invalid in v2.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Re: [layerindex-web][PATCH v2 09/12] layerindex: Add collection and version to layerbranch
Quoted 7 lines · Paul Eggleton
Quoted 4 lines · On Fri, 07 Oct 2016 11:57:18 Liam R. Howlett wrote:
Collection and version will be pulled from the layer.conf if it exists and dependencies will be resolved by first checking for layers with the dependency name and then checking for collections. It is necessary to shutdown tinfoil to avoid bitbake complaining about multiple instances.
That last bit of the commit message is now invalid in v2.
I have removed the errant comment (last line) from the commit message in the current 'mhatle/django18' branch.
Quoted 3 lines
Cheers,
Paul
PATCH v2 10/12 layerindexer: Add layer recommends support
In reply to Liam R. HowlettParse layer.conf and add dependencies that are not required from LAYERRECOMMENDS_<name>. Update the layerindex/template to support recommends. Uses bitbake parsing code & checks versions.
--- layerindex/models.py | 7 +++++++ layerindex/tools/import_layer.py | 2 +- layerindex/update.py | 1 + layerindex/update_layer.py | 1 + layerindex/utils.py | 9 +++++++-- templates/layerindex/detail.html | 32 +++++++++++++++++++++----------- 6 files changed, 38 insertions(+), 14 deletions(-)
layerindex/models.py +7 -0
@@ -211,6 +211,12 @@ class LayerBranch(models.Model): return "%s: %s" % (self.layer.name, self.branch.name) + def get_required(self): + return self.dependencies_set.filter(required=True) + + def get_recommends(self): + return self.dependencies_set.filter(required=False) + class LayerMaintainer(models.Model): MAINTAINER_STATUS_CHOICES = ( ('A', 'Active'),
@@ -232,6 +238,7 @@ class LayerMaintainer(models.Model): class LayerDependency(models.Model): layerbranch = models.ForeignKey(LayerBranch, related_name='dependencies_set') dependency = models.ForeignKey(LayerItem, related_name='dependents_set') + required = models.BooleanField(default=True) class Meta: verbose_name_plural = "Layer dependencies"
layerindex/tools/import_layer.py +1 -1
@@ -382,7 +382,7 @@ def main(): layerconfparser.shutdown() if config_data: utils.add_dependencies(layerbranch, config_data, logger=logger) - + utils.add_recommends(layerbranch, config_data, logger=logger) # Get some extra meta-information readme_files = glob.glob(os.path.join(layerdir, 'README*'))
layerindex/update.py +1 -0
@@ -235,6 +235,7 @@ def main(): continue utils.add_dependencies(layerbranch, config_data, logger=logger) + utils.add_recommends(layerbranch, config_data, logger=logger) finally: layerconfparser.shutdown()
layerindex/update_layer.py +1 -0
@@ -283,6 +283,7 @@ def main(): layerconfparser.shutdown() sys.exit(1) utils.add_dependencies(layerbranch, layer_config_data, logger=logger) + utils.add_recommends(layerbranch, layer_config_data, logger=logger) utils.set_layerbranch_collection_version(layerbranch, layer_config_data, logger=logger) layerbranch.save()
layerindex/utils.py +7 -2
@@ -62,9 +62,12 @@ def get_dependency_layer(depname, version_str=None, logger=None): return None def add_dependencies(layerbranch, config_data, logger=None): - _add_dependency("LAYERDEPENDS", 'dependency', layerbranch, config_data, logger) + _add_dependency("LAYERDEPENDS", 'dependency', layerbranch, config_data, logger=logger) -def _add_dependency(var, name, layerbranch, config_data, logger=None): +def add_recommends(layerbranch, config_data, logger=None): + _add_dependency("LAYERRECOMMENDS", 'recommends', layerbranch, config_data, logger=logger, required=False) + +def _add_dependency(var, name, layerbranch, config_data, logger=None, required=True): from layerindex.models import LayerBranch, LayerDependency layer_name = layerbranch.layer.name
@@ -97,6 +100,7 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None): logger.error('Error getting %s %s for %s\n%s' %(name, dep. layer_name, str(vse))) continue + # No layer found. if not dep_layer: if logger: logger.error('Cannot resolve %s %s (version %s) for %s' % (name, dep, ver_str, layer_name))
@@ -114,6 +118,7 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None): layerdep = LayerDependency() layerdep.layerbranch = layerbranch layerdep.dependency = dep_layer + layerdep.required = required layerdep.save() def set_layerbranch_collection_version(layerbranch, config_data, logger=None):
templates/layerindex/detail.html +21 -11
@@ -139,17 +139,27 @@ <div class="span4 pull-right description"> {% if layerbranch.dependencies_set.count > 0 %} - <div class="well dependency-well"> - <h3>Dependencies </h3> - <p>The {{ layeritem.name }} layer depends upon:</p> - <ul> - {% for dep in layerbranch.dependencies_set.all %} - <li><a href="{% url 'layer_item' url_branch dep.dependency.name %}">{{ dep.dependency.name }}</a></li> - {% endfor %} - </ul> - </div> <!-- end of well --> - {% endif %} - + <div class="well dependency-well"> + {% if layerbranch.get_required.count > 0 %} + <h3>Dependencies </h3> + <p>The {{ layeritem.name }} layer depends upon:</p> + <ul> + {% for dep in layerbranch.get_required %} + <li><a href="{% url 'layer_item' url_branch dep.dependency.name %}">{{ dep.dependency.name }}</a></li> + {% endfor %} + </ul> + {% endif %} <!-- end of layerbranch.get_required.count --> + {% if layerbranch.get_recommends.count > 0 %} + <h3>Recommends </h3> + <p>The {{ layeritem.name }} layer recommends:</p> + <ul> + {% for rec in layerbranch.get_recommends %} + <li><a href="{% url 'layer_item' url_branch rec.dependency.name %}">{{ rec.dependency.name }}</a></li> + {% endfor %} + </ul> + {% endif %} <!-- end of layerbranch.get_recommends.count --> + </div> <!-- end of well --> + {% endif %} <!-- end of layerbranch.dependencies_set.count --> </div> <!-- end of span4 --> </div> <!-- end of row-fluid --> </div> <!-- end of container-fluid -->
Signature · 1 line
1.9.1
PATCH v2 11/12 recipeparse: remove unnecessary else statement.
In reply to Liam R. HowlettCode clean up.
--- layerindex/recipeparse.py | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
layerindex/recipeparse.py +8 -9
@@ -134,15 +134,14 @@ def detect_file_type(path, subdir_start): if res: typename = 'machine' return (typename, None, res.group(1)) - else: - res = bbclass_re.match(subpath) - if res: - typename = 'bbclass' - return (typename, None, res.group(1)) - res = distro_conf_re.match(subpath) - if res: - typename = 'distro' - return (typename, None, res.group(1)) + res = bbclass_re.match(subpath) + if res: + typename = 'bbclass' + return (typename, None, res.group(1)) + res = distro_conf_re.match(subpath) + if res: + typename = 'distro' + return (typename, None, res.group(1)) if typename == 'recipe' or typename == 'bbappend': if subdir_start:
Signature · 1 line
1.9.1
PATCH v2 12/12 layerindex/update_layer.py: Preserve the recipedependency files
In reply to Liam R. HowlettFrom: Mark Hatle <[email protected]>
In order to keep primary keys from constantly changing, preverse the exist keys as much as possible.
--- layerindex/update_layer.py | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
layerindex/update_layer.py +18 -1
@@ -88,13 +88,30 @@ def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir): if depstr.startswith(layerdir_start) and not depstr.endswith('/conf/layer.conf'): filedeps.append(os.path.relpath(depstr, repodir)) from layerindex.models import RecipeFileDependency - RecipeFileDependency.objects.filter(recipe=recipe).delete() + + recipedeps_delete = [] + + recipedeps = RecipeFileDependency.objects.filter(recipe=recipe) + + for values in recipedeps.values('path'): + if 'path' in values: + recipedeps_delete.append(values['path']) + for filedep in filedeps: + if filedep in recipedeps_delete: + recipedeps_delete.remove(filedep) + continue + # New item, add it... recipedep = RecipeFileDependency() recipedep.layerbranch = recipe.layerbranch recipedep.recipe = recipe recipedep.path = filedep recipedep.save() + + for filedep in recipedeps_delete: + logger.debug('%s: removing %s' % (recipe.layerbranch, filedep)) + recipedeps.filter(path=filedep).delete() + except KeyboardInterrupt: raise except BaseException as e:
Signature · 1 line
1.9.1
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:
git://git.yoctoproject.org/layerindex-web mhatle/django18
BTW for me upgrading from paule/django18 to this version. I only had to run 'manage.py syncdb'.
This created the new database and added the missing field for the recommend.
I then ran update.py --forcereload and everything showed up as expected.
--Mark
Quoted 52 lines · Liam R. Howlett
This set of patches adds a number of features to the layerindex code:
- Adds the ability to set the actual-branch to import_layer. - Adds distro to the database and web interface. - Adds collection & version information to layerbranch. - Uses layer name and collection name to determine dependencies andrecommends.
- Addsimport_projectto simplify pulling all layers and openembedded-core into a layer index.Liam R. Howlett (11):
import_layer: Add --actual-branch optionlayerindex/tools/import_layer.py: Sanitize layer name.layerindex/tools/import_layer.py: Avoid failing if there is any layer
to add.
layerindex/utils: Update runcmd to decode binary strings to strings.
layerindex: Add distro to web interface and model. layerindex/tools/import_project: Addimport_projectlayerindex/recipeparse.py: refactorsetup_tinfoil,
checkout_layer_branch,parse_layer_conftoutils.py
layerindex: Detect dependencies from layer.conf files
layerindex: Add collection and version to layerbranch
layerindexer: Add layer recommends support
recipeparse: remove unnecessary else statement.Mark Hatle (1):
layerindex/update_layer.py: Preserve the recipedependency fileslayerindex/admin.py | 11 +++ layerindex/bulkchange.py | 4 +- layerindex/layerconfparse.py | 50 +++++++++++ layerindex/models.py | 23 +++++ layerindex/recipeparse.py | 53 +++-------- layerindex/restviews.py | 10 ++- layerindex/tools/import_classic.py | 2 + layerindex/tools/import_layer.py | 47 +++++++++- layerindex/tools/import_project.py | 180 +++++++++++++++++++++++++++++++++++++ layerindex/update.py | 42 ++++++++- layerindex/update_layer.py | 87 ++++++++++++++++-- layerindex/urls.py | 3 + layerindex/urls_branch.py | 6 +- layerindex/utils.py | 159 +++++++++++++++++++++++++++++++- layerindex/views.py | 29 +++++- templates/layerindex/detail.html | 55 +++++++++--- templates/layerindex/distros.html | 76 ++++++++++++++++ templates/layerindex/layers.html | 1 + templates/layerindex/machines.html | 1 + templates/layerindex/recipes.html | 1 + 20 files changed, 767 insertions(+), 73 deletions(-) create mode 100644 layerindex/layerconfparse.py create mode 100755 layerindex/tools/import_project.py create mode 100644 templates/layerindex/distros.html
Quoted 9 lines · On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:
git://git.yoctoproject.org/layerindex-webmhatle/django18BTW for me upgrading from paule/django18 to this version. I only had to run '
manage.pysyncdb'.This created the new database and added the missing field for the recommend.
I'm not sure how that worked there but it doesn't work here. If I take a database that works on paule/django18, run syncdb then open the admin interface and click on "Layer branches" I get:
no such column: layerindex_layerbranch.collection
I'm afraid we really do need to create migrations whenever we change the models.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 10 lines · Paul Eggleton
Quoted 7 lines · On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:
git://git.yoctoproject.org/layerindex-webmhatle/django18
Quoted 9 lines
Quoted 4 lines
BTW for me upgrading from paule/django18 to this version. I only had to run '
manage.pysyncdb'.This created the new database and added the missing field for the recommend.
I'm not sure how that worked there but it doesn't work here. If I take a database that works on paule/django18, run syncdb then open the admin interface and click on "Layer branches" I get:
We found a severe lack of instructions for creating migrations. There were instructions for adding and deleting full DBs but not changing them. We will need pointers or help to resolve this.
Quoted 1 line
no such column: layerindex_layerbranch.collection
I never got that, but I'm also running SQLite. May be the difference.
Quoted 2 lines
I'm afraid we really do need to create migrations whenever we change the models.
Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.
Quoted 7 lines
Cheers,
PaulSignature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 17 lines · Mark
Quoted 8 lines
Quoted 6 lines · Paul Eggleton
On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:
git://git.yoctoproject.org/layerindex-webmhatle/django18Quoted 8 lines
Quoted 5 lines
BTW for me upgrading from paule/django18 to this version. I only had to run '
manage.pysyncdb'.This created the new database and added the missing field for the recommend.>
I'm not sure how that worked there but it doesn't work here. If I take a database that works on paule/django18, run syncdb then open the admin interface and click on "Layer branches" I get:
Quoted 7 lines
We found a severe lack of instructions for creating migrations. There were instructions for adding and deleting full DBs but not changing them. We will need pointers or help to resolve this.
Quoted 1 line
no such column: layerindex_layerbranch.collection
I never got that, but I'm also running SQLite. May be the difference.
I'm using sqlite here also.
Quoted 5 lines
Quoted 2 lines
I'm afraid we really do need to create migrations whenever we change the models.
Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.
I'm happy to help directly if needed, but did you come across this already?
https://docs.djangoproject.com/en/1.8/topics/migrations/
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 31 lines · Paul Eggleton
Quoted 17 lines · Mark
Quoted 8 lines
Quoted 6 lines · Paul Eggleton
On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:
git://git.yoctoproject.org/layerindex-webmhatle/django18Quoted 8 lines
Quoted 5 lines
BTW for me upgrading from paule/django18 to this version. I only had to run '
manage.pysyncdb'.This created the new database and added the missing field for the recommend.>
I'm not sure how that worked there but it doesn't work here. If I take a database that works on paule/django18, run syncdb then open the admin interface and click on "Layer branches" I get:
Quoted 7 lines
We found a severe lack of instructions for creating migrations. There were instructions for adding and deleting full DBs but not changing them. We will need pointers or help to resolve this.
Quoted 1 line
no such column: layerindex_layerbranch.collection
I never got that, but I'm also running SQLite. May be the difference.
I'm using sqlite here also.
Quoted 3 lines
Quoted 2 lines
I'm afraid we really do need to create migrations whenever we change the models.
Quoted 6 lines
Quoted 2 lines
Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.
I'm happy to help directly if needed, but did you come across this already?
Unfortunately that is the exactly document I was looking at and could not figure it out.
I can the various commands, makemigrations and related.. and the system simply told me "no migrations necessary".. and didn't generate any.
So we were never able to figure out what a migration should look like or how to generate it.
If you (or someone else) can help us figure out how to create a migration, I suspect after the first one --- future ones will be much easier to generate.
--Mark
Quoted 3 lines
Cheers,
Paul
Quoted 51 lines · On Mon, 10 Oct 2016 12:47:11 Mark Hatle wrote:
Quoted 36 lines · Paul Eggleton
Quoted 21 lines · Mark
Quoted 14 lines
On Oct 10, 2016, at 2:54 AM, Paul Eggleton
<[email protected]>Quoted 10 lines · wrote:
On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:
git://git.yoctoproject.org/layerindex-webmhatle/django18BTW for me upgrading from paule/django18 to this version. I only had to
run 'manage.pysyncdb'.Quoted 7 lines
Quoted 2 lines
This created the new database and added the missing field for the recommend.>
I'm not sure how that worked there but it doesn't work here. If I take a database that works on paule/django18, run syncdb then open the admin
interface and click on "Layer branches" I get:
Quoted 8 lines
We found a severe lack of instructions for creating migrations. There were
instructions for adding and deleting full DBs but not changing them. We will need pointers or help to resolve this.Quoted 1 line
no such column: layerindex_layerbranch.collection
I never got that, but I'm also running SQLite. May be the difference.
I'm using sqlite here also.
Quoted 3 lines
Quoted 2 lines
I'm afraid we really do need to create migrations whenever we change the models.
Quoted 7 lines
Quoted 2 lines
Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.
I'm happy to help directly if needed, but did you come across this already?
Unfortunately that is the exactly document I was looking at and could not figure it out.
I can the various commands, makemigrations and related.. and the system simply told me "no migrations necessary".. and didn't generate any.
Quoted 6 lines
So we were never able to figure out what a migration should look like or how to generate it.
If you (or someone else) can help us figure out how to create a migration, I suspect after the first one --- future ones will be much easier to generate.
OK, so here's a step-by-step procedure - I suspect you'll need to stash and unstash your config changes along the way as well:
1) Ensure you are using a database created at the place you want to start, i.e. in this case paule/django18.
2) Determine which commits make changes to the models. This is one easy way:
$ git log -p paule/django18..mhatle/django18 layerindex/models.py
3) We don't have an initial migration, so we need to create that first in a single commit before all of the others, so to do that we'll direct the rebase to edit the top commit on paule/django18 but instead of modifying that we insert a commit after it. We also want to edit each commit found in step 1 that made changes so we can add the migrations to them.
$ git checkout mhatle/django18 $ git rebase -i paule/django18^ (now set the command to "e" on the oldest commit, in this case 6f8201fe, and do the same for all the commits you found in step 1, then save & quit the editor)
4) Create the initial migration:
$ python3 manage.py makemigrations layerindex
Migrations for 'layerindex':
0001_initial.py:
- Create model BBAppend
- Create model BBClass
- Create model Branch
- Create model LayerBranch
- Create model LayerDependency
- Create model LayerItem
- Create model LayerMaintainer
- Create model LayerNote
- Create model Machine
- Create model PythonEnvironment
- Create model Recipe
- Create model RecipeChange
- Create model RecipeChangeset
- Create model RecipeFileDependency
- Create model ClassicRecipe
- Add field recipe to recipefiledependency
- Add field changeset to recipechange
- Add field recipe to recipechange
- Add field layerbranch to recipe
- Add field dependency to layerdependency
- Add field layerbranch to layerdependency
- Add field layer to layerbranch
- Add field update_environment to branch
- Add field layerbranch to bbclass
- Add field layerbranch to bbappend
- Add field cover_layerbranch to classicrecipe5) Apply the migrations to the database - this time we make it "fake" because the tables already exist:
$ python manage.py migrate layerindex --fake-initial
6) Create the initial migrations commit. (We won't need to do this ever again)
$ git add layerindex/migrations/ $ git commit
7) Continue the rebase:
$ git rebase --continue
8) Create the migration for the changes in this commit:
$ python manage.py makemigrations layerindex
Migrations for 'layerindex':
0002_distro.py:
- Create model Distro9) Apply the migrations (*not* fake this time):
$ python manage.py migrate layerindex
10) Amend the migrations to the current commit:
$ git add layerindex/migrations/ $ git commit --amend
11) Continue the rebase:
$ git rebase --continue
12) Repeat steps 8-11 until the rebase finishes.
That should do it. It gets more complicated if you rename columns, or introduce columns that need initial data beyond a simple default, split models etc. but in this case when you're simply adding or removing columns it will take care of creating the migrations for you as long as you know the steps (and I admit it's a bit awkward if you've never done it before - I have the advantage of having done this already, not exactly with Django's migrations but with South and that was pretty similar).
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 153 lines · On Tue, 11 Oct 2016 09:19:15 Paul Eggleton wrote:
Quoted 54 lines · On Mon, 10 Oct 2016 12:47:11 Mark Hatle wrote:
Quoted 39 lines · Paul Eggleton
Quoted 28 lines · Mark
Quoted 14 lines
On Oct 10, 2016, at 2:54 AM, Paul Eggleton
<[email protected]>Quoted 10 lines · wrote:
On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:
git://git.yoctoproject.org/layerindex-webmhatle/django18BTW for me upgrading from paule/django18 to this version. I only had to
run 'manage.pysyncdb'.Quoted 8 lines
Quoted 2 lines
This created the new database and added the missing field for the recommend.>
I'm not sure how that worked there but it doesn't work here. If I take a
database that works on paule/django18, run syncdb then open the admininterface and click on "Layer branches" I get:
We found a severe lack of instructions for creating migrations. There were
instructions for adding and deleting full DBs but not changing them. We
will need pointers or help to resolve this.Quoted 3 lines
Quoted 1 line
no such column: layerindex_layerbranch.collection
I never got that, but I'm also running SQLite. May be the difference.
I'm using sqlite here also.
Quoted 4 lines
Quoted 3 lines
I'm afraid we really do need to create migrations whenever we change the
models.Quoted 7 lines
Quoted 2 lines
Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.
I'm happy to help directly if needed, but did you come across this already?
Unfortunately that is the exactly document I was looking at and could not figure it out.
I can the various commands, makemigrations and related.. and the system simply told me "no migrations necessary".. and didn't generate any.
Quoted 6 lines
So we were never able to figure out what a migration should look like or how to generate it.
If you (or someone else) can help us figure out how to create a migration, I suspect after the first one --- future ones will be much easier to generate.
OK, so here's a step-by-step procedure - I suspect you'll need to stash and unstash your config changes along the way as well:
1) Ensure you are using a database created at the place you want to start, i.e. in this case paule/django18.
2) Determine which commits make changes to the models. This is one easy way:
$ git log -p paule/django18..mhatle/django18 layerindex/models.py3) We don't have an initial migration, so we need to create that first in a single commit before all of the others, so to do that we'll direct the rebase to edit the top commit on paule/django18 but instead of modifying that we insert a commit after it. We also want to edit each commit found in step 1 that made changes so we can add the migrations to them.
$ git checkout mhatle/django18 $ git rebase -i paule/django18^ (now set the command to "e" on the oldest commit, in this case 6f8201fe, and do the same for all the commits you found in step 1, then save & quit the editor)4) Create the initial migration:
$ python3 manage.py makemigrations layerindex Migrations for 'layerindex': 0001_initial.py: - Create model BBAppend - Create model BBClass - Create model Branch - Create model LayerBranch - Create model LayerDependency - Create model LayerItem - Create model LayerMaintainer - Create model LayerNote - Create model Machine - Create model PythonEnvironment - Create model Recipe - Create model RecipeChange - Create model RecipeChangeset - Create model RecipeFileDependency - Create model ClassicRecipe - Add field recipe to recipefiledependency - Add field changeset to recipechange - Add field recipe to recipechange - Add field layerbranch to recipe - Add field dependency to layerdependency - Add field layerbranch to layerdependency - Add field layer to layerbranch - Add field update_environment to branch - Add field layerbranch to bbclass - Add field layerbranch to bbappend - Add field cover_layerbranch to classicrecipe5) Apply the migrations to the database - this time we make it "fake" because the tables already exist:
$ python manage.py migrate layerindex --fake-initial6) Create the initial migrations commit. (We won't need to do this ever again)
$ git add layerindex/migrations/ $ git commit7) Continue the rebase:
$ git rebase --continue8) Create the migration for the changes in this commit:
$ python manage.py makemigrations layerindex Migrations for 'layerindex': 0002_distro.py: - Create model Distro9) Apply the migrations (*not* fake this time):
$ python manage.py migrate layerindex10) Amend the migrations to the current commit:
$ git add layerindex/migrations/ $ git commit --amend11) Continue the rebase:
$ git rebase --continue12) Repeat steps 8-11 until the rebase finishes.
Quoted 7 lines
That should do it. It gets more complicated if you rename columns, or introduce columns that need initial data beyond a simple default, split models etc. but in this case when you're simply adding or removing columns it will take care of creating the migrations for you as long as you know the steps (and I admit it's a bit awkward if you've never done it before - I have the advantage of having done this already, not exactly with Django's migrations but with South and that was pretty similar).
Heh, I knew I would muck something up - where you see "python" in the above, read "python3". (In my virtualenv-based setup it doesn't make any difference, but that may not be true for yours.)
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 162 lines · Paul Eggleton
Quoted 153 lines · On Tue, 11 Oct 2016 09:19:15 Paul Eggleton wrote:
Quoted 54 lines · On Mon, 10 Oct 2016 12:47:11 Mark Hatle wrote:
Quoted 39 lines · Paul Eggleton
Quoted 28 lines · Mark
Quoted 14 lines
On Oct 10, 2016, at 2:54 AM, Paul Eggleton
<[email protected]>Quoted 10 lines · wrote:
On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:
git://git.yoctoproject.org/layerindex-webmhatle/django18BTW for me upgrading from paule/django18 to this version. I only had to
run 'manage.pysyncdb'.Quoted 8 lines
Quoted 2 lines
This created the new database and added the missing field for the recommend.>
I'm not sure how that worked there but it doesn't work here. If I take a
database that works on paule/django18, run syncdb then open the admininterface and click on "Layer branches" I get:
We found a severe lack of instructions for creating migrations. There were
instructions for adding and deleting full DBs but not changing them. We
will need pointers or help to resolve this.Quoted 3 lines
Quoted 1 line
no such column: layerindex_layerbranch.collection
I never got that, but I'm also running SQLite. May be the difference.
I'm using sqlite here also.
Quoted 4 lines
Quoted 3 lines
I'm afraid we really do need to create migrations whenever we change the
models.Quoted 7 lines
Quoted 2 lines
Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.
I'm happy to help directly if needed, but did you come across this already?
Unfortunately that is the exactly document I was looking at and could not figure it out.
I can the various commands, makemigrations and related.. and the system simply told me "no migrations necessary".. and didn't generate any.
Quoted 6 lines
So we were never able to figure out what a migration should look like or how to generate it.
If you (or someone else) can help us figure out how to create a migration, I suspect after the first one --- future ones will be much easier to generate.
OK, so here's a step-by-step procedure - I suspect you'll need to stash and unstash your config changes along the way as well:
1) Ensure you are using a database created at the place you want to start, i.e. in this case paule/django18.
2) Determine which commits make changes to the models. This is one easy way:
$ git log -p paule/django18..mhatle/django18 layerindex/models.py3) We don't have an initial migration, so we need to create that first in a single commit before all of the others, so to do that we'll direct the rebase to edit the top commit on paule/django18 but instead of modifying that we insert a commit after it. We also want to edit each commit found in step 1 that made changes so we can add the migrations to them.
$ git checkout mhatle/django18 $ git rebase -i paule/django18^ (now set the command to "e" on the oldest commit, in this case 6f8201fe, and do the same for all the commits you found in step 1, then save & quit the editor)4) Create the initial migration:
$ python3 manage.py makemigrations layerindex Migrations for 'layerindex': 0001_initial.py: - Create model BBAppend - Create model BBClass - Create model Branch - Create model LayerBranch - Create model LayerDependency - Create model LayerItem - Create model LayerMaintainer - Create model LayerNote - Create model Machine - Create model PythonEnvironment - Create model Recipe - Create model RecipeChange - Create model RecipeChangeset - Create model RecipeFileDependency - Create model ClassicRecipe - Add field recipe to recipefiledependency - Add field changeset to recipechange - Add field recipe to recipechange - Add field layerbranch to recipe - Add field dependency to layerdependency - Add field layerbranch to layerdependency - Add field layer to layerbranch - Add field update_environment to branch - Add field layerbranch to bbclass - Add field layerbranch to bbappend - Add field cover_layerbranch to classicrecipe5) Apply the migrations to the database - this time we make it "fake" because the tables already exist:
$ python manage.py migrate layerindex --fake-initial6) Create the initial migrations commit. (We won't need to do this ever again)
$ git add layerindex/migrations/ $ git commit7) Continue the rebase:
$ git rebase --continue8) Create the migration for the changes in this commit:
$ python manage.py makemigrations layerindex Migrations for 'layerindex': 0002_distro.py: - Create model Distro9) Apply the migrations (*not* fake this time):
$ python manage.py migrate layerindex10) Amend the migrations to the current commit:
$ git add layerindex/migrations/ $ git commit --amend11) Continue the rebase:
$ git rebase --continue12) Repeat steps 8-11 until the rebase finishes.
Quoted 7 lines
That should do it. It gets more complicated if you rename columns, or introduce columns that need initial data beyond a simple default, split models etc. but in this case when you're simply adding or removing columns it will take care of creating the migrations for you as long as you know the steps (and I admit it's a bit awkward if you've never done it before - I have the advantage of having done this already, not exactly with Django's migrations but with South and that was pretty similar).
Quoted 3 lines
Heh, I knew I would muck something up - where you see "python" in the above, read "python3". (In my virtualenv-based setup it doesn't make any difference, but that may not be true for yours.)
No problem. The steps above did work for me. Migrations have now been added to the mhatle/django18 branch.
I think what we were missing was the setup of the initial migration, step #4 above. Without doing that you get messages like no migrations necessary and such -- very confusing.
Thanks for the help, let me know if there is anything else that needs to be adjusted.
--Mark
Quoted 3 lines
Cheers,
Paul