Skip to contentSkip to conversation

Add Distro, dependency and recommends detection, and import_project

PATCH In review Liam R. Howlett 3 participants 25 messages Oct 7, 2016 – Oct 11, 2016

lore
Share & tools
lore
#0 lore raw

v2 cover Add Distro, dependency and recommends detection, and import_project

This set of patches adds a number of features to the layerindex code:

 -  Adds the ability to set the actual-branch to import_layer.
 -  Adds distro to the database and web interface.
 -  Adds collection & version information to layerbranch.
 -  Uses layer name and collection name to determine dependencies and

recommends.
- Adds import_project to simplify pulling all layers and openembedded-core into a layer index.

Liam R. Howlett (11):
import_layer: Add --actual-branch option layerindex/tools/import_layer.py: Sanitize layer name. layerindex/tools/import_layer.py: Avoid failing if there is any layer
to add.
layerindex/utils: Update runcmd to decode binary strings to strings.
layerindex: Add distro to web interface and model. layerindex/tools/import_project: Add import_project layerindex/recipeparse.py: refactor setup_tinfoil,
checkout_layer_branch, parse_layer_conf to utils.py
layerindex: Detect dependencies from layer.conf files
layerindex: Add collection and version to layerbranch
layerindexer: Add layer recommends support
recipeparse: remove unnecessary else statement.

Mark Hatle (1):
layerindex/update_layer.py: Preserve the recipedependency files

 layerindex/admin.py                |  11 +++
 layerindex/bulkchange.py           |   4 +-
 layerindex/layerconfparse.py       |  50 +++++++++++
 layerindex/models.py               |  23 +++++
 layerindex/recipeparse.py          |  53 +++--------
 layerindex/restviews.py            |  10 ++-
 layerindex/tools/import_classic.py |   2 +
 layerindex/tools/import_layer.py   |  47 +++++++++-
 layerindex/tools/import_project.py | 180 +++++++++++++++++++++++++++++++++++++
 layerindex/update.py               |  42 ++++++++-
 layerindex/update_layer.py         |  87 ++++++++++++++++--
 layerindex/urls.py                 |   3 +
 layerindex/urls_branch.py          |   6 +-
 layerindex/utils.py                | 159 +++++++++++++++++++++++++++++++-
 layerindex/views.py                |  29 +++++-
 templates/layerindex/detail.html   |  55 +++++++++---
 templates/layerindex/distros.html  |  76 ++++++++++++++++
 templates/layerindex/layers.html   |   1 +
 templates/layerindex/machines.html |   1 +
 templates/layerindex/recipes.html  |   1 +
 20 files changed, 767 insertions(+), 73 deletions(-)
 create mode 100644 layerindex/layerconfparse.py
 create mode 100755 layerindex/tools/import_project.py
 create mode 100644 templates/layerindex/distros.html
Signature · 1 line
1.9.1
#1 lore raw

PATCH v2 1/12 import_layer: Add --actual-branch option

Allow users to set actual-branch from the command line import of layers.

---
 layerindex/tools/import_layer.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
layerindex/tools/import_layer.py +8 -2
index 0521e1c..9b5da22 100755
@@ -200,6 +200,9 @@ def main():
     parser.add_option("-q", "--quiet",
             help = "Hide all output except error messages",
             action="store_const", const=logging.ERROR, dest="loglevel")
+    parser.add_option("-a", "--actual-branch",
+            help = "Set actual branch",
+            action="store", dest="actual_branch")
 
     options, args = parser.parse_args(sys.argv)
 
@@ -273,10 +276,13 @@ def main():
                 logger.error("Fetch failed: %s" % str(e))
                 sys.exit(1)
 
-            actual_branch = ''
+            actual_branch = 'master'
+            if (options.actual_branch):
+                actual_branch = options.actual_branch
             try:
-                out = utils.runcmd("git checkout origin/master", repodir, logger=logger)
+                out = utils.runcmd("git checkout origin/%s" % actual_branch, repodir, logger=logger)
             except subprocess.CalledProcessError:
+                actual_branch = None
                 branches = utils.runcmd("git branch -r", repodir, logger=logger)
                 for line in branches.splitlines():
                     if 'origin/HEAD ->' in line:
Signature · 1 line
1.9.1
#2 lore raw

PATCH v2 2/12 layerindex/tools/import_layer.py: Sanitize layer name.

In reply to Liam R. Howlett

Django will produce a cryptic error message if layers are added with invalid names. Sanitize the layer names when trying to add them.

---
 layerindex/tools/import_layer.py | 6 ++++++
 1 file changed, 6 insertions(+)
layerindex/tools/import_layer.py +6 -0
index 9b5da22..fefef0c 100755
@@ -181,6 +181,8 @@ def get_github_layerinfo(layer_url, username = None, password = None):
 
 
 def main():
+    valid_layer_name = re.compile('[-\w]+$')
+
     parser = optparse.OptionParser(
         usage = """
     %prog [options] <url> [name]""")
@@ -222,6 +224,10 @@ def main():
             if layer_name.endswith('.git'):
                 layer_name = layer_name[:-4]
 
+    if not valid_layer_name.match(layer_name):
+        logger.error('Invalid layer name "%s" -  Layer name can only include letters, numbers and dashes.', layer_name)
+        sys.exit(1)
+
     if options.github_auth:
         if not ':' in options.github_auth:
             logger.error('--github-auth value must be specified as username:password')
Signature · 1 line
1.9.1
#3 lore raw

PATCH v2 3/12 layerindex/tools/import_layer.py: Avoid failing if there is any layer to add.

In reply to Liam R. Howlett

Subdirectories are scanned when adding layers. If any of the subdirectories or root directory layers already exist in the database, then the addition fails. This changes that behaviour to report the failure as a warning and remove it from the list. That way, if a repo has a new layer added it can be rescanned without issue. Layers being rescanned are checked against the vcs_url to ensure there is not a name collision. A name collision without the same vcs_url will still produce a hard failure.

Note that multiple layers with the same vcs_url are supported in the error reporting even though this should never happen.

---
 layerindex/tools/import_layer.py | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
layerindex/tools/import_layer.py +18 -2
index fefef0c..184c5cc 100755
@@ -334,8 +334,20 @@ def main():
                 else:
                     subdir = ''
                 if LayerItem.objects.filter(name=layer.name).exists():
-                    logger.error('A layer named "%s" already exists in the database' % layer_name)
-                    sys.exit(1)
+                    if LayerItem.objects.filter(name=layer.name).exclude(vcs_url=layer.vcs_url).exists():
+                        conflict_list = LayerItem.objects.filter(name=layer.name).exclude(vcs_url=layer.vcs_url)
+                        conflict_list_urls = []
+                        for conflict in conflict_list:
+                            conflict_list_urls.append(conflict.vcs_url)
+                        cln = ', '.join(conflict_list_urls)
+                        logger.error('A layer named "%s" already exists in the database.  Possible name collision with %s.vcs_url = %s' % (layer.name, layer.name, cln))
+                        sys.exit(1)
+                    else:
+                        logger.info('The layer named "%s" already exists in the database. Skipping this layer with same vcs_url' % layer.name)
+                        layer_paths = [x for x in layer_paths if x != layerdir]
+                        continue
+
+
 
                 logger.info('Creating layer %s' % layer.name)
                 # Guess layer type
@@ -411,6 +423,10 @@ def main():
 
                 layer.save()
 
+            if not layer_paths:
+                logger.error('No layers added.')
+                sys.exit(1);
+
             if options.dryrun:
                 raise DryRunRollbackException()
     except DryRunRollbackException:
Signature · 1 line
1.9.1
#4 lore raw

PATCH v2 4/12 layerindex/utils: Update runcmd to decode binary strings to strings.

In reply to Liam R. Howlett

Convert binary strings to strings and strip leading/trailing whitespace prior to returning errors and output.

---
 layerindex/utils.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
layerindex/utils.py +6 -2
index 50268e0..23b81f5 100644
@@ -32,7 +32,8 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None):
         execute command, raise CalledProcessError if fail
         return output if succeed
     """
-    #logger.debug("run cmd '%s' in %s" % (cmd, os.getcwd() if destdir is None else destdir))
+    if logger:
+        logger.debug("run cmd '%s' in %s" % (cmd, os.getcwd() if destdir is None else destdir))
     out = tempfile.TemporaryFile()
     try:
         subprocess.check_call(cmd, stdout=out, stderr=out, cwd=destdir, shell=True)
@@ -40,6 +41,7 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None):
         out.seek(0)
         if printerr:
             output = out.read()
+            output = output.decode('ascii').strip()
             if logger:
                 logger.error("%s" % output)
             else:
@@ -49,7 +51,9 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None):
 
     out.seek(0)
     output = out.read()
-    #logger.debug("output: %s" % output.rstrip() )
+    output = output.decode('ascii').strip()
+    if logger:
+        logger.debug("output: %s" % output.rstrip() )
     return output
 
 def setup_django():
Signature · 1 line
1.9.1
#5 lore raw

PATCH v2 5/12 layerindex: Add distro to web interface and model.

In reply to Liam R. Howlett

Add the distros to the index. This looks a lot like the machines and allows users to search for a particular distro.

---
 layerindex/admin.py                | 11 ++++++
 layerindex/models.py               | 14 +++++++
 layerindex/recipeparse.py          |  5 +++
 layerindex/restviews.py            | 10 ++++-
 layerindex/tools/import_classic.py |  2 +
 layerindex/update_layer.py         | 46 ++++++++++++++++++++++-
 layerindex/urls.py                 |  3 ++
 layerindex/urls_branch.py          |  6 ++-
 layerindex/views.py                | 29 ++++++++++++++-
 templates/layerindex/detail.html   | 23 ++++++++++++
 templates/layerindex/distros.html  | 76 ++++++++++++++++++++++++++++++++++++++
 templates/layerindex/layers.html   |  1 +
 templates/layerindex/machines.html |  1 +
 templates/layerindex/recipes.html  |  1 +
 14 files changed, 224 insertions(+), 4 deletions(-)
 create mode 100644 templates/layerindex/distros.html
layerindex/admin.py +11 -0
index accb954..7339ce1 100644
@@ -75,6 +75,16 @@ class MachineAdmin(admin.ModelAdmin):
     def has_delete_permission(self, request, obj=None):
         return False
 
+class DistroAdmin(admin.ModelAdmin):
+    search_fields = ['name']
+    list_filter = ['layerbranch__layer__name', 'layerbranch__branch__name']
+    readonly_fields = Distro._meta.get_all_field_names()
+    def has_add_permission(self, request, obj=None):
+        return False
+    def has_delete_permission(self, request, obj=None):
+        return False
+
+
 class BBAppendAdmin(admin.ModelAdmin):
     search_fields = ['filename']
     list_filter = ['layerbranch__layer__name', 'layerbranch__branch__name']
@@ -111,6 +121,7 @@ admin.site.register(LayerNote, LayerNoteAdmin)
 admin.site.register(Recipe, RecipeAdmin)
 admin.site.register(RecipeFileDependency)
 admin.site.register(Machine, MachineAdmin)
+admin.site.register(Distro, DistroAdmin)
 admin.site.register(BBAppend, BBAppendAdmin)
 admin.site.register(BBClass, BBClassAdmin)
 admin.site.register(RecipeChangeset, RecipeChangesetAdmin)
layerindex/models.py +14 -0
index 3fd16d4..2343ba7 100644
@@ -361,6 +361,20 @@ class Machine(models.Model):
     def __str__(self):
         return '%s (%s)' % (self.name, self.layerbranch.layer.name)
 
+class Distro(models.Model):
+    layerbranch = models.ForeignKey(LayerBranch)
+    name = models.CharField(max_length=255)
+    description = models.CharField(max_length=255)
+
+    updated = models.DateTimeField(auto_now=True)
+
+    def vcs_web_url(self):
+        url = self.layerbranch.file_url(os.path.join('conf/distro/%s.conf' % self.name))
+        return url or ''
+
+    def __str__(self):
+        return '%s (%s)' % (self.name, self.layerbranch.layer.name)
+
 
 class BBAppend(models.Model):
     layerbranch = models.ForeignKey(LayerBranch)
layerindex/recipeparse.py +5 -0
index 7f995ec..a9cecfc 100644
@@ -152,6 +152,7 @@ def get_var_files(fn, varlist, d):
     return varfiles
 
 machine_conf_re = re.compile(r'conf/machine/([^/.]*).conf$')
+distro_conf_re = re.compile(r'conf/distro/([^/.]*).conf$')
 bbclass_re = re.compile(r'classes/([^/.]*).bbclass$')
 def detect_file_type(path, subdir_start):
     typename = None
@@ -171,6 +172,10 @@ def detect_file_type(path, subdir_start):
             if res:
                 typename = 'bbclass'
                 return (typename, None, res.group(1))
+            res = distro_conf_re.match(subpath)
+            if res:
+                typename = 'distro'
+                return (typename, None, res.group(1))
 
     if typename == 'recipe' or typename == 'bbappend':
         if subdir_start:
layerindex/restviews.py +9 -1
index b33d3d1..57f1552 100644
@@ -1,4 +1,4 @@
-from layerindex.models import Branch, LayerItem, LayerNote, LayerBranch, LayerDependency, Recipe, Machine
+from layerindex.models import Branch, LayerItem, LayerNote, LayerBranch, LayerDependency, Recipe, Machine, Distro
 from rest_framework import viewsets, serializers
 from layerindex.querysethelper import params_to_queryset, get_search_tuple
 
@@ -56,3 +56,11 @@ class MachineSerializer(serializers.ModelSerializer):
 class MachineViewSet(ParametricSearchableModelViewSet):
     queryset = Machine.objects.all()
     serializer_class = MachineSerializer
+
+class DistroSerializer(serializers.ModelSerializer):
+    class Meta:
+        model = Distro
+
+class DistroViewSet(ParametricSearchableModelViewSet):
+    queryset = Distro.objects.all()
+    serializer_class = DistroSerializer
layerindex/tools/import_classic.py +2 -0
index 45ccaa9..80c49af 100755
@@ -160,6 +160,7 @@ def main():
         layerdir_start = os.path.normpath(oeclassicpath) + os.sep
         layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
         layermachines = Machine.objects.filter(layerbranch=layerbranch)
+        layerdistros = Distro.objects.filter(layerbranch=layerbranch)
         layerappends = BBAppend.objects.filter(layerbranch=layerbranch)
         layerclasses = BBClass.objects.filter(layerbranch=layerbranch)
 
@@ -172,6 +173,7 @@ def main():
 
         layerrecipes.delete()
         layermachines.delete()
+        layerdistros.delete()
         layerappends.delete()
         layerclasses.delete()
         for root, dirs, files in os.walk(oeclassicpath):
layerindex/update_layer.py +45 -1

100 lines not on this page · raw message

layerindex/urls.py +3 -0
index cbd29ad..e37db24 100644
@@ -21,6 +21,7 @@ router.register(r'layerBranches', restviews.LayerBranchViewSet)
 router.register(r'layerDependencies', restviews.LayerDependencyViewSet)
 router.register(r'recipes', restviews.RecipeViewSet)
 router.register(r'machines', restviews.MachineViewSet)
+router.register(r'distros', restviews.DistroViewSet)
 
 urlpatterns = patterns('',
     url(r'^$',
@@ -37,6 +38,8 @@ urlpatterns = patterns('',
         RedirectView.as_view(url=reverse_lazy('recipe_search', args=('master',)), permanent=False)),
     url(r'^machines/$',
         RedirectView.as_view(url=reverse_lazy('machine_search', args=('master',)), permanent=False)),
+    url(r'^distros/$',
+        RedirectView.as_view(url=reverse_lazy('distro_search', args=('master',)), permanent=False)),
  
     url(r'^submit/$', edit_layer_view, {'template_name': 'layerindex/submitlayer.html'}, name="submit_layer"),
     url(r'^submit/thanks$',
layerindex/urls_branch.py +5 -1
index 3313290..dbfb8a1 100644
@@ -7,7 +7,7 @@
 from django.conf.urls import *
 from django.views.defaults import page_not_found
 from django.core.urlresolvers import reverse_lazy
-from layerindex.views import LayerListView, RecipeSearchView, MachineSearchView, PlainTextListView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, RedirectParamsView, DuplicatesView
+from layerindex.views import LayerListView, RecipeSearchView, MachineSearchView, DistroSearchView, PlainTextListView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, RedirectParamsView, DuplicatesView
 
 urlpatterns = patterns('',
     url(r'^$', 
@@ -28,6 +28,10 @@ urlpatterns = patterns('',
         MachineSearchView.as_view(
             template_name='layerindex/machines.html'),
             name='machine_search'),
+    url(r'^distros/$',
+        DistroSearchView.as_view(
+            template_name='layerindex/distros.html'),
+            name='distro_search'),
     url(r'^edit/(?P<slug>[-\w]+)/$', edit_layer_view, {'template_name': 'layerindex/editlayer.html'}, name="edit_layer"),
     url(r'^duplicates/$',
         DuplicatesView.as_view(
layerindex/views.py +28 -1
index 3f9525d..06045ae 100644
@@ -10,7 +10,7 @@ from django.http import HttpResponse, HttpResponseRedirect, HttpResponseForbidde
 from django.core.urlresolvers import reverse, reverse_lazy, resolve
 from django.core.exceptions import PermissionDenied
 from django.template import RequestContext
-from layerindex.models import Branch, LayerItem, LayerMaintainer, LayerBranch, LayerDependency, LayerNote, Recipe, Machine, BBClass, BBAppend, RecipeChange, RecipeChangeset, ClassicRecipe
+from layerindex.models import Branch, LayerItem, LayerMaintainer, LayerBranch, LayerDependency, LayerNote, Recipe, Machine, Distro, BBClass, BBAppend, RecipeChange, RecipeChangeset, ClassicRecipe
 from datetime import datetime
 from itertools import chain
 from django.views.generic import TemplateView, DetailView, ListView
@@ -326,6 +326,7 @@ class LayerDetailView(DetailView):
         if layerbranch:
             context['layerbranch'] = layerbranch
             context['machines'] = layerbranch.machine_set.order_by('name')
+            context['distros'] = layerbranch.distro_set.order_by('name')
             context['appends'] = layerbranch.bbappend_set.order_by('filename')
             context['classes'] = layerbranch.bbclass_set.order_by('name')
         context['url_branch'] = self.kwargs['branch']
@@ -596,6 +597,32 @@ class MachineSearchView(ListView):
         return context
 
 
+class DistroSearchView(ListView):
+    context_object_name = 'distro_list'
+    paginate_by = 50
+
+    def get_queryset(self):
+        _check_url_branch(self.kwargs)
+        query_string = self.request.GET.get('q', '')
+        init_qs = Distro.objects.filter(layerbranch__branch__name=self.kwargs['branch'])
+        if query_string.strip():
+            entry_query = simplesearch.get_query(query_string, ['name', 'description'])
+            return init_qs.filter(entry_query).order_by('name', 'layerbranch__layer')
+
+        if 'q' in self.request.GET:
+            return init_qs.order_by('name', 'layerbranch__layer')
+
+        # Be consistent with RecipeSearchView
+        return Distro.objects.none()
+
+    def get_context_data(self, **kwargs):
+        context = super(DistroSearchView, self).get_context_data(**kwargs)
+        context['search_keyword'] = self.request.GET.get('q', '')
+        context['url_branch'] = self.kwargs['branch']
+        context['this_url_name'] = resolve(self.request.path_info).url_name
+        return context
+
+
 class PlainTextListView(ListView):
     def render_to_response(self, context):
         "Returns a plain text response rendering of the template"
templates/layerindex/detail.html +23 -0
index d0d11c0..4bd7ecf 100644
@@ -168,6 +168,9 @@
             {% if classes.count > 0 %}
                 <li><a href="#classes" data-toggle="tab">Classes</a></li>
             {% endif %}
+            {% if distros.count > 0 %}
+                <li><a href="#distros" data-toggle="tab">Distros</a></li>
+            {% endif %}
         </ul>
 
         <div class="tab-content">
@@ -265,6 +268,26 @@
                     </table>
                 </div>
             {% endif %}
+            {% if distros.count > 0 %}
+                <div class="tab-pane" id="distros">
+                    <div class="navbar">
+                        <div class="navbar-inner">
+                                <a class="brand pull-left">{{ layeritem.name }} distros</a>
+                        </div>
+                    </div>
+
+                    <table class="table table-bordered">
+                        <tbody>
+                            {% for distro in distros %}
+                                <tr>
+                                    <td><a href="{{ distro.vcs_web_url }}">{{ distro.name }}</a></td>
+                                    <td>{{ distro.description }}</td>
+                                </tr>
+                            {% endfor %}
+                        </tbody>
+                    </table>
+                </div>
+            {% endif %}
         </div>
  
 
templates/layerindex/distros.html new file +76 -0

76 lines not on this page · raw message

templates/layerindex/layers.html +1 -0
index cfc4ebd..b11ff2f 100644
@@ -21,6 +21,7 @@
                             <li class="active"><a href="{% url 'layer_list' url_branch %}">Layers</a></li>
                             <li><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li>
                             <li><a href="{% url 'machine_search' url_branch %}">Machines</a></li>
+                            <li><a href="{% url 'distro_search' url_branch %}">Distros</a></li>
 {% endautoescape %}
 {% endblock %}
 
templates/layerindex/machines.html +1 -0
index e31433c..2a9f947 100644
@@ -20,6 +20,7 @@
                             <li><a href="{% url 'layer_list' url_branch %}">Layers</a></li>
                             <li><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li>
                             <li class="active"><a href="{% url 'machine_search' url_branch %}">Machines</a></li>
+                            <li><a href="{% url 'distro_search' url_branch %}">Distros</a></li>
 {% endautoescape %}
 {% endblock %}
 
templates/layerindex/recipes.html +1 -0
index 74f3bb4..60a2667 100644
@@ -20,6 +20,7 @@
                             <li><a href="{% url 'layer_list' url_branch %}">Layers</a></li>
                             <li class="active"><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li>
                             <li><a href="{% url 'machine_search' url_branch %}">Machines</a></li>
+                            <li><a href="{% url 'distro_search' url_branch %}">Distros</a></li>
 {% endautoescape %}
 {% endblock %}
 
Signature · 1 line
1.9.1
#6 lore raw

Re: [layerindex-web][PATCH v2 05/12] layerindex: Add distro to web interface and model.

This looks good, just a few notes about this bit:

layerindex/update_layer.py def update_machine_conf_file(path, machine): · line 127

Quoted 10 lines · On Fri, 07 Oct 2016 11:57:14 Liam R. Howlett wrote:
layerindex/update_layer.py +10 -0
+def update_distro_conf_file(path, distro):
+    logger.debug('Updating distro %s' % path)
+    desc = ""
+    with open(path, 'r') as f:
+        for line in f:
+            if line.startswith('#@NAME:'):
+                desc = line[7:].strip()
+            if line.startswith('#@DESCRIPTION:'):
+                desc = line[14:].strip()
+                desc = re.sub(r'Distribtuion configuration for( running)*( an)*( the)*', '', desc) 

Aside from the typo, that last line isn't needed for distros.

Also, unlike machines, the readable distro name actually gets set in a proper variable (DISTRO_NAME) and though some distro conf files do still use the meta-comments, a lot don't use them at all. Technically being a variable, DISTRO_NAME could be set in an inc file rather than the distro conf file, and often it is, so we should really parse to get it rather than scraping it.

(I'm happy if we sort this out as a later improvement, I just wanted to make a note of it now.)

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#7 lore raw

Re: [layerindex-web][PATCH v2 05/12] layerindex: Add distro to web interface and model.

Quoted 16 lines · Paul Eggleton

This looks good, just a few notes about this bit:

layerindex/update_layer.py def update_machine_conf_file(path, machine): · line 127

Quoted 10 lines · On Fri, 07 Oct 2016 11:57:14 Liam R. Howlett wrote:
layerindex/update_layer.py +10 -0
+def update_distro_conf_file(path, distro):
+    logger.debug('Updating distro %s' % path)
+    desc = ""
+    with open(path, 'r') as f:
+        for line in f:
+            if line.startswith('#@NAME:'):
+                desc = line[7:].strip()
+            if line.startswith('#@DESCRIPTION:'):
+                desc = line[14:].strip()
+                desc = re.sub(r'Distribtuion configuration for( running)*( an)*( the)*', '', desc) 

Aside from the typo, that last line isn't needed for distros.

Quoted 8 lines

Also, unlike machines, the readable distro name actually gets set in a proper variable (DISTRO_NAME) and though some distro conf files do still use the meta-comments, a lot don't use them at all. Technically being a variable, DISTRO_NAME could be set in an inc file rather than the distro conf file, and often it is, so we should really parse to get it rather than scraping it.

(I'm happy if we sort this out as a later improvement, I just wanted to make a note of it now.)

I have fixed the typo in the current mhatle/django18 layer.

Quoted 3 lines

Cheers,
Paul

#8 lore raw

PATCH v2 6/12 layerindex/tools/import_project: Add import_project

In reply to Liam R. Howlett

import_project will scan through a project and find any layer and add it to the database by calling import_layer on each layer. This differs from import_layer as it tires to figure out the remote url and uses the subdirectory (if one exists) as the name.

---
 layerindex/tools/import_project.py | 180 +++++++++++++++++++++++++++++++++++++
 1 file changed, 180 insertions(+)
 create mode 100755 layerindex/tools/import_project.py
layerindex/tools/import_project.py new file +180 -0
new file mode 100755
index 0000000..511282d
@@ -0,0 +1,180 @@
+#!/usr/bin/python3
+
+# Import a project into the database.
+#  This will scan through the directories in a project and find any layer and
+#  call import_layer.
+#
+#
+# Copyright (C) 2016 Wind River Systems
+# Author: Liam R. Howlett <[email protected]>
+#
+# Licensed under the MIT license, see COPYING.MIT for details
+
+from git import Repo
+from urllib.parse import urlparse
+import logging
+import optparse
+import os, fnmatch
+import sys
+
+sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__), '..')))
+
+import import_layer
+import update
+
+import utils
+
+
+
+
+class ImportProject:
+    logger = utils.logger_create('ProjectIndexImport')
+
+    def find_layers(self, path):
+        self.logger.debug("finding layer..");
+        result = []
+        for root, dirs, files in os.walk(path, followlinks=True):
+            for name in fnmatch.filter(files, 'layer.conf'):
+                if not root.endswith('conf'):
+                    continue
+
+                self.logger.debug("Found %s" % root)
+                result.append(root)
+        return result
+
+
+    def main(self):
+        parser = optparse.OptionParser(
+            usage = """
+            %prog [options] [directory]""")
+
+        parser.add_option("-d", "--debug",
+            help = "Enable debug output",
+            action="store_const", const=logging.DEBUG, dest="loglevel", default=logging.INFO)
+        parser.add_option("-n", "--dry-run",
+            help = "Don't write any data back to the database",
+            action="store_true", dest="dryrun")
+
+        self.options, args = parser.parse_args(sys.argv)
+
+        self.logger.setLevel(self.options.loglevel)
+
+        if len(args) == 1:
+            print("Please provide a directory.");
+            sys.exit(1)
+
+        install_dir = args[1]
+        lc_list = self.find_layers(install_dir)
+        core_layer = self.add_core(lc_list)
+        if core_layer:
+            lc_list.remove(core_layer)
+
+
+
+        for layer in lc_list:
+            self.add_layer(layer)
+
+    def add_layer(self, layer):
+        self.logger.debug("Processing layer %s" % layer);
+        try:
+            git_dir = utils.runcmd("git rev-parse --show-toplevel", destdir=layer, logger=self.logger)
+        except Exception as e:
+            self.logger.error("Cannot get root dir for layer %s: %s - Skipping." % (layer, str(e)))
+            return 1
+
+        repo = Repo(git_dir)
+        actual_branch = repo.active_branch.name
+
+
+        layer_name = layer.split('/')[-2]
+
+        layer_subdir = None
+        if os.path.basename(git_dir) != layer_name:
+            layer_subdir = layer_name
+
+        layer_name = self.get_layer_name(layer)
+
+        for i in [1, 2, 3]:
+            try:
+                git_url = utils.runcmd("git config --get remote.origin.url", destdir=git_dir, logger=self.logger)
+            except Exception as e:
+                self.logger.info("Cannot get remote.origin.url for git dir %s: %s" % (git_dir, str(e)))
+
+            if not os.path.exists(git_url):
+                # Assume this is remote.
+                self.logger.debug("Found git url = %s" % git_url)
+                break;
+            self.logger.debug("Iterating to find git url into %s" % git_dir)
+            git_dir = git_url
+
+        cmd = ['import_layer.py']
+        if self.options.loglevel == logging.DEBUG:
+            cmd.append("-d")
+        if layer_subdir:
+            cmd.append("-s")
+            cmd.append(layer_subdir)
+
+        if actual_branch:
+            cmd.append("-a")
+            cmd.append(actual_branch)
+        cmd.append(git_url)
+        cmd.append(layer_name)
+        prefix = "Calling"
+
+        if self.options.dryrun:
+            prefix = "Would Call"
+
+
+        self.logger.info("%s import_layer.main with %s for dir %s" % (prefix, str(cmd), layer))
+        sys.argv = cmd
+        if not self.options.dryrun:
+            try:
+                import_layer.main();
+            except SystemExit as see:
+                return see.code
+        return 0
+
+    def get_layer_name(self, layerconfdir):
+        layer_name = layerconfdir.split('/')[-2]
+        self.logger.debug('getting layer %s' % layerconfdir)
+        layer_conf = os.path.join(layerconfdir, 'layer.conf')
+        if os.path.isfile(layer_conf):
+            with open(layer_conf) as conf:
+                for line in conf:
+                    if 'BBLAYERS_LAYERINDEX_NAME' in line:
+                        layer_name = line.split('=')[1].strip(' "\n')
+        return layer_name
+
+    def add_core(self, layers):
+        utils.setup_django()
+        core = None
+        import settings
+        for layer in layers:
+            layer_name = self.get_layer_name(layer)
+            if layer_name == settings.CORE_LAYER_NAME:
+                if self.add_layer(layer):
+                    self.logger.info('Failed to add core layer\n')
+                core = layer
+                self.update()
+                break
+        return core
+
+    def update(self):
+        update_py = os.path.realpath(os.path.join(os.path.dirname(__file__), '../update.py'))
+        cmd = [update_py]
+        if self.options.loglevel == logging.DEBUG:
+            cmd.append("-d")
+        sys.argv = cmd
+        self.logger.info("update")
+        if not self.options.dryrun:
+            try:
+                update.main();
+            except SystemExit:
+                return 1
+
+        return 0
+
+
+if __name__ == "__main__":
+    x = ImportProject()
+    x.main()
Signature · 1 line
1.9.1
#9 lore raw

PATCH v2 7/12 layerindex/recipeparse.py: refactor setup_tinfoil, checkout_layer_branch, parse_layer_conf to utils.py

In reply to Liam R. Howlett

Move functions to utils to be used by other classes. _

---
 layerindex/bulkchange.py  |  4 ++--
 layerindex/recipeparse.py | 39 +++-----------------------------------
 layerindex/utils.py       | 48 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 53 insertions(+), 38 deletions(-)
layerindex/bulkchange.py +2 -2
index 8102571..ac9079f 100644
@@ -42,9 +42,9 @@ def generate_patches(tinfoil, fetchdir, changeset, outputdir):
                     patchname = "%s.patch" % layer.name
                     patches.append(patchname)
                     layerfetchdir = os.path.join(fetchdir, layer.get_fetch_dir())
-                    recipeparse.checkout_layer_branch(layerbranch, layerfetchdir)
+                    utils.checkout_layer_branch(layerbranch, layerfetchdir)
                     layerdir = os.path.join(layerfetchdir, layerbranch.vcs_subdir)
-                    config_data_copy = recipeparse.setup_layer(tinfoil.config_data, fetchdir, layerdir, layer, layerbranch)
+                    config_data_copy = utils.setup_layer(tinfoil.config_data, fetchdir, layerdir, layer, layerbranch)
                     if outfile:
                         outfile.close()
                     outfile = open(os.path.join(tmpoutdir, patchname), 'w')
layerindex/recipeparse.py +3 -36
index a9cecfc..91a083b 100644
@@ -20,31 +20,6 @@ class RecipeParseError(Exception):
     def __str__(self):
         return self.msg
 
-def _setup_tinfoil(bitbakepath, enable_tracking):
-    sys.path.insert(0, bitbakepath + '/lib')
-    import bb.tinfoil
-    import bb.cooker
-    import bb.data
-    try:
-        tinfoil = bb.tinfoil.Tinfoil(tracking=enable_tracking)
-    except TypeError:
-        # old API
-        tinfoil = bb.tinfoil.Tinfoil()
-        if enable_tracking:
-            tinfoil.cooker.enableDataTracking()
-    tinfoil.prepare(config_only = True)
-
-    return tinfoil
-
-def _parse_layer_conf(layerdir, data):
-    data.setVar('LAYERDIR', str(layerdir))
-    if hasattr(bb, "cookerdata"):
-        # Newer BitBake
-        data = bb.cookerdata.parse_config_file(os.path.join(layerdir, "conf", "layer.conf"), data)
-    else:
-        # Older BitBake (1.18 and below)
-        data = bb.cooker._parse(os.path.join(layerdir, "conf", "layer.conf"), data)
-    data.expandVarref('LAYERDIR')
 
 
 def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout=False, classic=False, logger=None):
@@ -97,7 +72,7 @@ def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout
     tempdir = tempfile.mkdtemp(dir=settings.TEMP_BASE_DIR)
     os.chdir(tempdir)
 
-    tinfoil = _setup_tinfoil(bitbakepath, enable_tracking)
+    tinfoil = utils.setup_tinfoil(bitbakepath, enable_tracking)
 
     # Ensure TMPDIR exists (or insane.bbclass will blow up trying to write to the QA log)
     oe_tmpdir = tinfoil.config_data.getVar('TMPDIR', True)
@@ -110,14 +85,6 @@ def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout
 
     return (tinfoil, tempdir)
 
-def checkout_layer_branch(layerbranch, repodir, logger=None):
-    if layerbranch.actual_branch:
-        branchname = layerbranch.actual_branch
-    else:
-        branchname = layerbranch.branch.name
-    out = utils.runcmd("git checkout origin/%s" % branchname, repodir, logger=logger)
-    out = utils.runcmd("git clean -f -x", repodir, logger=logger)
-
 def setup_layer(config_data, fetchdir, layerdir, layer, layerbranch):
     # Parse layer.conf files for this layer and its dependencies
     # This is necessary not just because BBPATH needs to be set in order
@@ -125,7 +92,7 @@ def setup_layer(config_data, fetchdir, layerdir, layer, layerbranch):
     # or across layers, but also because custom variable values might be
     # set in layer.conf.
     config_data_copy = bb.data.createCopy(config_data)
-    _parse_layer_conf(layerdir, config_data_copy)
+    utils.parse_layer_conf(layerdir, config_data_copy)
     for dep in layerbranch.dependencies_set.all():
         depurldir = dep.dependency.get_fetch_dir()
         deprepodir = os.path.join(fetchdir, depurldir)
@@ -133,7 +100,7 @@ def setup_layer(config_data, fetchdir, layerdir, layer, layerbranch):
         if not deplayerbranch:
             raise RecipeParseError('Dependency %s of layer %s does not have branch record for branch %s' % (dep.dependency.name, layer.name, layerbranch.branch.name))
         deplayerdir = os.path.join(deprepodir, deplayerbranch.vcs_subdir)
-        _parse_layer_conf(deplayerdir, config_data_copy)
+        utils.parse_layer_conf(deplayerdir, config_data_copy)
     config_data_copy.delVar('LAYERDIR')
     return config_data_copy
 
layerindex/utils.py +48 -0

54 lines not on this page · raw message

Signature · 1 line
1.9.1
#10 lore raw

PATCH v2 8/12 layerindex: Detect dependencies from layer.conf files

In reply to Liam R. Howlett

Read dependencies from layer.conf and try to create the LayerDependency entry by looking up the correct database object. Dependencies are found by layer name only - no collection support. layer.conf parsing is handled by the bitbake code.

Once all layers are added, the dependencies have to be rechecked in case the layers are not added in order.

---
 layerindex/layerconfparse.py     | 50 ++++++++++++++++++++++++
 layerindex/tools/import_layer.py | 11 ++++++
 layerindex/update.py             | 41 +++++++++++++++++++-
 layerindex/update_layer.py       | 20 ++++++----
 layerindex/utils.py              | 82 ++++++++++++++++++++++++++++++++++++++++
 5 files changed, 196 insertions(+), 8 deletions(-)
 create mode 100644 layerindex/layerconfparse.py
layerindex/layerconfparse.py new file +50 -0
new file mode 100644
index 0000000..3a453bd
@@ -0,0 +1,50 @@
+# Utility functions for parsing layer.conf using bitbake within layerindex-web
+#
+# Copyright (C) 2016 Wind River Systems
+# Author: Liam R. Howlett <[email protected]>
+#
+# Licensed under the MIT license, see COPYING.MIT for details
+#
+
+import sys
+import os
+import os.path
+import utils
+import tempfile
+import re
+
+class LayerConfParse:
+    def __init__(self, enable_tracking=False, logger=None, bitbakepath=None, tinfoil=None):
+        import settings
+        self.logger = logger
+
+        if not bitbakepath:
+            fetchdir = settings.LAYER_FETCH_DIR
+            bitbakepath = os.path.join(fetchdir, 'bitbake')
+        self.bbpath = bitbakepath
+
+        # Set up BBPATH.
+        os.environ['BBPATH'] = str("%s" % self.bbpath)
+        self.tinfoil = tinfoil
+
+        if not self.tinfoil:
+            self.tinfoil = utils.setup_tinfoil(self.bbpath, enable_tracking)
+
+        self.config_data_copy = bb.data.createCopy(self.tinfoil.config_data)
+
+    def parse_layer(self, layerbranch, layerdir):
+
+        utils.checkout_layer_branch(layerbranch, layerdir, self.logger)
+
+
+        # This is not a valid layer, parsing will cause exception.
+        if not utils.is_layer_valid(layerdir):
+            return None
+
+        utils.parse_layer_conf(layerdir, self.config_data_copy, logger=self.logger)
+        return self.config_data_copy
+
+    def shutdown(self):
+        self.tinfoil.shutdown()
+
+
layerindex/tools/import_layer.py +11 -0
index 184c5cc..1daaeb2 100755
@@ -19,6 +19,7 @@ import glob
 import utils
 import logging
 import subprocess
+from layerconfparse import LayerConfParse
 
 class DryRunRollbackException(Exception):
     pass
@@ -367,11 +368,21 @@ def main():
                 if layer.name != settings.CORE_LAYER_NAME:
                     if not core_layer:
                         core_layer = utils.get_layer(settings.CORE_LAYER_NAME)
+
                     if core_layer:
+                        logger.debug('Adding dep %s to %s' % (core_layer.name, layer.name))
                         layerdep = LayerDependency()
                         layerdep.layerbranch = layerbranch
                         layerdep.dependency = core_layer
                         layerdep.save()
+                    try:
+                        layerconfparser = LayerConfParse(logger=logger)
+                        config_data = layerconfparser.parse_layer(layerbranch, layerdir)
+                    finally:
+                        layerconfparser.shutdown()
+                    if config_data:
+                        utils.add_dependencies(layerbranch, config_data, logger=logger)
+
 
                 # Get some extra meta-information
                 readme_files = glob.glob(os.path.join(layerdir, 'README*'))
layerindex/update.py +40 -1

71 lines not on this page · raw message

layerindex/update_layer.py +13 -7
index 3f50547..addeb4c 100644
@@ -196,7 +196,6 @@ def main():
     except recipeparse.RecipeParseError as e:
         logger.error(str(e))
         sys.exit(1)
-
     # Clear the default value of SUMMARY so that we can use DESCRIPTION instead if it hasn't been set
     tinfoil.config_data.setVar('SUMMARY', '')
     # Clear the default value of DESCRIPTION so that we can see where it's not set
@@ -244,7 +243,7 @@ def main():
                 layerbranch = LayerBranch()
                 layerbranch.layer = layer
                 layerbranch.branch = branch
-                layerbranch_source = layer.get_layerbranch('master')
+                layerbranch_source = layer.get_layerbranch(branch)
                 if not layerbranch_source:
                     layerbranch_source = layer.get_layerbranch(None)
                 if layerbranch_source:
@@ -256,11 +255,6 @@ def main():
                         maintainer.id = None
                         maintainer.layerbranch = layerbranch
                         maintainer.save()
-                    for dep in layerbranch_source.dependencies_set.all():
-                        dep.pk = None
-                        dep.id = None
-                        dep.layerbranch = layerbranch
-                        dep.save()
 
             if layerbranch.vcs_subdir and not options.nocheckout:
                 # Find latest commit in subdirectory
@@ -280,6 +274,17 @@ def main():
 
             layerdir = os.path.join(repodir, layerbranch.vcs_subdir)
             layerdir_start = os.path.normpath(layerdir) + os.sep
+
+            from layerconfparse import LayerConfParse
+            layerconfparser = LayerConfParse(logger=logger, tinfoil=tinfoil)
+            layer_config_data = layerconfparser.parse_layer(layerbranch, layerdir)
+            if not layer_config_data:
+                logger.info("Skipping update of layer %s for branch %s - conf/layer.conf may have parse issues" % (layer.name, branchdesc))
+                layerconfparser.shutdown()
+                sys.exit(1)
+            utils.add_dependencies(layerbranch, layer_config_data, logger=logger)
+            layerbranch.save()
+
             layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
             layermachines = Machine.objects.filter(layerbranch=layerbranch)
             layerdistros = Distro.objects.filter(layerbranch=layerbranch)
@@ -655,6 +660,7 @@ def main():
         import traceback
         traceback.print_exc()
 
+    tinfoil.shutdown()
     shutil.rmtree(tempdir)
     sys.exit(0)
 
layerindex/utils.py +82 -0

88 lines not on this page · raw message

Signature · 1 line
1.9.1
#11 lore raw

PATCH v2 9/12 layerindex: Add collection and version to layerbranch

In reply to Liam R. Howlett

Collection and version will be pulled from the layer.conf if it exists and dependencies will be resolved by first checking for layers with the dependency name and then checking for collections. It is necessary to shutdown tinfoil to avoid bitbake complaining about multiple instances.

---
 layerindex/models.py       |  2 ++
 layerindex/update_layer.py |  1 +
 layerindex/utils.py        | 20 ++++++++++++++++++--
 3 files changed, 21 insertions(+), 2 deletions(-)
layerindex/models.py +2 -0
index 2343ba7..cbfb244 100644
@@ -130,6 +130,8 @@ class LayerItem(models.Model):
 class LayerBranch(models.Model):
     layer = models.ForeignKey(LayerItem)
     branch = models.ForeignKey(Branch)
+    collection = models.CharField('Layer Collection', max_length=40, null=True, help_text='Name of the layer that could be used in the list of dependencies - can only contain letters, numbers and dashes')
+    version = models.CharField('Layer Version', max_length=10, null=True, blank=True, help_text='The layer version for this particular branch.')
     vcs_subdir = models.CharField('Repository subdirectory', max_length=40, blank=True, help_text='Subdirectory within the repository where the layer is located, if not in the root (usually only used if the repository contains more than one layer)')
     vcs_last_fetch = models.DateTimeField('Last successful fetch', blank=True, null=True)
     vcs_last_rev = models.CharField('Last revision fetched', max_length=80, blank=True)
layerindex/update_layer.py +1 -0
index addeb4c..40d1f16 100644
@@ -283,6 +283,7 @@ def main():
                 layerconfparser.shutdown()
                 sys.exit(1)
             utils.add_dependencies(layerbranch, layer_config_data, logger=logger)
+            utils.set_layerbranch_collection_version(layerbranch, layer_config_data, logger=logger)
             layerbranch.save()
 
             layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
layerindex/utils.py +18 -2
index 018478e..170d202 100644
@@ -30,8 +30,10 @@ def get_layer(layername):
 def get_dependency_layer(depname, version_str=None, logger=None):
     from layerindex.models import LayerItem, LayerBranch
 
-    # Get any LayerBranch with a layer that has a name that matches the depname
-    res = list(LayerBranch.objects.filter(layer__name=depname))
+    # Get any LayerBranch with a layer that has a name that matches depmod, or
+    # a LayerBranch that has the collection name depmod.
+    res = list(LayerBranch.objects.filter(layer__name=depname)) + \
+          list(LayerBranch.objects.filter(collection=depname))
 
     # Nothing found, return.
     if not res:
@@ -68,6 +70,10 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None):
     layer_name = layerbranch.layer.name
     var_name = layer_name
 
+    if layerbranch.collection:
+        var_name = layerbranch.collection
+
+
     dep_list = config_data.getVar("%s_%s" % (var, var_name), True)
 
     if not dep_list:
@@ -104,11 +110,21 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None):
 
         if logger:
             logger.debug('Adding %s %s to %s' % (name, dep_layer.name, layer_name))
+
         layerdep = LayerDependency()
         layerdep.layerbranch = layerbranch
         layerdep.dependency = dep_layer
         layerdep.save()
 
+def set_layerbranch_collection_version(layerbranch, config_data, logger=None):
+
+            layerbranch.collection = config_data.getVar('BBFILE_COLLECTIONS', True)
+            ver_str = "LAYERVERSION_"
+            if layerbranch.collection:
+                layerbranch.collection = layerbranch.collection.strip()
+                ver_str += layerbranch.collection
+                layerbranch.version = config_data.getVar(ver_str, True)
+
 def setup_tinfoil(bitbakepath, enable_tracking):
     sys.path.insert(0, bitbakepath + '/lib')
     import bb.tinfoil
Signature · 1 line
1.9.1
#12 lore raw

Re: [layerindex-web][PATCH v2 09/12] layerindex: Add collection and version to layerbranch

Quoted 4 lines · On Fri, 07 Oct 2016 11:57:18 Liam R. Howlett wrote:

Collection and version will be pulled from the layer.conf if it exists and dependencies will be resolved by first checking for layers with the dependency name and then checking for collections. It is necessary to shutdown tinfoil to avoid bitbake complaining about multiple instances.

That last bit of the commit message is now invalid in v2.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#13 lore raw

Re: [layerindex-web][PATCH v2 09/12] layerindex: Add collection and version to layerbranch

Quoted 7 lines · Paul Eggleton
Quoted 4 lines · On Fri, 07 Oct 2016 11:57:18 Liam R. Howlett wrote:

Collection and version will be pulled from the layer.conf if it exists and dependencies will be resolved by first checking for layers with the dependency name and then checking for collections. It is necessary to shutdown tinfoil to avoid bitbake complaining about multiple instances.

That last bit of the commit message is now invalid in v2.

I have removed the errant comment (last line) from the commit message in the current 'mhatle/django18' branch.

Quoted 3 lines

Cheers,
Paul

#14 lore raw

PATCH v2 10/12 layerindexer: Add layer recommends support

In reply to Liam R. Howlett

Parse layer.conf and add dependencies that are not required from LAYERRECOMMENDS_<name>. Update the layerindex/template to support recommends. Uses bitbake parsing code & checks versions.

---
 layerindex/models.py             |  7 +++++++
 layerindex/tools/import_layer.py |  2 +-
 layerindex/update.py             |  1 +
 layerindex/update_layer.py       |  1 +
 layerindex/utils.py              |  9 +++++++--
 templates/layerindex/detail.html | 32 +++++++++++++++++++++-----------
 6 files changed, 38 insertions(+), 14 deletions(-)
layerindex/models.py +7 -0
index cbfb244..869d4a3 100644
@@ -211,6 +211,12 @@ class LayerBranch(models.Model):
         return "%s: %s" % (self.layer.name, self.branch.name)
 
 
+    def get_required(self):
+        return self.dependencies_set.filter(required=True)
+
+    def get_recommends(self):
+        return self.dependencies_set.filter(required=False)
+
 class LayerMaintainer(models.Model):
     MAINTAINER_STATUS_CHOICES = (
         ('A', 'Active'),
@@ -232,6 +238,7 @@ class LayerMaintainer(models.Model):
 class LayerDependency(models.Model):
     layerbranch = models.ForeignKey(LayerBranch, related_name='dependencies_set')
     dependency = models.ForeignKey(LayerItem, related_name='dependents_set')
+    required = models.BooleanField(default=True)
 
     class Meta:
         verbose_name_plural = "Layer dependencies"
layerindex/tools/import_layer.py +1 -1
index 1daaeb2..377a0f0 100755
@@ -382,7 +382,7 @@ def main():
                         layerconfparser.shutdown()
                     if config_data:
                         utils.add_dependencies(layerbranch, config_data, logger=logger)
-
+                        utils.add_recommends(layerbranch, config_data, logger=logger)
 
                 # Get some extra meta-information
                 readme_files = glob.glob(os.path.join(layerdir, 'README*'))
layerindex/update.py +1 -0
index 8684d36..6f93099 100755
@@ -235,6 +235,7 @@ def main():
                         continue
 
                     utils.add_dependencies(layerbranch, config_data, logger=logger)
+                    utils.add_recommends(layerbranch, config_data, logger=logger)
             finally:
                 layerconfparser.shutdown()
 
layerindex/update_layer.py +1 -0
index 40d1f16..27d4ebe 100644
@@ -283,6 +283,7 @@ def main():
                 layerconfparser.shutdown()
                 sys.exit(1)
             utils.add_dependencies(layerbranch, layer_config_data, logger=logger)
+            utils.add_recommends(layerbranch, layer_config_data, logger=logger)
             utils.set_layerbranch_collection_version(layerbranch, layer_config_data, logger=logger)
             layerbranch.save()
 
layerindex/utils.py +7 -2
index 170d202..484d3fe 100644
@@ -62,9 +62,12 @@ def get_dependency_layer(depname, version_str=None, logger=None):
     return None
 
 def add_dependencies(layerbranch, config_data, logger=None):
-    _add_dependency("LAYERDEPENDS", 'dependency', layerbranch, config_data, logger)
+    _add_dependency("LAYERDEPENDS", 'dependency', layerbranch, config_data, logger=logger)
 
-def _add_dependency(var, name, layerbranch, config_data, logger=None):
+def add_recommends(layerbranch, config_data, logger=None):
+    _add_dependency("LAYERRECOMMENDS", 'recommends', layerbranch, config_data, logger=logger, required=False)
+
+def _add_dependency(var, name, layerbranch, config_data, logger=None, required=True):
     from layerindex.models import LayerBranch, LayerDependency
 
     layer_name = layerbranch.layer.name
@@ -97,6 +100,7 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None):
                 logger.error('Error getting %s %s for %s\n%s' %(name, dep. layer_name, str(vse)))
             continue
 
+        # No layer found.
         if not dep_layer:
             if logger:
                 logger.error('Cannot resolve %s %s (version %s) for %s' % (name, dep, ver_str, layer_name))
@@ -114,6 +118,7 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None):
         layerdep = LayerDependency()
         layerdep.layerbranch = layerbranch
         layerdep.dependency = dep_layer
+        layerdep.required = required
         layerdep.save()
 
 def set_layerbranch_collection_version(layerbranch, config_data, logger=None):
templates/layerindex/detail.html +21 -11
index 4bd7ecf..9d3ee05 100644
@@ -139,17 +139,27 @@
 
                 <div class="span4 pull-right description">
                     {% if layerbranch.dependencies_set.count > 0 %}
-                         <div class="well dependency-well">
-                            <h3>Dependencies </h3>
-                            <p>The {{ layeritem.name }} layer depends upon:</p>
-                            <ul>
-                                {% for dep in layerbranch.dependencies_set.all %}
-                                    <li><a href="{% url 'layer_item' url_branch dep.dependency.name %}">{{ dep.dependency.name }}</a></li>
-                                {% endfor %}
-                            </ul>
-                         </div> <!-- end of well -->
-                    {% endif %}
-
+                        <div class="well dependency-well">
+                            {% if layerbranch.get_required.count > 0 %}
+                                <h3>Dependencies </h3>
+                                <p>The {{ layeritem.name }} layer depends upon:</p>
+                                <ul>
+                                    {% for dep in layerbranch.get_required %}
+                                        <li><a href="{% url 'layer_item' url_branch dep.dependency.name %}">{{ dep.dependency.name }}</a></li>
+                                    {% endfor %}
+                                </ul>
+                            {% endif %} <!-- end of layerbranch.get_required.count -->
+                            {% if layerbranch.get_recommends.count > 0 %}
+                                <h3>Recommends </h3>
+                                <p>The {{ layeritem.name }} layer recommends:</p>
+                                <ul>
+                                    {% for rec in layerbranch.get_recommends %}
+                                        <li><a href="{% url 'layer_item' url_branch rec.dependency.name %}">{{ rec.dependency.name }}</a></li>
+                                    {% endfor %}
+                                </ul>
+                            {% endif %} <!-- end of layerbranch.get_recommends.count -->
+                        </div> <!-- end of well -->
+                    {% endif %} <!-- end of layerbranch.dependencies_set.count -->
                 </div> <!-- end of span4 -->
             </div>  <!-- end of row-fluid -->
         </div> <!-- end of container-fluid -->
Signature · 1 line
1.9.1
#15 lore raw

PATCH v2 11/12 recipeparse: remove unnecessary else statement.

In reply to Liam R. Howlett

Code clean up.

---
 layerindex/recipeparse.py | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
layerindex/recipeparse.py +8 -9
index 91a083b..dd85bc3 100644
@@ -134,15 +134,14 @@ def detect_file_type(path, subdir_start):
         if res:
             typename = 'machine'
             return (typename, None, res.group(1))
-        else:
-            res = bbclass_re.match(subpath)
-            if res:
-                typename = 'bbclass'
-                return (typename, None, res.group(1))
-            res = distro_conf_re.match(subpath)
-            if res:
-                typename = 'distro'
-                return (typename, None, res.group(1))
+        res = bbclass_re.match(subpath)
+        if res:
+            typename = 'bbclass'
+            return (typename, None, res.group(1))
+        res = distro_conf_re.match(subpath)
+        if res:
+            typename = 'distro'
+            return (typename, None, res.group(1))
 
     if typename == 'recipe' or typename == 'bbappend':
         if subdir_start:
Signature · 1 line
1.9.1
#16 lore raw

PATCH v2 12/12 layerindex/update_layer.py: Preserve the recipedependency files

In reply to Liam R. Howlett

From: Mark Hatle <[email protected]>

In order to keep primary keys from constantly changing, preverse the exist keys as much as possible.

---
 layerindex/update_layer.py | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
layerindex/update_layer.py +18 -1
index 27d4ebe..ec40806 100644
@@ -88,13 +88,30 @@ def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir):
             if depstr.startswith(layerdir_start) and not depstr.endswith('/conf/layer.conf'):
                 filedeps.append(os.path.relpath(depstr, repodir))
         from layerindex.models import RecipeFileDependency
-        RecipeFileDependency.objects.filter(recipe=recipe).delete()
+
+        recipedeps_delete = []
+
+        recipedeps = RecipeFileDependency.objects.filter(recipe=recipe)
+
+        for values in recipedeps.values('path'):
+            if 'path' in values:
+                recipedeps_delete.append(values['path'])
+
         for filedep in filedeps:
+            if filedep in recipedeps_delete:
+                recipedeps_delete.remove(filedep)
+                continue
+            # New item, add it...
             recipedep = RecipeFileDependency()
             recipedep.layerbranch = recipe.layerbranch
             recipedep.recipe = recipe
             recipedep.path = filedep
             recipedep.save()
+
+        for filedep in recipedeps_delete:
+            logger.debug('%s: removing %s' % (recipe.layerbranch, filedep))
+            recipedeps.filter(path=filedep).delete()
+
     except KeyboardInterrupt:
         raise
     except BaseException as e:
Signature · 1 line
1.9.1
#17 lore raw
In reply to Liam R. Howlett

FYI, I have made sure these are re-based on top of paule/django18 and pushed to:

git://git.yoctoproject.org/layerindex-web mhatle/django18

BTW for me upgrading from paule/django18 to this version. I only had to run 'manage.py syncdb'.

This created the new database and added the missing field for the recommend.

I then ran update.py --forcereload and everything showed up as expected.

--Mark

Quoted 52 lines · Liam R. Howlett

This set of patches adds a number of features to the layerindex code:

 -  Adds the ability to set the actual-branch to import_layer.
 -  Adds distro to the database and web interface.
 -  Adds collection & version information to layerbranch.
 -  Uses layer name and collection name to determine dependencies and

recommends.
- Adds import_project to simplify pulling all layers and openembedded-core into a layer index.

Liam R. Howlett (11):
import_layer: Add --actual-branch option layerindex/tools/import_layer.py: Sanitize layer name. layerindex/tools/import_layer.py: Avoid failing if there is any layer
to add.
layerindex/utils: Update runcmd to decode binary strings to strings.
layerindex: Add distro to web interface and model. layerindex/tools/import_project: Add import_project layerindex/recipeparse.py: refactor setup_tinfoil,
checkout_layer_branch, parse_layer_conf to utils.py
layerindex: Detect dependencies from layer.conf files
layerindex: Add collection and version to layerbranch
layerindexer: Add layer recommends support
recipeparse: remove unnecessary else statement.

Mark Hatle (1):
layerindex/update_layer.py: Preserve the recipedependency files

 layerindex/admin.py                |  11 +++
 layerindex/bulkchange.py           |   4 +-
 layerindex/layerconfparse.py       |  50 +++++++++++
 layerindex/models.py               |  23 +++++
 layerindex/recipeparse.py          |  53 +++--------
 layerindex/restviews.py            |  10 ++-
 layerindex/tools/import_classic.py |   2 +
 layerindex/tools/import_layer.py   |  47 +++++++++-
 layerindex/tools/import_project.py | 180 +++++++++++++++++++++++++++++++++++++
 layerindex/update.py               |  42 ++++++++-
 layerindex/update_layer.py         |  87 ++++++++++++++++--
 layerindex/urls.py                 |   3 +
 layerindex/urls_branch.py          |   6 +-
 layerindex/utils.py                | 159 +++++++++++++++++++++++++++++++-
 layerindex/views.py                |  29 +++++-
 templates/layerindex/detail.html   |  55 +++++++++---
 templates/layerindex/distros.html  |  76 ++++++++++++++++
 templates/layerindex/layers.html   |   1 +
 templates/layerindex/machines.html |   1 +
 templates/layerindex/recipes.html  |   1 +
 20 files changed, 767 insertions(+), 73 deletions(-)
 create mode 100644 layerindex/layerconfparse.py
 create mode 100755 layerindex/tools/import_project.py
 create mode 100644 templates/layerindex/distros.html
#18 lore raw
Quoted 9 lines · On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:

FYI, I have made sure these are re-based on top of paule/django18 and pushed to:

git://git.yoctoproject.org/layerindex-web mhatle/django18

BTW for me upgrading from paule/django18 to this version. I only had to run 'manage.py syncdb'.

This created the new database and added the missing field for the recommend.

I'm not sure how that worked there but it doesn't work here. If I take a database that works on paule/django18, run syncdb then open the admin interface and click on "Layer branches" I get:

no such column: layerindex_layerbranch.collection

I'm afraid we really do need to create migrations whenever we change the models.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#19 lore raw
Quoted 10 lines · Paul Eggleton
Quoted 7 lines · On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:

FYI, I have made sure these are re-based on top of paule/django18 and pushed to:

git://git.yoctoproject.org/layerindex-web mhatle/django18

Quoted 9 lines
Quoted 4 lines

BTW for me upgrading from paule/django18 to this version. I only had to run 'manage.py syncdb'.

This created the new database and added the missing field for the recommend.

I'm not sure how that worked there but it doesn't work here. If I take a database that works on paule/django18, run syncdb then open the admin interface and click on "Layer branches" I get:

We found a severe lack of instructions for creating migrations. There were instructions for adding and deleting full DBs but not changing them. We will need pointers or help to resolve this.

Quoted 1 line

no such column: layerindex_layerbranch.collection

I never got that, but I'm also running SQLite. May be the difference.

Quoted 2 lines

I'm afraid we really do need to create migrations whenever we change the models.

Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.

Quoted 7 lines

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#20 lore raw
Quoted 17 lines · Mark
Quoted 8 lines
Quoted 6 lines · Paul Eggleton

On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:

git://git.yoctoproject.org/layerindex-web mhatle/django18

Quoted 8 lines
Quoted 5 lines

BTW for me upgrading from paule/django18 to this version. I only had to run 'manage.py syncdb'.

This created the new database and added the missing field for the recommend.>

I'm not sure how that worked there but it doesn't work here. If I take a database that works on paule/django18, run syncdb then open the admin interface and click on "Layer branches" I get:

Quoted 7 lines

We found a severe lack of instructions for creating migrations. There were instructions for adding and deleting full DBs but not changing them. We will need pointers or help to resolve this.

Quoted 1 line

no such column: layerindex_layerbranch.collection

I never got that, but I'm also running SQLite. May be the difference.

I'm using sqlite here also.

Quoted 5 lines
Quoted 2 lines

I'm afraid we really do need to create migrations whenever we change the models.

Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.

I'm happy to help directly if needed, but did you come across this already?

https://docs.djangoproject.com/en/1.8/topics/migrations/

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#21 lore raw
Quoted 31 lines · Paul Eggleton
Quoted 17 lines · Mark
Quoted 8 lines
Quoted 6 lines · Paul Eggleton

On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:

git://git.yoctoproject.org/layerindex-web mhatle/django18

Quoted 8 lines
Quoted 5 lines

BTW for me upgrading from paule/django18 to this version. I only had to run 'manage.py syncdb'.

This created the new database and added the missing field for the recommend.>

I'm not sure how that worked there but it doesn't work here. If I take a database that works on paule/django18, run syncdb then open the admin interface and click on "Layer branches" I get:

Quoted 7 lines

We found a severe lack of instructions for creating migrations. There were instructions for adding and deleting full DBs but not changing them. We will need pointers or help to resolve this.

Quoted 1 line

no such column: layerindex_layerbranch.collection

I never got that, but I'm also running SQLite. May be the difference.

I'm using sqlite here also.

Quoted 3 lines
Quoted 2 lines

I'm afraid we really do need to create migrations whenever we change the models.

Quoted 6 lines
Quoted 2 lines

Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.

I'm happy to help directly if needed, but did you come across this already?

https://docs.djangoproject.com/en/1.8/topics/migrations/

Unfortunately that is the exactly document I was looking at and could not figure it out.

I can the various commands, makemigrations and related.. and the system simply told me "no migrations necessary".. and didn't generate any.

So we were never able to figure out what a migration should look like or how to generate it.

If you (or someone else) can help us figure out how to create a migration, I suspect after the first one --- future ones will be much easier to generate.

--Mark

Quoted 3 lines

Cheers,
Paul

#22 lore raw
Quoted 51 lines · On Mon, 10 Oct 2016 12:47:11 Mark Hatle wrote:
Quoted 36 lines · Paul Eggleton
Quoted 21 lines · Mark
Quoted 14 lines

On Oct 10, 2016, at 2:54 AM, Paul Eggleton
<[email protected]>

Quoted 10 lines · wrote:

On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:

git://git.yoctoproject.org/layerindex-web mhatle/django18

BTW for me upgrading from paule/django18 to this version. I only had to
run 'manage.py syncdb'.

Quoted 7 lines
Quoted 2 lines

This created the new database and added the missing field for the recommend.>

I'm not sure how that worked there but it doesn't work here. If I take a database that works on paule/django18, run syncdb then open the admin

interface and click on "Layer branches" I get:

Quoted 8 lines

We found a severe lack of instructions for creating migrations. There were
instructions for adding and deleting full DBs but not changing them. We will need pointers or help to resolve this.

Quoted 1 line

no such column: layerindex_layerbranch.collection

I never got that, but I'm also running SQLite. May be the difference.

I'm using sqlite here also.

Quoted 3 lines
Quoted 2 lines

I'm afraid we really do need to create migrations whenever we change the models.

Quoted 7 lines
Quoted 2 lines

Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.

I'm happy to help directly if needed, but did you come across this already?

https://docs.djangoproject.com/en/1.8/topics/migrations/

Unfortunately that is the exactly document I was looking at and could not figure it out.

I can the various commands, makemigrations and related.. and the system simply told me "no migrations necessary".. and didn't generate any.

Quoted 6 lines

So we were never able to figure out what a migration should look like or how to generate it.

If you (or someone else) can help us figure out how to create a migration, I suspect after the first one --- future ones will be much easier to generate.

OK, so here's a step-by-step procedure - I suspect you'll need to stash and unstash your config changes along the way as well:

1) Ensure you are using a database created at the place you want to start, i.e. in this case paule/django18.

2) Determine which commits make changes to the models. This is one easy way:

$ git log -p paule/django18..mhatle/django18 layerindex/models.py

3) We don't have an initial migration, so we need to create that first in a single commit before all of the others, so to do that we'll direct the rebase to edit the top commit on paule/django18 but instead of modifying that we insert a commit after it. We also want to edit each commit found in step 1 that made changes so we can add the migrations to them.

$ git checkout mhatle/django18
$ git rebase -i paule/django18^
(now set the command to "e" on the oldest commit, in this case 6f8201fe, and 
do the same for all the commits you found in step 1, then save & quit the 
editor)

4) Create the initial migration:

$ python3 manage.py makemigrations layerindex
Migrations for 'layerindex':
  0001_initial.py:
    - Create model BBAppend
    - Create model BBClass
    - Create model Branch
    - Create model LayerBranch
    - Create model LayerDependency
    - Create model LayerItem
    - Create model LayerMaintainer
    - Create model LayerNote
    - Create model Machine
    - Create model PythonEnvironment
    - Create model Recipe
    - Create model RecipeChange
    - Create model RecipeChangeset
    - Create model RecipeFileDependency
    - Create model ClassicRecipe
    - Add field recipe to recipefiledependency
    - Add field changeset to recipechange
    - Add field recipe to recipechange
    - Add field layerbranch to recipe
    - Add field dependency to layerdependency
    - Add field layerbranch to layerdependency
    - Add field layer to layerbranch
    - Add field update_environment to branch
    - Add field layerbranch to bbclass
    - Add field layerbranch to bbappend
    - Add field cover_layerbranch to classicrecipe

5) Apply the migrations to the database - this time we make it "fake" because the tables already exist:

$ python manage.py migrate layerindex --fake-initial

6) Create the initial migrations commit. (We won't need to do this ever again)

$ git add layerindex/migrations/ 
$ git commit

7) Continue the rebase:

$ git rebase --continue

8) Create the migration for the changes in this commit:

$ python manage.py makemigrations layerindex
Migrations for 'layerindex':
  0002_distro.py:
    - Create model Distro

9) Apply the migrations (*not* fake this time):

$ python manage.py migrate layerindex

10) Amend the migrations to the current commit:

$ git add layerindex/migrations/
$ git commit --amend

11) Continue the rebase:

$ git rebase --continue

12) Repeat steps 8-11 until the rebase finishes.

That should do it. It gets more complicated if you rename columns, or introduce columns that need initial data beyond a simple default, split models etc. but in this case when you're simply adding or removing columns it will take care of creating the migrations for you as long as you know the steps (and I admit it's a bit awkward if you've never done it before - I have the advantage of having done this already, not exactly with Django's migrations but with South and that was pretty similar).

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#23 lore raw
Quoted 153 lines · On Tue, 11 Oct 2016 09:19:15 Paul Eggleton wrote:
Quoted 54 lines · On Mon, 10 Oct 2016 12:47:11 Mark Hatle wrote:
Quoted 39 lines · Paul Eggleton
Quoted 28 lines · Mark
Quoted 14 lines

On Oct 10, 2016, at 2:54 AM, Paul Eggleton
<[email protected]>

Quoted 10 lines · wrote:

On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:

git://git.yoctoproject.org/layerindex-web mhatle/django18

BTW for me upgrading from paule/django18 to this version. I only had to
run 'manage.py syncdb'.

Quoted 8 lines
Quoted 2 lines

This created the new database and added the missing field for the recommend.>

I'm not sure how that worked there but it doesn't work here. If I take a
database that works on paule/django18, run syncdb then open the admin

interface and click on "Layer branches" I get:

We found a severe lack of instructions for creating migrations. There were
instructions for adding and deleting full DBs but not changing them. We
will need pointers or help to resolve this.

Quoted 3 lines
Quoted 1 line

no such column: layerindex_layerbranch.collection

I never got that, but I'm also running SQLite. May be the difference.

I'm using sqlite here also.

Quoted 4 lines
Quoted 3 lines

I'm afraid we really do need to create migrations whenever we change the
models.

Quoted 7 lines
Quoted 2 lines

Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.

I'm happy to help directly if needed, but did you come across this already?

https://docs.djangoproject.com/en/1.8/topics/migrations/

Unfortunately that is the exactly document I was looking at and could not figure it out.

I can the various commands, makemigrations and related.. and the system simply told me "no migrations necessary".. and didn't generate any.

Quoted 6 lines

So we were never able to figure out what a migration should look like or how to generate it.

If you (or someone else) can help us figure out how to create a migration, I suspect after the first one --- future ones will be much easier to generate.

OK, so here's a step-by-step procedure - I suspect you'll need to stash and unstash your config changes along the way as well:

1) Ensure you are using a database created at the place you want to start, i.e. in this case paule/django18.

2) Determine which commits make changes to the models. This is one easy way:

$ git log -p paule/django18..mhatle/django18 layerindex/models.py

3) We don't have an initial migration, so we need to create that first in a single commit before all of the others, so to do that we'll direct the rebase to edit the top commit on paule/django18 but instead of modifying that we insert a commit after it. We also want to edit each commit found in step 1 that made changes so we can add the migrations to them.

$ git checkout mhatle/django18
$ git rebase -i paule/django18^
(now set the command to "e" on the oldest commit, in this case 6f8201fe, and
do the same for all the commits you found in step 1, then save & quit the
editor)

4) Create the initial migration:

$ python3 manage.py makemigrations layerindex
Migrations for 'layerindex':
  0001_initial.py:
    - Create model BBAppend
    - Create model BBClass
    - Create model Branch
    - Create model LayerBranch
    - Create model LayerDependency
    - Create model LayerItem
    - Create model LayerMaintainer
    - Create model LayerNote
    - Create model Machine
    - Create model PythonEnvironment
    - Create model Recipe
    - Create model RecipeChange
    - Create model RecipeChangeset
    - Create model RecipeFileDependency
    - Create model ClassicRecipe
    - Add field recipe to recipefiledependency
    - Add field changeset to recipechange
    - Add field recipe to recipechange
    - Add field layerbranch to recipe
    - Add field dependency to layerdependency
    - Add field layerbranch to layerdependency
    - Add field layer to layerbranch
    - Add field update_environment to branch
    - Add field layerbranch to bbclass
    - Add field layerbranch to bbappend
    - Add field cover_layerbranch to classicrecipe

5) Apply the migrations to the database - this time we make it "fake" because the tables already exist:

$ python manage.py migrate layerindex --fake-initial

6) Create the initial migrations commit. (We won't need to do this ever again)

$ git add layerindex/migrations/
$ git commit

7) Continue the rebase:

$ git rebase --continue

8) Create the migration for the changes in this commit:

$ python manage.py makemigrations layerindex
Migrations for 'layerindex':
  0002_distro.py:
    - Create model Distro

9) Apply the migrations (*not* fake this time):

$ python manage.py migrate layerindex

10) Amend the migrations to the current commit:

$ git add layerindex/migrations/
$ git commit --amend

11) Continue the rebase:

$ git rebase --continue

12) Repeat steps 8-11 until the rebase finishes.

Quoted 7 lines

That should do it. It gets more complicated if you rename columns, or introduce columns that need initial data beyond a simple default, split models etc. but in this case when you're simply adding or removing columns it will take care of creating the migrations for you as long as you know the steps (and I admit it's a bit awkward if you've never done it before - I have the advantage of having done this already, not exactly with Django's migrations but with South and that was pretty similar).

Heh, I knew I would muck something up - where you see "python" in the above, read "python3". (In my virtualenv-based setup it doesn't make any difference, but that may not be true for yours.)

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#24 lore raw
Quoted 162 lines · Paul Eggleton
Quoted 153 lines · On Tue, 11 Oct 2016 09:19:15 Paul Eggleton wrote:
Quoted 54 lines · On Mon, 10 Oct 2016 12:47:11 Mark Hatle wrote:
Quoted 39 lines · Paul Eggleton
Quoted 28 lines · Mark
Quoted 14 lines

On Oct 10, 2016, at 2:54 AM, Paul Eggleton
<[email protected]>

Quoted 10 lines · wrote:

On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:

git://git.yoctoproject.org/layerindex-web mhatle/django18

BTW for me upgrading from paule/django18 to this version. I only had to
run 'manage.py syncdb'.

Quoted 8 lines
Quoted 2 lines

This created the new database and added the missing field for the recommend.>

I'm not sure how that worked there but it doesn't work here. If I take a
database that works on paule/django18, run syncdb then open the admin

interface and click on "Layer branches" I get:

We found a severe lack of instructions for creating migrations. There were
instructions for adding and deleting full DBs but not changing them. We
will need pointers or help to resolve this.

Quoted 3 lines
Quoted 1 line

no such column: layerindex_layerbranch.collection

I never got that, but I'm also running SQLite. May be the difference.

I'm using sqlite here also.

Quoted 4 lines
Quoted 3 lines

I'm afraid we really do need to create migrations whenever we change the
models.

Quoted 7 lines
Quoted 2 lines

Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.

I'm happy to help directly if needed, but did you come across this already?

https://docs.djangoproject.com/en/1.8/topics/migrations/

Unfortunately that is the exactly document I was looking at and could not figure it out.

I can the various commands, makemigrations and related.. and the system simply told me "no migrations necessary".. and didn't generate any.

Quoted 6 lines

So we were never able to figure out what a migration should look like or how to generate it.

If you (or someone else) can help us figure out how to create a migration, I suspect after the first one --- future ones will be much easier to generate.

OK, so here's a step-by-step procedure - I suspect you'll need to stash and unstash your config changes along the way as well:

1) Ensure you are using a database created at the place you want to start, i.e. in this case paule/django18.

2) Determine which commits make changes to the models. This is one easy way:

$ git log -p paule/django18..mhatle/django18 layerindex/models.py

3) We don't have an initial migration, so we need to create that first in a single commit before all of the others, so to do that we'll direct the rebase to edit the top commit on paule/django18 but instead of modifying that we insert a commit after it. We also want to edit each commit found in step 1 that made changes so we can add the migrations to them.

$ git checkout mhatle/django18
$ git rebase -i paule/django18^
(now set the command to "e" on the oldest commit, in this case 6f8201fe, and
do the same for all the commits you found in step 1, then save & quit the
editor)

4) Create the initial migration:

$ python3 manage.py makemigrations layerindex
Migrations for 'layerindex':
  0001_initial.py:
    - Create model BBAppend
    - Create model BBClass
    - Create model Branch
    - Create model LayerBranch
    - Create model LayerDependency
    - Create model LayerItem
    - Create model LayerMaintainer
    - Create model LayerNote
    - Create model Machine
    - Create model PythonEnvironment
    - Create model Recipe
    - Create model RecipeChange
    - Create model RecipeChangeset
    - Create model RecipeFileDependency
    - Create model ClassicRecipe
    - Add field recipe to recipefiledependency
    - Add field changeset to recipechange
    - Add field recipe to recipechange
    - Add field layerbranch to recipe
    - Add field dependency to layerdependency
    - Add field layerbranch to layerdependency
    - Add field layer to layerbranch
    - Add field update_environment to branch
    - Add field layerbranch to bbclass
    - Add field layerbranch to bbappend
    - Add field cover_layerbranch to classicrecipe

5) Apply the migrations to the database - this time we make it "fake" because the tables already exist:

$ python manage.py migrate layerindex --fake-initial

6) Create the initial migrations commit. (We won't need to do this ever again)

$ git add layerindex/migrations/
$ git commit

7) Continue the rebase:

$ git rebase --continue

8) Create the migration for the changes in this commit:

$ python manage.py makemigrations layerindex
Migrations for 'layerindex':
  0002_distro.py:
    - Create model Distro

9) Apply the migrations (*not* fake this time):

$ python manage.py migrate layerindex

10) Amend the migrations to the current commit:

$ git add layerindex/migrations/
$ git commit --amend

11) Continue the rebase:

$ git rebase --continue

12) Repeat steps 8-11 until the rebase finishes.

Quoted 7 lines

That should do it. It gets more complicated if you rename columns, or introduce columns that need initial data beyond a simple default, split models etc. but in this case when you're simply adding or removing columns it will take care of creating the migrations for you as long as you know the steps (and I admit it's a bit awkward if you've never done it before - I have the advantage of having done this already, not exactly with Django's migrations but with South and that was pretty similar).

Quoted 3 lines

Heh, I knew I would muck something up - where you see "python" in the above, read "python3". (In my virtualenv-based setup it doesn't make any difference, but that may not be true for yours.)

No problem. The steps above did work for me. Migrations have now been added to the mhatle/django18 branch.

I think what we were missing was the setup of the initial migration, step #4 above. Without doing that you get messages like no migrations necessary and such -- very confusing.

Thanks for the help, let me know if there is anything else that needs to be adjusted.

--Mark

Quoted 3 lines

Cheers,
Paul