Layer index fixes

PATCH In review Paul Eggleton 2 participants 5 messages Aug 30, 2016

Share & tools

Paul Eggleton

author

2016-08-30 21:05 UTC #0 lore raw

cover Layer index fixes

Some minor fixes for the recent python3 work in the layer index, but more importantly fix parsing in master for the multiconfig changes that recently went into bitbake.

NOTE: this depends on the bitbake patchset I just sent in order to work.

The following changes since commit 1d76228675e93add50dd5216c43e00f977e1aaa8:

Use functools.reduce instead of reduce (2016-07-04 09:51:21 +1200)

are available in the git repository at:

git://git.yoctoproject.org/layerindex-web paule/mc-fixes
http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/mc-fixes

Paul Eggleton (3):
Use python3 commands in README
Fix output decoding/reporting for bulk change patch download update_layer.py: fix up for bitbake API change

 README                             |  6 +++---
 layerindex/tools/import_classic.py | 11 +++++++----
 layerindex/update_layer.py         | 17 ++++++++++-------
 layerindex/views.py                |  3 +++
 4 files changed, 23 insertions(+), 14 deletions(-)

Signature · 1 line

2.5.5

Paul Eggleton

author

2016-08-30 21:05 UTC #1 lore raw

PATCH 1/3 Use python3 commands in README

If you're in a virtualenv it doesn't make a difference, but outside of one it will.

Signed-off-by Paul Eggleton <[email protected]>

---
 README | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

README +3 -3

index fa5c3b2..8437089 100644
@@ -47,15 +47,15 @@ Setup instructions:
 2. Run the following commands within the layerindex-web directory to
    initialise the database:
 
-   python manage.py syncdb
-   python manage.py migrate
+   python3 manage.py syncdb
+   python3 manage.py migrate
 
    You should answer "yes" when asked to create an admin account.
 
 3. You can test the web application locally by setting DEBUG = True in
    settings.py and running the following:
 
-   python manage.py runserver
+   python3 manage.py runserver
 
    Then visit http://127.0.0.1:8000/layerindex/ with your browser. As
    with all Django applications there is an admin interface available

Signature · 1 line

2.5.5

Paul Eggleton

author

2016-08-30 21:05 UTC #2 lore raw

PATCH 2/3 Fix output decoding/reporting for bulk change patch download

In reply to Paul Eggleton

With Python 3 we need to take care of decoding the output so we can treat it as a string. At the same time, it's more useful to see the output string rather than the exception if there is some output.

Signed-off-by Paul Eggleton <[email protected]>

---
 layerindex/views.py | 3 +++
 1 file changed, 3 insertions(+)

layerindex/views.py +3 -0

index dfe4453..62f5268 100644
@@ -237,8 +237,11 @@ def bulk_change_patch_view(request, pk):
     except Exception as e:
         output = getattr(e, 'output', None)
         if output:
+            output = output.decode('utf-8', errors="ignore")
             if 'timeout' in output:
                 return HttpResponse('Failed to generate patches: timed out waiting for lock. Please try again shortly.', content_type='text/plain')
+            else:
+                return HttpResponse('Failed to generate patches: %s' % output, content_type='text/plain')
         return HttpResponse('Failed to generate patches: %s' % e, content_type='text/plain')
     # FIXME better error handling

Signature · 1 line

2.5.5

Paul Eggleton

author

2016-08-30 21:05 UTC #3 lore raw

PATCH 3/3 update_layer.py: fix up for bitbake API change

In reply to Paul Eggleton

The multiconfig changes broke the calls here to loadDataFull(). To avoid this being an issue in future, make use of tinfoil's new parse_recipe_file() function (if available) to isolate the code here from any future changes to bitbake's internal code.

Part of the fix for [YOCTO #10192].

Signed-off-by Paul Eggleton <[email protected]>

---
 layerindex/tools/import_classic.py | 11 +++++++----
 layerindex/update_layer.py         | 17 ++++++++++-------
 2 files changed, 17 insertions(+), 11 deletions(-)

layerindex/tools/import_classic.py +7 -4

index ee99246..45ccaa9 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Import OE-Classic recipe data into the layer index database
 #

@@ -27,11 +27,14 @@ import recipeparse
 logger = utils.logger_create('LayerIndexUpdate')
 
 
-def update_recipe_file(data, path, recipe, layerdir_start, repodir):
+def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir):
     fn = str(os.path.join(path, recipe.filename))
     try:
         logger.debug('Updating recipe %s' % fn)
-        envdata = bb.cache.Cache.loadDataFull(fn, [], data)
+        if hasattr(tinfoil, 'parse_recipe_file'):
+            envdata = tinfoil.parse_recipe_file(fn, appends=False, config_data=data)
+        else:
+            envdata = bb.cache.Cache.loadDataFull(fn, [], data)
         envdata.setVar('SRCPV', 'X')
         envdata.setVar('SRCDATE', 'X')
         envdata.setVar('SRCREV', 'X')

@@ -182,7 +185,7 @@ def main():
                     recipe.layerbranch = layerbranch
                     recipe.filename = filename
                     recipe.filepath = filepath
-                    update_recipe_file(config_data_copy, root, recipe, layerdir_start, oeclassicpath)
+                    update_recipe_file(tinfoil, config_data_copy, root, recipe, layerdir_start, oeclassicpath)
                     recipe.save()
 
         layerbranch.vcs_last_fetch = datetime.now()

layerindex/update_layer.py +10 -7

index fd8d3d4..13b508f 100644
@@ -54,11 +54,14 @@ def split_recipe_fn(path):
         pv = "1.0"
     return (pn, pv)
 
-def update_recipe_file(data, path, recipe, layerdir_start, repodir):
+def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir):
     fn = str(os.path.join(path, recipe.filename))
     try:
         logger.debug('Updating recipe %s' % fn)
-        envdata = bb.cache.Cache.loadDataFull(fn, [], data)
+        if hasattr(tinfoil, 'parse_recipe_file'):
+            envdata = tinfoil.parse_recipe_file(fn, appends=False, config_data=data)
+        else:
+            envdata = bb.cache.Cache.loadDataFull(fn, [], data)
         envdata.setVar('SRCPV', 'X')
         recipe.pn = envdata.getVar("PN", True)
         recipe.pv = envdata.getVar("PV", True)

@@ -356,7 +359,7 @@ def main():
                                     recipe.filepath = newfilepath
                                     recipe.filename = newfilename
                                     recipe.save()
-                                    update_recipe_file(config_data_copy, os.path.join(layerdir, newfilepath), recipe, layerdir_start, repodir)
+                                    update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, newfilepath), recipe, layerdir_start, repodir)
                                     updatedrecipes.add(os.path.join(oldfilepath, oldfilename))
                                     updatedrecipes.add(os.path.join(newfilepath, newfilename))
                                 else:

@@ -471,7 +474,7 @@ def main():
                                 results = layerrecipes.filter(filepath=filepath).filter(filename=filename)[:1]
                                 if results:
                                     recipe = results[0]
-                                    update_recipe_file(config_data_copy, os.path.join(layerdir, filepath), recipe, layerdir_start, repodir)
+                                    update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, filepath), recipe, layerdir_start, repodir)
                                     recipe.save()
                                     updatedrecipes.add(recipe.full_path())
                             elif typename == 'machine':

@@ -487,7 +490,7 @@ def main():
 
                     for recipe in dirtyrecipes:
                         if not recipe.full_path() in updatedrecipes:
-                            update_recipe_file(config_data_copy, os.path.join(layerdir, recipe.filepath), recipe, layerdir_start, repodir)
+                            update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, recipe.filepath), recipe, layerdir_start, repodir)
                 else:
                     # Collect recipe data from scratch

@@ -513,7 +516,7 @@ def main():
                                 # Recipe still exists, update it
                                 results = layerrecipes.filter(id=v['id'])[:1]
                                 recipe = results[0]
-                                update_recipe_file(config_data_copy, root, recipe, layerdir_start, repodir)
+                                update_recipe_file(tinfoil, config_data_copy, root, recipe, layerdir_start, repodir)
                             else:
                                 # Recipe no longer exists, mark it for later on
                                 layerrecipes_delete.append(v)

@@ -575,7 +578,7 @@ def main():
                     recipe.filename = os.path.basename(added)
                     root = os.path.dirname(added)
                     recipe.filepath = os.path.relpath(root, layerdir)
-                    update_recipe_file(config_data_copy, root, recipe, layerdir_start, repodir)
+                    update_recipe_file(tinfoil, config_data_copy, root, recipe, layerdir_start, repodir)
                     recipe.save()
 
                 for deleted in layerrecipes_delete:

Signature · 1 line

2.5.5

Andreas Müller

2016-08-30 21:32 UTC #4 lore raw

In reply to Paul Eggleton

Quoted 9 lines · Paul Eggleton

Some minor fixes for the recent python3 work in the layer index, but more importantly fix parsing in master for the multiconfig changes that recently went into bitbake.
NOTE: this depends on the bitbake patchset I just sent in order to work.
The following changes since commit 1d76228675e93add50dd5216c43e00f977e1aaa8:

Thanks

Andreas