Skip to contentSkip to conversation

Fixes for updates

PATCH Superseded Tim Orling 1 participant 5 messages Apr 25, 2022 – Apr 28, 2022

lore
Share & tools
lore

Tim Orling

author
#0 lore raw

cover Fixes for updates

This series includes fixes to allow update.py to run on 'master', by providing a check for the BB_ENV_PASSTHROUGHS_ADDITIONS variable renaming in bitbake 2.0+.

To do so, an 'is_commit_ancestor' check is added to allow for checking if a particular commit hash is present in a checked out repo (e.g. bitbake).

Also included is an improvement on the prior patch to ignore 'core' in BBFILES_COLLECTIONS (logging a debug message rather than a warning for the case where 'core' is ignored).

The following changes since commit 4dd7f0ee8f31df927c3862677dc59636988d8dfd:

layerindex/models.py: add Inactive-Upstream (2022-04-18 12:56:51 -0700)

are available in the Git repository at:

git://git.yoctoproject.org/layerindex-web timo/fixes2
http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=timo/fixes2

Tim Orling (3):
layerindex/utils.py: add is_commit_ancestor check recipe{desc,parse}.py: BB_ENV_PASSTHROUGH_ADDITIONS layerindex/utils.py: ignore 'core' in BBFILES_COLLECTIONS

 layerindex/recipedesc.py  |  8 +++++++-
 layerindex/recipeparse.py |  7 ++++++-
 layerindex/utils.py       | 29 ++++++++++++++++++++++++++++-
 3 files changed, 41 insertions(+), 3 deletions(-)
Signature · 1 line
2.32.0
author
#1 lore raw

PATCH v2 2/3 recipe{desc,parse}.py: BB_ENV_PASSTHROUGH_ADDITIONS

ERROR: Variable BB_ENV_EXTRAWHITE has been renamed to BB_ENV_PASSTHROUGH_ADDITIONS
ERROR: Variable BB_ENV_EXTRAWHITE from the shell environment has been renamed to BB_ENV_PASSTHROUGH_ADDITIONS
ERROR: Exiting to allow enviroment variables to be corrected

Replace BB_ENV_EXTRAWHITE with new variable BB_ENV_PASSTHROUGH_ADDITIONS

In order to be backward compatible with older branches, we must first check for the presence of the bitbake commit which implemented the variable name change, using layerindex.utils.is_commit_ancestor().

---
Changes in v2:
* Use new is_commit_ancestor check to check for the specific bitbake commit that introduced the variable renaming. This provides for backward compatibility in older branches (before bitbake 2.0).

 layerindex/recipedesc.py  | 8 +++++++-
 layerindex/recipeparse.py | 7 ++++++-
 2 files changed, 13 insertions(+), 2 deletions(-)
layerindex/recipedesc.py +7 -1
index ee7f2fe..a844186 100644
@@ -39,6 +39,7 @@ def main():
     from layerindex.models import LayerItem, Recipe
     from django.db import transaction
     import settings
+    from layerindex.utils import is_commit_ancestor
 
     setup_environ(settings)
 
@@ -62,8 +63,13 @@ def main():
             print("Unable to find bitbake by searching BITBAKEDIR, specified path '%s' or its parent, or PATH" % basepath)
             sys.exit(1)
 
+    # Commit "bitbake: Rename environment filtering variables"
+    bb_var_rename_commit = "87104b6a167188921da157c7dba45938849fb22a"
     # Skip sanity checks
-    os.environ['BB_ENV_EXTRAWHITE'] = 'DISABLE_SANITY_CHECKS'
+    if is_commit_ancestor(bitbakepath, bb_var_rename_commit, logger=logger):
+        os.environ['BB_ENV_PASSTHROUGH_ADDITIONS'] = 'DISABLE_SANITY_CHECKS'
+    else:
+        os.environ['BB_ENV_EXTRAWHITE'] = 'DISABLE_SANITY_CHECKS'
     os.environ['DISABLE_SANITY_CHECKS'] = '1'
 
     sys.path.extend([bitbakepath + '/lib'])
layerindex/recipeparse.py +6 -1
index c918677..6202745 100644
@@ -35,8 +35,13 @@ def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout
             bitbake_ref = 'origin/%s' % branch.bitbake_branch
         utils.checkout_repo(bitbakepath, bitbake_ref, logger=logger)
 
+    # Commit "bitbake: Rename environment filtering variables"
+    bb_var_rename_commit = "87104b6a167188921da157c7dba45938849fb22a"
     # Skip sanity checks
-    os.environ['BB_ENV_EXTRAWHITE'] = 'DISABLE_SANITY_CHECKS'
+    if utils.is_commit_ancestor(bitbakepath, bb_var_rename_commit, logger=logger):
+        os.environ['BB_ENV_PASSTHROUGH_ADDITIONS'] = 'DISABLE_SANITY_CHECKS'
+    else:
+        os.environ['BB_ENV_EXTRAWHITE'] = 'DISABLE_SANITY_CHECKS'
     os.environ['DISABLE_SANITY_CHECKS'] = '1'
 
     fetchdir = settings.LAYER_FETCH_DIR
Signature · 1 line
2.32.0
author
#2 lore raw

PATCH v2 3/3 layerindex/utils.py: ignore 'core' in BBFILES_COLLECTIONS

In reply to Tim Orling

Many layers append BBFILE_COLLECTIONS and therefore have 'core <layer>'

During update.py, this means we are likely not handling the collection we expect:

WARNING: /opt/workdir/git___git_openembedded_org_meta-openembedded/meta-oe: multiple collections found, handling first one (core) only BBFILE_COLLECTIONS = "core"

---
Changes in v2:
* Use logger.debug instead of warning for the case where 'core' is ignored

 layerindex/utils.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
layerindex/utils.py +6 -1
index 46e109e..414bbe5 100644
@@ -44,7 +44,12 @@ def get_layer_var(config_data, var, logger):
         collection = collection_list[0]
         layerdir = config_data.getVar('LAYERDIR', True)
         if len(collection_list) > 1:
-            logger.warn('%s: multiple collections found, handling first one (%s) only' % (layerdir, collection))
+            if collection_list[0] == 'core':
+                # Many layers append BBFILE_COLLECTIONS and therefore have 'core <layer>'
+                collection = collection_list[1]
+                logger.debug('%s: multiple collections found, ignoring the first one (\'core\') and handling (%s) only' % (layerdir, collection))
+            else:
+                logger.warn('%s: multiple collections found, handling first one (%s) only' % (layerdir, collection))
         if var == 'BBFILE_COLLECTIONS':
             return collection
     value = config_data.getVar('%s_%s' % (var, collection), True)
Signature · 1 line
2.32.0
author
#3 lore raw

PATCH 1/3 layerindex/utils.py: add is_commit_ancestor check

In reply to Tim Orling

Add a helper function to check if a given SHA1 hash is an ancestor in the currently checkout out branch, using:

git merge-branch --is-ancestor <commit> HEAD

NOTE: This will not match commits which have been cherry-picked.

---
 layerindex/utils.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
layerindex/utils.py +22 -0
index 32be16d..46e109e 100644
@@ -229,6 +229,28 @@ def explode_dep_versions2(bitbakepath, deps):
     import bb.utils
     return bb.utils.explode_dep_versions2(deps)
 
+def is_commit_ancestor(repodir, commit, logger):
+    """
+    Check if the given SHA1 hash is an ancestor in the currently checked out branch.
+    NOTE: This will not match commits which have been cherry-picked.
+    """
+    try:
+        # check if commit is a sha1 hash
+        if re.match('[0-9a-f]{40}', commit):
+            # check if the commit is an ancestor
+            contained = runcmd(['git', 'merge-base', '--is-ancestor', '%s' % commit, 'HEAD'], repodir, logger=logger)
+            return True
+        else:
+            raise Exception('is_commit_ancestor: "commit" must be a SHA1 hash')
+    except CalledProcessError as e:
+            if e.returncode == 1:
+                # commit is not an ancestor
+                return False
+            else:
+                raise e
+    except Exception as esc:
+        logger.warn(esc)
+
 def checkout_repo(repodir, commit, logger, force=False):
     """
     Check out a revision in a repository, ensuring that untracked/uncommitted
Signature · 1 line
2.32.0
author
#4 lore raw

PATCH v2 1/3 layerindex/utils.py: add is_commit_ancestor check

Add a helper function to check if a given SHA1 hash is an ancestor in the currently checked out branch, using:

git merge-branch --is-ancestor <commit> HEAD

NOTE: This will not match commits which have been cherry-picked.

---
Changes in v2:
* Fix import of (subprocess.)CalledProcessError

 layerindex/utils.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
layerindex/utils.py +22 -0
index 32be16d..aaf8c52 100644
@@ -229,6 +229,28 @@ def explode_dep_versions2(bitbakepath, deps):
     import bb.utils
     return bb.utils.explode_dep_versions2(deps)
 
+def is_commit_ancestor(repodir, commit, logger):
+    """
+    Check if the given SHA1 hash is an ancestor in the currently checked out branch.
+    NOTE: This will not match commits which have been cherry-picked.
+    """
+    try:
+        # check if commit is a sha1 hash
+        if re.match('[0-9a-f]{40}', commit):
+            # check if the commit is an ancestor
+            contained = runcmd(['git', 'merge-base', '--is-ancestor', '%s' % commit, 'HEAD'], repodir, logger=logger)
+            return True
+        else:
+            raise Exception('is_commit_ancestor: "commit" must be a SHA1 hash')
+    except subprocess.CalledProcessError as e:
+            if e.returncode == 1:
+                # commit is not an ancestor
+                return False
+            else:
+                raise e
+    except Exception as esc:
+        logger.warn(esc)
+
 def checkout_repo(repodir, commit, logger, force=False):
     """
     Check out a revision in a repository, ensuring that untracked/uncommitted
Signature · 1 line
2.32.0