Skip to contentSkip to conversation

Patches from mailing list and fixes

PATCH New Tim Orling 1 participant 7 messages Dec 16, 2022

lore
Share & tools
lore

Tim Orling

author
#0 lore raw

cover Patches from mailing list and fixes

This series catches up on some patches that had already been sent to the mailing list, just re-iterating them here exactly as they will be merged, as well as a couple new patches.

A big "Thank You!" to Robert Yang for sharing fixes that Wind River was carrying internally.

One new patch, sent on Michael Halstead's behalf, represents the current production instance changes in order to meet the Python 3.8 minimum version requirement for current bitbake.

One additional new patch replaces deprecated distutils usage (which was only used for LooseVersion), with pkg_resources.parse_version instead.

The following changes since commit d8ab1beebeb5b49693b522d8618393620495bd74:

layerindex/utils.py: ignore 'core' in BBFILES_COLLECTIONS (2022-04-28 08:15:22 -0700)

are available in the Git repository at:

https://git.yoctoproject.org/layerindex-web timo/mailing-list-patches
http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=timo/mailing-list-patches

Michael Halstead (2):
Dockerfile: add captcha dependencies
Dockerfile: switch to ubuntu:jammy for >python3.8

Robert Yang (3):
utils.py: Use get_layer_var() to get BBFILE_COLLECTIONS admin.py: Remove has_delete_permission() utils.py: Fix for is_commit_ancestor()

Tim Orling (1):
global: replace deprecated distutils

 Dockerfile                             | 23 ++++++++++++-----------
 layerindex/admin.py                    | 17 -----------------
 layerindex/recipedesc.py               |  1 -
 layerindex/tools/import_classic.py     |  1 -
 layerindex/tools/import_otherdistro.py |  1 -
 layerindex/update.py                   |  4 ++--
 layerindex/update_layer.py             |  6 +++---
 layerindex/utils.py                    | 20 +++++++++++---------
 layerindex/views.py                    |  6 ++----
 rrs/tools/upgrade_history_internal.py  |  4 ++--
 10 files changed, 32 insertions(+), 51 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/6 Dockerfile: add captcha dependencies

From: Michael Halstead <[email protected]>

The current version of django-simple-captcha requires libz and freetype development libraries.

Refactor to apply. Fix typo. Fix previous mixed use of space vs tab.

---
 Dockerfile | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Dockerfile +3 -1
index 4b2870a..a6c7ad2 100644
@@ -31,7 +31,9 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update \
 	python3-mysqldb \
 	python3-dev \
 	python3-pil \
-        python3-wheel \
+	python3-wheel \
+	libz-dev \
+	libfreetype6-dev \
 	libjpeg-dev \
 	libmariadbclient-dev \
 	locales \
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH 2/6 utils.py: Use get_layer_var() to get BBFILE_COLLECTIONS

In reply to Tim Orling

From: Robert Yang <[email protected]>

The commit 38e6288c7dad579518b0de2b6664f23be58889b6 has added oe-core to BBLAYERS by default, and then there will be multiple BBFILE_COLLECTIONS which causes update.py failure to update layers (most of the dependencies or recommends are missing), use get_layer_var() to get BBFILE_COLLECTIONS to fix the problem since it can ignore the extra oe-core from BBFILE_COLLECTIONS.

Fix minor typos. Minor rewording.

---
 layerindex/utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
layerindex/utils.py +1 -1
index efd18e4..ed7fb47 100644
@@ -188,7 +188,7 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None, required=T
             logger.warn('Either set REMOVE_LAYER_DEPENDENCIES to remove them from the database, or fix conf/layer.conf')
 
 def set_layerbranch_collection_version(layerbranch, config_data, logger=None):
-    layerbranch.collection = config_data.getVar('BBFILE_COLLECTIONS', True)
+    layerbranch.collection = get_layer_var(config_data, 'BBFILE_COLLECTIONS', logger)
     ver_str = "LAYERVERSION_"
     if layerbranch.collection:
         layerbranch.collection = layerbranch.collection.strip()
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH 3/6 admin.py: Remove has_delete_permission()

In reply to Tim Orling

From: Robert Yang <[email protected]>

The following commit has added on_delete=models.CASCADE to the ForeignKey:

===
commit 2d526f9b0d363f3e442abc97ef9bd9fab37ee5e4

Author: Amber Elliot <[email protected]>
Date:   Thu Jun 25 15:57:34 2020 -0700

Updating models and imports for Django 2.2 upgrade.
===

The on_delete=models.CASCADE will remove the objects which reference the removed objects, so there will be errors when removing a LayerBranch with admin permissions:

===
Cannot delete layer branch
Deleting the selected layer branch would result in deleting related objects, but your account doesn't have permission to delete the following types of objects:

Append
distro
===

This is because has_delete_permission() always return False, remove it to use Django's implementation to fix the problem.

Minor rewording.

---
 layerindex/admin.py | 17 -----------------
 1 file changed, 17 deletions(-)
layerindex/admin.py +0 -17
index 9ed6102..f8b2f6e 100644
@@ -102,8 +102,6 @@ class RecipeAdmin(admin.ModelAdmin):
         return rofields
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class PackageConfigAdmin(admin.ModelAdmin):
     search_fields = ['feature', 'recipe__pn']
@@ -132,8 +130,6 @@ class ClassicRecipeAdmin(admin.ModelAdmin):
         return rofields
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class MachineAdmin(admin.ModelAdmin):
     search_fields = ['name']
@@ -141,8 +137,6 @@ class MachineAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in Machine._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class DistroAdmin(admin.ModelAdmin):
     search_fields = ['name']
@@ -150,9 +144,6 @@ class DistroAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in Distro._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
-
 
 class BBAppendAdmin(admin.ModelAdmin):
     search_fields = ['filename']
@@ -160,8 +151,6 @@ class BBAppendAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in BBAppend._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class BBClassAdmin(admin.ModelAdmin):
     search_fields = ['name']
@@ -169,8 +158,6 @@ class BBClassAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in BBClass._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class PatchAdmin(admin.ModelAdmin):
     search_fields = ['path']
@@ -178,8 +165,6 @@ class PatchAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in Patch._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class PatchDispositionAdmin(admin.ModelAdmin):
     fields = ['patch', 'user', 'disposition', 'comment']
@@ -199,8 +184,6 @@ class IncFileAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in IncFile._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class RecipeChangeInline(admin.StackedInline):
     model = RecipeChange
Signature · 1 line
2.34.1
author
#4 lore raw

PATCH 4/6 utils.py: Fix for is_commit_ancestor()

In reply to Tim Orling

From: Robert Yang <[email protected]>

The runcmd() would print an "ERROR" on failure which causes confusion since the failure is expected on old branches, so subprocess.getstatusoutput to fix the problem.

Minor rewording.

---
 layerindex/utils.py | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
layerindex/utils.py +10 -8
index ed7fb47..6566710 100644
@@ -243,16 +243,18 @@ def is_commit_ancestor(repodir, commit, logger):
         # check if commit is a sha1 hash
         if re.match('[0-9a-f]{40}', commit):
             # check if the commit is an ancestor
-            contained = runcmd(['git', 'merge-base', '--is-ancestor', '%s' % commit, 'HEAD'], repodir, logger=logger)
-            return True
-        else:
-            raise Exception('is_commit_ancestor: "commit" must be a SHA1 hash')
-    except subprocess.CalledProcessError as e:
-            if e.returncode == 1:
-                # commit is not an ancestor
+            cmd = "GIT_DIR=%s/.git git merge-base --is-ancestor %s HEAD" % (repodir, commit)
+            logger.debug('Running "%s"' % cmd)
+            ret, output = subprocess.getstatusoutput(cmd)
+            if ret == 0:
+                return True
+            elif ret == 1:
+                logger.debug('output: %s' % output)
                 return False
             else:
-                raise e
+                raise Exception('Failed to run command: %s: ret: %s, output: %s' % (cmd, ret, output))
+        else:
+            raise Exception('is_commit_ancestor: "commit" must be a SHA1 hash')
     except Exception as esc:
         logger.warn(esc)
 
Signature · 1 line
2.34.1
author
#5 lore raw

PATCH 5/6 Dockerfile: switch to ubuntu:jammy for >python3.8

In reply to Tim Orling

From: Michael Halstead <[email protected]>

ubuntu:jammy provides python 3.10.6 which supports current bitbake. Dropped unneeded python2 packages.
Moved locale generation high to reduce error messages during image building.

---
 Dockerfile | 21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)
Dockerfile +10 -11
index a6c7ad2..d8c38e1 100644
@@ -1,6 +1,6 @@
 # See README for how to use this.
 
-FROM debian:buster
+FROM ubuntu:jammy
 LABEL maintainer="Michael Halstead <[email protected]>"
 
 ENV PYTHONUNBUFFERED=1 \
@@ -17,30 +17,29 @@ ENV PYTHONUNBUFFERED=1 \
 
 COPY requirements.txt /
 RUN DEBIAN_FRONTEND=noninteractive apt-get update \
+    && apt-get install -y locales \
+    && echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
+        && locale-gen en_US.UTF-8 \
+        && update-locale \
     && apt-get install -y --no-install-recommends \
 	autoconf \
 	g++ \
 	gcc \
 	make \
-	python-pip \
-	python-mysqldb \
-	python-dev \
-	python-pil \
-	python-wheel \
+	python2 \
 	python3-pip \
 	python3-mysqldb \
 	python3-dev \
 	python3-pil \
 	python3-wheel \
-	libz-dev \
+	zlib1g-dev \
 	libfreetype6-dev \
 	libjpeg-dev \
-	libmariadbclient-dev \
-	locales \
+	libmariadb-dev-compat \
 	netcat-openbsd \
 	curl \
 	wget \
-	git-core \
+	git \
 	vim \
 	rpm2cpio \
 	rpm \
@@ -51,7 +50,7 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update \
     && pip3 install gunicorn \
     && pip3 install setuptools \
     && pip3 install -r /requirements.txt \
-    && apt-get purge -y autoconf g++ make python3-dev libjpeg-dev libmariadbclient-dev \
+    && apt-get purge -y autoconf g++ make python3-dev libjpeg-dev \
 	&& apt-get autoremove -y \
 	&& rm -rf /var/lib/apt/lists/* \
 	&& apt-get clean
Signature · 1 line
2.34.1
author
#6 lore raw

PATCH 6/6 global: replace deprecated distutils

In reply to Tim Orling

Replace distutils.version.LooseVersion with pkg_resource.parse_version

https://majornetwork.net/2021/05/comparing-version-numbers-in-python/

[YOCTO #14990]

---
 layerindex/recipedesc.py               | 1 -
 layerindex/tools/import_classic.py     | 1 -
 layerindex/tools/import_otherdistro.py | 1 -
 layerindex/update.py                   | 4 ++--
 layerindex/update_layer.py             | 6 +++---
 layerindex/views.py                    | 6 ++----
 rrs/tools/upgrade_history_internal.py  | 4 ++--
 7 files changed, 9 insertions(+), 14 deletions(-)
layerindex/recipedesc.py +0 -1
index a844186..33aef31 100644
@@ -13,7 +13,6 @@ import logging
 import subprocess
 from datetime import datetime
 import fnmatch
-from distutils.version import LooseVersion
 
 
 logger = None
layerindex/tools/import_classic.py +0 -1
index 0216221..44f8f88 100755
@@ -22,7 +22,6 @@ import fnmatch
 import re
 import tempfile
 import shutil
-from distutils.version import LooseVersion
 import utils
 import recipeparse
 
layerindex/tools/import_otherdistro.py +0 -1
index 0c6640f..5c00115 100755
@@ -23,7 +23,6 @@ import subprocess
 import string
 import shlex
 import codecs
-from distutils.version import LooseVersion
 
 sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__), '..')))
 sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__), 'lib')))
layerindex/update.py +2 -2
index 5a2642b..7de9c0d 100755
@@ -17,7 +17,7 @@ import codecs
 import logging
 import subprocess
 from datetime import datetime, timedelta
-from distutils.version import LooseVersion
+from pkg_resources import parse_version
 import utils
 import operator
 import re
@@ -130,7 +130,7 @@ def extract_value(valuename, output):
         return ''
 
 def main():
-    if LooseVersion(git.__version__) < '0.3.1':
+    if parse_version(git.__version__) < parse_version('0.3.1'):
         logger.error("Version of GitPython is too old, please install GitPython (python-git) 0.3.1 or later in order to use this script")
         sys.exit(1)
 
layerindex/update_layer.py +3 -3
index e8b210e..e0ecc94 100644
@@ -19,7 +19,7 @@ import re
 import tempfile
 import shutil
 import errno
-from distutils.version import LooseVersion
+from pkg_resources import parse_version
 import itertools
 import utils
 import recipeparse
@@ -239,7 +239,7 @@ def update_distro_conf_file(path, distro, d):
         distro.description = desc
 
 def main():
-    if LooseVersion(git.__version__) < '0.3.1':
+    if parse_version(git.__version__) < parse_version('0.3.1'):
         logger.error("Version of GitPython is too old, please install GitPython (python-git) 0.3.1 or later in order to use this script")
         sys.exit(1)
 
@@ -808,7 +808,7 @@ def main():
         logger.error(traceback.format_exc().rstrip())
         sys.exit(1)
     finally:
-        if tinfoil and (LooseVersion(bb.__version__) > LooseVersion("1.27")):
+        if tinfoil and (parse_version(bb.__version__) > parse_version("1.27")):
             tinfoil.shutdown()
 
     if tempdir:
layerindex/views.py +2 -4
index 6f22c0c..e660578 100644
@@ -41,7 +41,6 @@ from django.views.generic.base import RedirectView
 from django.views.generic.edit import (CreateView, DeleteView, FormView,
                                        UpdateView)
 from django_registration.backends.activation.views import RegistrationView
-from pkg_resources import parse_version
 from reversion.models import Revision
 
 import settings
@@ -1736,7 +1735,6 @@ class BranchCompareView(FormView):
     form_class = BranchComparisonForm
 
     def get_recipes(self, from_branch, to_branch, layer_ids):
-        from distutils.version import LooseVersion
         class BranchComparisonResult:
             def __init__(self, pn, short_desc):
                 self.pn = pn
@@ -1791,9 +1789,9 @@ class BranchCompareView(FormView):
         changed = []
         removed = []
         for _, recipe in sorted(recipes.items(), key=lambda item: item[0]):
-            recipe.from_versions = sorted(recipe.from_versions, key=lambda item: LooseVersion(item.pv))
+            recipe.from_versions = sorted(recipe.from_versions, key=lambda item: parse_version(item.pv))
             from_version_exprs = [x.version_expr() for x in recipe.from_versions]
-            recipe.to_versions = sorted(recipe.to_versions, key=lambda item: LooseVersion(item.pv))
+            recipe.to_versions = sorted(recipe.to_versions, key=lambda item: parser_version(item.pv))
             to_version_exprs = [x.version_expr() for x in recipe.to_versions]
             if not from_version_exprs:
                 added.append(recipe)
rrs/tools/upgrade_history_internal.py +2 -2
index e3a96d9..2ba96b1 100644
@@ -16,7 +16,7 @@ import os
 import optparse
 import logging
 import re
-from distutils.version import LooseVersion
+from pkg_resources import parse_version
 import git
 from datetime import datetime
 import calendar
@@ -561,7 +561,7 @@ def generate_history(options, layerbranch_id, commit, logger):
             pass
 
     finally:
-        if tinfoil and hasattr(tinfoil, 'shutdown') and (LooseVersion(bb.__version__) > LooseVersion("1.27")):
+        if tinfoil and hasattr(tinfoil, 'shutdown') and (parse_version(bb.__version__) > parse_version("1.27")):
             tinfoil.shutdown()
         utils.rmtree_force(tempdir)
 
Signature · 1 line
2.34.1