Skip to contentSkip to conversation

python/*: remove spurious CLEANBROKENs

PATCH New Ross Burton 1 participant 4 messages Sep 4

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/4 python/*: remove spurious CLEANBROKENs

CLEANBROKEN doesn't do anything with the current build classes as they override base_do_configure().

---
 meta/recipes-devtools/python/python3-babel_2.18.0.bb | 2 --
 meta/recipes-devtools/python/python3-click_8.5.0.bb  | 2 --
 meta/recipes-devtools/python/python3-jinja2_3.1.6.bb | 2 --
 meta/recipes-devtools/python/python3-lxml_6.1.2.bb   | 2 --
 4 files changed, 8 deletions(-)
meta/recipes-devtools/python/python3-babel_2.18.0.bb +0 -2
index 26847372bf3..cf2752f6a23 100644
@@ -11,8 +11,6 @@ CVE_PRODUCT = "pocoo:babel"
 
 S = "${UNPACKDIR}/babel-${PV}"
 
-CLEANBROKEN = "1"
-
 RDEPENDS:${PN} += " \
     python3-codecs \
     python3-difflib \
meta/recipes-devtools/python/python3-click_8.5.0.bb +0 -2
index 88844676928..05a8fb343a7 100644
@@ -25,8 +25,6 @@ do_install_ptest:append() {
     cp -rf ${S}/docs ${D}${PTEST_PATH}/
 }
 
-CLEANBROKEN = "1"
-
 RDEPENDS:${PN} += "\
     python3-io \
     python3-threading \
meta/recipes-devtools/python/python3-jinja2_3.1.6.bb +0 -2
index 2e96eb0dcb3..add266095df 100644
@@ -10,8 +10,6 @@ PYPI_PACKAGE = "jinja2"
 
 CVE_PRODUCT = "jinja2 jinja"
 
-CLEANBROKEN = "1"
-
 inherit pypi python_flit_core ptest-python-pytest
 
 do_install_ptest:append() {
meta/recipes-devtools/python/python3-lxml_6.1.2.bb +0 -2
index a0162cab3ff..17c5d769a0c 100644
@@ -40,5 +40,3 @@ BBCLASSEXTEND = "native nativesdk"
 RDEPENDS:${PN} += "libxml2 libxslt python3-compression"
 
 CVE_PRODUCT = "lxml:lxml"
-
-CLEANBROKEN = "1"
Signature · 1 line
2.43.0
#1 lore raw

PATCH v2 2/4 classes/pypi: only set downloadprefix in SRC_URI if needed

Don't bloat the SRC_URI by setting a redundant downloadprefix if it is not required.

---
 meta/classes-recipe/pypi.bbclass | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
meta/classes-recipe/pypi.bbclass +7 -2
index 0897994bb65..b42b171e151 100644
@@ -25,8 +25,13 @@ def pypi_src_uri(d):
     """
     package = d.getVar('PYPI_PACKAGE')
     archive_name = d.expand('${PYPI_PACKAGE}-${PV}.${PYPI_PACKAGE_EXT}')
-    archive_downloadname = d.getVar('PYPI_ARCHIVE_NAME_PREFIX') + archive_name
-    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s;downloadfilename=%s' % (package[0], package, archive_name, archive_downloadname)
+    url = 'https://files.pythonhosted.org/packages/source/%s/%s/%s' % (package[0], package, archive_name)
+
+    download_prefix = d.getVar("PYPI_ARCHIVE_NAME_PREFIX")
+    if download_prefix:
+        url += ";downloadfilename=" + download_prefix + archive_name
+
+    return url
 
 def pypi_normalize(d):
     """"
Signature · 1 line
2.43.0
#2 lore raw

PATCH v2 3/4 classes/pypi: rename pypi_package() to pypi_default_package()

In reply to Ross Burton

Make it clear that this function returns a default value.

---
 meta/classes-recipe/pypi.bbclass | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
meta/classes-recipe/pypi.bbclass +6 -2
index b42b171e151..eb13ffe7e71 100644
@@ -4,7 +4,11 @@
 # SPDX-License-Identifier: MIT
 #
 
-def pypi_package(d):
+def pypi_default_package(d):
+    """
+    Return a reasonable guess for the PyPI package name by
+    stripping any python- prefix from PN.
+    """
     bpn = d.getVar('BPN')
     if bpn.startswith('python-'):
         return bpn[7:]
@@ -13,7 +17,7 @@ def pypi_package(d):
     return bpn
 
 # The PyPi package name (defaults to PN without the python3- prefix)
-PYPI_PACKAGE ?= "${@pypi_package(d)}"
+PYPI_PACKAGE ?= "${@pypi_default_package(d)}"
 # The file extension of the source archive
 PYPI_PACKAGE_EXT ?= "tar.gz"
 # An optional prefix for the download file in the case of name collisions
Signature · 1 line
2.43.0
#3 lore raw

PATCH v2 4/4 pypi: improve upstream check regex

In reply to Ross Burton

Currently the upstream release checker fetches the Simple API release index which lists all of the uploaded files. However the naming rules have evolved over time[1] so matching the value of PYPI_PACKAGE fails if the sdist filename was not normalised but then is normalised.

Remove UPSTREAM_CHECK_PYPI_PACKAGE which is realistically unused[2] and instead transform the PYPI_PACKAGE into a regular expression that handles both - and _ being used in the filename.

Change the version number match so that it handles .postN suffixes, as defined in the Version Specifiers documentation[3]. We don't want to match any alphanumeric characters as this would also match development snapshots and release candidates.

Finally escape the . between the version and the extension, so this matches a literal . and not any character.

Across oe-core and meta-oe this improves the upstream detection quite significantly:

- 93 recipes move from a broken check to up-to-date
- 21 recipes move from a broken check to needing an update
- 30 recipes move from up-to-date to needing an update

[1] Specifically PEP-0427 introduced normalisation of special characters to underscores and lowercasing.
[2] No users in oe-core, three in meta-oe which are no longer needed after this change.
[3] https://packaging.python.org/en/latest/specifications/version-specifiers/

---
 meta/classes-recipe/pypi.bbclass | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
meta/classes-recipe/pypi.bbclass +6 -4
index eb13ffe7e71..08788fcb9a9 100644
@@ -39,8 +39,7 @@ def pypi_src_uri(d):
 
 def pypi_normalize(d):
     """"
-        Normalize the package names to match PEP625 (https://peps.python.org/pep-0625/).
-        For non-compliant packages, maintainers can set UPSTREAM_CHECK_PYPI_PACKAGE to override the normalization
+    Normalize the package names to match PEP625 (https://peps.python.org/pep-0625/).
     """
     import re
     return re.sub(r"[-_.]+", "-", d.getVar('PYPI_PACKAGE')).lower()
@@ -52,7 +51,10 @@ SECTION = "devel/python"
 SRC_URI:prepend = "${PYPI_SRC_URI} "
 S = "${UNPACKDIR}/${PYPI_PACKAGE}-${PV}"
 
-UPSTREAM_CHECK_PYPI_PACKAGE ?= "${PYPI_PACKAGE}"
+def pypi_normalize_regex(d):
+    # Use a regex wildcard instead of hyphen as the filenames
+    # may or may not have been normalised properly.
+    return pypi_normalize(d).replace("-", "[_-]")
 
 # Use the simple repository API rather than the potentially unstable project URL
 # More information on the pypi API specification is avaialble here:
@@ -60,7 +62,7 @@ UPSTREAM_CHECK_PYPI_PACKAGE ?= "${PYPI_PACKAGE}"
 #
 # NOTE: All URLs for the simple API MUST request canonical normalized URLs per the spec
 UPSTREAM_CHECK_URI ?= "https://pypi.org/simple/${@pypi_normalize(d)}/"
-UPSTREAM_CHECK_REGEX ?= "(?i)${UPSTREAM_CHECK_PYPI_PACKAGE}-(?P<pver>(\d+[\.\-_]*)+).(tar\.gz|tgz|zip|tar\.bz2)"
+UPSTREAM_CHECK_REGEX ?= "(?i)${@pypi_normalize_regex(d)}-(?P<pver>(\d+(\.[\d\-]+)*(\.post\d+)?))\.(tar\.gz|tgz|zip|tar\.bz2)"
 
 CVE_PRODUCT ?= "python:${PYPI_PACKAGE}"
 
Signature · 1 line
2.43.0