Skip to contentSkip to conversation

classes/pypi: update the default UPSTREAM_CHECK_URI to use the simple repo API

PATCH Superseded Derek Straka 3 participants 9 messages Dec 12, 2024 – Dec 13, 2024

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 classes/pypi: update the default UPSTREAM_CHECK_URI to use the simple repo API

Update the UPSTREAM_CHECK_URI to leverage the simple repo API. The project URLs require javascript which breaks the version checking fetch and subsequent logic. The simple repo API provides similar functionality with a well defined spec which is used by tools such as pip. Also update the UPSTREAM_CHECK_REGEX to be compatible with the information retrieved via the API

---
 meta/classes-recipe/pypi.bbclass | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
meta/classes-recipe/pypi.bbclass +6 -2
index c6bbe8119a..15172e97b4 100644
@@ -37,7 +37,11 @@ S = "${WORKDIR}/${PYPI_PACKAGE}-${PV}"
 
 # Replace any '_' characters in the pypi URI with '-'s to follow the PyPi website naming conventions
 UPSTREAM_CHECK_PYPI_PACKAGE ?= "${@d.getVar('PYPI_PACKAGE').replace('_', '-')}"
-UPSTREAM_CHECK_URI ?= "https://pypi.org/project/${UPSTREAM_CHECK_PYPI_PACKAGE}/"
-UPSTREAM_CHECK_REGEX ?= "/${UPSTREAM_CHECK_PYPI_PACKAGE}/(?P<pver>(\d+[\.\-_]*)+)/"
+
+# Use the simple repository API rather than the potentially unstable project URL
+# More information on the pypi API specification is avaialble here:
+# https://packaging.python.org/en/latest/specifications/simple-repository-api/
+UPSTREAM_CHECK_URI ?= "https://pypi.org/simple/${UPSTREAM_CHECK_PYPI_PACKAGE}/"
+UPSTREAM_CHECK_REGEX ?= "${UPSTREAM_CHECK_PYPI_PACKAGE}-(?P<pver>(\d+[\.\-_]*)+).(tar\.gz|tgz)"
 
 CVE_PRODUCT ?= "python:${PYPI_PACKAGE}"
Signature · 1 line
2.34.1
#1 lore raw
Quoted 6 lines · Derek Straka via lists.openembedded.org

Update the UPSTREAM_CHECK_URI to leverage the simple repo API. The project URLs require javascript which breaks the version checking fetch and subsequent logic. The simple repo API provides similar functionality with a well defined spec which is used by tools such as pip. Also update the UPSTREAM_CHECK_REGEX to be compatible with the information retrieved via the API

Unfortunately even with this patch, there's a large amount of UNKNOWN_BROKEN in the output of 'devtool check-upgrade-status'. Can you please check those, hopefully it's a simple tweak?

Alex

#2 lore raw

Hi Alex,

Thanks for your note. I’m working through the remaining downstream recipe changes today which should address the rest of the UNKNOWN_BROKEN recipes.

While looking at it yesterday, the download packages come primarily in two archetypes:
1. Those that replace ‘_’ with ‘-‘ in the source archives
2. Those that leave the ‘_’ ONLY in the archives

Given that, I think it’s unlikely there’s a clean fix in the bbclass without a more invasive change to the upstream check logic. I can, however, package all the changes for one-core into a single patchset and submit a v2. That will at least address all the core updates in one fell swoop.

Does that sound reasonable?

-Derek

Quoted 16 lines · Alexander Kanavin
Quoted 6 lines · Derek Straka via lists.openembedded.org

Update the UPSTREAM_CHECK_URI to leverage the simple repo API. The project URLs require javascript which breaks the version checking fetch and subsequent logic. The simple repo API provides similar functionality with a well defined spec which is used by tools such as pip. Also update the UPSTREAM_CHECK_REGEX to be compatible with the information retrieved via the API

Unfortunately even with this patch, there's a large amount of UNKNOWN_BROKEN in the output of 'devtool check-upgrade-status'. Can you please check those, hopefully it's a simple tweak?

Alex

#3 lore raw
Quoted 9 lines · Derek Straka

Thanks for your note. I’m working through the remaining downstream recipe changes today which should address the rest of the UNKNOWN_BROKEN recipes.

While looking at it yesterday, the download packages come primarily in two archetypes:
1. Those that replace ‘_’ with ‘-‘ in the source archives
2. Those that leave the ‘_’ ONLY in the archives

Given that, I think it’s unlikely there’s a clean fix in the bbclass without a more invasive change to the upstream check logic. I can, however, package all the changes for one-core into a single patchset and submit a v2. That will at least address all the core updates in one fell swoop.

Does that sound reasonable?

Seems so, yes.

Historically pypi upstream checks have been a pain, as there has been a constant stream of seemingly random breaking changes, of two types:

1. _ being replaced by - and vice versa
2. CamelCasing being replaced by lowercasing and vice versa.

I haven't been able to figure out any pattern in this, or come up with a universal check. If you can simply fix up core recipes to not return UNKNOWN_BROKEN, I'd appreciate.

Alex

#4 lore raw

Agreed. It's painful at times because there isn't a complete standardization in pypi naming conventions. I' sent a v2 that is intended to resolve all of the oe-core UNKNOWN_BROKEN python recipes. I'll move to those in meta-python next.

Quoted 32 lines · Alexander Kanavin
Quoted 1 line · Derek Straka

Thanks for your note. I’m working through the remaining downstream

recipe changes today which should address the rest of the UNKNOWN_BROKEN recipes.

Quoted 2 lines

While looking at it yesterday, the download packages come primarily in

two archetypes:

Quoted 4 lines

1. Those that replace ‘_’ with ‘-‘ in the source archives
2. Those that leave the ‘_’ ONLY in the archives

Given that, I think it’s unlikely there’s a clean fix in the bbclass

without a more invasive change to the upstream check logic. I can, however, package all the changes for one-core into a single patchset and submit a v2. That will at least address all the core updates in one fell swoop.

Quoted 2 lines

Does that sound reasonable?

Seems so, yes.

Historically pypi upstream checks have been a pain, as there has been a constant stream of seemingly random breaking changes, of two types:

1. _ being replaced by - and vice versa
2. CamelCasing being replaced by lowercasing and vice versa.

I haven't been able to figure out any pattern in this, or come up with a universal check. If you can simply fix up core recipes to not return UNKNOWN_BROKEN, I'd appreciate.

Alex

#5 lore raw
In reply to Alexander Kanavin
Quoted 17 lines · Alexander Kanavin via lists.openembedded.org
Quoted 9 lines · Derek Straka

Thanks for your note. I’m working through the remaining downstream recipe changes today which should address the rest of the UNKNOWN_BROKEN recipes.

While looking at it yesterday, the download packages come primarily in two archetypes:
1. Those that replace ‘_’ with ‘-‘ in the source archives
2. Those that leave the ‘_’ ONLY in the archives

Given that, I think it’s unlikely there’s a clean fix in the bbclass without a more invasive change to the upstream check logic. I can, however, package all the changes for one-core into a single patchset and submit a v2. That will at least address all the core updates in one fell swoop.

Does that sound reasonable?

Seems so, yes.

Historically pypi upstream checks have been a pain, as there has been a constant stream of seemingly random breaking changes, of two types:

Quoted 6 lines

1. _ being replaced by - and vice versa
2. CamelCasing being replaced by lowercasing and vice versa.

I haven't been able to figure out any pattern in this, or come up with a universal check. If you can simply fix up core recipes to not return UNKNOWN_BROKEN, I'd appreciate.

I’ve some partial branches that attempt to bring sanity to this but yes, it’s a mess.

The good news is that https://peps.python.org/pep-0625/ says that sdist filenames should be normalised and from what I can tell everything but setuptools does normalise, and the use of setuptools is falling.

The simple update API says that the project name is normalised, so we can add a little normalise function:

def pypi_normalize(s):

    import re
    return re.sub(r"[-_.]+", "_", s).lower()

And use that to turn the PYPI_PACKAGE into the right thing, surely?

Rationalising this is the source of at least three wip branches I have locally, so I’d love to see it sorted.

Ross

#6 lore raw

Hi Ross,

I'll give that normalization another shot. I saw a couple older packages not following the normalized filenames, but those could be outliers. Thanks for the pointer.

-Derek

Quoted 53 lines · Ross Burton
Quoted 3 lines · Alexander Kanavin via lists.openembedded.org
Quoted 1 line · Derek Straka

Thanks for your note. I’m working through the remaining downstream

recipe changes today which should address the rest of the UNKNOWN_BROKEN recipes.

Quoted 2 lines
Quoted 2 lines

While looking at it yesterday, the download packages come primarily in

two archetypes:

Quoted 4 lines
Quoted 4 lines

1. Those that replace ‘_’ with ‘-‘ in the source archives
2. Those that leave the ‘_’ ONLY in the archives

Given that, I think it’s unlikely there’s a clean fix in the bbclass

without a more invasive change to the upstream check logic. I can, however, package all the changes for one-core into a single patchset and submit a v2. That will at least address all the core updates in one fell swoop.

Quoted 8 lines
Quoted 2 lines

Does that sound reasonable?

Seems so, yes.

Historically pypi upstream checks have been a pain, as there has been a constant stream of seemingly random breaking changes, of two types:

Quoted 6 lines

1. _ being replaced by - and vice versa
2. CamelCasing being replaced by lowercasing and vice versa.

I haven't been able to figure out any pattern in this, or come up with a universal check. If you can simply fix up core recipes to not return UNKNOWN_BROKEN, I'd appreciate.

I’ve some partial branches that attempt to bring sanity to this but yes, it’s a mess.

The good news is that https://peps.python.org/pep-0625/ says that sdist filenames should be normalised and from what I can tell everything but setuptools does normalise, and the use of setuptools is falling.

The simple update API says that the project name is normalised, so we can add a little normalise function:

def pypi_normalize(s):

    import re
    return re.sub(r"[-_.]+", "_", s).lower()

And use that to turn the PYPI_PACKAGE into the right thing, surely?

Rationalising this is the source of at least three wip branches I have locally, so I’d love to see it sorted.

Ross

#7 lore raw

Unfortunately, there's a large swath of packages in both oe-core (>30) in meta-python (>70) that do not follow PEP625. I'll send a v3 patchset that normalizes the URLs and sdist filenames for the future (assuming folks standardize to follow PEP625 in the future), but we'll still carry a large number of special cases in the interim.

Quoted 65 lines · Derek Straka

Hi Ross,

I'll give that normalization another shot. I saw a couple older packages not following the normalized filenames, but those could be outliers. Thanks for the pointer.

-Derek

Quoted 53 lines · Ross Burton
Quoted 3 lines · Alexander Kanavin via lists.openembedded.org
Quoted 1 line · Derek Straka

Thanks for your note. I’m working through the remaining downstream

recipe changes today which should address the rest of the UNKNOWN_BROKEN recipes.

Quoted 2 lines
Quoted 2 lines

While looking at it yesterday, the download packages come primarily in

two archetypes:

Quoted 4 lines
Quoted 4 lines

1. Those that replace ‘_’ with ‘-‘ in the source archives
2. Those that leave the ‘_’ ONLY in the archives

Given that, I think it’s unlikely there’s a clean fix in the bbclass

without a more invasive change to the upstream check logic. I can, however, package all the changes for one-core into a single patchset and submit a v2. That will at least address all the core updates in one fell swoop.

Quoted 8 lines
Quoted 2 lines

Does that sound reasonable?

Seems so, yes.

Historically pypi upstream checks have been a pain, as there has been a constant stream of seemingly random breaking changes, of two types:

Quoted 6 lines

1. _ being replaced by - and vice versa
2. CamelCasing being replaced by lowercasing and vice versa.

I haven't been able to figure out any pattern in this, or come up with a universal check. If you can simply fix up core recipes to not return UNKNOWN_BROKEN, I'd appreciate.

I’ve some partial branches that attempt to bring sanity to this but yes, it’s a mess.

The good news is that https://peps.python.org/pep-0625/ says that sdist filenames should be normalised and from what I can tell everything but setuptools does normalise, and the use of setuptools is falling.

The simple update API says that the project name is normalised, so we can add a little normalise function:

def pypi_normalize(s):

    import re
    return re.sub(r"[-_.]+", "_", s).lower()

And use that to turn the PYPI_PACKAGE into the right thing, surely?

Rationalising this is the source of at least three wip branches I have locally, so I’d love to see it sorted.

Ross

#8 lore raw
In reply to Ross Burton
Quoted 1 line · Ross Burton

Rationalising this is the source of at least three wip branches I have locally, so I’d love to see it sorted.

Hit sent too early.

One of my WIP branches basically made some semantic changes, where PYPI_PACKAGE is the canonical package name (as in, what you type when pip-installing) and PYPI_ARCHIVE_NAME is the normalised name (for sdist, etc). The fun is then building everything and setting variables where the package name can’t be derived from the recipe name (eg python3-babel has a PYPI_PACKAGE of Babel) or where the archive isn’t actually normalised (eg more-itertools has a more-itertools sdist but more_itertools wheel).

I _think_ this was the right direction to take at least…

Ross