Skip to contentSkip to conversation

pypi: Allow override of PyPI archive name

PATCH In review Paul Barker 3 participants 4 messages Aug 10, 2021 – Aug 13, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pypi: Allow override of PyPI archive name

Some packages on PyPI don't follow the usual expectations for archive naming. For example, the archive for asyncio-mqtt 0.10.0 is named asyncio_mqtt-0.10.0.tar.gz (with an underscore instead of the dash used in the package name).

To handle these edge cases a new PYPI_ARCHIVE_NAME variable is introduced. By default this is set to the expected archive name based on the PyPI package name, version and extension but it can be set to a different value if needed in a recipe which inherits the pypi class.

---
 meta/classes/pypi.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/classes/pypi.bbclass +3 -3
index 272c220bca..9405d58601 100644
@@ -8,12 +8,12 @@ def pypi_package(d):
 
 PYPI_PACKAGE ?= "${@pypi_package(d)}"
 PYPI_PACKAGE_EXT ?= "tar.gz"
+PYPI_ARCHIVE_NAME ?= "${PYPI_PACKAGE}-${PV}.${PYPI_PACKAGE_EXT}"
 
 def pypi_src_uri(d):
     package = d.getVar('PYPI_PACKAGE')
-    package_ext = d.getVar('PYPI_PACKAGE_EXT')
-    pv = d.getVar('PV')
-    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s' % (package[0], package, package, pv, package_ext)
+    archive_name = d.getVar('PYPI_ARCHIVE_NAME')
+    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s' % (package[0], package, archive_name)
 
 PYPI_SRC_URI ?= "${@pypi_src_uri(d)}"
 
Signature · 1 line
2.31.1
#1 lore raw
Quoted 10 lines · Paul Barker

Some packages on PyPI don't follow the usual expectations for archive naming. For example, the archive for asyncio-mqtt 0.10.0 is named asyncio_mqtt-0.10.0.tar.gz (with an underscore instead of the dash used in the package name).

To handle these edge cases a new PYPI_ARCHIVE_NAME variable is introduced. By default this is set to the expected archive name based on the PyPI package name, version and extension but it can be set to a different value if needed in a recipe which inherits the pypi class.

how many such packages are there roughly ? does it deserve to be addressed via introducing another variable I wonder.

Quoted 30 lines
---
  meta/classes/pypi.bbclass | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
meta/classes/pypi.bbclass +3 -3
index 272c220bca..9405d58601 100644
@@ -8,12 +8,12 @@ def pypi_package(d):
 
  PYPI_PACKAGE ?= "${@pypi_package(d)}"
  PYPI_PACKAGE_EXT ?= "tar.gz"
+PYPI_ARCHIVE_NAME ?= "${PYPI_PACKAGE}-${PV}.${PYPI_PACKAGE_EXT}"
 
  def pypi_src_uri(d):
      package = d.getVar('PYPI_PACKAGE')
-    package_ext = d.getVar('PYPI_PACKAGE_EXT')
-    pv = d.getVar('PV')
-    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s' % (package[0], package, package, pv, package_ext)
+    archive_name = d.getVar('PYPI_ARCHIVE_NAME')
+    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s' % (package[0], package, archive_name)
 
  PYPI_SRC_URI ?= "${@pypi_src_uri(d)}"
 
#2 lore raw
Quoted 8 lines · Khem Raj
Quoted 5 lines · Paul Barker

Some packages on PyPI don't follow the usual expectations for archive naming. For example, the archive for asyncio-mqtt 0.10.0 is named asyncio_mqtt-0.10.0.tar.gz (with an underscore instead of the dash used in the package name).

Quoted 9 lines
Quoted 5 lines

To handle these edge cases a new PYPI_ARCHIVE_NAME variable is introduced. By default this is set to the expected archive name based on the PyPI package name, version and extension but it can be set to a different value if needed in a recipe which inherits the pypi class.

how many such packages are there roughly ? does it deserve to be addressed via introducing another variable I wonder.

There are 29 packages in meta-oe/meta-python that fail to check latest upstream version and so far (13/29) it has been for very similar dash vs. underscore in the SRC_URI (from local AUH run). Not sure it is the complete solution, but it might help. Might need to apply to the upstream regex too in that case?

Quoted 37 lines
Quoted 10 lines
---
  meta/classes/pypi.bbclass | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
meta/classes/pypi.bbclass +0 -0
index 272c220bca..9405d58601 100644

meta/classes/pypi.bbclass def pypi_package(d): · line 16

Quoted 10 lines
meta/classes/pypi.bbclass +1 -3
@@ -8,12 +8,12 @@ def pypi_package(d):
 
  PYPI_PACKAGE ?= "${@pypi_package(d)}"
  PYPI_PACKAGE_EXT ?= "tar.gz"
+PYPI_ARCHIVE_NAME ?= "${PYPI_PACKAGE}-${PV}.${PYPI_PACKAGE_EXT}"
 
  def pypi_src_uri(d):
      package = d.getVar('PYPI_PACKAGE')
-    package_ext = d.getVar('PYPI_PACKAGE_EXT')
-    pv = d.getVar('PV')
-    return '

https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s' % (package[0], package, package, pv, package_ext)

meta/classes/pypi.bbclass def pypi_package(d): · line 16

Quoted 2 lines
meta/classes/pypi.bbclass +2 -0
+    archive_name = d.getVar('PYPI_ARCHIVE_NAME')
+    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s' %

(package[0], package, archive_name)

Quoted 8 lines
meta/classes/pypi.bbclass +0 -0
  PYPI_SRC_URI ?= "${@pypi_src_uri(d)}"
 
 
#3 lore raw
Quoted 20 lines · Tim Orling

On Thu, Aug 12, 2021 at 5:58 PM Khem Raj <[email protected] <mailto:[email protected]>> wrote:

On 8/10/21 4:03 AM, Paul Barker wrote:
> Some packages on PyPI don't follow the usual expectations for archive > naming. For example, the archive for asyncio-mqtt 0.10.0 is named > asyncio_mqtt-0.10.0.tar.gz (with an underscore instead of the
dash used
> in the package name).
>
> To handle these edge cases a new PYPI_ARCHIVE_NAME variable is > introduced. By default this is set to the expected archive name
based on
> the PyPI package name, version and extension but it can be set to a > different value if needed in a recipe which inherits the pypi class. >

Quoted 5 lines

how many such packages are there roughly ? does it deserve to be addressed via introducing another variable I wonder.

There are 29 packages in meta-oe/meta-python that fail to check latest upstream version and so far (13/29) it has been for very similar dash vs. underscore in the SRC_URI (from local AUH run). Not sure it is the complete solution, but it might help. Might need to apply to the upstream regex too in that case?

OK thats some number and yes extending it for upstream check would be an improvement.

Quoted 42 lines

     > Signed-off-by: Paul Barker <[email protected]
    <mailto:[email protected]>>
     > ---
     >   meta/classes/pypi.bbclass | 6 +++---
     >   1 file changed, 3 insertions(+), 3 deletions(-)
     >
     > diff --git a/meta/classes/pypi.bbclass b/meta/classes/pypi.bbclass
     > index 272c220bca..9405d58601 100644
     > --- a/meta/classes/pypi.bbclass
     > +++ b/meta/classes/pypi.bbclass
     > @@ -8,12 +8,12 @@ def pypi_package(d):
     >
     >   PYPI_PACKAGE ?= "${@pypi_package(d)}"
     >   PYPI_PACKAGE_EXT ?= "tar.gz"
     > +PYPI_ARCHIVE_NAME ?= "${PYPI_PACKAGE}-${PV}.${PYPI_PACKAGE_EXT}"
     >
     >   def pypi_src_uri(d):
     >       package = d.getVar('PYPI_PACKAGE')
     > -    package_ext = d.getVar('PYPI_PACKAGE_EXT')
     > -    pv = d.getVar('PV')
     > -    return
    'https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s
    <https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s>' %
    (package[0], package, package, pv, package_ext)
     > +    archive_name = d.getVar('PYPI_ARCHIVE_NAME')
     > +    return
    'https://files.pythonhosted.org/packages/source/%s/%s/%s
    <https://files.pythonhosted.org/packages/source/%s/%s/%s>' %
    (package[0], package, archive_name)
     >
     >   PYPI_SRC_URI ?= "${@pypi_src_uri(d)}"
     >
     >
     >
     >
     >
     >