Skip to contentSkip to conversation

bindings: python: optionally include module in sdist

PATCH Superseded Phil Howard 2 participants 4 messages Oct 24, 2023 GPIO

v7 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v7 cover bindings: python: optionally include module in sdist

This changeset vendors the gpiod library into the Python package.

Why?

So that setup.py can produce an sdist that is installable irrespective of the availability or version of a distro-supplied libgpiod.

This prevents a libgpiod pypi package install balking because the distro libgpiod is outdated or otherwise incompatible. This happens when attempting to install the current libgpiod from pypi onto - for example - the Debian Bookworm based Raspberry Pi OS or Ubuntu 23.10 Mantic which both ship with libgpiod v1.6.3.

The availability of a distro agnostic package also ensures that libgpiod can be installed via pypi into an isolated virtual environment, safely specified as a dependency for Python packages and allows Python developers to target the newest API version irrespective of their distro supplied libgpiod.

This is essential, since a venv is now widely *required* for user Python projects due to recommendations in pep-688 [1]

For Raspberry Pi this sdist can also be converted into a precompiled wheel by piwheels [2] which is, by default, added to Raspberry Pi OS as a pip index.

How?

If "LINK_SYSTEM_LIBGPIOD=1" is not specified and a valid "GPIOD_VERSION" is supplied then setup.py will first verify the requested version meets or exceeds "GPIOD_MINIMUM_VERSION".

If this check passes it will fetch the requested, stable tarball from mirrors.edge.kernel.org, verify the checksum, unpack the "lib" and "include" directories and vendor them into the output sdist.

It will also drop a "gpiod-version.txt" so that the sdist knows what GPIOD_VERSION_STR to pass into the gpiod build, eg:

GPIOD_VERSION="2.1.0" python3 -m build . --sdist

Will output dist/libgpiod-2.1.0.tar.gz vendoring libgpiod v2.1.0.

Runtime dependencies for fetching and vendoring a tarball are only imported when an sdist or build_ext command is triggered with "GPIOD_VERSION".

When a user builds or installs an sdist (via pip install or otherwise) if a "gpiod-version.txt" exists and "LINK_SYSTEM_LIBGPIOD=1" is not specified in their environment then the gpiod._ext C Extension is amended to include all of the vendored C sources for gpiod and the resulting module build will function independently of the system libgpiod.

Fetching libgpiod tarballs is an effort to reconsile the fact that both libgpiod and the Python bindings live in the same source tree but are versioned independently of each other. Bugfixes and changes to Python bindings should not vendor unstable, development versions of libgpiod.

No effort has been made to allow an unstable, vendored build since it's assumed developers will build and install libgpiod and use "LINK_SYSTEM_LIBGPIOD".

While the output sdist will - by default - build a standalone gpiod module it's possible for the end user to supply the "LINK_SYSTEM_LIBGPIOD=1" env variable to override this behaviour and attempt to link their system libgpiod (this will fail if it's incompatible with the bindings) eg:

LINK_SYSTEM_LIBPGIOD=1 pip install libgpiod

[1] - https://peps.python.org/pep-0668/
[2] - https://www.piwheels.org/

Changes v6 -> v7
- Added GPIOD_MINIMUM_VERSION to catch builds against incompatible libgpiod
- Set GPIOD_MINIMUM_VERSION to 2.1.0
- New requirement "packaging" added to pyproject.toml

Changes v5 -> v6
- Change SRC_BASE_URL to mirrors.edge.kernel.org
- Fetch sha256sums.asc and verify tarball signature
- Moved tarfile and urllib imports to fetch_tarball for fewer runtime deps
- Download and extract into a temporary directory
- Changed print to log.info
- Changed print/return errors to raise BaseError
- Check for lib/include in build tree and bail early

Changes v4 -> v5
- Move logic from main program flow to build_ext and sdist overloads
- GPIOD_VERSION_STR is now GPIOD_VERSION
- Fetch sources from a libgpiod tarball if GPIOD_VERSION is supplied
- Removed changes to bindings/python/Makefile.am
- Moved and rephrased comment about build_ext rmtree("tests")

Changes v3 -> v4
- Check for lack of "GPIOD_VERSION_STR" and revert to original behaviour
- Add status messages to setup.py, hinting at the package build mode

Changes v2 -> v3
- Pass in correct "GPIOD_VERSION_STR" from build
- Output an sdist build for pypi

Changes v1 -> v2
- Switch from symlinks to explicit file copy

Phil Howard (1):
bindings: python: optionally include module in sdist

 bindings/python/MANIFEST.in    |   5 +
 bindings/python/pyproject.toml |   2 +-
 bindings/python/setup.py       | 196 +++++++++++++++++++++++++++++++--
 3 files changed, 191 insertions(+), 12 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH v7 1/1 bindings: python: optionally include module in sdist

Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.

Add two new environment variables "LINK_SYSTEM_LIBGPIOD" and "GPIOD_VERSION" to control what kind of package setup.py will build.

In order to build an sdist or wheel package with a vendored libgpiod a version must be specified via the "GPIOD_VERSION" environment variable.

This will instruct setup.py to verify the given version against the list in sha256sums.asc and ensure it meets or exceeds a GPIOD_MINIMUM_VERSION required for compatibility with the bindings.

It will then fetch the tarball matching the requested version from mirrors.edge.kernel.org, verify the sha256 signature, unpack it, and copy the lib and include directories into the package root so they can be included in sdist or used to build a binary wheel.

eg: GPIOD_VERSION=2.1.0 python3 setup.py sdist

Will build a source distribution with gpiod version 2.1.0 source included.

It will also save the gpiod version into "gpiod-version.txt" so that it can be passed to the build when the sdist is built by pip.

Requiring an explicit version ensures that the Python bindings - which can be changed and versions independent of libgpiod - are built against a stable libgpiod release.

In order to force a package with vendored gpiod source to link the system libgpiod, the "LINK_SYSTEM_LIBGPIOD" environment variable can be used:

eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod

---
 bindings/python/MANIFEST.in    |   5 +
 bindings/python/pyproject.toml |   2 +-
 bindings/python/setup.py       | 196 +++++++++++++++++++++++++++++++--
 3 files changed, 191 insertions(+), 12 deletions(-)
bindings/python/MANIFEST.in +5 -0
index 459b317..66fe9b8 100644
@@ -3,6 +3,7 @@
 
 include setup.py
 include README.md
+include gpiod-version.txt
 
 recursive-include gpiod *.py
 recursive-include tests *.py
@@ -12,3 +13,7 @@ recursive-include gpiod/ext *.h
 
 recursive-include tests/gpiosim *.c
 recursive-include tests/procname *.c
+
+recursive-include lib *.c
+recursive-include lib *.h
+recursive-include include *.h
bindings/python/pyproject.toml +1 -1
index fcf6bbe..f6bf43c 100644
@@ -2,4 +2,4 @@
 # SPDX-FileCopyrightText: 2023 Phil Howard <[email protected]>
 
 [build-system]
-requires = ["setuptools", "wheel"]
+requires = ["setuptools", "wheel", "packaging"]
bindings/python/setup.py +185 -11

228 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw
Quoted 84 lines · Phil Howard

Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.

Add two new environment variables "LINK_SYSTEM_LIBGPIOD" and "GPIOD_VERSION" to control what kind of package setup.py will build.

In order to build an sdist or wheel package with a vendored libgpiod a version must be specified via the "GPIOD_VERSION" environment variable.

This will instruct setup.py to verify the given version against the list in sha256sums.asc and ensure it meets or exceeds a GPIOD_MINIMUM_VERSION required for compatibility with the bindings.

It will then fetch the tarball matching the requested version from mirrors.edge.kernel.org, verify the sha256 signature, unpack it, and copy the lib and include directories into the package root so they can be included in sdist or used to build a binary wheel.

eg: GPIOD_VERSION=2.1.0 python3 setup.py sdist

Will build a source distribution with gpiod version 2.1.0 source included.

It will also save the gpiod version into "gpiod-version.txt" so that it can be passed to the build when the sdist is built by pip.

Requiring an explicit version ensures that the Python bindings - which can be changed and versions independent of libgpiod - are built against a stable libgpiod release.

In order to force a package with vendored gpiod source to link the system libgpiod, the "LINK_SYSTEM_LIBGPIOD" environment variable can be used:

eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod

---
 bindings/python/MANIFEST.in    |   5 +
 bindings/python/pyproject.toml |   2 +-
 bindings/python/setup.py       | 196 +++++++++++++++++++++++++++++++--
 3 files changed, 191 insertions(+), 12 deletions(-)
bindings/python/MANIFEST.in +5 -0
index 459b317..66fe9b8 100644
@@ -3,6 +3,7 @@
 
 include setup.py
 include README.md
+include gpiod-version.txt
 
 recursive-include gpiod *.py
 recursive-include tests *.py
@@ -12,3 +13,7 @@ recursive-include gpiod/ext *.h
 
 recursive-include tests/gpiosim *.c
 recursive-include tests/procname *.c
+
+recursive-include lib *.c
+recursive-include lib *.h
+recursive-include include *.h
bindings/python/pyproject.toml +1 -1
index fcf6bbe..f6bf43c 100644
@@ -2,4 +2,4 @@
 # SPDX-FileCopyrightText: 2023 Phil Howard <[email protected]>
 
 [build-system]
-requires = ["setuptools", "wheel"]
+requires = ["setuptools", "wheel", "packaging"]
bindings/python/setup.py +2 -2
index c8db0a0..ac8b241 100644
@@ -1,24 +1,201 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 # SPDX-FileCopyrightText: 2022 Bartosz Golaszewski <[email protected]>
 
-from os import environ, path
-from setuptools import setup, Extension, find_packages
+from os import getenv, path, unlink
+from shutil import copytree, rmtree

bindings/python/setup.py line 14

Quoted 10 lines
bindings/python/setup.py +8 -1
+
+from setuptools import Extension, find_packages, setup
 from setuptools.command.build_ext import build_ext as orig_build_ext
-from shutil import rmtree
+from setuptools.command.sdist import log
+from setuptools.command.sdist import sdist as orig_sdist
+from setuptools.errors import BaseError
+
+LINK_SYSTEM_LIBGPIOD = getenv("LINK_SYSTEM_LIBGPIOD") == "1"
+GPIOD_MINIMUM_VERSION = "2.1.0"

I'm wondering how to handle it. We need these changes in the v2.1 release but without the release it will not work. I think we'll need to bite the bullet and just apply this, do the v2.1 libgpiod release, do the v2.1.0 python release and then upload to pypi.

bindings/python/setup.py line 15

Quoted 1 line
bindings/python/setup.py +1 -0
+GPIOD_VERSION = getenv("GPIOD_VERSION")

I thought I mentioned it before but I cannot find it now. Let's call this LIBGPIOD_VERSION and LIBGPIOD_MINIMUM_VERSION to emphasise that we mean the C library and not the wider GPIOD project.

bindings/python/setup.py line 16

Quoted 1 line
bindings/python/setup.py +1 -0
+GPIOD_WITH_TESTS = getenv("GPIOD_WITH_TESTS") == "1"

This is fine with the GPIOD_ prefix as it's a global setting in the project.

Quoted 23 lines
bindings/python/setup.py +23 -0
+SRC_BASE_URL = "https://mirrors.edge.kernel.org/pub/software/libs/libgpiod/"
+TAR_FILENAME = "libgpiod-{version}.tar.gz"
+ASC_FILENAME = "sha256sums.asc"
+SHA256_CHUNK_SIZE = 2048
+
+# __version__
+with open("gpiod/version.py", "r") as fd:
+    exec(fd.read())
+
+
+def sha256(filename):
+    """
+    Return a sha256sum for a specific filename, loading the file in chunks
+    to avoid potentially excessive memory use.
+    """
+    from hashlib import sha256
+
+    sha256sum = sha256()
+    with open(filename, "rb") as f:
+        for chunk in iter(lambda: f.read(SHA256_CHUNK_SIZE), b""):
+            sha256sum.update(chunk)
+
+    return sha256sum.hexdigest()

bindings/python/setup.py line 49

Quoted 10 lines
bindings/python/setup.py +10 -0
+
+
+def find_sha256sum(asc_file, tar_filename):
+    """
+    Search through a local copy of sha256sums.asc for a specific filename
+    and return the associated sha256 sum.
+    """
+    with open(asc_file, "r") as f:
+        for line in f:
+            if line[66:-1] == tar_filename:

I can't say I like it. Could you maybe try to split it first and then see if the second token exists and check if it equals tar_filename?

Quoted 178 lines
bindings/python/setup.py +140 -9
+                return line.split("  ")[0]
+
+    raise BaseError(f"no signature found for {tar_filename}")
+
+
+def fetch_tarball(command):
+    """
+    Verify the requested GPIOD_VERSION tarball exists in sha256sums.asc,
+    fetch it from https://mirrors.edge.kernel.org/pub/software/libs/libgpiod/
+    and verify its sha256sum.
+
+    If the check passes, extract the tarball and copy the lib and include
+    dirs into our source tree.
+    """
+
+    # If no GPIOD_VERSION is specified in env, just run the command
+    if GPIOD_VERSION is None:
+        return command
+
+    # If GPIOD_VERSION is specified, apply the tarball wrapper
+    def wrapper(self):
+        # Just-in-time import of tarfile and urllib.request so these are
+        # not required for Yocto to build a vendored or linked package
+        import tarfile
+        from tempfile import TemporaryDirectory
+        from urllib.request import urlretrieve
+
+        from packaging.version import Version
+
+        # Early exit for build tree with dirty lib/include dirs
+        for check_dir in "lib", "include":
+            if path.isdir(f"./{check_dir}"):
+                raise BaseError(f"refusing to overwrite ./{check_dir}")
+
+        with TemporaryDirectory(prefix="libgpiod-") as temp_dir:
+            tarball_filename = TAR_FILENAME.format(version=GPIOD_VERSION)
+            tarball_url = f"{SRC_BASE_URL}{tarball_filename}"
+            asc_url = f"{SRC_BASE_URL}{ASC_FILENAME}"
+
+            log.info(f"fetching: {asc_url}")
+
+            asc_filename, _ = urlretrieve(asc_url, path.join(temp_dir, ASC_FILENAME))
+
+            tarball_sha256 = find_sha256sum(asc_filename, tarball_filename)
+
+            if Version(GPIOD_VERSION) < Version(GPIOD_MINIMUM_VERSION):
+                raise BaseError(f"requires gpiod>={GPIOD_MINIMUM_VERSION}")
+
+            log.info(f"fetching: {tarball_url}")
+
+            downloaded_tarball, _ = urlretrieve(
+                tarball_url, path.join(temp_dir, tarball_filename)
+            )
+
+            log.info(f"verifying: {tarball_filename}")
+            if sha256(downloaded_tarball) != tarball_sha256:
+                raise BaseError(f"signature mismatch for {tarball_filename}")
+
+            # Unpack the downloaded tarball
+            log.info(f"unpacking: {tarball_filename}")
+            with tarfile.open(downloaded_tarball) as f:
+                f.extractall(temp_dir)
+
+            # Copy the include and lib directories we need to build libgpiod
+            base_dir = path.join(temp_dir, f"libgpiod-{GPIOD_VERSION}")
+            copytree(path.join(base_dir, "include"), "./include")
+            copytree(path.join(base_dir, "lib"), "./lib")
+
+        # Save the gpiod version for sdist
+        open("gpiod-version.txt", "w").write(GPIOD_VERSION)
+
+        # Run the command
+        command(self)
+
+        # Clean up the build directory
+        rmtree("./lib", ignore_errors=True)
+        rmtree("./include", ignore_errors=True)
+        unlink("gpiod-version.txt")
+
+    return wrapper
 
 
 class build_ext(orig_build_ext):
     """
-    setuptools install all C extentions even if they're excluded in setup().
-    As a workaround - remove the tests directory right after all extensions
-    were built (and possibly copied to the source directory if inplace is set).
+    Wrap build_ext to amend the module sources and settings to build
+    the bindings and gpiod into a combined module when a version is
+    specified and LINK_SYSTEM_LIBGPIOD=1 is not present in env.
+
+    run is wrapped with @fetch_tarball in order to fetch the sources
+    needed to build binary wheels when GPIOD_VERSION is specified, eg:
+
+    GPIOD_VERSION="2.0.2" python3 -m build .
     """
 
+    @fetch_tarball
     def run(self):
+        # Try to get the gpiod version from the .txt file included in sdist
+        try:
+            gpiod_version = open("gpiod-version.txt", "r").read()
+        except OSError:
+            gpiod_version = GPIOD_VERSION
+
+        if gpiod_version and not LINK_SYSTEM_LIBGPIOD:
+            # When building the extension from an sdist with a vendored
+            # amend gpiod._ext sources and settings accordingly.
+            gpiod_ext = self.ext_map["gpiod._ext"]
+            gpiod_ext.sources += [
+                "lib/chip.c",
+                "lib/chip-info.c",
+                "lib/edge-event.c",
+                "lib/info-event.c",
+                "lib/internal.c",
+                "lib/line-config.c",
+                "lib/line-info.c",
+                "lib/line-request.c",
+                "lib/line-settings.c",
+                "lib/misc.c",
+                "lib/request-config.c",
+            ]
+            gpiod_ext.libraries = []
+            gpiod_ext.include_dirs = ["include", "lib", "gpiod/ext"]
+            gpiod_ext.extra_compile_args.append(
+                f'-DGPIOD_VERSION_STR="{gpiod_version}"',
+            )
+
         super().run()
+
+        # We don't ever want the module tests directory in our package
+        # since this might include gpiosim._ext or procname._ext from a
+        # previous dirty build tree.
         rmtree(path.join(self.build_lib, "tests"), ignore_errors=True)
 
 
+class sdist(orig_sdist):
+    """
+    Wrap sdist in order to fetch the libgpiod source files for vendoring
+    into a source distribution.
+
+    run is wrapped with @fetch_tarball in order to fetch the sources
+    needed to build binary wheels when GPIOD_VERSION is specified, eg:
+
+    GPIOD_VERSION="2.0.2" python3 -m build . --sdist
+    """
+
+    @fetch_tarball
+    def run(self):
+        super().run()
+
+
 gpiod_ext = Extension(
     "gpiod._ext",
     sources=[
@@ -50,19 +227,16 @@ procname_ext = Extension(
 )
 
 extensions = [gpiod_ext]
-if environ.get("GPIOD_WITH_TESTS") == "1":
+if GPIOD_WITH_TESTS:
     extensions.append(gpiosim_ext)
     extensions.append(procname_ext)
 
-with open("gpiod/version.py", "r") as fd:
-    exec(fd.read())
-
 setup(
     name="gpiod",
     packages=find_packages(exclude=["tests", "tests.*"]),
     python_requires=">=3.9.0",
     ext_modules=extensions,
-    cmdclass={"build_ext": build_ext},
+    cmdclass={"build_ext": build_ext, "sdist": sdist},
     version=__version__,
     author="Bartosz Golaszewski",
     author_email="[email protected]",
--
Quoted 2 lines

2.34.1

Other than the above, this looks pretty nice. I think we're getting there.

Bart

#3 lore raw
Quoted 93 lines · Bartosz Golaszewski
Quoted 91 lines · Phil Howard

Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.

Add two new environment variables "LINK_SYSTEM_LIBGPIOD" and "GPIOD_VERSION" to control what kind of package setup.py will build.

In order to build an sdist or wheel package with a vendored libgpiod a version must be specified via the "GPIOD_VERSION" environment variable.

This will instruct setup.py to verify the given version against the list in sha256sums.asc and ensure it meets or exceeds a GPIOD_MINIMUM_VERSION required for compatibility with the bindings.

It will then fetch the tarball matching the requested version from mirrors.edge.kernel.org, verify the sha256 signature, unpack it, and copy the lib and include directories into the package root so they can be included in sdist or used to build a binary wheel.

eg: GPIOD_VERSION=2.1.0 python3 setup.py sdist

Will build a source distribution with gpiod version 2.1.0 source included.

It will also save the gpiod version into "gpiod-version.txt" so that it can be passed to the build when the sdist is built by pip.

Requiring an explicit version ensures that the Python bindings - which can be changed and versions independent of libgpiod - are built against a stable libgpiod release.

In order to force a package with vendored gpiod source to link the system libgpiod, the "LINK_SYSTEM_LIBGPIOD" environment variable can be used:

eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod

---
 bindings/python/MANIFEST.in    |   5 +
 bindings/python/pyproject.toml |   2 +-
 bindings/python/setup.py       | 196 +++++++++++++++++++++++++++++++--
 3 files changed, 191 insertions(+), 12 deletions(-)
bindings/python/MANIFEST.in +5 -0
index 459b317..66fe9b8 100644
@@ -3,6 +3,7 @@
 
 include setup.py
 include README.md
+include gpiod-version.txt
 
 recursive-include gpiod *.py
 recursive-include tests *.py
@@ -12,3 +13,7 @@ recursive-include gpiod/ext *.h
 
 recursive-include tests/gpiosim *.c
 recursive-include tests/procname *.c
+
+recursive-include lib *.c
+recursive-include lib *.h
+recursive-include include *.h
bindings/python/pyproject.toml +1 -1
index fcf6bbe..f6bf43c 100644
@@ -2,4 +2,4 @@
 # SPDX-FileCopyrightText: 2023 Phil Howard <[email protected]>
 
 [build-system]
-requires = ["setuptools", "wheel"]
+requires = ["setuptools", "wheel", "packaging"]
bindings/python/setup.py +7 -3
index c8db0a0..ac8b241 100644
@@ -1,24 +1,201 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 # SPDX-FileCopyrightText: 2022 Bartosz Golaszewski <[email protected]>
 
-from os import environ, path
-from setuptools import setup, Extension, find_packages
+from os import getenv, path, unlink
+from shutil import copytree, rmtree
+
+from setuptools import Extension, find_packages, setup
 from setuptools.command.build_ext import build_ext as orig_build_ext
-from shutil import rmtree
+from setuptools.command.sdist import log
+from setuptools.command.sdist import sdist as orig_sdist
+from setuptools.errors import BaseError
Quoted 8 lines

bindings/python/setup.py line 14

Quoted 3 lines
bindings/python/setup.py +3 -0
+
+LINK_SYSTEM_LIBGPIOD = getenv("LINK_SYSTEM_LIBGPIOD") == "1"
+GPIOD_MINIMUM_VERSION = "2.1.0"

I'm wondering how to handle it. We need these changes in the v2.1 release but without the release it will not work. I think we'll need to bite the bullet and just apply this, do the v2.1 libgpiod release, do the v2.1.0 python release and then upload to pypi.

It's a bit of a chicken and egg problem. But yes, we'll have to release then follow
with the pypi release once the package is available to download and build the sdist.

Quoted 6 lines

bindings/python/setup.py line 15

Quoted 2 lines
bindings/python/setup.py +1 -0
+GPIOD_VERSION = getenv("GPIOD_VERSION")

I thought I mentioned it before but I cannot find it now. Let's call this LIBGPIOD_VERSION and LIBGPIOD_MINIMUM_VERSION to emphasise that we mean the C library and not the wider GPIOD project.

Will change!

Quoted 33 lines

bindings/python/setup.py line 16

Quoted 2 lines
bindings/python/setup.py +1 -0
+GPIOD_WITH_TESTS = getenv("GPIOD_WITH_TESTS") == "1"

This is fine with the GPIOD_ prefix as it's a global setting in the project.

Quoted 28 lines
bindings/python/setup.py +28 -0
+SRC_BASE_URL = "https://mirrors.edge.kernel.org/pub/software/libs/libgpiod/"
+TAR_FILENAME = "libgpiod-{version}.tar.gz"
+ASC_FILENAME = "sha256sums.asc"
+SHA256_CHUNK_SIZE = 2048
+
+# __version__
+with open("gpiod/version.py", "r") as fd:
+    exec(fd.read())
+
+
+def sha256(filename):
+    """
+    Return a sha256sum for a specific filename, loading the file in chunks
+    to avoid potentially excessive memory use.
+    """
+    from hashlib import sha256
+
+    sha256sum = sha256()
+    with open(filename, "rb") as f:
+        for chunk in iter(lambda: f.read(SHA256_CHUNK_SIZE), b""):
+            sha256sum.update(chunk)
+
+    return sha256sum.hexdigest()
+
+
+def find_sha256sum(asc_file, tar_filename):
+    """
+    Search through a local copy of sha256sums.asc for a specific filename
Quoted 8 lines

bindings/python/setup.py line 49

Quoted 5 lines
bindings/python/setup.py +5 -0
+    and return the associated sha256 sum.
+    """
+    with open(asc_file, "r") as f:
+        for line in f:
+            if line[66:-1] == tar_filename:

I can't say I like it. Could you maybe try to split it first and then see if the second token exists and check if it equals tar_filename?

Will change! I didn't expect the length of a sha256sum to change but yes the strip and split method is a little bit clearer about intent.

Quoted 184 lines
Quoted 179 lines
bindings/python/setup.py +140 -9
+                return line.split("  ")[0]
+
+    raise BaseError(f"no signature found for {tar_filename}")
+
+
+def fetch_tarball(command):
+    """
+    Verify the requested GPIOD_VERSION tarball exists in sha256sums.asc,
+    fetch it from https://mirrors.edge.kernel.org/pub/software/libs/libgpiod/
+    and verify its sha256sum.
+
+    If the check passes, extract the tarball and copy the lib and include
+    dirs into our source tree.
+    """
+
+    # If no GPIOD_VERSION is specified in env, just run the command
+    if GPIOD_VERSION is None:
+        return command
+
+    # If GPIOD_VERSION is specified, apply the tarball wrapper
+    def wrapper(self):
+        # Just-in-time import of tarfile and urllib.request so these are
+        # not required for Yocto to build a vendored or linked package
+        import tarfile
+        from tempfile import TemporaryDirectory
+        from urllib.request import urlretrieve
+
+        from packaging.version import Version
+
+        # Early exit for build tree with dirty lib/include dirs
+        for check_dir in "lib", "include":
+            if path.isdir(f"./{check_dir}"):
+                raise BaseError(f"refusing to overwrite ./{check_dir}")
+
+        with TemporaryDirectory(prefix="libgpiod-") as temp_dir:
+            tarball_filename = TAR_FILENAME.format(version=GPIOD_VERSION)
+            tarball_url = f"{SRC_BASE_URL}{tarball_filename}"
+            asc_url = f"{SRC_BASE_URL}{ASC_FILENAME}"
+
+            log.info(f"fetching: {asc_url}")
+
+            asc_filename, _ = urlretrieve(asc_url, path.join(temp_dir, ASC_FILENAME))
+
+            tarball_sha256 = find_sha256sum(asc_filename, tarball_filename)
+
+            if Version(GPIOD_VERSION) < Version(GPIOD_MINIMUM_VERSION):
+                raise BaseError(f"requires gpiod>={GPIOD_MINIMUM_VERSION}")
+
+            log.info(f"fetching: {tarball_url}")
+
+            downloaded_tarball, _ = urlretrieve(
+                tarball_url, path.join(temp_dir, tarball_filename)
+            )
+
+            log.info(f"verifying: {tarball_filename}")
+            if sha256(downloaded_tarball) != tarball_sha256:
+                raise BaseError(f"signature mismatch for {tarball_filename}")
+
+            # Unpack the downloaded tarball
+            log.info(f"unpacking: {tarball_filename}")
+            with tarfile.open(downloaded_tarball) as f:
+                f.extractall(temp_dir)
+
+            # Copy the include and lib directories we need to build libgpiod
+            base_dir = path.join(temp_dir, f"libgpiod-{GPIOD_VERSION}")
+            copytree(path.join(base_dir, "include"), "./include")
+            copytree(path.join(base_dir, "lib"), "./lib")
+
+        # Save the gpiod version for sdist
+        open("gpiod-version.txt", "w").write(GPIOD_VERSION)
+
+        # Run the command
+        command(self)
+
+        # Clean up the build directory
+        rmtree("./lib", ignore_errors=True)
+        rmtree("./include", ignore_errors=True)
+        unlink("gpiod-version.txt")
+
+    return wrapper
 
 
 class build_ext(orig_build_ext):
     """
-    setuptools install all C extentions even if they're excluded in setup().
-    As a workaround - remove the tests directory right after all extensions
-    were built (and possibly copied to the source directory if inplace is set).
+    Wrap build_ext to amend the module sources and settings to build
+    the bindings and gpiod into a combined module when a version is
+    specified and LINK_SYSTEM_LIBGPIOD=1 is not present in env.
+
+    run is wrapped with @fetch_tarball in order to fetch the sources
+    needed to build binary wheels when GPIOD_VERSION is specified, eg:
+
+    GPIOD_VERSION="2.0.2" python3 -m build .
     """
 
+    @fetch_tarball
     def run(self):
+        # Try to get the gpiod version from the .txt file included in sdist
+        try:
+            gpiod_version = open("gpiod-version.txt", "r").read()
+        except OSError:
+            gpiod_version = GPIOD_VERSION
+
+        if gpiod_version and not LINK_SYSTEM_LIBGPIOD:
+            # When building the extension from an sdist with a vendored
+            # amend gpiod._ext sources and settings accordingly.
+            gpiod_ext = self.ext_map["gpiod._ext"]
+            gpiod_ext.sources += [
+                "lib/chip.c",
+                "lib/chip-info.c",
+                "lib/edge-event.c",
+                "lib/info-event.c",
+                "lib/internal.c",
+                "lib/line-config.c",
+                "lib/line-info.c",
+                "lib/line-request.c",
+                "lib/line-settings.c",
+                "lib/misc.c",
+                "lib/request-config.c",
+            ]
+            gpiod_ext.libraries = []
+            gpiod_ext.include_dirs = ["include", "lib", "gpiod/ext"]
+            gpiod_ext.extra_compile_args.append(
+                f'-DGPIOD_VERSION_STR="{gpiod_version}"',
+            )
+
         super().run()
+
+        # We don't ever want the module tests directory in our package
+        # since this might include gpiosim._ext or procname._ext from a
+        # previous dirty build tree.
         rmtree(path.join(self.build_lib, "tests"), ignore_errors=True)
 
 
+class sdist(orig_sdist):
+    """
+    Wrap sdist in order to fetch the libgpiod source files for vendoring
+    into a source distribution.
+
+    run is wrapped with @fetch_tarball in order to fetch the sources
+    needed to build binary wheels when GPIOD_VERSION is specified, eg:
+
+    GPIOD_VERSION="2.0.2" python3 -m build . --sdist
+    """
+
+    @fetch_tarball
+    def run(self):
+        super().run()
+
+
 gpiod_ext = Extension(
     "gpiod._ext",
     sources=[
@@ -50,19 +227,16 @@ procname_ext = Extension(
 )
 
 extensions = [gpiod_ext]
-if environ.get("GPIOD_WITH_TESTS") == "1":
+if GPIOD_WITH_TESTS:
     extensions.append(gpiosim_ext)
     extensions.append(procname_ext)
 
-with open("gpiod/version.py", "r") as fd:
-    exec(fd.read())
-
 setup(
     name="gpiod",
     packages=find_packages(exclude=["tests", "tests.*"]),
     python_requires=">=3.9.0",
     ext_modules=extensions,
-    cmdclass={"build_ext": build_ext},
+    cmdclass={"build_ext": build_ext, "sdist": sdist},
     version=__version__,
     author="Bartosz Golaszewski",
     author_email="[email protected]",
--
Quoted 2 lines

2.34.1

Other than the above, this looks pretty nice. I think we're getting there.

Quoted 1 line

Bart

Whew, it's been a journey!