bindings: python: optionally include module in sdist
v9 cover bindings: python: optionally include module in sdist
This changeset vendors the gpiod library into the Python package.
Why?
So that setup.py can produce an sdist that is installable irrespective of the availability or version of a distro-supplied libgpiod.
This prevents a libgpiod pypi package install balking because the distro libgpiod is outdated or otherwise incompatible. This happens when attempting to install the current libgpiod from pypi onto - for example - the Debian Bookworm based Raspberry Pi OS or Ubuntu 23.10 Mantic which both ship with libgpiod v1.6.3.
The availability of a distro agnostic package also ensures that libgpiod can be installed via pypi into an isolated virtual environment, safely specified as a dependency for Python packages and allows Python developers to target the newest API version irrespective of their distro supplied libgpiod.
This is essential, since a venv is now widely *required* for user Python projects due to recommendations in pep-688 [1]
For Raspberry Pi this sdist can also be converted into a precompiled wheel by piwheels [2] which is, by default, added to Raspberry Pi OS as a pip index.
How?
If "LINK_SYSTEM_LIBGPIOD=1" is not specified and a valid "LIBGPIOD_VERSION" is supplied then setup.py will first verify the requested version meets or exceeds "GPIOD_MINIMUM_VERSION".
If this check passes it will fetch the requested, stable tarball from mirrors.edge.kernel.org, verify the checksum, unpack the "lib" and "include" directories and vendor them into the output sdist.
It will also drop a "libgpiod-version.txt" so that the sdist knows what GPIOD_VERSION_STR to pass into the gpiod build, eg:
LIBGPIOD_VERSION="2.1.0" python3 -m build . --sdist
Will output dist/libgpiod-2.1.0.tar.gz vendoring libgpiod v2.1.0.
If "libgpiod-version.txt" exists and the version in "libgpiod-version.txt" matches the requested LIBGPIOD_VERSION then lib and include are assumed to already exist and the tarball fetch will be skipped. This prevents build from trying to fetch the tarball twice when called with:
LIBGPIOD_VERSION="2.1.0" python3 -m build .
Runtime dependencies for fetching and vendoring a tarball are only imported when an sdist or build_ext command is triggered with "LIBGPIOD_VERSION".
When a user builds or installs an sdist (via pip install or otherwise) if a "libgpiod-version.txt" exists and "LINK_SYSTEM_LIBGPIOD=1" is not specified in their environment then the gpiod._ext C Extension is amended to include all of the vendored C sources for gpiod and the resulting module build will function independently of the system libgpiod.
Fetching libgpiod tarballs is an effort to reconsile the fact that both libgpiod and the Python bindings live in the same source tree but are versioned independently of each other. Bugfixes and changes to Python bindings should not vendor unstable, development versions of libgpiod.
No effort has been made to allow an unstable, vendored build since it's assumed developers will build and install libgpiod and use "LINK_SYSTEM_LIBGPIOD".
While the output sdist will - by default - build a standalone gpiod module it's possible for the end user to supply the "LINK_SYSTEM_LIBGPIOD=1" env variable to override this behaviour and attempt to link their system libgpiod (this will fail if it's incompatible with the bindings) eg:
LINK_SYSTEM_LIBPGIOD=1 pip install libgpiod
[1] - https://peps.python.org/pep-0668/
[2] - https://www.piwheels.org/
Changes v8 -> v9
- Removed Version testing hack
Changes v7 -> v8
- Tweak sha256sums parsing for better clarity
- Add check for libgpiod-version.txt to prevent "build" trying to refetch
- Added a URL to metadata to stop setup.py complaining
- Change GPIOD_MINIMUM_VERSION to LIBGPIOD_MINIMUM_VERSION
- Change GPIOD_VERSION to LIBGPIOD_VERSION
Changes v6 -> v7
- Added GPIOD_MINIMUM_VERSION to catch builds against incompatible libgpiod
- Set GPIOD_MINIMUM_VERSION to 2.1.0
- New requirement "packaging" added to pyproject.toml
Changes v5 -> v6
- Change SRC_BASE_URL to mirrors.edge.kernel.org
- Fetch sha256sums.asc and verify tarball signature
- Moved tarfile and urllib imports to fetch_tarball for fewer runtime deps
- Download and extract into a temporary directory
- Changed print to log.info
- Changed print/return errors to raise BaseError
- Check for lib/include in build tree and bail early
Changes v4 -> v5
- Move logic from main program flow to build_ext and sdist overloads
- GPIOD_VERSION_STR is now GPIOD_VERSION
- Fetch sources from a libgpiod tarball if GPIOD_VERSION is supplied
- Removed changes to bindings/python/Makefile.am
- Moved and rephrased comment about build_ext rmtree("tests")
Changes v3 -> v4
- Check for lack of "GPIOD_VERSION_STR" and revert to original behaviour
- Add status messages to setup.py, hinting at the package build mode
Changes v2 -> v3
- Pass in correct "GPIOD_VERSION_STR" from build
- Output an sdist build for pypi
Changes v1 -> v2
- Switch from symlinks to explicit file copy
Phil Howard (1):
bindings: python: optionally include module in sdist
bindings/python/MANIFEST.in | 5 + bindings/python/pyproject.toml | 2 +- bindings/python/setup.py | 212 +++++++++++++++++++++++++++++++-- 3 files changed, 207 insertions(+), 12 deletions(-)
Signature · 1 line
2.34.1
PATCH v9 1/1 bindings: python: optionally include module in sdist
Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.
Add two new environment variables "LINK_SYSTEM_LIBGPIOD" and "LIBGPIOD_VERSION" to control what kind of package setup.py will build.
In order to build an sdist or wheel package with a vendored libgpiod a version must be specified via the "LIBGPIOD_VERSION" environment variable.
This will instruct setup.py to verify the given version against the list in sha256sums.asc and ensure it meets or exceeds a LIBGPIOD_MINIMUM_VERSION required for compatibility with the bindings.
It will then fetch the tarball matching the requested version from mirrors.edge.kernel.org, verify the sha256 signature, unpack it, and copy the lib and include directories into the package root so they can be included in sdist or used to build a binary wheel.
eg: LIBGPIOD_VERSION=2.1.0 python3 setup.py sdist
Will build a source distribution with gpiod version 2.1.0 source included.
It will also save the gpiod version into "libgpiod-version.txt" so that it can be passed to the build when the sdist is built by pip.
Requiring an explicit version ensures that the Python bindings - which can be changed and versions independent of libgpiod - are built against a stable libgpiod release.
In order to force a package with vendored gpiod source to link the system libgpiod, the "LINK_SYSTEM_LIBGPIOD" environment variable can be used:
eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod
--- bindings/python/MANIFEST.in | 5 + bindings/python/pyproject.toml | 2 +- bindings/python/setup.py | 212 +++++++++++++++++++++++++++++++-- 3 files changed, 207 insertions(+), 12 deletions(-)
bindings/python/MANIFEST.in +5 -0
@@ -3,6 +3,7 @@ include setup.py include README.md +include libgpiod-version.txt recursive-include gpiod *.py recursive-include tests *.py
@@ -12,3 +13,7 @@ recursive-include gpiod/ext *.h recursive-include tests/gpiosim *.c recursive-include tests/procname *.c + +recursive-include lib *.c +recursive-include lib *.h +recursive-include include *.h
bindings/python/pyproject.toml +1 -1
@@ -2,4 +2,4 @@ # SPDX-FileCopyrightText: 2023 Phil Howard <[email protected]> [build-system] -requires = ["setuptools", "wheel"] +requires = ["setuptools", "wheel", "packaging"]
bindings/python/setup.py +201 -11
244 lines not on this page · raw message
Signature · 1 line
2.34.1
Quoted 30 lines · Phil Howard
Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.
Add two new environment variables "
LINK_SYSTEM_LIBGPIOD" and "LIBGPIOD_VERSION" to control what kind of packagesetup.pywill build.In order to build an sdist or wheel package with a vendored libgpiod a version must be specified via the "
LIBGPIOD_VERSION" environment variable.This will instruct
setup.pyto verify the given version against the list in sha256sums.asc and ensure it meets or exceeds aLIBGPIOD_MINIMUM_VERSIONrequired for compatibility with the bindings.It will then fetch the tarball matching the requested version from mirrors.edge.kernel.org, verify the sha256 signature, unpack it, and copy the lib and include directories into the package root so they can be included in sdist or used to build a binary wheel.
eg: LIBGPIOD_VERSION=2.1.0 python3
setup.pysdistWill build a source distribution with gpiod version 2.1.0 source included.
It will also save the gpiod version into "libgpiod-version.txt" so that it can be passed to the build when the sdist is built by pip.
Requiring an explicit version ensures that the Python bindings - which can be changed and versions independent of libgpiod - are built against a stable libgpiod release.
Quoted 7 lines
In order to force a package with vendored gpiod source to link the system libgpiod, the "
LINK_SYSTEM_LIBGPIOD" environment variable can be used:eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod
Signed-off-by Phil Howard <[email protected]>---
This now looks good to me. I'll leave it here until Friday and if there are no objections (Kent, would you mind reviewing this one?), I'll apply it to the master branch, do the v2.1 release of libgpiod, then create the v2.1.0 release of python bindings (with sdist packaging libgpiod v2.1) and upload it to gpiod @ pypi. Hopefully this will work correctly.
Bart
Quoted 5 lines · Bartosz Golaszewski
Quoted 1 line · Phil Howard
This now looks good to me. I'll leave it here until Friday and if there are no objections (Kent, would you mind reviewing this one?),
I've got nothing to add.
Cheers,
Kent.
Quoted 5 lines · Kent Gibson
Quoted 3 lines · Bartosz Golaszewski
Quoted 1 line · Phil Howard
Quoted 8 lines
Quoted 2 lines
This now looks good to me. I'll leave it here until Friday and if there are no objections (Kent, would you mind reviewing this one?),
I've got nothing to add.
Cheers,
Kent.
Ok, I will actually delay it for a week until next Friday because I'll be off Mon-Thu and I don't want to leave stuff broken if anything goes wrong.
Bart
So, today's the day?
What's the plan?
Quoted 21 lines · Bartosz Golaszewski
Quoted 5 lines · Kent Gibson
Quoted 3 lines · Bartosz Golaszewski
Quoted 1 line · Phil Howard
Quoted 8 lines
Quoted 2 lines
This now looks good to me. I'll leave it here until Friday and if there are no objections (Kent, would you mind reviewing this one?),
I've got nothing to add.
Cheers,
Kent.Ok, I will actually delay it for a week until next Friday because I'll be off Mon-Thu and I don't want to leave stuff broken if anything goes wrong.
Bart
Signature · 10 lines
Philip Howard
Technology & Lifestyle Writer
gadgetoid.com
Gadgetoid gadg-et-oid [gaj-it-oid]
-adjective
1. having the characteristics or form of a gadget; resembling a
mechanical contrivance or device.Quoted 5 lines · Phil Howard
So, today's the day?
What's the plan?
Yes! I'll apply this, release libgpiod v2.1, generate the python v2.1.0 release from it and upload it to pypi - this time to the gpiod package.
Bart
Quoted 36 lines · Bartosz Golaszewski
Quoted 16 lines · Kent Gibson
Quoted 13 lines · Bartosz Golaszewski
Quoted 5 lines · Phil Howard
Quoted 1 line
This now looks good to me. I'll leave it here until Friday and if there are no objections (Kent, would you mind reviewing this one?),
I've got nothing to add.
Cheers,
Kent.Quoted 5 lines
Ok, I will actually delay it for a week until next Friday because I'll be off Mon-Thu and I don't want to leave stuff broken if anything goes wrong.
Bart
--
Philip Howard
Technology & Lifestyle Writer
gadgetoid.comGadgetoid gadg-et-oid [gaj-it-oid]
-adjective
1. having the characteristics or form of a gadget; resembling a mechanical contrivance or device.
A quick aside, if modified to run on your system, does this script segfault?
```
import gpiod
from gpiod.line import Direction, Value
a = gpiod.request_lines("/dev/gpiochip0", consumer="test", config={
23: gpiod.LineSettings(direction=Direction.OUTPUT,
output_value=Value.INACTIVE)
})
b = gpiod.request_lines("/dev/gpiochip0", consumer="test", config={
24: gpiod.LineSettings(direction=Direction.OUTPUT,
output_value=Value.INACTIVE)
})
a.release()
b.release()
```Quoted 50 lines · Bartosz Golaszewski
Quoted 5 lines · Phil Howard
So, today's the day?
What's the plan?
Yes! I'll apply this, release libgpiod v2.1, generate the python v2.1.0 release from it and upload it to pypi - this time to the gpiod package.
Bart
Quoted 36 lines · Bartosz Golaszewski
Quoted 16 lines · Kent Gibson
Quoted 13 lines · Bartosz Golaszewski
Quoted 5 lines · Phil Howard
Quoted 1 line
This now looks good to me. I'll leave it here until Friday and if there are no objections (Kent, would you mind reviewing this one?),
I've got nothing to add.
Cheers,
Kent.Quoted 5 lines
Ok, I will actually delay it for a week until next Friday because I'll be off Mon-Thu and I don't want to leave stuff broken if anything goes wrong.
Bart
--
Philip Howard
Technology & Lifestyle Writer
gadgetoid.comGadgetoid gadg-et-oid [gaj-it-oid]
-adjective
1. having the characteristics or form of a gadget; resembling a mechanical contrivance or device.
Quoted 13 lines · Phil Howard
A quick aside, if modified to run on your system, does this script segfault?
``` import gpiod from gpiod.line import Direction, Value a = gpiod.request_lines("/dev/gpiochip0", consumer="test", config={ 23: gpiod.LineSettings(direction=Direction.OUTPUT, output_value=Value.INACTIVE) })
Quoted 8 lines
b = gpiod.request_lines("/dev/gpiochip0", consumer="test", config={
24: gpiod.LineSettings(direction=Direction.OUTPUT,
output_value=Value.INACTIVE)
})
a.release()
b.release()
```Nope, works fine. What is the stack trace?
Bartosz
Quoted 53 lines · Bartosz Golaszewski
Quoted 50 lines · Phil Howard
Quoted 5 lines
So, today's the day?
What's the plan?
Yes! I'll apply this, release libgpiod v2.1, generate the python v2.1.0 release from it and upload it to pypi - this time to the gpiod package.
Bart
Quoted 36 lines · Bartosz Golaszewski
Quoted 16 lines · Kent Gibson
Quoted 13 lines · Bartosz Golaszewski
Quoted 5 lines · Phil Howard
Quoted 1 line
This now looks good to me. I'll leave it here until Friday and if there are no objections (Kent, would you mind reviewing this one?),
I've got nothing to add.
Cheers,
Kent.Quoted 5 lines
Ok, I will actually delay it for a week until next Friday because I'll be off Mon-Thu and I don't want to leave stuff broken if anything goes wrong.
Bart
--
Philip Howard
Technology & Lifestyle Writer
gadgetoid.comGadgetoid gadg-et-oid [gaj-it-oid]
-adjective
1. having the characteristics or form of a gadget; resembling a mechanical contrivance or device.