bindings: python: optionally include module in sdist
v4 cover bindings: python: optionally include module in sdist
This changeset vendors the gpiod library into the Python package.
Why?
So that setup.py can produce an sdist that is installable irrespective of the availability or version of a distro-supplied libgpiod.
This prevents a libgpiod pypi package install balking because the distro libgpiod is outdated or otherwise incompatible. This happens when attempting to install the current libgpiod from pypi onto - for example - the Debian Bookworm based Raspberry Pi OS or Ubuntu 23.10 Mantic which both ship with libgpiod v1.6.3.
The availability of a distro agnostic package also ensures that libgpiod can be installed via pypi into an isolated virtual environment, safely specified as a dependency for Python packages and allows Python developers to target the newest API version irrespective of their distro supplied libgpiod.
This is essential, since a venv is now widely *required* for user Python projects due to recommendations in pep-688 [1]
For Raspberry Pi this sdist can also be converted into a precompiled wheel by piwheels [2] which is, by default, added to Raspberry Pi OS as a pip index.
How?
If "LINK_SYSTEM_LIBGPIOD=1" is not specified and "GPIOD_VERSION_STR" is supplied then the gpiod._ext C Extension is amended to include all of the C sources for gpiod, so it can be built as a standalone module without depending upon a shared distro library.
The gpiod sources are included by copying the "lib" and "include" directories up to the parent module, and updating "MANIFEST.in" to include the source files when an sdist is built. Additionally the gpiod version string is included in "libgpiod-version-str.txt" so that it is available for source builds.
bindings/python/Makefile.am has been extended so that "make" will now produce a .tar.gz source distribution in bindings/python/dist which is suitable for uploading to pypi and can be built and installed by any user with python3-dev installed.
[1] - https://peps.python.org/pep-0668/
[2] - https://www.piwheels.org/
Changes v3 -> v4
- Check for lack of "GPIOD_VERSION_STR" and revert to original behaviour
- Add status messages to setup.py, hinting at the package build mode
Changes v2 -> v3
- Pass in correct "GPIOD_VERSION_STR" from build
- Output an sdist build for pypi
Changes v1 -> v2
- Switch from symlinks to explicit file copy
Phil Howard (1):
bindings: python: optionally include module in sdist
bindings/python/MANIFEST.in | 5 ++ bindings/python/Makefile.am | 3 + bindings/python/setup.py | 122 +++++++++++++++++++++++++++++++----- 3 files changed, 114 insertions(+), 16 deletions(-)
Signature · 1 line
2.34.1
PATCH v4 1/1 bindings: python: optionally include module in sdist
Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.
Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" so that the sdist package can optionally be built and linked against a compatible system libgpiod.
eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod
Update build to add an additional sdist target for upload to pypi. Call setup.py with "GPIOD_VERSION_STR" set, which triggers a vendored package build. "GPIOD_VERSION_STR" is saved to "gpiod-version-str.txt" and included in the sdist for standalone builds.
"GPIOD_VERSION_STR" must be specified in order to produce a standalone buildable sdist package, this requirement implicitly preserves the old build behaviour.
--- bindings/python/MANIFEST.in | 5 ++ bindings/python/Makefile.am | 3 + bindings/python/setup.py | 122 +++++++++++++++++++++++++++++++----- 3 files changed, 114 insertions(+), 16 deletions(-)
bindings/python/MANIFEST.in +5 -0
@@ -2,6 +2,7 @@ # SPDX-FileCopyrightText: 2023 Bartosz Golaszewski <[email protected]> include setup.py +include gpiod-version-str.txt recursive-include gpiod *.py recursive-include tests *.py
@@ -11,3 +12,7 @@ recursive-include gpiod/ext *.h recursive-include tests/gpiosim *.c recursive-include tests/procname *.c + +recursive-include lib *.c +recursive-include lib *.h +recursive-include include *.h
bindings/python/Makefile.am +3 -0
@@ -16,6 +16,9 @@ all-local: $(PYTHON) setup.py build_ext --inplace \ --include-dirs=$(top_srcdir)/include/:$(top_srcdir)/tests/gpiosim/ \ --library-dirs=$(top_builddir)/lib/.libs/:$(top_srcdir)/tests/gpiosim/.libs/ + GPIOD_VERSION_STR=$(VERSION_STR) \ + $(PYTHON) setup.py sdist + install-exec-local: GPIOD_WITH_TESTS= \
bindings/python/setup.py +106 -16
155 lines not on this page · raw message
Signature · 1 line
2.34.1
Quoted 17 lines · Phil Howard
Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.
Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" so that the sdist package can optionally be built and linked against a compatible system libgpiod.
eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod
Update build to add an additional sdist target for upload to pypi. Call
setup.pywith "GPIOD_VERSION_STR" set, which triggers a vendored package build. "GPIOD_VERSION_STR" is saved to "gpiod-version-str.txt" and included in the sdist for standalone builds.
Quoted 6 lines
"
GPIOD_VERSION_STR" must be specified in order to produce a standalone buildable sdist package, this requirement implicitly preserves the old build behaviour.Signed-off-by Phil Howard <[email protected]>---
Alright, this looks and works well. Before I merge it: the package still shows up in 'pip freeze' as libgpiod. Now that we control the name 'gpiod' on pypi, should we change it to 'gpiod' instead?
Once I apply the patch - how would we go about making a new proper release? Do you also want to add a better homepage to setup.py so that users of pypi don't bounce off the nonexistent documentation?
Bart
Quoted 23 lines · Bartosz Golaszewski
Quoted 21 lines · Phil Howard
Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.
Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" so that the sdist package can optionally be built and linked against a compatible system libgpiod.
eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod
Update build to add an additional sdist target for upload to pypi. Call
setup.pywith "GPIOD_VERSION_STR" set, which triggers a vendored package build. "GPIOD_VERSION_STR" is saved to "gpiod-version-str.txt" and included in the sdist for standalone builds."
GPIOD_VERSION_STR" must be specified in order to produce a standalone buildable sdist package, this requirement implicitly preserves the old build behaviour.
Quoted 6 lines
Quoted 2 lines
Signed-off-by Phil Howard <[email protected]>---
Alright, this looks and works well. Before I merge it: the package still shows up in 'pip freeze' as libgpiod. Now that we control the name 'gpiod' on pypi, should we change it to 'gpiod' instead?
Yes, it's conventional to have the module and package name match for single module packages.
Quoted 2 lines
Once I apply the patch - how would we go about making a new proper release?
The "libgpiod-2.0.1.tar.gz" (or "gpiod-2.0.1.tar.gz" if we change it) should be uploaded to pypi via the utility "twine" (pip install twine). You can upload versions to testpypi by specifying "--repository testpypi" which, combined with a "-postN" version suffix, can help work through bugs in packaging or README formatting.
You can also run "twine check dist/*" to precheck for packaging issues.
I'm not sure if we should yank prior releases of the Python gpiod but some note in the README paying lip service to the change from unofficial Python to official C-bindings would be wise.
Quoted 2 lines
Do you also want to add a better homepage to
setup.pyso that users of pypi don't bounce off the nonexistent documentation?
Yes, we need a basic introduction and a couple of succinct examples.
I have some half done edits to the README somewhere locally which I was holding back until after we got the packaging working and until I could ask: What format would you prefer for the README?
Quoted 2 lines
Bart
Quoted 38 lines · Phil Howard
Quoted 23 lines · Bartosz Golaszewski
Quoted 21 lines · Phil Howard
Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.
Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" so that the sdist package can optionally be built and linked against a compatible system libgpiod.
eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod
Update build to add an additional sdist target for upload to pypi. Call
setup.pywith "GPIOD_VERSION_STR" set, which triggers a vendored package build. "GPIOD_VERSION_STR" is saved to "gpiod-version-str.txt" and included in the sdist for standalone builds."
GPIOD_VERSION_STR" must be specified in order to produce a standalone buildable sdist package, this requirement implicitly preserves the old build behaviour.Quoted 6 lines
Quoted 2 lines
Signed-off-by Phil Howard <[email protected]>---
Alright, this looks and works well. Before I merge it: the package still shows up in 'pip freeze' as libgpiod. Now that we control the name 'gpiod' on pypi, should we change it to 'gpiod' instead?
Yes, it's conventional to have the module and package name match for single module packages.
Quoted 2 lines
Once I apply the patch - how would we go about making a new proper release?
Quoted 7 lines
The "libgpiod-2.0.1.tar.gz" (or "gpiod-2.0.1.tar.gz" if we change it) should be uploaded to pypi via the utility "twine" (pip install twine). You can upload versions to testpypi by specifying "--repository testpypi" which, combined with a "-postN" version suffix, can help work through bugs in packaging or README formatting.
You can also run "twine check dist/*" to precheck for packaging issues.
Sure, this is how I uploaded the existing releases. I was thinking more about versioning. We haven't added new interfaces to the code so I think the version should be set to v2.0.2 as technically this is a bug-fix release + build improvements.
Quoted 5 lines
I'm not sure if we should yank prior releases of the Python gpiod but some note in the README paying lip service to the change from unofficial Python to official C-bindings would be wise.
You mean out of the "gpiod" repo, not "libgpiod"? Because I assume this is where we'll upload the new release.
Quoted 9 lines
Quoted 2 lines
Do you also want to add a better homepage to
setup.pyso that users of pypi don't bounce off the nonexistent documentation?Yes, we need a basic introduction and a couple of succinct examples.
I have some half done edits to the README somewhere locally which I was holding back until after we got the packaging working and until I could ask: What format would you prefer for the README?
Whatever works best or is the standard for pypi. You are the expert. :)
Bart
Quoted 2 lines
Quoted 2 lines
Bart
Quoted 46 lines · Bartosz Golaszewski
Quoted 44 lines · Phil Howard
Quoted 23 lines · Bartosz Golaszewski
Quoted 21 lines · Phil Howard
Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.
Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" so that the sdist package can optionally be built and linked against a compatible system libgpiod.
eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod
Update build to add an additional sdist target for upload to pypi. Call
setup.pywith "GPIOD_VERSION_STR" set, which triggers a vendored package build. "GPIOD_VERSION_STR" is saved to "gpiod-version-str.txt" and included in the sdist for standalone builds."
GPIOD_VERSION_STR" must be specified in order to produce a standalone buildable sdist package, this requirement implicitly preserves the old build behaviour.Quoted 6 lines
Quoted 2 lines
Signed-off-by Phil Howard <[email protected]>---
Alright, this looks and works well. Before I merge it: the package still shows up in 'pip freeze' as libgpiod. Now that we control the name 'gpiod' on pypi, should we change it to 'gpiod' instead?
Yes, it's conventional to have the module and package name match for single module packages.
Quoted 2 lines
Once I apply the patch - how would we go about making a new proper release?
The "libgpiod-2.0.1.tar.gz" (or "gpiod-2.0.1.tar.gz" if we change it) should be uploaded to pypi via the utility "twine" (pip install twine). You can upload versions to testpypi by specifying "--repository testpypi" which, combined with a "-postN" version suffix, can help work through bugs in packaging or README formatting.
Quoted 6 lines
Quoted 1 line
You can also run "twine check dist/*" to precheck for packaging issues.
Sure, this is how I uploaded the existing releases. I was thinking more about versioning. We haven't added new interfaces to the code so I think the version should be set to v2.0.2 as technically this is a bug-fix release + build improvements.
Agreed.
Quoted 8 lines
Quoted 5 lines
I'm not sure if we should yank prior releases of the Python gpiod but some note in the README paying lip service to the change from unofficial Python to official C-bindings would be wise.
You mean out of the "gpiod" repo, not "libgpiod"? Because I assume this is where we'll upload the new release.
Yes since new packages will be uploaded to the "gpiod" pypi repo they are going to be seen as an "update" to the old "gpiod".
Something like "For the old unofficial Python gpiod install with:
pip install gpiod==1.5.4" should suffice.
There's good evidence [1] that gpiod is at least moderately popular so we should expect some minor fallout from people who refuse to pin their deps.
[1] - https://pypistats.org/packages/gpiod
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Do you also want to add a better homepage to
setup.pyso that users of pypi don't bounce off the nonexistent documentation?
Quoted 8 lines
Quoted 6 lines
Yes, we need a basic introduction and a couple of succinct examples.
I have some half done edits to the README somewhere locally which I was holding back until after we got the packaging working and until I could ask: What format would you prefer for the README?
Whatever works best or is the standard for pypi. You are the expert. :)
I prefer Markdown because it's what I know and use. I don't believe any specific format is preferred these days- though the fallback default is 'text/x-rst'
Quoted 5 lines
Bart
Quoted 2 lines
Quoted 2 lines
Bart
Quoted 53 lines · Phil Howard
Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.
Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" so that the sdist package can optionally be built and linked against a compatible system libgpiod.
eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod
Update build to add an additional sdist target for upload to pypi. Call
setup.pywith "GPIOD_VERSION_STR" set, which triggers a vendored package build. "GPIOD_VERSION_STR" is saved to "gpiod-version-str.txt" and included in the sdist for standalone builds."
GPIOD_VERSION_STR" must be specified in order to produce a standalone buildable sdist package, this requirement implicitly preserves the old build behaviour.Signed-off-by Phil Howard <[email protected]>--- bindings/python/MANIFEST.in | 5 ++ bindings/python/Makefile.am | 3 + bindings/python/setup.py | 122 +++++++++++++++++++++++++++++++----- 3 files changed, 114 insertions(+), 16 deletions(-)bindings/python/MANIFEST.in +5
-0@@ -2,6 +2,7 @@ # SPDX-FileCopyrightText: 2023 Bartosz Golaszewski <[email protected]> include setup.py +include gpiod-version-str.txt recursive-include gpiod *.py recursive-include tests *.py@@ -11,3 +12,7 @@ recursive-include gpiod/ext *.h recursive-include tests/gpiosim *.c recursive-include tests/procname *.c + +recursive-include lib *.c +recursive-include lib *.h +recursive-include include *.hbindings/python/Makefile.am +0
-0
bindings/python/Makefile.am all-local: · line 19
Quoted 4 lines
bindings/python/Makefile.am +1
-0@@ -16,6 +16,9 @@ all-local: $(PYTHON) setup.py build_ext --inplace \ --include-dirs=$(top_srcdir)/include/:$(top_srcdir)/tests/gpiosim/ \ --library-dirs=$(top_builddir)/lib/.libs/:$(top_srcdir)/tests/gpiosim/.libs/ + GPIOD_VERSION_STR=$(VERSION_STR) \
I suppose I can remove this line now?
Bart
[snip]
Quoted 53 lines · Bartosz Golaszewski
Quoted 51 lines · Phil Howard
Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.
Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" so that the sdist package can optionally be built and linked against a compatible system libgpiod.
eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod
Update build to add an additional sdist target for upload to pypi. Call
setup.pywith "GPIOD_VERSION_STR" set, which triggers a vendored package build. "GPIOD_VERSION_STR" is saved to "gpiod-version-str.txt" and included in the sdist for standalone builds."
GPIOD_VERSION_STR" must be specified in order to produce a standalone buildable sdist package, this requirement implicitly preserves the old build behaviour.Signed-off-by Phil Howard <[email protected]>--- bindings/python/MANIFEST.in | 5 ++ bindings/python/Makefile.am | 3 + bindings/python/setup.py | 122 +++++++++++++++++++++++++++++++----- 3 files changed, 114 insertions(+), 16 deletions(-)bindings/python/MANIFEST.in +5
-0@@ -2,6 +2,7 @@ # SPDX-FileCopyrightText: 2023 Bartosz Golaszewski <[email protected]> include setup.py +include gpiod-version-str.txt recursive-include gpiod *.py recursive-include tests *.py@@ -11,3 +12,7 @@ recursive-include gpiod/ext *.h recursive-include tests/gpiosim *.c recursive-include tests/procname *.c + +recursive-include lib *.c +recursive-include lib *.h +recursive-include include *.hbindings/python/Makefile.am +0
-1--- a/bindings/python/Makefile.am
Quoted 8 lines
bindings/python/Makefile.am all-local: · line 19
Quoted 6 lines
+++ b/bindings/python/Makefile.am
bindings/python/Makefile.am +1
-0@@ -16,6 +16,9 @@ all-local: $(PYTHON) setup.py build_ext --inplace \ --include-dirs=$(top_srcdir)/include/:$(top_srcdir)/tests/gpiosim/ \ --library-dirs=$(top_builddir)/lib/.libs/:$(top_srcdir)/tests/gpiosim/.libs/ + GPIOD_VERSION_STR=$(VERSION_STR) \I suppose I can remove this line now?
This is right before a separate invocation of the build, which might want to be split into another Makefile target for clarity.
This is required to build an sdist package which includes a valid "gpiod-version-str.txt" and can be built standalone, but doesn't change the original "build_ext" call.
Quoted 4 lines
Bart
[snip]
Quoted 17 lines · Phil Howard
Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.
Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" so that the sdist package can optionally be built and linked against a compatible system libgpiod.
eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod
Update build to add an additional sdist target for upload to pypi. Call
setup.pywith "GPIOD_VERSION_STR" set, which triggers a vendored package build. "GPIOD_VERSION_STR" is saved to "gpiod-version-str.txt" and included in the sdist for standalone builds.
Quoted 6 lines
"
GPIOD_VERSION_STR" must be specified in order to produce a standalone buildable sdist package, this requirement implicitly preserves the old build behaviour.Signed-off-by Phil Howard <[email protected]>---
I just realized one issue with this. Even if we indicate the version of libgpiod as v2.0.2 using the environment variable but actually create it from the master branch then the build-in code will really be the development branch not v2.0.2 stable release.
This is precisely the kind of issue that linking to system lib allows to avoid. :)
Anyway, maybe we would need an additional environment variable that could be used to point setup.py to the actual source tree we want to use, which may be different from the tree we're using to generate the sdist package? Typically we'd want to release the pypi package with the most recent stable release of libgpiod.
Bart
Quoted 34 lines · Bartosz Golaszewski
Quoted 17 lines · Phil Howard
Optionally vendor libgpiod source into sdist so that the Python module can be built from source, even with a missing or mismatched system libgpiod.
Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" so that the sdist package can optionally be built and linked against a compatible system libgpiod.
eg: LINK_SYSTEM_LIBGPIOD=1 pip install libgpiod
Update build to add an additional sdist target for upload to pypi. Call
setup.pywith "GPIOD_VERSION_STR" set, which triggers a vendored package build. "GPIOD_VERSION_STR" is saved to "gpiod-version-str.txt" and included in the sdist for standalone builds.Quoted 6 lines
"
GPIOD_VERSION_STR" must be specified in order to produce a standalone buildable sdist package, this requirement implicitly preserves the old build behaviour.Signed-off-by Phil Howard <[email protected]>---
I just realized one issue with this. Even if we indicate the version of libgpiod as v2.0.2 using the environment variable but actually create it from the master branch then the build-in code will really be the development branch not v2.0.2 stable release.
This is precisely the kind of issue that linking to system lib allows to avoid. :)
Quoted 7 lines
Anyway, maybe we would need an additional environment variable that could be used to point
setup.pyto the actual source tree we want to use, which may be different from the tree we're using to generate the sdist package? Typically we'd want to release the pypi package with the most recent stable release of libgpiod.Bart
Or we could even do it online and fetch the libgpiod tarball from the official repo[1] at kernel.org?
Bart
[1] https://mirrors.edge.kernel.org/pub/software/libs/libgpiod/