Skip to contentSkip to conversation

bindings: python: optionally include (...)

PATCH Applied Phil Howard 2 participants 6 messages Oct 12, 2023 – Oct 13, 2023 GPIO

v2 of 9 Applied Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v2 cover bindings: python: optionally include (...)

This changeset vendors the gpiod library into the Python package, and adds a pyproject.toml for minimum compatibility with modern Python packaging.

Why?

So that setup.py can produce an sdist that is installable irrespective of the availability or version of a distro-supplied libgpiod.

This prevents a libgpiod pypi package install balking because the distro libgpiod is outdated or otherwise incompatible. This happens when attempting to install the current libgpiod from pypi onto - for example - the Debian Bookworm based Raspberry Pi OS.

The availability of a distro agnostic package also ensures that libgpiod can be installed via pypi into an isolated virtual environment, safely specified as a dependency for Python packages and allows Python developers to target the newest API version irrespective of their distro supplied libgpiod.

This is essential, since a venv is now widely *required* for user Python projects due to recommendations in pep-688 - https://peps.python.org/pep-0668/

For Raspberry Pi this sdist can also be converted into a precompiled wheel by piwheels [1] which is, by default, added to Raspberry Pi OS as a pip index.

How?

If "LINK_SYSTEM_LIBGPIOD=1" is not specified then the gpiod._ext C Extension is amended to include all of the C sources for gpiod, so it can be built as a standalone module without depending upon a shared distro library.

The gpiod sources are included by copying the lib and include directories up to the parent module, and updating MANIFEST.in to include the source files when an sdist is built.

The resulting source distribution can then be uploaded to pypi and from there can be built and installed by any user with python3-dev installed.

[1] - https://www.piwheels.org/

Phil Howard (2):
bindings: python: optionally include module in sdist
bindings: python: add pyproject.toml, pep 518

 bindings/python/MANIFEST.in    |  4 ++
 bindings/python/pyproject.toml |  5 ++
 bindings/python/setup.py       | 95 ++++++++++++++++++++++++++++------
 3 files changed, 89 insertions(+), 15 deletions(-)
 create mode 100644 bindings/python/pyproject.toml
Signature · 1 line
2.34.1
#1 lore raw

PATCH v2 1/2 bindings: python: optionally include module in sdist

Build libgpiod into Python module for build_ext or bdist_wheel.

Include libgpiod source in sdist so that the Python module can be built from source by end users, even with a missing or mismatched system libgpiod.

Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" to generate a module via build_ext or bdist_wheel that links against system libgpiod.

---
 bindings/python/MANIFEST.in |  4 ++
 bindings/python/setup.py    | 95 +++++++++++++++++++++++++++++++------
 2 files changed, 84 insertions(+), 15 deletions(-)
bindings/python/MANIFEST.in +4 -0
index c7124d4..eff8977 100644
@@ -11,3 +11,7 @@ recursive-include gpiod/ext *.h
 
 recursive-include tests/gpiosim *.c
 recursive-include tests/procname *.c
+
+recursive-include lib *.c
+recursive-include lib *.h
+recursive-include include *.h
bindings/python/setup.py +80 -15

126 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw

Re: [libgpiod][PATCH v2 1/2] bindings: python: optionally include module in sdist

Quoted 131 lines · Phil Howard

Build libgpiod into Python module for build_ext or bdist_wheel.

Include libgpiod source in sdist so that the Python module can be built from source by end users, even with a missing or mismatched system libgpiod.

Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" to generate a module via build_ext or bdist_wheel that links against system libgpiod.

---
 bindings/python/MANIFEST.in |  4 ++
 bindings/python/setup.py    | 95 +++++++++++++++++++++++++++++++------
 2 files changed, 84 insertions(+), 15 deletions(-)
bindings/python/MANIFEST.in +4 -0
index c7124d4..eff8977 100644
@@ -11,3 +11,7 @@ recursive-include gpiod/ext *.h
 
 recursive-include tests/gpiosim *.c
 recursive-include tests/procname *.c
+
+recursive-include lib *.c
+recursive-include lib *.h
+recursive-include include *.h
bindings/python/setup.py +71 -9
index fd674aa..e3b571c 100644
@@ -4,7 +4,30 @@
 from os import environ, path
 from setuptools import setup, Extension, find_packages
 from setuptools.command.build_ext import build_ext as orig_build_ext
-from shutil import rmtree
+from setuptools.command.sdist import sdist as orig_sdist
+from shutil import rmtree, copytree
+
+
+def copy_libgpiod_files(func):
+    """
+    In order to include the lib and include directories in the sdist
+    we must temporarily copy them up into the python bindings directory.
+
+    If "./lib" exists we are building from an sdist package and will not
+    try to copy the files again.
+    """
+
+    def wrapper(self):
+        copy_src = not path.exists("./lib")
+        if copy_src:
+            copytree("../../lib", "./lib")
+            copytree("../../include", "./include")
+        func(self)
+        if copy_src:
+            rmtree("./lib")
+            rmtree("./include")
+
+    return wrapper
 
 
 class build_ext(orig_build_ext):
@@ -14,24 +37,69 @@ class build_ext(orig_build_ext):
     were built (and possibly copied to the source directory if inplace is set).
     """
 
+    @copy_libgpiod_files
     def run(self):
         super().run()
         rmtree(path.join(self.build_lib, "tests"), ignore_errors=True)
 
 
+class sdist(orig_sdist):
+    """
+    Wrap sdist so that we can copy the lib and include files into . where
+    MANIFEST.in will include them in the source package.
+    """
+
+    @copy_libgpiod_files
+    def run(self):
+        super().run()
+
+
+with open("gpiod/version.py", "r") as fd:
+    exec(fd.read())
+
+sources = [
+    # gpiod Python bindings
+    "gpiod/ext/chip.c",
+    "gpiod/ext/common.c",
+    "gpiod/ext/line-config.c",
+    "gpiod/ext/line-settings.c",
+    "gpiod/ext/module.c",
+    "gpiod/ext/request.c",
+]
+
+if environ.get("LINK_SYSTEM_LIBGPIOD") == "1":
+    libraries = ["gpiod"]
+    include_dirs = ["gpiod"]
+else:
+    sources += [
+        # gpiod library
+        "lib/chip.c",
+        "lib/chip-info.c",
+        "lib/edge-event.c",
+        "lib/info-event.c",
+        "lib/internal.c",
+        "lib/line-config.c",
+        "lib/line-info.c",
+        "lib/line-request.c",
+        "lib/line-settings.c",
+        "lib/misc.c",
+        "lib/request-config.c",
+    ]
+    libraries = []
+    include_dirs = ["include", "lib", "gpiod/ext"]
+
+
 gpiod_ext = Extension(
     "gpiod._ext",
-    sources=[
-        "gpiod/ext/chip.c",
-        "gpiod/ext/common.c",
-        "gpiod/ext/line-config.c",
-        "gpiod/ext/line-settings.c",
-        "gpiod/ext/module.c",
-        "gpiod/ext/request.c",
-    ],

bindings/python/setup.py class build_ext(orig_build_ext): · line 101

Quoted 10 lines
bindings/python/setup.py +7 -2
+    libraries=libraries,
+    sources=sources,
     define_macros=[("_GNU_SOURCE", "1")],
-    libraries=["gpiod"],
-    extra_compile_args=["-Wall", "-Wextra"],
+    include_dirs=include_dirs,
+    extra_compile_args=[
+        "-Wall",
+        "-Wextra",
+        '-DGPIOD_VERSION_STR="{}"'.format(__version__),

This is incorrect. The version we want to pass here is the version of libgpiod, not the version of python bindings which is now decoupled from the former.

I'm not sure how to correctly do it. We can read the contents of configure.ac for one and get it from there. Maybe you have a better idea.

Bart

Quoted 24 lines
bindings/python/setup.py +2 -5
+    ],
 )
 
 gpiosim_ext = Extension(
@@ -54,15 +122,12 @@ if "GPIOD_WITH_TESTS" in environ and environ["GPIOD_WITH_TESTS"] == "1":
     extensions.append(gpiosim_ext)
     extensions.append(procname_ext)
 
-with open("gpiod/version.py", "r") as fd:
-    exec(fd.read())
-
 setup(
     name="libgpiod",
     packages=find_packages(exclude=["tests", "tests.*"]),
     python_requires=">=3.9.0",
     ext_modules=extensions,
-    cmdclass={"build_ext": build_ext},
+    cmdclass={"build_ext": build_ext, "sdist": sdist},
     version=__version__,
     author="Bartosz Golaszewski",
     author_email="[email protected]",
--

2.34.1

#3 lore raw

Re: [libgpiod][PATCH v2 1/2] bindings: python: optionally include module in sdist

Quoted 148 lines · Bartosz Golaszewski
Quoted 131 lines · Phil Howard

Build libgpiod into Python module for build_ext or bdist_wheel.

Include libgpiod source in sdist so that the Python module can be built from source by end users, even with a missing or mismatched system libgpiod.

Add optional environment variable "LINK_SYSTEM_LIBGPIOD=1" to generate a module via build_ext or bdist_wheel that links against system libgpiod.

---
 bindings/python/MANIFEST.in |  4 ++
 bindings/python/setup.py    | 95 +++++++++++++++++++++++++++++++------
 2 files changed, 84 insertions(+), 15 deletions(-)
bindings/python/MANIFEST.in +4 -0
index c7124d4..eff8977 100644
@@ -11,3 +11,7 @@ recursive-include gpiod/ext *.h
 
 recursive-include tests/gpiosim *.c
 recursive-include tests/procname *.c
+
+recursive-include lib *.c
+recursive-include lib *.h
+recursive-include include *.h
bindings/python/setup.py +71 -9
index fd674aa..e3b571c 100644
@@ -4,7 +4,30 @@
 from os import environ, path
 from setuptools import setup, Extension, find_packages
 from setuptools.command.build_ext import build_ext as orig_build_ext
-from shutil import rmtree
+from setuptools.command.sdist import sdist as orig_sdist
+from shutil import rmtree, copytree
+
+
+def copy_libgpiod_files(func):
+    """
+    In order to include the lib and include directories in the sdist
+    we must temporarily copy them up into the python bindings directory.
+
+    If "./lib" exists we are building from an sdist package and will not
+    try to copy the files again.
+    """
+
+    def wrapper(self):
+        copy_src = not path.exists("./lib")
+        if copy_src:
+            copytree("../../lib", "./lib")
+            copytree("../../include", "./include")
+        func(self)
+        if copy_src:
+            rmtree("./lib")
+            rmtree("./include")
+
+    return wrapper
 
 
 class build_ext(orig_build_ext):
@@ -14,24 +37,69 @@ class build_ext(orig_build_ext):
     were built (and possibly copied to the source directory if inplace is set).
     """
 
+    @copy_libgpiod_files
     def run(self):
         super().run()
         rmtree(path.join(self.build_lib, "tests"), ignore_errors=True)
 
 
+class sdist(orig_sdist):
+    """
+    Wrap sdist so that we can copy the lib and include files into . where
+    MANIFEST.in will include them in the source package.
+    """
+
+    @copy_libgpiod_files
+    def run(self):
+        super().run()
+
+
+with open("gpiod/version.py", "r") as fd:
+    exec(fd.read())
+
+sources = [
+    # gpiod Python bindings
+    "gpiod/ext/chip.c",
+    "gpiod/ext/common.c",
+    "gpiod/ext/line-config.c",
+    "gpiod/ext/line-settings.c",
+    "gpiod/ext/module.c",
+    "gpiod/ext/request.c",
+]
+
+if environ.get("LINK_SYSTEM_LIBGPIOD") == "1":
+    libraries = ["gpiod"]
+    include_dirs = ["gpiod"]
+else:
+    sources += [
+        # gpiod library
+        "lib/chip.c",
+        "lib/chip-info.c",
+        "lib/edge-event.c",
+        "lib/info-event.c",
+        "lib/internal.c",
+        "lib/line-config.c",
+        "lib/line-info.c",
+        "lib/line-request.c",
+        "lib/line-settings.c",
+        "lib/misc.c",
+        "lib/request-config.c",
+    ]
+    libraries = []
+    include_dirs = ["include", "lib", "gpiod/ext"]
+
+
 gpiod_ext = Extension(
     "gpiod._ext",
-    sources=[
-        "gpiod/ext/chip.c",
-        "gpiod/ext/common.c",
-        "gpiod/ext/line-config.c",
-        "gpiod/ext/line-settings.c",
-        "gpiod/ext/module.c",
-        "gpiod/ext/request.c",
-    ],

bindings/python/setup.py class build_ext(orig_build_ext): · line 101

Quoted 10 lines
bindings/python/setup.py +7 -2
+    libraries=libraries,
+    sources=sources,
     define_macros=[("_GNU_SOURCE", "1")],
-    libraries=["gpiod"],
-    extra_compile_args=["-Wall", "-Wextra"],
+    include_dirs=include_dirs,
+    extra_compile_args=[
+        "-Wall",
+        "-Wextra",
+        '-DGPIOD_VERSION_STR="{}"'.format(__version__),

This is incorrect. The version we want to pass here is the version of libgpiod, not the version of python bindings which is now decoupled from the former.

Quoted 3 lines

I'm not sure how to correctly do it. We can read the contents of configure.ac for one and get it from there. Maybe you have a better idea.

Since configure.ac wont be available in the sdist package we'll need to copy the version number somewhere.

Since we should probably be plumbing this into autoconf anyway I think this var should be passed in the same way as GPIOD_WITH_TESTS, then setup.py can grab it from environ with a failover to loading from a packaged "gpiod-version-str.txt".

A little out of my wheelhouse, but I'll take a shot.

Quoted 27 lines

Bart

Quoted 24 lines
bindings/python/setup.py +2 -5
+    ],
 )
 
 gpiosim_ext = Extension(
@@ -54,15 +122,12 @@ if "GPIOD_WITH_TESTS" in environ and environ["GPIOD_WITH_TESTS"] == "1":
     extensions.append(gpiosim_ext)
     extensions.append(procname_ext)
 
-with open("gpiod/version.py", "r") as fd:
-    exec(fd.read())
-
 setup(
     name="libgpiod",
     packages=find_packages(exclude=["tests", "tests.*"]),
     python_requires=">=3.9.0",
     ext_modules=extensions,
-    cmdclass={"build_ext": build_ext},
+    cmdclass={"build_ext": build_ext, "sdist": sdist},
     version=__version__,
     author="Bartosz Golaszewski",
     author_email="[email protected]",
--

2.34.1

Signature · 10 lines
Philip Howard
Technology & Lifestyle Writer
gadgetoid.com

Gadgetoid gadg-et-oid [gaj-it-oid]

                                     -adjective

1. having the characteristics or form of a gadget; resembling a
mechanical contrivance or device.
#4 lore raw

PATCH v2 2/2 bindings: python: add pyproject.toml, pep 518

In reply to Phil Howard

Add pyproject.toml to prevent spurious deprecation warnings from pip.

---
 bindings/python/pyproject.toml | 5 +++++
 1 file changed, 5 insertions(+)
 create mode 100644 bindings/python/pyproject.toml
bindings/python/pyproject.toml new file +5 -0
new file mode 100644
index 0000000..fcf6bbe
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+# SPDX-FileCopyrightText: 2023 Phil Howard <[email protected]>
+
+[build-system]
+requires = ["setuptools", "wheel"]
Signature · 1 line
2.34.1
#5 lore raw

Re: [libgpiod][PATCH v2 2/2] bindings: python: add pyproject.toml, pep 518

Quoted 21 lines · Phil Howard

Add pyproject.toml to prevent spurious deprecation warnings from pip.

---
 bindings/python/pyproject.toml | 5 +++++
 1 file changed, 5 insertions(+)
 create mode 100644 bindings/python/pyproject.toml
bindings/python/pyproject.toml new file +5 -1
new file mode 100644
index 0000000..fcf6bbe
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+# SPDX-FileCopyrightText: 2023 Phil Howard <[email protected]>
+
+[build-system]
+requires = ["setuptools", "wheel"]
--
Quoted 2 lines

2.34.1

Applied, thanks!

Bart