Skip to contentSkip to conversation

bindings: python: optionally include module (...)

PATCH Applied Phil Howard 3 participants 10 messages Oct 11, 2023 – Oct 12, 2023 GPIO

v1 of 9 Applied Latest is v9 →

lore
Share & tools
lore
#0 lore raw

cover bindings: python: optionally include module (...)

This changeset vendors the gpiod library into the Python package, adds a pyproject.toml for minimum compatibility with modern Python packaging standards and sets "python_requires" to 3.9, reflecting the minimum version required by these bindings as established in commit <b436d05809b1> ("bindings:
python: replace PyModule_AddObjectRef() with PyModule_AddObject()").

Why?

So that setup.py can produce an sdist that is installable irrespective of the availability or version of a distro-supplied libgpiod.

This prevents a libgpiod pypi package install balking because the distro libgpiod is outdated or otherwise incompatible. This happens when attempting to install the current libgpiod from pypi onto - for example - the Debian Bookworm based Raspberry Pi OS.

The availability of a distro agnostic package also ensures that libgpiod can be installed via pypi into an isolated virtual environment, safely specified as a dependency for Python packages and allows Python developers to target the newest API version irrespective of their distro supplied libgpiod.

This is essential, since a venv is now widely *required* for user Python projects due to recommendations in pep-688 - https://peps.python.org/pep-0668/

For Raspberry Pi this sdist can also be converted into a precompiled wheel by piwheels [1] which is, by default, added to Raspberry Pi OS as a pip index.

How?

If "USE_SYSTEM_GPIOD=1" is not specified then the gpiod._ext C Extension is amended to include all of the C sources for gpiod, so it can be built as a standalone module without depending upon a shared distro library.

The gpiod sources are included by symlinking the lib and include directories up to the parent module, and updating MANIFEST.in to include the source files when an sdist is built.

The resulting source distribution can then be uploaded to pypi and from there can be built and installed by any user with python3-dev installed.

[1] - https://www.piwheels.org/

Phil Howard (3):
bindings: python: optionally include module in sdist
bindings: python: add pyproject.toml, pep 518
bindings: python: require python 3.9.0

 bindings/python/MANIFEST.in    |  4 +++
 bindings/python/include        |  1 +
 bindings/python/lib            |  1 +
 bindings/python/pyproject.toml |  2 ++
 bindings/python/setup.py       | 58 ++++++++++++++++++++++++++--------
 5 files changed, 53 insertions(+), 13 deletions(-)
 create mode 120000 bindings/python/include
 create mode 120000 bindings/python/lib
 create mode 100644 bindings/python/pyproject.toml
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/3 bindings: python: optionally include module in sdist

Build gpiod into Python module.

Optional environment var USE_SYSTEM_GPIO=1 to
generate a module that depends upon system gpiod.

---
 bindings/python/MANIFEST.in |  4 +++
 bindings/python/include     |  1 +
 bindings/python/lib         |  1 +
 bindings/python/setup.py    | 57 ++++++++++++++++++++++++++++---------
 4 files changed, 50 insertions(+), 13 deletions(-)
 create mode 120000 bindings/python/include
 create mode 120000 bindings/python/lib
bindings/python/MANIFEST.in +4 -0
index c7124d4..eff8977 100644
@@ -11,3 +11,7 @@ recursive-include gpiod/ext *.h
 
 recursive-include tests/gpiosim *.c
 recursive-include tests/procname *.c
+
+recursive-include lib *.c
+recursive-include lib *.h
+recursive-include include *.h
bindings/python/include new file +1 -0
new file mode 120000
index 0000000..fcffffb
@@ -0,0 +1 @@
+../../include
\ No newline at end of file
bindings/python/lib new file +1 -0
new file mode 120000
index 0000000..58677dd
@@ -0,0 +1 @@
+../../lib
\ No newline at end of file
bindings/python/setup.py +44 -13

72 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw

Re: [libgpiod][PATCH 1/3] bindings: python: optionally include module in sdist

Quoted 4 lines · Phil Howard

Build gpiod into Python module.

Optional environment var USE_SYSTEM_GPIO=1 to
generate a module that depends upon system gpiod.

...

Quoted 6 lines
bindings/python/include new file +1 -0
@@ -0,0 +1 @@
+../../include

\ No newline at end of file

These lines are bothering me, why the new line can't be added to all affected files?

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#3 lore raw

Re: [libgpiod][PATCH 1/3] bindings: python: optionally include module in sdist

Quoted 8 lines · Andy Shevchenko
Quoted 4 lines · Phil Howard

Build gpiod into Python module.

Optional environment var USE_SYSTEM_GPIO=1 to
generate a module that depends upon system gpiod.

...

*sigh*

Quoted 9 lines
Quoted 7 lines
bindings/python/include new file +1 -0
@@ -0,0 +1 @@
+../../include

\ No newline at end of file

These lines are bothering me, why the new line can't be added to all affected files?

Is it convention for symlinks to include a newline, is it even possible?

I'm not super sure about the symlink approach, actually.

It's the path of least complexity but after some research into the usage of "shutil.copytree"
to copy dependent and packaged files at build time I'm starting to wonder if a more complex
setup.py isn't necessarily a bad idea.

Quoted 6 lines

--
With Best Regards,
Andy Shevchenko

Signature · 10 lines
Philip Howard
Technology & Lifestyle Writer
gadgetoid.com

Gadgetoid gadg-et-oid [gaj-it-oid]

                                     -adjective

1. having the characteristics or form of a gadget; resembling a
mechanical contrivance or device.
#4 lore raw

Re: [libgpiod][PATCH 1/3] bindings: python: optionally include module in sdist

Quoted 26 lines · Phil Howard
Quoted 8 lines · Andy Shevchenko
Quoted 4 lines · Phil Howard

Build gpiod into Python module.

Optional environment var USE_SYSTEM_GPIO=1 to
generate a module that depends upon system gpiod.

...

*sigh*

Quoted 9 lines
Quoted 7 lines
bindings/python/include new file +1 -0
@@ -0,0 +1 @@
+../../include

\ No newline at end of file

These lines are bothering me, why the new line can't be added to all affected files?

Is it convention for symlinks to include a newline, is it even possible?

Quoted 8 lines

I'm not super sure about the symlink approach, actually.

It's the path of least complexity but after some research into the usage of "shutil.copytree"
to copy dependent and packaged files at build time I'm starting to wonder if a more complex
setup.py isn't necessarily a bad idea.

How about not having links in the repo but creating them as required at build-time?

Bart

Quoted 19 lines
Quoted 6 lines

--
With Best Regards,
Andy Shevchenko

--
Philip Howard
Technology & Lifestyle Writer
gadgetoid.com

Gadgetoid gadg-et-oid [gaj-it-oid]

-adjective

1. having the characteristics or form of a gadget; resembling a mechanical contrivance or device.

#5 lore raw

Re: [libgpiod][PATCH 1/3] bindings: python: optionally include module in sdist

In reply to Phil Howard
Quoted 4 lines · Phil Howard

Build gpiod into Python module.

Optional environment var USE_SYSTEM_GPIO=1 to

Let's call it USE_SYSTEM_GPIOD or USE_SYSTEM_LIBGPIOD or even LINK_SYSTEM_LIBGPIOD?

Quoted 30 lines

generate a module that depends upon system gpiod.

---
 bindings/python/MANIFEST.in |  4 +++
 bindings/python/include     |  1 +
 bindings/python/lib         |  1 +
 bindings/python/setup.py    | 57 ++++++++++++++++++++++++++++---------
 4 files changed, 50 insertions(+), 13 deletions(-)
 create mode 120000 bindings/python/include
 create mode 120000 bindings/python/lib
bindings/python/MANIFEST.in +4 -0
index c7124d4..eff8977 100644
@@ -11,3 +11,7 @@ recursive-include gpiod/ext *.h
 
 recursive-include tests/gpiosim *.c
 recursive-include tests/procname *.c
+
+recursive-include lib *.c
+recursive-include lib *.h
+recursive-include include *.h
bindings/python/include new file +0 -0
new file mode 120000
index 0000000..fcffffb

bindings/python/include line 1

Quoted 2 lines
bindings/python/include new file +1 -0
@@ -0,0 +1 @@
+../../include
\ No newline at end of file

Addressing Andy's remark: I think this is just how git generates diffs for links, nothing we can do about it.

I would prefer not to have links in the repo anyway, it would be great if we could create them at build-time.

Bart

Quoted 89 lines
bindings/python/lib new file +1 -0
new file mode 120000
index 0000000..58677dd
@@ -0,0 +1 @@
+../../lib
\ No newline at end of file
bindings/python/setup.py +44 -14
index 66b7908..2e25981 100644
@@ -19,19 +19,53 @@ class build_ext(orig_build_ext):
         rmtree(path.join(self.build_lib, "tests"), ignore_errors=True)
 
 
+with open("gpiod/version.py", "r") as fd:
+    exec(fd.read())
+
+
+sources = [
+    # gpiod Python bindings
+    "gpiod/ext/chip.c",
+    "gpiod/ext/common.c",
+    "gpiod/ext/line-config.c",
+    "gpiod/ext/line-settings.c",
+    "gpiod/ext/module.c",
+    "gpiod/ext/request.c",
+]
+
+if "USE_SYSTEM_GPIOD" in environ and environ["USE_SYSTEM_GPIOD"] == "1":
+    libraries = ["gpiod"]
+    include_dirs = ["gpiod"]
+else:
+    sources += [
+        # gpiod library
+        "lib/chip.c",
+        "lib/chip-info.c",
+        "lib/edge-event.c",
+        "lib/info-event.c",
+        "lib/internal.c",
+        "lib/line-config.c",
+        "lib/line-info.c",
+        "lib/line-request.c",
+        "lib/line-settings.c",
+        "lib/misc.c",
+        "lib/request-config.c",
+    ]
+    libraries = []
+    include_dirs = ["include", "lib", "gpiod/ext"]
+
+
 gpiod_ext = Extension(
     "gpiod._ext",
-    sources=[
-        "gpiod/ext/chip.c",
-        "gpiod/ext/common.c",
-        "gpiod/ext/line-config.c",
-        "gpiod/ext/line-settings.c",
-        "gpiod/ext/module.c",
-        "gpiod/ext/request.c",
-    ],
+    libraries=libraries,
+    sources=sources,
     define_macros=[("_GNU_SOURCE", "1")],
-    libraries=["gpiod"],
-    extra_compile_args=["-Wall", "-Wextra"],
+    include_dirs=include_dirs,
+    extra_compile_args=[
+        "-Wall",
+        "-Wextra",
+        '-DGPIOD_VERSION_STR="{}"'.format(__version__),
+    ],
 )
 
 gpiosim_ext = Extension(
@@ -54,9 +88,6 @@ if "GPIOD_WITH_TESTS" in environ and environ["GPIOD_WITH_TESTS"] == "1":
     extensions.append(gpiosim_ext)
     extensions.append(procname_ext)
 
-with open("gpiod/version.py", "r") as fd:
-    exec(fd.read())
-
 setup(
     name="libgpiod",
     packages=find_packages(exclude=["tests", "tests.*"]),
--

2.34.1

#6 lore raw

PATCH 2/3 bindings: python: add pyproject.toml, pep 518

In reply to Phil Howard

Add pyproject.toml to prevent spurious deprecation warnings from pip.

---
 bindings/python/pyproject.toml | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 bindings/python/pyproject.toml
bindings/python/pyproject.toml new file +2 -0
new file mode 100644
index 0000000..d1e6ae6
@@ -0,0 +1,2 @@
+[build-system]
+requires = ["setuptools", "wheel"]
Signature · 1 line
2.34.1
#7 lore raw

Re: [libgpiod][PATCH 2/3] bindings: python: add pyproject.toml, pep 518

Quoted 18 lines · Phil Howard

Add pyproject.toml to prevent spurious deprecation warnings from pip.

---
 bindings/python/pyproject.toml | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 bindings/python/pyproject.toml
bindings/python/pyproject.toml new file +2 -1
new file mode 100644
index 0000000..d1e6ae6
@@ -0,0 +1,2 @@
+[build-system]
+requires = ["setuptools", "wheel"]
--
Quoted 2 lines

2.34.1

Please add licensing and copyright headers like what you see in all the other files. I know this is a very small configuration file but otherwise reuse lint complains.

Bart

#8 lore raw

PATCH 3/3 bindings: python: require python 3.9.0

In reply to Phil Howard

Required minimum version for PyModule_AddType helper.

---
 bindings/python/setup.py | 1 +
 1 file changed, 1 insertion(+)
bindings/python/setup.py +1 -0
index 2e25981..fbba7ab 100644
@@ -91,6 +91,7 @@ if "GPIOD_WITH_TESTS" in environ and environ["GPIOD_WITH_TESTS"] == "1":
 setup(
     name="libgpiod",
     packages=find_packages(exclude=["tests", "tests.*"]),
+    python_requires=">=3.9.0",
     ext_modules=extensions,
     cmdclass={"build_ext": build_ext},
     version=__version__,
Signature · 1 line
2.34.1
#9 lore raw

Re: [libgpiod][PATCH 3/3] bindings: python: require python 3.9.0

Quoted 21 lines · Phil Howard

Required minimum version for PyModule_AddType helper.

---
 bindings/python/setup.py | 1 +
 1 file changed, 1 insertion(+)
bindings/python/setup.py +1 -1
index 2e25981..fbba7ab 100644
@@ -91,6 +91,7 @@ if "GPIOD_WITH_TESTS" in environ and environ["GPIOD_WITH_TESTS"] == "1":
 setup(
     name="libgpiod",
     packages=find_packages(exclude=["tests", "tests.*"]),
+    python_requires=">=3.9.0",
     ext_modules=extensions,
     cmdclass={"build_ext": build_ext},
     version=__version__,
--
Quoted 2 lines

2.34.1

Applied, thanks!

Bart