Skip to contentSkip to conversation

bindings: python: standalone build tooling for tests

PATCH In review Phil Howard 2 participants 3 messages Nov 8, 2023 – Nov 9, 2023 GPIO

lore
Share & tools
lore
#0 lore raw

v2 cover bindings: python: standalone build tooling for tests

After much delibration about how in-place test builds should work, I have pulled out all of the test extension code from setup.py into a new build_tests.py and updated Makefile.am to invoke it.

This replaces the old python3 setup.py build_ext --inplace with enough setuptools tooling to replicate that behaviour, while also putting it in a better place to avoid problems arising from the deprecation of direct setup.py invocation [1].

Since receipients of the gpiod Python bindings (in tar.gz source or .whl binary format) won't need the test C extensions (and can't run them anyway without the accompanying Python modules) the test extension code has been removed from setup.py rather than duplicated into this new file.

The new build_tests.py creates a temporary directory and redirects all build-time output into it. A final step - handled by build_ext and internal to setuptools - then moves the built _ext.<python_version>-<arch>-<os>.so C extensions into gpiod, tests/gpiosim and tests/procname where they can be imported.

Since release packages and tests no longer share the build/ directory I have removed the "rmtree" hack from the "build_ext" override in setup.py. There should be no cases where test extensions can accidentally end up being installed into the system/user Python environment.

To build and run tests:
- make python-tests
- (sudo) make python-tests-run (I need root for the gpiosim module)

Tested with setuptools==68.2.2 (Sep 2023), setuptools==59.8.0 (Dec 2021) and setuptools==42.0.2 (Dec 2019).

Changes v1 -> v2:
- Add the gpiod ext module to build_tests.py (using glob for sources)
- Formalize the command for running tests by adding it to Makefile.am
- Add build_tests.py output artefacts to Makefile.am's clean-local target
- Sort distutils.log imports (DEBUG before set_verbosity)

Phil Howard (1):
bindings: python: standalone build tooling for tests

 bindings/python/Makefile.am    | 33 +++++++++---
 bindings/python/build_tests.py | 95 ++++++++++++++++++++++++++++++++++
 bindings/python/setup.py       | 28 +---------
 3 files changed, 121 insertions(+), 35 deletions(-)
 create mode 100644 bindings/python/build_tests.py
Signature · 1 line
2.34.1
#1 lore raw

PATCH v2 1/1 bindings: python: standalone build tooling for tests

Move extension definitions and tooling for building tests into build_tests.py and update Makefile.am to call it with appropriate path prefixes.

build_tests.py will perform a standalone build of the text extensions, keeping any build noise in a temporary directory and copying the final built modules automatically out to tests/gpiosim and tests/procname.

Add "python-tests-run" to Makefile.am so it's clear how to run the tests.

Add .so object files generated by build_test.py to Makefile.am's clean-local.

---
 bindings/python/Makefile.am    | 33 +++++++++---
 bindings/python/build_tests.py | 95 ++++++++++++++++++++++++++++++++++
 bindings/python/setup.py       | 28 +---------
 3 files changed, 121 insertions(+), 35 deletions(-)
 create mode 100644 bindings/python/build_tests.py
bindings/python/Makefile.am +25 -8
index 079ceb1..389da9d 100644
@@ -7,19 +7,36 @@ EXTRA_DIST = \
 
 if WITH_TESTS
 
-BUILD_TESTS = 1
+python-tests:
+	TOP_SRCDIR=$(abs_top_builddir) \
+	TOP_BUILDDIR=$(abs_top_builddir) \
+	$(PYTHON) build_tests.py
+
+python-tests-run:
+	PYTHONPATH=$(abs_top_srcdir)/bindings/python
+	LD_LIBRARY_PATH=$(abs_top_builddir)/lib/.libs/:\
+$(abs_top_builddir)/tests/gpiosim/.libs/ \
+    python3 -B -m tests
+
+else
+
+python-tests:
 
 endif
 
-all-local:
-	GPIOD_WITH_TESTS=$(BUILD_TESTS) \
-	$(PYTHON) setup.py build_ext --inplace \
-		--include-dirs=$(top_srcdir)/include/:$(top_srcdir)/tests/gpiosim/ \
-		--library-dirs=$(top_builddir)/lib/.libs/:$(top_srcdir)/tests/gpiosim/.libs/
+clean-local:
+	rm -rf dist
+	rm -f gpiod/_ext.*.so
+	rm -f tests/*/_ext.*.so
+
+all-local: python-tests
+	CFLAGS="-I$(abs_top_srcdir)/include/ -I$(abs_top_srcdir)/tests/gpiosim \
+	-L$(abs_top_builddir)/lib/.libs/ -L$(abs_top_builddir)/tests/gpiosim/.libs/" \
+	$(PYTHON) -m build
 
 install-exec-local:
-	GPIOD_WITH_TESTS= \
-	$(PYTHON) setup.py install --prefix=$(DESTDIR)$(prefix)
+	$(PYTHON) -m pip install dist/*.whl \
+	--prefix=$(DESTDIR)$(prefix)
 
 SUBDIRS = gpiod
 
bindings/python/build_tests.py new file +95 -0

95 lines not on this page · raw message

bindings/python/setup.py +1 -27
index e8704d5..9607a28 100644
@@ -13,7 +13,6 @@ from setuptools.errors import BaseError
 LINK_SYSTEM_LIBGPIOD = getenv("LINK_SYSTEM_LIBGPIOD") == "1"
 LIBGPIOD_MINIMUM_VERSION = "2.1"
 LIBGPIOD_VERSION = getenv("LIBGPIOD_VERSION")
-GPIOD_WITH_TESTS = getenv("GPIOD_WITH_TESTS") == "1"
 SRC_BASE_URL = "https://mirrors.edge.kernel.org/pub/software/libs/libgpiod/"
 TAR_FILENAME = "libgpiod-{version}.tar.gz"
 ASC_FILENAME = "sha256sums.asc"
@@ -189,11 +188,6 @@ class build_ext(orig_build_ext):
 
         super().run()
 
-        # We don't ever want the module tests directory in our package
-        # since this might include gpiosim._ext or procname._ext from a
-        # previous dirty build tree.
-        rmtree(path.join(self.build_lib, "tests"), ignore_errors=True)
-
 
 class sdist(orig_sdist):
     """
@@ -226,32 +220,12 @@ gpiod_ext = Extension(
     extra_compile_args=["-Wall", "-Wextra"],
 )
 
-gpiosim_ext = Extension(
-    "tests.gpiosim._ext",
-    sources=["tests/gpiosim/ext.c"],
-    define_macros=[("_GNU_SOURCE", "1")],
-    libraries=["gpiosim"],
-    extra_compile_args=["-Wall", "-Wextra"],
-)
-
-procname_ext = Extension(
-    "tests.procname._ext",
-    sources=["tests/procname/ext.c"],
-    define_macros=[("_GNU_SOURCE", "1")],
-    extra_compile_args=["-Wall", "-Wextra"],
-)
-
-extensions = [gpiod_ext]
-if GPIOD_WITH_TESTS:
-    extensions.append(gpiosim_ext)
-    extensions.append(procname_ext)
-
 setup(
     name="gpiod",
     url="https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git",
     packages=find_packages(exclude=["tests", "tests.*"]),
     python_requires=">=3.9.0",
-    ext_modules=extensions,
+    ext_modules=[gpiod_ext],
     cmdclass={"build_ext": build_ext, "sdist": sdist},
     version=__version__,
     author="Bartosz Golaszewski",
Signature · 1 line
2.34.1
#2 lore raw
Quoted 9 lines · Phil Howard

Move extension definitions and tooling for building tests into build_tests.py and update Makefile.am to call it with appropriate path prefixes.

build_tests.py will perform a standalone build of the text extensions, keeping any build noise in a temporary directory and copying the final built modules automatically out to tests/gpiosim and tests/procname.

Quoted 7 lines

Add "python-tests-run" to Makefile.am so it's clear how to run the tests.

Add .so object files generated by build_test.py to Makefile.am's clean-local.

---

Thanks, this works quite nicely. I applied it and will release python bindings v2.1.2 to pypi.

Bart