Skip to contentSkip to conversation

package/python-numpy: fixup numpy distutils for cross compilation

PATCH New Esben Haabendal 3 participants 5 messages Sep 30, 2019 – Jan 7, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/2 package/python-numpy: fixup numpy distutils for cross compilation

Fix problems using the numpy distutils extension for cross compilation, so that linking with npymath library will use target library when building target packages.

---
 .../0001-NPY_PKG_CONFIG_PATH.patch            | 85 +++++++++++++++++++
 package/python-numpy/python-numpy.mk          | 21 +++++
 2 files changed, 106 insertions(+)
 create mode 100644 package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch
package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch new file +85 -0
new file mode 100644
index 000000000000..3e8620a37cb8
@@ -0,0 +1,85 @@
+commit 153fc148eec60e5cbec0e80617f75a3a5dd2a3f8
+Author: Esben Haabendal <[email protected]>
+Date:   Thu Sep 12 21:59:58 2019 +0200
+
+    ENH: Allow NPY_PKG_CONFIG_PATH environment variable override
+    
+    Allow overriding npy-pkg-config directory using the NPY_PKG_CONFIG_PATH
+    environment variable, making it easier to use numpy in cross-compilation
+    setups.
+
+Upstream-Status: Accepted (scheduled for 1.18.0)
+Signed-off-by: Esben Haabendal <[email protected]>
+
+diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py
+index 1e10e92fdadf..0eaaeb7364d4 100644
+--- a/numpy/distutils/misc_util.py
++++ b/numpy/distutils/misc_util.py
+@@ -1687,6 +1687,43 @@ class Configuration(object):
+ 
+         and will be installed as foo.ini in the 'lib' subpath.
+ 
++        Cross-compilation
++        -----------------
++        When cross-compiling with numpy distutils, it might be necessary to
++        use modified npy-pkg-config files.  Using the default/generated files
++        will link with the host libraries (i.e. libnpymath.a).  For
++        cross-compilation you of-course need to link with target libraries,
++        while using the host Python installation.
++
++        You can copy out the numpy/core/lib/npy-pkg-config directory, add a
++        pkgdir value to the .ini files and set NPY_PKG_CONFIG_PATH environment
++        variable to point to the directory with the modified npy-pkg-config
++        files.
++
++        Example npymath.ini modified for cross-compilation::
++
++            [meta]
++            Name=npymath
++            Description=Portable, core math library implementing C99 standard
++            Version=0.1
++
++            [variables]
++            pkgname=numpy.core
++            pkgdir=/build/arm-linux-gnueabi/sysroot/usr/lib/python3.7/site-packages/numpy/core
++            prefix=${pkgdir}
++            libdir=${prefix}/lib
++            includedir=${prefix}/include
++
++            [default]
++            Libs=-L${libdir} -lnpymath
++            Cflags=-I${includedir}
++            Requires=mlib
++
++            [msvc]
++            Libs=/LIBPATH:${libdir} npymath.lib
++            Cflags=/INCLUDE:${includedir}
++            Requires=mlib
++
+         """
+         if subst_dict is None:
+             subst_dict = {}
+@@ -2092,9 +2129,22 @@ def get_numpy_include_dirs():
+     return include_dirs
+ 
+ def get_npy_pkg_dir():
+-    """Return the path where to find the npy-pkg-config directory."""
++    """Return the path where to find the npy-pkg-config directory.
++
++    If the NPY_PKG_CONFIG_PATH environment variable is set, the value of that
++    is returned.  Otherwise, a path inside the location of the numpy module is
++    returned.
++
++    The NPY_PKG_CONFIG_PATH can be useful when cross-compiling, maintaining
++    customized npy-pkg-config .ini files for the cross-compilation
++    environment, and using them when cross-compiling.
++
++    """
+     # XXX: import here for bootstrapping reasons
+     import numpy
++    d = os.environ.get('NPY_PKG_CONFIG_PATH')
++    if d is not None:
++        return d
+     d = os.path.join(os.path.dirname(numpy.__file__),
+             'core', 'lib', 'npy-pkg-config')
+     return d
package/python-numpy/python-numpy.mk +21 -0
index 3b474efa6e6c..1c7dd17adad0 100644
@@ -30,6 +30,27 @@ define PYTHON_NUMPY_CONFIGURE_CMDS
 	echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg
 endef
 
+# The numpy distutils extensions are not very cross friendly.  It comes with
+# it's own pkg-config look-alike, which we are patching to allow overriding
+# where it locates the configuration files.  This allows us to use fixed up
+# target configuration files, which we make sure includes full path to the
+# target staging area, so that when building for target, we actually use the
+# target libraries.  Without this, target builds using numpy distutils
+# extensions (such as fx. python-scipy) will use the host libraries, which
+# obviously will not work.
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+PYTHON_NUMPY_STAGING_DIR = $(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/numpy
+else
+PYTHON_NUMPY_STAGING_DIR = $(STAGING_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/numpy
+endif
+PYTHON_NUMPY_NPY_PKG_CONFIG_PATH=$(PYTHON_NUMPY_STAGING_DIR)/core/lib/npy-pkg-config
+define PYTHON_NUMPY_FIXUP_NPY_PKG_CONFIG_FILES
+	sed -e '/^pkgdir=/d' \
+	    -e '/^prefix=/i pkgdir=$(PYTHON_NUMPY_STAGING_DIR)/core' \
+	    -i $(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH)/npymath.ini
+endef
+PYTHON_NUMPY_POST_INSTALL_STAGING_HOOKS += PYTHON_NUMPY_FIXUP_NPY_PKG_CONFIG_FILES
+
 # Some package may include few headers from NumPy, so let's install it
 # in the staging area.
 PYTHON_NUMPY_INSTALL_STAGING = YES
Signature · 1 line
2.23.0
#1 lore raw

PATCH v5 2/2 package/python-scipy: new package

See http://scipy.github.io/devdocs/building/linux.html
for SciPy build requirements:

* BLAS and LAPACK libraries
* C and Fortran compilers
* Python header files
* Cython compiler

For now, clapack package is used to provide both BLAS and LAPACK libraries, as that is what is supported by the current python-numpy recipe which python-scipy depends on.

With additional work, it should be possible to improve both python-numpy and python-scipy to support other BLAS and LAPACK library implementations, but it is probably best to keep such changes aligned.

A site.cfg file is used to configure numpy distutils extension to find include files and libraries in stagging.

Note, host-python-numpy is needed in order to use numpy distutils extension from it.

---
 DEVELOPERS                             |  1 +
 package/Config.in                      |  1 +
 package/python-scipy/Config.in         | 23 +++++++++++++++
 package/python-scipy/python-scipy.hash |  9 ++++++
 package/python-scipy/python-scipy.mk   | 40 ++++++++++++++++++++++++++
 5 files changed, 74 insertions(+)
 create mode 100644 package/python-scipy/Config.in
 create mode 100644 package/python-scipy/python-scipy.hash
 create mode 100644 package/python-scipy/python-scipy.mk
DEVELOPERS +1 -0
index 8cbf55fd1013..499fa5954ffe 100644
@@ -727,6 +727,7 @@ F:	package/szip/
 N:	Esben Haabendal <[email protected]>
 F:	boot/gummiboot/
 F:	package/python-kiwisolver/
+F:	package/python-scipy/
 
 N:	Etienne Carriere <[email protected]>
 F:	boot/optee-os/
package/Config.in +1 -0
index 713ef79edb9e..1a98ad9b2bcb 100644
@@ -1059,6 +1059,7 @@ menu "External python modules"
 	source "package/python-scapy/Config.in"
 	source "package/python-scapy3k/Config.in"
 	source "package/python-schedule/Config.in"
+	source "package/python-scipy/Config.in"
 	source "package/python-sdnotify/Config.in"
 	source "package/python-secretstorage/Config.in"
 	source "package/python-see/Config.in"
package/python-scipy/Config.in new file +23 -0
new file mode 100644
index 000000000000..071ceb2b70a6
@@ -0,0 +1,23 @@
+config BR2_PACKAGE_PYTHON_SCIPY
+	bool "python-scipy"
+	depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
+	depends on !BR2_powerpc || BR2_TOOLCHAIN_USES_GLIBC # clapack
+        depends on !BR2_m68k_cf # clapack
+	depends on BR2_TOOLCHAIN_USES_GLIBC || BR2_TOOLCHAIN_USES_MUSL # python-numpy
+	depends on BR2_TOOLCHAIN_HAS_FORTRAN
+	depends on BR2_PACKAGE_PYTHON3
+	select BR2_PACKAGE_HOST_PYTHON_CYTHON
+	select BR2_PACKAGE_PYTHON_NUMPY
+	select BR2_PACKAGE_CLAPACK
+	help
+	  The SciPy library is one of the core packages that make up the SciPy
+	  stack. It provides many user-friendly and efficient numerical
+	  routines such as routines for numerical integration, interpolation,
+	  optimization, linear algebra and statistics.
+
+	  https://www.scipy.org/scipylib/
+
+comment "python-scipy needs toolchain w/ fortran and glibc or musl"
+	depends on BR2_PACKAGE_PYTHON3
+	depends on !BR2_TOOLCHAIN_HAS_FORTRAN || \
+		(!BR2_TOOLCHAIN_USES_GLIBC && !BR2_TOOLCHAIN_USES_MUSL)
package/python-scipy/python-scipy.hash new file +9 -0
new file mode 100644
index 000000000000..6b0c3605a3b8
@@ -0,0 +1,9 @@
+# Locally generated
+sha256 4ea68de2840cf7d35c58464412d21f6f154cab7fba610c11002603ee0b9e2372  python-scipy-1.3.1.tar.gz
+sha256 904942f7df1dc0f4a48af8d72405854fd96497bd1fe9ddb2c69d1797c22dfcd7  LICENSE.txt
+sha256 37e64a498894ac7c3b070023e3689e954a8ecf8a23b90968d09a455f1b4f7b35  scipy/linalg/src/lapack_deprecations/LICENSE
+sha256 606209a000716c5f66e33e180ce08434b96ed17db4975ab9723c6b5fbcc89609  scipy/ndimage/LICENSE.txt
+sha256 3df9207af2fdb861af0ae3b22026f163b9bcfab4e525dc4943afe2ffa3f77624  scipy/optimize/tnc/LICENSE
+sha256 f0cedf52503b2d42b83411a0a16e6fefac346dfad8fddc66f50050150123470c  scipy/sparse/linalg/dsolve/SuperLU/License.txt
+sha256 0926566f9f406178d1214f8cc796e166b1213dd7c05e0c5b461a8b8ac9e50bbe  scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING
+sha256 51a5a08d537b34febb851c761496ead461f90b6c3c46a5248780d63870124e9c  scipy/spatial/qhull_src/COPYING.txt
package/python-scipy/python-scipy.mk new file +40 -0
new file mode 100644
index 000000000000..bad392121da6
@@ -0,0 +1,40 @@
+################################################################################
+#
+# python-scipy
+#
+################################################################################
+
+PYTHON_SCIPY_VERSION = 1.3.1
+PYTHON_SCIPY_SITE = $(call github,scipy,scipy,v$(PYTHON_SCIPY_VERSION))
+PYTHON_SCIPY_LICENSE = BSD-3-Clause, BSD-2-Clause, BSD, BSD-Style, \
+	Apache-2.0, MIT
+PYTHON_SCIPY_LICENSE_FILES = \
+	LICENSE.txt \
+	scipy/linalg/src/lapack_deprecations/LICENSE \
+	scipy/ndimage/LICENSE.txt \
+	scipy/optimize/tnc/LICENSE \
+	scipy/sparse/linalg/dsolve/SuperLU/License.txt \
+	scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING \
+	scipy/spatial/qhull_src/COPYING.txt
+PYTHON_SCIPY_SETUP_TYPE = setuptools
+PYTHON_SCIPY_DEPENDENCIES += host-python-cython host-python-numpy python-numpy \
+	clapack
+
+PYTHON_SCIPY_BUILD_OPTS = config_fc --fcompiler=gnu95
+PYTHON_SCIPY_ENV += F90=$(TARGET_CROSS)gfortran F77=$(TARGET_CROSS)gfortran
+
+# Provide system configuration options to numpy distutils extenions, telling
+# to find all include files and libraries in staging directory.
+define PYTHON_SCIPY_CONFIGURE_CMDS
+	-rm -f $(@D)/site.cfg
+	echo "[DEFAULT]" >> $(@D)/site.cfg
+	echo "library_dirs = $(STAGING_DIR)/usr/lib" >> $(@D)/site.cfg
+	echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg
+endef
+
+# Use the target numpy pkg-config configuration files modified for target
+# cross-compilation.  Without this, numpy distutils will cause the linker to
+# link with host libnpymath.a.
+PYTHON_SCIPY_ENV += NPY_PKG_CONFIG_PATH=$(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH)
+
+$(eval $(python-package))
Signature · 1 line
2.23.0
#2 lore raw

PATCH v5 2/2 package/python-scipy: new package

Quoted 17 lines · Esben Haabendal

See http://scipy.github.io/devdocs/building/linux.html
for SciPy build requirements:

* BLAS and LAPACK libraries
* C and Fortran compilers
* Python header files
* Cython compiler

For now, clapack package is used to provide both BLAS and LAPACK libraries, as that is what is supported by the current python-numpy recipe which python-scipy depends on.

With additional work, it should be possible to improve both python-numpy and python-scipy to support other BLAS and LAPACK library implementations, but it is probably best to keep such changes aligned.

Quoted 8 lines

A site.cfg file is used to configure numpy distutils extension to find include files and libraries in stagging.

Note, host-python-numpy is needed in order to use numpy distutils extension from it.

---

Thanks for the great work.

Tested the build via arm64 configuration and verified import in sample python test.

#3 lore raw

PATCH v5 1/2 package/python-numpy: fixup numpy distutils for cross compilation

In reply to Esben Haabendal

Hello Esben,

Quoted 5 lines · On Mon, 30 Sep 2019 10:43:25 +0200 Esben Haabendal <[email protected]> wrote:

Fix problems using the numpy distutils extension for cross compilation, so that linking with npymath library will use target library when building target packages.

When you submit new version of patches, could you include a changelog of what has changed from the previous version ?

You have posted several iterations, with no indication of the changes between the versions, which makes it difficult to keep track of why new versions are being posted.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

PATCH v5 1/2 package/python-numpy: fixup numpy distutils for cross compilation

Quoted 5 lines · Thomas Petazzoni

Hello Esben,

On Mon, 30 Sep 2019 10:43:25 +0200
Esben Haabendal <[email protected]> wrote:

Quoted 8 lines
Quoted 5 lines

Fix problems using the numpy distutils extension for cross compilation, so that linking with npymath library will use target library when building target packages.

When you submit new version of patches, could you include a changelog of what has changed from the previous version ?

I definitely can. I will use a cover letter with changelog in the future.

Quoted 3 lines

You have posted several iterations, with no indication of the changes between the versions, which makes it difficult to keep track of why new versions are being posted.

Sorry about that.

/Esben