Skip to contentSkip to conversation

host-python-numpy - confused about host openblas dependency

Buildroot Applied Graeme Smecher 3 participants 6 messages Jan 29, 2024 – Feb 29, 2024

lore
Share & tools
lore
#0 lore raw

Hi all,

Several packages depend on host-python-numpy:

buildroot$ grep host-python-numpy -rl package/ package/python-pythran/python-pythran.mk package/python-matplotlib/python-matplotlib.mk package/python-matplotlib/Config.in package/piglit/piglit.mk package/piglit/Config.in package/gnuradio/gnuradio.mk package/gnuradio/Config.in package/python-scipy/python-scipy.mk package/python-scipy/Config.in package/uhd/uhd.mk package/uhd/Config.in

However, this package appears to rely on a host-openblas dependency that's implied but never really spelled out.

My builds fail as follows:

	buldroot$ make
	>>> host-python-numpy 1.25.0 Configuring
	rm -rf /path/to/buildroot/output/build/host-python-numpy-1.25.0//build
	mkdir -p /path/to/buildroot/output/build/host-python-numpy-1.25.0//build
	GIT_DIR=. PATH="/path/to/buildroot/output/host/bin:/path/to/buildroot/output/host/sbin:/home/gsmecher/.local/bin:/home/gsmecher/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin" PKG_CONFIG="/path/to/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/path/to/buildroot/output/host/lib/pkgconfig:/path/to/buildroot/output/host/share/pkgconfig" AR="/usr/bin/ar" AS="/usr/bin/as" LD="/usr/bin/ld" NM="/usr/bin/nm" CC="/usr/bin/gcc" GCC="/usr/bin/gcc" CXX="/usr/bin/g++" CPP="/usr/bin/cpp" OBJCOPY="/usr/bin/objcopy" RANLIB="/usr/bin/ranlib" CPPFLAGS="-I/path/to/buildroot/output/host/include" CFLAGS="-O2 -I/path/to/buildroot/output/host/include" CXXFLAGS="-O2 -I/path/to/buildroot/output/host/include" LDFLAGS="-L/path/to/buildroot/output/host/lib -Wl,-rpath,/path/to/buildroot/output/host/lib" INTLTOOL_PERL=/usr/bin/perl  PYTHONNOUSERSITE=y /path/to/buildroot/output/host/bin/me

son setup --prefix=/path/to/buildroot/output/host --libdir=lib --sysconfdir=/path/to/buildroot/output/host/etc --localstatedir=/path/to/buildroot/output/host/var --default-library=shared --buildtype=release --wrap-mode=nodownload -Dstrip=true /path/to/buildroot/output/build/host-python-numpy-1.25.0/ /path/to/buildroot/output/build/host-python-numpy-1.25.0//build
The Meson build system
Version: 1.1.1
Source dir: /path/to/buildroot/output/build/host-python-numpy-1.25.0
Build dir: /path/to/buildroot/output/build/host-python-numpy-1.25.0/build
Build type: native build
Project name: NumPy
Project version: 1.24.0.dev0
C compiler for the host machine: /usr/bin/gcc (gcc 12.2.0 "gcc (Debian 12.2.0-14) 12.2.0")
C linker for the host machine: /usr/bin/gcc ld.bfd 2.40
C++ compiler for the host machine: /usr/bin/g++ (gcc 12.2.0 "g++ (Debian 12.2.0-14) 12.2.0")
C++ linker for the host machine: /usr/bin/g++ ld.bfd 2.40
Cython compiler for the host machine: cython (cython 0.29.36)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python3 found: YES (/path/to/buildroot/output/host/bin/python3)
Found pkg-config: /path/to/buildroot/output/host/bin/pkg-config (1.6.3)
Has header "Python.h" with dependency python-3.11: YES
Compiler for C supports arguments -fno-strict-aliasing: YES
Library m found: YES
Found CMake: /path/to/buildroot/output/host/bin/cmake (3.27.7)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: NO (tried pkgconfig and cmake)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: YES 0.3.21
Dependency openblas found: YES 0.3.21 (cached)
Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py)
Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py)

output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None>

A full log can be found at /path/to/buildroot/output/build/host-python-numpy-1.25.0/build/meson-logs/meson-log.txt
make: *** [package/pkg-generic.mk:273: /path/to/buildroot/output/build/host-python-numpy-1.25.0/.stamp_configured] Error 1

Any suggestions? There are enough packages depending on host-numpy that I expect it's something unusual on my end. If I force the build to proceed using system pkg-config I can coax it into proceeding - but this is clearly a hack.

thanks,
Graeme
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#1 lore raw

Graeme, All,

On 2024-01-29 11:54 -0800, Graeme Smecher spake thusly:

Quoted 1 line

Several packages depend on host-python-numpy:

[--SNIP--]

Quoted 2 lines

However, this package appears to rely on a host-openblas dependency that's implied but never really spelled out.

[--SNIP--]

Quoted 1 line

Any suggestions?

See package/python-numpy/python-numpy.mk: it has proper lapack and blas handling for the target variant, but indeed the host variant is left in the dark.

Quoted 2 lines

There are enough packages depending on host-numpy that I expect it's something unusual on my end.

Or rather just happenstance that you found it first...

Quoted 2 lines

If I force the build to proceed using system pkg-config I can coax it into proceeding - but this is clearly a hack.

Indeed it is not the proper solutoin. The proper way is to edit package/python-numpy/python-numpy.mk and add:

    HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack=""

Hopefully they are not needed for the host variant, because we not have host-lapack or host-openblas.

Care to send a patch, please?

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

PATCH 1/1 host-python-numpy: correct implied host-lapack/host-blas dependency.

From: Graeme Smecher <[email protected]>

Before this patch, host-python-numpy attempted (and failed) to detect OpenBLAS as follows:

buildroot$ make
>>> host-python-numpy 1.25.0 Configuring
[...]
Run-time dependency openblas found: NO (tried pkgconfig and cmake)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: YES 0.3.21
Dependency openblas found: YES 0.3.21 (cached)
Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py) Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py)

output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None> [...]

Two obvious ways to correct this:

- Add some kind of host-lapack / host-openblas dependency to host-python-numpy, or
- Tell host-python-numpy not to look for (or rely on) external code.

Because the host numpy environment shouldn't be doing any numerical heavy lifting, we opt for the latter here.

---
 package/python-numpy/python-numpy.mk | 4 ++++
 1 file changed, 4 insertions(+)
package/python-numpy/python-numpy.mk +4 -0
index b52e0da17d..cc5dfc9224 100644
@@ -36,6 +36,10 @@ else
 PYTHON_NUMPY_CONF_OPTS += -Dblas=""
 endif
 
+# Rather than add a host-blas or host-lapack dependencies, just use unoptimized,
+# in-tree code.
+HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack=""
+
 # Fixup the npymath.ini prefix path with actual target staging area where
 # numpy core was built. Without this, target builds using numpy distutils
 # extensions like python-scipy, python-numba cannot find -lnpymath since
Signature · 6 lines
2.39.2

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

Re: [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency.

Graeme, All,

+James

On 2024-01-29 12:40 -0800, [email protected] spake thusly:

Quoted 20 lines

From: Graeme Smecher <[email protected]>

Before this patch, host-python-numpy attempted (and failed) to detect OpenBLAS as follows:

buildroot$ make
>>> host-python-numpy 1.25.0 Configuring
[...]
Run-time dependency openblas found: NO (tried pkgconfig and cmake)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: YES 0.3.21
Dependency openblas found: YES 0.3.21 (cached)
Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py) Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py)

output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None> [...]

Two obvious ways to correct this:

Quoted 8 lines

- Add some kind of host-lapack / host-openblas dependency to host-python-numpy, or
- Tell host-python-numpy not to look for (or rely on) external code.

Because the host numpy environment shouldn't be doing any numerical heavy lifting, we opt for the latter here.

Good commit log, but there is a missing piece to it, see below...

James: as far as I understand, this was the situation before the bump to 1.25.0 and the switch to meson, because we did not have the BLAS=None and LAPACK=None options before that either, but had no such failure report.

So I guess the defaults changed with the switch to meson, which might explain why we did not notice before?

If so, we need to add that info to the commit log, to decide whether this needs backporting to 2023.11, and what to do about 2023.02 LTS that does not have 1.25.

Regards,
Yann E. MORIN.

Quoted 22 lines
---
 package/python-numpy/python-numpy.mk | 4 ++++
 1 file changed, 4 insertions(+)
package/python-numpy/python-numpy.mk +4 -0
index b52e0da17d..cc5dfc9224 100644
@@ -36,6 +36,10 @@ else
 PYTHON_NUMPY_CONF_OPTS += -Dblas=""
 endif
 
+# Rather than add a host-blas or host-lapack dependencies, just use unoptimized,
+# in-tree code.
+HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack=""
+
 # Fixup the npymath.ini prefix path with actual target staging area where
 # numpy core was built. Without this, target builds using numpy distutils
 # extensions like python-scipy, python-numba cannot find -lnpymath since
Signature · 1 line
2.39.2
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw

Re: [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency.

In reply to [email protected]

Graeme, All,

On 2024-01-29 12:40 -0800, [email protected] spake thusly:

Quoted 15 lines

From: Graeme Smecher <[email protected]>

Before this patch, host-python-numpy attempted (and failed) to detect OpenBLAS as follows:

buildroot$ make
>>> host-python-numpy 1.25.0 Configuring
[...]
Run-time dependency openblas found: NO (tried pkgconfig and cmake)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: YES 0.3.21
Dependency openblas found: YES 0.3.21 (cached)
Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py) Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py)

Quoted 2 lines

output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None> [...]

So I dug in the Numpy git history, and before we bumped to 1.25, it was using distutils, which would only warn and disable BLAS support if not found, but the meson switch made the default to fail.

Quoted 10 lines

Two obvious ways to correct this:

- Add some kind of host-lapack / host-openblas dependency to host-python-numpy, or
- Tell host-python-numpy not to look for (or rely on) external code.

Because the host numpy environment shouldn't be doing any numerical heavy lifting, we opt for the latter here.

Applied to master, with an extended commit log about the history, thanks.

Regards,
Yann E. MORIN.

Quoted 26 lines
---
 package/python-numpy/python-numpy.mk | 4 ++++
 1 file changed, 4 insertions(+)
package/python-numpy/python-numpy.mk +4 -0
index b52e0da17d..cc5dfc9224 100644
@@ -36,6 +36,10 @@ else
 PYTHON_NUMPY_CONF_OPTS += -Dblas=""
 endif
 
+# Rather than add a host-blas or host-lapack dependencies, just use unoptimized,
+# in-tree code.
+HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack=""
+
 # Fixup the npymath.ini prefix path with actual target staging area where
 # numpy core was built. Without this, target builds using numpy distutils
 # extensions like python-scipy, python-numba cannot find -lnpymath since
Signature · 6 lines
2.39.2

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#5 lore raw

Re: [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency.

Quoted 37 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Graeme, All,
On 2024-01-29 12:40 -0800, [email protected] spake thusly:

Quoted 18 lines

From: Graeme Smecher <[email protected]>

Before this patch, host-python-numpy attempted (and failed) to detect OpenBLAS as follows:

buildroot$ make

Quoted 1 line
Quoted 1 line
Quoted 1 line

host-python-numpy 1.25.0 Configuring

[...]
Run-time dependency openblas found: NO (tried pkgconfig and cmake)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: YES 0.3.21
Dependency openblas found: YES 0.3.21 (cached)
Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py) Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py)

Quoted 4 lines

output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44:
ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None>
[...]

So I dug in the Numpy git history, and before we bumped to 1.25, it was using distutils, which would only warn and disable BLAS support if not found, but the meson switch made the default to fail.

Quoted 6 lines

Two obvious ways to correct this:

- Add some kind of host-lapack / host-openblas dependency to host-python-numpy, or
- Tell host-python-numpy not to look for (or rely on) external code.

Quoted 7 lines
Quoted 4 lines

Because the host numpy environment shouldn't be doing any numerical heavy lifting, we opt for the latter here.

Applied to master, with an extended commit log about the history, thanks.

Committed to 2023.11.x, thanks.

Signature · 5 lines
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot