Skip to contentSkip to conversation

package/python-numpy: fix occasional build and run-time failure with lapack

PATCH New Romain Naour 6 participants 22 messages Aug 1, 2019 – Aug 18, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/5 package/python-numpy: fix occasional build and run-time failure with lapack

From: Alexandre PAYEN <[email protected]>

python-numpy fail at run-time because cblas reference is missing in BR2_PYTHON_NUMPY_SITE_CFG_LIBS.
clapack is the C implementation of lapack so when using clapack python numpy depends on clapack.
cblas is a blas layer which allow to call blas routine from C code. So python numpy need cblas instead of blas to work correclty

So :
- add cblas reference in BR2_PYTHON_NUMPY_SITE_CFG_LIBS when BR2_PACKAGE_CLAPACK = y

Fixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/
http://lists.busybox.net/pipermail/buildroot/2019-June/252380.html

Initial patch from Giulio Benetti :
[v1] http://patchwork.ozlabs.org/patch/1100100/
[v2] http://patchwork.ozlabs.org/patch/1100208/

---
 package/python-numpy/python-numpy.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/python-numpy/python-numpy.mk +1 -1
index 5d2fbfc7ad..aa8052dca4 100644
@@ -17,7 +17,7 @@ PYTHON_NUMPY_SETUP_TYPE = setuptools
 
 ifeq ($(BR2_PACKAGE_CLAPACK),y)
 PYTHON_NUMPY_DEPENDENCIES += clapack
-PYTHON_NUMPY_SITE_CFG_LIBS += blas lapack
+PYTHON_NUMPY_SITE_CFG_LIBS += cblas lapack
 else
 PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
 endif
Signature · 1 line
2.20.1
#1 lore raw

PATCH 2/5 package/liblapack: add liblapack as virtual package with lapack and clapack as providers.

From: Alexandre PAYEN <[email protected]>

lapack and clapack are both generating the same liblapack.so but does not have the same symbols. So the run-time is broke. They have to exclude each other. A liblapack virtual package havev been created. It include lapack and clapack as providers.
The armadillo packet require clapack to build, but lapack with complex support works as well.

The upcomming python-scipy package require lapack as liblapack provider, we add the ability to force the lapack implementation.

So :
- add package/liblapack directory and do the virtual configuration
- add BR2_LAPACK_COMPLEX in armadillo configuration when BR2_PACKAGE_LAPACK = y
- Add

---
v6: remove Config.in from clapack and lapack
Rework liblapack virtual package (Config.in)

---
 package/Config.in              |  3 +-
 package/armadillo/Config.in    |  3 +-
 package/armadillo/armadillo.mk |  2 +-
 package/clapack/Config.in      | 28 -----------
 package/clapack/clapack.mk     |  1 +
 package/lapack/Config.in       | 21 --------
 package/lapack/lapack.mk       |  1 +
 package/liblapack/Config.in    | 91 ++++++++++++++++++++++++++++++++++
 package/liblapack/liblapack.mk |  7 +++
 9 files changed, 104 insertions(+), 53 deletions(-)
 delete mode 100644 package/clapack/Config.in
 delete mode 100644 package/lapack/Config.in
 create mode 100644 package/liblapack/Config.in
 create mode 100644 package/liblapack/liblapack.mk
package/Config.in +1 -2
index b7472b9bea..5f408f4de9 100644
@@ -1666,7 +1666,6 @@ menu "Other"
 	source "package/c-capnproto/Config.in"
 	source "package/capnproto/Config.in"
 	source "package/clang/Config.in"
-	source "package/clapack/Config.in"
 	source "package/classpath/Config.in"
 	source "package/cmocka/Config.in"
 	source "package/cppcms/Config.in"
@@ -1689,7 +1688,6 @@ menu "Other"
 	source "package/gsl/Config.in"
 	source "package/gtest/Config.in"
 	source "package/jemalloc/Config.in"
-	source "package/lapack/Config.in"
 	source "package/libargtable2/Config.in"
 	source "package/libatomic_ops/Config.in"
 	source "package/libb64/Config.in"
@@ -1715,6 +1713,7 @@ menu "Other"
 	source "package/libglob/Config.in"
 	source "package/libical/Config.in"
 	source "package/libite/Config.in"
+	source "package/liblapack/Config.in"
 	source "package/liblinear/Config.in"
 	source "package/libloki/Config.in"
 	source "package/libnpth/Config.in"
package/armadillo/Config.in +2 -1
index b2b61a3233..62c15fbdbe 100644
@@ -12,7 +12,8 @@ config BR2_PACKAGE_ARMADILLO
 	depends on BR2_INSTALL_LIBSTDCPP
 	depends on !BR2_powerpc || BR2_TOOLCHAIN_USES_GLIBC # clapack
 	depends on !BR2_m68k_cf # clapack
-	select BR2_PACKAGE_CLAPACK
+	select BR2_PACKAGE_LIBLAPACK
+	select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK
 	help
 	  Armadillo: An Open Source C++ Linear Algebra Library for
 	  Fast Prototyping and Computationally Intensive Experiments.
package/armadillo/armadillo.mk +1 -1
index b5fe403a7a..75c402a9ca 100644
@@ -7,7 +7,7 @@
 ARMADILLO_VERSION = 7.900.1
 ARMADILLO_SOURCE = armadillo-$(ARMADILLO_VERSION).tar.xz
 ARMADILLO_SITE = https://downloads.sourceforge.net/project/arma
-ARMADILLO_DEPENDENCIES = clapack
+ARMADILLO_DEPENDENCIES = liblapack
 ARMADILLO_INSTALL_STAGING = YES
 ARMADILLO_LICENSE = MPL-2.0
 ARMADILLO_LICENSE_FILES = LICENSE.txt
package/clapack/Config.in deleted +0 -28
deleted file mode 100644
index a912eb6c91..0000000000
@@ -1,28 +0,0 @@
-comment "clapack needs a glibc toolchain"
-	depends on BR2_powerpc
-	depends on !BR2_TOOLCHAIN_USES_GLIBC
-
-config BR2_PACKAGE_CLAPACK
-	bool "cblas/clapack"
-	# _fpu_control is used on PowerPC, but not available with
-	# uClibc or musl
-	depends on !BR2_powerpc || BR2_TOOLCHAIN_USES_GLIBC
-	# assembler: Error: value out of range
-	depends on !BR2_m68k_cf
-	help
-	  BLAS and LAPACK C implementation (f2c'ed version of).
-
-	  http://www.netlib.org/clapack/
-
-config BR2_PACKAGE_CLAPACK_ARITH_H
-	string "Custom BLAS arith.h"
-	depends on BR2_PACKAGE_CLAPACK
-	help
-	  To optimize BLAS library for the hardware, an 'arith.h' header
-	  should be provided.
-
-	  If empty, the library will not be optimized by the compiler.
-
-	  In any case an 'arithchk' program is built
-	  (but not installed), to be run on the target to generate this
-	  arith.h header.
package/clapack/clapack.mk +1 -0
index 9a083f3223..280ea20e0f 100644
@@ -14,6 +14,7 @@ CLAPACK_LICENSE = HPND (libf2c), BSD-3-Clause (libblas and liblapack)
 CLAPACK_LICENSE_FILES = F2CLIBS/libf2c/Notice COPYING
 CLAPACK_SITE = http://www.netlib.org/clapack
 CLAPACK_INSTALL_STAGING = YES
+CLAPACK_PROVIDES = liblapack
 
 ifneq ($(BR2_PACKAGE_CLAPACK_ARITH_H),)
 CLAPACK_CONF_OPTS += -DARITH_H=$(BR2_PACKAGE_CLAPACK_ARITH_H)
package/lapack/Config.in deleted +0 -21
deleted file mode 100644
index 9687ace82d..0000000000
@@ -1,21 +0,0 @@
-comment "lapack/blas needs a toolchain w/ fortran"
-	depends on !(BR2_powerpc && BR2_TOOLCHAIN_USES_UCLIBC)
-	depends on !BR2_TOOLCHAIN_HAS_FORTRAN
-
-config BR2_PACKAGE_LAPACK
-	bool "lapack/blas"
-	depends on BR2_TOOLCHAIN_HAS_FORTRAN
-	# _fpu_control is used on PowerPC, but not available with uClibc
-	depends on !(BR2_powerpc && BR2_TOOLCHAIN_USES_UCLIBC)
-	help
-	  LAPACK and BLAS FORTRAN implementation. This package
-	  installs two libraries: libblas and liblapack.
-
-	  http://www.netlib.org/lapack/
-
-config BR2_PACKAGE_LAPACK_COMPLEX
-	bool "Complex/Complex16 support"
-	default y
-	depends on BR2_PACKAGE_LAPACK
-	help
-	  Builds support for COMPLEX and COMPLEX16 data types.
package/lapack/lapack.mk +1 -0
index 3b59214e4b..8d3fdc6bcf 100644
@@ -11,6 +11,7 @@ LAPACK_SITE = http://www.netlib.org/lapack
 LAPACK_INSTALL_STAGING = YES
 LAPACK_SUPPORTS_IN_SOURCE_BUILD = NO
 LAPACK_CONF_OPTS = -DLAPACKE=ON -DCBLAS=ON
+LAPACK_PROVIDES = liblapack
 
 ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
 LAPACK_CONF_OPTS += -DBUILD_COMPLEX=ON -DBUILD_COMPLEX16=ON
package/liblapack/Config.in new file +91 -0

91 lines not on this page · raw message

package/liblapack/liblapack.mk new file +7 -0
new file mode 100644
index 0000000000..06cece60f9
@@ -0,0 +1,7 @@
+################################################################################
+#
+# liblapack
+#
+################################################################################
+
+$(eval $(virtual-package))
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/5 package/liblapack: add liblapack as virtual package with lapack and clapack as providers.

Hi Alex, Romain,

Quoted 2 lines · Romain Naour

lapack and clapack are both generating the same liblapack.so but does not have the same symbols. So the run-time is broke. They have to exclude each other.

We discussed this at the hackaton and reviewed all the options that were
discussed before on the mailing list, and we concluded that the simplest solution is to remove clapack entirely. Everything that uses clapack (i.e., armadillo and numpy) can use lapack as well, as you mention, so the only reason to keep clapack would be to support toolchains without Fortran compiler. Most of our external toolchains do have Fortran support, and I expect that anybody who cares about maths wouldn't mind enabling Fortran in their toolchains.

If anybody disagrees with that, now is the time to speak up :-)

So, I've marked the rest of the series as Changes Requested.

Regards,
Arnout

Quoted 4 lines

A liblapack virtual package havev been created. It include lapack and clapack as providers.
The armadillo packet require clapack to build, but lapack with complex support works as well.

#3 lore raw

PATCH 2/5 package/liblapack: add liblapack as virtual package with lapack and clapack as providers.

Quoted 19 lines · On Sat, Aug 3, 2019, 12:23 Arnout Vandecappelle <[email protected]> wrote:

Hi Alex, Romain,

Quoted 1 line · Romain Naour

lapack and clapack are both generating the same liblapack.so but does

not have

Quoted 1 line

the same symbols. So the run-time is broke. They have to exclude each

other.

We discussed this at the hackaton and reviewed all the options that were
discussed before on the mailing list, and we concluded that the simplest solution is to remove clapack entirely. Everything that uses clapack (i.e., armadillo and numpy) can use lapack as well, as you mention, so the only reason
to keep clapack would be to support toolchains without Fortran compiler. Most of
our external toolchains do have Fortran support, and I expect that anybody who
cares about maths wouldn't mind enabling Fortran in their toolchains.

Quoted 2 lines

If anybody disagrees with that, now is the time to speak up :-)

I'm going to analyze this change in detail next week, as we've been in this situation and just now were working to switch from clapack to openblas as backend to armadillo.

I'll report back...

Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190803/a5ed1da9/attachment.html>

#4 lore raw

PATCH 2/5 package/liblapack: add liblapack as virtual package with lapack and clapack as providers.

In reply to Arnout Vandecappelle

El s?b., 3 ago. 2019 a las 12:23, Arnout Vandecappelle (<[email protected]>) escribi?:

Quoted 15 lines

Hi Alex, Romain,

Quoted 2 lines · Romain Naour

lapack and clapack are both generating the same liblapack.so but does not have the same symbols. So the run-time is broke. They have to exclude each other.

We discussed this at the hackaton and reviewed all the options that were
discussed before on the mailing list, and we concluded that the simplest solution is to remove clapack entirely. Everything that uses clapack (i.e., armadillo and numpy) can use lapack as well, as you mention, so the only reason to keep clapack would be to support toolchains without Fortran compiler. Most of our external toolchains do have Fortran support, and I expect that anybody who cares about maths wouldn't mind enabling Fortran in their toolchains.

Quoted 2 lines

If anybody disagrees with that, now is the time to speak up :-)

There may be cases where a toolchain is provided by a third-party, e.g. a silicon vendor like Marvell.
In this case, Fortran is not necessarily provided. Depending on that third-party, it may or may not be easy to get this fixed.

In our case, we have some boards with a toolchain from Marvell (no Fortran) and others where we build the toolchains ourselves. In both cases we have armadillo with clapack. Recently, we wanted to use openblas, and getting clapack to work as lapack backend for openblas did not seem to work. So for those boards were it mattered, we enabled fortran and let openblas use its own lapack implementation. For the boards with Marvell toolchain, we planned to keep the existing armadillo+clapack situation.

So removing clapack altogether would not be desirable for that case.

Note also that openblas also has lapack inside its sources. So it could also a provider for liblapack in the case of a virtual-package situation.

I searched but could not find extensive discussion on the mailing list regarding this topic.
What is the actual problem with introducing a virtual-package?

Thanks,
Thomas

#5 lore raw

PATCH 2/5 package/liblapack: add liblapack as virtual package with lapack and clapack as providers.

Hi Thomas, all.

Le lun. 5 ao?t 2019 ? 14:27, Thomas De Schampheleire < [email protected]> a ?crit :

Quoted 40 lines

El s?b., 3 ago. 2019 a las 12:23, Arnout Vandecappelle (<[email protected]>) escribi?:

Quoted 5 lines

Hi Alex, Romain,

Quoted 1 line · Romain Naour

lapack and clapack are both generating the same liblapack.so but does

not have

Quoted 1 line
Quoted 1 line

the same symbols. So the run-time is broke. They have to exclude each

other.

Quoted 4 lines

We discussed this at the hackaton and reviewed all the options that were
discussed before on the mailing list, and we concluded that the simplest solution is to remove clapack entirely. Everything that uses clapack

(i.e.,

Quoted 1 line

armadillo and numpy) can use lapack as well, as you mention, so the only

reason

Quoted 1 line

to keep clapack would be to support toolchains without Fortran compiler.

Most of

Quoted 1 line

our external toolchains do have Fortran support, and I expect that

anybody who

Quoted 4 lines

cares about maths wouldn't mind enabling Fortran in their toolchains.

If anybody disagrees with that, now is the time to speak up :-)

There may be cases where a toolchain is provided by a third-party, e.g. a silicon vendor like Marvell.
In this case, Fortran is not necessarily provided. Depending on that third-party, it may or may not be easy to get this fixed.

In our case, we have some boards with a toolchain from Marvell (no Fortran) and others where we build the toolchains ourselves. In both cases we have armadillo with clapack. Recently, we wanted to use openblas, and getting clapack to work as lapack backend for openblas did not seem to work. So for those boards were it mattered, we enabled fortran and let openblas use its own lapack implementation. For the boards with Marvell toolchain, we planned to keep the existing armadillo+clapack situation.

Quoted 6 lines

So removing clapack altogether would not be desirable for that case.

Note also that openblas also has lapack inside its sources. So it could also a provider for liblapack in the case of a virtual-package situation.

Okay, good to know, I will check in this way also. I will check which package use
libopenblas.so and how it use it.

Quoted 5 lines

I searched but could not find extensive discussion on the mailing list regarding this topic.
What is the actual problem with introducing a virtual-package?

So, for want I know and understand :
- lapack and clapack generate the same library files : liblapack.so and libblas.so.
- lapack generate libcblas.so and clapack does not.
- lapack can be built with complex support. It double the size of the library so it
should add a lot of symbols. I guess clapack is already built with complex support but I'm not sure ...

A new idee is to introduce virtual package to generate :
- liblapack.so
- libblas.so
- libcblas.so
with lapack and clapack as providers.
As you said Thomas, openblas has it own lapack implementation so, we could add openblas as a provider and create a symbolic link to liblapack.so.
I will inspect those implementations using readefl or a similar tool and try to
assert that we can use this solution.

Regards,
Aalx.

Quoted 3 lines

Thanks,
Thomas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190805/b428865f/attachment.html>

#6 lore raw

PATCH 2/5 package/liblapack: add liblapack as virtual package with lapack and clapack as providers.

Hi all.

Well, after digging lapack(with and without complex),clapack and openblas I got
some new insight.

Here is what I did :
I gather the different libraries (lapack without complex, lapack with complex,
clapack and openblas) and gather from LAPACK[1] and BLAS[2] documentation the
function list.
Then, for each function, I ran nm command on the library with a simple shell script. nm command can output symbols from the symtable of an ELF binary thanks
to -D option.

Here is the correct line option from man nm

```
nm -D OBJFILE
   Display the dynamic symbols rather than the normal symbols. This is only
   meaningful for dynamic objects, such as certain types of shared
libraries.
```

Here is the script

```
#!/bin/sh

nm -D $library_file > /tmp/tmp_nm
for line in `cat $function_file` # for each line in the file
do
    grep -i $line /tmp/tmp_nm # check if the function is in the nm dump
    if [[ $? -ne 0 ]]
    then
        echo $line >> $library_file.not_found # add to the not found file
    else
        echo $line >> $library_file.found # add to the found file
    fi
done
```

After running this script on every library with the good function set, I could
determine what is missing or not.
I will use the function set I link to perform my analysis.
So, first of all, clapack have complex support. This mean :
(clapack-)libblas.so and (lapack-complex-)libblas.so have the same functions and
(clapack-)liblapack.so and (lapack-complex-)liblapack.so have the same functions
except one symbol which differs : geqpfp in not in clapack implementation.

In theory, we could use clapack or lapack with complex support, it will change
(almost) annything.

Openblas[3] now,
openblas.so and (lapack-complex-)liblapack.so have the same functions openblas.so and (lapack-complex-)libblas.so have the same functions.

Theoretically, we could make two symbolic link on openblas.so named:
- liblapack.so
- libblas.so
and use it as lapack and blas implementation.

I dig deeper into openblas and I realise that openblas does not provide a fresh
lapack implementation. Openblas just compile lapack from netlib and link it to
provide it own library openblas.lib.
So, what about blas implementation. I'm not sure if openblas provide its own or
not ...
There is this wierd rule in openblas Makefile ...

```
blas-test:
 (cd $(NETLIB_LAPACK_DIR)/BLAS/TESTING && rm -f x* *.out)
 $(MAKE) -j 1 -C $(NETLIB_LAPACK_DIR) blas_testing
 (cd $(NETLIB_LAPACK_DIR)/BLAS/TESTING && cat *.out)
```

It clearly use the test from the netlib implementation of BLAS ...

Well ... I think, we could use those three packages as providers for
- liblapack.so
- libblas.so
(by doing symlinks on Openblas).

I still have to:
- dig into cblas and check if openblas, lapack(complex) and clapack are compatible,
- perform the tests with the symlinks on openblas
- replace the files from clapack with the one from lapack.

What do you think of this idee on virtual packages with those three providers ?

Something else about Openblas. It use netlib lapack, and doing so, need a fortran compiler, so Thomas, you cannot use Openblas for lapack implementation
on your fortran-less project ...

Regards,
Aalx.

[1]: http://www.netlib.org/lapack/lapacke.html#_real_functions
[2]: http://www.netlib.org/blas/#_blas_routines
[3]: https://github.com/xianyi/OpenBLAS

Le lun. 5 ao?t 2019 ? 14:49, Alexandre PAYEN <[email protected]> a ?crit :

Quoted 82 lines

Hi Thomas, all.

Le lun. 5 ao?t 2019 ? 14:27, Thomas De Schampheleire < patrickdepinguin at gmail.com> a ?crit :

Quoted 41 lines

El s?b., 3 ago. 2019 a las 12:23, Arnout Vandecappelle (<[email protected]>) escribi?:

Quoted 5 lines

Hi Alex, Romain,

Quoted 1 line · Romain Naour

lapack and clapack are both generating the same liblapack.so but does

not have

Quoted 1 line
Quoted 1 line

the same symbols. So the run-time is broke. They have to exclude each

other.

Quoted 2 lines

We discussed this at the hackaton and reviewed all the options that

were

Quoted 2 lines

discussed before on the mailing list, and we concluded that the simplest solution is to remove clapack entirely. Everything that uses clapack

(i.e.,

Quoted 1 line

armadillo and numpy) can use lapack as well, as you mention, so the

only reason

Quoted 1 line

to keep clapack would be to support toolchains without Fortran

compiler. Most of

Quoted 1 line

our external toolchains do have Fortran support, and I expect that

anybody who

Quoted 4 lines

cares about maths wouldn't mind enabling Fortran in their toolchains.

If anybody disagrees with that, now is the time to speak up :-)

There may be cases where a toolchain is provided by a third-party, e.g. a silicon vendor like Marvell.
In this case, Fortran is not necessarily provided. Depending on that third-party, it may or may not be easy to get this fixed.

In our case, we have some boards with a toolchain from Marvell (no Fortran) and others where we build the toolchains ourselves. In both cases we have armadillo with clapack. Recently, we wanted to use openblas, and getting clapack to work as lapack backend for openblas did not seem to work. So for those boards were it mattered, we enabled fortran and let openblas use its own lapack implementation. For the boards with Marvell toolchain, we planned to keep the existing armadillo+clapack situation.

Quoted 6 lines

So removing clapack altogether would not be desirable for that case.

Note also that openblas also has lapack inside its sources. So it could also a provider for liblapack in the case of a virtual-package situation.

Okay, good to know, I will check in this way also. I will check which package use
libopenblas.so and how it use it.

Quoted 5 lines

I searched but could not find extensive discussion on the mailing list regarding this topic.
What is the actual problem with introducing a virtual-package?

So, for want I know and understand :
- lapack and clapack generate the same library files : liblapack.so and libblas.so.
- lapack generate libcblas.so and clapack does not.
- lapack can be built with complex support. It double the size of the library so it
should add a lot of symbols. I guess clapack is already built with complex support but I'm not sure ...

A new idee is to introduce virtual package to generate :
- liblapack.so
- libblas.so
- libcblas.so
with lapack and clapack as providers.
As you said Thomas, openblas has it own lapack implementation so, we could add openblas as a provider and create a symbolic link to liblapack.so.
I will inspect those implementations using readefl or a similar tool and try to
assert that we can use this solution.

Quoted 8 lines

Regards,
Aalx.

Quoted 3 lines

Thanks,
Thomas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190806/5a3c0186/attachment.html>

#7 lore raw

PATCH 2/5 package/liblapack: add liblapack as virtual package with lapack and clapack as providers.

Hi all,

I realise many thinks in this lapack/clapack problem. First of all, clapack is completly out dated ...
The last release is version 3.2.1 when lapack is at version 3.8.0 Also, since ten years, no one touch it ...
I really suggest to drop this package from buildroot ...

OpenBLAS is an implementation of BLAS. It can also generate a lapack implementation if needed. This implementation is lapack from netlib (lapack already present in buildroot). So, we should remove this from OpenBLAS and use the already present lapack in the build system.

Also, lapack from netlib advise to use another BLAS implementation than netlib BLAS. Perfect, we can use OpenBLAS implementation.

I suggest to make a virtual package libblas with OpenBLAS as provider ATLAS could be added later. And depending from this virtual package, a liblapack virtual package with lapack, clapack more could be added. This is the easiest way. But be carefull because lapack generate a libblas.so file so, just the liblapack.so should be built.

The second solution is to keep the same package architecture as now but linking OpenBLAS and lapack. So if they are both selected, OpenBLAS should be built with lapack, and lapack should not build libblas.so. Else, only lapack should build normaly or only libblas.so from OpenBLAS.

I think the maintain cost for the second solution is higher. Also, it is simplier to build the first solution.

Tell me what you think of that and I will work on a patch.

Regards,
Alexandre

Le mar. 6 ao?t 2019 ? 17:57, Alexandre PAYEN <[email protected]> a ?crit :

Quoted 107 lines

Hi all.

Well, after digging lapack(with and without complex),clapack and openblas I got
some new insight.

Here is what I did :
I gather the different libraries (lapack without complex, lapack with complex,
clapack and openblas) and gather from LAPACK[1] and BLAS[2] documentation the
function list.
Then, for each function, I ran nm command on the library with a simple shell
script. nm command can output symbols from the symtable of an ELF binary thanks
to -D option.

Here is the correct line option from man nm

```
nm -D OBJFILE
   Display the dynamic symbols rather than the normal symbols. This is only
   meaningful for dynamic objects, such as certain types of shared
libraries.
```

Here is the script

```
#!/bin/sh

nm -D $library_file > /tmp/tmp_nm
for line in `cat $function_file` # for each line in the file
do
    grep -i $line /tmp/tmp_nm # check if the function is in the nm dump
    if [[ $? -ne 0 ]]
    then
        echo $line >> $library_file.not_found # add to the not found file
    else
        echo $line >> $library_file.found # add to the found file
    fi
done
```

After running this script on every library with the good function set, I could
determine what is missing or not.
I will use the function set I link to perform my analysis. So, first of all, clapack have complex support. This mean :
(clapack-)libblas.so and (lapack-complex-)libblas.so have the same functions and
(clapack-)liblapack.so and (lapack-complex-)liblapack.so have the same functions
except one symbol which differs : geqpfp in not in clapack implementation.

In theory, we could use clapack or lapack with complex support, it will change
(almost) annything.

Openblas[3] now,
openblas.so and (lapack-complex-)liblapack.so have the same functions openblas.so and (lapack-complex-)libblas.so have the same functions.

Theoretically, we could make two symbolic link on openblas.so named:
- liblapack.so
- libblas.so
and use it as lapack and blas implementation.

I dig deeper into openblas and I realise that openblas does not provide a fresh
lapack implementation. Openblas just compile lapack from netlib and link it to
provide it own library openblas.lib.
So, what about blas implementation. I'm not sure if openblas provide its own or
not ...
There is this wierd rule in openblas Makefile ...

```
blas-test:
 (cd $(NETLIB_LAPACK_DIR)/BLAS/TESTING && rm -f x* *.out)
 $(MAKE) -j 1 -C $(NETLIB_LAPACK_DIR) blas_testing
 (cd $(NETLIB_LAPACK_DIR)/BLAS/TESTING && cat *.out)
```

It clearly use the test from the netlib implementation of BLAS ...

Well ... I think, we could use those three packages as providers for
- liblapack.so
- libblas.so
(by doing symlinks on Openblas).

I still have to:
- dig into cblas and check if openblas, lapack(complex) and clapack are compatible,
- perform the tests with the symlinks on openblas
- replace the files from clapack with the one from lapack.

What do you think of this idee on virtual packages with those three providers ?

Something else about Openblas. It use netlib lapack, and doing so, need a fortran compiler, so Thomas, you cannot use Openblas for lapack implementation
on your fortran-less project ...

Regards,
Aalx.

Quoted 99 lines

[1]: http://www.netlib.org/lapack/lapacke.html#_real_functions
[2]: http://www.netlib.org/blas/#_blas_routines
[3]: https://github.com/xianyi/OpenBLAS

Le lun. 5 ao?t 2019 ? 14:49, Alexandre PAYEN <[email protected]> a ?crit :

Quoted 91 lines

Hi Thomas, all.

Le lun. 5 ao?t 2019 ? 14:27, Thomas De Schampheleire < patrickdepinguin at gmail.com> a ?crit :

Quoted 42 lines

El s?b., 3 ago. 2019 a las 12:23, Arnout Vandecappelle (<[email protected]>) escribi?:

Quoted 5 lines

Hi Alex, Romain,

Quoted 1 line · Romain Naour

lapack and clapack are both generating the same liblapack.so but

does not have

Quoted 1 line
Quoted 1 line

the same symbols. So the run-time is broke. They have to exclude

each other.

Quoted 2 lines

We discussed this at the hackaton and reviewed all the options that

were

Quoted 1 line

discussed before on the mailing list, and we concluded that the

simplest

Quoted 1 line

solution is to remove clapack entirely. Everything that uses clapack

(i.e.,

Quoted 1 line

armadillo and numpy) can use lapack as well, as you mention, so the

only reason

Quoted 1 line

to keep clapack would be to support toolchains without Fortran

compiler. Most of

Quoted 1 line

our external toolchains do have Fortran support, and I expect that

anybody who

Quoted 4 lines

cares about maths wouldn't mind enabling Fortran in their toolchains.

If anybody disagrees with that, now is the time to speak up :-)

There may be cases where a toolchain is provided by a third-party, e.g. a silicon vendor like Marvell.
In this case, Fortran is not necessarily provided. Depending on that third-party, it may or may not be easy to get this fixed.

In our case, we have some boards with a toolchain from Marvell (no Fortran) and others where we build the toolchains ourselves. In both cases we have armadillo with clapack. Recently, we wanted to use openblas, and getting clapack to work as lapack backend for openblas did not seem to work. So for those boards were it mattered, we enabled fortran and let openblas use its own lapack implementation. For the boards with Marvell toolchain, we planned to keep the existing armadillo+clapack situation.

Quoted 6 lines

So removing clapack altogether would not be desirable for that case.

Note also that openblas also has lapack inside its sources. So it could also a provider for liblapack in the case of a virtual-package situation.

Okay, good to know, I will check in this way also. I will check which package use
libopenblas.so and how it use it.

Quoted 5 lines

I searched but could not find extensive discussion on the mailing list regarding this topic.
What is the actual problem with introducing a virtual-package?

So, for want I know and understand :
- lapack and clapack generate the same library files : liblapack.so and libblas.so.
- lapack generate libcblas.so and clapack does not.
- lapack can be built with complex support. It double the size of the library so it
should add a lot of symbols. I guess clapack is already built with complex support but I'm not sure ...

A new idee is to introduce virtual package to generate :
- liblapack.so
- libblas.so
- libcblas.so
with lapack and clapack as providers.
As you said Thomas, openblas has it own lapack implementation so, we could add openblas as a provider and create a symbolic link to liblapack.so.
I will inspect those implementations using readefl or a similar tool and try to
assert that we can use this solution.

Regards,
Aalx.

Quoted 3 lines

Thanks,
Thomas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190809/bf884a76/attachment.html>

#8 lore raw

PATCH 2/5 package/liblapack: add liblapack as virtual package with lapack and clapack as providers.

Hi Alex,

Thanks for continuing this investigation...

Since I looked at the lapack/numpy hell during the hackaton, I'm kind of
responsible to follow up on this :-(

Quoted 7 lines · Alexandre PAYEN

Hi all,

I realise many thinks in this lapack/clapack problem. First of all, clapack is completly out dated ...
The last release is version 3.2.1 when lapack is at version 3.8.0 Also, since ten years, no one touch it ...
I really suggest to drop this package from buildroot ...

I expect that ThomasDS wouldn't like that :-)

It is pretty darn annoying though that we're stuck with this old clapack,
because it is not 100% identical any more - cfr. the missing function you identified.

Quoted 4 lines

OpenBLAS is an implementation of BLAS. It can also generate a lapack implementation if needed. This implementation is lapack from netlib (lapack already present in buildroot). So, we should remove this from OpenBLAS and use the already present lapack in the build system.

+1 to that, it really is the 3.8.0 version that is bundled in openblas.

However, openblas appears to also have a C implementation of lapack. It doesn't
appear to be built with our .mk file for some reason though.

Quoted 8 lines

Also, lapack from netlib advise to use another BLAS implementation than netlib BLAS. Perfect, we can use OpenBLAS implementation.

I suggest to make a virtual package libblas with OpenBLAS as provider ATLAS could be added later. And depending from this virtual package, a liblapack virtual package with lapack, clapack more could be added. This is the easiest way. But be carefull because lapack generate a libblas.so file so, just the liblapack.so should be built.

It's a little more complicated I'm afraid... Based on what I've seen in numpy,
I think there are in fact 4 different interfaces:

lapack (= fortran ABI)-> provided by lapack, clapack lapacke (= C ABI) -> provided by lapack
blas (= fortran ABI) -> provided by lapack, clapack cblas (= C ABI) -> provided by lapack

I don't really understand which of these openblas provides.

As far as I understand, the preferred combination is cblas + lapack. If I
remember correctly from our (= Romain and me) analysis two weeks ago, numpy will try to use lapack+cblas and fall back to blas if it can't find cblas.

Quoted 5 lines

The second solution is to keep the same package architecture as now but linking OpenBLAS and lapack. So if they are both selected, OpenBLAS should be built with lapack, and lapack should not build libblas.so. Else, only lapack should build normaly or only libblas.so from OpenBLAS.

So you're saying:

* openblas only -> only libblas.so
* lapack only -> libblas.so, libcblas.so, liblapack.so
* both -> libblas.so from openblas, libcblas.o and liblapack.so from lapack

Is that correct?

The ideal would be if it turns out that openblas can build everything without
Fortran compiler, and if the packages that can use lapack (armadillo and numpy) work with either lapack or openblas. Then we could just get rid of clapack and make a single virtual package. Or even better, we could remove both clapack and lapack and use openblas's bundled lapack-netlib if there is a Fortran, and openblas's C implementation of lapack if there is no Fortran.

But I don't know if that is even possible...

Regards,
Arnout

Quoted 201 lines

I think the maintain cost for the second solution is higher. Also, it is simplier to build the first solution.

Tell me what you think of that and I will work on a patch.

Regards,
Alexandre

Le?mar. 6 ao?t 2019 ??17:57, Alexandre PAYEN <[email protected] <mailto:[email protected]>> a ?crit?:

Hi all.

Well, after digging lapack(with and without complex),clapack and openblas I got some new insight.

Here is what I did :
I gather the different libraries (lapack without complex, lapack with complex, clapack and openblas) and gather from LAPACK[1] and BLAS[2] documentation the function list.
Then, for each function, I ran nm command on the library with a simple shell script. nm command can output symbols from the symtable of an ELF binary thanks to -D option.

Here is the correct line option from man nm
```
nm -D OBJFILE
? ?Display the dynamic symbols rather than the normal symbols. This is only ? ?meaningful for dynamic objects, such as certain types of shared libraries. ```

Here is the script
```
#!/bin/sh

nm -D $library_file > /tmp/tmp_nm
for line in cat $function_file # for each line in the file do
? ? grep -i $line /tmp/tmp_nm # check if the function is in the nm dump ? ? if [[ $? -ne 0 ]]
? ? then
? ? ? ? echo $line >> $library_file.not_found # add to the not found file ? ? else
? ? ? ? echo $line >> $library_file.found # add to the found file ? ? fi
done
```

After running this script on every library with the good function set, I could determine what is missing or not.
I will use the function set I link to perform my analysis.
So, first of all, clapack have complex support. This mean :
(clapack-)libblas.so and (lapack-complex-)libblas.so have the same functions and (clapack-)liblapack.so and (lapack-complex-)liblapack.so have the same functions except one symbol which differs : geqpfp in not in clapack implementation.

In theory, we could use clapack or lapack with complex support, it will change (almost) annything.

Openblas[3] now,
openblas.so and (lapack-complex-)liblapack.so have the same functions openblas.so and (lapack-complex-)libblas.so have the same functions.

Theoretically, we could make two symbolic link on openblas.so named:
- liblapack.so
- libblas.so
and use it as lapack and blas implementation.

I dig deeper into openblas and I realise that openblas does not provide a fresh lapack implementation. Openblas just compile lapack from netlib and link it to provide it own library openblas.lib.
So, what about blas implementation. I'm not sure if openblas provide its own or not ...
There is this wierd rule in openblas Makefile ...
```
blas-test:
?(cd $(NETLIB_LAPACK_DIR)/BLAS/TESTING && rm -f x* *.out)
?$(MAKE) -j 1 -C $(NETLIB_LAPACK_DIR) blas_testing
?(cd $(NETLIB_LAPACK_DIR)/BLAS/TESTING && cat *.out)
```
It clearly use the test from the netlib implementation of BLAS ...

Well ... I think, we could use those three packages as providers for
- liblapack.so
- libblas.so
(by doing symlinks on Openblas).

I still have to:
- dig into cblas and check if openblas, lapack(complex) and clapack are compatible,
- perform the tests with the symlinks on openblas
- replace the files from clapack with the one from lapack.

What do you think of this idee on virtual packages with those three providers ?

Something else about Openblas. It use netlib lapack, and doing so, need a fortran compiler, so Thomas, you cannot use Openblas for lapack implementation on your fortran-less project ...

Regards,
Aalx.

[1]: http://www.netlib.org/lapack/lapacke.html#_real_functions
[2]: http://www.netlib.org/blas/#_blas_routines
[3]: https://github.com/xianyi/OpenBLAS

Le?lun. 5 ao?t 2019 ??14:49, Alexandre PAYEN <[email protected] <mailto:[email protected]>> a ?crit?:

Hi Thomas, all.

Le?lun. 5 ao?t 2019 ??14:27, Thomas De Schampheleire <patrickdepinguin at gmail.com <mailto:[email protected]>> a ?crit?:

El s?b., 3 ago. 2019 a las 12:23, Arnout Vandecappelle (<arnout at mind.be <mailto:[email protected]>>) escribi?:
>
>? Hi Alex, Romain,
>
> On 01/08/2019 23:09, Romain Naour wrote:
> > lapack and clapack are both generating the same liblapack.so but does not have
> > the same symbols. So the run-time is broke. They have to exclude each other.
>
>? We discussed this at the hackaton and reviewed all the options that were
> discussed before on the mailing list, and we concluded that the simplest
> solution is to remove clapack entirely. Everything that uses clapack (i.e.,
> armadillo and numpy) can use lapack as well, as you mention, so the only reason
> to keep clapack would be to support toolchains without Fortran compiler. Most of
> our external toolchains do have Fortran support, and I expect that anybody who
> cares about maths wouldn't mind enabling Fortran in their toolchains. >
>? If anybody disagrees with that, now is the time to speak up :-) >

There may be cases where a toolchain is provided by a third-party, e.g. a silicon vendor like Marvell.
In this case, Fortran is not necessarily provided. Depending on that third-party, it may or may not be easy to get this fixed.

In our case, we have some boards with a toolchain from Marvell (no Fortran) and others where we build the toolchains ourselves. In both cases we have armadillo with clapack. Recently, we wanted to use openblas, and getting clapack to work as lapack backend for openblas did not seem to work. So for those boards were it mattered, we enabled fortran and let openblas use its own lapack implementation. For the boards with Marvell toolchain, we planned to keep the existing armadillo+clapack situation.

So removing clapack altogether would not be desirable for that case.

Note also that openblas also has lapack inside its sources. So it could also a provider for liblapack in the case of a virtual-package situation.

Okay, good to know, I will check in this way also. I will check which package use
libopenblas.so and how it use it.

I searched but could not find extensive discussion on the mailing list regarding this topic.
What is the actual problem with introducing a virtual-package?

So, for want I know and understand :
- lapack and clapack generate the same library files : liblapack.so and libblas.so.
- lapack generate libcblas.so and clapack does not.
- lapack can be built with complex support. It double the size of the library so it
should add a lot of symbols. I guess clapack is already built with complex support but I'm not sure ...

A new idee is to introduce virtual package to generate :
- liblapack.so
- libblas.so
- libcblas.so
with lapack and clapack as providers.
As you said Thomas, openblas has it own lapack implementation so, we could add openblas as a provider and create a symbolic link to liblapack.so.
I will inspect those implementations using readefl or a similar tool and try to
assert that we can use this solution.

Regards,
Aalx.

Thanks,
Thomas

#9 lore raw

PATCH 2/5 package/liblapack: add liblapack as virtual package with lapack and clapack as providers.

Hi, Arnout, Thomas, Alexandre,

Le 15/08/2019 ? 15:03, Arnout Vandecappelle a ?crit?:

Quoted 6 lines

Hi Alex,

Thanks for continuing this investigation...

Since I looked at the lapack/numpy hell during the hackaton, I'm kind of
responsible to follow up on this :-(

Thanks for your time Arnout :)

Quoted 10 lines · Alexandre PAYEN
Quoted 7 lines

Hi all,

I realise many thinks in this lapack/clapack problem. First of all, clapack is completly out dated ...
The last release is version 3.2.1 when lapack is at version 3.8.0 Also, since ten years, no one touch it ...
I really suggest to drop this package from buildroot ...

Quoted 5 lines

I expect that ThomasDS wouldn't like that :-)

It is pretty darn annoying though that we're stuck with this old clapack,
because it is not 100% identical any more - cfr. the missing function you identified.

@Thomas: Do you know about libflame?
https://www.cs.utexas.edu/~flame/web/libFLAME.html

Especially that part:

"Independence from Fortran and LAPACK. The libflame development team is pleased to offer a high-performance linear algebra solution that is 100% Fortran-free. libflame is a C-only implementation and does not depend on any external Fortran libraries, such as LAPACK. That said, we happily provide an optional backward compatibility layer, lapack2flame, that maps legacy LAPACK routine invocations to their corresponding native C implementations in libflame."

Indeed, we can build libflame without gfortran maybe you can replace your old clapack? (To be tested).

Note: libflame is yet another blas/cblas/lapack provider :)

Quoted 21 lines
Quoted 5 lines

OpenBLAS is an implementation of BLAS. It can also generate a lapack implementation if needed. This implementation is lapack from netlib (lapack already present in buildroot). So, we should remove this from OpenBLAS and use the already present lapack in the build system.

+1 to that, it really is the 3.8.0 version that is bundled in openblas.

However, openblas appears to also have a C implementation of lapack. It doesn't
appear to be built with our .mk file for some reason though.

Quoted 8 lines

Also, lapack from netlib advise to use another BLAS implementation than netlib BLAS. Perfect, we can use OpenBLAS implementation.

I suggest to make a virtual package libblas with OpenBLAS as provider ATLAS could be added later. And depending from this virtual package, a liblapack virtual package with lapack, clapack more could be added. This is the easiest way. But be carefull because lapack generate a libblas.so file so, just the liblapack.so should be built.

Quoted 7 lines

It's a little more complicated I'm afraid... Based on what I've seen in numpy,
I think there are in fact 4 different interfaces:

lapack (= fortran ABI)-> provided by lapack, clapack lapacke (= C ABI) -> provided by lapack
blas (= fortran ABI) -> provided by lapack, clapack cblas (= C ABI) -> provided by lapack

The new numpy release (1.17.0) now support for building NumPy with the libFLAME linear algebra package as the LAPACK implementation:

https://github.com/numpy/numpy/releases/tag/v1.17.0

Quoted 14 lines

I don't really understand which of these openblas provides.

As far as I understand, the preferred combination is cblas + lapack. If I
remember correctly from our (= Romain and me) analysis two weeks ago, numpy will try to use lapack+cblas and fall back to blas if it can't find cblas.

Quoted 5 lines

The second solution is to keep the same package architecture as now but linking OpenBLAS and lapack. So if they are both selected, OpenBLAS should be built with lapack, and lapack should not build libblas.so. Else, only lapack should build normaly or only libblas.so from OpenBLAS.

Quoted 7 lines

So you're saying:

* openblas only -> only libblas.so
* lapack only -> libblas.so, libcblas.so, liblapack.so
* both -> libblas.so from openblas, libcblas.o and liblapack.so from lapack

Is that correct?

That what I understand.
I'm wondering if we should use libcblas.so from openblas instead.

Quoted 10 lines

The ideal would be if it turns out that openblas can build everything without
Fortran compiler, and if the packages that can use lapack (armadillo and numpy) work with either lapack or openblas. Then we could just get rid of clapack and make a single virtual package. Or even better, we could remove both clapack and lapack and use openblas's bundled lapack-netlib if there is a Fortran, and openblas's C implementation of lapack if there is no Fortran.

But I don't know if that is even possible...

I don't think so.

The libflame seems our only hope :)

Best regards,
Romain

Quoted 211 lines

Regards,
Arnout

Quoted 194 lines

I think the maintain cost for the second solution is higher. Also, it is simplier to build the first solution.

Tell me what you think of that and I will work on a patch.

Regards,
Alexandre

Le?mar. 6 ao?t 2019 ??17:57, Alexandre PAYEN <[email protected] <mailto:[email protected]>> a ?crit?:

Hi all.

Well, after digging lapack(with and without complex),clapack and openblas I got some new insight.

Here is what I did :
I gather the different libraries (lapack without complex, lapack with complex, clapack and openblas) and gather from LAPACK[1] and BLAS[2] documentation the function list.
Then, for each function, I ran nm command on the library with a simple shell script. nm command can output symbols from the symtable of an ELF binary thanks to -D option.

Here is the correct line option from man nm
```
nm -D OBJFILE
? ?Display the dynamic symbols rather than the normal symbols. This is only ? ?meaningful for dynamic objects, such as certain types of shared libraries. ```

Here is the script
```
#!/bin/sh

nm -D $library_file > /tmp/tmp_nm
for line in cat $function_file # for each line in the file do
? ? grep -i $line /tmp/tmp_nm # check if the function is in the nm dump ? ? if [[ $? -ne 0 ]]
? ? then
? ? ? ? echo $line >> $library_file.not_found # add to the not found file ? ? else
? ? ? ? echo $line >> $library_file.found # add to the found file ? ? fi
done
```

After running this script on every library with the good function set, I could determine what is missing or not.
I will use the function set I link to perform my analysis.
So, first of all, clapack have complex support. This mean :
(clapack-)libblas.so and (lapack-complex-)libblas.so have the same functions and (clapack-)liblapack.so and (lapack-complex-)liblapack.so have the same functions except one symbol which differs : geqpfp in not in clapack implementation.

In theory, we could use clapack or lapack with complex support, it will change (almost) annything.

Openblas[3] now,
openblas.so and (lapack-complex-)liblapack.so have the same functions openblas.so and (lapack-complex-)libblas.so have the same functions.

Theoretically, we could make two symbolic link on openblas.so named:
- liblapack.so
- libblas.so
and use it as lapack and blas implementation.

I dig deeper into openblas and I realise that openblas does not provide a fresh lapack implementation. Openblas just compile lapack from netlib and link it to provide it own library openblas.lib.
So, what about blas implementation. I'm not sure if openblas provide its own or not ...
There is this wierd rule in openblas Makefile ...
```
blas-test:
?(cd $(NETLIB_LAPACK_DIR)/BLAS/TESTING && rm -f x* *.out)
?$(MAKE) -j 1 -C $(NETLIB_LAPACK_DIR) blas_testing
?(cd $(NETLIB_LAPACK_DIR)/BLAS/TESTING && cat *.out)
```
It clearly use the test from the netlib implementation of BLAS ...

Well ... I think, we could use those three packages as providers for
- liblapack.so
- libblas.so
(by doing symlinks on Openblas).

I still have to:
- dig into cblas and check if openblas, lapack(complex) and clapack are compatible,
- perform the tests with the symlinks on openblas
- replace the files from clapack with the one from lapack.

What do you think of this idee on virtual packages with those three providers ?

Something else about Openblas. It use netlib lapack, and doing so, need a fortran compiler, so Thomas, you cannot use Openblas for lapack implementation on your fortran-less project ...

Regards,
Aalx.

[1]: http://www.netlib.org/lapack/lapacke.html#_real_functions
[2]: http://www.netlib.org/blas/#_blas_routines
[3]: https://github.com/xianyi/OpenBLAS

Le?lun. 5 ao?t 2019 ??14:49, Alexandre PAYEN <[email protected] <mailto:[email protected]>> a ?crit?:

Hi Thomas, all.

Le?lun. 5 ao?t 2019 ??14:27, Thomas De Schampheleire <patrickdepinguin at gmail.com <mailto:[email protected]>> a ?crit?:

El s?b., 3 ago. 2019 a las 12:23, Arnout Vandecappelle (<arnout at mind.be <mailto:[email protected]>>) escribi?:
>
>? Hi Alex, Romain,
>
> On 01/08/2019 23:09, Romain Naour wrote:
> > lapack and clapack are both generating the same liblapack.so but does not have
> > the same symbols. So the run-time is broke. They have to exclude each other.
>
>? We discussed this at the hackaton and reviewed all the options that were
> discussed before on the mailing list, and we concluded that the simplest
> solution is to remove clapack entirely. Everything that uses clapack (i.e.,
> armadillo and numpy) can use lapack as well, as you mention, so the only reason
> to keep clapack would be to support toolchains without Fortran compiler. Most of
> our external toolchains do have Fortran support, and I expect that anybody who
> cares about maths wouldn't mind enabling Fortran in their toolchains. >
>? If anybody disagrees with that, now is the time to speak up :-) >

There may be cases where a toolchain is provided by a third-party, e.g. a silicon vendor like Marvell.
In this case, Fortran is not necessarily provided. Depending on that third-party, it may or may not be easy to get this fixed.

In our case, we have some boards with a toolchain from Marvell (no Fortran) and others where we build the toolchains ourselves. In both cases we have armadillo with clapack. Recently, we wanted to use openblas, and getting clapack to work as lapack backend for openblas did not seem to work. So for those boards were it mattered, we enabled fortran and let openblas use its own lapack implementation. For the boards with Marvell toolchain, we planned to keep the existing armadillo+clapack situation.

So removing clapack altogether would not be desirable for that case.

Note also that openblas also has lapack inside its sources. So it could also a provider for liblapack in the case of a virtual-package situation.

Okay, good to know, I will check in this way also. I will check which package use
libopenblas.so and how it use it.

I searched but could not find extensive discussion on the mailing list regarding this topic.
What is the actual problem with introducing a virtual-package?

So, for want I know and understand :
- lapack and clapack generate the same library files : liblapack.so and libblas.so.
- lapack generate libcblas.so and clapack does not.
- lapack can be built with complex support. It double the size of the library so it
should add a lot of symbols. I guess clapack is already built with complex support but I'm not sure ...

A new idee is to introduce virtual package to generate :
- liblapack.so
- libblas.so
- libcblas.so
with lapack and clapack as providers.
As you said Thomas, openblas has it own lapack implementation so, we could add openblas as a provider and create a symbolic link to liblapack.so.
I will inspect those implementations using readefl or a similar tool and try to
assert that we can use this solution.

Quoted 7 lines

Regards,
Aalx.

Thanks,
Thomas

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#10 lore raw

PATCH 3/5 package/python-numpy: use liblapack virtual package

In reply to Romain Naour

From: Alexandre PAYEN <[email protected]>

With the virtual packet liblapack, numpy can be build using both providers (clapack or lapack).

With lapack as provider we needs to select BR2_PACKAGE_LAPACK_COMPLEX.

---
 package/python-numpy/Config.in       | 1 +
 package/python-numpy/python-numpy.mk | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
package/python-numpy/Config.in +1 -0
index c337675ceb..6d0b373413 100644
@@ -16,6 +16,7 @@ config BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
 config BR2_PACKAGE_PYTHON_NUMPY
 	bool "python-numpy"
 	depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
+	select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK
 	help
 	  NumPy is the fundamental package for scientific computing
 	  with Python.
package/python-numpy/python-numpy.mk +2 -2
index aa8052dca4..602eb3bad9 100644
@@ -15,8 +15,8 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \
 			numpy/core/src/multiarray/dragon4.c
 PYTHON_NUMPY_SETUP_TYPE = setuptools
 
-ifeq ($(BR2_PACKAGE_CLAPACK),y)
-PYTHON_NUMPY_DEPENDENCIES += clapack
+ifeq ($(BR2_PACKAGE_LIBLAPACK),y)
+PYTHON_NUMPY_DEPENDENCIES += liblapack
 PYTHON_NUMPY_SITE_CFG_LIBS += cblas lapack
 else
 PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
Signature · 1 line
2.20.1
#11 lore raw

PATCH 4/5 package/python-numpy: disable numpy if fenv.h is not provided by libc

In reply to Romain Naour

From: Damien DUVAL <[email protected]>

With a c library which does not provide fenv.h, it won't work at runtime:
Crash after an "import numpy" on python.

Since numpy v1.16.0:
"Alpine Linux (and other musl c library distros) support
We now default to use fenv.h for floating point status error reporting. Previously we had a broken default that sometimes would not report underflow, overflow, and invalid floating point operations. Now we can support non-glibc distrubutions like Alpine Linux as long as they ship fenv.h."
Also there is a lack of define in glibc for ARC architecture and python numpy cannot build.
The removed patches were just fixes for the build for uclibc and glibc for ARC

Configurations which does not provide fenv.h (like uclibc) won't be able to select the python-numpy package to avoid errors at runtime.

---
 .../0001-Don-t-use-fenv.h-on-uClibc.patch     | 30 ------------------
 ...-no-FPU-exceptions-bits-on-ARC-glibc.patch | 31 -------------------
 package/python-numpy/Config.in                | 10 ++++++
 3 files changed, 10 insertions(+), 61 deletions(-)
 delete mode 100644 package/python-numpy/0001-Don-t-use-fenv.h-on-uClibc.patch
 delete mode 100644 package/python-numpy/0002-FIX-no-FPU-exceptions-bits-on-ARC-glibc.patch
package/python-numpy/0001-Don-t-use-fenv.h-on-uClibc.patch deleted +0 -30
deleted file mode 100644
index 8b3937a1ba..0000000000
@@ -1,30 +0,0 @@
-From 52b47439d17463304e5bd7974dec17ced0b0f24a Mon Sep 17 00:00:00 2001
-From: Fabrice Fontaine <[email protected]>
-Date: Sat, 16 Mar 2019 10:38:27 +0100
-Subject: [PATCH] Don't use <fenv.h> on uClibc
-
-Signed-off-by: Thomas Petazzoni <[email protected]>
-Signed-off-by: Alexey Brodkin <[email protected]>
-Signed-off-by: Fabrice Fontaine <[email protected]>
-[Upstream status: https://github.com/numpy/numpy/pull/13137]
----
- numpy/core/src/npymath/ieee754.c.src | 3 ++-
- 1 file changed, 2 insertions(+), 1 deletion(-)
-
-diff --git a/numpy/core/src/npymath/ieee754.c.src b/numpy/core/src/npymath/ieee754.c.src
-index d960838c8..f3f15f841 100644
---- a/numpy/core/src/npymath/ieee754.c.src
-+++ b/numpy/core/src/npymath/ieee754.c.src
-@@ -681,7 +681,8 @@ void npy_set_floatstatus_invalid(void)
-     fp_raise_xcp(FP_INVALID);
- }
- 
--#elif defined(_MSC_VER) || (defined(__osf__) && defined(__alpha))
-+#elif defined(_MSC_VER) || (defined(__osf__) && defined(__alpha)) || \
-+      defined (__UCLIBC__)
- 
- /*
-  * By using a volatile floating point value,
--- 
-2.14.1
-
package/python-numpy/0002-FIX-no-FPU-exceptions-bits-on-ARC-glibc.patch deleted +0 -31
deleted file mode 100644
index ece52118d7..0000000000
@@ -1,31 +0,0 @@
-From 1125f1ee33324bc91b4e8dd9da49163af572d04a Mon Sep 17 00:00:00 2001
-From: Fabrice Fontaine <[email protected]>
-Date: Sat, 16 Mar 2019 10:48:25 +0100
-Subject: [PATCH] FIX: no FPU exceptions bits on ARC glibc
-
-The FPU exceptions bits are missing in fenv.h in glibc for ARC
-architecture.
-
-Signed-off-by: Evgeniy Didin <[email protected]>
-Signed-off-by: Fabrice Fontaine <[email protected]>
-[Upstream status: https://github.com/numpy/numpy/pull/13137]
----
- numpy/core/src/npymath/ieee754.c.src | 2 +-
- 1 file changed, 1 insertion(+), 1 deletion(-)
-
-diff --git a/numpy/core/src/npymath/ieee754.c.src b/numpy/core/src/npymath/ieee754.c.src
-index f3f15f841..3f66b24a4 100644
---- a/numpy/core/src/npymath/ieee754.c.src
-+++ b/numpy/core/src/npymath/ieee754.c.src
-@@ -682,7 +682,7 @@ void npy_set_floatstatus_invalid(void)
- }
- 
- #elif defined(_MSC_VER) || (defined(__osf__) && defined(__alpha)) || \
--      defined (__UCLIBC__)
-+      defined (__UCLIBC__) || (defined(__arc__) && defined(__GLIBC__))
- 
- /*
-  * By using a volatile floating point value,
--- 
-2.14.1
-
package/python-numpy/Config.in +10 -0
index 6d0b373413..bcd51e6000 100644
@@ -16,6 +16,10 @@ config BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
 config BR2_PACKAGE_PYTHON_NUMPY
 	bool "python-numpy"
 	depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
+	# python-numpy needs fenv.h which is not provided by uclibc
+	depends on BR2_TOOLCHAIN_USES_GLIBC || BR2_TOOLCHAIN_USES_MUSL
+	# python-numpy is not compatible with glibc for ARC architecture.
+	depends on !(BR2_TOOLCHAIN_USES_GLIBC && BR2_arc)
 	select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK
 	help
 	  NumPy is the fundamental package for scientific computing
@@ -25,3 +29,9 @@ config BR2_PACKAGE_PYTHON_NUMPY
 	  C library.
 
 	  http://www.numpy.org/
+
+comment "python-numpy needs glibc or musl"
+	depends on !(BR2_TOOLCHAIN_USES_GLIBC || BR2_TOOLCHAIN_USES_MUSL)
+
+comment "python-numpy is not compatible with glibc for ARC architecture"
+	depends on BR2_TOOLCHAIN_USES_GLIBC && BR2_arc
Signature · 1 line
2.20.1
#12 lore raw

PATCH 4/5 package/python-numpy: disable numpy if fenv.h is not provided by libc

Hi Romain, all,

Quoted 15 lines

-----Original Message-----
From: Romain Naour <[email protected]>
Sent: Friday, August 2, 2019 12:09 AM
To: buildroot at buildroot.org
Cc: Damien DUVAL <[email protected]>; Alexandre PAYEN <[email protected]>; Alexey Brodkin <[email protected]>; Romain Naour <[email protected]>; Thomas Petazzoni <[email protected]>; Alexey Brodkin <[email protected]>; Fabrice Fontaine <[email protected]>; Evgeniy Didin <[email protected]>
Subject: [PATCH 4/5] package/python-numpy: disable numpy if fenv.h is not provided by libc

From: Damien DUVAL <[email protected]>

With a c library which does not provide fenv.h, it won't work at runtime:
Crash after an "import numpy" on python.

Quoted 11 lines

Since numpy v1.16.0:
"Alpine Linux (and other musl c library distros) support
We now default to use fenv.h for floating point status error reporting. Previously we had a broken default that sometimes would not report underflow, overflow, and invalid floating point operations. Now we can support non-glibc distrubutions like Alpine Linux as long as they ship fenv.h."
Also there is a lack of define in glibc for ARC architecture and python numpy cannot build.
The removed patches were just fixes for the build for uclibc and glibc for ARC

Well we used to lack fenv.h in ARC's port back in the day but not any longer, check [1] as what we have in arc-2019.03 release so I guess since [2] commit be0aaaaecda5 ("toolchain: bump ARC tools to arc-2019.03 release") numpy should build and work for ARC with glibc as good as for other arches.

package/python-numpy/Config.in config BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS · line 23

Quoted 13 lines
package/python-numpy/Config.in +4 -0
index 6d0b373413..bcd51e6000 100644
@@ -16,6 +16,10 @@ config BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
 config BR2_PACKAGE_PYTHON_NUMPY
 	bool "python-numpy"
 	depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
+	# python-numpy needs fenv.h which is not provided by uclibc
+	depends on BR2_TOOLCHAIN_USES_GLIBC || BR2_TOOLCHAIN_USES_MUSL
+	# python-numpy is not compatible with glibc for ARC architecture.
+	depends on !(BR2_TOOLCHAIN_USES_GLIBC && BR2_arc)
 	select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK

Given my comment above can we try it on ARC?
I.e. what exactly is a test-case so that we may try it and confirm?

[1] https://github.com/foss-for-synopsys-dwc-arc-processors/glibc/blob/arc-2019.03-release/sysdeps/arc/bits/fenv.h
[2] https://git.buildroot.org/buildroot/commit/?id=be0aaaaecda56d16acf99a38319293913fd1e0b5

-Alexey

#13 lore raw

PATCH 4/5 package/python-numpy: disable numpy if fenv.h is not provided by libc

In reply to Romain Naour
Quoted 5 lines · Romain Naour

From: Damien DUVAL <[email protected]>

With a c library which does not provide fenv.h, it won't work at runtime:
Crash after an "import numpy" on python.

Quoted 11 lines

Since numpy v1.16.0:
"Alpine Linux (and other musl c library distros) support
We now default to use fenv.h for floating point status error reporting. Previously we had a broken default that sometimes would not report underflow, overflow, and invalid floating point operations. Now we can support non-glibc distrubutions like Alpine Linux as long as they ship fenv.h."
Also there is a lack of define in glibc for ARC architecture and python numpy cannot build.
The removed patches were just fixes for the build for uclibc and glibc for ARC

Since Alexey indicated that this is not needed any longer, I removed that part
and applied to master, thanks.

Alexey, could you do a runtime test (based on the one submitted by Romain) for
ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm asking you instead of doing it myself :-)

Regards,
Arnout

Quoted 8 lines

Configurations which does not provide fenv.h (like uclibc) won't be able to select the python-numpy package to avoid errors at runtime.

#14 lore raw

PATCH 4/5 package/python-numpy: disable numpy if fenv.h is not provided by libc

Hi Arnout,

Quoted 6 lines

Since Alexey indicated that this is not needed any longer, I removed that part
and applied to master, thanks.

Alexey, could you do a runtime test (based on the one submitted by Romain) for
ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm asking you instead of doing it myself :-)

Any hint on why QEMU is in the question?
I.e. is there any framework or test-suite that require QEMU? What about real boards, proprietary simulators?

-Alexey

#15 lore raw

PATCH 4/5 package/python-numpy: disable numpy if fenv.h is not provided by libc

Quoted 11 lines · Alexey Brodkin

Hi Arnout,

Quoted 6 lines

Since Alexey indicated that this is not needed any longer, I removed that part
and applied to master, thanks.

Alexey, could you do a runtime test (based on the one submitted by Romain) for
ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm asking you instead of doing it myself :-)

Any hint on why QEMU is in the question?
I.e. is there any framework or test-suite that require QEMU?

Yes, the Buildroot runtime tests in support/testing use qemu.

Regards,
Arnout

Quoted 4 lines

What about real boards, proprietary simulators?

-Alexey

#16 lore raw

PATCH 4/5 package/python-numpy: disable numpy if fenv.h is not provided by libc

Hi Arnout,

Quoted 19 lines

-----Original Message-----
From: Arnout Vandecappelle <[email protected]>
Sent: Saturday, August 3, 2019 3:25 PM
To: Alexey Brodkin <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Alexandre PAYEN <[email protected]>; Damien DUVAL <[email protected]>; Evgeniy Didin <[email protected]>; Fabrice Fontaine <[email protected]>; Romain Naour <[email protected]>; buildroot at buildroot.org
Subject: Re: [Buildroot] [PATCH 4/5] package/python-numpy: disable numpy if fenv.h is not provided by libc

Quoted 5 lines · Alexey Brodkin

Hi Arnout,

Quoted 3 lines

Since Alexey indicated that this is not needed any longer, I removed that part
and applied to master, thanks.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Alexey, could you do a runtime test (based on the one submitted by Romain) for
ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm asking you instead of doing it myself :-)

Any hint on why QEMU is in the question?
I.e. is there any framework or test-suite that require QEMU?

Yes, the Buildroot runtime tests in support/testing use qemu.

That's cool, though I haven't tried that yet so may I ask what do I need to run for testing numpy in particular?

-Alexey

P.S. Pls pardon my noob's questions - if there's an answer in form of one-liner that would be super cool, otherwise docs will work.

#17 lore raw

PATCH 4/5 package/python-numpy: disable numpy if fenv.h is not provided by libc

Quoted 25 lines · Alexey Brodkin

Hi Arnout,

Quoted 23 lines

-----Original Message-----
From: Arnout Vandecappelle <[email protected]>
Sent: Saturday, August 3, 2019 3:25 PM
To: Alexey Brodkin <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Alexandre PAYEN <[email protected]>; Damien DUVAL <[email protected]>; Evgeniy Didin <[email protected]>; Fabrice Fontaine <[email protected]>; Romain Naour <[email protected]>; buildroot at buildroot.org
Subject: Re: [Buildroot] [PATCH 4/5] package/python-numpy: disable numpy if fenv.h is not provided by libc

Quoted 9 lines · Alexey Brodkin

Hi Arnout,

Quoted 6 lines

Since Alexey indicated that this is not needed any longer, I removed that part
and applied to master, thanks.

Alexey, could you do a runtime test (based on the one submitted by Romain) for
ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm asking you instead of doing it myself :-)

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Any hint on why QEMU is in the question?
I.e. is there any framework or test-suite that require QEMU?

Yes, the Buildroot runtime tests in support/testing use qemu.

That's cool, though I haven't tried that yet so may I ask what do I need to run for testing numpy in particular?

Specifically for numpy, there is no runtime test yet - Romain submitted a patch
[1] but it needs a new iteration.

But I realise now that in your case the runtime testing infra isn't that useful
since you anyway need to change the test completely to get the right arch and kernel. So yes, by all means, test it on the board :-)

You just need to add this to the config:

BR2_PACKAGE_PYTHON3=y
BR2_PACKAGE_PYTHON_NUMPY=y

And run this in Python:

import numpy as np
list = [1,2,3,4]
arr = np.array(list)
print (arr)

Regards,
Arnout

[1] http://patchwork.ozlabs.org/patch/1140773/

Quoted 6 lines

-Alexey

P.S. Pls pardon my noob's questions - if there's an answer in form of one-liner that would be super cool, otherwise docs will work.

#18 lore raw

PATCH 4/5 package/python-numpy: disable numpy if fenv.h is not provided by libc

Hi Arnout, all,

Quoted 61 lines

-----Original Message-----
From: Arnout Vandecappelle <[email protected]>
Sent: Saturday, August 3, 2019 5:12 PM
To: Alexey Brodkin <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Alexandre PAYEN <[email protected]>; Damien DUVAL <[email protected]>; Evgeniy Didin <[email protected]>; Fabrice Fontaine <[email protected]>; Romain Naour <[email protected]>; buildroot at buildroot.org
Subject: Re: [Buildroot] [PATCH 4/5] package/python-numpy: disable numpy if fenv.h is not provided by libc

Quoted 8 lines · Alexey Brodkin

Hi Arnout,

Quoted 6 lines

-----Original Message-----
From: Arnout Vandecappelle <[email protected]>
Sent: Saturday, August 3, 2019 3:25 PM
To: Alexey Brodkin <[email protected]>
Cc: Thomas Petazzoni <[email protected]>; Alexandre PAYEN <[email protected]>; Damien DUVAL <[email protected]>; Evgeniy Didin

<[email protected]>;

Quoted 3 lines
Quoted 3 lines

Fabrice Fontaine <[email protected]>; Romain Naour <[email protected]>; buildroot at buildroot.org
Subject: Re: [Buildroot] [PATCH 4/5] package/python-numpy: disable numpy if fenv.h is not provided

by

Quoted 14 lines
Quoted 14 lines

libc

Quoted 9 lines · Alexey Brodkin

Hi Arnout,

Quoted 6 lines

Since Alexey indicated that this is not needed any longer, I removed that part
and applied to master, thanks.

Alexey, could you do a runtime test (based on the one submitted by Romain) for
ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm asking you instead of doing it myself :-)

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Any hint on why QEMU is in the question?
I.e. is there any framework or test-suite that require QEMU?

Yes, the Buildroot runtime tests in support/testing use qemu.

That's cool, though I haven't tried that yet so may I ask what do I need to run for testing numpy in particular?

Specifically for numpy, there is no runtime test yet - Romain submitted a patch
[1] but it needs a new iteration.

But I realise now that in your case the runtime testing infra isn't that useful
since you anyway need to change the test completely to get the right arch and kernel. So yes, by all means, test it on the board :-)

You just need to add this to the config:

BR2_PACKAGE_PYTHON3=y
BR2_PACKAGE_PYTHON_NUMPY=y

Quoted 6 lines

And run this in Python:

import numpy as np
list = [1,2,3,4]
arr = np.array(list)
print (arr)

It works perfectly fine:
------------------------------>8---------------------------------

# python
Python 3.7.4 (default, Aug  6 2019, 12:07:44)
[GCC 8.3.1 20190225] on linux
Type "help", "copyright", "credits" or "license" for more information.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

import numpy as np
list = [1,2,3,4]

arr = np.array(list)
print (arr)

[1 2 3 4]

Quoted 1 line
Quoted 1 line
Quoted 1 line
# uname -a
Linux buildroot 5.2.5 #2 PREEMPT Tue Aug 6 12:10:30 MSK 2019 arc GNU/Linux
------------------------------>8---------------------------------

-Alexey

#19 lore raw

PATCH 5/5 support/testing: add python-numpy tests

In reply to Romain Naour

This commit add 3 runtime tests for python-numpy.

The current state of python-numpy is not good since several runtime issue has been discovered while working on new python packages that are based on it.

First, we have a runtime issue while testing python-numpy with clapack option enabled:
(glibc toolchain)

python code:
import numpy as np
list = [1,2,3,4]
arr = np.array(list)
print arr

python Error:
ImportError: /usr/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-arm-linux-gnueabi.so: undefined symbol: cblas_sgemm

Since python-numpy is not covered by the Buildroot testing infrastructure, so we add 3 runtime tests for the following cases:
- Without Lapack support
- With Clapack
- With Lapack

We have to use an external toolchain based on glibc since the uClibc-ng support of python-numpy has been removed due to the following runtime issue:

python Error:
code object from '/usr/lib/python3.7/site-packages/numpy/core/overrides.pyc' /usr/bin/python3.7: symbol 'npy_clear_floatstatus_barrier': can't resolve symbol /usr/bin/python3.7: symbol 'npy_get_floatstatus_barrier': can't resolve symbol

This issue is only present on uClibc-ng system (tested on arm and aarch64).

Cc Asaf Kahlon <[email protected]>
Cc Samuel Martin <[email protected]>
Cc Yegor Yefremov <[email protected]>
---
 .../tests/package/sample_python_numpy.py      |  5 +
 .../tests/package/test_python_numpy.py        | 91 +++++++++++++++++++
 2 files changed, 96 insertions(+)
 create mode 100644 support/testing/tests/package/sample_python_numpy.py
 create mode 100644 support/testing/tests/package/test_python_numpy.py
support/testing/tests/package/sample_python_numpy.py new file +5 -0
new file mode 100644
index 0000000000..8e8d36ea4c
@@ -0,0 +1,5 @@
+# From https://www.tutorialspoint.com/scipy/scipy_basic_functionality.htm
+import numpy as np
+list = [1,2,3,4]
+arr = np.array(list)
+print (arr)
support/testing/tests/package/test_python_numpy.py new file +91 -0

91 lines not on this page · raw message

Signature · 1 line
2.20.1
#20 lore raw

PATCH 5/5 support/testing: add python-numpy tests

Quoted 1 line · Romain Naour

This commit add 3 runtime tests for python-numpy.

As discussed face to face, please make two changes:

- rebase on top of Ricardo's updated kernel so we don't need to rebuild the kernel;

- add a test that we're *really* using lapack, because in one of my attempts I wasn't using lapack and the test still succeeded.

Regards,
Arnout

#21 lore raw

PATCH 1/5 package/python-numpy: fix occasional build and run-time failure with lapack

In reply to Romain Naour
Quoted 9 lines · Romain Naour

From: Alexandre PAYEN <[email protected]>

python-numpy fail at run-time because cblas reference is missing in BR2_PYTHON_NUMPY_SITE_CFG_LIBS.
clapack is the C implementation of lapack so when using clapack python numpy depends on clapack.
cblas is a blas layer which allow to call blas routine from C code. So python numpy need cblas instead of blas to work correclty

Quoted 3 lines

So :
- add cblas reference in BR2_PYTHON_NUMPY_SITE_CFG_LIBS when BR2_PACKAGE_CLAPACK = y

It turns out that this only works because libcblas.so doesn't exist when
installing clapack - it is provided by lapack, but not by clapack. Since libcblas.so doesn't exist, by adding cblas to the libraries line, the libraries get ignored entirely. Putting 'foo' on it works as well.

So this led us to believe that the libraries line maybe isn't needed. And
indeed, just removing it works as well.

There are a lot of corner cases to be checked here, so the runtime test really
is essential...

Thanks you Alex and Giulio for working on this!

Regards,
Arnout

Quoted 30 lines

Fixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/
http://lists.busybox.net/pipermail/buildroot/2019-June/252380.html

Initial patch from Giulio Benetti :
[v1] http://patchwork.ozlabs.org/patch/1100100/
[v2] http://patchwork.ozlabs.org/patch/1100208/

---
 package/python-numpy/python-numpy.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/python-numpy/python-numpy.mk +1 -1
index 5d2fbfc7ad..aa8052dca4 100644
@@ -17,7 +17,7 @@ PYTHON_NUMPY_SETUP_TYPE = setuptools
 
 ifeq ($(BR2_PACKAGE_CLAPACK),y)
 PYTHON_NUMPY_DEPENDENCIES += clapack
-PYTHON_NUMPY_SITE_CFG_LIBS += blas lapack
+PYTHON_NUMPY_SITE_CFG_LIBS += cblas lapack
 else
 PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
 endif