package/python-numpy: fix occasional build failure with lapack
PATCH 1/1 package/python-numpy: fix occasional build failure with lapack
python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in BR2_PYTHON_NUMPY_DEPENDENCIES.
Then build failure is due to missing BR2_PACKAGE_LAPACK_COMPLEX that provides some functions in lapack libraries needed by python-numpy.
So:
- add lapack to BR2_PYTHON_NUMPY_DEPENDENCIES when BR2_PACKAGE_LAPACK = y
- substitute ifeq check "$(BR2_PACKAGE_LAPACK),y" with "$(BR2_PACKAGE_LAPACK_COMPLEX),y" because python-numpy needs COMPLEX functions and BR2_PACKAGE_LAPACK_COMPLEX inherits BR2_PACKAGE_LAPACK
Fixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/
--- package/python-numpy/python-numpy.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
package/python-numpy/python-numpy.mk +2 -2
@@ -15,8 +15,8 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ numpy/core/src/multiarray/dragon4.c PYTHON_NUMPY_SETUP_TYPE = setuptools -ifeq ($(BR2_PACKAGE_CLAPACK),y) -PYTHON_NUMPY_DEPENDENCIES += clapack +ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y) +PYTHON_NUMPY_DEPENDENCIES += clapack lapack PYTHON_NUMPY_SITE_CFG_LIBS += blas lapack else PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
Signature · 1 line
2.17.1
PATCH 1/1 package/python-numpy: fix occasional build failure with lapack
Giulio, All,
On 2019-05-15 16:41 +0200, Giulio Benetti spake thusly:
Quoted 26 lines
python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in
BR2_PYTHON_NUMPY_DEPENDENCIES.
Then build failure is due to missingBR2_PACKAGE_LAPACK_COMPLEXthat provides some functions in lapack libraries needed by python-numpy.So:
- add lapack toBR2_PYTHON_NUMPY_DEPENDENCIESwhenBR2_PACKAGE_LAPACK= y
- substitute ifeq check "$(BR2_PACKAGE_LAPACK),y" with "$(BR2_PACKAGE_LAPACK_COMPLEX),y" because python-numpy needs COMPLEX functions andBR2_PACKAGE_LAPACK_COMPLEXinheritsBR2_PACKAGE_LAPACKFixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/Signed-off-by Giulio Benetti <[email protected]>--- package/python-numpy/python-numpy.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)package/python-numpy/python-numpy.mk +0
-0
package/python-numpy/python-numpy.mk PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ · line 19
Quoted 7 lines
package/python-numpy/python-numpy.mk +2
-2@@ -15,8 +15,8 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ numpy/core/src/multiarray/dragon4.c PYTHON_NUMPY_SETUP_TYPE = setuptools -ifeq ($(BR2_PACKAGE_CLAPACK),y) -PYTHON_NUMPY_DEPENDENCIES += clapack +ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y) +PYTHON_NUMPY_DEPENDENCIES += clapack lapack
This is not correct, because tehre is no relation between lapack and clapack. So, we can have lapack enabled but not clapack. Your code will cause a build failure when clapack is not enabled.
So, you probably want something like:
ifeq ($(BR2_PACKAGE_CLAPACK),y)
PYTHON_NUMPY_DEPENDENCIES_LAPACK += clapack
endif
ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
PYTHON_NUMPY_DEPENDENCIES_LAPACK += lapack
endif
ifneq ($(PYTHON_NUMPY_DEPENDENCIES_LAPACK),)
PYTHON_NUMPY_DEPENDENCIES += $(PYTHON_NUMPY_DEPENDENCIES_LAPACK)
else
PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
endifBut beware of the above, it might not yet be correct: if clapack and lapack (without complex) are both enabled, then you may still en up in the current situation.
So, you may need to refine it even further, with something like:
ifeq ($(BR2_PACKAGE_CLAPACK),y)
PYTHON_NUMPY_DEPENDENCIES += clapack
PYTHON_NUMPY_ENV += BLAS=clapack LAPACK=clapack
else ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
PYTHON_NUMPY_DEPENDENCIES += lapack
PYTHON_NUMPY_ENV += BLAS=lapack LAPACK=lapack
else
PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
endif(check what BLAS= and LAPACK= expect as values.)
Also, is the depenency on clapack really needed? Can python-numpy really use clapack?
Regards,
Yann E. MORIN.
Quoted 6 lines
package/python-numpy/python-numpy.mk +0
-0PYTHON_NUMPY_SITE_CFG_LIBS += blas lapack else PYTHON_NUMPY_ENV += BLAS=None LAPACK=NoneSignature · 1 line
2.17.1
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 package/python-numpy: fix occasional build failure with lapack
Hello Yann,
Il 15/05/2019 17:12, Yann E. MORIN ha scritto:
Quoted 32 lines
Giulio, All,
On 2019-05-15 16:41 +0200, Giulio Benetti spake thusly:
Quoted 29 lines
python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in
BR2_PYTHON_NUMPY_DEPENDENCIES.
Then build failure is due to missingBR2_PACKAGE_LAPACK_COMPLEXthat provides some functions in lapack libraries needed by python-numpy.So:
- add lapack toBR2_PYTHON_NUMPY_DEPENDENCIESwhenBR2_PACKAGE_LAPACK= y
- substitute ifeq check "$(BR2_PACKAGE_LAPACK),y" with "$(BR2_PACKAGE_LAPACK_COMPLEX),y" because python-numpy needs COMPLEX functions andBR2_PACKAGE_LAPACK_COMPLEXinheritsBR2_PACKAGE_LAPACKFixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/Signed-off-by Giulio Benetti <[email protected]>--- package/python-numpy/python-numpy.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)package/python-numpy/python-numpy.mk +0
-0@@ -15,8 +15,8 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ numpy/core/src/multiarray/dragon4.c PYTHON_NUMPY_SETUP_TYPE = setuptools
Quoted 8 lines
package/python-numpy/python-numpy.mk PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ · line 19
Quoted 4 lines
package/python-numpy/python-numpy.mk +2
-2-ifeq ($(BR2_PACKAGE_CLAPACK),y) -PYTHON_NUMPY_DEPENDENCIES += clapack +ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y) +PYTHON_NUMPY_DEPENDENCIES += clapack lapackThis is not correct, because tehre is no relation between lapack and clapack. So, we can have lapack enabled but not clapack. Your code will cause a build failure when clapack is not enabled.
Right, they should be exclusive.
Quoted 14 lines
So, you probably want something like:
ifeq ($(BR2_PACKAGE_CLAPACK),y) PYTHON_NUMPY_DEPENDENCIES_LAPACK += clapack endif ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y) PYTHON_NUMPY_DEPENDENCIES_LAPACK += lapack endif ifneq ($(PYTHON_NUMPY_DEPENDENCIES_LAPACK),) PYTHON_NUMPY_DEPENDENCIES += $(PYTHON_NUMPY_DEPENDENCIES_LAPACK) else PYTHON_NUMPY_ENV += BLAS=None LAPACK=None endif
Quoted 3 lines
But beware of the above, it might not yet be correct: if clapack and lapack (without complex) are both enabled, then you may still en up in the current situation.
Right.
Quoted 12 lines
So, you may need to refine it even further, with something like:
ifeq ($(BR2_PACKAGE_CLAPACK),y) PYTHON_NUMPY_DEPENDENCIES += clapack PYTHON_NUMPY_ENV += BLAS=clapack LAPACK=clapack else ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y) PYTHON_NUMPY_DEPENDENCIES += lapack PYTHON_NUMPY_ENV += BLAS=lapack LAPACK=lapack else PYTHON_NUMPY_ENV += BLAS=None LAPACK=None endif
Quoted 4 lines
(check what BLAS= and LAPACK= expect as values.)
Also, is the depenency on clapack really needed? Can python-numpy really use clapack?
Judging from here:
https://www.numpy.org/devdocs/user/building.html#blas
clapack seems to not be supported.
But AIK clapack should be the same as lapack but written entirely in C instead of Fortran(later translated in C in lapack during build), so it should work with clapack too.
I try to build with that and check.
Thanks for reviewing and proposing alternatives.
Kind regards
Signature · 10 lines
Giulio Benetti CTO MICRONOVA SRL Sede: Via A. Niedda 3 - 35010 Vigonza (PD) Tel. 049/8931563 - Fax 049/8931346 Cod.Fiscale - P.IVA 02663420285 Capitale Sociale ? 26.000 i.v. Iscritta al Reg. Imprese di Padova N. 02663420285 Numero R.E.A. 258642
Quoted 10 lines
Regards,
Yann E. MORIN.Quoted 6 lines
package/python-numpy/python-numpy.mk +0
-0PYTHON_NUMPY_SITE_CFG_LIBS += blas lapack else PYTHON_NUMPY_ENV += BLAS=None LAPACK=NoneSignature · 1 line
2.17.1
PATCH 1/1 package/python-numpy: fix occasional build failure with lapack
Il 15/05/2019 21:58, Giulio Benetti ha scritto:
Quoted 83 lines
Hello Yann,
Il 15/05/2019 17:12, Yann E. MORIN ha scritto:
Quoted 32 lines
Giulio, All,
On 2019-05-15 16:41 +0200, Giulio Benetti spake thusly:
Quoted 29 lines
python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in
BR2_PYTHON_NUMPY_DEPENDENCIES.
Then build failure is due to missingBR2_PACKAGE_LAPACK_COMPLEXthat provides some functions in lapack libraries needed by python-numpy.So:
- add lapack toBR2_PYTHON_NUMPY_DEPENDENCIESwhenBR2_PACKAGE_LAPACK= y
- substitute ifeq check "$(BR2_PACKAGE_LAPACK),y" with "$(BR2_PACKAGE_LAPACK_COMPLEX),y" because python-numpy needs COMPLEX functions andBR2_PACKAGE_LAPACK_COMPLEXinheritsBR2_PACKAGE_LAPACKFixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/Signed-off-by Giulio Benetti <[email protected]>--- package/python-numpy/python-numpy.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)package/python-numpy/python-numpy.mk +0
-0@@ -15,8 +15,8 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ numpy/core/src/multiarray/dragon4.c PYTHON_NUMPY_SETUP_TYPE = setuptoolsQuoted 8 lines
package/python-numpy/python-numpy.mk PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ · line 19
Quoted 4 lines
package/python-numpy/python-numpy.mk +2
-2-ifeq ($(BR2_PACKAGE_CLAPACK),y) -PYTHON_NUMPY_DEPENDENCIES += clapack +ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y) +PYTHON_NUMPY_DEPENDENCIES += clapack lapackThis is not correct, because tehre is no relation between lapack and clapack. So, we can have lapack enabled but not clapack. Your code will cause a build failure when clapack is not enabled.
Right, they should be exclusive.
Quoted 14 lines
So, you probably want something like:
ifeq ($(BR2_PACKAGE_CLAPACK),y) PYTHON_NUMPY_DEPENDENCIES_LAPACK += clapack endif ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y) PYTHON_NUMPY_DEPENDENCIES_LAPACK += lapack endif ifneq ($(PYTHON_NUMPY_DEPENDENCIES_LAPACK),) PYTHON_NUMPY_DEPENDENCIES += $(PYTHON_NUMPY_DEPENDENCIES_LAPACK) else PYTHON_NUMPY_ENV += BLAS=None LAPACK=None endifQuoted 3 lines
But beware of the above, it might not yet be correct: if clapack and lapack (without complex) are both enabled, then you may still en up in the current situation.
Right.
Quoted 12 lines
So, you may need to refine it even further, with something like:
ifeq ($(BR2_PACKAGE_CLAPACK),y) PYTHON_NUMPY_DEPENDENCIES += clapack PYTHON_NUMPY_ENV += BLAS=clapack LAPACK=clapack else ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y) PYTHON_NUMPY_DEPENDENCIES += lapack PYTHON_NUMPY_ENV += BLAS=lapack LAPACK=lapack else PYTHON_NUMPY_ENV += BLAS=None LAPACK=None endifQuoted 4 lines
(check what BLAS= and LAPACK= expect as values.)
Also, is the depenency on clapack really needed? Can python-numpy really use clapack?
Quoted 7 lines
Judging from here:
https://www.numpy.org/devdocs/user/building.html#blas
clapack seems to not be supported.
But AIK clapack should be the same as lapack but written entirely in C instead of Fortran(later translated in C in lapack during build), so it should work with clapack too.
I try to build with that and check.
I've successfully built python-numpy with these accelerators:
- lapack
- clapack(c verion of lapack(fortran))
- openblas
Speaking in IRC with Yann came out the possibility to specify LAPACK order with NPY_LAPACK_ORDER.
Another thing that could be done(after fixing current patch) is to add the choice in Config.in of which accelerator to use between these:
- lapack
- clapack
- openblas
IMHO I would like to impose LAPACK and BLAS instead of setting NPY_LAPACK_ORDER and NPY_BLAS_ORDER, since Buildroot is meant to be kinda "static" after building.
So I would:
- send a V2 patch to fix build failure using lapack only then:
- follow with a second patch adding such choice
What about this?
Kind regards
Signature · 10 lines
Giulio Benetti CTO MICRONOVA SRL Sede: Via A. Niedda 3 - 35010 Vigonza (PD) Tel. 049/8931563 - Fax 049/8931346 Cod.Fiscale - P.IVA 02663420285 Capitale Sociale ? 26.000 i.v. Iscritta al Reg. Imprese di Padova N. 02663420285 Numero R.E.A. 258642
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in BR2_PYTHON_NUMPY_DEPENDENCIES. clapack is present instead, but it's wrong since we're checking $(BR2_PACKAGE_LAPACK) not $(BR2_PACKAGE_CLAPACK). Once clapack is substituted with lapack build failure is due to missing BR2_PACKAGE_LAPACK_COMPLEX that provides some functions in lapack libraries needed by python-numpy.
So:
- substitute clapack with lapack in BR2_PYTHON_NUMPY_DEPENDENCIES when BR2_PACKAGE_LAPACK = y
- substitute ifeq check "$(BR2_PACKAGE_LAPACK),y" with "$(BR2_PACKAGE_LAPACK_COMPLEX),y" because python-numpy needs COMPLEX functions and BR2_PACKAGE_LAPACK_COMPLEX inherits BR2_PACKAGE_LAPACK
Fixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/
--- package/python-numpy/python-numpy.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
package/python-numpy/python-numpy.mk +2 -2
@@ -15,8 +15,8 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ numpy/core/src/multiarray/dragon4.c PYTHON_NUMPY_SETUP_TYPE = setuptools -ifeq ($(BR2_PACKAGE_CLAPACK),y) -PYTHON_NUMPY_DEPENDENCIES += clapack +ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y) +PYTHON_NUMPY_DEPENDENCIES += lapack PYTHON_NUMPY_SITE_CFG_LIBS += blas lapack else PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
Signature · 1 line
2.17.1
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
Hello,
Quoted 5 lines · On Wed, 15 May 2019 23:03:42 +0200 Giulio Benetti…
python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in
BR2_PYTHON_NUMPY_DEPENDENCIES. clapack is present instead, but it's wrong since we're checking $(BR2_PACKAGE_LAPACK) not $(BR2_PACKAGE_CLAPACK).
I don't follow here: in the current code, we're checking BR2_PACKAGE_CLAPACK and adding clapack to PYTHON_NUMPY_DEPENDENCIES, which is consistent.
In order to better understand the problem. Without this patch:
- Does python-numpy builds fine when neither clapack nor lapack are enabled ?
- Does python-numpy builds fine when clapack is enabled, but not lapack ?
- What happens when both clapack and lapack are enabled ?
Also, perhaps we need:
select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK
in the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
Hello Thomas, All,
Il 18/05/2019 22:13, Thomas Petazzoni ha scritto:
Quoted 11 lines
Hello,
Quoted 5 lines · On Wed, 15 May 2019 23:03:42 +0200 Giulio Benetti…
python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in
BR2_PYTHON_NUMPY_DEPENDENCIES. clapack is present instead, but it's wrong since we're checking $(BR2_PACKAGE_LAPACK) not $(BR2_PACKAGE_CLAPACK).
Quoted 3 lines
I don't follow here: in the current code, we're checking
BR2_PACKAGE_CLAPACKand adding clapack toPYTHON_NUMPY_DEPENDENCIES, which is consistent.
You're right, I've done a mistake, I've substituted clapack with lapack. So need to send a v3 with this corrected.
The problem was due to cohexistence of lapack and clapack, but they are both(like openblas) accelerators that can exist once(I mean or lapack, or clapack).
Here I was asking about inserting a menu to choose which accelerator to use:
http://lists.busybox.net/pipermail/buildroot/2019-May/250273.html
Can you take a look and answer with your opinion?
Quoted 4 lines
In order to better understand the problem. Without this patch:
- Does python-numpy builds fine when neither clapack nor lapack are enabled ?
Yes it does(it works without accelerators).
Quoted 2 lines
- Does python-numpy builds fine when clapack is enabled, but not lapack ?
Yes
Quoted 1 line
- What happens when both clapack and lapack are enabled ?
Python-numpy will choose the accelerator according to this list:
https://www.numpy.org/devdocs/user/building.html#accelerated-blas-lapack-libraries
Quoted 6 lines
Also, perhaps we need:
select
BR2_PACKAGE_LAPACK_COMPLEXifBR2_PACKAGE_LAPACKin the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.
If lapack selected yes, but that should come after adding a choice menu IMHO.
Kind regards
Signature · 10 lines
Giulio Benetti CTO MICRONOVA SRL Sede: Via A. Niedda 3 - 35010 Vigonza (PD) Tel. 049/8931563 - Fax 049/8931346 Cod.Fiscale - P.IVA 02663420285 Capitale Sociale ? 26.000 i.v. Iscritta al Reg. Imprese di Padova N. 02663420285 Numero R.E.A. 258642
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
Hello Thomas,
Il 19/05/2019 16:47, Giulio Benetti ha scritto:
Quoted 40 lines
Hello Thomas, All,
Il 18/05/2019 22:13, Thomas Petazzoni ha scritto:
Quoted 11 lines
Hello,
Quoted 5 lines · On Wed, 15 May 2019 23:03:42 +0200 Giulio Benetti…
python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in
BR2_PYTHON_NUMPY_DEPENDENCIES. clapack is present instead, but it's wrong since we're checking $(BR2_PACKAGE_LAPACK) not $(BR2_PACKAGE_CLAPACK).Quoted 3 lines
I don't follow here: in the current code, we're checking
BR2_PACKAGE_CLAPACKand adding clapack toPYTHON_NUMPY_DEPENDENCIES, which is consistent.You're right, I've done a mistake, I've substituted clapack with lapack. So need to send a v3 with this corrected.
The problem was due to cohexistence of lapack and clapack, but they are both(like openblas) accelerators that can exist
once(I mean or lapack, or clapack).
Here I was asking about inserting a menu to choose which accelerator to use:
http://lists.busybox.net/pipermail/buildroot/2019-May/250273.htmlCan you take a look and answer with your opinion?
Quoted 4 lines
In order to better understand the problem. Without this patch:
? - Does python-numpy builds fine when neither clapack nor lapack are ??? enabled ?
Yes it does(it works without accelerators).
Quoted 2 lines
? - Does python-numpy builds fine when clapack is enabled, but not ??? lapack ?
Quoted 6 lines
Yes
Quoted 1 line
? - What happens when both clapack and lapack are enabled ?
Python-numpy will choose the accelerator according to this list:
https://www.numpy.org/devdocs/user/building.html#accelerated-blas-lapack-libraries
This ^^^^^^^^^^^^^ is wrong.
The problem is that both lapack and clapack library names are equal:
liblapack.so
So what happens here is that python-numpy will use clapack headers, then it will link with liblapack.so causing linking error. Because liblapack.so is from lapack(not clapack) due to alphabetical order when building.
Basically clapack is the c implementation of lapack, but in lapack it's been implemented lapacke(a c interface) that makes clapack useless at this point.
I see 2 options:
- remove clapack completely because it's the same as lapacke(embedded in lapack)
- make clapack and lapack exclusive selectable in their Config.in like:
(clapack Config.in)
depends on !BR2_PACKAGE_LAPACK
and:
(lapack Config.in)
depends on !BR2_PACKAGE_CLAPACK
But IMHO we should proceed with dropping clapack.
Quoted 4 lines
Quoted 4 lines
Also, perhaps we need:
????select
BR2_PACKAGE_LAPACK_COMPLEXifBR2_PACKAGE_LAPACK
This ^^^^^^^^^^ instead never built succesfully.
It built successfully only because I was trying to build with:
make python-numpy-dirclean python-numpy
instead of:
make clean all
So I had a mess of LAPACK libraries in my environment(sorry).
Best regards--
Giulio Benetti
CTO
MICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642
Quoted 7 lines
Quoted 2 lines
in the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.
If lapack selected yes, but that should come after adding a choice menu IMHO.
Kind regards
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
Adding Samuel+,
Il 20/05/2019 19:48, Giulio Benetti ha scritto:
Quoted 113 lines
Hello Thomas,
Il 19/05/2019 16:47, Giulio Benetti ha scritto:
Quoted 40 lines
Hello Thomas, All,
Il 18/05/2019 22:13, Thomas Petazzoni ha scritto:
Quoted 11 lines
Hello,
Quoted 5 lines · On Wed, 15 May 2019 23:03:42 +0200 Giulio Benetti…
python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in
BR2_PYTHON_NUMPY_DEPENDENCIES. clapack is present instead, but it's wrong since we're checking $(BR2_PACKAGE_LAPACK) not $(BR2_PACKAGE_CLAPACK).Quoted 3 lines
I don't follow here: in the current code, we're checking
BR2_PACKAGE_CLAPACKand adding clapack toPYTHON_NUMPY_DEPENDENCIES, which is consistent.You're right, I've done a mistake, I've substituted clapack with lapack. So need to send a v3 with this corrected.
The problem was due to cohexistence of lapack and clapack, but they are both(like openblas) accelerators that can exist
once(I mean or lapack, or clapack).
Here I was asking about inserting a menu to choose which accelerator to use:
http://lists.busybox.net/pipermail/buildroot/2019-May/250273.htmlCan you take a look and answer with your opinion?
Quoted 4 lines
In order to better understand the problem. Without this patch:
? - Does python-numpy builds fine when neither clapack nor lapack are ??? enabled ?
Yes it does(it works without accelerators).
Quoted 2 lines
? - Does python-numpy builds fine when clapack is enabled, but not ??? lapack ?
Quoted 6 lines
Yes
Quoted 1 line
? - What happens when both clapack and lapack are enabled ?
Python-numpy will choose the accelerator according to this list:
https://www.numpy.org/devdocs/user/building.html#accelerated-blas-lapack-librariesThis ^^^^^^^^^^^^^ is wrong.
The problem is that both lapack and clapack library names are equal:
liblapack.soSo what happens here is that python-numpy will use clapack headers, then it will link with liblapack.so causing linking error. Because liblapack.so is from lapack(not clapack) due to alphabetical order when building.
Basically clapack is the c implementation of lapack, but in lapack it's been implemented lapacke(a c interface) that makes clapack useless at this point.
I see 2 options:
- remove clapack completely because it's the same as lapacke(embedded in lapack)
- make clapack and lapack exclusive selectable in their Config.in like:
? (clapack Config.in)
? depends on !BR2_PACKAGE_LAPACK
? and:
? (lapack Config.in)
? depends on !BR2_PACKAGE_CLAPACKBut IMHO we should proceed with dropping clapack.
Quoted 4 lines
Quoted 4 lines
Also, perhaps we need:
????select
BR2_PACKAGE_LAPACK_COMPLEXifBR2_PACKAGE_LAPACKThis ^^^^^^^^^^ instead never built succesfully.
It built successfully only because I was trying to build with:
make python-numpy-dirclean python-numpy
instead of:
make clean allSo I had a mess of LAPACK libraries in my environment(sorry).
Best regards--
Giulio Benetti
CTOMICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642Quoted 7 lines
Quoted 2 lines
in the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.
If lapack selected yes, but that should come after adding a choice menu IMHO.
Kind regards
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
Giulio Benetti CTO MICRONOVA SRL Sede: Via A. Niedda 3 - 35010 Vigonza (PD) Tel. 049/8931563 - Fax 049/8931346 Cod.Fiscale - P.IVA 02663420285 Capitale Sociale ? 26.000 i.v. Iscritta al Reg. Imprese di Padova N. 02663420285 Numero R.E.A. 258642
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
Quoted 9 lines · Giulio Benetti
Adding Samuel+,
Il 20/05/2019 19:48, Giulio Benetti ha scritto:
Quoted 6 lines
Hello Thomas,
Il 19/05/2019 16:47, Giulio Benetti ha scritto:
Quoted 3 lines
Hello Thomas, All,
Il 18/05/2019 22:13, Thomas Petazzoni ha scritto:
[snip]
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
? - What happens when both clapack and lapack are enabled ?
Python-numpy will choose the accelerator according to this list:
https://www.numpy.org/devdocs/user/building.html#accelerated-blas-lapack-librariesThis ^^^^^^^^^^^^^ is wrong.
The problem is that both lapack and clapack library names are equal:
liblapack.so
So, you are saying that if you build both lapack and clapack, you get only one
.so file?
Quoted 9 lines
Quoted 9 lines
So what happens here is that python-numpy will use clapack headers, then it will link with liblapack.so causing linking error. Because liblapack.so is from lapack(not clapack) due to alphabetical order when building.
Basically clapack is the c implementation of lapack, but in lapack it's been implemented lapacke(a c interface) that makes clapack useless at this point.
Quoted 9 lines
Quoted 9 lines
I see 2 options:
- remove clapack completely because it's the same as lapacke(embedded in lapack)
- make clapack and lapack exclusive selectable in their Config.in like:
?? (clapack Config.in)
?? depends on !BR2_PACKAGE_LAPACK
?? and:
?? (lapack Config.in)
?? depends on !BR2_PACKAGE_CLAPACK
We indeed have to make the mutually exclusive. There are a number of ways that
we can do that.
A. Add a dependency (not like above because it's a circular dependency).
B. Turn it into a virtual package.
C. Turn it into a choice (in addition to making it a virtual package).
The latter is probably the proper way. Cfr. how it was done for
openssl/libressl and libjpeg/turbo-jpeg.
Quoted 1 line
Quoted 1 line
But IMHO we should proceed with dropping clapack.
Why? it is maintained, and it is a dependency of armadillo...
I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.
We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.
So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available. So in that sense, it would actually be more appropriate to remove lapack...
But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).
Regards,
Arnout
Quoted 36 lines
Quoted 35 lines
Quoted 3 lines
Quoted 3 lines
Also, perhaps we need:
????select
BR2_PACKAGE_LAPACK_COMPLEXifBR2_PACKAGE_LAPACKThis ^^^^^^^^^^ instead never built succesfully.
It built successfully only because I was trying to build with:
make python-numpy-dirclean python-numpy
instead of:
make clean allSo I had a mess of LAPACK libraries in my environment(sorry).
Best regards--
Giulio Benetti
CTOMICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642Quoted 6 lines
Quoted 2 lines
in the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.
If lapack selected yes, but that should come after adding a choice menu IMHO.
Kind regards
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
Hello Arnout,
readding Samuel+,
Il 26/05/2019 11:43, Arnout Vandecappelle ha scritto:
Quoted 15 lines · Giulio Benetti
Quoted 9 lines
Adding Samuel+,
Il 20/05/2019 19:48, Giulio Benetti ha scritto:
Quoted 6 lines
Hello Thomas,
Il 19/05/2019 16:47, Giulio Benetti ha scritto:
Quoted 3 lines
Hello Thomas, All,
Il 18/05/2019 22:13, Thomas Petazzoni ha scritto:
[snip]
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
? - What happens when both clapack and lapack are enabled ?
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Python-numpy will choose the accelerator according to this list:
https://www.numpy.org/devdocs/user/building.html#accelerated-blas-lapack-librariesThis ^^^^^^^^^^^^^ is wrong.
The problem is that both lapack and clapack library names are equal:
liblapack.soSo, you are saying that if you build both lapack and clapack, you get only one
.so file?
Exactly and lapack during install overwrite clapacks's liblapack.so with its liblapack.so
Quoted 23 lines
Quoted 10 lines
Quoted 10 lines
So what happens here is that python-numpy will use clapack headers, then it will link with liblapack.so causing linking error. Because liblapack.so is from lapack(not clapack) due to alphabetical order when building.
Basically clapack is the c implementation of lapack, but in lapack it's been implemented lapacke(a c interface) that makes clapack useless at this point.
Quoted 9 lines
Quoted 9 lines
I see 2 options:
- remove clapack completely because it's the same as lapacke(embedded in lapack)
- make clapack and lapack exclusive selectable in their Config.in like:
?? (clapack Config.in)
?? depends on !BR2_PACKAGE_LAPACK
?? and:
?? (lapack Config.in)
?? depends on !BR2_PACKAGE_CLAPACKWe indeed have to make the mutually exclusive. There are a number of ways that
we can do that.
Quoted 8 lines
A. Add a dependency (not like above because it's a circular dependency).
B. Turn it into a virtual package.
C. Turn it into a choice (in addition to making it a virtual package).
The latter is probably the proper way. Cfr. how it was done for
openssl/libressl and libjpeg/turbo-jpeg.
I agree with using option C. Otherwise there will always be problems like this.
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
But IMHO we should proceed with dropping clapack.
Why? it is maintained, and it is a dependency of armadillo...
I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.
Quoted 5 lines
We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available.
Yes, I thought about it too and this is why it makes more sense option C IMHO.
Quoted 5 lines
So in that sense, it
would actually be more appropriate to remove lapack...But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).
Yes that can be done. But at this point I would make it with option C.
Do I proceed that way?
PS: I always try to add to CC Samuel and everytime it gets removed, is there a reason?
Best regards
Signature · 10 lines
Giulio Benetti CTO MICRONOVA SRL Sede: Via A. Niedda 3 - 35010 Vigonza (PD) Tel. 049/8931563 - Fax 049/8931346 Cod.Fiscale - P.IVA 02663420285 Capitale Sociale ? 26.000 i.v. Iscritta al Reg. Imprese di Padova N. 02663420285 Numero R.E.A. 258642
Quoted 39 lines
Regards,
ArnoutQuoted 36 lines
Quoted 35 lines
Quoted 3 lines
Quoted 3 lines
Also, perhaps we need:
????select
BR2_PACKAGE_LAPACK_COMPLEXifBR2_PACKAGE_LAPACKThis ^^^^^^^^^^ instead never built succesfully.
It built successfully only because I was trying to build with:
make python-numpy-dirclean python-numpy
instead of:
make clean allSo I had a mess of LAPACK libraries in my environment(sorry).
Best regards--
Giulio Benetti
CTOMICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642Quoted 6 lines
Quoted 2 lines
in the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.
If lapack selected yes, but that should come after adding a choice menu IMHO.
Kind regards
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
In reply to Arnout VandecappelleQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Hi,
Quoted 12 lines
Quoted 1 line
Quoted 1 line
But IMHO we should proceed with dropping clapack.
Why? it is maintained, and it is a dependency of armadillo...
I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.
Quoted 6 lines
So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available. So in that sense, it would actually be more appropriate to remove lapack...But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).
Not knowing anything about (c)lapack (or fortran), is there any performance advantage using lapack over clapack? Otherwise just always using clapack seems like the simplest solution.
Signature · 1 line
Bye, Peter Korsgaard
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
Quoted 20 lines · Peter Korsgaard
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Hi,
Quoted 15 lines
Quoted 1 line
Quoted 1 line
But IMHO we should proceed with dropping clapack.
Why? it is maintained, and it is a dependency of armadillo...
I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available. So in that sense, it would actually be more appropriate to remove lapack...
Quoted 6 lines
Quoted 2 lines
But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).Not knowing anything about (c)lapack (or fortran), is there any performance advantage using lapack over clapack? Otherwise just always using clapack seems like the simplest solution.
Sounds OK to me.
Bernd, you're the only one who ever made a real contribution to lapack. Thoughts?
Benjamin, you originally contributed lapack. Any ideas?
Regards,
Arnout
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
Sub options ate my last reply. Re-posting to list.
Quoted 31 lines · Arnout Vandecappelle
Quoted 20 lines · Peter Korsgaard
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Hi,
Quoted 15 lines
Quoted 1 line
Quoted 1 line
But IMHO we should proceed with dropping clapack.
Why? it is maintained, and it is a dependency of armadillo...
I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available. So in that sense, it would actually be more appropriate to remove lapack...Quoted 6 lines
Quoted 2 lines
But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).Not knowing anything about (c)lapack (or fortran), is there any performance advantage using lapack over clapack? Otherwise just always using clapack seems like the simplest solution.
Quoted 5 lines
Sounds OK to me.
Bernd, you're the only one who ever made a real contribution to lapack. Thoughts?
Benjamin, you originally contributed lapack. Any ideas?
I think lapack also might serve as a useful reference design for anyone trying to bring an external Fortran package into their build.
If one is much more difficult to maintain, it makes sense to get rid of that one. I think in this case clapack might be pretty well maintained, but in general it's nice to avoid intermediate sources as they are just another level of indirection where problems can occur.
Don't feel very strongly one way or the other. At this point, both will be in the history so breadcrumbs are there if someone wants to use the package that has support dropped.
Cheers,
Ben
Quoted 3 lines
Regards,
Arnout
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
Hi Giulio, All,
Le 28/05/2019 ? 07:34, Benjamin Kamath a ?crit?:
Quoted 26 lines
Sub options ate my last reply. Re-posting to list.
Quoted 23 lines · Arnout Vandecappelle
Quoted 19 lines · Peter Korsgaard
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Hi,
Quoted 15 lines
Quoted 1 line
Quoted 1 line
But IMHO we should proceed with dropping clapack.
Why? it is maintained, and it is a dependency of armadillo...
I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available. So in that sense, it would actually be more appropriate to remove lapack...
Maybe we should convince Thomas to generate toolchains from toolchain-builder project with gFortran :
http://patchwork.ozlabs.org/patch/1113806/
Quoted 17 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).Not knowing anything about (c)lapack (or fortran), is there any performance advantage using lapack over clapack? Otherwise just always using clapack seems like the simplest solution.
Quoted 5 lines
Sounds OK to me.
Bernd, you're the only one who ever made a real contribution to lapack. Thoughts?
Benjamin, you originally contributed lapack. Any ideas?
I think lapack also might serve as a useful reference design for anyone trying to bring an external Fortran package into their build.
Quoted 8 lines
If one is much more difficult to maintain, it makes sense to get rid of that one. I think in this case clapack might be pretty well maintained, but in general it's nice to avoid intermediate sources as they are just another level of indirection where problems can occur.
Don't feel very strongly one way or the other. At this point, both will be in the history so breadcrumbs are there if someone wants to use the package that has support dropped.
Actually python-numpy doesn't seems to work at all due to a runtime issue:
http://patchwork.ozlabs.org/patch/1114198/
http://patchwork.ozlabs.org/patch/1112759/
(It would be great if python-numpy is runtime tested in gitlab)
Also, using uClibc fail too for another reason.
Sorry, I missed this thread while working on theses patches...
As a side project, I'm testing python-scipy package that require gFortran to build. For now Python is stuck while importing the library for some reason...
Best regards,
Romain
Quoted 12 lines
Cheers,
BenQuoted 3 lines
Regards,
Arnout_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack
In reply to Arnout VandecappelleAm Sun, 26 May 2019 14:29:26 +0200 schrieb Arnout Vandecappelle:
Quoted 2 lines
Bernd, you're the only one who ever made a real contribution to lapack. Thoughts?
Hi Arnout,
while it is true that I sent a version bump for lapack nearly two years ago this was part of a broader approach to bump many outdated packages. I never used lapack and therefore I am unable to contribute to the current discussion, sorry.
Regards, Bernd