Skip to contentSkip to conversation

package/python-numpy: fix occasional build failure with lapack

PATCH New Giulio Benetti 8 participants 16 messages May 15, 2019 – Jul 3, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package/python-numpy: fix occasional build failure with lapack

python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in BR2_PYTHON_NUMPY_DEPENDENCIES.
Then build failure is due to missing BR2_PACKAGE_LAPACK_COMPLEX that provides some functions in lapack libraries needed by python-numpy.

So:
- add lapack to BR2_PYTHON_NUMPY_DEPENDENCIES when BR2_PACKAGE_LAPACK = y
- substitute ifeq check "$(BR2_PACKAGE_LAPACK),y" with "$(BR2_PACKAGE_LAPACK_COMPLEX),y" because python-numpy needs COMPLEX functions and BR2_PACKAGE_LAPACK_COMPLEX inherits BR2_PACKAGE_LAPACK

Fixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/

---
 package/python-numpy/python-numpy.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
package/python-numpy/python-numpy.mk +2 -2
index 28dccf8be5..a61246615a 100644
@@ -15,8 +15,8 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \
 			numpy/core/src/multiarray/dragon4.c
 PYTHON_NUMPY_SETUP_TYPE = setuptools
 
-ifeq ($(BR2_PACKAGE_CLAPACK),y)
-PYTHON_NUMPY_DEPENDENCIES += clapack
+ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
+PYTHON_NUMPY_DEPENDENCIES += clapack lapack
 PYTHON_NUMPY_SITE_CFG_LIBS += blas lapack
 else
 PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/1 package/python-numpy: fix occasional build failure with lapack

Giulio, All,

On 2019-05-15 16:41 +0200, Giulio Benetti spake thusly:

Quoted 26 lines

python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in BR2_PYTHON_NUMPY_DEPENDENCIES.
Then build failure is due to missing BR2_PACKAGE_LAPACK_COMPLEX that provides some functions in lapack libraries needed by python-numpy.

So:
- add lapack to BR2_PYTHON_NUMPY_DEPENDENCIES when BR2_PACKAGE_LAPACK = y
- substitute ifeq check "$(BR2_PACKAGE_LAPACK),y" with "$(BR2_PACKAGE_LAPACK_COMPLEX),y" because python-numpy needs COMPLEX functions and BR2_PACKAGE_LAPACK_COMPLEX inherits BR2_PACKAGE_LAPACK

Fixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/

---
 package/python-numpy/python-numpy.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
package/python-numpy/python-numpy.mk +0 -0
index 28dccf8be5..a61246615a 100644

package/python-numpy/python-numpy.mk PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ · line 19

Quoted 7 lines
package/python-numpy/python-numpy.mk +2 -2
@@ -15,8 +15,8 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \
 			numpy/core/src/multiarray/dragon4.c
 PYTHON_NUMPY_SETUP_TYPE = setuptools
 
-ifeq ($(BR2_PACKAGE_CLAPACK),y)
-PYTHON_NUMPY_DEPENDENCIES += clapack
+ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
+PYTHON_NUMPY_DEPENDENCIES += clapack lapack

This is not correct, because tehre is no relation between lapack and clapack. So, we can have lapack enabled but not clapack. Your code will cause a build failure when clapack is not enabled.

So, you probably want something like:

    ifeq ($(BR2_PACKAGE_CLAPACK),y)
    PYTHON_NUMPY_DEPENDENCIES_LAPACK += clapack
    endif
    ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
    PYTHON_NUMPY_DEPENDENCIES_LAPACK += lapack
    endif
    ifneq ($(PYTHON_NUMPY_DEPENDENCIES_LAPACK),)
    PYTHON_NUMPY_DEPENDENCIES += $(PYTHON_NUMPY_DEPENDENCIES_LAPACK)
    else
    PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
    endif

But beware of the above, it might not yet be correct: if clapack and lapack (without complex) are both enabled, then you may still en up in the current situation.

So, you may need to refine it even further, with something like:

    ifeq ($(BR2_PACKAGE_CLAPACK),y)
    PYTHON_NUMPY_DEPENDENCIES += clapack
    PYTHON_NUMPY_ENV += BLAS=clapack LAPACK=clapack
    else ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
    PYTHON_NUMPY_DEPENDENCIES += lapack
    PYTHON_NUMPY_ENV += BLAS=lapack LAPACK=lapack
    else
    PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
    endif

(check what BLAS= and LAPACK= expect as values.)

Also, is the depenency on clapack really needed? Can python-numpy really use clapack?

Regards,
Yann E. MORIN.

Quoted 6 lines
package/python-numpy/python-numpy.mk +0 -0
 PYTHON_NUMPY_SITE_CFG_LIBS += blas lapack
 else
 PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
Signature · 1 line
2.17.1
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#2 lore raw

PATCH 1/1 package/python-numpy: fix occasional build failure with lapack

Hello Yann,

Il 15/05/2019 17:12, Yann E. MORIN ha scritto:

Quoted 32 lines

Giulio, All,

On 2019-05-15 16:41 +0200, Giulio Benetti spake thusly:

Quoted 29 lines

python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in BR2_PYTHON_NUMPY_DEPENDENCIES.
Then build failure is due to missing BR2_PACKAGE_LAPACK_COMPLEX that provides some functions in lapack libraries needed by python-numpy.

So:
- add lapack to BR2_PYTHON_NUMPY_DEPENDENCIES when BR2_PACKAGE_LAPACK = y
- substitute ifeq check "$(BR2_PACKAGE_LAPACK),y" with "$(BR2_PACKAGE_LAPACK_COMPLEX),y" because python-numpy needs COMPLEX functions and BR2_PACKAGE_LAPACK_COMPLEX inherits BR2_PACKAGE_LAPACK

Fixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/

---
  package/python-numpy/python-numpy.mk | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
package/python-numpy/python-numpy.mk +0 -0
index 28dccf8be5..a61246615a 100644
@@ -15,8 +15,8 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \
  			numpy/core/src/multiarray/dragon4.c
  PYTHON_NUMPY_SETUP_TYPE = setuptools
 
Quoted 8 lines

package/python-numpy/python-numpy.mk PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ · line 19

Quoted 4 lines
package/python-numpy/python-numpy.mk +2 -2
-ifeq ($(BR2_PACKAGE_CLAPACK),y)
-PYTHON_NUMPY_DEPENDENCIES += clapack
+ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
+PYTHON_NUMPY_DEPENDENCIES += clapack lapack

This is not correct, because tehre is no relation between lapack and clapack. So, we can have lapack enabled but not clapack. Your code will cause a build failure when clapack is not enabled.

Right, they should be exclusive.

Quoted 14 lines

So, you probably want something like:

     ifeq ($(BR2_PACKAGE_CLAPACK),y)
     PYTHON_NUMPY_DEPENDENCIES_LAPACK += clapack
     endif
     ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
     PYTHON_NUMPY_DEPENDENCIES_LAPACK += lapack
     endif
     ifneq ($(PYTHON_NUMPY_DEPENDENCIES_LAPACK),)
     PYTHON_NUMPY_DEPENDENCIES += $(PYTHON_NUMPY_DEPENDENCIES_LAPACK)
     else
     PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
     endif
Quoted 3 lines

But beware of the above, it might not yet be correct: if clapack and lapack (without complex) are both enabled, then you may still en up in the current situation.

Right.

Quoted 12 lines

So, you may need to refine it even further, with something like:

     ifeq ($(BR2_PACKAGE_CLAPACK),y)
     PYTHON_NUMPY_DEPENDENCIES += clapack
     PYTHON_NUMPY_ENV += BLAS=clapack LAPACK=clapack
     else ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
     PYTHON_NUMPY_DEPENDENCIES += lapack
     PYTHON_NUMPY_ENV += BLAS=lapack LAPACK=lapack
     else
     PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
     endif
Quoted 4 lines

(check what BLAS= and LAPACK= expect as values.)

Also, is the depenency on clapack really needed? Can python-numpy really use clapack?

Judging from here:
https://www.numpy.org/devdocs/user/building.html#blas
clapack seems to not be supported.
But AIK clapack should be the same as lapack but written entirely in C instead of Fortran(later translated in C in lapack during build), so it should work with clapack too.
I try to build with that and check.

Thanks for reviewing and proposing alternatives.

Kind regards

Signature · 10 lines
Giulio Benetti
CTO

MICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642
Quoted 10 lines

Regards,
Yann E. MORIN.

Quoted 6 lines
package/python-numpy/python-numpy.mk +0 -0
  PYTHON_NUMPY_SITE_CFG_LIBS += blas lapack
  else
  PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
Signature · 1 line
2.17.1
#3 lore raw

PATCH 1/1 package/python-numpy: fix occasional build failure with lapack

Il 15/05/2019 21:58, Giulio Benetti ha scritto:

Quoted 83 lines

Hello Yann,

Il 15/05/2019 17:12, Yann E. MORIN ha scritto:

Quoted 32 lines

Giulio, All,

On 2019-05-15 16:41 +0200, Giulio Benetti spake thusly:

Quoted 29 lines

python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in BR2_PYTHON_NUMPY_DEPENDENCIES.
Then build failure is due to missing BR2_PACKAGE_LAPACK_COMPLEX that provides some functions in lapack libraries needed by python-numpy.

So:
- add lapack to BR2_PYTHON_NUMPY_DEPENDENCIES when BR2_PACKAGE_LAPACK = y
- substitute ifeq check "$(BR2_PACKAGE_LAPACK),y" with "$(BR2_PACKAGE_LAPACK_COMPLEX),y" because python-numpy needs COMPLEX functions and BR2_PACKAGE_LAPACK_COMPLEX inherits BR2_PACKAGE_LAPACK

Fixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/

---
   package/python-numpy/python-numpy.mk | 4 ++--
   1 file changed, 2 insertions(+), 2 deletions(-)
package/python-numpy/python-numpy.mk +0 -0
index 28dccf8be5..a61246615a 100644
@@ -15,8 +15,8 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \
   			numpy/core/src/multiarray/dragon4.c
   PYTHON_NUMPY_SETUP_TYPE = setuptools
 
Quoted 8 lines

package/python-numpy/python-numpy.mk PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ · line 19

Quoted 4 lines
package/python-numpy/python-numpy.mk +2 -2
-ifeq ($(BR2_PACKAGE_CLAPACK),y)
-PYTHON_NUMPY_DEPENDENCIES += clapack
+ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
+PYTHON_NUMPY_DEPENDENCIES += clapack lapack

This is not correct, because tehre is no relation between lapack and clapack. So, we can have lapack enabled but not clapack. Your code will cause a build failure when clapack is not enabled.

Right, they should be exclusive.

Quoted 14 lines

So, you probably want something like:

      ifeq ($(BR2_PACKAGE_CLAPACK),y)
      PYTHON_NUMPY_DEPENDENCIES_LAPACK += clapack
      endif
      ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
      PYTHON_NUMPY_DEPENDENCIES_LAPACK += lapack
      endif
      ifneq ($(PYTHON_NUMPY_DEPENDENCIES_LAPACK),)
      PYTHON_NUMPY_DEPENDENCIES += $(PYTHON_NUMPY_DEPENDENCIES_LAPACK)
      else
      PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
      endif
Quoted 3 lines

But beware of the above, it might not yet be correct: if clapack and lapack (without complex) are both enabled, then you may still en up in the current situation.

Right.

Quoted 12 lines

So, you may need to refine it even further, with something like:

      ifeq ($(BR2_PACKAGE_CLAPACK),y)
      PYTHON_NUMPY_DEPENDENCIES += clapack
      PYTHON_NUMPY_ENV += BLAS=clapack LAPACK=clapack
      else ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
      PYTHON_NUMPY_DEPENDENCIES += lapack
      PYTHON_NUMPY_ENV += BLAS=lapack LAPACK=lapack
      else
      PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
      endif
Quoted 4 lines

(check what BLAS= and LAPACK= expect as values.)

Also, is the depenency on clapack really needed? Can python-numpy really use clapack?

Quoted 7 lines

Judging from here:
https://www.numpy.org/devdocs/user/building.html#blas
clapack seems to not be supported.
But AIK clapack should be the same as lapack but written entirely in C instead of Fortran(later translated in C in lapack during build), so it should work with clapack too.
I try to build with that and check.

I've successfully built python-numpy with these accelerators:
- lapack
- clapack(c verion of lapack(fortran))
- openblas

Speaking in IRC with Yann came out the possibility to specify LAPACK order with NPY_LAPACK_ORDER.

Another thing that could be done(after fixing current patch) is to add the choice in Config.in of which accelerator to use between these:
- lapack
- clapack
- openblas

IMHO I would like to impose LAPACK and BLAS instead of setting NPY_LAPACK_ORDER and NPY_BLAS_ORDER, since Buildroot is meant to be kinda "static" after building.

So I would:
- send a V2 patch to fix build failure using lapack only then:
- follow with a second patch adding such choice

What about this?

Kind regards

Signature · 10 lines
Giulio Benetti
CTO

MICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642
#4 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in BR2_PYTHON_NUMPY_DEPENDENCIES. clapack is present instead, but it's wrong since we're checking $(BR2_PACKAGE_LAPACK) not $(BR2_PACKAGE_CLAPACK). Once clapack is substituted with lapack build failure is due to missing BR2_PACKAGE_LAPACK_COMPLEX that provides some functions in lapack libraries needed by python-numpy.

So:
- substitute clapack with lapack in BR2_PYTHON_NUMPY_DEPENDENCIES when BR2_PACKAGE_LAPACK = y
- substitute ifeq check "$(BR2_PACKAGE_LAPACK),y" with "$(BR2_PACKAGE_LAPACK_COMPLEX),y" because python-numpy needs COMPLEX functions and BR2_PACKAGE_LAPACK_COMPLEX inherits BR2_PACKAGE_LAPACK

Fixes:
http://autobuild.buildroot.net/results/50f/50f7f09a9f830cd7b94f8fc83c09fc3d39297d3d/

---
 package/python-numpy/python-numpy.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
package/python-numpy/python-numpy.mk +2 -2
index 28dccf8be5..af210041a1 100644
@@ -15,8 +15,8 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \
 			numpy/core/src/multiarray/dragon4.c
 PYTHON_NUMPY_SETUP_TYPE = setuptools
 
-ifeq ($(BR2_PACKAGE_CLAPACK),y)
-PYTHON_NUMPY_DEPENDENCIES += clapack
+ifeq ($(BR2_PACKAGE_LAPACK_COMPLEX),y)
+PYTHON_NUMPY_DEPENDENCIES += lapack
 PYTHON_NUMPY_SITE_CFG_LIBS += blas lapack
 else
 PYTHON_NUMPY_ENV += BLAS=None LAPACK=None
Signature · 1 line
2.17.1
#5 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

Hello,

Quoted 5 lines · On Wed, 15 May 2019 23:03:42 +0200 Giulio Benetti…

python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in BR2_PYTHON_NUMPY_DEPENDENCIES. clapack is present instead, but it's wrong since we're checking $(BR2_PACKAGE_LAPACK) not $(BR2_PACKAGE_CLAPACK).

I don't follow here: in the current code, we're checking BR2_PACKAGE_CLAPACK and adding clapack to PYTHON_NUMPY_DEPENDENCIES, which is consistent.

In order to better understand the problem. Without this patch:

- Does python-numpy builds fine when neither clapack nor lapack are enabled ?

- Does python-numpy builds fine when clapack is enabled, but not lapack ?

- What happens when both clapack and lapack are enabled ?

Also, perhaps we need:

select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK

in the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#6 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

Hello Thomas, All,

Il 18/05/2019 22:13, Thomas Petazzoni ha scritto:

Quoted 11 lines

Hello,

Quoted 5 lines · On Wed, 15 May 2019 23:03:42 +0200 Giulio Benetti…

python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in BR2_PYTHON_NUMPY_DEPENDENCIES. clapack is present instead, but it's wrong since we're checking $(BR2_PACKAGE_LAPACK) not $(BR2_PACKAGE_CLAPACK).

Quoted 3 lines

I don't follow here: in the current code, we're checking BR2_PACKAGE_CLAPACK and adding clapack to PYTHON_NUMPY_DEPENDENCIES, which is consistent.

You're right, I've done a mistake, I've substituted clapack with lapack. So need to send a v3 with this corrected.

The problem was due to cohexistence of lapack and clapack, but they are both(like openblas) accelerators that can exist once(I mean or lapack, or clapack).
Here I was asking about inserting a menu to choose which accelerator to use:
http://lists.busybox.net/pipermail/buildroot/2019-May/250273.html

Can you take a look and answer with your opinion?

Quoted 4 lines

In order to better understand the problem. Without this patch:

- Does python-numpy builds fine when neither clapack nor lapack are enabled ?

Yes it does(it works without accelerators).

Quoted 2 lines

- Does python-numpy builds fine when clapack is enabled, but not lapack ?

Yes

Quoted 1 line

- What happens when both clapack and lapack are enabled ?

Python-numpy will choose the accelerator according to this list:
https://www.numpy.org/devdocs/user/building.html#accelerated-blas-lapack-libraries

Quoted 6 lines

Also, perhaps we need:

select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK

in the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.

If lapack selected yes, but that should come after adding a choice menu IMHO.

Kind regards

Signature · 10 lines
Giulio Benetti
CTO

MICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642
#7 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

Hello Thomas,

Il 19/05/2019 16:47, Giulio Benetti ha scritto:

Quoted 40 lines

Hello Thomas, All,

Il 18/05/2019 22:13, Thomas Petazzoni ha scritto:

Quoted 11 lines

Hello,

Quoted 5 lines · On Wed, 15 May 2019 23:03:42 +0200 Giulio Benetti…

python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in BR2_PYTHON_NUMPY_DEPENDENCIES. clapack is present instead, but it's wrong since we're checking $(BR2_PACKAGE_LAPACK) not $(BR2_PACKAGE_CLAPACK).

Quoted 3 lines

I don't follow here: in the current code, we're checking BR2_PACKAGE_CLAPACK and adding clapack to PYTHON_NUMPY_DEPENDENCIES, which is consistent.

You're right, I've done a mistake, I've substituted clapack with lapack. So need to send a v3 with this corrected.

The problem was due to cohexistence of lapack and clapack, but they are both(like openblas) accelerators that can exist once(I mean or lapack, or clapack).
Here I was asking about inserting a menu to choose which accelerator to use:
http://lists.busybox.net/pipermail/buildroot/2019-May/250273.html

Can you take a look and answer with your opinion?

Quoted 4 lines

In order to better understand the problem. Without this patch:

? - Does python-numpy builds fine when neither clapack nor lapack are ??? enabled ?

Yes it does(it works without accelerators).

Quoted 2 lines

? - Does python-numpy builds fine when clapack is enabled, but not ??? lapack ?

Quoted 6 lines

Yes

Quoted 1 line

? - What happens when both clapack and lapack are enabled ?

Python-numpy will choose the accelerator according to this list:
https://www.numpy.org/devdocs/user/building.html#accelerated-blas-lapack-libraries

This ^^^^^^^^^^^^^ is wrong.
The problem is that both lapack and clapack library names are equal:
liblapack.so

So what happens here is that python-numpy will use clapack headers, then it will link with liblapack.so causing linking error. Because liblapack.so is from lapack(not clapack) due to alphabetical order when building.

Basically clapack is the c implementation of lapack, but in lapack it's been implemented lapacke(a c interface) that makes clapack useless at this point.

I see 2 options:
- remove clapack completely because it's the same as lapacke(embedded in lapack)
- make clapack and lapack exclusive selectable in their Config.in like:
(clapack Config.in)
depends on !BR2_PACKAGE_LAPACK
and:
(lapack Config.in)
depends on !BR2_PACKAGE_CLAPACK

But IMHO we should proceed with dropping clapack.

Quoted 4 lines
Quoted 4 lines

Also, perhaps we need:

????select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK

This ^^^^^^^^^^ instead never built succesfully.
It built successfully only because I was trying to build with:
make python-numpy-dirclean python-numpy
instead of:
make clean all

So I had a mess of LAPACK libraries in my environment(sorry).

Best regards--
Giulio Benetti
CTO

MICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642

Quoted 7 lines
Quoted 2 lines

in the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.

If lapack selected yes, but that should come after adding a choice menu IMHO.

Kind regards

#8 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

Adding Samuel+,

Il 20/05/2019 19:48, Giulio Benetti ha scritto:

Quoted 113 lines

Hello Thomas,

Il 19/05/2019 16:47, Giulio Benetti ha scritto:

Quoted 40 lines

Hello Thomas, All,

Il 18/05/2019 22:13, Thomas Petazzoni ha scritto:

Quoted 11 lines

Hello,

Quoted 5 lines · On Wed, 15 May 2019 23:03:42 +0200 Giulio Benetti…

python-numpy build fails only if lapack is built before python-numpy itself, and this doesn't always happen because lapack dependency is missing in BR2_PYTHON_NUMPY_DEPENDENCIES. clapack is present instead, but it's wrong since we're checking $(BR2_PACKAGE_LAPACK) not $(BR2_PACKAGE_CLAPACK).

Quoted 3 lines

I don't follow here: in the current code, we're checking BR2_PACKAGE_CLAPACK and adding clapack to PYTHON_NUMPY_DEPENDENCIES, which is consistent.

You're right, I've done a mistake, I've substituted clapack with lapack. So need to send a v3 with this corrected.

The problem was due to cohexistence of lapack and clapack, but they are both(like openblas) accelerators that can exist once(I mean or lapack, or clapack).
Here I was asking about inserting a menu to choose which accelerator to use:
http://lists.busybox.net/pipermail/buildroot/2019-May/250273.html

Can you take a look and answer with your opinion?

Quoted 4 lines

In order to better understand the problem. Without this patch:

? - Does python-numpy builds fine when neither clapack nor lapack are ??? enabled ?

Yes it does(it works without accelerators).

Quoted 2 lines

? - Does python-numpy builds fine when clapack is enabled, but not ??? lapack ?

Quoted 6 lines

Yes

Quoted 1 line

? - What happens when both clapack and lapack are enabled ?

Python-numpy will choose the accelerator according to this list:
https://www.numpy.org/devdocs/user/building.html#accelerated-blas-lapack-libraries

This ^^^^^^^^^^^^^ is wrong.
The problem is that both lapack and clapack library names are equal:
liblapack.so

So what happens here is that python-numpy will use clapack headers, then it will link with liblapack.so causing linking error. Because liblapack.so is from lapack(not clapack) due to alphabetical order when building.

Basically clapack is the c implementation of lapack, but in lapack it's been implemented lapacke(a c interface) that makes clapack useless at this point.

I see 2 options:
- remove clapack completely because it's the same as lapacke(embedded in lapack)
- make clapack and lapack exclusive selectable in their Config.in like:
? (clapack Config.in)
? depends on !BR2_PACKAGE_LAPACK
? and:
? (lapack Config.in)
? depends on !BR2_PACKAGE_CLAPACK

But IMHO we should proceed with dropping clapack.

Quoted 4 lines
Quoted 4 lines

Also, perhaps we need:

????select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK

This ^^^^^^^^^^ instead never built succesfully.
It built successfully only because I was trying to build with:
make python-numpy-dirclean python-numpy
instead of:
make clean all

So I had a mess of LAPACK libraries in my environment(sorry).

Best regards--
Giulio Benetti
CTO

MICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642

Quoted 7 lines
Quoted 2 lines

in the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.

If lapack selected yes, but that should come after adding a choice menu IMHO.

Kind regards

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Signature · 10 lines
Giulio Benetti
CTO

MICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642
#9 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

Quoted 9 lines · Giulio Benetti

Adding Samuel+,

Il 20/05/2019 19:48, Giulio Benetti ha scritto:

Quoted 6 lines

Hello Thomas,

Il 19/05/2019 16:47, Giulio Benetti ha scritto:

Quoted 3 lines

Hello Thomas, All,

Il 18/05/2019 22:13, Thomas Petazzoni ha scritto:

[snip]

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

? - What happens when both clapack and lapack are enabled ?

Python-numpy will choose the accelerator according to this list:
https://www.numpy.org/devdocs/user/building.html#accelerated-blas-lapack-libraries

This ^^^^^^^^^^^^^ is wrong.
The problem is that both lapack and clapack library names are equal:
liblapack.so

So, you are saying that if you build both lapack and clapack, you get only one
.so file?

Quoted 9 lines
Quoted 9 lines

So what happens here is that python-numpy will use clapack headers, then it will link with liblapack.so causing linking error. Because liblapack.so is from lapack(not clapack) due to alphabetical order when building.

Basically clapack is the c implementation of lapack, but in lapack it's been implemented lapacke(a c interface) that makes clapack useless at this point.

Quoted 9 lines
Quoted 9 lines

I see 2 options:
- remove clapack completely because it's the same as lapacke(embedded in lapack)
- make clapack and lapack exclusive selectable in their Config.in like:
?? (clapack Config.in)
?? depends on !BR2_PACKAGE_LAPACK
?? and:
?? (lapack Config.in)
?? depends on !BR2_PACKAGE_CLAPACK

We indeed have to make the mutually exclusive. There are a number of ways that
we can do that.

A. Add a dependency (not like above because it's a circular dependency).

B. Turn it into a virtual package.

C. Turn it into a choice (in addition to making it a virtual package).

The latter is probably the proper way. Cfr. how it was done for
openssl/libressl and libjpeg/turbo-jpeg.

Quoted 1 line
Quoted 1 line

But IMHO we should proceed with dropping clapack.

Why? it is maintained, and it is a dependency of armadillo...

I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.

We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.

So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available. So in that sense, it would actually be more appropriate to remove lapack...

But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).

Regards,
Arnout

Quoted 36 lines
Quoted 35 lines
Quoted 3 lines
Quoted 3 lines

Also, perhaps we need:

????select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK

This ^^^^^^^^^^ instead never built succesfully.
It built successfully only because I was trying to build with:
make python-numpy-dirclean python-numpy
instead of:
make clean all

So I had a mess of LAPACK libraries in my environment(sorry).

Best regards--
Giulio Benetti
CTO

MICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642

Quoted 6 lines
Quoted 2 lines

in the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.

If lapack selected yes, but that should come after adding a choice menu IMHO.

Kind regards

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#10 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

Hello Arnout,

readding Samuel+,

Il 26/05/2019 11:43, Arnout Vandecappelle ha scritto:

Quoted 15 lines · Giulio Benetti
Quoted 9 lines

Adding Samuel+,

Il 20/05/2019 19:48, Giulio Benetti ha scritto:

Quoted 6 lines

Hello Thomas,

Il 19/05/2019 16:47, Giulio Benetti ha scritto:

Quoted 3 lines

Hello Thomas, All,

Il 18/05/2019 22:13, Thomas Petazzoni ha scritto:

[snip]

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

? - What happens when both clapack and lapack are enabled ?

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Python-numpy will choose the accelerator according to this list:
https://www.numpy.org/devdocs/user/building.html#accelerated-blas-lapack-libraries

This ^^^^^^^^^^^^^ is wrong.
The problem is that both lapack and clapack library names are equal:
liblapack.so

So, you are saying that if you build both lapack and clapack, you get only one
.so file?

Exactly and lapack during install overwrite clapacks's liblapack.so with its liblapack.so

Quoted 23 lines
Quoted 10 lines
Quoted 10 lines

So what happens here is that python-numpy will use clapack headers, then it will link with liblapack.so causing linking error. Because liblapack.so is from lapack(not clapack) due to alphabetical order when building.

Basically clapack is the c implementation of lapack, but in lapack it's been implemented lapacke(a c interface) that makes clapack useless at this point.

Quoted 9 lines
Quoted 9 lines

I see 2 options:
- remove clapack completely because it's the same as lapacke(embedded in lapack)
- make clapack and lapack exclusive selectable in their Config.in like:
?? (clapack Config.in)
?? depends on !BR2_PACKAGE_LAPACK
?? and:
?? (lapack Config.in)
?? depends on !BR2_PACKAGE_CLAPACK

We indeed have to make the mutually exclusive. There are a number of ways that
we can do that.

Quoted 8 lines

A. Add a dependency (not like above because it's a circular dependency).

B. Turn it into a virtual package.

C. Turn it into a choice (in addition to making it a virtual package).

The latter is probably the proper way. Cfr. how it was done for
openssl/libressl and libjpeg/turbo-jpeg.

I agree with using option C. Otherwise there will always be problems like this.

Quoted 10 lines
Quoted 2 lines
Quoted 2 lines

But IMHO we should proceed with dropping clapack.

Why? it is maintained, and it is a dependency of armadillo...

I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.

Quoted 5 lines

We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.

So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available.

Yes, I thought about it too and this is why it makes more sense option C IMHO.

Quoted 5 lines

So in that sense, it
would actually be more appropriate to remove lapack...

But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).

Yes that can be done. But at this point I would make it with option C.

Do I proceed that way?

PS: I always try to add to CC Samuel and everytime it gets removed, is there a reason?

Best regards

Signature · 10 lines
Giulio Benetti
CTO

MICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642
Quoted 39 lines

Regards,
Arnout

Quoted 36 lines
Quoted 35 lines
Quoted 3 lines
Quoted 3 lines

Also, perhaps we need:

????select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK

This ^^^^^^^^^^ instead never built succesfully.
It built successfully only because I was trying to build with:
make python-numpy-dirclean python-numpy
instead of:
make clean all

So I had a mess of LAPACK libraries in my environment(sorry).

Best regards--
Giulio Benetti
CTO

MICRONOVA SRL
Sede: Via A. Niedda 3 - 35010 Vigonza (PD)
Tel. 049/8931563 - Fax 049/8931346
Cod.Fiscale - P.IVA 02663420285
Capitale Sociale ? 26.000 i.v.
Iscritta al Reg. Imprese di Padova N. 02663420285
Numero R.E.A. 258642

Quoted 6 lines
Quoted 2 lines

in the Config.in file, so that it's a bit easier for users: they don't have to know they need that specific sub-option of lapack.

If lapack selected yes, but that should come after adding a choice menu IMHO.

Kind regards

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#11 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

In reply to Arnout Vandecappelle
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arnout" == Arnout Vandecappelle <[email protected]> writes:

Hi,

Quoted 12 lines
Quoted 1 line
Quoted 1 line

But IMHO we should proceed with dropping clapack.

Why? it is maintained, and it is a dependency of armadillo...

I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.

We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.

Quoted 6 lines

So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available. So in that sense, it would actually be more appropriate to remove lapack...

But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).

Not knowing anything about (c)lapack (or fortran), is there any performance advantage using lapack over clapack? Otherwise just always using clapack seems like the simplest solution.

Signature · 1 line
Bye, Peter Korsgaard
#12 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

Quoted 20 lines · Peter Korsgaard
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arnout" == Arnout Vandecappelle <[email protected]> writes:

Hi,

Quoted 15 lines
Quoted 1 line
Quoted 1 line

But IMHO we should proceed with dropping clapack.

Why? it is maintained, and it is a dependency of armadillo...

I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.

We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.

So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available. So in that sense, it would actually be more appropriate to remove lapack...

Quoted 6 lines
Quoted 2 lines

But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).

Not knowing anything about (c)lapack (or fortran), is there any performance advantage using lapack over clapack? Otherwise just always using clapack seems like the simplest solution.

Sounds OK to me.

Bernd, you're the only one who ever made a real contribution to lapack. Thoughts?

Benjamin, you originally contributed lapack. Any ideas?

Regards,
Arnout

#13 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

Sub options ate my last reply. Re-posting to list.

Quoted 31 lines · Arnout Vandecappelle
Quoted 20 lines · Peter Korsgaard
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arnout" == Arnout Vandecappelle <[email protected]> writes:

Hi,

Quoted 15 lines
Quoted 1 line
Quoted 1 line

But IMHO we should proceed with dropping clapack.

Why? it is maintained, and it is a dependency of armadillo...

I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.

We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.

So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available. So in that sense, it would actually be more appropriate to remove lapack...

Quoted 6 lines
Quoted 2 lines

But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).

Not knowing anything about (c)lapack (or fortran), is there any performance advantage using lapack over clapack? Otherwise just always using clapack seems like the simplest solution.

Quoted 5 lines

Sounds OK to me.

Bernd, you're the only one who ever made a real contribution to lapack. Thoughts?

Benjamin, you originally contributed lapack. Any ideas?

I think lapack also might serve as a useful reference design for anyone trying to bring an external Fortran package into their build.

If one is much more difficult to maintain, it makes sense to get rid of that one. I think in this case clapack might be pretty well maintained, but in general it's nice to avoid intermediate sources as they are just another level of indirection where problems can occur.

Don't feel very strongly one way or the other. At this point, both will be in the history so breadcrumbs are there if someone wants to use the package that has support dropped.

Cheers,
Ben

Quoted 3 lines

Regards,
Arnout

#14 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

Hi Giulio, All,

Le 28/05/2019 ? 07:34, Benjamin Kamath a ?crit?:

Quoted 26 lines

Sub options ate my last reply. Re-posting to list.

Quoted 23 lines · Arnout Vandecappelle
Quoted 19 lines · Peter Korsgaard
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arnout" == Arnout Vandecappelle <[email protected]> writes:

Hi,

Quoted 15 lines
Quoted 1 line
Quoted 1 line

But IMHO we should proceed with dropping clapack.

Why? it is maintained, and it is a dependency of armadillo...

I had a deeper look, and actually lapack and clapack are the same, just
compiled differently: lapack is built with the fortran compiler, clapack is generated with f2c (by upstream) and built with the C compiler. AFAIU, anything that uses lapack should also be able to use clapack and vice versa.

We have two packages like that: numpy and armadillo. They both seem to support
both lapack and clapack.

So, it seems indeed removing clapack is the appropriate thing. Except: lapack
requires a fortran compiler, which is not always available. So in that sense, it would actually be more appropriate to remove lapack...

Maybe we should convince Thomas to generate toolchains from toolchain-builder project with gFortran :

http://patchwork.ozlabs.org/patch/1113806/

Quoted 17 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

But maybe it would be better to normally use lapack, and only enable clapack
when lapack isn't available (i.e. when there's no fortran compiler).

Not knowing anything about (c)lapack (or fortran), is there any performance advantage using lapack over clapack? Otherwise just always using clapack seems like the simplest solution.

Quoted 5 lines

Sounds OK to me.

Bernd, you're the only one who ever made a real contribution to lapack. Thoughts?

Benjamin, you originally contributed lapack. Any ideas?

I think lapack also might serve as a useful reference design for anyone trying to bring an external Fortran package into their build.

Quoted 8 lines

If one is much more difficult to maintain, it makes sense to get rid of that one. I think in this case clapack might be pretty well maintained, but in general it's nice to avoid intermediate sources as they are just another level of indirection where problems can occur.

Don't feel very strongly one way or the other. At this point, both will be in the history so breadcrumbs are there if someone wants to use the package that has support dropped.

Actually python-numpy doesn't seems to work at all due to a runtime issue:

http://patchwork.ozlabs.org/patch/1114198/

http://patchwork.ozlabs.org/patch/1112759/
(It would be great if python-numpy is runtime tested in gitlab)

Also, using uClibc fail too for another reason.

Sorry, I missed this thread while working on theses patches...

As a side project, I'm testing python-scipy package that require gFortran to build. For now Python is stuck while importing the library for some reason...

Best regards,
Romain

Quoted 12 lines

Cheers,
Ben

Quoted 3 lines

Regards,
Arnout

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#15 lore raw

PATCH v2 1/1 package/python-numpy: fix occasional build failure with lapack

In reply to Arnout Vandecappelle

Am Sun, 26 May 2019 14:29:26 +0200 schrieb Arnout Vandecappelle:

Quoted 2 lines

Bernd, you're the only one who ever made a real contribution to lapack. Thoughts?

Hi Arnout,

while it is true that I sent a version bump for lapack nearly two years ago this was part of a broader approach to bump many outdated packages. I never used lapack and therefore I am unable to contribute to the current discussion, sorry.

Regards, Bernd