meson: bump version to 0.48.1

Bump version to 0.48.1 and re-add the RPATH patch.

Signed-off-by Eric Le Bihan <[email protected]>

Bump version to 0.48.1 and re-add the RPATH patch.

Signed-off-by Eric Le Bihan <[email protected]>

---
 ...y-fix-RPATH-if-install_rpath-is-not-empty.patch | 31 ++++++++++++++++++++++
 package/meson/meson.hash                           |  4 +--
 package/meson/meson.mk                             |  2 +-
 3 files changed, 34 insertions(+), 3 deletions(-)
 create mode 100644 package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch new file +0 -0

new file mode 100644
index 0000000000..752042f7ec

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch new file +26 -0

+---
+ mesonbuild/minstall.py | 8 ++++++++
+ 1 file changed, 8 insertions(+)
+
+diff --git a/mesonbuild/minstall.py b/mesonbuild/minstall.py
+index 4615b6dd..dd476ea8 100644
+--- a/mesonbuild/minstall.py
++++ b/mesonbuild/minstall.py
+@@ -440,6 +440,14 @@ class Installer:
+                 try:
+                     depfixer.fix_rpath(outname, install_rpath, final_path,
+                                        install_name_mappings, verbose=False)
++                    # Buildroot check-host-rpath script expects RPATH
++                    # But if install_rpath is empty, it will stripped.
++                    # So, preserve it in this case
++                    if install_rpath:
++                        depfixer.fix_rpath(outname, install_rpath, final_path,
++                                           install_name_mappings, verbose=False)
++                    else:
++                        print("Skipping RPATH fixing")
+                 except SystemExit as e:
+                     if isinstance(e.code, int) and e.code == 0:
+                         pass
+-- 
+2.14.4
+

package/meson/meson.hash +2 -2

index 4a4923bd12..656f51f01d 100644
@@ -1,4 +1,4 @@
 # Locally calculated after checking pgp signature
-# https://github.com/mesonbuild/meson/releases/download/0.47.1/meson-0.47.1.tar.gz.asc
-sha256 d673de79f7bab064190a5ea06140eaa8415efb386d0121ba549f6d66c555ada6 meson-0.47.1.tar.gz
+# https://github.com/mesonbuild/meson/releases/download/0.48.1/meson-0.48.1.tar.gz.asc
+sha256 425d12edbb81498314926afa93a3eb2fca4ca6929cbe0f8abeb6ab45d1d9fa8f meson-0.48.1.tar.gz
 sha256 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 COPYING

package/meson/meson.mk +0 -0

index 08e5c505c9..f24aef06fc 100644

Quoted 1 line

Quoted 1 line

host-meson 0.48.1 Building

Hello Eric,

Quoted 16 lines · Eric Le Bihan

Bump version to 0.48.1 and re-add the RPATH patch.

Signed-off-by Eric Le Bihan <[email protected]>

---
 ...y-fix-RPATH-if-install_rpath-is-not-empty.patch | 31 ++++++++++++++++++++++
 package/meson/meson.hash                           |  4 +--
 package/meson/meson.mk                             |  2 +-
 3 files changed, 34 insertions(+), 3 deletions(-)
 create mode 100644 package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch new file +0 -0

new file mode 100644
index 0000000000..752042f7ec

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch line 5

Quoted 5 lines

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch new file +5 -0

@@ -0,0 +1,31 @@
+From cb5af88b1102ea8379323f95b9588c0bef71f175 Mon Sep 17 00:00:00 2001
+From: Eric Le Bihan <[email protected]>
+Date: Sat, 14 Jul 2018 11:18:45 +0200
+Subject: [PATCH] Only fix RPATH if install_rpath is not empty
+  

Signed-off-by line missing here...

Quoted 42 lines

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch new file +26 -0

+---
+ mesonbuild/minstall.py | 8 ++++++++
+ 1 file changed, 8 insertions(+)
+
+diff --git a/mesonbuild/minstall.py b/mesonbuild/minstall.py
+index 4615b6dd..dd476ea8 100644
+--- a/mesonbuild/minstall.py
++++ b/mesonbuild/minstall.py
+@@ -440,6 +440,14 @@ class Installer:
+                 try:
+                     depfixer.fix_rpath(outname, install_rpath, final_path,
+                                        install_name_mappings, verbose=False)
++                    # Buildroot check-host-rpath script expects RPATH
++                    # But if install_rpath is empty, it will stripped.
++                    # So, preserve it in this case
++                    if install_rpath:
++                        depfixer.fix_rpath(outname, install_rpath, final_path,
++                                           install_name_mappings, verbose=False)
++                    else:
++                        print("Skipping RPATH fixing")
+                 except SystemExit as e:
+                     if isinstance(e.code, int) and e.code == 0:
+                         pass
+-- 
+2.14.4
+

package/meson/meson.hash +2 -2

index 4a4923bd12..656f51f01d 100644
@@ -1,4 +1,4 @@
 # Locally calculated after checking pgp signature
-# https://github.com/mesonbuild/meson/releases/download/0.47.1/meson-0.47.1.tar.gz.asc
-sha256 d673de79f7bab064190a5ea06140eaa8415efb386d0121ba549f6d66c555ada6 meson-0.47.1.tar.gz
+# https://github.com/mesonbuild/meson/releases/download/0.48.1/meson-0.48.1.tar.gz.asc
+sha256 425d12edbb81498314926afa93a3eb2fca4ca6929cbe0f8abeb6ab45d1d9fa8f meson-0.48.1.tar.gz
 sha256 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 COPYING

package/meson/meson.mk +0 -0

index 08e5c505c9..f24aef06fc 100644

package/meson/meson.mk line 10

Quoted 8 lines

package/meson/meson.mk +1 -1

@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-MESON_VERSION = 0.47.1
+MESON_VERSION = 0.48.1
 MESON_SITE = https://github.com/mesonbuild/meson/releases/download/$(MESON_VERSION)
 MESON_LICENSE = Apache-2.0
 MESON_LICENSE_FILES = COPYING  

With the patch applied I get the following error:

Quoted 1 line

Quoted 1 line

Quoted 1 line

host-meson 0.48.1 Building

(cd .../build/host-meson-0.48.1//; PATH=".../host/bin:.../host/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/sbin" PYTHONNOUSERSITE=1 .../host/bin/python3 setup.py build ) Traceback (most recent call last):
File "setup.py", line 26, in <module>
from setuptools import setup
ModuleNotFoundError: No module named 'setuptools'

	$ ls -d build/host-python*
build/host-python-2.7.15
build/host-python3-3.7.0
build/host-python-setuptools-40.0.0

Regards,
Peter
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Quoted 14 lines · Peter Seiderer

Hello Eric,

Quoted 10 lines · Eric Le Bihan

Bump version to 0.48.1 and re-add the RPATH patch.

Signed-off-by Eric Le Bihan <[email protected]>

---
 ...y-fix-RPATH-if-install_rpath-is-not-empty.patch | 31 ++++++++++++++++++++++
 package/meson/meson.hash                           |  4 +--
 package/meson/meson.mk                             |  2 +-
 3 files changed, 34 insertions(+), 3 deletions(-)
 create mode 100644 package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch

Quoted 5 lines

Quoted 5 lines

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch new file +0 -0

new file mode 100644
index 0000000000..752042f7ec

Quoted 9 lines

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch line 5

Quoted 6 lines

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch new file +5 -0

@@ -0,0 +1,31 @@
+From cb5af88b1102ea8379323f95b9588c0bef71f175 Mon Sep 17 00:00:00 2001
+From: Eric Le Bihan <[email protected]>
+Date: Sat, 14 Jul 2018 11:18:45 +0200
+Subject: [PATCH] Only fix RPATH if install_rpath is not empty
+

Signed-off-by line missing here...

Quoted 9 lines

With the patch applied I get the following error:

Quoted 1 line

Quoted 1 line

Quoted 1 line

host-meson 0.48.1 Building

(cd .../build/host-meson-0.48.1//; PATH=".../host/bin:.../host/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/sbin" PYTHONNOUSERSITE=1 .../host/bin/python3 setup.py build ) Traceback (most recent call last):
File "setup.py", line 26, in <module>
from setuptools import setup
ModuleNotFoundError: No module named 'setuptools'

Quoted 4 lines

	$ ls -d build/host-python*
build/host-python-2.7.15
build/host-python3-3.7.0
build/host-python-setuptools-40.0.0

graph-depends.pdf shows the following host-meson dependencies:

- host-meson --> host-python3
- host-meson --> host-python-setuptools --> host-python
- host-meson --> host-ninja --> host-python

Maybe host-python-setuptools are build for the wrong python version (python2 instead of python3)?

Hi!

Quoted 16 lines · Peter Seiderer

Quoted 14 lines · Peter Seiderer

Hello Eric,

Quoted 10 lines · Eric Le Bihan

Bump version to 0.48.1 and re-add the RPATH patch.

Signed-off-by Eric Le Bihan <[email protected]>

---
 ...y-fix-RPATH-if-install_rpath-is-not-empty.patch | 31 ++++++++++++++++++++++
 package/meson/meson.hash                           |  4 +--
 package/meson/meson.mk                             |  2 +-
 3 files changed, 34 insertions(+), 3 deletions(-)
 create mode 100644 package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch

Quoted 14 lines

Quoted 5 lines

Quoted 5 lines

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch new file +0 -0

new file mode 100644
index 0000000000..752042f7ec

Quoted 9 lines

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch line 5

Quoted 6 lines

package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch new file +5 -0

@@ -0,0 +1,31 @@
+From cb5af88b1102ea8379323f95b9588c0bef71f175 Mon Sep 17 00:00:00 2001
+From: Eric Le Bihan <[email protected]>
+Date: Sat, 14 Jul 2018 11:18:45 +0200
+Subject: [PATCH] Only fix RPATH if install_rpath is not empty
+  

Signed-off-by line missing here...

Good catch! Will fix.

Quoted 14 lines

Quoted 9 lines

With the patch applied I get the following error:

Quoted 1 line

Quoted 1 line

Quoted 1 line

host-meson 0.48.1 Building

(cd .../build/host-meson-0.48.1//; PATH=".../host/bin:.../host/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/sbin" PYTHONNOUSERSITE=1 .../host/bin/python3 setup.py build ) Traceback (most recent call last):
File "setup.py", line 26, in <module>
from setuptools import setup
ModuleNotFoundError: No module named 'setuptools'

Quoted 4 lines

	$ ls -d build/host-python*
build/host-python-2.7.15
build/host-python3-3.7.0
build/host-python-setuptools-40.0.0  

Quoted 7 lines

graph-depends.pdf shows the following host-meson dependencies:

- host-meson --> host-python3
- host-meson --> host-python-setuptools --> host-python
- host-meson --> host-ninja --> host-python

Maybe host-python-setuptools are build for the wrong python version (python2 instead of python3)?

I got the same issue and thought that was related to using an old build environment and not a fresh one.

The package host-python-setuptools was using $(HOST_DIR)/bin/python, symlinked to $(HOST_DIR)/bin/python2.7. Symlinking to python3 and rebuilding host-python-setuptools solves host-meson build issue, but that is not a proper solution.

I'm a bit puzzled... If for some reason host-meson had been built before a package requiring host-python2 and host-python-setuptools, would the build of this package fail due to host-python-setuptools having been built for host-python3?

Thanks for the review.

Regards,

--
ELB
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Hi!

Quoted 1 line · Peter Seiderer

On Fri, 19 Oct 2018 20:30:50 +0200, Peter Seiderer <[email protected]> wrote:

Quoted 13 lines

Quoted 12 lines

With the patch applied I get the following error:

Quoted 1 line

Quoted 1 line

Quoted 1 line

host-meson 0.48.1 Building

(cd .../build/host-meson-0.48.1//; PATH=".../host/bin:.../host/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/sbin" PYTHONNOUSERSITE=1 .../host/bin/python3 setup.py build ) Traceback (most recent call last):
File "setup.py", line 26, in <module>
from setuptools import setup
ModuleNotFoundError: No module named 'setuptools'>>>

	$ ls -d build/host-python*
build/host-python-2.7.15
build/host-python3-3.7.0
build/host-python-setuptools-40.0.0

Quoted 7 lines

graph-depends.pdf shows the following host-meson dependencies:

- host-meson --> host-python3
- host-meson --> host-python-setuptools --> host-python
- host-meson --> host-ninja --> host-python

Maybe host-python-setuptools are build for the wrong python version (python2 instead of python3)?

I got the same issue and thought that was related to using an old build environment and not a fresh one.

The package host-python-setuptools was using $(HOST_DIR)/bin/python, symlinked to $(HOST_DIR)/bin/python2.7. Symlinking to python3 and rebuilding host-python-setuptools solves host-meson build issue, but that is not a proper solution.

I'm a bit puzzled... If for some reason host-meson had been built before a package requiring host-python2 and host-python-setuptools, would the build of this package fail due to host-python-setuptools having been built for host-python3?

Thanks for the review.

Regards,

--
ELB
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Hi Eric,

Quoted 4 lines · Eric Le Bihan

Hi!

Quoted 1 line · Peter Seiderer

On Fri, 19 Oct 2018 20:30:50 +0200, Peter Seiderer <[email protected]> wrote:

[snip]

Quoted 12 lines

Quoted 12 lines

Quoted 12 lines

With the patch applied I get the following error:

Quoted 1 line

Quoted 1 line

Quoted 1 line

host-meson 0.48.1 Building

(cd .../build/host-meson-0.48.1//; PATH=".../host/bin:.../host/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/sbin" PYTHONNOUSERSITE=1 .../host/bin/python3 setup.py build ) Traceback (most recent call last):
File "setup.py", line 26, in <module>
from setuptools import setup
ModuleNotFoundError: No module named 'setuptools'>>>

	$ ls -d build/host-python*
build/host-python-2.7.15
build/host-python3-3.7.0
build/host-python-setuptools-40.0.0

Quoted 7 lines

Quoted 7 lines

graph-depends.pdf shows the following host-meson dependencies:

- host-meson --> host-python3
- host-meson --> host-python-setuptools --> host-python
- host-meson --> host-ninja --> host-python

Maybe host-python-setuptools are build for the wrong python version (python2 instead of python3)?

That is correct. When target python or python3 is selected, then the
corresponding host python is also selected, and *all* host python packages are built for that specific python version. But if only target python is selected, then you will get host-python as well, but you can also still depend on host-python3. In that case, however, all the host python packages will be built only for python2, not for python3. Hence the breakage.

I'm surprised though that we don't see this in the autobuilders... Your patch
doesn't change anything there, does it?

Quoted 9 lines

I got the same issue and thought that was related to using an old build environment and not a fresh one.

The package host-python-setuptools was using $(HOST_DIR)/bin/python, symlinked to $(HOST_DIR)/bin/python2.7. Symlinking to python3 and rebuilding host-python-setuptools solves host-meson build issue, but that is not a proper solution.

Quoted 4 lines

I'm a bit puzzled... If for some reason host-meson had been built before a package requiring host-python2 and host-python-setuptools, would the build of this package fail due to host-python-setuptools having been built for host-python3?

No, it purely depends on what you have selected for the target.

Hm, actually, if you have *nothing* selected for the target, it gets even more
confusing, because any host package that depends on python 2-or-3 will default to python2. I guess it then depends on the build order which version gets used... Bad!

Yann is working on a series to support side-by-side python2 and python3 on the
host and target. That would solve this issue.

Regards,
Arnout

Quoted 11 lines

Thanks for the review.

Regards,

--
ELB
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

On 20/10/2018 13:57, Arnout Vandecappelle wrote:

Quoted 8 lines

Quoted 1 line

Maybe host-python-setuptools are build for the wrong python version (python2 instead of python3)?

That is correct. When target python or python3 is selected, then the
corresponding host python is also selected, and *all* host python packages are built for that specific python version. But if only target python is selected, then you will get host-python as well, but you can also still depend on host-python3. In that case, however, all the host python packages will be built only for python2, not for python3. Hence the breakage.

Quoted 2 lines

I'm surprised though that we don't see this in the autobuilders... Your patch
doesn't change anything there, does it?

?It actually does... meson 0.47.1 still had a fallback on distutils if setuptools wasn't available, but 0.48.1 removed that fallback.

Quoted 1 line · Arnout Vandecappelle

On 20/10/2018 13:57, Arnout Vandecappelle wrote:

[snip]

Quoted 8 lines

Quoted 8 lines

Quoted 1 line

Maybe host-python-setuptools are build for the wrong python version (python2 instead of python3)?

That is correct. When target python or python3 is selected, then the
corresponding host python is also selected, and *all* host python packages are built for that specific python version. But if only target python is selected, then you will get host-python as well, but you can also still depend on host-python3. In that case, however, all the host python packages will be built only for python2, not for python3. Hence the breakage.

Quoted 4 lines

Quoted 2 lines

I'm surprised though that we don't see this in the autobuilders... Your patch
doesn't change anything there, does it?

It actually does... meson 0.47.1 still had a fallback on distutils if setuptools wasn't available, but 0.48.1 removed that fallback.

Long discussion an the BR developer meeting, here is the conclusion. Adding Yegor and Asaf in Cc as python maintainers.

The issue we have at the moment is that all host python packages are built for Python 2, not Python 3, *unless* Python 3 is selected for the target.

However, *most* packages support Python 3 as well.

So, the idea is to default to Python 3 for the host. In other words, all host packages will use Python 3 (independent of what is going on the target), except for the packages that work on python2 only. This step by itself is quite a bit of work because there are many packages that can use python2 OR python3.

This leaves us with two problems.

First of all, there may be python-2-only host packages that depend on some other host-python package that would only get built for python3. We investigated that, and it looks OK: there are only 4 python-2-only host packages: gdb, python-cheetah, python-enum34, and python-yieldfrom. The latter two are compatibility libraries so obviously only available on Python2. python-yieldfrom is not used at all. python-enum34 is only used by setools, in case it needs to be built for host-python2, so that should be OK. host-python-cheetah is only used by gr-osmosdr; bumping python-cheetah probably enables it for Python 3 so would solve that issue. And gdb doesn't need any other python package, just python itself. So all this looks fixable.

The bigger problem, is when a setup.py script tries to import a module from some python package. The clearest one of those is python-setuptools: any python package that uses setuptools will import setuptools using the same host-python version as on the target. So when you have python2 on the target, the setup.py script will be run in python2, so we need host-python-setuptools for python2. The solution there, we think, is to add a separate host-python2-setuptools package that is used instead of host-python-setuptools in case the target is python2. Unfortunately there are quite a few others that may need the same treatment. Some of them, however, probably don't really need to be imported in the setup.py script so they can be removed as a dependency. Others, however, will need the same treatment. In some cases, we might be able to just build both python2 and python3 versions (e.g. for python-sip this should be possible).

Finally, there are most likely some corner cases that we didn't think of...

So it looks like this will be a pretty big migration. Therefore, for the time being, it may be easier to just add a host-python3-setuptools package and use that in meson.

By the way, lirc-tools has the same problem. It adds python3 host-python-setuptools to its dependencies, but the setuptools don't get imported so it just doesn't work (I think). Baruch, care to take a look at that?

Hi Arnout,

Quoted 57 lines · Arnout Vandecappelle

Quoted 1 line · Arnout Vandecappelle

On 20/10/2018 13:57, Arnout Vandecappelle wrote:

[snip]

Quoted 8 lines

Quoted 8 lines

Quoted 1 line

Maybe host-python-setuptools are build for the wrong python version (python2 instead of python3)?

That is correct. When target python or python3 is selected, then the
corresponding host python is also selected, and *all* host python packages are built for that specific python version. But if only target python is selected, then you will get host-python as well, but you can also still depend on host-python3. In that case, however, all the host python packages will be built only for python2, not for python3. Hence the breakage.

Quoted 4 lines

Quoted 2 lines

I'm surprised though that we don't see this in the autobuilders... Your patch
doesn't change anything there, does it?

It actually does... meson 0.47.1 still had a fallback on distutils if setuptools wasn't available, but 0.48.1 removed that fallback.

Long discussion an the BR developer meeting, here is the conclusion. Adding Yegor and Asaf in Cc as python maintainers.

The issue we have at the moment is that all host python packages are built for Python 2, not Python 3, *unless* Python 3 is selected for the target.

However, *most* packages support Python 3 as well.

So, the idea is to default to Python 3 for the host. In other words, all host packages will use Python 3 (independent of what is going on the target), except for the packages that work on python2 only. This step by itself is quite a bit of work because there are many packages that can use python2 OR python3.

This leaves us with two problems.

First of all, there may be python-2-only host packages that depend on some other host-python package that would only get built for python3. We investigated that, and it looks OK: there are only 4 python-2-only host packages: gdb, python-cheetah, python-enum34, and python-yieldfrom. The latter two are compatibility libraries so obviously only available on Python2. python-yieldfrom is not used at all. python-enum34 is only used by setools, in case it needs to be built for host-python2, so that should be OK. host-python-cheetah is only used by gr-osmosdr; bumping python-cheetah probably enables it for Python 3 so would solve that issue. And gdb doesn't need any other python package, just python itself. So all this looks fixable.

The bigger problem, is when a setup.py script tries to import a module from some python package. The clearest one of those is python-setuptools: any python package that uses setuptools will import setuptools using the same host-python version as on the target. So when you have python2 on the target, the setup.py script will be run in python2, so we need host-python-setuptools for python2. The solution there, we think, is to add a separate host-python2-setuptools package that is used instead of host-python-setuptools in case the target is python2. Unfortunately there are quite a few others that may need the same treatment. Some of them, however, probably don't really need to be imported in the setup.py script so they can be removed as a dependency. Others, however, will need the same treatment. In some cases, we might be able to just build both python2 and python3 versions (e.g. for python-sip this should be possible).

Finally, there are most likely some corner cases that we didn't think of...

Quoted 7 lines

So it looks like this will be a pretty big migration. Therefore, for the time being, it may be easier to just add a host-python3-setuptools package and use that in meson.

By the way, lirc-tools has the same problem. It adds python3 host-python-setuptools to its dependencies, but the setuptools don't get imported so it just doesn't work (I think). Baruch, care to take a look at that?

I'm not that much of a Python expert so I might be missing something here.

The python-pkg/setup.py file in the lirc-tools source tree have this:

from setuptools import setup, Extension

A build test with python3 enabled installs lirc python and binary code under /usr/lib/python3.7/site-packages/lirc in the target.

So what is it that "doesn't work"? Please enlighten me.

Thanks,
baruch

Quoted 73 lines · Baruch Siach

Quoted 57 lines · Arnout Vandecappelle

Quoted 1 line · Arnout Vandecappelle

On 20/10/2018 13:57, Arnout Vandecappelle wrote:

[snip]

Quoted 8 lines

Quoted 8 lines

Quoted 1 line

Maybe host-python-setuptools are build for the wrong python version (python2 instead of python3)?

That is correct. When target python or python3 is selected, then the
corresponding host python is also selected, and *all* host python packages are built for that specific python version. But if only target python is selected, then you will get host-python as well, but you can also still depend on host-python3. In that case, however, all the host python packages will be built only for python2, not for python3. Hence the breakage.

Quoted 4 lines

Quoted 2 lines

I'm surprised though that we don't see this in the autobuilders... Your patch
doesn't change anything there, does it?

It actually does... meson 0.47.1 still had a fallback on distutils if setuptools wasn't available, but 0.48.1 removed that fallback.

Long discussion an the BR developer meeting, here is the conclusion. Adding Yegor and Asaf in Cc as python maintainers.

The issue we have at the moment is that all host python packages are built for Python 2, not Python 3, *unless* Python 3 is selected for the target.

However, *most* packages support Python 3 as well.

So, the idea is to default to Python 3 for the host. In other words, all host packages will use Python 3 (independent of what is going on the target), except for the packages that work on python2 only. This step by itself is quite a bit of work because there are many packages that can use python2 OR python3.

This leaves us with two problems.

First of all, there may be python-2-only host packages that depend on some other host-python package that would only get built for python3. We investigated that, and it looks OK: there are only 4 python-2-only host packages: gdb, python-cheetah, python-enum34, and python-yieldfrom. The latter two are compatibility libraries so obviously only available on Python2. python-yieldfrom is not used at all. python-enum34 is only used by setools, in case it needs to be built for host-python2, so that should be OK. host-python-cheetah is only used by gr-osmosdr; bumping python-cheetah probably enables it for Python 3 so would solve that issue. And gdb doesn't need any other python package, just python itself. So all this looks fixable.

The bigger problem, is when a setup.py script tries to import a module from some python package. The clearest one of those is python-setuptools: any python package that uses setuptools will import setuptools using the same host-python version as on the target. So when you have python2 on the target, the setup.py script will be run in python2, so we need host-python-setuptools for python2. The solution there, we think, is to add a separate host-python2-setuptools package that is used instead of host-python-setuptools in case the target is python2. Unfortunately there are quite a few others that may need the same treatment. Some of them, however, probably don't really need to be imported in the setup.py script so they can be removed as a dependency. Others, however, will need the same treatment. In some cases, we might be able to just build both python2 and python3 versions (e.g. for python-sip this should be possible).

Finally, there are most likely some corner cases that we didn't think of...

Quoted 7 lines

So it looks like this will be a pretty big migration. Therefore, for the time being, it may be easier to just add a host-python3-setuptools package and use that in meson.

By the way, lirc-tools has the same problem. It adds python3 host-python-setuptools to its dependencies, but the setuptools don't get imported so it just doesn't work (I think). Baruch, care to take a look at that?

I'm not that much of a Python expert so I might be missing something here.

The python-pkg/setup.py file in the lirc-tools source tree have this:

from setuptools import setup, Extension

Quoted 4 lines

A build test with python3 enabled installs lirc python and binary code under /usr/lib/python3.7/site-packages/lirc in the target.

So what is it that "doesn't work"? Please enlighten me.

In the case of meson python3 is enabled for it AND python2 for ninja (via meson dependency), in this case the host-setuptools are build/installed only for python2 and the meson build fails...

Hi Peter,

Quoted 79 lines · Peter Seiderer

Quoted 77 lines · Baruch Siach

Quoted 57 lines · Arnout Vandecappelle

Quoted 1 line · Arnout Vandecappelle

On 20/10/2018 13:57, Arnout Vandecappelle wrote:

[snip]

Quoted 8 lines

Quoted 8 lines

Quoted 1 line

Maybe host-python-setuptools are build for the wrong python version (python2 instead of python3)?

That is correct. When target python or python3 is selected, then the
corresponding host python is also selected, and *all* host python packages are built for that specific python version. But if only target python is selected, then you will get host-python as well, but you can also still depend on host-python3. In that case, however, all the host python packages will be built only for python2, not for python3. Hence the breakage.

Quoted 4 lines

Quoted 2 lines

I'm surprised though that we don't see this in the autobuilders... Your patch
doesn't change anything there, does it?

It actually does... meson 0.47.1 still had a fallback on distutils if setuptools wasn't available, but 0.48.1 removed that fallback.

Long discussion an the BR developer meeting, here is the conclusion. Adding Yegor and Asaf in Cc as python maintainers.

The issue we have at the moment is that all host python packages are built for Python 2, not Python 3, *unless* Python 3 is selected for the target.

However, *most* packages support Python 3 as well.

So, the idea is to default to Python 3 for the host. In other words, all host packages will use Python 3 (independent of what is going on the target), except for the packages that work on python2 only. This step by itself is quite a bit of work because there are many packages that can use python2 OR python3.

This leaves us with two problems.

First of all, there may be python-2-only host packages that depend on some other host-python package that would only get built for python3. We investigated that, and it looks OK: there are only 4 python-2-only host packages: gdb, python-cheetah, python-enum34, and python-yieldfrom. The latter two are compatibility libraries so obviously only available on Python2. python-yieldfrom is not used at all. python-enum34 is only used by setools, in case it needs to be built for host-python2, so that should be OK. host-python-cheetah is only used by gr-osmosdr; bumping python-cheetah probably enables it for Python 3 so would solve that issue. And gdb doesn't need any other python package, just python itself. So all this looks fixable.

The bigger problem, is when a setup.py script tries to import a module from some python package. The clearest one of those is python-setuptools: any python package that uses setuptools will import setuptools using the same host-python version as on the target. So when you have python2 on the target, the setup.py script will be run in python2, so we need host-python-setuptools for python2. The solution there, we think, is to add a separate host-python2-setuptools package that is used instead of host-python-setuptools in case the target is python2. Unfortunately there are quite a few others that may need the same treatment. Some of them, however, probably don't really need to be imported in the setup.py script so they can be removed as a dependency. Others, however, will need the same treatment. In some cases, we might be able to just build both python2 and python3 versions (e.g. for python-sip this should be possible).

Finally, there are most likely some corner cases that we didn't think of...

Quoted 7 lines

So it looks like this will be a pretty big migration. Therefore, for the time being, it may be easier to just add a host-python3-setuptools package and use that in meson.

By the way, lirc-tools has the same problem. It adds python3 host-python-setuptools to its dependencies, but the setuptools don't get imported so it just doesn't work (I think). Baruch, care to take a look at that?

I'm not that much of a Python expert so I might be missing something here.

The python-pkg/setup.py file in the lirc-tools source tree have this:

from setuptools import setup, Extension

A build test with python3 enabled installs lirc python and binary code under /usr/lib/python3.7/site-packages/lirc in the target.

So what is it that "doesn't work"? Please enlighten me.

Quoted 3 lines

In the case of meson python3 is enabled for it AND python2 for ninja (via meson dependency), in this case the host-setuptools are build/installed only for python2 and the meson build fails...

So mason needs host-python3 (via HOST_MESON_NEEDS_HOST_PYTHON), but since BR2_PACKAGE_PYTHON3 is disabled, ninja drags in host-python as well. Is that correct?

I tried to reproduce something similar with BR2_PACKAGE_PYTHON3 enabled using this config:

BR2_aarch64=y
BR2_cortex_a72=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_PACKAGE_LIRC_TOOLS=y
BR2_PACKAGE_PYTHON3=y
BR2_PACKAGE_MONGODB=y

mongodb depends on host-scons that makes host-python enabled in addition to host-python3 for target python3. Yet the build of 'mongodb lirc-tools' targets succeeds, with the expected files in the lirc python package on target.

Quoted 34 lines · Arnout Vandecappelle

Hi Eric,

Quoted 4 lines · Eric Le Bihan

Hi!

Quoted 1 line · Peter Seiderer

On Fri, 19 Oct 2018 20:30:50 +0200, Peter Seiderer <[email protected]> wrote:

[snip]

Quoted 12 lines

Quoted 12 lines

Quoted 12 lines

With the patch applied I get the following error:

Quoted 1 line

Quoted 1 line

Quoted 1 line

host-meson 0.48.1 Building

(cd .../build/host-meson-0.48.1//; PATH=".../host/bin:.../host/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/sbin" PYTHONNOUSERSITE=1 .../host/bin/python3 setup.py build ) Traceback (most recent call last):
File "setup.py", line 26, in <module>
from setuptools import setup
ModuleNotFoundError: No module named 'setuptools'>>>

	$ ls -d build/host-python*
build/host-python-2.7.15
build/host-python3-3.7.0
build/host-python-setuptools-40.0.0

Quoted 7 lines

Quoted 7 lines

graph-depends.pdf shows the following host-meson dependencies:

- host-meson --> host-python3
- host-meson --> host-python-setuptools --> host-python
- host-meson --> host-ninja --> host-python

Maybe host-python-setuptools are build for the wrong python version (python2 instead of python3)?

That is correct. When target python or python3 is selected, then the
corresponding host python is also selected, and *all* host python packages are built for that specific python version. But if only target python is selected, then you will get host-python as well, but you can also still depend on host-python3. In that case, however, all the host python packages will be built only for python2, not for python3. Hence the breakage.

Quoted 2 lines

I'm surprised though that we don't see this in the autobuilders... Your patch
doesn't change anything there, does it?

?It actually does... meson 0.47.1 still had a fallback on distutils if setuptools wasn't available, but 0.48.1 removed that fallback.