libxcb: Add inherit python3native
PATCH 1/1 libxcb: Add inherit python3native
During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.
--- meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 + 1 file changed, 1 insertion(+)
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1 -0
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*" FILES_${PN} = "${libdir}/libxcb.so.*" inherit autotools pkgconfig features_check +inherit python3native # The libxau and others requires x11 in DISTRO_FEATURES REQUIRED_DISTRO_FEATURES = "x11"
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
Quoted 15 lines · Josh Lehan via lists.openembedded.org
During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.
Signed-off-by Josh Lehan <[email protected]>---
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +1 file changed, 1 insertion(+)meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +0
-0@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"FILES_${PN} = "${libdir}/libxcb.so.*"
Quoted 3 lines
inherit autotools pkgconfig
features_check
+inherit python3native
Could you provide a little more information about how this is breaking things?
Whilst python is used by the libxcb build, I believe it shouldn't change the output, it doesn't in our reproducibity tests.
Adding a python-native dependency does slow the build down a bit so if we don't need it, it would be good to avoid it.
Cheers,
Richard
Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:
Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.
Traceback (most recent call last):
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
line 3338, in <module>
from xcbgen.state import Module
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'
(/usr/lib/python3.9/fractions.py)
The above error text repeats for many of the files it tries to compile within libxcb.
In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941 <(650)%20733-8941>
Quoted 42 lines · Richard Purdie
Review
Quoted 9 lines · Josh Lehan via lists.openembedded.org
During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.
Signed-off-by Josh Lehan <[email protected]>--- meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 + 1 file changed, 1 insertion(+)(quoted diff) +0
-0b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28
Quoted 9 lines
index
f0a223a19c..40c7b1cc3c100644meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1
-0@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*" FILES_${PN} = "${libdir}/libxcb.so.*" inherit autotools pkgconfig features_check +inherit python3nativeCould you provide a little more information about how this is breaking things?
Whilst python is used by the libxcb build, I believe it shouldn't change the
output, it doesn't in our reproducibity tests.Adding a python-native dependency does slow the build down a bit so if we don't need it, it would be good to avoid it.
Cheers,
Richard
I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?
Alex
Quoted 101 lines · Josh Lehan via lists.openembedded.org
Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:
Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.Traceback (most recent call last):
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
line 3338, in <module>
from xcbgen.state import Module
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'
(/usr/lib/python3.9/fractions.py)The above error text repeats for many of the files it tries to compile within libxcb.
In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941 <(650)%20733-8941>
Quoted 42 lines · Richard Purdie
Review
Quoted 9 lines · Josh Lehan via lists.openembedded.org
During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.
Signed-off-by Josh Lehan <[email protected]>--- meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 + 1 file changed, 1 insertion(+)(quoted diff) +0
-0b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28
Quoted 9 lines
index
f0a223a19c..40c7b1cc3c100644meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1
-0@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*" FILES_${PN} = "${libdir}/libxcb.so.*" inherit autotools pkgconfig features_check +inherit python3nativeCould you provide a little more information about how this is breaking things?
Whilst python is used by the libxcb build, I believe it shouldn't change the
output, it doesn't in our reproducibity tests.Adding a python-native dependency does slow the build down a bit so if we don't need it, it would be good to avoid it.
Cheers,
Richard
Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.
Alex
Quoted 110 lines · Alexander Kanavin
I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?
Alex
Quoted 102 lines · Josh Lehan via lists.openembedded.org
Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:
Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.Traceback (most recent call last):
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
line 3338, in <module>
from xcbgen.state import Module
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'
(/usr/lib/python3.9/fractions.py)The above error text repeats for many of the files it tries to compile within libxcb.
In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941 <(650)%20733-8941>
Quoted 43 lines · Richard Purdie
Review
Quoted 9 lines · Josh Lehan via lists.openembedded.org
During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.
Signed-off-by Josh Lehan <[email protected]>--- meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 + 1 file changed, 1 insertion(+)(quoted diff) +0
-0b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28
Quoted 9 lines
index
f0a223a19c..40c7b1cc3c100644meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1
-0@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*" FILES_${PN} = "${libdir}/libxcb.so.*" inherit autotools pkgconfig features_check +inherit python3nativeCould you provide a little more information about how this is breaking things?
Whilst python is used by the libxcb build, I believe it shouldn't change the
output, it doesn't in our reproducibity tests.Adding a python-native dependency does slow the build down a bit so if we
don't need it, it would be good to avoid it.Cheers,
Richard
Thanks. It's true, my local build is a little bit outdated. I'm running OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.
However, there is the same underlying problem, even if I upgraded. It looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.
As for "from fractions import gcd" becoming an error in 3.9, there was this
change: https://bugs.python.org/issue39350
It appears fractions.gcd was deprecated for a while. It was moved to math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941
Quoted 118 lines · Alexander Kanavin
Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.
Alex
Quoted 110 lines · Alexander Kanavin
I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?
Alex
Quoted 102 lines · Josh Lehan via lists.openembedded.org
Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:
Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.Traceback (most recent call last):
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
line 3338, in <module>
from xcbgen.state import Module
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'
(/usr/lib/python3.9/fractions.py)The above error text repeats for many of the files it tries to compile within libxcb.
In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941 <(650)%20733-8941>
Quoted 43 lines · Richard Purdie
Review
Quoted 9 lines · Josh Lehan via lists.openembedded.org
During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.
Signed-off-by Josh Lehan <[email protected]>--- meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 + 1 file changed, 1 insertion(+)(quoted diff) +0
-0b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28
Quoted 9 lines
index
f0a223a19c..40c7b1cc3c100644meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1
-0@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*" FILES_${PN} = "${libdir}/libxcb.so.*" inherit autotools pkgconfig features_check +inherit python3nativeCould you provide a little more information about how this is breaking things?
Whilst python is used by the libxcb build, I believe it shouldn't change the
output, it doesn't in our reproducibity tests.Adding a python-native dependency does slow the build down a bit so if we
don't need it, it would be good to avoid it.Cheers,
Richard
As a matter of policy, yocto recipes default to using python from the host, and are only switched to native python when there host python does not come with needed 3rd party modules (which are then pulled in via -native DEPENDS), or there is some other issue that cannot be resolved with host python. Native python dependency lengthens the dependency chains, which is especially critical for -native recipes.
In this case, the problem is that libxcb in your build is too old for the latest python 3.9, so you need to either update libxcb or backport an upstream fix to make it work with 3.9 again. Your patch would not resolve the issue, as native python in latest oe-core master is also 3.9, and the failure would be the same.
Alex
Quoted 151 lines · Josh Lehan
Thanks. It's true, my local build is a little bit outdated. I'm running OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.
However, there is the same underlying problem, even if I upgraded. It looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.
As for "from fractions import gcd" becoming an error in 3.9, there was this change: https://bugs.python.org/issue39350
It appears fractions.gcd was deprecated for a while. It was moved to math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941
Quoted 118 lines · Alexander Kanavin
Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.
Alex
Quoted 110 lines · Alexander Kanavin
I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?
Alex
Quoted 102 lines · Josh Lehan via lists.openembedded.org
Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:
Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.Traceback (most recent call last):
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
line 3338, in <module>
from xcbgen.state import Module
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'
(/usr/lib/python3.9/fractions.py)The above error text repeats for many of the files it tries to compile within libxcb.
In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941 <(650)%20733-8941>
Quoted 43 lines · Richard Purdie
Review
Quoted 9 lines · Josh Lehan via lists.openembedded.org
During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.
Signed-off-by Josh Lehan <[email protected]>--- meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 + 1 file changed, 1 insertion(+)(quoted diff) +0
-0b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28
Quoted 9 lines
index
f0a223a19c..40c7b1cc3c100644meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1
-0@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*" FILES_${PN} = "${libdir}/libxcb.so.*" inherit autotools pkgconfig features_check +inherit python3nativeCould you provide a little more information about how this is breaking things?
Whilst python is used by the libxcb build, I believe it shouldn't change the
output, it doesn't in our reproducibity tests.Adding a python-native dependency does slow the build down a bit so if we
don't need it, it would be good to avoid it.Cheers,
Richard
It's correct because the package it depends on (xcb-proto-native) is already inherit python3native. This package's xcbgen invocation is broken because it is using a combination of files generated for native python but executed through host python (look at the site-packages version). It can't bloat our dependency tree any more as it already implicitly depends on python3-native.
Quoted 113 lines · Alexander Kanavin
As a matter of policy, yocto recipes default to using python from the host, and are only switched to native python when there host python does not come with needed 3rd party modules (which are then pulled in via -native DEPENDS), or there is some other issue that cannot be resolved with host python. Native python dependency lengthens the dependency chains, which is especially critical for -native recipes.
In this case, the problem is that libxcb in your build is too old for the latest python 3.9, so you need to either update libxcb or backport an upstream fix to make it work with 3.9 again. Your patch would not resolve the issue, as native python in latest oe-core master is also 3.9, and the failure would be the same.
Alex
Quoted 102 lines · Josh Lehan
Thanks. It's true, my local build is a little bit outdated. I'm running OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.
However, there is the same underlying problem, even if I upgraded. It looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.
As for "from fractions import gcd" becoming an error in 3.9, there was this change: https://bugs.python.org/issue39350
It appears fractions.gcd was deprecated for a while. It was moved to math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941
Quoted 86 lines · Alexander Kanavin
Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.
Alex
Quoted 80 lines · Alexander Kanavin
I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?
Alex
Quoted 74 lines · Josh Lehan via lists.openembedded.org
Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:
Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.Traceback (most recent call last):
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py", line 3338, in <module>
from xcbgen.state import Module
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions' (/usr/lib/python3.9/fractions.py)The above error text repeats for many of the files it tries to compile within libxcb.
In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941
Quoted 38 lines · Richard Purdie
Quoted 13 lines · Josh Lehan via lists.openembedded.org
During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.
Signed-off-by Josh Lehan <[email protected]>--- meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 + 1 file changed, 1 insertion(+)meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +0
-0meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28
Quoted 5 lines
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1
-0@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*" FILES_${PN} = "${libdir}/libxcb.so.*" inherit autotools pkgconfig features_check +inherit python3nativeCould you provide a little more information about how this is breaking things?
Whilst python is used by the libxcb build, I believe it shouldn't change the output, it doesn't in our reproducibity tests.
Adding a python-native dependency does slow the build down a bit so if we don't need it, it would be good to avoid it.
Cheers,
Richard
That seems correct, but if we force the native python 3.9 to be used, is the current libxcb/xcbgen in master compatible with it (e.g. not anymore using the removed feautre)?
Alex
Quoted 182 lines · William Kennington
It's correct because the package it depends on (xcb-proto-native) is already
inherit python3native. This package's xcbgen invocation is broken because it is using a combination of files generated for native python but executed through host python (look at the site-packages version). It can't bloat our dependency tree any more as it already implicitly depends on python3-native.Quoted 2 lines · Alexander Kanavin
As a matter of policy, yocto recipes default to using python from the
host, and are only switched to native python when there host python does not come with needed 3rd party modules (which are then pulled in via -native DEPENDS), or there is some other issue that cannot be resolved with host python. Native python dependency lengthens the dependency chains, which is especially critical for -native recipes.
Quoted 2 lines
In this case, the problem is that libxcb in your build is too old for
the latest python 3.9, so you need to either update libxcb or backport an upstream fix to make it work with 3.9 again. Your patch would not resolve the issue, as native python in latest oe-core master is also 3.9, and the failure would be the same.
Quoted 6 lines
Alex
Quoted 2 lines · Josh Lehan
Thanks. It's true, my local build is a little bit outdated. I'm running
OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.
Quoted 2 lines
Quoted 2 lines
However, there is the same underlying problem, even if I upgraded. It
looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.
Quoted 2 lines
Quoted 2 lines
As for "from fractions import gcd" becoming an error in 3.9, there was
this change: https://bugs.python.org/issue39350
Quoted 2 lines
Quoted 2 lines
It appears fractions.gcd was deprecated for a while. It was moved to
math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).
Quoted 8 lines
Quoted 8 lines
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941
On Tue, Apr 13, 2021 at 1:08 PM Alexander Kanavin <
Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Also note: oe-core master already has 3.9, so you need to find out why
'3.8' shows up in install paths as well. Something doesn't compute here.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Alex
On Tue, 13 Apr 2021 at 22:07, Alexander Kanavin <
Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I think you need to dig a little deeper. Why 'from fractions import
gcd' is a problem with host python3.9 but not a problem with native python 3.9?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Alex
On Tue, 13 Apr 2021 at 20:58, Josh Lehan via lists.openembedded.org
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Hi there, thanks. I'm not entirely sure what "python3native" does
exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.Traceback (most recent call last):
File"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py", line 3338, in <module>
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
from xcbgen.state import Module
File"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
from xcbgen import matcher
File"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
from xcbgen.xtypes import *
File"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
from xcbgen.align import Alignment, AlignmentLog
File"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'(
/usr/lib/python3.9/fractions.py)Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The above error text repeats for many of the files it tries to
compile within libxcb.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
In general, it seems the safest approach is to standardize on a
Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Josh
Josh Lehan | Software Engineer | [email protected] | +1
650-733-8941
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
On Tue, Apr 13, 2021 at 2:45 AM Richard Purdie <
Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Tue, 2021-04-13 at 02:24 -0700, Josh Lehan via
Quoted 9 lines · lists.openembedded.org
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Review
Quoted 9 lines
During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.
Signed-off-by Josh Lehan <[email protected]>--- meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 + 1 file changed, 1 insertion(+)(quoted diff) +0
-0b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
index
f0a223a19c..40c7b1cc3c100644meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +0
-0@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*" FILES_${PN} = "${libdir}/libxcb.so.*"Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28
Quoted 3 lines
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1
-0inherit autotools pkgconfig features_check +inherit python3nativeCould you provide a little more information about how this is
breaking things?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Whilst python is used by the libxcb build, I believe it shouldn't
change the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
output, it doesn't in our reproducibity tests.
Adding a python-native dependency does slow the build down a bit so
if we
Quoted 16 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
don't need it, it would be good to avoid it.
Cheers,
Richard
Quoted 7 lines · William A. Kennington III via lists.openembedded.org
It's correct because the package it depends on (xcb-proto-native) is already
inherit python3native. This package's xcbgen invocation is broken because it is using a combination of files generated for native python but executed through host python (look at the site-packages version). It can't bloat our dependency tree any more as it already implicitly depends on python3-native.
I tend to agree. Although this makes me think if we have more such mix and match happening elsewhere too
in that case, would it be better to always use a captured version of python3 on native build hosts and ignore distro
version of python3 always something like buildtools-tarball should then become mandatory.
Quoted 119 lines
Quoted 113 lines · Alexander Kanavin
As a matter of policy, yocto recipes default to using python from the host, and are only switched to native python when there host python does not come with needed 3rd party modules (which are then pulled in via -native DEPENDS), or there is some other issue that cannot be resolved with host python. Native python dependency lengthens the dependency chains, which is especially critical for -native recipes.
In this case, the problem is that libxcb in your build is too old for the latest python 3.9, so you need to either update libxcb or backport an upstream fix to make it work with 3.9 again. Your patch would not resolve the issue, as native python in latest oe-core master is also 3.9, and the failure would be the same.
Alex
Quoted 102 lines · Josh Lehan
Thanks. It's true, my local build is a little bit outdated. I'm running OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.
However, there is the same underlying problem, even if I upgraded. It looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.
As for "from fractions import gcd" becoming an error in 3.9, there was this change: https://bugs.python.org/issue39350
It appears fractions.gcd was deprecated for a while. It was moved to math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941
Quoted 86 lines · Alexander Kanavin
Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.
Alex
Quoted 80 lines · Alexander Kanavin
I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?
Alex
Quoted 74 lines · Josh Lehan via lists.openembedded.org
Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:
Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.Traceback (most recent call last):
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py", line 3338, in <module>
from xcbgen.state import Module
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions' (/usr/lib/python3.9/fractions.py)The above error text repeats for many of the files it tries to compile within libxcb.
In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).
Josh
Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941
Quoted 38 lines · Richard Purdie
Quoted 13 lines · Josh Lehan via lists.openembedded.org
During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.
Signed-off-by Josh Lehan <[email protected]>--- meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 + 1 file changed, 1 insertion(+)meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +0
-0meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28
Quoted 5 lines
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1
-0@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*" FILES_${PN} = "${libdir}/libxcb.so.*" inherit autotools pkgconfig features_check +inherit python3nativeCould you provide a little more information about how this is breaking things?
Whilst python is used by the libxcb build, I believe it shouldn't change the output, it doesn't in our reproducibity tests.
Adding a python-native dependency does slow the build down a bit so if we don't need it, it would be good to avoid it.
Cheers,
Richard