Skip to contentSkip to conversation

libxcb: Add inherit python3native

PATCH In review [email protected] 5 participants 10 messages Apr 13, 2021 – Apr 14, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 libxcb: Add inherit python3native

During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.

---
 meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1 -0
index f0a223a19c..40c7b1cc3c 100644
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
 FILES_${PN} = "${libdir}/libxcb.so.*"
 
 inherit autotools pkgconfig features_check
+inherit python3native
 
 # The libxau and others requires x11 in DISTRO_FEATURES
 REQUIRED_DISTRO_FEATURES = "x11"
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
#1 lore raw
Quoted 15 lines · Josh Lehan via lists.openembedded.org

During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.

---
 meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +

 1 file changed, 1 insertion(+)
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +0 -0
index f0a223a19c..40c7b1cc3c 100644
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"

 FILES_${PN} = "${libdir}/libxcb.so.*"

Quoted 3 lines

 inherit autotools pkgconfig features_check
+inherit python3native

Could you provide a little more information about how this is breaking things?

Whilst python is used by the libxcb build, I believe it shouldn't change the output, it doesn't in our reproducibity tests.

Adding a python-native dependency does slow the build down a bit so if we don't need it, it would be good to avoid it.

Cheers,

Richard

author
#2 lore raw

Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:

Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.

Traceback (most recent call last):
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
line 3338, in <module>
from xcbgen.state import Module
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'
(/usr/lib/python3.9/fractions.py)

The above error text repeats for many of the files it tries to compile within libxcb.

In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941 <(650)%20733-8941>

Quoted 42 lines · Richard Purdie

Review

Quoted 9 lines · Josh Lehan via lists.openembedded.org

During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.

---
 meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
 1 file changed, 1 insertion(+)
(quoted diff) +0 -0

b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28

Quoted 9 lines

index f0a223a19c..40c7b1cc3c 100644

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1 -0
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
 FILES_${PN} = "${libdir}/libxcb.so.*"
 
 inherit autotools pkgconfig features_check
+inherit python3native
 

Could you provide a little more information about how this is breaking things?

Whilst python is used by the libxcb build, I believe it shouldn't change the
output, it doesn't in our reproducibity tests.

Adding a python-native dependency does slow the build down a bit so if we don't need it, it would be good to avoid it.

Cheers,

Richard

#3 lore raw

I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?

Alex

Quoted 101 lines · Josh Lehan via lists.openembedded.org

Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:

Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.

Traceback (most recent call last):
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
line 3338, in <module>
from xcbgen.state import Module
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'
(/usr/lib/python3.9/fractions.py)

The above error text repeats for many of the files it tries to compile within libxcb.

In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941 <(650)%20733-8941>

Quoted 42 lines · Richard Purdie

Review

Quoted 9 lines · Josh Lehan via lists.openembedded.org

During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.

---
 meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
 1 file changed, 1 insertion(+)
(quoted diff) +0 -0

b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28

Quoted 9 lines

index f0a223a19c..40c7b1cc3c 100644

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1 -0
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
 FILES_${PN} = "${libdir}/libxcb.so.*"
 
 inherit autotools pkgconfig features_check
+inherit python3native
 

Could you provide a little more information about how this is breaking things?

Whilst python is used by the libxcb build, I believe it shouldn't change the
output, it doesn't in our reproducibity tests.

Adding a python-native dependency does slow the build down a bit so if we don't need it, it would be good to avoid it.

Cheers,

Richard

#4 lore raw

Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.

Alex

Quoted 110 lines · Alexander Kanavin

I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?

Alex

Quoted 102 lines · Josh Lehan via lists.openembedded.org

Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:

Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.

Traceback (most recent call last):
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
line 3338, in <module>
from xcbgen.state import Module
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'
(/usr/lib/python3.9/fractions.py)

The above error text repeats for many of the files it tries to compile within libxcb.

In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941 <(650)%20733-8941>

Quoted 43 lines · Richard Purdie

Review

Quoted 9 lines · Josh Lehan via lists.openembedded.org

During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.

---
 meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
 1 file changed, 1 insertion(+)
(quoted diff) +0 -0

b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28

Quoted 9 lines

index f0a223a19c..40c7b1cc3c 100644

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1 -0
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
 FILES_${PN} = "${libdir}/libxcb.so.*"
 
 inherit autotools pkgconfig features_check
+inherit python3native
 

Could you provide a little more information about how this is breaking things?

Whilst python is used by the libxcb build, I believe it shouldn't change the
output, it doesn't in our reproducibity tests.

Adding a python-native dependency does slow the build down a bit so if we
don't need it, it would be good to avoid it.

Cheers,

Richard

author
#5 lore raw

Thanks. It's true, my local build is a little bit outdated. I'm running OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.

However, there is the same underlying problem, even if I upgraded. It looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.

As for "from fractions import gcd" becoming an error in 3.9, there was this
change: https://bugs.python.org/issue39350

It appears fractions.gcd was deprecated for a while. It was moved to math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941

Quoted 118 lines · Alexander Kanavin

Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.

Alex

Quoted 110 lines · Alexander Kanavin

I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?

Alex

Quoted 102 lines · Josh Lehan via lists.openembedded.org

Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:

Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.

Traceback (most recent call last):
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
line 3338, in <module>
from xcbgen.state import Module
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'
(/usr/lib/python3.9/fractions.py)

The above error text repeats for many of the files it tries to compile within libxcb.

In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941 <(650)%20733-8941>

Quoted 43 lines · Richard Purdie

Review

Quoted 9 lines · Josh Lehan via lists.openembedded.org

During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.

---
 meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
 1 file changed, 1 insertion(+)
(quoted diff) +0 -0

b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28

Quoted 9 lines

index f0a223a19c..40c7b1cc3c 100644

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1 -0
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
 FILES_${PN} = "${libdir}/libxcb.so.*"
 
 inherit autotools pkgconfig features_check
+inherit python3native
 

Could you provide a little more information about how this is breaking things?

Whilst python is used by the libxcb build, I believe it shouldn't change the
output, it doesn't in our reproducibity tests.

Adding a python-native dependency does slow the build down a bit so if we
don't need it, it would be good to avoid it.

Cheers,

Richard

#6 lore raw

As a matter of policy, yocto recipes default to using python from the host, and are only switched to native python when there host python does not come with needed 3rd party modules (which are then pulled in via -native DEPENDS), or there is some other issue that cannot be resolved with host python. Native python dependency lengthens the dependency chains, which is especially critical for -native recipes.

In this case, the problem is that libxcb in your build is too old for the latest python 3.9, so you need to either update libxcb or backport an upstream fix to make it work with 3.9 again. Your patch would not resolve the issue, as native python in latest oe-core master is also 3.9, and the failure would be the same.

Alex

Quoted 151 lines · Josh Lehan

Thanks. It's true, my local build is a little bit outdated. I'm running OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.

However, there is the same underlying problem, even if I upgraded. It looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.

As for "from fractions import gcd" becoming an error in 3.9, there was this change: https://bugs.python.org/issue39350

It appears fractions.gcd was deprecated for a while. It was moved to math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941

Quoted 118 lines · Alexander Kanavin

Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.

Alex

Quoted 110 lines · Alexander Kanavin

I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?

Alex

Quoted 102 lines · Josh Lehan via lists.openembedded.org

Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:

Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.

Traceback (most recent call last):
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
line 3338, in <module>
from xcbgen.state import Module
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'
(/usr/lib/python3.9/fractions.py)

The above error text repeats for many of the files it tries to compile within libxcb.

In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941 <(650)%20733-8941>

Quoted 43 lines · Richard Purdie

Review

Quoted 9 lines · Josh Lehan via lists.openembedded.org

During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.

---
 meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
 1 file changed, 1 insertion(+)
(quoted diff) +0 -0

b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28

Quoted 9 lines

index f0a223a19c..40c7b1cc3c 100644

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1 -0
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
 FILES_${PN} = "${libdir}/libxcb.so.*"
 
 inherit autotools pkgconfig features_check
+inherit python3native
 

Could you provide a little more information about how this is breaking things?

Whilst python is used by the libxcb build, I believe it shouldn't change the
output, it doesn't in our reproducibity tests.

Adding a python-native dependency does slow the build down a bit so if we
don't need it, it would be good to avoid it.

Cheers,

Richard

#7 lore raw

It's correct because the package it depends on (xcb-proto-native) is already inherit python3native. This package's xcbgen invocation is broken because it is using a combination of files generated for native python but executed through host python (look at the site-packages version). It can't bloat our dependency tree any more as it already implicitly depends on python3-native.

Quoted 113 lines · Alexander Kanavin

As a matter of policy, yocto recipes default to using python from the host, and are only switched to native python when there host python does not come with needed 3rd party modules (which are then pulled in via -native DEPENDS), or there is some other issue that cannot be resolved with host python. Native python dependency lengthens the dependency chains, which is especially critical for -native recipes.

In this case, the problem is that libxcb in your build is too old for the latest python 3.9, so you need to either update libxcb or backport an upstream fix to make it work with 3.9 again. Your patch would not resolve the issue, as native python in latest oe-core master is also 3.9, and the failure would be the same.

Alex

Quoted 102 lines · Josh Lehan

Thanks. It's true, my local build is a little bit outdated. I'm running OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.

However, there is the same underlying problem, even if I upgraded. It looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.

As for "from fractions import gcd" becoming an error in 3.9, there was this change: https://bugs.python.org/issue39350

It appears fractions.gcd was deprecated for a while. It was moved to math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941

Quoted 86 lines · Alexander Kanavin

Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.

Alex

Quoted 80 lines · Alexander Kanavin

I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?

Alex

Quoted 74 lines · Josh Lehan via lists.openembedded.org

Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:

Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.

Traceback (most recent call last):
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py", line 3338, in <module>
from xcbgen.state import Module
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions' (/usr/lib/python3.9/fractions.py)

The above error text repeats for many of the files it tries to compile within libxcb.

In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941

Quoted 38 lines · Richard Purdie
Quoted 13 lines · Josh Lehan via lists.openembedded.org

During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.

---
 meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +0 -0
index f0a223a19c..40c7b1cc3c 100644

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28

Quoted 5 lines
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1 -0
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
 FILES_${PN} = "${libdir}/libxcb.so.*"
 
 inherit autotools pkgconfig features_check
+inherit python3native
 

Could you provide a little more information about how this is breaking things?

Whilst python is used by the libxcb build, I believe it shouldn't change the output, it doesn't in our reproducibity tests.

Adding a python-native dependency does slow the build down a bit so if we don't need it, it would be good to avoid it.

Cheers,

Richard

#8 lore raw

That seems correct, but if we force the native python 3.9 to be used, is the current libxcb/xcbgen in master compatible with it (e.g. not anymore using the removed feautre)?

Alex

Quoted 182 lines · William Kennington

It's correct because the package it depends on (xcb-proto-native) is already inherit python3native. This package's xcbgen invocation is broken because it is using a combination of files generated for native python but executed through host python (look at the site-packages version). It can't bloat our dependency tree any more as it already implicitly depends on python3-native.

Quoted 2 lines · Alexander Kanavin

As a matter of policy, yocto recipes default to using python from the

host, and are only switched to native python when there host python does not come with needed 3rd party modules (which are then pulled in via -native DEPENDS), or there is some other issue that cannot be resolved with host python. Native python dependency lengthens the dependency chains, which is especially critical for -native recipes.

Quoted 2 lines

In this case, the problem is that libxcb in your build is too old for

the latest python 3.9, so you need to either update libxcb or backport an upstream fix to make it work with 3.9 again. Your patch would not resolve the issue, as native python in latest oe-core master is also 3.9, and the failure would be the same.

Quoted 6 lines

Alex

Quoted 2 lines · Josh Lehan

Thanks. It's true, my local build is a little bit outdated. I'm running

OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.

Quoted 2 lines
Quoted 2 lines

However, there is the same underlying problem, even if I upgraded. It

looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.

Quoted 2 lines
Quoted 2 lines

As for "from fractions import gcd" becoming an error in 3.9, there was

this change: https://bugs.python.org/issue39350

Quoted 2 lines
Quoted 2 lines

It appears fractions.gcd was deprecated for a while. It was moved to

math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).

Quoted 8 lines
Quoted 8 lines

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941

On Tue, Apr 13, 2021 at 1:08 PM Alexander Kanavin <

Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines

Also note: oe-core master already has 3.9, so you need to find out why

'3.8' shows up in install paths as well. Something doesn't compute here.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Alex

On Tue, 13 Apr 2021 at 22:07, Alexander Kanavin <

Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I think you need to dig a little deeper. Why 'from fractions import

gcd' is a problem with host python3.9 but not a problem with native python 3.9?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Alex

On Tue, 13 Apr 2021 at 20:58, Josh Lehan via lists.openembedded.org

Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Hi there, thanks. I'm not entirely sure what "python3native" does

exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.

Traceback (most recent call last):
File

"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py", line 3338, in <module>

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

from xcbgen.state import Module
File

"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

from xcbgen import matcher
File

"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

from xcbgen.xtypes import *
File

"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

from xcbgen.align import Alignment, AlignmentLog
File

"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'

(/usr/lib/python3.9/fractions.py)

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The above error text repeats for many of the files it tries to

compile within libxcb.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

In general, it seems the safest approach is to standardize on a

Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Josh

Josh Lehan | Software Engineer | [email protected] | +1

650-733-8941

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

On Tue, Apr 13, 2021 at 2:45 AM Richard Purdie <

Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

On Tue, 2021-04-13 at 02:24 -0700, Josh Lehan via

Quoted 9 lines · lists.openembedded.org
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Review

Quoted 9 lines

During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.

---
 meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
 1 file changed, 1 insertion(+)
(quoted diff) +0 -0

b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

index f0a223a19c..40c7b1cc3c 100644

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +0 -0
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
 FILES_${PN} = "${libdir}/libxcb.so.*"
 
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28

Quoted 3 lines
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1 -0
 inherit autotools pkgconfig features_check
+inherit python3native
 

Could you provide a little more information about how this is

breaking things?

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Whilst python is used by the libxcb build, I believe it shouldn't

change the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

output, it doesn't in our reproducibity tests.

Adding a python-native dependency does slow the build down a bit so

if we

Quoted 16 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 10 lines

don't need it, it would be good to avoid it.

Cheers,

Richard

#9 lore raw
In reply to William A. Kennington III
Quoted 7 lines · William A. Kennington III via lists.openembedded.org

It's correct because the package it depends on (xcb-proto-native) is already inherit python3native. This package's xcbgen invocation is broken because it is using a combination of files generated for native python but executed through host python (look at the site-packages version). It can't bloat our dependency tree any more as it already implicitly depends on python3-native.

I tend to agree. Although this makes me think if we have more such mix and match happening elsewhere too
in that case, would it be better to always use a captured version of python3 on native build hosts and ignore distro
version of python3 always something like buildtools-tarball should then become mandatory.

Quoted 119 lines
Quoted 113 lines · Alexander Kanavin

As a matter of policy, yocto recipes default to using python from the host, and are only switched to native python when there host python does not come with needed 3rd party modules (which are then pulled in via -native DEPENDS), or there is some other issue that cannot be resolved with host python. Native python dependency lengthens the dependency chains, which is especially critical for -native recipes.

In this case, the problem is that libxcb in your build is too old for the latest python 3.9, so you need to either update libxcb or backport an upstream fix to make it work with 3.9 again. Your patch would not resolve the issue, as native python in latest oe-core master is also 3.9, and the failure would be the same.

Alex

Quoted 102 lines · Josh Lehan

Thanks. It's true, my local build is a little bit outdated. I'm running OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.

However, there is the same underlying problem, even if I upgraded. It looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.

As for "from fractions import gcd" becoming an error in 3.9, there was this change: https://bugs.python.org/issue39350

It appears fractions.gcd was deprecated for a while. It was moved to math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941

Quoted 86 lines · Alexander Kanavin

Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.

Alex

Quoted 80 lines · Alexander Kanavin

I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?

Alex

Quoted 74 lines · Josh Lehan via lists.openembedded.org

Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:

Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path. If not, you will need to create a .pth file or define $PYTHONPATH to extend the path.
Refer to the README file in xcb/proto for more info.

Traceback (most recent call last):
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py", line 3338, in <module>
from xcbgen.state import Module
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
from xcbgen import matcher
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
from xcbgen.xtypes import *
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
from xcbgen.align import Alignment, AlignmentLog
File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions' (/usr/lib/python3.9/fractions.py)

The above error text repeats for many of the files it tries to compile within libxcb.

In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).

Josh

Josh Lehan | Software Engineer | [email protected] | +1 650-733-8941

Quoted 38 lines · Richard Purdie
Quoted 13 lines · Josh Lehan via lists.openembedded.org

During compilation, the local Python environment leaks into the Bitbake build. Adding "inherit python3native" fixes this.

---
 meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +0 -0
index f0a223a19c..40c7b1cc3c 100644

meta/recipes-graphics/xorg-lib/libxcb_1.14.bb PACKAGES_DYNAMIC = "^${PN}-.*" · line 28

Quoted 5 lines
meta/recipes-graphics/xorg-lib/libxcb_1.14.bb +1 -0
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
 FILES_${PN} = "${libdir}/libxcb.so.*"
 
 inherit autotools pkgconfig features_check
+inherit python3native
 

Could you provide a little more information about how this is breaking things?

Whilst python is used by the libxcb build, I believe it shouldn't change the output, it doesn't in our reproducibity tests.

Adding a python-native dependency does slow the build down a bit so if we don't need it, it would be good to avoid it.

Cheers,

Richard