Skip to contentSkip to conversation

python3-rpds-py: Create a symlink for cython module on musl

PATCH In review Khem Raj 3 participants 7 messages Jun 3, 2024 – Jun 5, 2024

lore
Share & tools
lore

Khem Raj

author
#0 lore raw

PATCH 1/1 python3-rpds-py: Create a symlink for cython module on musl

loader expects it to be called with 'linux-gnu' e.g rpds.cpython-312-x86_64-linux-musl.so is created with musl but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so

Lets create the symlink to please the loader.

Fixes
ImportError while importing test module '/usr/lib/python3-rpds-py/ptest/tests/test_hash_trie_map.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../python3.12/importlib/__init__.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_hash_trie_map.py:36: in <module>
from rpds import HashTrieMap
../../python3.12/site-packages/rpds/__init__.py:1: in <module>
from .rpds import *
E ModuleNotFoundError: No module named 'rpds.rpds'
ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py

---
 .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +11 -0
index f46df1115c8..b1f0e100332 100644
@@ -27,4 +27,15 @@ do_install_ptest() {
     cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/
 }
 
+do_install:append() {
+    for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so
+    do
+        fname=`basename $f`
+        lname=`echo $fname | sed 's/musl/gnu/'`
+        if [ "$fname" != "$lname" ]; then
+            mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname
+        fi
+    done
+}
+
 BBCLASSEXTEND = "native nativesdk"
#1 lore raw
Quoted 46 lines · Khem Raj via lists.openembedded.org

loader expects it to be called with 'linux-gnu' e.g rpds.cpython-312-x86_64-linux-musl.so is created with musl but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so

Lets create the symlink to please the loader.

Fixes
ImportError while importing test module '/usr/lib/python3-rpds- py/ptest/tests/test_hash_trie_map.py'.
Hint: make sure your test modules/packages have valid Python names. Traceback:
../../python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) tests/test_hash_trie_map.py:36: in <module>
    from rpds import HashTrieMap ../../python3.12/site-packages/rpds/__init__.py:1: in <module> from .rpds import *
E   ModuleNotFoundError: No module named 'rpds.rpds'
ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py

---
 .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11
+++++++++++

 1 file changed, 11 insertions(+)
meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +0 -0
index f46df1115c8..b1f0e100332 100644
@@ -27,4 +27,15 @@ do_install_ptest() {

     cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/
 }

meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +11 -0
+do_install:append() {
+    for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so
+    do
+        fname=`basename $f`
+        lname=`echo $fname | sed 's/musl/gnu/'`
+        if [ "$fname" != "$lname" ]; then
+            mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname
+        fi
+    done
+}
+
Quoted 2 lines

 BBCLASSEXTEND = "native nativesdk"

This feels like the kind of workaround that will live forever. Can we not fix the loader to use the right name?

Also, there is not even a comment above saying why we need to do this.

Cheers,

Richard

author
#2 lore raw
Quoted 38 lines · Richard Purdie
Quoted 35 lines · Khem Raj via lists.openembedded.org

loader expects it to be called with 'linux-gnu' e.g rpds.cpython-312-x86_64-linux-musl.so is created with musl but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so

Lets create the symlink to please the loader.

Fixes
ImportError while importing test module '/usr/lib/python3-rpds- py/ptest/tests/test_hash_trie_map.py'.
Hint: make sure your test modules/packages have valid Python names. Traceback:
../../python3.12/importlib/__init__.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_hash_trie_map.py:36: in <module>
from rpds import HashTrieMap
../../python3.12/site-packages/rpds/__init__.py:1: in <module>
from .rpds import *
E ModuleNotFoundError: No module named 'rpds.rpds'
ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py

---
 .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11
+++++++++++
 1 file changed, 11 insertions(+)
meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +0 -0
index f46df1115c8..b1f0e100332 100644
@@ -27,4 +27,15 @@ do_install_ptest() {
     cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/
 }
 
Quoted 16 lines

meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb do_install_ptest() { · line 41

Quoted 13 lines
meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +11 -0
+do_install:append() {
+    for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so
+    do
+        fname=`basename $f`
+        lname=`echo $fname | sed 's/musl/gnu/'`
+        if [ "$fname" != "$lname" ]; then
+            mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname
+        fi
+    done
+}
+
 BBCLASSEXTEND = "native nativesdk"

This feels like the kind of workaround that will live forever. Can we not fix the loader to use the right name?

OE's python interpreter is cross-built and we end up using OSABI which is detected during build 'linux-gnu', this seems to work fine for most of python modules since they get the OSABI from the sysconfigdata so it all works even though the OSABI is called linux-gnu, however some modules like this one which are using maturin to build, seems to get the OSABI equivalent differently, I am no expert on maturin but maybe someone knows how it happens. This problem is seen with several python modules built with OE, see

https://github.com/meta-homeassistant/meta-homeassistant/issues/89
https://github.com/pypa/auditwheel/issues/349

also see how it is fixed in python3-pydantic-core on meta-python
https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pydantic-core_2.16.3.bb#n38

Hopefully musllinux ABI tag will be sorted in pypa and this will work better however, we also need to see how we are encoding OSABI in python builds

e.g. on qemux86-64 musl target I get

root@qemux86-64:~# python3
Python 3.12.3 (main, Apr  9 2024, 08:09:14) [Clang 19.0.0
(/home/kraj/work/llvm-project 4152c5f5ae88d8d5fe99d20b525b35f14db2 on
linux
Type "help", "copyright", "credits" or "license" for more information.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

import sysconfig
sysconfig.get_config_var('SOABI')

'cpython-312-x86_64-linux-gnu'

Quoted 3 lines

Also, there is not even a comment above saying why we need to do this.

I added the log showing how it does not work, I thought that is showing how it's not working at all when using musl. If you have better idea what I should add to commit msg to make it better, let me know I will send a v2.

Quoted 3 lines

Cheers,

Richard

#3 lore raw

I agree with RP, please get to the source of the mismatch. It’s a really unpleasant workaround.

Alex

Quoted 112 lines · Khem Raj via lists.openembedded.org
Quoted 38 lines · Richard Purdie
Quoted 35 lines · Khem Raj via lists.openembedded.org

loader expects it to be called with 'linux-gnu' e.g rpds.cpython-312-x86_64-linux-musl.so is created with musl but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so

Lets create the symlink to please the loader.

Fixes
ImportError while importing test module '/usr/lib/python3-rpds- py/ptest/tests/test_hash_trie_map.py'.
Hint: make sure your test modules/packages have valid Python names. Traceback:
../../python3.12/importlib/__init__.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_hash_trie_map.py:36: in <module>
from rpds import HashTrieMap
../../python3.12/site-packages/rpds/__init__.py:1: in <module>
from .rpds import *
E ModuleNotFoundError: No module named 'rpds.rpds'
ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py

---
 .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11
+++++++++++
 1 file changed, 11 insertions(+)
meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +0 -0
index f46df1115c8..b1f0e100332 100644
@@ -27,4 +27,15 @@ do_install_ptest() {
     cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/
 }
 
Quoted 16 lines

meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb do_install_ptest() { · line 41

Quoted 13 lines
meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +11 -0
+do_install:append() {
+    for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so
+    do
+        fname=`basename $f`
+        lname=`echo $fname | sed 's/musl/gnu/'`
+        if [ "$fname" != "$lname" ]; then
+            mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname
+        fi
+    done
+}
+
 BBCLASSEXTEND = "native nativesdk"

This feels like the kind of workaround that will live forever. Can we not fix the loader to use the right name?

OE's python interpreter is cross-built and we end up using OSABI which is detected during build 'linux-gnu', this seems to work fine for most of python modules since they get the OSABI from the sysconfigdata so it all works even though the OSABI is called linux-gnu, however some modules like this one which are using maturin to build, seems to get the OSABI equivalent differently, I am no expert on maturin but maybe someone knows how it happens. This problem is seen with several python modules built with OE, see

https://github.com/meta-homeassistant/meta-homeassistant/issues/89
https://github.com/pypa/auditwheel/issues/349

also see how it is fixed in python3-pydantic-core on meta-python

https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pydantic-core_2.16.3.bb#n38

Hopefully musllinux ABI tag will be sorted in pypa and this will work better
however, we also need to see how we are encoding OSABI in python builds

e.g. on qemux86-64 musl target I get

root@qemux86-64:~# python3
Python 3.12.3 (main, Apr  9 2024, 08:09:14) [Clang 19.0.0
(/home/kraj/work/llvm-project 4152c5f5ae88d8d5fe99d20b525b35f14db2 on
linux
Type "help", "copyright", "credits" or "license" for more information.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

import sysconfig
sysconfig.get_config_var('SOABI')

'cpython-312-x86_64-linux-gnu'

Quoted 3 lines

Also, there is not even a comment above saying why we need to do this.

I added the log showing how it does not work, I thought that is showing how it's not working at all when using musl. If you have better idea what I should
add to commit msg to make it better, let me know I will send a v2.

Quoted 3 lines

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#200275):
https://lists.openembedded.org/g/openembedded-core/message/200275
Mute This Topic: https://lists.openembedded.org/mt/106465307/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

author
#4 lore raw
In reply to Khem Raj
Quoted 78 lines · Khem Raj
Quoted 38 lines · Richard Purdie
Quoted 35 lines · Khem Raj via lists.openembedded.org

loader expects it to be called with 'linux-gnu' e.g rpds.cpython-312-x86_64-linux-musl.so is created with musl but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so

Lets create the symlink to please the loader.

Fixes
ImportError while importing test module '/usr/lib/python3-rpds- py/ptest/tests/test_hash_trie_map.py'.
Hint: make sure your test modules/packages have valid Python names. Traceback:
../../python3.12/importlib/__init__.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_hash_trie_map.py:36: in <module>
from rpds import HashTrieMap
../../python3.12/site-packages/rpds/__init__.py:1: in <module>
from .rpds import *
E ModuleNotFoundError: No module named 'rpds.rpds'
ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py

---
 .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11
+++++++++++
 1 file changed, 11 insertions(+)
meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +0 -0
index f46df1115c8..b1f0e100332 100644
@@ -27,4 +27,15 @@ do_install_ptest() {
     cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/
 }
 
Quoted 16 lines

meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb do_install_ptest() { · line 41

Quoted 13 lines
meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +11 -0
+do_install:append() {
+    for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so
+    do
+        fname=`basename $f`
+        lname=`echo $fname | sed 's/musl/gnu/'`
+        if [ "$fname" != "$lname" ]; then
+            mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname
+        fi
+    done
+}
+
 BBCLASSEXTEND = "native nativesdk"

This feels like the kind of workaround that will live forever. Can we not fix the loader to use the right name?

OE's python interpreter is cross-built and we end up using OSABI which is detected during build 'linux-gnu', this seems to work fine for most of python modules since they get the OSABI from the sysconfigdata so it all works even though the OSABI is called linux-gnu, however some modules like this one which are using maturin to build, seems to get the OSABI equivalent differently, I am no expert on maturin but maybe someone knows how it happens. This problem is seen with several python modules built with OE, see

https://github.com/meta-homeassistant/meta-homeassistant/issues/89
https://github.com/pypa/auditwheel/issues/349

also see how it is fixed in python3-pydantic-core on meta-python
https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pydantic-core_2.16.3.bb#n38

Hopefully musllinux ABI tag will be sorted in pypa and this will work better however, we also need to see how we are encoding OSABI in python builds

e.g. on qemux86-64 musl target I get

Quoted 9 lines
root@qemux86-64:~# python3
Python 3.12.3 (main, Apr  9 2024, 08:09:14) [Clang 19.0.0
(/home/kraj/work/llvm-project 4152c5f5ae88d8d5fe99d20b525b35f14db2 on
linux
Type "help", "copyright", "credits" or "license" for more information.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

import sysconfig
sysconfig.get_config_var('SOABI')

'cpython-312-x86_64-linux-gnu'

I looked further in pyo and maturin infrastructure and it is using ustc --target=<tuple> --print cfg to compute EXT_SUFFIX equivalent value which is then used to build the C extension (if any) inside the wheel, thats why we get cpython-312-x86_64-linux-musl for suffix, what pyo/maturin is doing is right, however how OE does it in python itself seems to be wrong to me, its synthesizing the OSABI incorrectly.
where 'environment' value is not right. However, this is same everywhere so it works, so long as the modules are built using setuptools.
since SOABI is part of "_sysconfigdata" and its an OE generated file, Perhaps it can consider the musl/gnu difference and emit SOABI accordingly. Something like below

https://snips.sh/f/L6QspcdxjL

might be able to fix the python interpreter's logic.

Quoted 11 lines
Quoted 3 lines

Also, there is not even a comment above saying why we need to do this.

I added the log showing how it does not work, I thought that is showing how it's not working at all when using musl. If you have better idea what I should add to commit msg to make it better, let me know I will send a v2.

Quoted 3 lines

Cheers,

Richard

author
#5 lore raw
Quoted 104 lines · Khem Raj
Quoted 78 lines · Khem Raj
Quoted 38 lines · Richard Purdie
Quoted 35 lines · Khem Raj via lists.openembedded.org

loader expects it to be called with 'linux-gnu' e.g rpds.cpython-312-x86_64-linux-musl.so is created with musl but loader expects it to called rpds.cpython-312-x86_64-linux-gnu.so

Lets create the symlink to please the loader.

Fixes
ImportError while importing test module '/usr/lib/python3-rpds- py/ptest/tests/test_hash_trie_map.py'.
Hint: make sure your test modules/packages have valid Python names. Traceback:
../../python3.12/importlib/__init__.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_hash_trie_map.py:36: in <module>
from rpds import HashTrieMap
../../python3.12/site-packages/rpds/__init__.py:1: in <module>
from .rpds import *
E ModuleNotFoundError: No module named 'rpds.rpds'
ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py

---
 .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11
+++++++++++
 1 file changed, 11 insertions(+)
meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +0 -0
index f46df1115c8..b1f0e100332 100644
@@ -27,4 +27,15 @@ do_install_ptest() {
     cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/
 }
 
Quoted 16 lines

meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb do_install_ptest() { · line 41

Quoted 13 lines
meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +11 -0
+do_install:append() {
+    for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so
+    do
+        fname=`basename $f`
+        lname=`echo $fname | sed 's/musl/gnu/'`
+        if [ "$fname" != "$lname" ]; then
+            mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname
+        fi
+    done
+}
+
 BBCLASSEXTEND = "native nativesdk"

This feels like the kind of workaround that will live forever. Can we not fix the loader to use the right name?

OE's python interpreter is cross-built and we end up using OSABI which is detected during build 'linux-gnu', this seems to work fine for most of python modules since they get the OSABI from the sysconfigdata so it all works even though the OSABI is called linux-gnu, however some modules like this one which are using maturin to build, seems to get the OSABI equivalent differently, I am no expert on maturin but maybe someone knows how it happens. This problem is seen with several python modules built with OE, see

https://github.com/meta-homeassistant/meta-homeassistant/issues/89
https://github.com/pypa/auditwheel/issues/349

also see how it is fixed in python3-pydantic-core on meta-python
https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pydantic-core_2.16.3.bb#n38

Hopefully musllinux ABI tag will be sorted in pypa and this will work better however, we also need to see how we are encoding OSABI in python builds

e.g. on qemux86-64 musl target I get

Quoted 9 lines
root@qemux86-64:~# python3
Python 3.12.3 (main, Apr  9 2024, 08:09:14) [Clang 19.0.0
(/home/kraj/work/llvm-project 4152c5f5ae88d8d5fe99d20b525b35f14db2 on
linux
Type "help", "copyright", "credits" or "license" for more information.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

import sysconfig
sysconfig.get_config_var('SOABI')

'cpython-312-x86_64-linux-gnu'

I looked further in pyo and maturin infrastructure and it is using ustc --target=<tuple> --print cfg to compute EXT_SUFFIX equivalent value which is then used to build the C extension (if any) inside the wheel, thats why we get cpython-312-x86_64-linux-musl for suffix, what pyo/maturin is doing is right, however how OE does it in python itself seems to be wrong to me, its synthesizing the OSABI incorrectly.
where 'environment' value is not right. However, this is same everywhere so it works, so long as the modules are built using setuptools.
since SOABI is part of "_sysconfigdata" and its an OE generated file, Perhaps it can consider the musl/gnu difference and emit SOABI accordingly. Something like below

Quoted 3 lines

https://snips.sh/f/L6QspcdxjL

might be able to fix the python interpreter's logic.

To close this thread. I have sent an alternative patch which fixes it in python3 itself.
https://lore.kernel.org/openembedded-core/[email protected]/T/#t

Quoted 12 lines
Quoted 12 lines
Quoted 4 lines

Also, there is not even a comment above saying why we need to do this.

I added the log showing how it does not work, I thought that is showing how it's not working at all when using musl. If you have better idea what I should add to commit msg to make it better, let me know I will send a v2.

Quoted 3 lines

Cheers,

Richard

#6 lore raw

Thanks for getting to the root of the issue.

Alex

Quoted 150 lines · Khem Raj via lists.openembedded.org
Quoted 7 lines · Khem Raj
Quoted 5 lines · Khem Raj
Quoted 2 lines · Richard Purdie

On Mon, 2024-06-03 at 09:25 -0700, Khem Raj via

lists.openembedded.org

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines · wrote:

loader expects it to be called with 'linux-gnu' e.g rpds.cpython-312-x86_64-linux-musl.so is created with musl but loader expects it to called

rpds.cpython-312-x86_64-linux-gnu.so

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines

Lets create the symlink to please the loader.

Fixes
ImportError while importing test module '/usr/lib/python3-rpds- py/ptest/tests/test_hash_trie_map.py'.
Hint: make sure your test modules/packages have valid Python names. Traceback:
../../python3.12/importlib/__init__.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_hash_trie_map.py:36: in <module>
from rpds import HashTrieMap
../../python3.12/site-packages/rpds/__init__.py:1: in <module>
from .rpds import *
E ModuleNotFoundError: No module named 'rpds.rpds'
ERROR: tests/test_hash_trie_map.py:tests/test_hash_trie_map.py

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Review

Quoted 7 lines
---
 .../recipes-devtools/python/python3-rpds-py_0.18.1.bb | 11
+++++++++++
 1 file changed, 11 insertions(+)
(quoted diff) +0 -0

python3-rpds-py_0.18.1.bb

Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 11 lines

b/meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb index f46df1115c8..b1f0e100332 100644

meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +2 -0
@@ -27,4 +27,15 @@ do_install_ptest() {
     cp -rf ${S}/tests/* ${D}${PTEST_PATH}/tests/
 }
 
+do_install:append() {
+    for f in ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/rpds.cpython*.so

meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb do_install_ptest() { · line 41

Quoted 10 lines
meta/recipes-devtools/python/python3-rpds-py_0.18.1.bb +9 -0
+    do
+        fname=`basename $f`
+        lname=`echo $fname | sed 's/musl/gnu/'`
+        if [ "$fname" != "$lname" ]; then
+            mv $f ${D}/${PYTHON_SITEPACKAGES_DIR}/rpds/$lname
+        fi
+    done
+}
+
 BBCLASSEXTEND = "native nativesdk"

This feels like the kind of workaround that will live forever. Can we not fix the loader to use the right name?

Quoted 2 lines
Quoted 2 lines

OE's python interpreter is cross-built and we end up using OSABI which is detected during build 'linux-gnu', this seems to work fine for most

of

Quoted 2 lines
Quoted 2 lines

python modules since they get the OSABI from the sysconfigdata so it all works even though the OSABI is called linux-gnu, however some

modules

Quoted 2 lines
Quoted 2 lines

like this one which are using maturin to build, seems to get the OSABI equivalent differently, I am no expert on maturin but maybe someone

knows

Quoted 1 line
Quoted 1 line

how it happens. This problem is seen with several python modules built

with

Quoted 7 lines
Quoted 7 lines

OE, see

https://github.com/meta-homeassistant/meta-homeassistant/issues/89
https://github.com/pypa/auditwheel/issues/349

also see how it is fixed in python3-pydantic-core on meta-python

https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pydantic-core_2.16.3.bb#n38

Quoted 2 lines
Quoted 2 lines

Hopefully musllinux ABI tag will be sorted in pypa and this will work

better

Quoted 28 lines
Quoted 4 lines

however, we also need to see how we are encoding OSABI in python builds

e.g. on qemux86-64 musl target I get

Quoted 9 lines
root@qemux86-64:~# python3
Python 3.12.3 (main, Apr  9 2024, 08:09:14) [Clang 19.0.0
(/home/kraj/work/llvm-project 4152c5f5ae88d8d5fe99d20b525b35f14db2 on
linux
Type "help", "copyright", "credits" or "license" for more information.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

import sysconfig
sysconfig.get_config_var('SOABI')

'cpython-312-x86_64-linux-gnu'

I looked further in pyo and maturin infrastructure and it is using ustc --target=<tuple> --print cfg to compute EXT_SUFFIX equivalent value which is then used to build the C extension (if any) inside the wheel, thats why we get cpython-312-x86_64-linux-musl for suffix, what pyo/maturin is doing is right, however how OE does it in python itself seems to be wrong to me, its synthesizing the OSABI incorrectly.
where 'environment' value is not right. However, this is same everywhere so it works, so long as the modules are built using setuptools.
since SOABI is part of "_sysconfigdata" and its an OE generated file, Perhaps it can consider the musl/gnu difference and emit SOABI accordingly. Something like below

Quoted 3 lines

https://snips.sh/f/L6QspcdxjL

might be able to fix the python interpreter's logic.

To close this thread. I have sent an alternative patch which fixes it in python3 itself.

https://lore.kernel.org/openembedded-core/[email protected]/T/#t

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Also, there is not even a comment above saying why we need to do

this.

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

I added the log showing how it does not work, I thought that is

showing how

Quoted 1 line
Quoted 1 line

it's not working at all when using musl. If you have better idea what

I should

Quoted 5 lines
Quoted 5 lines

add to commit msg to make it better, let me know I will send a v2.

Quoted 3 lines

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#200355):
https://lists.openembedded.org/g/openembedded-core/message/200355
Mute This Topic: https://lists.openembedded.org/mt/106465307/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-