Skip to contentSkip to conversation

package/kmod: fix build with python 3.8

PATCH New James Hilliard 2 participants 5 messages Oct 20, 2019 – Oct 22, 2019 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package/kmod: fix build with python 3.8

Fixes:
- http://autobuild.buildroot.net/results/71bf937339705a520e047d12c146229ec42a76f2

Details:
- https://bugs.python.org/issue36721

---
 package/kmod/kmod.mk | 4 ++++
 1 file changed, 4 insertions(+)
package/kmod/kmod.mk +4 -0
index a5bcf2f2d6..a63fe99d91 100644
@@ -45,6 +45,10 @@ endif
 ifeq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),y)
 KMOD_DEPENDENCIES += $(if $(BR2_PACKAGE_PYTHON),python,python3)
 KMOD_CONF_OPTS += --enable-python
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+KMOD_CONF_ENV += \
+	PYTHON_LIBS="`$(STAGING_DIR)/usr/bin/python3-config --libs --embed`"
+endif
 endif
 
 ifeq ($(BR2_PACKAGE_KMOD_TOOLS),y)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/1 package/kmod: fix build with python 3.8

Quoted 5 lines · On Sun, 20 Oct 2019 13:38:18 -0600 James Hilliard <[email protected]>…

Fixes:
- http://autobuild.buildroot.net/results/71bf937339705a520e047d12c146229ec42a76f2

Details:
- https://bugs.python.org/issue36721

Reading this, I am not sure your fix is correct (though I am not sure what the correct fix is).

Indeed, in this bug report, Victor Stinner explains:

"""
There are two use cases for libpython:

* Build a C extension and load it in Python: use case called "pyext" by waf
* Embed Python into an application: use case called "pyembed" by waf """

The first case should not require -lpython3.8, and therefore python-config no longer returns -lpython3.8, unless you pass --embed (which means you are in the second case: you embed Python into an application).

But kmod is only providing Python bindings: it's a C extension to be loaded in Python, so we are in the first case, and we shouldn't need --embed.

What do you think ?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH 1/1 package/kmod: fix build with python 3.8

Quoted 4 lines · Thomas Petazzoni

On Sun, 20 Oct 2019 13:38:18 -0600
James Hilliard <[email protected]> wrote:

Quoted 8 lines
Quoted 5 lines

Fixes:
- http://autobuild.buildroot.net/results/71bf937339705a520e047d12c146229ec42a76f2

Details:
- https://bugs.python.org/issue36721

Reading this, I am not sure your fix is correct (though I am not sure what the correct fix is).

Yeah, not entirely sure but I saw other projects use this same fix effectively.

Quoted 15 lines

Indeed, in this bug report, Victor Stinner explains:

"""
There are two use cases for libpython:

* Build a C extension and load it in Python: use case called "pyext" by waf
* Embed Python into an application: use case called "pyembed" by waf """

The first case should not require -lpython3.8, and therefore python-config no longer returns -lpython3.8, unless you pass --embed (which means you are in the second case: you embed Python into an application).

Quoted 5 lines

But kmod is only providing Python bindings: it's a C extension to be loaded in Python, so we are in the first case, and we shouldn't need --embed.

What do you think ?

Yeah, it's not very clear, this fix is the same as these essentially however:
https://gitlab.freedesktop.org/gstreamer/gst-python/merge_requests/14/diffs
https://gitlab.com/ita1024/waf/merge_requests/2236/diffs

Quoted 6 lines

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#3 lore raw

PATCH 1/1 package/kmod: fix build with python 3.8

Quoted 3 lines · On Mon, 21 Oct 2019 21:12:44 +0200 James Hilliard <[email protected]>…
Quoted 2 lines

Reading this, I am not sure your fix is correct (though I am not sure what the correct fix is).

Yeah, not entirely sure but I saw other projects use this same fix effectively.

Yes, it "works", but it's quite weird, because the whole point of Victor Stinner's work was to make sure Python extensions were not linked with libpython.

Quoted 3 lines
Quoted 1 line

What do you think ?

Yeah, it's not very clear, this fix is the same as these essentially however:
https://gitlab.freedesktop.org/gstreamer/gst-python/merge_requests/14/diffs

This is indeed a Python binding.

Quoted 1 line

https://gitlab.com/ita1024/waf/merge_requests/2236/diffs

This is a generic build system, and it only adds --embed when the "pyembed" flag is used. It could simply be that this flag is used for applications/libraries than embed Python, and not for Python extensions. Indeed, the original entry in the Python issue tracker mentioned that waf had "pyext" for Python extensions and "pyembed" for apps/libs embedding Python.

It turns out that Victor Stinner (the author of the changes) is a former colleague of mine from university times. I'll try to get some feedback from him about this.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

PATCH 1/1 package/kmod: fix build with python 3.8

In reply to James Hilliard

Hello,

Review

Quoted 4 lines · On Sun, 20 Oct 2019 13:38:18 -0600 James Hilliard <[email protected]>…
(quoted diff) +4 -0
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+KMOD_CONF_ENV += \
+	PYTHON_LIBS="`$(STAGING_DIR)/usr/bin/python3-config --libs --embed`"
+endif

For kmod, the problem is the same as libselinux, just not the same option is used. libselinux is passing "-z defs", while kmod is passing --no-undefined, but they are exactly the same option.

So, for kmod, I believe the proper fix is:

Makefile.am +1 -1
index c5c2f06..8e9c90d 100644
@@ -173,7 +173,7 @@ CPYTHON_MODULE_CFLAGS = \
        $(AM_CFLAGS) -DCPYTHON_COMPILING_IN_PYPY=0 \
        $(PYTHON_NOWARN) $(PYTHON_CFLAGS) \
        -fvisibility=default
-CPYTHON_MODULE_LDFLAGS = $(AM_LDFLAGS) -module -avoid-version -shared
+CPYTHON_MODULE_LDFLAGS = $(AM_LDFLAGS) -module -avoid-version -shared -Wl,-z,undefs
 
 if BUILD_PYTHON
 pkgpyexec_LTLIBRARIES = \

Indeed, kmod globally uses -Wl,--no-undefined when linking, but specifically for building the Python extension, we want to disable that and allow undefined symbols to exist, so we override it with -Wl,-z,undefs.

Best regards,

Thomas Petazzoni

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com