Skip to contentSkip to conversation

package/dbus-python: Fix build with python 3.8

PATCH Superseded Grzegorz Blach 3 participants 6 messages Oct 23, 2019 – Oct 25, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package/dbus-python: Fix build with python 3.8

test/import-repeatedly.c uses an embedded python interpreter and PYTHON_EXTRA_LIBS is used only for building this test case, so set PYTHON_EXTRA_LIBS with python3-config --libs --embed to build this test case and don't link the module with libpython3.so.

Fixes:
- http://autobuild.buildroot.org/results/b30/b308eeb5c5d95ab9f1dbfc19f9183f2ba3ba0ce3/
- http://autobuild.buildroot.org/results/0dd/0dd9203f859b97ee5a3b6358644c26f8ab784ed8/
and many similar failures.

---
Changes v1 -> v2:
- Set PYTHON_EXTRA_LIBS with python3-config --libs --embed instead of appending --embed to PYTHON_LIBS

---
 package/dbus-python/dbus-python.mk | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
package/dbus-python/dbus-python.mk +4 -2
index eaf4e5f483..d48208f1d1 100644
@@ -35,14 +35,16 @@ DBUS_PYTHON_DEPENDENCIES += python3 host-python3
 DBUS_PYTHON_CONF_ENV += \
 	PYTHON=$(HOST_DIR)/bin/python3 \
 	PYTHON_INCLUDES="`$(STAGING_DIR)/usr/bin/python3-config --includes`" \
-	PYTHON_LIBS="`$(STAGING_DIR)/usr/bin/python3-config --ldflags`"
+	PYTHON_LIBS="`$(STAGING_DIR)/usr/bin/python3-config --ldflags`" \
+	PYTHON_EXTRA_LIBS="`$(STAGING_DIR)/usr/bin/python3-config --libs --embed`"
 
 HOST_DBUS_PYTHON_DEPENDENCIES += host-python3
 
 HOST_DBUS_PYTHON_CONF_ENV += \
 	PYTHON=$(HOST_DIR)/bin/python3 \
 	PYTHON_INCLUDES="`$(HOST_DIR)/usr/bin/python3-config --includes`" \
-	PYTHON_LIBS="`$(HOST_DIR)/usr/bin/python3-config --ldflags`"
+	PYTHON_LIBS="`$(HOST_DIR)/usr/bin/python3-config --ldflags`" \
+	PYTHON_EXTRA_LIBS="`$(HOST_DIR)/usr/bin/python3-config --libs --embed`"
 endif
 
 $(eval $(autotools-package))
Signature · 1 line
2.23.0
#1 lore raw

PATCH 1/1 package/dbus-python: Fix build with python 3.8

Quoted 10 lines · On Wed, 23 Oct 2019 21:19:21 +0000 Grzegorz Blach <[email protected]> wrote:

test/import-repeatedly.c uses an embedded python interpreter and PYTHON_EXTRA_LIBS is used only for building this test case, so set PYTHON_EXTRA_LIBS with python3-config --libs --embed to build this test case and don't link the module with libpython3.so.

Fixes:
- http://autobuild.buildroot.org/results/b30/b308eeb5c5d95ab9f1dbfc19f9183f2ba3ba0ce3/
- http://autobuild.buildroot.org/results/0dd/0dd9203f859b97ee5a3b6358644c26f8ab784ed8/
and many similar failures.

Quoted 5 lines

---
Changes v1 -> v2:
- Set PYTHON_EXTRA_LIBS with python3-config --libs --embed instead of appending --embed to PYTHON_LIBS

This doesn't change the fix, which I think still isn't correct (but I haven't checked in details). A Python extension should not need the --embed option.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH 1/1 package/dbus-python: Fix build with python 3.8

Quoted 19 lines · On Thu, 24 Oct 2019 10:05:34 +0200 Thomas Petazzoni…
Quoted 10 lines · On Wed, 23 Oct 2019 21:19:21 +0000 Grzegorz Blach <[email protected]> wrote:

test/import-repeatedly.c uses an embedded python interpreter and PYTHON_EXTRA_LIBS is used only for building this test case, so set PYTHON_EXTRA_LIBS with python3-config --libs --embed to build this test case and don't link the module with libpython3.so.

Fixes:
- http://autobuild.buildroot.org/results/b30/b308eeb5c5d95ab9f1dbfc19f9183f2ba3ba0ce3/
- http://autobuild.buildroot.org/results/0dd/0dd9203f859b97ee5a3b6358644c26f8ab784ed8/
and many similar failures.

Quoted 5 lines

---
Changes v1 -> v2:
- Set PYTHON_EXTRA_LIBS with python3-config --libs --embed instead of appending --embed to PYTHON_LIBS

Quoted 3 lines

This doesn't change the fix, which I think still isn't correct (but I haven't checked in details). A Python extension should not need the --embed option.

And I replied too fast, without reading your commit log. Reading it, it makes more sense. However, I think it should be fixed in the dbus-python build system itself, so that the fix can be upstreamed. Could you have a look at that ?

Of course, it's not trivial, because you need to pass --embed for Python >= 3.8, but not for older versions of Python.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw

PATCH 1/1 package/dbus-python: Fix build with python 3.8

Quoted 26 lines · Thomas Petazzoni
Quoted 19 lines · On Thu, 24 Oct 2019 10:05:34 +0200 Thomas Petazzoni…
Quoted 10 lines · On Wed, 23 Oct 2019 21:19:21 +0000 Grzegorz Blach <[email protected]> wrote:

test/import-repeatedly.c uses an embedded python interpreter and PYTHON_EXTRA_LIBS is used only for building this test case, so set PYTHON_EXTRA_LIBS with python3-config --libs --embed to build this test case and don't link the module with libpython3.so.

Fixes:
- http://autobuild.buildroot.org/results/b30/b308eeb5c5d95ab9f1dbfc19f9183f2ba3ba0ce3/
- http://autobuild.buildroot.org/results/0dd/0dd9203f859b97ee5a3b6358644c26f8ab784ed8/
and many similar failures.

Quoted 5 lines

---
Changes v1 -> v2:
- Set PYTHON_EXTRA_LIBS with python3-config --libs --embed instead of appending --embed to PYTHON_LIBS

Quoted 3 lines

This doesn't change the fix, which I think still isn't correct (but I haven't checked in details). A Python extension should not need the --embed option.

Quoted 7 lines

And I replied too fast, without reading your commit log. Reading it, it makes more sense. However, I think it should be fixed in the dbus-python build system itself, so that the fix can be upstreamed. Could you have a look at that ?

Of course, it's not trivial, because you need to pass --embed for Python >= 3.8, but not for older versions of Python.

Actually, it looks like this should be retrieved from sysconfig. So maybe it's
just a matter of setting the proper environment variables like we do in pkg-python? In other words, I think the entire logic in dbus-python.mk could be replaced with:

DBUS_PYTHON_CONF_ENV = $(PKG_PYTHON_SETUPTOOLS_ENV) HOST_DBUS_PYTHON_CONF_ENV = $(HOST_PKG_PYTHON_SETUPTOOLS_ENV)

We may still need to set PYTHON=$(HOST_DIR)/bin/python[23] as well.

Regards,
Arnout

#4 lore raw

PATCH v3 1/1 package/dbus-python: Fix build with python 3.8

In reply to Grzegorz Blach

Python 3.8 doesn't allow python modules to be linked with libpythonX.Y.so, but applications that embed a python interpreter still need to be linked with libpythonX.Y.so. As a solution -lpythonX.Y need to be moved from PYTHON_LIBS to PYTHON_EXTRA_LIBS, but only for python 3.8 or newer.

---
Changes v2 -> v3:
- Patch m4/ax_python_devel.m4 instead of setting PYTHON_EXTRA_LIBS

Changes v1 -> v2:
- Set PYTHON_EXTRA_LIBS with python3-config --libs --embed instead of appending --embed to PYTHON_LIBS

---
 .../0001-Fix-build-with-python-3.8.patch      | 95 +++++++++++++++++++
 package/dbus-python/dbus-python.mk            |  4 +
 2 files changed, 99 insertions(+)
 create mode 100644 package/dbus-python/0001-Fix-build-with-python-3.8.patch
package/dbus-python/0001-Fix-build-with-python-3.8.patch new file +95 -0
new file mode 100644
index 0000000000..db4a0d0b10
@@ -0,0 +1,95 @@
+From 1d7695681f1db3f6e3b487abbffc9665f1422bf4 Mon Sep 17 00:00:00 2001
+From: Grzegorz Blach <[email protected]>
+Date: Thu, 24 Oct 2019 15:42:55 +0000
+Subject: [PATCH] Fix build with python 3.8
+
+Python 3.8 doesn't allow python modules to be linked with
+libpythonX.Y.so, but applications that embed a python interpreter
+still need to be linked with libpythonX.Y.so. As a solution
+-lpythonX.Y need to be moved from PYTHON_LIBS to PYTHON_EXTRA_LIBS,
+but only for python 3.8 or newer.
+
+Signed-off-by: Grzegorz Blach <[email protected]>
+---
+ m4/ax_python_devel.m4 | 36 +++++++++++++++++++++++-------------
+ 1 file changed, 23 insertions(+), 13 deletions(-)
+
+diff --git a/m4/ax_python_devel.m4 b/m4/ax_python_devel.m4
+index 44dbd83..0de0c6e 100644
+--- a/m4/ax_python_devel.m4
++++ b/m4/ax_python_devel.m4
+@@ -40,6 +40,7 @@
+ #   Copyright (c) 2009 Matteo Settenvini <[email protected]>
+ #   Copyright (c) 2009 Horst Knorr <[email protected]>
+ #   Copyright (c) 2013 Daniel Mullner <[email protected]>
++#   Copyright (c) 2019 Grzegorz Blach <[email protected]>
+ #
+ #   This program is free software: you can redistribute it and/or modify it
+ #   under the terms of the GNU General Public License as published by the
+@@ -67,7 +68,7 @@
+ #   modified version of the Autoconf Macro, you may extend this special
+ #   exception to the GPL to apply to your modified version as well.
+ 
+-#serial 21
++#serial 21 + python 3.8 fixes
+ 
+ AU_ALIAS([AC_PYTHON_DEVEL], [AX_PYTHON_DEVEL])
+ AC_DEFUN([AX_PYTHON_DEVEL],[
+@@ -198,6 +199,11 @@ EOD`
+ 		AC_DEFINE_UNQUOTED([HAVE_PYTHON], ["$ac_python_version"],
+                                    [If available, contains the Python version number currently in use.])
+ 
++		# Check if we should omit -lpython in PYTHON_LIBS
++		ac_omit_python_library=`$PYTHON -c "import sys; \
++			ver = sys.version.split ()[[0]]; \
++			print (ver >= '3.8.0')"`
++
+ 		# First, the library directory:
+ 		ac_python_libdir=`cat<<EOD | $PYTHON -
+ 
+@@ -223,18 +229,22 @@ EOD`
+ 		# credits goes to momjian, I think. I'd like to put the right name
+ 		# in the credits, if someone can point me in the right direction... ?
+ 		#
+-		if test -n "$ac_python_libdir" -a -n "$ac_python_library"
+-		then
+-			# use the official shared library
+-			ac_python_library=`echo "$ac_python_library" | sed "s/^lib//"`
+-			PYTHON_LIBS="-L$ac_python_libdir -l$ac_python_library"
++		if test "$ac_omit_python_library" = "True"; then
++			PYTHON_LIBS="-L$ac_python_libdir"
+ 		else
+-			# old way: use libpython from python_configdir
+-			ac_python_libdir=`$PYTHON -c \
+-			  "from distutils.sysconfig import get_python_lib as f; \
+-			  import os; \
+-			  print (os.path.join(f(plat_specific=1, standard_lib=1), 'config'));"`
+-			PYTHON_LIBS="-L$ac_python_libdir -lpython$ac_python_version"
++			if test -n "$ac_python_libdir" -a -n "$ac_python_library"
++			then
++				# use the official shared library
++				ac_python_library=`echo "$ac_python_library" | sed "s/^lib//"`
++				PYTHON_LIBS="-L$ac_python_libdir -l$ac_python_library"
++			else
++				# old way: use libpython from python_configdir
++				ac_python_libdir=`$PYTHON -c \
++				  "from distutils.sysconfig import get_python_lib as f; \
++				  import os; \
++				  print (os.path.join(f(plat_specific=1, standard_lib=1), 'config'));"`
++				PYTHON_LIBS="-L$ac_python_libdir -lpython$ac_python_version"
++			fi
+ 		fi
+ 
+ 		if test -z "PYTHON_LIBS"; then
+@@ -265,7 +275,7 @@ EOD`
+ 	if test -z "$PYTHON_EXTRA_LIBS"; then
+ 	   PYTHON_EXTRA_LIBS=`$PYTHON -c "import distutils.sysconfig; \
+                 conf = distutils.sysconfig.get_config_var; \
+-                print (conf('LIBS') + ' ' + conf('SYSLIBS'))"`
++                print ('-lpython' + (conf('LDVERSION') or conf('VERSION')) + ' ' + conf('LIBS') + ' ' + conf('SYSLIBS'))"`
+ 	fi
+ 	AC_MSG_RESULT([$PYTHON_EXTRA_LIBS])
+ 	AC_SUBST(PYTHON_EXTRA_LIBS)
+-- 
+2.23.0
+
package/dbus-python/dbus-python.mk +4 -0
index eaf4e5f483..201c7130ec 100644
@@ -15,6 +15,10 @@ DBUS_PYTHON_CONF_OPTS = --disable-html-docs --disable-api-docs
 HOST_DBUS_PYTHON_DEPENDENCIES = host-dbus-glib
 HOST_DBUS_PYTHON_CONF_OPTS = --disable-html-docs --disable-api-docs
 
+# We patch m4/ax_python_devel.m4
+DBUS_PYTHON_AUTORECONF = YES
+HOST_DBUS_PYTHON_AUTORECONF = YES
+
 ifeq ($(BR2_PACKAGE_PYTHON),y)
 DBUS_PYTHON_DEPENDENCIES += python host-python
 
Signature · 1 line
2.23.0
#5 lore raw

PATCH 1/1 package/dbus-python: Fix build with python 3.8

In reply to Grzegorz Blach
Quoted 11 lines · On Wed, 23 Oct 2019 21:19:21 +0000 Grzegorz Blach <[email protected]> wrote:

test/import-repeatedly.c uses an embedded python interpreter and PYTHON_EXTRA_LIBS is used only for building this test case, so set PYTHON_EXTRA_LIBS with python3-config --libs --embed to build this test case and don't link the module with libpython3.so.

Fixes:
- http://autobuild.buildroot.org/results/b30/b308eeb5c5d95ab9f1dbfc19f9183f2ba3ba0ce3/
- http://autobuild.buildroot.org/results/0dd/0dd9203f859b97ee5a3b6358644c26f8ab784ed8/
and many similar failures.

Guess what, in the end, after investigating, I agree that this patch was the right solution! PYTHON_LIBS is used when building Python extensions and PYTHON_EXTRA_LIBS is used when building code that embeds a Python interpreter.

So this patch made perfect sense. Of course, it would be *much* better for ax_python_devel.m4 to use python-config, rather than do its convoluted logic, but your patch is simple is easy, so I've applied.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com