Skip to contentSkip to conversation

package/libselinux: fix build with python 3.8

PATCH New James Hilliard 2 participants 11 messages Oct 20, 2019 – Oct 25, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package/libselinux: fix build with python 3.8

Fixes:
- http://autobuild.buildroot.net/results/839e81c5b968c47842b4d0c25b902397af8ea5a5

Details:
- https://bugs.python.org/issue36721
- https://github.com/python/cpython/commit/7efc526e5cfb929a79c192ac2dcf7eb78d3a4401

---
 package/libselinux/libselinux.mk | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
package/libselinux/libselinux.mk +7 -2
index c707c8a25e..75f01a84b7 100644
@@ -33,7 +33,7 @@ endif
 ifeq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),y)
 ifeq ($(BR2_PACKAGE_PYTHON3),y)
 LIBSELINUX_DEPENDENCIES += python3 host-swig
-LIBSELINUX_PYINC = -I$(STAGING_DIR)/usr/include/python$(PYTHON3_VERSION_MAJOR)m
+LIBSELINUX_PYINC = -I$(STAGING_DIR)/usr/include/python$(PYTHON3_VERSION_MAJOR)
 LIBSELINUX_PYLIBVER = python$(PYTHON3_VERSION_MAJOR)
 else ifeq ($(BR2_PACKAGE_PYTHON),y)
 LIBSELINUX_DEPENDENCIES += python host-swig
@@ -47,6 +47,11 @@ LIBSELINUX_MAKE_OPTS += \
 	PYSITEDIR=$(TARGET_DIR)/usr/lib/$(LIBSELINUX_PYLIBVER)/site-packages \
 	SWIG_LIB="$(HOST_DIR)/share/swig/$(SWIG_VERSION)/"
 
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+LIBSELINUX_MAKE_OPTS += \
+	PYLIBS="`$(STAGING_DIR)/usr/bin/python3-config --libs --embed`"
+endif
+
 LIBSELINUX_MAKE_INSTALL_TARGETS += install-pywrap
 
 # dependencies are broken and result in file truncation errors at link
@@ -84,7 +89,7 @@ HOST_LIBSELINUX_DEPENDENCIES = \
 
 ifeq ($(BR2_PACKAGE_PYTHON3),y)
 HOST_LIBSELINUX_DEPENDENCIES += host-python3
-HOST_LIBSELINUX_PYINC = -I$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR)m/
+HOST_LIBSELINUX_PYINC = -I$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR)/
 HOST_LIBSELINUX_PYLIBVER = python$(PYTHON3_VERSION_MAJOR)
 else
 HOST_LIBSELINUX_DEPENDENCIES += host-python
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/1 package/libselinux: fix build with python 3.8

package/libselinux/libselinux.mk endif · line 36

Quoted 10 lines · On Sun, 20 Oct 2019 14:31:19 -0600 James Hilliard <[email protected]>…
package/libselinux/libselinux.mk +1 -1
index c707c8a25e..75f01a84b7 100644
@@ -33,7 +33,7 @@ endif
 ifeq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),y)
 ifeq ($(BR2_PACKAGE_PYTHON3),y)
 LIBSELINUX_DEPENDENCIES += python3 host-swig
-LIBSELINUX_PYINC = -I$(STAGING_DIR)/usr/include/python$(PYTHON3_VERSION_MAJOR)m
+LIBSELINUX_PYINC = -I$(STAGING_DIR)/usr/include/python$(PYTHON3_VERSION_MAJOR)

Considering the number of packages that hardcode this path, perhaps we should have a variable provided by some common code, so that if it changes again in the future, we don't have to chase down multiple packages ?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH 1/1 package/libselinux: fix build with python 3.8

Quoted 4 lines · Thomas Petazzoni

On Sun, 20 Oct 2019 14:31:19 -0600
James Hilliard <[email protected]> wrote:

Quoted 15 lines

package/libselinux/libselinux.mk endif · line 36

Quoted 10 lines
package/libselinux/libselinux.mk +1 -1
index c707c8a25e..75f01a84b7 100644
@@ -33,7 +33,7 @@ endif
 ifeq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),y)
 ifeq ($(BR2_PACKAGE_PYTHON3),y)
 LIBSELINUX_DEPENDENCIES += python3 host-swig
-LIBSELINUX_PYINC = -I$(STAGING_DIR)/usr/include/python$(PYTHON3_VERSION_MAJOR)m
+LIBSELINUX_PYINC = -I$(STAGING_DIR)/usr/include/python$(PYTHON3_VERSION_MAJOR)

Considering the number of packages that hardcode this path, perhaps we should have a variable provided by some common code, so that if it changes again in the future, we don't have to chase down multiple packages ?

Maybe, although there aren't all that many from what I can tell and some seem to use slightly different variables. It doesn't seem all that likely that this will change
in the near future again at least since this change was only finalized after many
python releases.

Quoted 6 lines

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#3 lore raw

PATCH 1/1 package/libselinux: fix build with python 3.8

In reply to James Hilliard

package/libselinux/libselinux.mk LIBSELINUX_MAKE_OPTS += \ · line 53

Quoted 4 lines · On Sun, 20 Oct 2019 14:31:19 -0600 James Hilliard <[email protected]>…
package/libselinux/libselinux.mk +4 -0
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+LIBSELINUX_MAKE_OPTS += \
+	PYLIBS="`$(STAGING_DIR)/usr/bin/python3-config --libs --embed`"
+endif

Same question: libselinux is not embedding Python, it is providing a Python extension written in C, so according to what's in
https://bugs.python.org/issue36721, we should not need --embed.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

PATCH 1/1 package/libselinux: fix build with python 3.8

Quoted 4 lines · Thomas Petazzoni

On Sun, 20 Oct 2019 14:31:19 -0600
James Hilliard <[email protected]> wrote:

Quoted 8 lines

package/libselinux/libselinux.mk LIBSELINUX_MAKE_OPTS += \ · line 53

Quoted 4 lines
package/libselinux/libselinux.mk +4 -0
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+LIBSELINUX_MAKE_OPTS += \
+     PYLIBS="`$(STAGING_DIR)/usr/bin/python3-config --libs --embed`"
+endif

Same question: libselinux is not embedding Python, it is providing a Python extension written in C, so according to what's in
https://bugs.python.org/issue36721, we should not need --embed.

I think it might depend on if python is built with static vs shared libraries.

Quoted 6 lines

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#5 lore raw

PATCH 1/1 package/libselinux: fix build with python 3.8

Quoted 14 lines · On Mon, 21 Oct 2019 21:25:52 +0200 James Hilliard <[email protected]>…
Quoted 4 lines · Thomas Petazzoni

On Sun, 20 Oct 2019 14:31:19 -0600
James Hilliard <[email protected]> wrote:

Quoted 8 lines

package/libselinux/libselinux.mk LIBSELINUX_MAKE_OPTS += \ · line 53

Quoted 4 lines
package/libselinux/libselinux.mk +4 -0
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+LIBSELINUX_MAKE_OPTS += \
+     PYLIBS="`$(STAGING_DIR)/usr/bin/python3-config --libs --embed`"
+endif  

Same question: libselinux is not embedding Python, it is providing a Python extension written in C, so according to what's in
https://bugs.python.org/issue36721, we should not need --embed.

Quoted 1 line

I think it might depend on if python is built with static vs shared libraries.

That is not really what the bug report says. It says the reason to not link Python extensions with libpython was to allow loading Python modules in a statically-compiled Python, if I understood correctly.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#6 lore raw

PATCH 1/1 package/libselinux: fix build with python 3.8

Quoted 18 lines · Thomas Petazzoni
Quoted 14 lines · On Mon, 21 Oct 2019 21:25:52 +0200 James Hilliard <[email protected]>…
Quoted 12 lines · Thomas Petazzoni

package/libselinux/libselinux.mk LIBSELINUX_MAKE_OPTS += \ · line 53

Quoted 4 lines · On Sun, 20 Oct 2019 14:31:19 -0600 James Hilliard <[email protected]>…
package/libselinux/libselinux.mk +4 -0
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+LIBSELINUX_MAKE_OPTS += \
+     PYLIBS="`$(STAGING_DIR)/usr/bin/python3-config --libs --embed`"
+endif

Same question: libselinux is not embedding Python, it is providing a Python extension written in C, so according to what's in
https://bugs.python.org/issue36721, we should not need --embed.

Quoted 5 lines
Quoted 1 line

I think it might depend on if python is built with static vs shared libraries.

That is not really what the bug report says. It says the reason to not link Python extensions with libpython was to allow loading Python modules in a statically-compiled Python, if I understood correctly.

I think it's probably due to a quirk with the kmod build system where it's doing something weird here that's different from the typical setuptool/distutils build process:
https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git/tree/Makefile.am?id=58133a96c894c043e48c74ddf0bfe8db90bac62f#n178

Quoted 6 lines

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#7 lore raw

PATCH 1/1 package/libselinux: fix build with python 3.8

In reply to Thomas Petazzoni

Hello,

Quoted 8 lines · On Mon, 21 Oct 2019 23:18:48 +0200 Thomas Petazzoni…
Quoted 4 lines
Quoted 3 lines

Same question: libselinux is not embedding Python, it is providing a Python extension written in C, so according to what's in
https://bugs.python.org/issue36721, we should not need --embed.

I think it might depend on if python is built with static vs shared libraries.

That is not really what the bug report says. It says the reason to not link Python extensions with libpython was to allow loading Python modules in a statically-compiled Python, if I understood correctly.

So, if you try to build python-alsaaudio, which has some native code, you will see that it builds fine. The resulting .so Python extension is not linked against libpython, even though it calls functions like PyList_New(). This is fine, as the module will be dlopened() in the Python interpreter, which will provide PyList_New().

libselinux also uses PyList_New(), but complains that it is not available. So I looked at the libselinux link of the Python extension.

When I use the link command from the Makefile, i.e:

arm-linux-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -Os -I../include -D_GNU_SOURCE -DNO_ANDROID_BACKEND -lpcre -lpthread -L. -shared -o python-3.8audit2why.so python-3.8audit2why.lo -lselinux -l:libsepol.a -Wl,-soname,audit2why.so,--version-script=audit2why.map,-z,defs

Then the build fails with lots of undefined symbols, such as PyList_New().

If I change the command line to just:

arm-linux-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -Os -I../include -D_GNU_SOURCE -DNO_ANDROID_BACKEND -lpcre -lpthread -L. -shared -o python-3.8audit2why.so python-3.8audit2why.lo -lselinux -l:libsepol.a -Wl,-soname,audit2why.so

i.e, I simply drop the version script. Then doh, it builds fine!

It does have undefined references to PyList_New, but it is not linked against libpython:

$ arm-linux-gnu-readelf -W -a python-3.8audit2why.so | grep PyList_New
0003c29c  00002316 R_ARM_JUMP_SLOT        00000000   PyList_New
    35: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND PyList_New
  1145: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND PyList_New
$ arm-linux-gnu-readelf -W -a python-3.8audit2why.so | grep NEEDED
 0x00000001 (NEEDED)                     Shared library: [libpcre.so.1]
 0x00000001 (NEEDED)                     Shared library: [libselinux.so.1]
 0x00000001 (NEEDED)                     Shared library: [libc.so.0]

So as you can see linking against libpython is *not* the solution, it is just a workaround for another real problem. I have not found (yet) what is the problem with the versioning script, but I guess we should research in this direction.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#8 lore raw

PATCH 1/1 package/libselinux: fix build with python 3.8

Hello,

Quoted 4 lines · On Tue, 22 Oct 2019 09:40:25 +0200 Thomas Petazzoni…

So as you can see linking against libpython is *not* the solution, it is just a workaround for another real problem. I have not found (yet) what is the problem with the versioning script, but I guess we should research in this direction.

So, it is not the versioning script. In fact, when testing I removed both the --version-script option and the -z defs option. But it is actually the -z defs option that causes the problem.

And indeed:

-z defs

Report unresolved symbol references from regular object files. This is done even if the linker is creating a non-symbolic shared library. The switch --[no-]allow-shlib-undefined controls the behaviour for reporting unresolved references found in shared libraries being linked in.

Using -z defs in this particular case doesn't make any sense: we do acknowledge that we can have remaining unresolved symbols. Then will be provided by the Python interpreter.

So I believe the right fix is:

+++ src/Makefile 2019-10-22 09:44:26.700158005 +0200
@@ -165,7 +165,7 @@
$(CC) $(filter-out -Werror, $(CFLAGS)) $(PYINC) -fPIC -DSHARED -c -o $@ $<

$(AUDIT2WHYSO): $(AUDIT2WHYLOBJ) $(LIBSEPOLA)
- $(CC) $(CFLAGS) $(LDFLAGS) -L. -shared -o $@ $^ -lselinux $(LDLIBS_LIBSEPOLA) $(PYLIBS) -Wl,-soname,audit2why.so,--version-script=audit2why.map,-z,defs
+ $(CC) $(CFLAGS) $(LDFLAGS) -L. -shared -o $@ $^ -lselinux $(LDLIBS_LIBSEPOLA) $(PYLIBS) -Wl,-soname,audit2why.so,--version-script=audit2why.map

%.o: %.c policy.h
$(CC) $(CFLAGS) $(TLSFLAGS) -c -o $@ $<

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#9 lore raw

PATCH 1/1 package/libselinux: fix build with python 3.8

Hello,

+Matt Weber, SELinux stuff inside.

Quoted 1 line · On Tue, 22 Oct 2019 09:45:00 +0200 Thomas Petazzoni…

So I believe the right fix is:

With this fix, the module builds properly. With
http://patchwork.ozlabs.org/patch/1181164/, it gets installed to the target. However, importing the module fails:

Python 3.8.0 (default, Oct 21 2019, 21:15:12)
[GCC 4.9.4] on linux
Type "help", "copyright", "credits" or "license" for more information.

Quoted 1 line
Quoted 1 line
Quoted 1 line

import selinux

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "usr/lib/python3.8/site-packages/selinux/__init__.py", line 13, in <module>
ImportError: cannot import name '_selinux' from partially initialized module 'selinux' (most likely due to a circular import) (/usr/lib/python3.8/site-packages/selinux/__init__.pyc)

One interesting thing is that the name of the _selinux module is incorrect:

_selinux.cpython-38-x86_64-linux-gnu.so

See the "x86_64-linux-gnu" ? A properly working module has the following name:

alsaaudio.cpython-38-arm-linux-gnueabi.so

You might think _selinux.cpython-38-x86_64-linux-gnu.so is built for x86-64, but it's not, it's properly cross-compiled for ARM:

$ file _selinux.cpython-38-x86_64-linux-gnu.so 
_selinux.cpython-38-x86_64-linux-gnu.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, stripped

More investigation is needed.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#10 lore raw

PATCH 1/1 package/libselinux: fix build with python 3.8

Quoted 13 lines · On Tue, 22 Oct 2019 11:10:17 +0200 Thomas Petazzoni…

Hello,

+Matt Weber, SELinux stuff inside.

Quoted 1 line · On Tue, 22 Oct 2019 09:45:00 +0200 Thomas Petazzoni…

So I believe the right fix is:

With this fix, the module builds properly. With
http://patchwork.ozlabs.org/patch/1181164/, it gets installed to the target. However, importing the module fails:

Quoted 8 lines

Python 3.8.0 (default, Oct 21 2019, 21:15:12)
[GCC 4.9.4] on linux
Type "help", "copyright", "credits" or "license" for more information.

Quoted 1 line
Quoted 1 line
Quoted 1 line

import selinux

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "usr/lib/python3.8/site-packages/selinux/__init__.py", line 13, in <module>
ImportError: cannot import name '_selinux' from partially initialized module 'selinux' (most likely due to a circular import) (/usr/lib/python3.8/site-packages/selinux/__init__.pyc)

This problem, as well as the initial -z defs issue, has been resolved upstream by:

https://github.com/SELinuxProject/selinux/commit/2efa06857575e4118e91ca250b6b92da68b130d5

I'm now trying to backport that to 2.9.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com