Skip to contentSkip to conversation

package/python-iptables: add explicit dependency on dynamic libs

PATCH Superseded Frank Vanbever 4 participants 17 messages Feb 26, 2020 – Jan 6, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 package/python-iptables: add explicit dependency on dynamic libs

---
 package/python-iptables/Config.in | 4 ++++
 1 file changed, 4 insertions(+)
package/python-iptables/Config.in +4 -0
index e55359963e..a35577bad3 100644
@@ -1,7 +1,11 @@
 config BR2_PACKAGE_PYTHON_IPTABLES
 	bool "python-iptables"
+	depends on !BR2_STATIC_LIBS
 	select BR2_PACKAGE_IPTABLES # runtime dependency
 	help
 	  Python bindings for iptables.
 
 	  https://github.com/ldx/python-iptables
+
+comment "python-iptables needs a toolchain w/ dynamic library"
+	depends on BR2_STATIC_LIBS
Signature · 1 line
2.20.1
#1 lore raw

PATCH 2/3 package/python-iptables: try known libc instead of find_library()

ctypes.uitl.find_library() depends on gcc and friends to detect the location of a given shared library. Given that these are not available on the target and that python-iptables depends on this functionality we need to work around this. The SONAMEs of the libc are well known so we try the known ones for glibc, uClibc and musl.

---
 ...-Add-separate-mechanism-to-load-libc.patch | 90 +++++++++++++++++++
 1 file changed, 90 insertions(+)
 create mode 100644 package/python-iptables/0001-Add-separate-mechanism-to-load-libc.patch
package/python-iptables/0001-Add-separate-mechanism-to-load-libc.patch new file +90 -0
new file mode 100644
index 0000000000..1a928b4235
@@ -0,0 +1,90 @@
+From e3557528d7cdcdc2c579212be8837bc9b54635a4 Mon Sep 17 00:00:00 2001
+From: Frank Vanbever <[email protected]>
+Date: Thu, 20 Feb 2020 12:14:08 +0100
+Subject: [PATCH] Add separate mechanism to load libc
+
+ctypes.util.find_library() always returns None for systems which do not have the
+tools installed to determine the location of a given shared library (i.e.
+ldconfig, gcc, objdump). If find_libary() fails attempt to load known libc by
+SONAME.
+
+Upstream: https://github.com/ldx/python-iptables/pull/300
+
+Signed-off-by: Frank Vanbever <[email protected]>
+---
+ iptc/ip4tc.py   |  4 ++--
+ iptc/util.py    | 16 ++++++++++++++++
+ iptc/xtables.py |  4 ++--
+ 3 files changed, 20 insertions(+), 4 deletions(-)
+
+diff --git a/iptc/ip4tc.py b/iptc/ip4tc.py
+index 4c5d690..4ddd2dc 100644
+--- a/iptc/ip4tc.py
++++ b/iptc/ip4tc.py
+@@ -9,7 +9,7 @@ import socket
+ import struct
+ import weakref
+ 
+-from .util import find_library, load_kernel
++from .util import find_library, load_kernel, find_libc
+ from .xtables import (XT_INV_PROTO, NFPROTO_IPV4, XTablesError, xtables,
+                       xt_align, xt_counters, xt_entry_target, xt_entry_match)
+ 
+@@ -26,7 +26,7 @@ if not hasattr(socket, 'IPPROTO_SCTP'):
+ 
+ _IFNAMSIZ = 16
+ 
+-_libc = ct.CDLL("libc.so.6")
++_libc = find_libc()
+ _get_errno_loc = _libc.__errno_location
+ _get_errno_loc.restype = ct.POINTER(ct.c_int)
+ _malloc = _libc.malloc
+diff --git a/iptc/util.py b/iptc/util.py
+index ae5fb9b..e6b1649 100644
+--- a/iptc/util.py
++++ b/iptc/util.py
+@@ -109,3 +109,19 @@ def find_library(*names):
+             major = int(m.group(1))
+         return lib, major
+     return None, None
++
++
++def find_libc():
++    lib = ctypes.util.find_library('c')
++    if lib is not None:
++        return ctypes.CDLL(lib, mode=ctypes.RTLD_GLOBAL)
++
++    libnames = ['libc.so.6', 'libc.so.0', 'libc.so']
++    for name in libnames:
++        try:
++            lib = ctypes.CDLL(name, mode=ctypes.RTLD_GLOBAL)
++            return lib
++        except:
++            pass
++
++    return None
+diff --git a/iptc/xtables.py b/iptc/xtables.py
+index 93bc080..cf21029 100644
+--- a/iptc/xtables.py
++++ b/iptc/xtables.py
+@@ -6,7 +6,7 @@ import sys
+ import weakref
+ 
+ from . import version
+-from .util import find_library
++from .util import find_library, find_libc
+ from .errors import *
+ 
+ XT_INV_PROTO = 0x40  # invert the sense of PROTO
+@@ -792,7 +792,7 @@ class xtables_target(ct.Union):
+                 ("v12", _xtables_target_v12)]
+ 
+ 
+-_libc, _ = find_library("c")
++_libc = find_libc()
+ _optind = ct.c_long.in_dll(_libc, "optind")
+ _optarg = ct.c_char_p.in_dll(_libc, "optarg")
+ 
+-- 
+2.20.1
+
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/3 package/python-iptables: try known libc instead of find_library()

Hello Frank,

Quoted 1 line · On Wed, 26 Feb 2020 15:26:16 +0100 Frank Vanbever…

ctypes.uitl.find_library() depends on gcc and friends to detect the location of

Typo: uitl -> util.

Quoted 8 lines

a given shared library. Given that these are not available on the target and that python-iptables depends on this functionality we need to work around this. The SONAMEs of the libc are well known so we try the known ones for glibc, uClibc and musl.

I think the solution is nice and good, well done. I remember thinking about the issue a while ago, but couldn't come up with a simple way of detecting the C library file name from the target. Testing this limited hardcoded set of SONAMEs seems like a reasonable approach.

Quoted 1 line

+Upstream: https://github.com/ldx/python-iptables/pull/300

It has even been merged upstream, so this could even be replaced by a link to the upstream commit now.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw

PATCH 3/3 package/python-iptables: use installed iptables by default

In reply to Frank Vanbever

python-iptables depends on ctypes.util.find_library() which does not work due to the absence of gcc and friends on target. The location of the xtables library and the iptables modules can be configured through environment variables. Within the scope of buildroot we can determine what these should be at build time and replace the calls to os.getenv() with the correct value.

---
 package/python-iptables/Config.in          | 2 +-
 package/python-iptables/python-iptables.mk | 9 +++++++++
 2 files changed, 10 insertions(+), 1 deletion(-)
package/python-iptables/Config.in +1 -1
index a35577bad3..0e24283d06 100644
@@ -1,7 +1,7 @@
 config BR2_PACKAGE_PYTHON_IPTABLES
 	bool "python-iptables"
 	depends on !BR2_STATIC_LIBS
-	select BR2_PACKAGE_IPTABLES # runtime dependency
+	select BR2_PACKAGE_IPTABLES
 	help
 	  Python bindings for iptables.
 
package/python-iptables/python-iptables.mk +9 -0
index 66e478a89a..086d5b2457 100644
@@ -9,5 +9,14 @@ PYTHON_IPTABLES_SITE = https://files.pythonhosted.org/packages/08/5e/16a5ca35c42
 PYTHON_IPTABLES_SETUP_TYPE = setuptools
 PYTHON_IPTABLES_LICENSE = Apache-2.0
 PYTHON_IPTABLES_LICENSE_FILES = NOTICE
+PYTHON_IPTABLES_DEPENDENCIES = iptables
+
+define PYTHON_IPTABLES_SET_XTABLES_ENV_VARS
+	XTABLES_VERSION=`awk '/XTABLES_VERSION_CODE/ {print $$NF}' $(STAGING_DIR)/usr/include/xtables-version.h`; \
+	sed -i "s/os.getenv(\"PYTHON_IPTABLES_XTABLES_VERSION\")/$$XTABLES_VERSION/" $(@D)/iptc/xtables.py; \
+	sed -i "s/os.getenv(\"XTABLES_LIBDIR\")/\"\/usr\/lib\/xtables\"/" $(@D)/iptc/xtables.py
+endef
+
+PYTHON_IPTABLES_PRE_BUILD_HOOKS += PYTHON_IPTABLES_SET_XTABLES_ENV_VARS
 
 $(eval $(python-package))
Signature · 1 line
2.20.1
#4 lore raw

PATCH 3/3 package/python-iptables: use installed iptables by default

Hello Frank,

Quoted 5 lines · On Wed, 26 Feb 2020 15:26:17 +0100 Frank Vanbever…
package/python-iptables/python-iptables.mk +0 -0
index 66e478a89a..086d5b2457 100644

package/python-iptables/python-iptables.mk PYTHON_IPTABLES_SITE = https://files.pythonhosted.org/packages/08/5e/16a5ca35c42 · line 17

Quoted 9 lines
package/python-iptables/python-iptables.mk +6 -0
@@ -9,5 +9,14 @@ PYTHON_IPTABLES_SITE = https://files.pythonhosted.org/packages/08/5e/16a5ca35c42
 PYTHON_IPTABLES_SETUP_TYPE = setuptools
 PYTHON_IPTABLES_LICENSE = Apache-2.0
 PYTHON_IPTABLES_LICENSE_FILES = NOTICE
+PYTHON_IPTABLES_DEPENDENCIES = iptables
+
+define PYTHON_IPTABLES_SET_XTABLES_ENV_VARS
+	XTABLES_VERSION=`awk '/XTABLES_VERSION_CODE/ {print $$NF}' $(STAGING_DIR)/usr/include/xtables-version.h`; \
+	sed -i "s/os.getenv(\"PYTHON_IPTABLES_XTABLES_VERSION\")/$$XTABLES_VERSION/" $(@D)/iptc/xtables.py; \
+	sed -i "s/os.getenv(\"XTABLES_LIBDIR\")/\"\/usr\/lib\/xtables\"/" $(@D)/iptc/xtables.py

This last line is not using the XTABLES_VERSION variable, so it does not need to be a continuation line from the previous commands.

Also, use % as a sed separator instead of / so that you don't have to escape all the slashes.

But overall, is there a better way ? Like some official way to pass these values at build time as setup.py options, with the environment variable taking precedence if available ? That would make this hopefully acceptable by upstream. Perhaps the setup.py logic could even check the xtables-version.h by itself ?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

PATCH 3/3 package/python-iptables: use installed iptables by default

Hi Thomas,

Thank you for the comments.

Quoted 5 lines

But overall, is there a better way ? Like some official way to pass these values at build time as setup.py options, with the environment variable taking precedence if available ? That would make this hopefully acceptable by upstream. Perhaps the setup.py logic could even check the xtables-version.h by itself ?

For installations from source this would work. [1] However Python also has the option of distributing Wheel binary packages, where this logic would not be executed. [2] This would mean that we put the burden of handling the difference between installations on the upstream.

I took a second look and actually I think I can propose a workaround to the upstream. There's 2 problems that need to be tackled.

The first one is loading libxtables.so. iptables installs a libxtables.so symlink without any version. CDLL has no issues with this so it can just try and load that if all the other attempts fail.

The second problem is the location of the XTABLES_LIBDIR. Right now it tries all paths that are mentioned in the output of ldconfig -N -v. I guess it wouldn't be that much of a stretch to test some sensible locations (/usr/ local/lib/xtables, /usr/lib/xtables etc) if that should fail, in the same vein as the libc solution. It's not clean but it gets the job done.

I'll see if I can whip something up and submit it upstream.

Best regards,
Frank

[1] https://stackoverflow.com/questions/20288711/post-install-script-with-python-setuptools
[2] https://github.com/pypa/setuptools/issues/1782

#6 lore raw

PATCH 3/3 package/python-iptables: use installed iptables by default

Hi Thomas,

Quoted 1 line

I'll see if I can whip something up and submit it upstream.

I gave it another try but it seems that I was a bit too optimistic yesterday. The xtables module requires the ABI version to cast the return values for matches and targets to something it can access.

I have a final last ditch theory about how this could be fixed, but to be honest it's really starting to look like more trouble than it's worth.

The ABI version information is embedded into struct xtables_match and
struct xtables_target. You should be able to get to these just from a handle
to the libxtables.so and the location of the iptables extensions (i.e. xtables_libdir), for which you could use the same solution as for libc and test a number of sensible locations if ldconfig is unavailable. The xtables_{match,target} structs always have the version field as the first field in the returned struct so I think you could do an xtables_find_{target,match} and do a ctypes.cast to a dummy ctypes.struct class with just the version field.

The way the module is written though requires the ABI version global variable to instantiate the xtables class which contains the logic that would allow you to find a match/target. So you get into this chicken or the egg type situation where you need to know the ABI version to get the workaround way to get the ABI version to work.

Best regards,
Frank

#7 lore raw

PATCH 1/3 package/python-iptables: add explicit dependency on dynamic libs

In reply to Frank Vanbever

package/python-iptables/Config.in line 3

Quoted 13 lines · On Wed, 26 Feb 2020 15:26:15 +0100 Frank Vanbever…
---
 package/python-iptables/Config.in | 4 ++++
 1 file changed, 4 insertions(+)
package/python-iptables/Config.in +1 -0
index e55359963e..a35577bad3 100644
@@ -1,7 +1,11 @@
 config BR2_PACKAGE_PYTHON_IPTABLES
 	bool "python-iptables"
+	depends on !BR2_STATIC_LIBS

This needs a bit more explanation, at least a short comment above. Even your commit log is empty.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#8 lore raw

PATCH v2 1/3 package/python-iptables: add explicit dependency on dynamic libs

In reply to Frank Vanbever

The package uses ctypes.CDLL extensively which only makes sense when dynamic libraries are available.

---
v1 - > v2:
- Add some additional explanation in commit message.

---
 package/python-iptables/Config.in | 4 ++++
 1 file changed, 4 insertions(+)
package/python-iptables/Config.in +4 -0
index e55359963e..a35577bad3 100644
@@ -1,7 +1,11 @@
 config BR2_PACKAGE_PYTHON_IPTABLES
 	bool "python-iptables"
+	depends on !BR2_STATIC_LIBS
 	select BR2_PACKAGE_IPTABLES # runtime dependency
 	help
 	  Python bindings for iptables.
 
 	  https://github.com/ldx/python-iptables
+
+comment "python-iptables needs a toolchain w/ dynamic library"
+	depends on BR2_STATIC_LIBS
Signature · 1 line
2.20.1
#9 lore raw

PATCH v2 2/3 package/python-iptables: try known libc instead of find_library()

ctypes.util.find_library() depends on gcc and friends to detect the location of a given shared library. Given that these are not available on the target and that python-iptables depends on this functionality we need to work around this. The SONAMEs of the libc are well known so we try the known ones for glibc, uClibc and musl.

----
v1 -> v2:
- Change URL in patch from pull request to commit

---
 ...-Add-separate-mechanism-to-load-libc.patch | 90 +++++++++++++++++++
 1 file changed, 90 insertions(+)
 create mode 100644 package/python-iptables/0001-Add-separate-mechanism-to-load-libc.patch
package/python-iptables/0001-Add-separate-mechanism-to-load-libc.patch new file +90 -0
new file mode 100644
index 0000000000..2a344c2f55
@@ -0,0 +1,90 @@
+From e3557528d7cdcdc2c579212be8837bc9b54635a4 Mon Sep 17 00:00:00 2001
+From: Frank Vanbever <[email protected]>
+Date: Thu, 20 Feb 2020 12:14:08 +0100
+Subject: [PATCH] Add separate mechanism to load libc
+
+ctypes.util.find_library() always returns None for systems which do not have the
+tools installed to determine the location of a given shared library (i.e.
+ldconfig, gcc, objdump). If find_libary() fails attempt to load known libc by
+SONAME.
+
+Upstream: https://github.com/ldx/python-iptables/commit/e3557528d7cdcdc2c579212be8837bc9b54635a4
+
+Signed-off-by: Frank Vanbever <[email protected]>
+---
+ iptc/ip4tc.py   |  4 ++--
+ iptc/util.py    | 16 ++++++++++++++++
+ iptc/xtables.py |  4 ++--
+ 3 files changed, 20 insertions(+), 4 deletions(-)
+
+diff --git a/iptc/ip4tc.py b/iptc/ip4tc.py
+index 4c5d690..4ddd2dc 100644
+--- a/iptc/ip4tc.py
++++ b/iptc/ip4tc.py
+@@ -9,7 +9,7 @@ import socket
+ import struct
+ import weakref
+ 
+-from .util import find_library, load_kernel
++from .util import find_library, load_kernel, find_libc
+ from .xtables import (XT_INV_PROTO, NFPROTO_IPV4, XTablesError, xtables,
+                       xt_align, xt_counters, xt_entry_target, xt_entry_match)
+ 
+@@ -26,7 +26,7 @@ if not hasattr(socket, 'IPPROTO_SCTP'):
+ 
+ _IFNAMSIZ = 16
+ 
+-_libc = ct.CDLL("libc.so.6")
++_libc = find_libc()
+ _get_errno_loc = _libc.__errno_location
+ _get_errno_loc.restype = ct.POINTER(ct.c_int)
+ _malloc = _libc.malloc
+diff --git a/iptc/util.py b/iptc/util.py
+index ae5fb9b..e6b1649 100644
+--- a/iptc/util.py
++++ b/iptc/util.py
+@@ -109,3 +109,19 @@ def find_library(*names):
+             major = int(m.group(1))
+         return lib, major
+     return None, None
++
++
++def find_libc():
++    lib = ctypes.util.find_library('c')
++    if lib is not None:
++        return ctypes.CDLL(lib, mode=ctypes.RTLD_GLOBAL)
++
++    libnames = ['libc.so.6', 'libc.so.0', 'libc.so']
++    for name in libnames:
++        try:
++            lib = ctypes.CDLL(name, mode=ctypes.RTLD_GLOBAL)
++            return lib
++        except:
++            pass
++
++    return None
+diff --git a/iptc/xtables.py b/iptc/xtables.py
+index 93bc080..cf21029 100644
+--- a/iptc/xtables.py
++++ b/iptc/xtables.py
+@@ -6,7 +6,7 @@ import sys
+ import weakref
+ 
+ from . import version
+-from .util import find_library
++from .util import find_library, find_libc
+ from .errors import *
+ 
+ XT_INV_PROTO = 0x40  # invert the sense of PROTO
+@@ -792,7 +792,7 @@ class xtables_target(ct.Union):
+                 ("v12", _xtables_target_v12)]
+ 
+ 
+-_libc, _ = find_library("c")
++_libc = find_libc()
+ _optind = ct.c_long.in_dll(_libc, "optind")
+ _optarg = ct.c_char_p.in_dll(_libc, "optarg")
+ 
+-- 
+2.20.1
+
Signature · 1 line
2.20.1
#10 lore raw

PATCH v2 3/3 package/python-iptables: use installed iptables by default

In reply to Frank Vanbever

python-iptables depends on ctypes.util.find_library() which does not work due to the absence of gcc and friends on target. The location of the xtables library and the iptables modules can be configured through environment variables. Within the scope of buildroot we can determine what these should be at build time and replace the calls to os.getenv() with the correct value.

---
v1 -> v2:
- Remove unnecessary continuation in pre-build hook
- Change sed separator to %

---
 package/python-iptables/Config.in          | 2 +-
 package/python-iptables/python-iptables.mk | 9 +++++++++
 2 files changed, 10 insertions(+), 1 deletion(-)
package/python-iptables/Config.in +1 -1
index a35577bad3..0e24283d06 100644
@@ -1,7 +1,7 @@
 config BR2_PACKAGE_PYTHON_IPTABLES
 	bool "python-iptables"
 	depends on !BR2_STATIC_LIBS
-	select BR2_PACKAGE_IPTABLES # runtime dependency
+	select BR2_PACKAGE_IPTABLES
 	help
 	  Python bindings for iptables.
 
package/python-iptables/python-iptables.mk +9 -0
index 66e478a89a..38c6ee3950 100644
@@ -9,5 +9,14 @@ PYTHON_IPTABLES_SITE = https://files.pythonhosted.org/packages/08/5e/16a5ca35c42
 PYTHON_IPTABLES_SETUP_TYPE = setuptools
 PYTHON_IPTABLES_LICENSE = Apache-2.0
 PYTHON_IPTABLES_LICENSE_FILES = NOTICE
+PYTHON_IPTABLES_DEPENDENCIES = iptables
+
+define PYTHON_IPTABLES_SET_XTABLES_ENV_VARS
+	XTABLES_VERSION=`awk '/XTABLES_VERSION_CODE/ {print $$NF}' $(STAGING_DIR)/usr/include/xtables-version.h`; \
+	sed -i "s%os.getenv(\"PYTHON_IPTABLES_XTABLES_VERSION\")%$$XTABLES_VERSION%" $(@D)/iptc/xtables.py
+	sed -i "s%os.getenv(\"XTABLES_LIBDIR\")%\"/usr/lib/xtables\"%" $(@D)/iptc/xtables.py
+endef
+
+PYTHON_IPTABLES_PRE_BUILD_HOOKS += PYTHON_IPTABLES_SET_XTABLES_ENV_VARS
 
 $(eval $(python-package))
Signature · 1 line
2.20.1
#11 lore raw

PATCH v3 1/4 package/python-iptables: add explicit dependency on dynamic libs

In reply to Frank Vanbever

The package uses ctypes.CDLL extensively which only makes sense when dynamic libraries are available.

---
v2 -> v3:
- None
v1 -> v2:
- Add some additional explanation in commit message.

---
 package/python-iptables/Config.in | 4 ++++
 1 file changed, 4 insertions(+)
package/python-iptables/Config.in +4 -0
index e55359963e..a35577bad3 100644
@@ -1,7 +1,11 @@
 config BR2_PACKAGE_PYTHON_IPTABLES
 	bool "python-iptables"
+	depends on !BR2_STATIC_LIBS
 	select BR2_PACKAGE_IPTABLES # runtime dependency
 	help
 	  Python bindings for iptables.
 
 	  https://github.com/ldx/python-iptables
+
+comment "python-iptables needs a toolchain w/ dynamic library"
+	depends on BR2_STATIC_LIBS
Signature · 1 line
2.20.1
#12 lore raw

PATCH v3 2/4 package/python-iptables: try known libc instead of find_library()

ctypes.util.find_library() depends on gcc and friends to detect the location of a given shared library. Given that these are not available on the target and that python-iptables depends on this functionality we need to work around this. The SONAMEs of the libc are well known so we try the known ones for glibc, uClibc and musl.

----
v2 -> v3:
- None
v1 -> v2:
- Change URL in patch from pull request to commit

---
 ...-Add-separate-mechanism-to-load-libc.patch | 90 +++++++++++++++++++
 1 file changed, 90 insertions(+)
 create mode 100644 package/python-iptables/0001-Add-separate-mechanism-to-load-libc.patch
package/python-iptables/0001-Add-separate-mechanism-to-load-libc.patch new file +90 -0
new file mode 100644
index 0000000000..2a344c2f55
@@ -0,0 +1,90 @@
+From e3557528d7cdcdc2c579212be8837bc9b54635a4 Mon Sep 17 00:00:00 2001
+From: Frank Vanbever <[email protected]>
+Date: Thu, 20 Feb 2020 12:14:08 +0100
+Subject: [PATCH] Add separate mechanism to load libc
+
+ctypes.util.find_library() always returns None for systems which do not have the
+tools installed to determine the location of a given shared library (i.e.
+ldconfig, gcc, objdump). If find_libary() fails attempt to load known libc by
+SONAME.
+
+Upstream: https://github.com/ldx/python-iptables/commit/e3557528d7cdcdc2c579212be8837bc9b54635a4
+
+Signed-off-by: Frank Vanbever <[email protected]>
+---
+ iptc/ip4tc.py   |  4 ++--
+ iptc/util.py    | 16 ++++++++++++++++
+ iptc/xtables.py |  4 ++--
+ 3 files changed, 20 insertions(+), 4 deletions(-)
+
+diff --git a/iptc/ip4tc.py b/iptc/ip4tc.py
+index 4c5d690..4ddd2dc 100644
+--- a/iptc/ip4tc.py
++++ b/iptc/ip4tc.py
+@@ -9,7 +9,7 @@ import socket
+ import struct
+ import weakref
+ 
+-from .util import find_library, load_kernel
++from .util import find_library, load_kernel, find_libc
+ from .xtables import (XT_INV_PROTO, NFPROTO_IPV4, XTablesError, xtables,
+                       xt_align, xt_counters, xt_entry_target, xt_entry_match)
+ 
+@@ -26,7 +26,7 @@ if not hasattr(socket, 'IPPROTO_SCTP'):
+ 
+ _IFNAMSIZ = 16
+ 
+-_libc = ct.CDLL("libc.so.6")
++_libc = find_libc()
+ _get_errno_loc = _libc.__errno_location
+ _get_errno_loc.restype = ct.POINTER(ct.c_int)
+ _malloc = _libc.malloc
+diff --git a/iptc/util.py b/iptc/util.py
+index ae5fb9b..e6b1649 100644
+--- a/iptc/util.py
++++ b/iptc/util.py
+@@ -109,3 +109,19 @@ def find_library(*names):
+             major = int(m.group(1))
+         return lib, major
+     return None, None
++
++
++def find_libc():
++    lib = ctypes.util.find_library('c')
++    if lib is not None:
++        return ctypes.CDLL(lib, mode=ctypes.RTLD_GLOBAL)
++
++    libnames = ['libc.so.6', 'libc.so.0', 'libc.so']
++    for name in libnames:
++        try:
++            lib = ctypes.CDLL(name, mode=ctypes.RTLD_GLOBAL)
++            return lib
++        except:
++            pass
++
++    return None
+diff --git a/iptc/xtables.py b/iptc/xtables.py
+index 93bc080..cf21029 100644
+--- a/iptc/xtables.py
++++ b/iptc/xtables.py
+@@ -6,7 +6,7 @@ import sys
+ import weakref
+ 
+ from . import version
+-from .util import find_library
++from .util import find_library, find_libc
+ from .errors import *
+ 
+ XT_INV_PROTO = 0x40  # invert the sense of PROTO
+@@ -792,7 +792,7 @@ class xtables_target(ct.Union):
+                 ("v12", _xtables_target_v12)]
+ 
+ 
+-_libc, _ = find_library("c")
++_libc = find_libc()
+ _optind = ct.c_long.in_dll(_libc, "optind")
+ _optarg = ct.c_char_p.in_dll(_libc, "optarg")
+ 
+-- 
+2.20.1
+
Signature · 1 line
2.20.1
#13 lore raw

PATCH v3 3/4 package/python-iptables: use installed iptables by default

In reply to Frank Vanbever

python-iptables depends on ctypes.util.find_library() which does not work due to the absence of gcc and friends on target. The location of the xtables library and the iptables modules can be configured through environment variables. Within the scope of buildroot we can determine what these should be at build time and replace the calls to os.getenv() with the correct value.

---
v2 -> v3:
- None
v1 -> v2:
- Remove unnecessary continuation in pre-build hook
- Change sed separator to %

---
 package/python-iptables/Config.in          | 2 +-
 package/python-iptables/python-iptables.mk | 9 +++++++++
 2 files changed, 10 insertions(+), 1 deletion(-)
package/python-iptables/Config.in +1 -1
index a35577bad3..0e24283d06 100644
@@ -1,7 +1,7 @@
 config BR2_PACKAGE_PYTHON_IPTABLES
 	bool "python-iptables"
 	depends on !BR2_STATIC_LIBS
-	select BR2_PACKAGE_IPTABLES # runtime dependency
+	select BR2_PACKAGE_IPTABLES
 	help
 	  Python bindings for iptables.
 
package/python-iptables/python-iptables.mk +9 -0
index 66e478a89a..38c6ee3950 100644
@@ -9,5 +9,14 @@ PYTHON_IPTABLES_SITE = https://files.pythonhosted.org/packages/08/5e/16a5ca35c42
 PYTHON_IPTABLES_SETUP_TYPE = setuptools
 PYTHON_IPTABLES_LICENSE = Apache-2.0
 PYTHON_IPTABLES_LICENSE_FILES = NOTICE
+PYTHON_IPTABLES_DEPENDENCIES = iptables
+
+define PYTHON_IPTABLES_SET_XTABLES_ENV_VARS
+	XTABLES_VERSION=`awk '/XTABLES_VERSION_CODE/ {print $$NF}' $(STAGING_DIR)/usr/include/xtables-version.h`; \
+	sed -i "s%os.getenv(\"PYTHON_IPTABLES_XTABLES_VERSION\")%$$XTABLES_VERSION%" $(@D)/iptc/xtables.py
+	sed -i "s%os.getenv(\"XTABLES_LIBDIR\")%\"/usr/lib/xtables\"%" $(@D)/iptc/xtables.py
+endef
+
+PYTHON_IPTABLES_PRE_BUILD_HOOKS += PYTHON_IPTABLES_SET_XTABLES_ENV_VARS
 
 $(eval $(python-package))
Signature · 1 line
2.20.1
#14 lore raw

Re: [Buildroot] [PATCH v3 3/4] package/python-iptables: use installed iptables by default

Quoted 9 lines · Frank Vanbever

python-iptables depends on ctypes.util.find_library() which does not work due to the absence of gcc and friends on target. The location of the xtables library and the iptables modules can be configured through environment variables. Within the scope of buildroot we can determine what these should be at build time and replace the calls to os.getenv() with the correct value.

This one had fallen through the cracks. I finally applied it to master.

To verify that it really solves a bug, I also added a runtime test which
indeed fails without this patch.

Regards,
Arnout

Quoted 36 lines

---
v2 -> v3:
- None
v1 -> v2:
- Remove unnecessary continuation in pre-build hook
- Change sed separator to %

---
  package/python-iptables/Config.in          | 2 +-
  package/python-iptables/python-iptables.mk | 9 +++++++++
  2 files changed, 10 insertions(+), 1 deletion(-)
package/python-iptables/Config.in +1 -1
index a35577bad3..0e24283d06 100644
@@ -1,7 +1,7 @@
  config BR2_PACKAGE_PYTHON_IPTABLES
  	bool "python-iptables"
  	depends on !BR2_STATIC_LIBS
-	select BR2_PACKAGE_IPTABLES # runtime dependency
+	select BR2_PACKAGE_IPTABLES
  	help
  	  Python bindings for iptables.
 
package/python-iptables/python-iptables.mk +1 -0
index 66e478a89a..38c6ee3950 100644
@@ -9,5 +9,14 @@ PYTHON_IPTABLES_SITE = https://files.pythonhosted.org/packages/08/5e/16a5ca35c42
  PYTHON_IPTABLES_SETUP_TYPE = setuptools
  PYTHON_IPTABLES_LICENSE = Apache-2.0
  PYTHON_IPTABLES_LICENSE_FILES = NOTICE
+PYTHON_IPTABLES_DEPENDENCIES = iptables

package/python-iptables/python-iptables.mk PYTHON_IPTABLES_SITE = https://files.pythonhosted.org/packages/08/5e/16a5ca35c42 · line 22

Quoted 10 lines
package/python-iptables/python-iptables.mk +8 -0
+
+define PYTHON_IPTABLES_SET_XTABLES_ENV_VARS
+	XTABLES_VERSION=`awk '/XTABLES_VERSION_CODE/ {print $$NF}' $(STAGING_DIR)/usr/include/xtables-version.h`; \
+	sed -i "s%os.getenv(\"PYTHON_IPTABLES_XTABLES_VERSION\")%$$XTABLES_VERSION%" $(@D)/iptc/xtables.py
+	sed -i "s%os.getenv(\"XTABLES_LIBDIR\")%\"/usr/lib/xtables\"%" $(@D)/iptc/xtables.py
+endef
+
+PYTHON_IPTABLES_PRE_BUILD_HOOKS += PYTHON_IPTABLES_SET_XTABLES_ENV_VARS
 
  $(eval $(python-package))

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#15 lore raw

PATCH v3 4/4 package/python-iptables: account for platform tag in extensions

In reply to Frank Vanbever

EXT_SUFFIX in Python versions > 3.5 contains a platform tag which only applies to cpython extensions. Given that ctypes.util.find_library does not work on the target due to the absence of the underlying tools '.so' needs to be added as a possible suffix for libraries to enable python-iptables to find the iptables shared libraries.

---
v2 -> v3:
- Add additional patch to fix python-iptables for python > 3.5

---
 ...o-as-additional-shared-object-suffix.patch | 57 +++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 package/python-iptables/0002-Add-.so-as-additional-shared-object-suffix.patch
package/python-iptables/0002-Add-.so-as-additional-shared-object-suffix.patch new file +57 -0
new file mode 100644
index 0000000000..41d49fc1e6
@@ -0,0 +1,57 @@
+From 899d25c511c6ce779b7153e9ae2e41055b30b9c5 Mon Sep 17 00:00:00 2001
+From: Frank Vanbever <[email protected]>
+Date: Mon, 9 Mar 2020 12:36:47 +0100
+Subject: [PATCH] Add '.so' as additional shared object suffix
+
+EXT_SUFFIX includes a platform information tag starting from Python 3.5 [0]
+For example:
+
+    >>> sysconfig.get_config_var("EXT_SUFFIX")
+    '.cpython-38-aarch64-linux-gnu.so'
+
+This suffix only applies to cpython extensions i.e. not to the iptables shared
+objects.
+
+Adding '.so' as an additional suffix for shared objects fixes the issue.
+
+Fixes: Issue #301
+
+Signed-off-by: Frank Vanbever <[email protected]>
+
+Backported from: 899d25c511c6ce779b7153e9ae2e41055b30b9c5
+
+[0]: https://docs.python.org/3/whatsnew/3.5.html#build-and-c-api-changes
+---
+ iptc/util.py | 13 ++++++++++---
+ 1 file changed, 10 insertions(+), 3 deletions(-)
+
+diff --git a/iptc/util.py b/iptc/util.py
+index e6b1649..04fe905 100644
+--- a/iptc/util.py
++++ b/iptc/util.py
+@@ -80,12 +80,19 @@ def _do_find_library(name):
+ 
+ 
+ def _find_library(*names):
++    exts = []
+     if version_info >= (3, 3):
+-        ext = get_config_var("EXT_SUFFIX")
++        exts.append(get_config_var("EXT_SUFFIX"))
+     else:
+-        ext = get_config_var('SO')
++        exts.append(get_config_var('SO'))
++
++    if version_info >= (3, 5):
++        exts.append('.so')
++
+     for name in names:
+-        libnames = [name, "lib" + name, name + ext, "lib" + name + ext]
++        libnames = [name, "lib" + name]
++        for ext in exts:
++            libnames += [name + ext, "lib" + name + ext]
+         libdir = os.environ.get('IPTABLES_LIBDIR', None)
+         if libdir is not None:
+             libdirs = libdir.split(':')
+-- 
+2.20.1
+
Signature · 1 line
2.20.1
#16 lore raw

PATCH v3 1/4 package/python-iptables: add explicit dependency on dynamic libs

In reply to Frank Vanbever

Frank, All,

On 2020-03-11 11:54 +0100, Frank Vanbever spake thusly:

Quoted 4 lines

The package uses ctypes.CDLL extensively which only makes sense when dynamic libraries are available.

Patches 1, 2, and 4, which are all backports of upstream commits, applied to master, thanks.

I'm still pondering how to improve patch 3, though, as I concur with Thomas that it would be nice if we could do better... I'm not gonna sweat it, though...

Regards,
Yann E. MORIN.

Quoted 34 lines

---
v2 -> v3:
- None
v1 -> v2:
- Add some additional explanation in commit message.

---
 package/python-iptables/Config.in | 4 ++++
 1 file changed, 4 insertions(+)
package/python-iptables/Config.in +4 -0
index e55359963e..a35577bad3 100644
@@ -1,7 +1,11 @@
 config BR2_PACKAGE_PYTHON_IPTABLES
 	bool "python-iptables"
+	depends on !BR2_STATIC_LIBS
 	select BR2_PACKAGE_IPTABLES # runtime dependency
 	help
 	  Python bindings for iptables.
 
 	  https://github.com/ldx/python-iptables
+
+comment "python-iptables needs a toolchain w/ dynamic library"
+	depends on BR2_STATIC_LIBS
Signature · 6 lines
2.20.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'