Skip to contentSkip to conversation

CFFI backend

PATCH New yegorslists at googlemail.com 2 participants 9 messages Jan 5, 2016 – Feb 3, 2016

lore
Share & tools
lore
#0 lore raw

v2 cover CFFI backend

This patch series introduces CFFI backend for Python packages. CFFI can cross-compile C library wrappers and create *.py and *.so files for the target rootfs. In order to do so CFFI uses pycparser to parse C heasers/sources. The target variant of CFFI is working with already prepared *.py and *.so files and doesn't need pycparser any longer.

The libffi headers patch is needed, because only headers for target variant will be copied to the proper destination and host packages, that don't use pkg-config cannot find libffi headers at its default location.

Yegor

#1 lore raw

PATCH v2 1/3 libffi: fix headers location for host-libffi

From: Yegor Yefremov <[email protected]>

There is a special hook for target-libffi, that moves headers from /usr/lib/libffi-version/include to /usr/include. This patch adds the same procedure for host-libffi.

---
 package/libffi/libffi.mk | 11 +++++++++++
 1 file changed, 11 insertions(+)
package/libffi/libffi.mk +11 -0
index 8f0f72d..47de8cd 100644
@@ -22,6 +22,17 @@ endef
 
 LIBFFI_POST_INSTALL_STAGING_HOOKS += LIBFFI_MOVE_STAGING_HEADERS
 
+# Move the headers to the usual location, and adjust the .pc file
+# accordingly.
+define HOST_LIBFFI_MOVE_HOST_HEADERS
+	mv $(HOST_DIR)/usr/lib/libffi-$(LIBFFI_VERSION)/include/*.h $(HOST_DIR)/usr/include/
+	$(SED) '/^includedir.*/d' -e '/^Cflags:.*/d' \
+		$(HOST_DIR)/usr/lib/pkgconfig/libffi.pc
+	rm -rf $(HOST_DIR)/usr/lib/libffi-*
+endef
+
+HOST_LIBFFI_POST_INSTALL_HOOKS += HOST_LIBFFI_MOVE_HOST_HEADERS
+
 # Remove headers that are not at the usual location from the target
 define LIBFFI_REMOVE_TARGET_HEADERS
 	$(RM) -rf $(TARGET_DIR)/usr/lib/libffi-$(LIBFFI_VERSION)
Signature · 1 line
2.1.4
#2 lore raw

PATCH v2 1/3 libffi: fix headers location for host-libffi

Hello,

Quoted 19 lines · yegorslists at googlemail.com

From: Yegor Yefremov <[email protected]>

There is a special hook for target-libffi, that moves headers from /usr/lib/libffi-version/include to /usr/include. This patch adds the same procedure for host-libffi.

---
 package/libffi/libffi.mk | 11 +++++++++++
 1 file changed, 11 insertions(+)
package/libffi/libffi.mk +0 -0
index 8f0f72d..47de8cd 100644
@@ -22,6 +22,17 @@ endef
 
 LIBFFI_POST_INSTALL_STAGING_HOOKS += LIBFFI_MOVE_STAGING_HEADERS
 

package/libffi/libffi.mk endef · line 34

Quoted 10 lines
package/libffi/libffi.mk +10 -0
+# Move the headers to the usual location, and adjust the .pc file
+# accordingly.
+define HOST_LIBFFI_MOVE_HOST_HEADERS
+	mv $(HOST_DIR)/usr/lib/libffi-$(LIBFFI_VERSION)/include/*.h $(HOST_DIR)/usr/include/
+	$(SED) '/^includedir.*/d' -e '/^Cflags:.*/d' \
+		$(HOST_DIR)/usr/lib/pkgconfig/libffi.pc
+	rm -rf $(HOST_DIR)/usr/lib/libffi-*
+endef
+
+HOST_LIBFFI_POST_INSTALL_HOOKS += HOST_LIBFFI_MOVE_HOST_HEADERS

I've applied, but after doing some changes. First, the existing staging fixup had a mistake, it was removing the directory from $(TARGET_DIR), while it should have been removed from $(STAGING_DIR). I've fixed that in a preparation commit:

https://git.busybox.net/buildroot/commit/?id=2b73538920107b6c020ce5c4c8d9f7f672549bb1

Once this was fixed, your new code to take care of headers in $(HOST_DIR) was exactly the same as the one taking care of the headers in $(STAGING_DIR). So I've factorized the code in a common function, which takes as argument the base directory. Which gives:

https://git.busybox.net/buildroot/commit/?id=f41091176bb3e8da6cf78760b69f188e9868a259

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#3 lore raw

PATCH v2 2/3 python-pycparser: new package

In reply to yegorslists at googlemail.com

From: Yegor Yefremov <[email protected]>

pycparser requires C preprocessor in order to parse C related files, hence available only as host package.

---
Changes:
v2: remove target package variant

 package/python-pycparser/python-pycparser.hash |  3 +++
 package/python-pycparser/python-pycparser.mk   | 14 ++++++++++++++
 2 files changed, 17 insertions(+)
 create mode 100644 package/python-pycparser/python-pycparser.hash
 create mode 100644 package/python-pycparser/python-pycparser.mk
package/python-pycparser/python-pycparser.hash new file +3 -0
new file mode 100644
index 0000000..a87bfd4
@@ -0,0 +1,3 @@
+# md5 from https://pypi.python.org/pypi?:action=show_md5&digest=a2bc8d28c923b4fe2b2c3b4b51a4f935, sha256 locally computed
+md5     a2bc8d28c923b4fe2b2c3b4b51a4f935  pycparser-2.14.tar.gz
+sha256  7959b4a74abdc27b312fed1c21e6caf9309ce0b29ea86b591fd2e99ecdf27f73  pycparser-2.14.tar.gz
package/python-pycparser/python-pycparser.mk new file +14 -0
new file mode 100644
index 0000000..328bc8a
@@ -0,0 +1,14 @@
+################################################################################
+#
+# python-pycparser
+#
+################################################################################
+
+PYTHON_PYCPARSER_VERSION = 2.14
+PYTHON_PYCPARSER_SOURCE = pycparser-$(PYTHON_PYCPARSER_VERSION).tar.gz
+PYTHON_PYCPARSER_SITE = https://pypi.python.org/packages/source/p/pycparser
+PYTHON_PYCPARSER_SETUP_TYPE = setuptools
+PYTHON_PYCPARSER_LICENSE = BSD-3c
+PYTHON_PYCPARSER_LICENSE_FILES = LICENSE
+
+$(eval $(host-python-package))
Signature · 1 line
2.1.4
#4 lore raw

PATCH v2 2/3 python-pycparser: new package

Dear yegorslists at googlemail.com,

Quoted 9 lines · yegorslists at googlemail.com

From: Yegor Yefremov <[email protected]>

pycparser requires C preprocessor in order to parse C related files, hence available only as host package.

---
Changes:
v2: remove target package variant

Applied, thanks.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#5 lore raw

PATCH v2 3/3 python-cffi: add host variant

In reply to yegorslists at googlemail.com

From: Yegor Yefremov <[email protected]>

Host variant is needed to cross-compile CFFI based C library wrappers.

---
Changes:
v2: add host-libffi dependency

 package/python-cffi/python-cffi.mk | 2 ++
 1 file changed, 2 insertions(+)
package/python-cffi/python-cffi.mk +2 -0
index 70fbe02..02c969b 100644
@@ -11,5 +11,7 @@ PYTHON_CFFI_SETUP_TYPE = setuptools
 PYTHON_CFFI_DEPENDENCIES = host-pkgconf libffi
 PYTHON_CFFI_LICENSE = MIT
 PYTHON_CFFI_LICENSE_FILES = LICENSE
+HOST_PYTHON_CFFI_DEPENDENCIES = host-python-pycparser host-libffi
 
 $(eval $(python-package))
+$(eval $(host-python-package))
Signature · 1 line
2.1.4
#6 lore raw

PATCH v2 3/3 python-cffi: add host variant

Hello Thomas, Yann,

Quoted 17 lines · On Tue, Jan 5, 2016 at 8:17 AM, <[email protected]> wrote:

From: Yegor Yefremov <[email protected]>

Host variant is needed to cross-compile CFFI based C library wrappers.

---
Changes:
v2: add host-libffi dependency

 package/python-cffi/python-cffi.mk | 2 ++
 1 file changed, 2 insertions(+)
package/python-cffi/python-cffi.mk +0 -0
index 70fbe02..02c969b 100644

package/python-cffi/python-cffi.mk PYTHON_CFFI_SETUP_TYPE = setuptools

Quoted 8 lines
package/python-cffi/python-cffi.mk +2 -1
@@ -11,5 +11,7 @@ PYTHON_CFFI_SETUP_TYPE = setuptools
 PYTHON_CFFI_DEPENDENCIES = host-pkgconf libffi
 PYTHON_CFFI_LICENSE = MIT
 PYTHON_CFFI_LICENSE_FILES = LICENSE
+HOST_PYTHON_CFFI_DEPENDENCIES = host-python-pycparser host-libffi
 
 $(eval $(python-package))
+$(eval $(host-python-package))
--

I have now following theory regarding checking dependencies in target and host variant.

If you look at python-tornado's setup.py file [1], you see, that they say, install_requires will be checked only for setuptools. This is also the case for host-python-cffi. The target variant will be installed with --single-version-externally-managed option, that performs "old-style" install [2] [3], i.e. without checking install_requires.

That's my understanding for now.

[1] https://github.com/tornadoweb/tornado/blob/master/setup.py#L126
[2] https://pythonhosted.org/setuptools/setuptools.html
[3] https://bitbucket.org/pypa/setuptools/src/e517fced669d54bcaf3da2d8ca468d29120fedfc/setuptools/command/install.py?at=default&fileviewer=file-view-default#install.py-58

Yegor

#7 lore raw

PATCH v2 3/3 python-cffi: add host variant

Quoted 39 lines · Yegor Yefremov

Hello Thomas, Yann,

Quoted 17 lines · On Tue, Jan 5, 2016 at 8:17 AM, <[email protected]> wrote:

From: Yegor Yefremov <[email protected]>

Host variant is needed to cross-compile CFFI based C library wrappers.

---
Changes:
v2: add host-libffi dependency

 package/python-cffi/python-cffi.mk | 2 ++
 1 file changed, 2 insertions(+)
package/python-cffi/python-cffi.mk +0 -0
index 70fbe02..02c969b 100644

package/python-cffi/python-cffi.mk PYTHON_CFFI_SETUP_TYPE = setuptools

Quoted 8 lines
package/python-cffi/python-cffi.mk +2 -1
@@ -11,5 +11,7 @@ PYTHON_CFFI_SETUP_TYPE = setuptools
 PYTHON_CFFI_DEPENDENCIES = host-pkgconf libffi
 PYTHON_CFFI_LICENSE = MIT
 PYTHON_CFFI_LICENSE_FILES = LICENSE
+HOST_PYTHON_CFFI_DEPENDENCIES = host-python-pycparser host-libffi
 
 $(eval $(python-package))
+$(eval $(host-python-package))
--

I have now following theory regarding checking dependencies in target and host variant.

If you look at python-tornado's setup.py file [1], you see, that they say, install_requires will be checked only for setuptools. This is also the case for host-python-cffi. The target variant will be installed with --single-version-externally-managed option, that performs "old-style" install [2] [3], i.e. without checking install_requires.

Quoted 5 lines

That's my understanding for now.

[1] https://github.com/tornadoweb/tornado/blob/master/setup.py#L126
[2] https://pythonhosted.org/setuptools/setuptools.html
[3] https://bitbucket.org/pypa/setuptools/src/e517fced669d54bcaf3da2d8ca468d29120fedfc/setuptools/command/install.py?at=default&fileviewer=file-view-default#install.py-58

This SO answer also points to this conclusion:
http://stackoverflow.com/a/22179371/1113139

Yegor

#8 lore raw

PATCH v2 3/3 python-cffi: add host variant

In reply to yegorslists at googlemail.com

Hello,

Quoted 17 lines · yegorslists at googlemail.com

From: Yegor Yefremov <[email protected]>

Host variant is needed to cross-compile CFFI based C library wrappers.

---
Changes:
v2: add host-libffi dependency

 package/python-cffi/python-cffi.mk | 2 ++
 1 file changed, 2 insertions(+)
package/python-cffi/python-cffi.mk +0 -0
index 70fbe02..02c969b 100644

package/python-cffi/python-cffi.mk PYTHON_CFFI_SETUP_TYPE = setuptools · line 17

Quoted 7 lines
package/python-cffi/python-cffi.mk +2 -0
@@ -11,5 +11,7 @@ PYTHON_CFFI_SETUP_TYPE = setuptools
 PYTHON_CFFI_DEPENDENCIES = host-pkgconf libffi
 PYTHON_CFFI_LICENSE = MIT
 PYTHON_CFFI_LICENSE_FILES = LICENSE
+HOST_PYTHON_CFFI_DEPENDENCIES = host-python-pycparser host-libffi
 
 $(eval $(python-package))
+$(eval $(host-python-package))

This wasn't building properly here, because python-cffi uses pkg-config to get the flags to link with libffi, and by default our pkg-config returns results that are valid to build stuff for the target, not for the host.

So, I've fixed that by passing a bunch of pkg-config environment variables when building the host package.

Also, you were missing the host-pkgconf dependency for the dependencies of host variant.

Committed with those issues fixed.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com