Skip to contentSkip to conversation

pyzmq: new package

PATCH New Michael Rommel 4 participants 5 messages Aug 30, 2013 – Sep 2, 2013

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pyzmq: new package

Hi,

as promised, here is the separate patch for pyzmq.

Michael.

PS: I noticed, that I missed in the previous e-mail to change the Config.in and remove the pyzmq line from it. Do I need to re-submit the patch for knock?

The python language bindings for zeromq.

Please note that the reason for the hardcoded version number in the python-pyzmq package is due to the following dilemma:
- the setup.py script tries to compile a test C program and runs it, to retrieve a version string
- if the cross compiler links it together, the result cannot be run on the host, due to different architectures
- if the host compiler would compile/link it, it would not link with the library version inside buildroot but with the library from the host

---
 package/Config.in                |    1 +
 package/python-pyzmq/Config.in   |   13 ++++++++++
 package/python-pyzmq/pyzmq.mk    |   50 ++++++++++++++++++++++++++++++++++++++
 package/python-pyzmq/pyzmq.patch |   28 +++++++++++++++++++++
 4 files changed, 92 insertions(+)
 create mode 100644 package/python-pyzmq/Config.in
 create mode 100644 package/python-pyzmq/pyzmq.mk
 create mode 100644 package/python-pyzmq/pyzmq.patch
package/Config.in +1 -0
index 97cd7da..0289fd1 100644
@@ -363,6 +363,7 @@ source "package/python-protobuf/Config.in"
 source "package/python-pygame/Config.in"
 source "package/python-pyparsing/Config.in"
 source "package/python-pyro/Config.in"
+source "package/python-pyzmq/Config.in"
 source "package/python-serial/Config.in"
 source "package/python-setuptools/Config.in"
 source "package/python-thrift/Config.in"
package/python-pyzmq/Config.in new file +13 -0
new file mode 100644
index 0000000..d9791c1
@@ -0,0 +1,13 @@
+config BR2_PACKAGE_PYZMQ
+	bool "pyzmq"
+	select BR2_PACKAGE_ZEROMQ
+        select BR2_PACKAGE_PYTHON
+	depends on BR2_USE_WCHAR
+	depends on BR2_USE_MMU
+	help
+	  This package contains the python language binding for zeromq.
+	  Due to issues with cross-compiling this package is hardcoded
+	  to ZeroMQ version 3.2.3
+
+	  See also: http://zeromq.org/bindings:python
+
package/python-pyzmq/pyzmq.mk new file +50 -0

50 lines not on this page · raw message

package/python-pyzmq/pyzmq.patch new file +28 -0
new file mode 100644
index 0000000..c660457
@@ -0,0 +1,28 @@
+--- pyzmq-13.0.2/buildutils/detect.py.orig	2013-08-10 00:49:28.242557978 +0200
++++ pyzmq-13.0.2/buildutils/detect.py	2013-08-10 00:44:35.197572704 +0200
+@@ -118,15 +118,17 @@ def detect_zmq(basedir, compiler=None, *
+             
+     efile = test_compilation(cfile, compiler=compiler, **compiler_attrs)
+     
+-    result = Popen(efile, stdout=PIPE, stderr=PIPE)
+-    so, se = result.communicate()
++    # result = Popen(efile, stdout=PIPE, stderr=PIPE)
++    # so, se = result.communicate()
+     # for py3k:
+-    so = so.decode()
+-    se = se.decode()
+-    if result.returncode:
+-        msg = "Error running version detection script:\n%s\n%s" % (so,se)
+-        logging.error(msg)
+-        raise IOError(msg)
++    #so = so.decode()
++    #se = se.decode()
++    #if result.returncode:
++    #    msg = "Error running version detection script:\n%s\n%s" % (so,se)
++    #    logging.error(msg)
++    #    raise IOError(msg)
++
++    so = "vers: ##PYZMQ_ZMQ_VERSION##"
+ 
+     handlers = {'vers':  lambda val: tuple(int(v) for v in val.split('.'))}
+ 
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH 1/1 pyzmq: new package

Quoted 7 lines · Michael Rommel

Hi,

as promised, here is the separate patch for pyzmq.

Michael.

PS: I noticed, that I missed in the previous e-mail to change the Config.in and remove the pyzmq line from it. Do I need to re-submit the patch for knock?

Yes, please send an updated patch taking into account the comments you received. As I said in the knock package, this explanation should be below '---'.

Quoted 16 lines

The python language bindings for zeromq.

Please note that the reason for the hardcoded version number in the python-pyzmq package is due to the following dilemma:
- the setup.py script tries to compile a test C program and runs it, to retrieve a version string
- if the cross compiler links it together, the result cannot be run on the host, due to different architectures
- if the host compiler would compile/link it, it would not link with the library version inside buildroot but with the library from the host

Quoted 9 lines
---
 package/Config.in                |    1 +
 package/python-pyzmq/Config.in   |   13 ++++++++++
 package/python-pyzmq/pyzmq.mk    |   50 ++++++++++++++++++++++++++++++++++++++
 package/python-pyzmq/pyzmq.patch |   28 +++++++++++++++++++++
 4 files changed, 92 insertions(+)
 create mode 100644 package/python-pyzmq/Config.in
 create mode 100644 package/python-pyzmq/pyzmq.mk
 create mode 100644 package/python-pyzmq/pyzmq.patch

Patches should be named according to this format:
<packagename>-<number>-<description>.patch, see
http://buildroot.uclibc.org/downloads/manual/manual.html#patch-policy

Quoted 19 lines
package/Config.in +1 -0
index 97cd7da..0289fd1 100644
@@ -363,6 +363,7 @@ source "package/python-protobuf/Config.in"
 source "package/python-pygame/Config.in"
 source "package/python-pyparsing/Config.in"
 source "package/python-pyro/Config.in"
+source "package/python-pyzmq/Config.in"
 source "package/python-serial/Config.in"
 source "package/python-setuptools/Config.in"
 source "package/python-thrift/Config.in"
package/python-pyzmq/Config.in new file +0 -0
new file mode 100644
index 0000000..d9791c1

package/python-pyzmq/Config.in line 4

Quoted 4 lines
package/python-pyzmq/Config.in new file +4 -0
@@ -0,0 +1,13 @@
+config BR2_PACKAGE_PYZMQ
+       bool "pyzmq"
+       select BR2_PACKAGE_ZEROMQ
+        select BR2_PACKAGE_PYTHON

This line is indented with spaces instead of one tab.

You will need to add some other lines here, because both zeromq as python depend on other packages, and as explained here:
http://buildroot.uclibc.org/downloads/manual/manual.html#depends-on-vs-select,
you will need to replicate these dependencies in this package.

package/python-pyzmq/Config.in line 6

Quoted 2 lines
package/python-pyzmq/Config.in new file +2 -0
+       depends on BR2_USE_WCHAR
+       depends on BR2_USE_MMU

For this type of dependencies, we generally add a comment in case the dependencies are not met, warning the user about this. Have a look at some other packages to get an idea.

Quoted 13 lines
package/python-pyzmq/Config.in new file +7 -0
+       help
+         This package contains the python language binding for zeromq.
+         Due to issues with cross-compiling this package is hardcoded
+         to ZeroMQ version 3.2.3
+
+         See also: http://zeromq.org/bindings:python
+
package/python-pyzmq/pyzmq.mk new file +0 -0
new file mode 100644
index 0000000..a7efcad

package/python-pyzmq/pyzmq.mk line 8

Quoted 8 lines
package/python-pyzmq/pyzmq.mk new file +8 -0
@@ -0,0 +1,50 @@
+##########################################
+#
+# pyzmq
+#
+##########################################
+
+PYZMQ_VERSION = 13.0.2
+PYZMQ_ZMQ_VERSION = 3.2.3

I haven't thought about better ways to do this, so I'll take it as a given that you need to have a fixed version.
But have you considered using $(ZEROMQ_VERSION) instead of the hardcoded 3.2.3. This way, pyzmq will not have to be updated when the zeromq version is bumped.

package/python-pyzmq/pyzmq.mk line 9

Quoted 1 line
package/python-pyzmq/pyzmq.mk new file +1 -0
+PYZMQ_SOURCE = pyzmq-$(PYZMQ_VERSION).tar.gz

This format is the default value for FOO_SOURCE, so this line can be ommitted.

package/python-pyzmq/pyzmq.mk line 13

Quoted 4 lines
package/python-pyzmq/pyzmq.mk new file +4 -0
+PYZMQ_SITE = https://pypi.python.org/packages/source/p/pyzmq/
+PYZMQ_LICENSE = LGPLv3 BSD-3c
+PYZMQ_LICENSE_FILES = COPYING.LESSER COPYING.BSD
+PYZMQ_INSTALL_TARGET = YES

FOO_INSTALL_TARGET = YES is the default, so line can be ommitted.

Quoted 12 lines
package/python-pyzmq/pyzmq.mk new file +12 -0
+
+define PYZMQ_POST_PATCH_FIXUP
+        (cd $(@D); \
+               cat buildutils/detect.py | \
+               sed -e "s/##PYZMQ_ZMQ_VERSION##/$(PYZMQ_ZMQ_VERSION)/" \
+                 >buildutils/detect.py.new; \
+               cp buildutils/detect.py.new buildutils/detect.py)
+endef
+
+PYZMQ_POST_PATCH_HOOKS += PYZMQ_POST_PATCH_FIXUP
+
+define PYZMQ_CONFIGURE_CMDS

package/python-pyzmq/pyzmq.mk line 35

Quoted 10 lines
package/python-pyzmq/pyzmq.mk new file +10 -0
+        (cd $(@D); CC="$(TARGET_CC)" CFLAGS="$(TARGET_CFLAGS)" \
+                LDSHARED="$(TARGET_CROSS)gcc -shared" \
+                CROSS_COMPILING=yes \
+                _python_sysroot=$(STAGING_DIR) \
+                _python_srcdir=$(BUILD_DIR)/python$(PYTHON_VERSION) \
+                _python_prefix=/usr \
+                _python_exec_prefix=/usr \
+                $(HOST_DIR)/usr/bin/python setup.py configure \
+               --zmq=$(@D)/../../staging/usr)
+endef

Python and zeromq will already have to be installed before pyzmq is built, so you will have to add a line like:
PYZMQ_DEPENDENCIES = python zeromq

Quoted 5 lines
package/python-pyzmq/pyzmq.mk new file +5 -0
+
+define PYZMQ_INSTALL_TARGET_CMDS
+        (cd $(@D); CC="$(TARGET_CC)" CFLAGS="$(TARGET_CFLAGS)" \
+                LDSHARED="$(TARGET_CROSS)gcc -shared" \
+                CROSS_COMPILING=yes \

package/python-pyzmq/pyzmq.mk line 50

Quoted 10 lines
package/python-pyzmq/pyzmq.mk new file +10 -0
+                _python_sysroot=$(STAGING_DIR) \
+                _python_srcdir=$(BUILD_DIR)/python$(PYTHON_VERSION) \
+                _python_prefix=/usr \
+                _python_exec_prefix=/usr \
+                $(HOST_DIR)/usr/bin/python setup.py install \
+               --prefix=$(TARGET_DIR)/usr)
+endef
+
+$(eval $(generic-package))
+

This empty line can be removed.

package/python-pyzmq/pyzmq.patch

Quoted 5 lines
package/python-pyzmq/pyzmq.patch new file +0 -0
new file mode 100644
index 0000000..c660457

Also check the manual about patches here: the patch should have a different name, and should contain an explanation and signed-off-by line.

Quoted 19 lines
package/python-pyzmq/pyzmq.patch new file +18 -0
@@ -0,0 +1,28 @@
+--- pyzmq-13.0.2/buildutils/detect.py.orig     2013-08-10 00:49:28.242557978 +0200
++++ pyzmq-13.0.2/buildutils/detect.py  2013-08-10 00:44:35.197572704 +0200
+@@ -118,15 +118,17 @@ def detect_zmq(basedir, compiler=None, *
+
+     efile = test_compilation(cfile, compiler=compiler, **compiler_attrs)
+
+-    result = Popen(efile, stdout=PIPE, stderr=PIPE)
+-    so, se = result.communicate()
++    # result = Popen(efile, stdout=PIPE, stderr=PIPE)
++    # so, se = result.communicate()
+     # for py3k:
+-    so = so.decode()
+-    se = se.decode()
+-    if result.returncode:
+-        msg = "Error running version detection script:\n%s\n%s" % (so,se)
+-        logging.error(msg)
+-        raise IOError(msg)
++    #so = so.decode()

package/python-pyzmq/pyzmq.patch line 28

Quoted 10 lines
package/python-pyzmq/pyzmq.patch new file +10 -0
++    #se = se.decode()
++    #if result.returncode:
++    #    msg = "Error running version detection script:\n%s\n%s" % (so,se)
++    #    logging.error(msg)
++    #    raise IOError(msg)
++
++    so = "vers: ##PYZMQ_ZMQ_VERSION##"
+
+     handlers = {'vers':  lambda val: tuple(int(v) for v in val.split('.'))}
+

Thanks for both of your contributions!

Best regards,
Thomas

#2 lore raw

PATCH 1/1 pyzmq: new package

Michael, Thomas,

A few comments below.

Quoted 8 lines · Thomas De Schampheleire
Quoted 8 lines

Please note that the reason for the hardcoded version number in the python-pyzmq package is due to the following dilemma:
- the setup.py script tries to compile a test C program and runs it, to retrieve a version string
- if the cross compiler links it together, the result cannot be run on the host, due to different architectures
- if the host compiler would compile/link it, it would not link with the library version inside buildroot but with the library from the host

I believe such explanations fall inside the .mk file itself, so that people looking at your code later can understand this 'bizarre' aspect of your package.

Quoted 6 lines

package/python-pyzmq/Config.in line 6

Quoted 2 lines
package/python-pyzmq/Config.in new file +2 -0
+       depends on BR2_USE_WCHAR
+       depends on BR2_USE_MMU

For this type of dependencies, we generally add a comment in case the dependencies are not met, warning the user about this. Have a look at some other packages to get an idea.

That's true for the WCHAR dependency, but not for the MMU dependency. So the comment should be:

comment "pyzmq requires wchar support in toolchain"
depends on !BR2_USE_WCHAR

Quoted 4 lines

package/python-pyzmq/Config.in line 10

Quoted 4 lines
package/python-pyzmq/Config.in new file +4 -0
+       help
+         This package contains the python language binding for zeromq.
+         Due to issues with cross-compiling this package is hardcoded
+         to ZeroMQ version 3.2.3

I wouldn't mention the version of ZeroMQ here, as it will most likely not be updated when ZeroMQ is upgraded in Buildroot.

Quoted 1 line
Quoted 1 line

+ See also: http://zeromq.org/bindings:python

No need for 'See also:', just put the upstream URL.

Quoted 11 lines

package/python-pyzmq/pyzmq.mk line 5

Quoted 11 lines
package/python-pyzmq/pyzmq.mk new file +5 -0
new file mode 100644
index 0000000..a7efcad
@@ -0,0 +1,50 @@
+##########################################
+#
+# pyzmq
+#
+##########################################

80 '#' should be used.

Quoted 8 lines

package/python-pyzmq/pyzmq.mk line 8

Quoted 2 lines
package/python-pyzmq/pyzmq.mk new file +2 -0
+PYZMQ_VERSION = 13.0.2
+PYZMQ_ZMQ_VERSION = 3.2.3

I haven't thought about better ways to do this, so I'll take it as a given that you need to have a fixed version.
But have you considered using $(ZEROMQ_VERSION) instead of the hardcoded 3.2.3. This way, pyzmq will not have to be updated when the zeromq version is bumped.

I agree.

Quoted 8 lines

package/python-pyzmq/pyzmq.mk line 9

Quoted 2 lines
package/python-pyzmq/pyzmq.mk new file +1 -0
+PYZMQ_SOURCE = pyzmq-$(PYZMQ_VERSION).tar.gz

This format is the default value for FOO_SOURCE, so this line can be ommitted.

package/python-pyzmq/pyzmq.mk line 12

Quoted 3 lines
package/python-pyzmq/pyzmq.mk new file +3 -0
+PYZMQ_SITE = https://pypi.python.org/packages/source/p/pyzmq/
+PYZMQ_LICENSE = LGPLv3 BSD-3c
+PYZMQ_LICENSE_FILES = COPYING.LESSER COPYING.BSD

Check whether it's really LGPLv3 or LGPLv3+ (i.e, whether it's LGPL version 3 only, or LGPL version 3 or later).

Quoted 5 lines

package/python-pyzmq/pyzmq.mk line 13

Quoted 1 line
package/python-pyzmq/pyzmq.mk new file +1 -0
+PYZMQ_INSTALL_TARGET = YES

FOO_INSTALL_TARGET = YES is the default, so line can be ommitted.

Quoted 7 lines

package/python-pyzmq/pyzmq.mk line 20

Quoted 7 lines
package/python-pyzmq/pyzmq.mk new file +7 -0
+
+define PYZMQ_POST_PATCH_FIXUP
+        (cd $(@D); \
+               cat buildutils/detect.py | \
+               sed -e "s/##PYZMQ_ZMQ_VERSION##/$(PYZMQ_ZMQ_VERSION)/" \
+                 >buildutils/detect.py.new; \
+               cp buildutils/detect.py.new buildutils/detect.py)

I think you should use $(SED) to do an in-place replacement. See for example what package/dmalloc/dmalloc.mk is doing. Something like:

	$(SED) 's/##PYZMQ_ZMQ_VERSION##/$(PYZMQ_ZMQ_VERSION)/' 	$(@D)/buildutils/detect.py
Quoted 7 lines

package/python-pyzmq/pyzmq.mk line 27

Quoted 7 lines
package/python-pyzmq/pyzmq.mk new file +7 -0
+endef
+
+PYZMQ_POST_PATCH_HOOKS += PYZMQ_POST_PATCH_FIXUP
+
+define PYZMQ_CONFIGURE_CMDS
+        (cd $(@D); CC="$(TARGET_CC)" CFLAGS="$(TARGET_CFLAGS)" \
+                LDSHARED="$(TARGET_CROSS)gcc -shared" \

LDSHARED="$(TARGET_CC) -shared" would be preferable I believe.

Quoted 7 lines

package/python-pyzmq/pyzmq.mk line 34

Quoted 7 lines
package/python-pyzmq/pyzmq.mk new file +7 -0
+                CROSS_COMPILING=yes \
+                _python_sysroot=$(STAGING_DIR) \
+                _python_srcdir=$(BUILD_DIR)/python$(PYTHON_VERSION) \
+                _python_prefix=/usr \
+                _python_exec_prefix=/usr \
+                $(HOST_DIR)/usr/bin/python setup.py configure \
+               --zmq=$(@D)/../../staging/usr)

No, this should be --zmq=$(STAGING_DIR)/usr

Quoted 6 lines

package/python-pyzmq/pyzmq.mk line 34

Quoted 1 line
package/python-pyzmq/pyzmq.mk new file +1 -0
+endef

Python and zeromq will already have to be installed before pyzmq is built, so you will have to add a line like:
PYZMQ_DEPENDENCIES = python zeromq

Quoted 11 lines

package/python-pyzmq/pyzmq.mk line 46

Quoted 11 lines
package/python-pyzmq/pyzmq.mk new file +11 -0
+
+define PYZMQ_INSTALL_TARGET_CMDS
+        (cd $(@D); CC="$(TARGET_CC)" CFLAGS="$(TARGET_CFLAGS)" \
+                LDSHARED="$(TARGET_CROSS)gcc -shared" \
+                CROSS_COMPILING=yes \
+                _python_sysroot=$(STAGING_DIR) \
+                _python_srcdir=$(BUILD_DIR)/python$(PYTHON_VERSION) \
+                _python_prefix=/usr \
+                _python_exec_prefix=/usr \
+                $(HOST_DIR)/usr/bin/python setup.py install \
+               --prefix=$(TARGET_DIR)/usr)

This prefix looks wrong. The prefix should normally always be '/usr'.

You also have lots of duplicated variables between the CONFIGURE_CMDS and INSTALL_TARGET_CMDS, it'd be great to factorize them.

Thanks!

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#3 lore raw

PATCH 1/1 pyzmq: new package

In reply to Michael Rommel

On 08/30/13 23:58, Michael Rommel wrote:
[snip]

package/python-pyzmq/Config.in line 4

Quoted 10 lines
package/python-pyzmq/Config.in new file +4 -0
new file mode 100644
index 0000000..d9791c1
@@ -0,0 +1,13 @@
+config BR2_PACKAGE_PYZMQ
+	bool "pyzmq"
+	select BR2_PACKAGE_ZEROMQ
+        select BR2_PACKAGE_PYTHON

For python packages, we use a 'depends on' here rather than a select.

Regards,
Arnout

Quoted 2 lines
package/python-pyzmq/Config.in new file +2 -0
+	depends on BR2_USE_WCHAR
+	depends on BR2_USE_MMU

[snip]

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#4 lore raw

PATCH 1/1 pyzmq: new package

Hi Arnout,

I'll test that again today in the evening. IIRC "make savedefconfig" complained
about a cyclic dependency, but I need to check that again. Thanks for pointing
that out!

Michael.

Quoted 22 lines · Arnout Vandecappelle

On 08/30/13 23:58, Michael Rommel wrote:
[snip]

package/python-pyzmq/Config.in line 4

Quoted 10 lines
package/python-pyzmq/Config.in new file +4 -0
new file mode 100644
index 0000000..d9791c1
@@ -0,0 +1,13 @@
+config BR2_PACKAGE_PYZMQ
+	bool "pyzmq"
+	select BR2_PACKAGE_ZEROMQ
+        select BR2_PACKAGE_PYTHON

For python packages, we use a 'depends on' here rather than a select.

Regards,
Arnout

Quoted 2 lines
package/python-pyzmq/Config.in new file +2 -0
+	depends on BR2_USE_WCHAR
+	depends on BR2_USE_MMU

[snip]

Signature · 1 line
Michael Rommel, Erlangen, Germany