new Avro packages
v4 cover new Avro packages
Apache Avro is a data serialization system, somewhat similar to protobuff or cap'n proto. Unlike them however, Avro schemas can be distributed in JSON files to be loaded at runtime, rather than language-specific generated code to use at compile time, even though the latter is also possible with Avro.
This series adds 2 packages for the Avro serialization format:
* 1 package for the C language bindings and files manipulation utilities
* 1 package for the Python3 language bindings, and a file manipulation utility
There are also Avro bindings for other languages such as C++, C#, Java, Javascript, Perl, PHP or Ruby. However, because I am not familiar with these ones and not using them, they are not part of this initial series.
---
Changes v1->v2:
* Use per-language archives from upstream
* Split new packages in multiple patches
* Keep my DEVELOPERS packages in alphabetical order
Changes v2->v3:
* avro-c depends on zlib, snappy if they are enabled
* NOTICE is not part of the license files
* fix musl patch format for avro-c
* display the prompt "avro-c" in menuconfig and add comment for shared libs
* add upstream url in Config.in
Changes v3->v4:
* Update upstream patch status
* avro-c needs host-pkgconf
* avro-c depends on xz when selected, enabling the lzma compression codec
---
Titouan Christophe (2):
package/avro-c: new package
package/python-avro: new package
DEVELOPERS | 2 ++ package/Config.in | 2 ++ package/avro-c/0001-Compile-on-musl.patch | 40 +++++++++++++++++++++++ package/avro-c/Config.in | 25 ++++++++++++++ package/avro-c/avro-c.hash | 5 +++ package/avro-c/avro-c.mk | 26 +++++++++++++++ package/python-avro/Config.in | 20 ++++++++++++ package/python-avro/python-avro.hash | 5 +++ package/python-avro/python-avro.mk | 14 ++++++++ 9 files changed, 139 insertions(+) create mode 100644 package/avro-c/0001-Compile-on-musl.patch create mode 100644 package/avro-c/Config.in create mode 100644 package/avro-c/avro-c.hash create mode 100644 package/avro-c/avro-c.mk create mode 100644 package/python-avro/Config.in create mode 100644 package/python-avro/python-avro.hash create mode 100644 package/python-avro/python-avro.mk
Signature · 1 line
2.23.0
PATCH v4 1/2 package/avro-c: new package
---
Dropped reviewed-by from Gilles because of changes in AVRO_C_DEPENDENCIES
--- DEVELOPERS | 1 + package/Config.in | 1 + package/avro-c/0001-Compile-on-musl.patch | 40 +++++++++++++++++++++++ package/avro-c/Config.in | 25 ++++++++++++++ package/avro-c/avro-c.hash | 5 +++ package/avro-c/avro-c.mk | 26 +++++++++++++++ 6 files changed, 98 insertions(+) create mode 100644 package/avro-c/0001-Compile-on-musl.patch create mode 100644 package/avro-c/Config.in create mode 100644 package/avro-c/avro-c.hash create mode 100644 package/avro-c/avro-c.mk
DEVELOPERS +1 -0
@@ -2418,6 +2418,7 @@ N: Timo Ketola <[email protected]> F: package/fbgrab/ N: Titouan Christophe <[email protected]> +F: package/avro-c/ F: package/mosquitto/ F: package/redis/
package/Config.in +1 -0
@@ -1724,6 +1724,7 @@ menu "Other" source "package/argp-standalone/Config.in" source "package/armadillo/Config.in" source "package/atf/Config.in" + source "package/avro-c/Config.in" source "package/bctoolbox/Config.in" source "package/bdwgc/Config.in" source "package/boost/Config.in"
package/avro-c/0001-Compile-on-musl.patch new file +40 -0
@@ -0,0 +1,40 @@ +From 4ada709e44e26a329d11359f10ab208ce758d863 Mon Sep 17 00:00:00 2001 +From: Titouan Christophe <[email protected]> +Date: Sun, 8 Dec 2019 01:55:59 +0100 +Subject: [PATCH] Allow avro C to be built on musl based systems. + +The type 'ssize_t' is defined in sys/types.h, and nowhere else +in the musl standard C library, so it should be included for the +compilation to succeed. + +This fixes several errors like: + + In file included from src/generic.c:29:0: + src/generic.c: In function 'avro_generic_value_new': + src/avro_generic_internal.h:63:39: + error: 'ssize_t' undeclared (first use in this function); + did you mean 'size_t'? + +Signed-off-by: Titouan Christophe <[email protected]> +[Upstream status: accepted, not yet released + https://github.com/apache/avro/pull/740] +--- + src/avro_generic_internal.h | 2 ++ + 1 file changed, 2 insertions(+) + +diff --git a/src/avro_generic_internal.h b/src/avro_generic_internal.h +index 709403c0..9843ed65 100644 +--- a/src/avro_generic_internal.h ++++ b/src/avro_generic_internal.h +@@ -24,6 +24,8 @@ extern "C" { + #define CLOSE_EXTERN + #endif + ++#include <sys/types.h> ++ + #include "avro/generic.h" + #include "avro/schema.h" + #include "avro/value.h" +-- +2.23.0 +
package/avro-c/Config.in new file +25 -0
@@ -0,0 +1,25 @@ +comment "avro-c needs a toolchain w/ dynamic library" + depends on BR2_STATIC_LIBS + +config BR2_PACKAGE_AVRO_C + bool "avro-c" + depends on !BR2_STATIC_LIBS # Avro unconditionally compiles shared libs + select BR2_PACKAGE_JANSSON + help + Select this option to install the Avro C language bindings, + and the command line tools avroappend, avrocat, avromod and + avropipe. + + Apache Avro is a data serialization system. + Avro provides: + - Rich data structures. + - A compact, fast, binary data format. + - A container file, to store persistent data. + - Remote procedure call (RPC). + - Simple integration with dynamic languages. + - Code generation is not required to read or write data + files nor to use or implement RPC protocols. + - Code generation as an optional optimization, + only worth implementing for statically typed languages. + + http://avro.apache.org/
package/avro-c/avro-c.hash new file +5 -0
@@ -0,0 +1,5 @@ +# From https://www-eu.apache.org/dist/avro/avro-1.9.1/c/avro-c-1.9.1.tar.gz.sha512 +sha512 68b1f44f870c9b6f0b2380da9e34d91148ff4398cb300f4bdd4e3e1ad00820acd9084b73232b00d4cd4935fb992b41dc65afdafcbea14a3d87259608688df904 avro-c-1.9.1.tar.gz + +# License files +sha256 d62488d6ba17132e92c23c03c80bfedc848267f96ab36489fec860f76cf6819a LICENSE
package/avro-c/avro-c.mk new file +26 -0
@@ -0,0 +1,26 @@ +################################################################################ +# +# avro-c +# +################################################################################ + +AVRO_C_VERSION = 1.9.1 +AVRO_C_SITE = https://www-eu.apache.org/dist/avro/avro-$(AVRO_C_VERSION)/c +AVRO_C_LICENSE = Apache-2.0 +AVRO_C_LICENSE_FILES = LICENSE +AVRO_C_INSTALL_STAGING = YES +AVRO_C_DEPENDENCIES = host-pkgconf jansson + +ifeq ($(BR2_PACKAGE_SNAPPY),y) +AVRO_C_DEPENDENCIES += snappy +endif + +ifeq ($(BR2_PACKAGE_ZLIB),y) +AVRO_C_DEPENDENCIES += zlib +endif + +ifeq ($(BR2_PACKAGE_XZ),y) +AVRO_C_DEPENDENCIES += xz +endif + +$(eval $(cmake-package))
Signature · 1 line
2.23.0
PATCH v4 1/2 package/avro-c: new package
Titouan, All,
On 2019-12-23 18:23 +0100, Titouan Christophe spake thusly:
Quoted 1 line
Signed-off-by Titouan Christophe <[email protected]>
The patch was accepted and merged upstreram, so I backported it directly, adapted the comment, and applied to master. Thanks!
Regards,
Yann E. MORIN.
Quoted 162 lines
---
Dropped reviewed-by from Gilles because of changes inAVRO_C_DEPENDENCIES--- DEVELOPERS | 1 + package/Config.in | 1 + package/avro-c/0001-Compile-on-musl.patch | 40 +++++++++++++++++++++++ package/avro-c/Config.in | 25 ++++++++++++++ package/avro-c/avro-c.hash | 5 +++ package/avro-c/avro-c.mk | 26 +++++++++++++++ 6 files changed, 98 insertions(+) create mode 100644 package/avro-c/0001-Compile-on-musl.patch create mode 100644 package/avro-c/Config.in create mode 100644 package/avro-c/avro-c.hash create mode 100644 package/avro-c/avro-c.mkDEVELOPERS +1
-0@@ -2418,6 +2418,7 @@ N: Timo Ketola <[email protected]> F: package/fbgrab/ N: Titouan Christophe <[email protected]> +F: package/avro-c/ F: package/mosquitto/ F: package/redis/package/Config.in +1
-0@@ -1724,6 +1724,7 @@ menu "Other" source "package/argp-standalone/Config.in" source "package/armadillo/Config.in" source "package/atf/Config.in" + source "package/avro-c/Config.in" source "package/bctoolbox/Config.in" source "package/bdwgc/Config.in" source "package/boost/Config.in"package/avro-c/0001-Compile-on-musl.patch new file +40
-0@@ -0,0 +1,40 @@ +From 4ada709e44e26a329d11359f10ab208ce758d863 Mon Sep 17 00:00:00 2001 +From: Titouan Christophe <[email protected]> +Date: Sun, 8 Dec 2019 01:55:59 +0100 +Subject: [PATCH] Allow avro C to be built on musl based systems. + +The type 'ssize_t' is defined in sys/types.h, and nowhere else +in the musl standard C library, so it should be included for the +compilation to succeed. + +This fixes several errors like: + + In file included from src/generic.c:29:0: + src/generic.c: In function 'avro_generic_value_new': + src/avro_generic_internal.h:63:39: + error: 'ssize_t' undeclared (first use in this function); + did you mean 'size_t'? + +Signed-off-by: Titouan Christophe <[email protected]> +[Upstream status: accepted, not yet released + https://github.com/apache/avro/pull/740] +--- + src/avro_generic_internal.h | 2 ++ + 1 file changed, 2 insertions(+) + +diff --git a/src/avro_generic_internal.h b/src/avro_generic_internal.h +index 709403c0..9843ed65 100644 +--- a/src/avro_generic_internal.h ++++ b/src/avro_generic_internal.h +@@ -24,6 +24,8 @@ extern "C" { + #define CLOSE_EXTERN + #endif + ++#include <sys/types.h> ++ + #include "avro/generic.h" + #include "avro/schema.h" + #include "avro/value.h" +-- +2.23.0 +package/avro-c/Config.in new file +25
-0@@ -0,0 +1,25 @@ +comment "avro-c needs a toolchain w/ dynamic library" + depends on BR2_STATIC_LIBS + +config BR2_PACKAGE_AVRO_C + bool "avro-c" + depends on !BR2_STATIC_LIBS # Avro unconditionally compiles shared libs + select BR2_PACKAGE_JANSSON + help + Select this option to install the Avro C language bindings, + and the command line tools avroappend, avrocat, avromod and + avropipe. + + Apache Avro is a data serialization system. + Avro provides: + - Rich data structures. + - A compact, fast, binary data format. + - A container file, to store persistent data. + - Remote procedure call (RPC). + - Simple integration with dynamic languages. + - Code generation is not required to read or write data + files nor to use or implement RPC protocols. + - Code generation as an optional optimization, + only worth implementing for statically typed languages. + + http://avro.apache.org/package/avro-c/avro-c.hash new file +5
-0@@ -0,0 +1,5 @@ +# From https://www-eu.apache.org/dist/avro/avro-1.9.1/c/avro-c-1.9.1.tar.gz.sha512 +sha512 68b1f44f870c9b6f0b2380da9e34d91148ff4398cb300f4bdd4e3e1ad00820acd9084b73232b00d4cd4935fb992b41dc65afdafcbea14a3d87259608688df904 avro-c-1.9.1.tar.gz + +# License files +sha256 d62488d6ba17132e92c23c03c80bfedc848267f96ab36489fec860f76cf6819a LICENSEpackage/avro-c/avro-c.mk new file +26
-0@@ -0,0 +1,26 @@ +################################################################################ +# +# avro-c +# +################################################################################ + +AVRO_C_VERSION = 1.9.1 +AVRO_C_SITE = https://www-eu.apache.org/dist/avro/avro-$(AVRO_C_VERSION)/c +AVRO_C_LICENSE = Apache-2.0 +AVRO_C_LICENSE_FILES = LICENSE +AVRO_C_INSTALL_STAGING = YES +AVRO_C_DEPENDENCIES = host-pkgconf jansson + +ifeq ($(BR2_PACKAGE_SNAPPY),y) +AVRO_C_DEPENDENCIES += snappy +endif + +ifeq ($(BR2_PACKAGE_ZLIB),y) +AVRO_C_DEPENDENCIES += zlib +endif + +ifeq ($(BR2_PACKAGE_XZ),y) +AVRO_C_DEPENDENCIES += xz +endif + +$(eval $(cmake-package))Signature · 1 line
2.23.0
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v4 2/2 package/python-avro: new package
In reply to Titouan Christophe--- DEVELOPERS | 1 + package/Config.in | 1 + package/python-avro/Config.in | 20 ++++++++++++++++++++ package/python-avro/python-avro.hash | 5 +++++ package/python-avro/python-avro.mk | 14 ++++++++++++++ 5 files changed, 41 insertions(+) create mode 100644 package/python-avro/Config.in create mode 100644 package/python-avro/python-avro.hash create mode 100644 package/python-avro/python-avro.mk
DEVELOPERS +1 -0
@@ -2420,6 +2420,7 @@ F: package/fbgrab/ N: Titouan Christophe <[email protected]> F: package/avro-c/ F: package/mosquitto/ +F: package/python-avro/ F: package/redis/ N: Trent Piepho <[email protected]>
package/Config.in +1 -0
@@ -854,6 +854,7 @@ menu "External python modules" source "package/python-attrs/Config.in" source "package/python-autobahn/Config.in" source "package/python-automat/Config.in" + source "package/python-avro/Config.in" source "package/python-babel/Config.in" source "package/python-backcall/Config.in" source "package/python-backports-abc/Config.in"
package/python-avro/Config.in new file +20 -0
@@ -0,0 +1,20 @@ +config BR2_PACKAGE_PYTHON_AVRO + bool "python-avro" + depends on BR2_PACKAGE_PYTHON3 + help + Select this option to install the Avro Python3 language + bindings, and the `avro` command line tool + + Apache Avro is a data serialization system. + Avro provides: + - Rich data structures. + - A compact, fast, binary data format. + - A container file, to store persistent data. + - Remote procedure call (RPC). + - Simple integration with dynamic languages. + - Code generation is not required to read or write data + files nor to use or implement RPC protocols. + - Code generation as an optional optimization, + only worth implementing for statically typed languages. + + http://avro.apache.org/
package/python-avro/python-avro.hash new file +5 -0
@@ -0,0 +1,5 @@ +# From https://www-eu.apache.org/dist/avro/avro-1.9.1/py3/avro-python3-1.9.1.tar.gz.sha512 +sha512 771ce6e5ca6315f5b9108cb4f5dd9488575cc3433ffdd744f96ce7989763728259c89475319fd9c411ed6b65c996b762548c50a697eb1fba9ab024112ca17daa avro-python3-1.9.1.tar.gz + +# License files +sha256 c79a7fea0e3cac04cd43f20e7b648e5a0ff8fa5344e644b0ee09ca1162b62747 avro/LICENSE
package/python-avro/python-avro.mk new file +14 -0
@@ -0,0 +1,14 @@ +################################################################################ +# +# python-avro +# +################################################################################ + +PYTHON_AVRO_VERSION = $(AVRO_C_VERSION) +PYTHON_AVRO_SITE = https://www-eu.apache.org/dist/avro/avro-1.9.1/py3 +PYTHON_AVRO_SOURCE = avro-python3-$(PYTHON_AVRO_VERSION).tar.gz +PYTHON_AVRO_LICENSE = Apache-2.0 +PYTHON_AVRO_LICENSE_FILES = avro/LICENSE +PYTHON_AVRO_SETUP_TYPE = setuptools + +$(eval $(python-package))
Signature · 1 line
2.23.0
Yann E. MORIN
AppliedPATCH v4 2/2 package/python-avro: new package
Titouan, All,
On 2019-12-23 18:23 +0100, Titouan Christophe spake thusly:
Quoted 2 lines
Signed-off-by Titouan Christophe <[email protected]>Reviewed-by Gilles Talis <[email protected]>
Applied to master, thanks.
Could you please follow-up with a run-time test for it, please? See for example: support/testing/tests/package/sample_python_txtorcon.py (or others that are more expressive, if you feel so inclined).
Regards,
Yann E. MORIN.
Quoted 95 lines
--- DEVELOPERS | 1 + package/Config.in | 1 + package/python-avro/Config.in | 20 ++++++++++++++++++++ package/python-avro/python-avro.hash | 5 +++++ package/python-avro/python-avro.mk | 14 ++++++++++++++ 5 files changed, 41 insertions(+) create mode 100644 package/python-avro/Config.in create mode 100644 package/python-avro/python-avro.hash create mode 100644 package/python-avro/python-avro.mkDEVELOPERS +1
-0@@ -2420,6 +2420,7 @@ F: package/fbgrab/ N: Titouan Christophe <[email protected]> F: package/avro-c/ F: package/mosquitto/ +F: package/python-avro/ F: package/redis/ N: Trent Piepho <[email protected]>package/Config.in +1
-0@@ -854,6 +854,7 @@ menu "External python modules" source "package/python-attrs/Config.in" source "package/python-autobahn/Config.in" source "package/python-automat/Config.in" + source "package/python-avro/Config.in" source "package/python-babel/Config.in" source "package/python-backcall/Config.in" source "package/python-backports-abc/Config.in"package/python-avro/Config.in new file +20
-0@@ -0,0 +1,20 @@ +config BR2_PACKAGE_PYTHON_AVRO + bool "python-avro" + depends on BR2_PACKAGE_PYTHON3 + help + Select this option to install the Avro Python3 language + bindings, and the `avro` command line tool + + Apache Avro is a data serialization system. + Avro provides: + - Rich data structures. + - A compact, fast, binary data format. + - A container file, to store persistent data. + - Remote procedure call (RPC). + - Simple integration with dynamic languages. + - Code generation is not required to read or write data + files nor to use or implement RPC protocols. + - Code generation as an optional optimization, + only worth implementing for statically typed languages. + + http://avro.apache.org/package/python-avro/python-avro.hash new file +5
-0@@ -0,0 +1,5 @@ +# From https://www-eu.apache.org/dist/avro/avro-1.9.1/py3/avro-python3-1.9.1.tar.gz.sha512 +sha512 771ce6e5ca6315f5b9108cb4f5dd9488575cc3433ffdd744f96ce7989763728259c89475319fd9c411ed6b65c996b762548c50a697eb1fba9ab024112ca17daa avro-python3-1.9.1.tar.gz + +# License files +sha256 c79a7fea0e3cac04cd43f20e7b648e5a0ff8fa5344e644b0ee09ca1162b62747 avro/LICENSEpackage/python-avro/python-avro.mk new file +14
-0@@ -0,0 +1,14 @@ +################################################################################ +# +# python-avro +# +################################################################################ + +PYTHON_AVRO_VERSION = $(AVRO_C_VERSION) +PYTHON_AVRO_SITE = https://www-eu.apache.org/dist/avro/avro-1.9.1/py3 +PYTHON_AVRO_SOURCE = avro-python3-$(PYTHON_AVRO_VERSION).tar.gz +PYTHON_AVRO_LICENSE = Apache-2.0 +PYTHON_AVRO_LICENSE_FILES = avro/LICENSE +PYTHON_AVRO_SETUP_TYPE = setuptools + +$(eval $(python-package))Signature · 1 line
2.23.0
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v4 2/2 package/python-avro: new package
Hello Yann,
Quoted 8 lines · Yann E. MORIN
Titouan, All,
On 2019-12-23 18:23 +0100, Titouan Christophe spake thusly:
Quoted 2 lines
Signed-off-by Titouan Christophe <[email protected]>Reviewed-by Gilles Talis <[email protected]>Applied to master, thanks.
Superb !
Quoted 4 lines
Could you please follow-up with a run-time test for it, please? See for example:
support/testing/tests/package/sample_python_txtorcon.py(or others that are more expressive, if you feel so inclined).
Sure, I'll send that soon.
Quoted 3 lines
Regards,
Yann E. MORIN.
Titouan