Adding package libgit2

While testing this package, there were lots of failures due to zlib-ng and libressl, but none due to libgit2. None of these failures appears in the autobuilders. Do they have trouble testing virtual packages ?

I tested this package using something like this script:

i=0
while true; do

	left=$i
	i=$((i+1))
	maybe () {
		local count=$#
		shift $((left % count))
		left=$((left / count))
		printf "%s\n" $1 | grep '.' || true
	}
	{
		printf '%s\n' BR2_PACKAGE_LIBGIT2=y
		maybe "" BR2_PACKAGE_LIBSSH=y
		maybe "" \
			"BR2_PACKAGE_LIBCURL=y" \
			"BR2_PACKAGE_LIBCURL=y BR2_PACKAGE_GNUTLS=y" \
			"BR2_PACKAGE_OPENSSL=y BR2_PACKAGE_LIBRESSL=y" \
			"BR2_PACKAGE_OPENSSL=y BR2_PACKAGE_LIBOPENSSL=y" \
		"BR2_PACKAGE_LIBCURL=y BR2_PACKAGE_OPENSSL=y BR2_PACKAGE_LIBRESSL=y" \
		"BR2_PACKAGE_LIBCURL=y BR2_PACKAGE_OPENSSL=y BR2_PACKAGE_LIBOPENSSL=y"
		maybe BR2_PACKAGE_LIBZLIB=y BR2_PACKAGE_ZLIB_NG=y
		[ "$left" != 0 ] && break
	} > test.config
	cat test.config
	./utils/test-pkg -c test.config -p libgit2 || exit 1

done | tee -a test-complete.log
echo "Summary (Legend: . -> OK, s -> SKIPPED, F -> FAILED)" sed -r -e '
:moreconf

/=y$/ {
	s/\n//
	s/BR2_PACKAGE_(.*)=y/,\1/
	s/^,/ /
	N
	b moreconf
}
:moreres
/: (FAILED|OK|SKIPPED)$/ {
	s/^(\S*) (\S+)\n.*: OK/\1. \2/
	s/^(\S*) (\S+)\n.*: FAILED/\1F \2/
	s/^(\S*) (\S+)\n.*: SKIPPED/\1s \2/
	N
	b moreres
}

s/\n.*//p
d
' test-complete.log

Summary (Legend: . -> OK, s -> SKIPPED, F -> FAILED) ............................................... LIBGIT2,LIBZLIB .......s.sss..s................................ LIBGIT2,LIBSSH,LIBZLIB ............................................... LIBGIT2,LIBCURL,LIBZLIB .......s.sss..s................................ LIBGIT2,LIBSSH,LIBCURL,LIBZLIB ....s..s..s...s........s....................... LIBGIT2,LIBCURL,GNUTLS,LIBZLIB ....s..s.sss..s........s....................... LIBGIT2,LIBSSH,LIBCURL,GNUTLS,LIBZLIB .......s.F.s..sFF.....F........................ LIBGIT2,OPENSSL,LIBRESSL,LIBZLIB .......s.sss..sFF.....F........................ LIBGIT2,LIBSSH,OPENSSL,LIBRESSL,LIBZLIB ............................................... LIBGIT2,OPENSSL,LIBOPENSSL,LIBZLIB .......s.sss..s................................ LIBGIT2,LIBSSH,OPENSSL,LIBOPENSSL,LIBZLIB .......s.F.s..sFF.....F........................ LIBGIT2,LIBCURL,OPENSSL,LIBRESSL,LIBZLIB .......s.sss..sFF.....F........................ LIBGIT2,LIBSSH,LIBCURL,OPENSSL,LIBRESSL,LIBZLIB ............................................... LIBGIT2,LIBCURL,OPENSSL,LIBOPENSSL,LIBZLIB .......s.sss..s................................ LIBGIT2,LIBSSH,LIBCURL,OPENSSL,LIBOPENSSL,LIBZLIB FFFF...F..FF..FFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,ZLIB_NG FFFF...s.sss..sFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,LIBSSH,ZLIB_NG FFFF...F..FF..FFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,LIBCURL,ZLIB_NG FFFF...s.sss..sFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,LIBSSH,LIBCURL,ZLIB_NG FFFFs..s..sF..sFFFFFFFFsFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,LIBCURL,GNUTLS,ZLIB_NG FFFFs..s.sss..sFFFFFFFFsFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,LIBSSH,LIBCURL,GNUTLS,ZLIB_NG FFFF...s.FFs..sFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,OPENSSL,LIBRESSL,ZLIB_NG FFFF...s.sss..sFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,LIBSSH,OPENSSL,LIBRESSL,ZLIB_NG FFFF...F..FF..FFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,OPENSSL,LIBOPENSSL,ZLIB_NG FFFF...s.sss..sFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,LIBSSH,OPENSSL,LIBOPENSSL,ZLIB_NG FFFF...s.FFs..sFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,LIBCURL,OPENSSL,LIBRESSL,ZLIB_NG FFFF...s.sss..sFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,LIBSSH,LIBCURL,OPENSSL,LIBRESSL,ZLIB_NG FFFF...F..FF..FFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,LIBCURL,OPENSSL,LIBOPENSSL,ZLIB_NG FFFF...s.sss..sFFFFFFFFFFFFFFF..F.F.FFFFFFFFF.. LIBGIT2,LIBSSH,LIBCURL,OPENSSL,LIBOPENSSL,ZLIB_NG

I identified 6 different failure reasons:

1) zlib-ng (armv5-ctng-linux-gnueabi, armv7-ctng-linux-gnueabihf, sourcery-arm-armv4t, sourcery-arm, sourcery-arm-thumb2)

-- The following features have been enabled:
[...]
* WITH_ACLE , Build with ACLE CRC
* WITH_NEON , Build with NEON intrinsics
* ACLE_CRC , Support CRC hash generation using the ACLE instruction set, using "-march=armv8-a+crc"
* NEON_FILLWINDOW , Support NEON instructions in fill_window_arm, using "-mfloat-abi=softfp -mfpu=neon"

[...]
armv5-ctng-linux-gnueabi-gcc: error: unrecognized argument in option '-march=armv8-a+crc'

This option apparently appeared with GCC 4.9. Should i send a patch ?

2) zlib-ng (br-aarch64-glibc, linaro-aarch64)

-- The following features have been disabled:

* WITH_ACLE , Build with ACLE CRC
* WITH_NEON , Build with NEON intrinsics

-- Configuring done
[...]
[ 96%] Linking C executable minigzip64 libz.so.1.2.11.zlib-ng: undefined reference to crc32_acle' libz.so.1.2.11.zlib-ng: undefined reference to adler32_neon' libz.so.1.2.11.zlib-ng: undefined reference to crc32_acle' libz.so.1.2.11.zlib-ng: undefined reference to adler32_neon'

3) zlib-ng (br-arm-full-static)
[ 86%] Linking C static library libz.a
/home/cavallar/br-test-pkg/br-arm-full-static/host/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib/libc.a(__uClibc_main.os): In function `__uClibc_fini':
__uClibc_main.c:(.text+0x164): undefined reference to `__fini_array_end'
__uClibc_main.c:(.text+0x168): undefined reference to `__fini_array_start'
__uClibc_main.c:(.text+0x16c): undefined reference to `__fini_array_start'
/home/cavallar/br-test-pkg/br-arm-full-static/host/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib/libc.a(__uClibc_main.os): In function `__uClibc_main':
__uClibc_main.c:(.text+0x59c): undefined reference to `__preinit_array_start'
__uClibc_main.c:(.text+0x5a0): undefined reference to `__preinit_array_end'
__uClibc_main.c:(.text+0x5a4): undefined reference to `__preinit_array_start'
__uClibc_main.c:(.text+0x5a8): undefined reference to `__init_array_start'
__uClibc_main.c:(.text+0x5ac): undefined reference to `__init_array_end'
__uClibc_main.c:(.text+0x5b0): undefined reference to `__init_array_start' /home/cavallar/br-test-pkg/br-arm-full-static/host/opt/ext-toolchain/bin/../lib/gcc/arm-buildroot-linux-uclibcgnueabi/6.4.0/../../../../arm-buildroot-linux-uclibcgnueabi/bin/ld: libz.so.1.2.11.zlib-ng: hidden symbol `__fini_array_end' isn't defined /home/cavallar/br-test-pkg/br-arm-full-static/host/opt/ext-toolchain/bin/../lib/gcc/arm-buildroot-linux-uclibcgnueabi/6.4.0/../../../../arm-buildroot-linux-uclibcgnueabi/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

I don't know much about static linking.

4) zlib-ng (all other toolchains)

-- The following features have been enabled:
[...]
* SSE2 , Support the SSE2 instruction set, using "-msse2"
[...]
/home/cavallar/br-test-pkg/br-arcle-hs38/build/zlib-ng-a17fefa147942edbf80ab22a16bcbc2419def12d/arch/x86/x86.c:17:10: fatal error: cpuid.h: No such file or directory

 #include <cpuid.h>
          ^~~~~~~~~

compilation terminated.

5) libressl (br-arm-full-nothread)
PATH="$HOME/br-test-pkg/br-arm-full-nothread/host/bin:$HOME/br-test-pkg/br-arm-full-nothread/host/sbin:$HOME/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games" /usr/bin/make -j25 -C $HOME/br-test-pkg/br-arm-full-nothread/build/libressl-2.6.4/ Making all in crypto
/usr/bin/make all-am
CC compat/arc4random.lo
In file included from compat/arc4random.h:16:0,
from compat/arc4random.c:66:
compat/arc4random_linux.h:28:21: fatal error: pthread.h: No such file or directory

 #include <pthread.h>
                     ^

compilation terminated.

Only this file seems to use locks. It could be patched away, or libressl could depends on threads, but i don't think kconfig will like that?

6) libressl (br-microblazeel-full, br-m68k-68040-full, br-openrisc-uclibc)
Making all in ocspcheck
CCLD ocspcheck
/home/cavallar/br-test-pkg/br-m68k-68040-full/build/libressl-2.6.4/crypto/.libs/libcrypto.so: undefined reference to `__register_atfork'
collect2: error: ld returned 1 exit status

They have a MMU, but uclibc still doesn't provide __register_atfork() ?

---
 DEVELOPERS                   |  3 +++
 package/Config.in            |  1 +
 package/libgit2/Config.in    |  9 +++++++++
 package/libgit2/libgit2.hash |  3 +++
 package/libgit2/libgit2.mk   | 25 +++++++++++++++++++++++++
 5 files changed, 41 insertions(+)
 create mode 100644 package/libgit2/Config.in
 create mode 100644 package/libgit2/libgit2.hash
 create mode 100644 package/libgit2/libgit2.mk

index 293bd73797..c291156476 100644
@@ -1300,6 +1300,9 @@ N:	Naumann Andreas <[email protected]>
 F:	package/evemu/
 F:	package/libevdev/
 
+N:	Nicolas Cavallari <[email protected]>
+F:	package/libgit2/
+
 N:	Nicolas Serafini <[email protected]>
 F:	package/exiv2/
 F:	package/nvidia-tegra23/nvidia-tegra23-binaries/

index 19dda8aad9..94b3654210 100644
@@ -1071,6 +1071,7 @@ menu "Database" (i have no idea where to put it)
 	source "package/hiredis/Config.in"
 	source "package/kompexsqlite/Config.in"
 	source "package/leveldb/Config.in"
+	source "package/libgit2/Config.in"
 	source "package/libpqxx/Config.in"
 	source "package/mongodb/Config.in"
 	source "package/mysql/Config.in"

new file mode 100644
index 0000000000..53ba4d4245
@@ -0,0 +1,9 @@
+config BR2_PACKAGE_LIBGIT2
+	bool "libgit2"
+	select BR2_PACKAGE_ZLIB
+	help
+	  libgit2 is a portable, pure C implementation of the Git core
+	  methods provided as a linkable library with a solid API,
+	  allowing to build Git functionality into your application.
+
+	  https://github.com/libgit2/libgit2

new file mode 100644
index 0000000000..f642bdb230
@@ -0,0 +1,3 @@
+# Manually generated
+sha256	6a62393e0ceb37d02fe0d5707713f504e7acac9006ef33da1e88960bd78b6eac  libgit2-v0.26.0.tar.gz
+sha256	d9a8038088df84fde493fa33a0f1e537252eeb9642122aa4b862690197152813  COPYING

new file mode 100644
index 0000000000..5dec01b337
@@ -0,0 +1,25 @@
+################################################################################
+#
+# libgit2
+#
+################################################################################
+
+LIBGIT2_VERSION = v0.26.0
+LIBGIT2_SITE = $(call github,libgit2,libgit2,$(LIBGIT2_VERSION))
+LIBGIT2_LICENSE = GPL-2.0 with linking exception
+LIBGIT2_LICENSE_FILES = COPYING
+LIBGIT2_INSTALL_STAGING = YES
+
+LIBGIT2_DEPENDENCIES = zlib \
+	$(if $(BR2_PACKAGE_LIBSSH),libssh) \
+	$(if $(BR2_PACKAGE_OPENSSL),openssl) \
+	$(if $(BR2_PACKAGE_LIBCURL),libcurl)
+
+# USE_ICONV looks for a Iconv.pc.  libiconv does not have it.
+LIBGIT2_CONF_OPTS = -DUSE_GSSAPI=OFF -DBUILD_CLAR=OFF -DUSE_ICONV=OFF \
+	-DTHREADSAFE=$(if $(BR2_TOOLCHAIN_HAS_THREADS),ON,OFF) \
+	-DUSE_SSH=$(if $(BR2_PACKAGE_LIBSSH),ON,OFF) \
+	-DCURL=$(if $(BR2_PACKAGE_LIBCURL),ON,OFF) \
+	-DUSE_OPENSSL=$(if $(BR2_PACKAGE_OPENSSL),ON,OFF)
+
+$(eval $(cmake-package))

2.15.1

Hello,

So, I've applied to next, but I had to change quite a few things that were not really correct:

[Thomas:
- Do not select BR2_PACKAGE_ZLIB, because zlib is an optional dependency.
- Handle optional dependencies in a more usual way in libgit2.mk:
group the addition in _DEPENDENCIES and in _CONF_OPTS for a given library together.
- libgit2 can optionally use libssh2, not libssh.
- Add the optional dependency on zlib.
- Always pass USE_ICONV=ON, the detection works perfectly fine, with both a C library providing iconv support built-in, and with libiconv. If neither provides iconv, it gets disabled automatically as expected.
- Add libiconv as an optional dependency.]

Thanks!

Thomas

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Hello, and thanks.

IIRC zlib is not really an optional dependency.? If zlib is not detected, then a bundled version of zlib is compiled in deps/zlib...

I don't think this is desirable, that's why i forced a dependency on zlib.

Hello,

OK, then indeed we want zlib to be a dependency. The thing is that you had a select BR2_PACKAGE_ZLIB, but zlib was not in <pkg>_DEPENDENCIES.

Don't hesitate to send a follow-up patch re-introducing zlib as a mandatory dependency.

Thomas

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Hello,

(Adam, Yann: questions for you below.)

It's not a problem with virtual packages, but with Kconfig choices. If you have a choice...endchoice block, with multiple options, then "make randpackageconfig" does not any randomization of such choices, and only the first option will ever be used.

So for the virtual packages that do have a "choice..endchoice", such as jpeg, openssl and zlib, indeed only the first implementation is going to be tested by the autobuilders. Yann had already spent a significant amount of time a few years ago trying to fix this, and I think he has lost more hairs than he really wanted to while trying to dive into the Kconfig code base :)

I believe we could probably fix this by tweaking utils/genrandconfig:
we randomly inject BR2_PACKAGE_ZLIB_NG=y or BR2_PACKAGE_LIBRESSL=y. But I'm not sure if "make randpackageconfig" will not override that, and if we can convince it to not do that.

Cc'ing Kconfig guru Yann E. Morin on this :-)

It is very bogus to use -march=armv8-+crc when building for ARM32-only cores.

Perhaps this:

# Enable NEON and ACLE on ARM
ifeq ($(BR2_arm),y)
ZLIB_NG_CONF_OPTS += -DWITH_ACLE=1 -DWITH_NEON=1
endif

isn't entirely correct.

They should probably be enabled for ARM64.

Which should perhaps fix this ?

The CMake build system is trying to build a shared library, while the toolchain only supports static linking. The CMakeLists.txt needs to be fixed to obey to -DBUILD_SHARED_LIBS, and not build the shared library if it's set to OFF.

Can you check what is the value of ${ARCH} in the CMakeLists.txt when such problem occurs ? It is printed in the configure output:

message(STATUS "Architecture: ${ARCH}")

ARCH is set to ${CMAKE_SYSTEM_PROCESSOR}, and CMAKE_SYSTEM_PROCESSOR is explicitly passed by Buildroot in package/pkg-cmake.mk. So it should be correct. zlib-ng CMakeLists.txt uses its ${ARCH} variable to decide whether the x86 code should be built or not:

if("${ARCH}" MATCHES "x86_64" OR "${ARCH}" MATCHES "AMD64")

    set(ARCHDIR "arch/x86")
    add_definitions(-DX86_64 -DX86_NOCHECK_SSE2 -DUNALIGNED_OK -DUNROLL_LESS)
    add_feature_info(SSE2 1 "Use the SSE2 instruction set, using \"${SSE2FLAG}\"")

So I guess there is something wrong going on with ${ARCH} or ${CMAKE_SYSTEM_PROCESSOR}, because a build targetting the ARC architecture should not have ARCH=x86_64 or AMD64.

Kconfig will be OK with this, since nobody selects libressl. However, I'm pretty we had discussed this issue before. Adam ?

I think this has been fixed in uClibc-ng as of commit 9e38e0aa45cca21d5023d0af94377f0e1e41d2f4, which appeared in uClibc-ng 1.0.27. I guess I haven't rebuilt the toolchains since then, causing this problem.

I could definitely rebuild the toolchains to fix this problem.

Best regards,

Thomas

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

Seems like git send-email scrubbed my CC to the maintainers

It is correct, e.g. "-- Architecture: or1k" for openrisc.

I have added --trace to cmake. There is this gem a few lines below:

else()
set(ARCHDIR "arch/x86")
add_definitions(-DX86 -DUNALIGNED_OK -DUNROLL_LESS) add_feature_info(SSE2 1 "Support the SSE2 instruction set, using
\"${SSE2FLAG}\"")
endif()

So anything not arm/aarch64/x86 is treated like x86 anyway...

Hello,

Which obviously is not going to work very well :)

I guess this should be fixed and/or reported to zlib-ng upstream. In the mean time, either we fix it, or we disallow zlib-ng on architectures other than arm/aarch64/x86.

Thomas

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

Nicolas, Thomas, All,

On 2018-02-14 16:36 +0100, Thomas Petazzoni spake thusly:

Yes, let's limit zlib-ng to the only arch upstream supports.

I'll send a patch...

Regards,
Yann E. MORIN.

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'