core: add host-gzip for cvs/git/svn backends

Hello All!

As reported a few times already, gzip on some systems is aliased to pigz, a parallel implementation of gzip that takes advantage of multi-{processor,core,thread} systems to compress in parallel.

However, the compressed streams generated by pigz, even though they are valid gzip streams, differ for the streams generated by gzip. In fact, this is caused by pigz adding more checksums than is done by gzip, and because the streams are arbitrarily cut in blocks, the coder starts in different conditions for each blocks.

So, add a host variant to gzip, and use that as a download dependency for packages that need to create compressed tarballs (cvs, git, and svn).

While at it, do two seemingly unrelated cleanups: both where noticed while investigating how to do the last patch (i.e. the questions were:
How do we handle tar so I just copy that? -> noticed the tar comments. Then, should I export GZIP? If so, where? -> noticed SSH.)

Regards,
Yann E. MORIN.

The following changes since commit 915c136c5c9abeb683b59fad4a0e05a25c1a1cc2

package/openocd: add missing host-pkgconf dependency (2018-11-16 23:28:50 +0100)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to ba548d03f79279b7f02605006a709ffc630e8639

support/dependencies: add a check for a suitable gzip (2018-11-17 18:12:51 +0100)

----------------------------------------------------------------
Yann E. MORIN (4):
core/download: drop the SSH command support/dependencies: treat BSD-tar like the other cases package/gzip: add host variant support/dependencies: add a check for a suitable gzip

 Config.in                               |  4 ----
 package/gzip/gzip.mk                    |  2 ++
 package/pkg-download.mk                 |  1 -
 package/pkg-generic.mk                  |  4 +++-
 support/dependencies/check-host-gzip.mk |  3 +++
 support/dependencies/check-host-gzip.sh | 21 +++++++++++++++++++++
 support/dependencies/check-host-tar.sh  |  9 +++++----
 7 files changed, 34 insertions(+), 10 deletions(-)
 create mode 100644 support/dependencies/check-host-gzip.mk
 create mode 100755 support/dependencies/check-host-gzip.sh

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

The ssh command was added back in 2011 with commit c61788f09 (GENTARGETS:
add support for scp://) and was used to check that the remote file existed, back when we supported 'make source-check'.

However, in 2017, with commit bf28a165d (pkg-{download, generic}: remove source-check), we actually removed support for source-check.

The SSH command however was not removed then, and stuck, even though nothing ever uses it It is not even exported in the environment, and scp does not use it either (it has -S to specify an ssh-compatible program).

Get rid of it.

---
 Config.in               | 4 ----
 package/pkg-download.mk | 1 -
 2 files changed, 5 deletions(-)

index 03e4eb3928..f965e9d6d8 100644
@@ -136,10 +136,6 @@ config BR2_SCP
 	string "Secure copy (scp) command"
 	default "scp"
 
-config BR2_SSH
-	string "Secure shell (ssh) command"
-	default "ssh"
-
 config BR2_HG
 	string "Mercurial (hg) command"
 	default "hg"

index 73ea2a69f8..6293e2985b 100644
@@ -15,7 +15,6 @@ export BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
 export HG := $(call qstrip,$(BR2_HG))
 export SCP := $(call qstrip,$(BR2_SCP))
-SSH := $(call qstrip,$(BR2_SSH))
 export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
 DL_WRAPPER = support/download/dl-wrapper

2.14.1

Hello,

Applied to next, thanks.

Thomas

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Currently, when we detect that tar is BSD-tar, we fake an unsupported version (major, minor) and rely on the version check to reject BSD-tar.

There is no reason to use such shenanigans, when we can simply reject it from the onset.

Simplify the logic:
- use positive logic in the condition
- directly exit in error

Also, comment that case like the other cases are commented.

---
 support/dependencies/check-host-tar.sh | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

index 0857307396..934cb61299 100755
@@ -20,10 +20,11 @@ major=`echo "$version" | cut -d. -f1`
 minor=`echo "$version" | cut -d. -f2`
 bugfix=`echo "$version" | cut -d. -f3`
 version_bsd=`$tar --version | grep 'bsdtar'`
-if [ ! -z "${version_bsd}" ] ; then 
-  # mark as invalid version - not all command line options are available
-  major=0
-  minor=0
+
+# BSD tar does not have all the command-line options
+if [ -n "${version_bsd}" ] ; then
+    # echo nothing: no suitable tar found
+    exit 1
 fi
 
 # Minimal version = 1.27 (previous versions do not correctly unpack archives

2.14.1

In case someone is building on a musl-based distro (Alpine), we do as for the target variant, and force the fflush_stdin detection.

We however do not do the /bin/sh trick, because we are building natively, so the shell check is working.

---
 package/gzip/gzip.mk | 2 ++
 1 file changed, 2 insertions(+)

index d48cb1aa15..8ac6ddd6ed 100644
@@ -12,6 +12,7 @@ GZIP_CONF_OPTS = --exec-prefix=/
 GZIP_LICENSE = GPL-3.0+
 GZIP_LICENSE_FILES = COPYING
 GZIP_CONF_ENV += gl_cv_func_fflush_stdin=yes
+HOST_GZIP_CONF_ENV += gl_cv_func_fflush_stdin=yes
 # configure substitutes $(SHELL) for the shell shebang in scripts like
 # gzexe. Unfortunately, the same $(SHELL) variable will also be used by
 # make to run its commands. Fortunately, /bin/sh is always a POSIX shell

@@ -21,3 +22,4 @@ GZIP_CONF_ENV += gl_cv_func_fflush_stdin=yes
 GZIP_CONF_ENV += ac_cv_path_shell=/bin/sh
 
 $(eval $(autotools-package))
+$(eval $(host-autotools-package))

2.14.1

Recently, some hash mismatch have been reported, both by users as well as autobuilder failures, about tarballs generated from git repositories.

This turned out to be caused by users having the 'gzip' command somehow aliased to 'pigz' (which stand for: parallel implementation of gzip, which takes advantage of multi-processor system to parallelise the compression).

Unfortunately, the output of pigz-compressed archives differ from that of gzip (even though they *are* valid gzip-compressed streams).

Add a dependency check that ensures that gzip is not pigz. If that is the case, define a conditional dependency to host-gzip, that is used as a download dependency for packages that will generate compressed files, i.e. cvs, git, and svn.

Fixes:

    http://autobuild.buildroot.org/results/330/3308271fc641cadb59dbf1b5ee529a84f79e6d5c/

---
Changes v1 -> v2:
- don't fail, but define the conditional dependency (Thomas)

---
 package/pkg-generic.mk                  |  4 +++-
 support/dependencies/check-host-gzip.mk |  3 +++
 support/dependencies/check-host-gzip.sh | 21 +++++++++++++++++++++
 3 files changed, 27 insertions(+), 1 deletion(-)
 create mode 100644 support/dependencies/check-host-gzip.mk
 create mode 100755 support/dependencies/check-host-gzip.sh

index f34f46afc8..ef890981bb 100644
@@ -583,7 +583,9 @@ $(2)_DEPENDENCIES += host-skeleton
 endif
 
 ifneq ($$(filter cvs git svn,$$($(2)_SITE_METHOD)),)
-$(2)_DOWNLOAD_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
+$(2)_DOWNLOAD_DEPENDENCIES += \
+	$(BR2_GZIP_HOST_DEPENDENCY) \
+	$(BR2_TAR_HOST_DEPENDENCY)
 endif
 
 ifeq ($$(filter host-tar host-skeleton host-fakedate,$(1)),)

new file mode 100644
index 0000000000..bf9a369a7d
@@ -0,0 +1,3 @@
+ifeq (,$(call suitable-host-package,gzip))
+BR2_GZIP_HOST_DEPENDENCY = host-gzip
+endif

new file mode 100755
index 0000000000..5f344c5f9b
@@ -0,0 +1,21 @@
+#!/bin/sh
+
+candidate="$1" # ignored
+
+gzip="$(which gzip)"
+if [ ! -x "${gzip}" ]; then
+    # echo nothing: no suitable gzip found
+    exit 1
+fi
+
+# gzip displays its version string on stdout
+# pigz displays its version string on stderr
+version="$("${gzip}" --version 2>&1)"
+case "${version}" in
+  (*pigz*)
+    # echo nothing: no suitable gzip found
+    exit 1
+    ;;
+esac
+
+printf "%s" "${gzip}"

2.14.1

Yann,

Matthew L Weber / Pr Software Engineer
Airborne Information Systems / RC Linux Secure Platforms
MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA
www.rockwellcollins.com

Note: Any Export License Required Information and License Restricted
Third Party Intellectual Property (TPIP) content must be encrypted and
sent to matthew.weber at corp.rockwellcollins.com.

Matthew, All,

On 2018-11-17 11:23 -0600, Matthew Weber spake thusly:

[--SNIP--]

[--SNIP--]

So, two things, that are contradictory one to the other:

1- we want reproducible builds,
2- we want fast builds

For 1, it would mean that we should build as much tools as possible. However, the more we build, the slower the build is.

For 2, we should rely as much as possible on distro-provided tools, However, the more we rely on the host, the less reproducible we get.

gzip has been rock stable over the years. IIRC, I took one of the first releases from way back 1993-or-so, and the latest one, 1.9; they were generating the exact same output, 25 years apart! That, is stability.

Given the goals of the gzip authors and maintainers, I don't expect they change anything to it anytime.

So, we really don't want to build it if the host provides it.

Now, we can't know what the future will be, and we can't predict what other tool is gonna change its behaviour, that we have to build our own. So, when you update to a newer host, you'll also have to adapt, even if that means adding a few new archives to your BR2_DL_DIR, yes.

If you want to be sure that, in the future, you'll be as reproducible as possible, then do a chroot. Even now, having a chroot ensures that all users/developpers of your project have a known and reproducible devel environment (no more "it builds for me" arguments!) You may even go further, and mandate a VM, and even go as far as having HW spares for the project lifetime (to run the VM on!).

As for Buildroot, I guess we're going to continue relying on the host tools when they meet our expectations.

Regards,
Yann E. MORIN.

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Yann,

I'm definitely not advocating for building all the tools and libraries we use from the host distro packages. The case I'm running into is when additional host dependency checks/builds are added over time to Buildroot, it changes the consistency of the necessary set of cached dl archives depending on the machine you execute on. I do agree using a standard container or VM instance is the way to capture and define that "consistent environment". More times then not, I find that I can't control the OS users use for a dev env (many devops teams, timelines, "favorite OS", financial constraints, engineer opinions :-) ).

Use cases
1) We have a Sandbox environment which is engineered to create consistent offline rebuilds from a given set off offline inputs. This sandbox environment can't change as often as the distro used for day to day development. ie. need lots of projects to use the consistent environment to get our money out of the setup/doc effort. Normally we'd update the environment every ~4yrs. This mis-match of distro/env versions results in us doing some additional test builds in the sandbox and our day-to-day envs to identify the conditional host pkg builds.
2) Corporate network/proxy and offline builds. A user prepares to take a set of files offline and collects their material on distro 14.x.y.z (when online) and then had the same distro but 14.x (offline) that triggered a dependency build requiring another archive.

Agree. What about adding the option that if only the reproducible option is enabled, then we build all host tools we have a version dependency on (ie. all those we'd normally just conditionally build)?

I'm actually worried/experiencing the opposite. It's when our distro versions are newer during development and we go back to a older OS for release or CI.

Yeah, the hard part is the $/time investment in those VM and dev environments means (at least for my company) they don't change as often and we've found you always end up with a different/new one on the next new project. As a Linux team supporting our own env and a series of dev configurations, we start to see some of the use cases appear. For instance I currently have a projects with dev envs close to my Buildroot build machine distro version and a project on the
fringe of support. Generally this spread of versions is Ok with our
projects only having a ~1-2yr development cycle before feature complete. It does mean we get caught occasionally by things like the conditional host dependencies. Internally we'll carry a patch to make this consistant but I figured I'd bring it up and see if collectively this would be a good upstream change.

Thanks for the feedback Yann!

Hi,

I think there are a number of use cases where BR2_REPRODUCIBLE would be interesting (E.G. we have discussed turning it on by default), but you do no want to pay the extra build time for building these host utilities.

So I'm open to an option to force building all host dependencies, but it should be keyed from a separate configuration option and not BR2_REPRODUCIBLE.

Bye, Peter Korsgaard

Hello,

I've applied patches 2 to 4 to the next branch. Thanks!

Thomas

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com