some download-related changes

From: "Yann E. MORIN"<[email protected]>

This variable contains extra envirnoment variables that we can not export since they are clashing with some build systems (eg. BUILD_DIR with u-boot).

So, we may need these variables for other ueses than the user's hooks for instrumentation. For example, we'll use them later on to export BUILD_DIR to the download helper scripts.

Signed-off-by "Yann E. MORIN"<[email protected]>

From: "Yann E. MORIN" <[email protected]>

The git download helper is getting a bit more complex, and does not raise an error when a PKG_VERSION is a sha1 that does not exist in the repository. Instead, it generates an empty archive.

Fixing it in the Makefile proves to be challenging, to say the least.

Move it into a shell script in support/download/git, which will make it much easier to read, maintain, fix an enhance in the future.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Peter Korsgaard <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
  package/pkg-download.mk | 16 +++-------------
  support/download/git    | 36 ++++++++++++++++++++++++++++++++++++
  2 files changed, 39 insertions(+), 13 deletions(-)

create mode 100755 support/download/git

package/pkg-download.mk +3 -13

index c00689b..8e6f7a3 100644
@@ -12,7 +12,7 @@ WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
  SVN := $(call qstrip,$(BR2_SVN))
  CVS := $(call qstrip,$(BR2_CVS))
  BZR := $(call qstrip,$(BR2_BZR))
-GIT := $(call qstrip,$(BR2_GIT))
+export GIT := $(call qstrip,$(BR2_GIT))
  HG := $(call qstrip,$(BR2_HG)) $(QUIET)
  SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
  SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)

@@ -84,18 +84,8 @@ github = https://github.com/$(1)/$(2)/tarball/$(3)
  # problems
  define DOWNLOAD_GIT
  	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	 ((test "`git ls-remote $($(PKG)_SITE) $($(PKG)_DL_VERSION)`" && \
-	   echo "Doing shallow clone" && \
-	   $(GIT) clone --depth 1 -b $($(PKG)_DL_VERSION) --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME)) || \
-	  (echo "Doing full clone" && \
-	   $(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME))) && \
-	pushd $($(PKG)_BASE_NAME) > /dev/null && \
-	$(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
-		gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
-	popd > /dev/null && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(EXTRA_ENV) support/download/git $($(PKG)_SITE) $($(PKG)_DL_VERSION) \
+					  $($(PKG)_BASE_NAME)/ $(DL_DIR)/$($(PKG)_SOURCE)
  endef
 
  # TODO: improve to check that the given PKG_DL_VERSION exists on the remote

support/download/git new file +10 -0

new file mode 100755
index 0000000..99ea1b2
@@ -0,0 +1,36 @@
+#!/bin/sh
+set -e
+
+# Download helper for git
+# Call it with:
+#   $1: git repo
+#   $2: git cset
+#   $3: prefix/
+#   $4: output file
+# And this environment:

support/download/git new file +16 -0

+
+if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
+    printf "Doing shallow clone\n"
+    ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" .git-tmp-repo
+else
+    printf "Doing full clone\n"
+    ${GIT} clone --bare "${repo}" .git-tmp-repo
+fi
+
+pushd .git-tmp-repo >/dev/null
+${GIT} archive --prefix="${prefix}" --format=tar "${cset}"  \
+|gzip -c >"${output}"
+popd >/dev/null
+
+rm -rf .git-tmp-repo
+popd >/dev/null

Quoted 11 lines · Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

The git download helper is getting a bit more complex, and does not raise an error when a PKG_VERSION is a sha1 that does not exist in the repository. Instead, it generates an empty archive.

Fixing it in the Makefile proves to be challenging, to say the least.

Move it into a shell script in support/download/git, which will make it much easier to read, maintain, fix an enhance in the future.

Quoted 3 lines

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Peter Korsgaard <[email protected]>

Cc Thomas Petazzoni <[email protected]>

This is a very welcome change, thanks for caring!

Just a minor note below, but looks ok anyway:

Acked-by Luca Ceresoli <[email protected]>

Quoted 21 lines

support/download/git new file +15 -0

new file mode 100755
index 0000000..99ea1b2
@@ -0,0 +1,36 @@
+#!/bin/sh
+set -e
+
+# Download helper for git
+# Call it with:
+#   $1: git repo
+#   $2: git cset
+#   $3: prefix/
+#   $4: output file
+# And this environment:
+#   GIT      :  the git command to call
+#   BUILD_DIR:  path to Buildroot's build dir
+
+repo="${1}"
+cset="${2}"

support/download/git line 20

Quoted 5 lines

support/download/git new file +5 -0

+prefix="${3}"
+output="${4}"
+
+pushd "${BUILD_DIR}" >/dev/null
+rm -rf .git-tmp-repo    # In case a previous attempt was interrupted

This rm did not exist in the original version. I would move this into a separate commit.

From: "Yann E. MORIN" <[email protected]>

The git download helper is getting a bit more complex, and does not raise an error when a PKG_VERSION is a sha1 that does not exist in the repository. Instead, it generates an empty archive.

Fixing it in the Makefile proves to be challenging, to say the least.

Move it into a shell script in support/download/git, which will make it much easier to read, maintain, fix an enhance in the future.

support/download/git new file +9 -0

new file mode 100755
index 0000000..99ea1b2
@@ -0,0 +1,36 @@
+#!/bin/sh
+set -e
+
+# Download helper for git
+# Call it with:
+#   $1: git repo
+#   $2: git cset
+#   $3: prefix/
+#   $4: output file

support/download/git new file +4 -0

+popd >/dev/null
+
+rm -rf .git-tmp-repo
+popd >/dev/null

Quoted 6 lines

From: "Yann E. MORIN" <[email protected]>

The git download helper is getting a bit more complex, and does not raise an error when a PKG_VERSION is a sha1 that does not exist in the repository. Instead, it generates an empty archive.

Quoted 4 lines

Fixing it in the Makefile proves to be challenging, to say the least.

Move it into a shell script in support/download/git, which will make it much easier to read, maintain, fix an enhance in the future.

This patch contains a few more changes than just moving it to a script.
These should ideally be done in separate patches.

[snip]

support/download/git

Quoted 5 lines

support/download/git new file +0 -0

new file mode 100755
index 0000000..99ea1b2

Quoted 1 line

+pushd "${BUILD_DIR}" >/dev/null

This used to be DL_DIR.

Note that I agree that BUILD_DIR is more appropriate.

Although the original indeed does a pushd, this is quite redundant when
it's done in a script. So I'd directly make it a simple cd (without redirection).

support/download/git line 24

Quoted 5 lines

support/download/git new file +5 -0

+rm -rf .git-tmp-repo    # In case a previous attempt was interrupted
+
+if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
+    printf "Doing shallow clone\n"
+    ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" .git-tmp-repo

It used to be extracted into $($(PKG)_BASE_NAME). That has the advantage
that it is unique, which makes it possible to do multiple downloads in parallel (which I sometimes do BTW, by manually starting downloads in separate shells).

But even better would be $($(PKG)_BASE_NAME).git-tmp-repo :-)

Quoted 4 lines

support/download/git new file +4 -0

+else
+    printf "Doing full clone\n"
+    ${GIT} clone --bare "${repo}" .git-tmp-repo
+fi

support/download/git line 32

Quoted 4 lines

support/download/git new file +4 -0

+
+pushd .git-tmp-repo >/dev/null
+${GIT} archive --prefix="${prefix}" --format=tar "${cset}"  \
+|gzip -c >"${output}"

We have about 20 instances of the | at the end of the line (as was the
original), and only 4 at the beginning of the line (and two of these were done by you...).

In the original, the / at the end of the prefix is added explicitly here in
the call to git archive. I think that makes more sense than adding the / in the call itself.

From: "Yann E. MORIN" <[email protected]>

If the PKG_VERSION is a sha1 that does not exist in the repository, the git helper just creates an empty archive instead of failing and erroring out.

This is because of the way we create the archive:

     git archive ${cset} |gzip -c >"${archive}"

In this construct, git does exit with a non-zero exit-code, but gzip does not, as it was successful in creating the archive (from an empty output, but gzip does not mind that).

Since a pipe exits with the exit-code of the right-most simple command (see POSIX.1-2008), the pipe above exits with gzip's exit-code. And we miss the error altogether.

This is most harmful, in that the error may not occur until much later, when we try to configure the package, with the error message buried deep down ipotentially thousands of lines above... :-(

Fix that by checking that the hash (or whatever treeish we've been passed, for that matters) does indeed exist in the repository, before we even attempt to create the archive.

Signed-off-by "Yann E. MORIN" <[email protected]>

Quoted 16 lines · Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

If the PKG_VERSION is a sha1 that does not exist in the repository, the git helper just creates an empty archive instead of failing and erroring out.

This is because of the way we create the archive:

    git archive ${cset} |gzip -c >"${archive}"

In this construct, git does exit with a non-zero exit-code, but gzip does not, as it was successful in creating the archive (from an empty output, but gzip does not mind that).

Since a pipe exits with the exit-code of the right-most simple command (see POSIX.1-2008), the pipe above exits with gzip's exit-code. And we miss the error altogether.

Quoted 3 lines

This is most harmful, in that the error may not occur until much later, when we try to configure the package, with the error message buried deep down ipotentially thousands of lines above... :-(

s/ipotentially/potentially/

Quoted 5 lines

Fix that by checking that the hash (or whatever treeish we've been passed, for that matters) does indeed exist in the repository, before we even attempt to create the archive.

Signed-off-by "Yann E. MORIN" <[email protected]>

Acked-by Luca Ceresoli <[email protected]>

From: "Yann E. MORIN" <[email protected]>

If the PKG_VERSION is a sha1 that does not exist in the repository, the git helper just creates an empty archive instead of failing and erroring out.

This is because of the way we create the archive:

     git archive ${cset} |gzip -c >"${archive}"

In this construct, git does exit with a non-zero exit-code, but gzip does not, as it was successful in creating the archive (from an empty output, but gzip does not mind that).

Since a pipe exits with the exit-code of the right-most simple command (see POSIX.1-2008), the pipe above exits with gzip's exit-code. And we miss the error altogether.

This is most harmful, in that the error may not occur until much later, when we try to configure the package, with the error message buried deep down ipotentially thousands of lines above... :-(

Fix that by checking that the hash (or whatever treeish we've been passed, for that matters) does indeed exist in the repository, before we even attempt to create the archive.

Signed-off-by "Yann E. MORIN" <[email protected]>

---
  support/download/git | 10 ++++++++--
  1 file changed, 8 insertions(+), 2 deletions(-)

support/download/git +0 -0

index 99ea1b2..508f8a1 100755

support/download/git +6 -0

+    ${GIT} archive --prefix="${prefix}" --format=tar "${cset}"  \
+    |gzip -c >"${output}"
+    res=0
+fi
  popd >/dev/null
 
  rm -rf .git-tmp-repo
  popd >/dev/null
+
+exit ${res}

Quoted 1 line

From: "Yann E. MORIN" <[email protected]>

support/download/git else · line 32

Quoted 4 lines

-${GIT} archive --prefix="${prefix}" --format=tar "${cset}" \
-|gzip -c >"${output}"

support/download/git +2 -0

+res=1
+if ${GIT} rev-list -n 1 "${cset}" >/dev/null 2>&1; then

When we've done a shallow clone, this check is redundant because we have
cloned exactly this ref.

So instead, I'd do the check in the full clone condition, with an explicit
exit.

Arnout, All,

On 2014-01-14 21:43 +0100, Arnout Vandecappelle spake thusly:

Quoted 2 lines · Yann E. MORIN

Quoted 1 line

From: "Yann E. MORIN" <[email protected]>

[--SNIP--]

Quoted 10 lines

support/download/git else · line 32

Quoted 4 lines

-${GIT} archive --prefix="${prefix}" --format=tar "${cset}" \
-|gzip -c >"${output}"

support/download/git +2 -0

+res=1
+if ${GIT} rev-list -n 1 "${cset}" >/dev/null 2>&1; then

When we've done a shallow clone, this check is redundant because we have
cloned exactly this ref.

So instead, I'd do the check in the full clone condition, with an explicit
exit.

I see. But it complexifies the script.

And doing the check a second time ensures we did not download garbage

Heck, there's even a race condition, since we check the remote, then clone it, and the ref may have vanished (granted, very unlikely, but yet...). :-p

Quoted 19 lines

Arnout, All,

On 2014-01-14 21:43 +0100, Arnout Vandecappelle spake thusly:

Quoted 2 lines · Yann E. MORIN

Quoted 1 line

From: "Yann E. MORIN" <[email protected]>

[--SNIP--]

Quoted 10 lines

support/download/git else · line 32

Quoted 4 lines

-${GIT} archive --prefix="${prefix}" --format=tar "${cset}" \
-|gzip -c >"${output}"

support/download/git +2 -0

+res=1
+if ${GIT} rev-list -n 1 "${cset}" >/dev/null 2>&1; then

When we've done a shallow clone, this check is redundant because we have
cloned exactly this ref.

So instead, I'd do the check in the full clone condition, with an explicit
exit.

I see. But it complexifies the script.

Quoted 5 lines

And doing the check a second time ensures we did not download garbage

Heck, there's even a race condition, since we check the remote, then clone it, and the ref may have vanished (granted, very unlikely, but yet...). :-p

True. I yield to your wisdom.

From: "Yann E. MORIN" <[email protected]>

Currently, the DOWNLOAD helper accepts one or two arguments:
- the source URI
- the source filename, if missing, derived from the URI

But we have no caller of DOWNLOAD that passes a second argument.

Signed-off-by "Yann E. MORIN" <[email protected]>

---
  package/pkg-download.mk | 5 ++---
  1 file changed, 2 insertions(+), 3 deletions(-)

package/pkg-download.mk +2 -3

index 8e6f7a3..f3354d1 100644
@@ -227,14 +227,13 @@ endef
  # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
  #
  # Argument 1 is the source location
-# Argument 2 is the source filename
  #
  # E.G. use like this:
-# $(call DOWNLOAD,$(FOO_SITE),$(FOO_SOURCE))
+# $(call DOWNLOAD,$(FOO_SITE))
  ################################################################################
 
  define DOWNLOAD
-	$(call DOWNLOAD_INNER,$(1),$(if $(2),$(2),$(notdir $(1))))
+	$(call DOWNLOAD_INNER,$(1),$(notdir $(1)))
  endef
 
  define DOWNLOAD_INNER

support/download/check-hash new file +23 -0

new file mode 100755
index 0000000..5cf708f
@@ -0,0 +1,38 @@
+#!/bin/sh
+set -e
+
+# Helper to check a file matches its known hash
+# Call it with:
+#   $1: the basename of the package's tarball
+#   $2: the full path to the file to check
+#   $3: the path of the file containing all the the expected hashes
+
+tarball="${1}"
+file="${2}"
+h_file="${3}"
+
+# Does the hash-file exist?
+if [ ! -f "${h_file}" ]; then
+    exit 0
+fi
+
+# Do we know a hash for that tarball?
+known=$( grep -E '^[[:xdigit:]]+[[:space:]]{2}'"${tarball}"'$$' "${h_file}" \
+         |cut -d ' ' -f 1
+       )
+if [ -z "${known}" ]; then

Quoted 33 lines

support/download/check-hash new file +27 -0

new file mode 100755
index 0000000..5cf708f
@@ -0,0 +1,38 @@
+#!/bin/sh
+set -e
+
+# Helper to check a file matches its known hash
+# Call it with:
+#   $1: the basename of the package's tarball
+#   $2: the full path to the file to check
+#   $3: the path of the file containing all the the expected hashes
+
+tarball="${1}"
+file="${2}"
+h_file="${3}"
+
+# Does the hash-file exist?
+if [ ! -f "${h_file}" ]; then
+    exit 0
+fi
+
+# Do we know a hash for that tarball?
+known=$( grep -E '^[[:xdigit:]]+[[:space:]]{2}'"${tarball}"'$$' "${h_file}" \
+         |cut -d ' ' -f 1
+       )
+if [ -z "${known}" ]; then
+    exit 0
+fi
+
+# Do the hashes match?

support/download/check-hash line 33

Quoted 6 lines

support/download/check-hash new file +6 -0

+hash=$( sha1sum "${file}" |cut -d ' ' -f 1 )
+if [ "${hash}" = "${known}" ]; then
+    exit 0
+fi
+
+printf "ERROR: %s has wrong SHA256\n" "${tarball}"

That's SHA1.

From: "Yann E. MORIN" <[email protected]>

Some of the packages that Buildroot might build are sensitive pacakges, related to security: openssl, dropbear, ca-certificates...

Some of those packages are download over plain http, because there is no way to get them over a secure channel, such as https.

In these dark times of pervasive surveillance, the potential for harm that a tampered package could generate, we may want to check the integrity of those sensitive packages.

So, each package may now provide a list of hashes for all files that needs to be downloaded, and Buildroot will just fail if any download file does not match its known hash.

The choosen hash function is SHA1 since it is widely available, though theoretical attacks have been devised (but no known practical attack is known).

Signed-off-by "Yann E. MORIN" <[email protected]>

---
Note: this is not a bullet-proof solution, since Buildroot may itself be compromised. But if we eventually sign our releases, then we secure the list of hashes at the same time. Only random snapshots from the repository may be at risk of tampering, although this is highly doubtfull, given how git stores its data.

Also, before we commit a list of hashes to the tree, we may want to setup a chain-of-trust to validate that thos hashes are correct. We may want to discuss this during our next developpers' day in Brussels in February.

Note-2: The laternative to sha1 would be sha2 (256- or 512-bit), but oldish "enterprise-class" distributions may be missing them entirely. sha256sum and sha512sum were added to coreutils in 2005-10-23, and RHEL5 seems to have them. But better be safe than sorry. If sha2 should be considered instead of sha1, then it is very easy to switch now. Switching later would require that we revalidate all packages that have hashes, which could prove to be quite time-demanding if we have lots of packages using hashes.

package/pkg-download.mk +4 -0

+	support/download/check-hash $(1) $(2) \
+		$($(PKG)_DIR_PREFIX)/$($(PKG)_NAME)/$($(PKG)_NAME).hash
+endef
+
  ################################################################################
  # The DOWNLOAD_* helpers are in charge of getting a working copy
  # of the source repository for their corresponding SCM,

Quoted 3 lines

From: "Yann E. MORIN" <[email protected]>
Some of the packages that Buildroot might build are sensitive pacakges, related to security: openssl, dropbear, ca-certificates...

Quoted 3 lines

So, each package may now provide a list of hashes for all files that needs to be downloaded, and Buildroot will just fail if any download file does not match its known hash.

Quoted 4 lines

Also, before we commit a list of hashes to the tree, we may want to setup a chain-of-trust to validate that thos hashes are correct. We may want to discuss this during our next developpers' day in Brussels in February.

I think the risk is small, because the package will be downloaded by
multiple users and autobuilders, so an incorrect hash in the buildroot sources will lead to download failure reports.

Quoted 8 lines

Note-2: The laternative to sha1 would be sha2 (256- or 512-bit), but oldish "enterprise-class" distributions may be missing them entirely. sha256sum and sha512sum were added to coreutils in 2005-10-23, and RHEL5 seems to have them. But better be safe than sorry. If sha2 should be considered instead of sha1, then it is very easy to switch now. Switching later would require that we revalidate all packages that have hashes, which could prove to be quite time-demanding if we have lots of packages using hashes.

We can be more future-safe by storing the hash that is used in the .hash
file itself.

Quoted 8 lines

package/pkg-download.mk +0 -0

index f3354d1..5627850 100644
@@ -58,6 +58,14 @@ domainseparator=$(if $(1),$(1),/)
 # github(user,package,version): returns site of github repository
 github = https://github.com/$(1)/$(2)/tarball/$(3)
 

package/pkg-download.mk domainseparator=$(if $(1),$(1),/) · line 64

Quoted 4 lines

package/pkg-download.mk +4 -0

+# Helper for checking a tarball's checksum
+# $(1): the basename of the tarball to check
+# $(2): the full path to the file to check
+define VERIFY_SHA256

VERIFY_HASH would be better.

Arnout, All,

On 2014-01-14 22:37 +0100, Arnout Vandecappelle spake thusly:

Quoted 1 line · Yann E. MORIN

Quoted 9 lines

Quoted 8 lines

Note-2: The laternative to sha1 would be sha2 (256- or 512-bit), but oldish "enterprise-class" distributions may be missing them entirely. sha256sum and sha512sum were added to coreutils in 2005-10-23, and RHEL5 seems to have them. But better be safe than sorry. If sha2 should be considered instead of sha1, then it is very easy to switch now. Switching later would require that we revalidate all packages that have hashes, which could prove to be quite time-demanding if we have lots of packages using hashes.

Quoted 2 lines

We can be more future-safe by storing the hash that is used in the .hash
file itself.

Hu?

If the hash file contains the following:

486fb55c3efa71148fe07895fd713ea3a5ae343a sha1 libfoo-1.2.3.tar.bz2

then you can now let the script check that the second field is sha1, and later you can support different hash methods. In that case, it is not necessary to update all the files when we want to switch to a new hash method.

(Incidentally, it also enables Gustavo's suggestion to use whatever
upstream provides.)

Quoted 5 lines

If the hash file contains the following:

486fb55c3efa71148fe07895fd713ea3a5ae343a sha1 libfoo-1.2.3.tar.bz2

Quoted 7 lines

then you can now let the script check that the second field is sha1, and later you can support different hash methods. In that case, it is not necessary to update all the files when we want to switch to a new hash method.

(Incidentally, it also enables Gustavo's suggestion to use whatever
upstream provides.)

Yes.
A little explanation on why upstream hashes should be used (my mail last night was a bit rushed out, busy busy).
When upstream releases a tarball normally it'll fire off an announcement mail to some mailing list with (hopefully) the hash for the tarball(s). Usually this hash(es) will also live in some project website hosting. If the website is compromised then the hash there can also be compromised and you are computing your hash on a bad tarball (if done when bumping version) or an altered hash in the website. Sent mail (mailing list archives, personal mail) is much harder to compromise all at once hence the original hash will still be true if all is dandy.
Of course there's the possibility that the developer machine has been compromised or some code has been "sneaked under the carpet" with peer review failing to notice, but then there's no hash to save you there and it's not the intention of this either.

I understand your concerns.

But, whether we do compute the hashes ourselves, or retrieve them from an annoucment mail (which by your own saying is not systematic), we can't blindly accept a patch that contains hashes without verifying them.

So, whenever Peter would be about to apply a package, he'd have to check for himself that the hashes are correct.

For that, he'd have to go the package's website, dig up the announcement mail from the list, and compare the hashes.

So, whether we compute them, or get them from the annoucement mail, Peter would still have to check them in the end, prior to applying the patch.

And we still do not have solved the problem of packages for which no hash has been publicly posted and archived.

So, all that comes to mind now is that we need signatures, not hashes. But not all packages have signed releases either. So we're back to square-one.

In the end, I wonder how much we do want to protect the downloads. I firmly believe that a set of hashes are enough for what we want to do. If a user is even most concerned about security, then he should (and would anyway) audit the download for integrity before shipping a product.

And since we do sign our releases (peter does it), a user can verify our releases easily, and assess that the hashes are legit.

Quoted 4 lines

Arnout, All,

On 2014-01-14 22:37 +0100, Arnout Vandecappelle spake thusly:

Quoted 1 line · Yann E. MORIN

[snip]

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

Note-2: The laternative to sha1 would be sha2 (256- or 512-bit), but oldish "enterprise-class" distributions may be missing them entirely. sha256sum and sha512sum were added to coreutils in 2005-10-23, and RHEL5 seems to have them. But better be safe than sorry. If sha2 should be considered instead of sha1, then it is very easy to switch now. Switching later would require that we revalidate all packages that have hashes, which could prove to be quite time-demanding if we have lots of packages using hashes.

Quoted 4 lines

Quoted 2 lines

We can be more future-safe by storing the hash that is used in the .hash
file itself.

Hu?

If the hash file contains the following:

486fb55c3efa71148fe07895fd713ea3a5ae343a sha1 libfoo-1.2.3.tar.bz2

then you can now let the script check that the second field is sha1, and later you can support different hash methods. In that case, it is not necessary to update all the files when we want to switch to a new hash method.

Arnout, All,

On 2014-01-15 09:22 +0100, Arnout Vandecappelle spake thusly:

Quoted 15 lines · Yann E. MORIN

Quoted 4 lines

Arnout, All,

On 2014-01-14 22:37 +0100, Arnout Vandecappelle spake thusly:

Quoted 1 line · Yann E. MORIN

[snip]

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

Note-2: The laternative to sha1 would be sha2 (256- or 512-bit), but oldish "enterprise-class" distributions may be missing them entirely. sha256sum and sha512sum were added to coreutils in 2005-10-23, and RHEL5 seems to have them. But better be safe than sorry. If sha2 should be considered instead of sha1, then it is very easy to switch now. Switching later would require that we revalidate all packages that have hashes, which could prove to be quite time-demanding if we have lots of packages using hashes.

Quoted 8 lines

Quoted 4 lines

Quoted 2 lines

We can be more future-safe by storing the hash that is used in the .hash
file itself.

Hu?

If the hash file contains the following:

486fb55c3efa71148fe07895fd713ea3a5ae343a sha1 libfoo-1.2.3.tar.bz2

OK, I see what you meant, now.

Quoted 4 lines

then you can now let the script check that the second field is sha1, and later you can support different hash methods. In that case, it is not necessary to update all the files when we want to switch to a new hash method.

However, that means the file is no longer the output of:
sha1sum files-to-check* >package.hash

or of any other hash utility: sha*sum all generates similarly-formatted outputs:
hash <two spaces> filename

Which was the reason I choose that format.

If we'd use your suggestion, we'd need a simple way to generate that file, or it'd be error prone.

The choosen hash function is SHA1 since it is widely available, though theoretical attacks have been devised (but no known practical attack is known).

The first part, patch 1 to 3, are a fix the git download helper, which is currently broken in case a PKG_VERSION is a sha1 that does not exist in the repository.

Since the make macro was a bit complex to read and maintain, Peter suggested it be moved to a shell script. If this change is OK, I'll move the other download helpers to their own shell script.

Quoted 7 lines

The first part, patch 1 to 3, are a fix the git download helper, which is currently broken in case a PKG_VERSION is a sha1 that does not exist in the repository.

Since the make macro was a bit complex to read and maintain, Peter suggested it be moved to a shell script. If this change is OK, I'll move the other download helpers to their own shell script.

I would actually move all of $(DOWNLOAD) into a single helper script. I
expect that that will give you a lot of refactoring opportunities.