Skip to contentSkip to conversation

Improve silent builds (branch yem/fabio/silent)

PATCH Superseded Yann E. MORIN 3 participants 12 messages Jan 1, 2015 – Jan 2, 2015 Kbuild

v9 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Yann E. MORIN

author
#0 lore raw

v9 cover Improve silent builds (branch yem/fabio/silent)

Hello All!

This patch set improves silent builds.

Silent builds are also nice when top-level parallel make is being used to reduce the output clutter.

Some downloaders, like git or wget, insist on reporting progress on stderr instead of stdout, so redirecting stdout is not enough, we need to pass the quiet information all the way down to the beackends, so they know they have to be silent. Sigh... :-(

Changes v8 -> v9:
- propagate to all download backends (Fabio)

Changes v7 -> v8:
- simplify by just consigning stdout to oblivion (Yann)

Regards,
Yann E. MORIN.

The following changes since commit b7e7b417fa1210a7f9c4e76ee747238981dcd012:

wpa_supplicant: fix indentation and change file permissions (2015-01-01 11:31:14 +0100)

are available in the git repository at:

git://git.busybox.net/~ymorin/git/buildroot yem/fabio/silent

for you to fetch changes up to 8745e57788ef111547ca0a3f4b407b09cafb20f0:

package/pkg-download: do not hard-code QUIET (2015-01-01 21:00:16 +0100)

----------------------------------------------------------------
Fabio Porcedda (2):
Makefile: improve detection of make "-s" flag
pkg-download: silence downloads if make is silent

Yann E. MORIN (2):
support/download: support -q in all download backends package/pkg-download: do not hard-code QUIET

 Makefile                    |  2 +-
 package/pkg-download.mk     | 16 ++++++++++++----
 support/download/bzr        | 23 ++++++++++++++++-------
 support/download/check-hash |  8 ++++++++
 support/download/cp         | 20 +++++++++++++++-----
 support/download/cvs        | 23 +++++++++++++++--------
 support/download/dl-wrapper | 13 +++++++------
 support/download/git        | 24 ++++++++++++++++--------
 support/download/hg         | 24 ++++++++++++++++--------
 support/download/scp        | 20 +++++++++++++++-----
 support/download/svn        | 22 +++++++++++++++-------
 support/download/wget       | 20 +++++++++++++++-----
 12 files changed, 151 insertions(+), 64 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
author
#1 lore raw

PATCH v9 1/4 Makefile: improve detection of make "-s" flag

From: Fabio Porcedda <[email protected]>

Because it's just checking the presence of the "s" character even a
make --warn-undefined-variables
is detected as a silent build.

Fix that by filtering out long options.

Signed-off-by Fabio Porcedda <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Acked-by "Yann E. MORIN" <[email protected]>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
index 5e0b4f2..9995df7 100644
@@ -304,7 +304,7 @@ GNU_HOST_NAME := $(shell support/gnuconfig/config.guess)
 TARGETS :=
 
 # silent mode requested?
-QUIET := $(if $(findstring s,$(MAKEFLAGS)),-q)
+QUIET := $(if $(findstring s,$(filter-out --%,$(MAKEFLAGS))),-q)
 
 # Strip off the annoying quoting
 ARCH := $(call qstrip,$(BR2_ARCH))
Signature · 1 line
1.9.1
Applied
#2 lore raw

PATCH v9 1/4 Makefile: improve detection of make "-s" flag

Dear Yann E. MORIN,

Quoted 6 lines · Yann E. MORIN

From: Fabio Porcedda <[email protected]>

Because it's just checking the presence of the "s" character even a
make --warn-undefined-variables
is detected as a silent build.

Quoted 8 lines

Fix that by filtering out long options.

Signed-off-by Fabio Porcedda <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Acked-by "Yann E. MORIN" <[email protected]>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Applied, thanks.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
author
#3 lore raw

PATCH v9 2/4 support/download: support -q in all download backends

In reply to Yann E. MORIN

Add an option flag to all backends, as well as the check-hash script, so as to silence download helpers when the user wants a silent build.

Additionaly, make the default verbose.

Inspired by Fabio's patch on git/svn.

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Fabio Porcedda <[email protected]>

---
Note: cvs, cp and scp are not tested, because we do not have in-tree packages using those transport methods.

---
 support/download/bzr        | 23 ++++++++++++++++-------
 support/download/check-hash |  8 ++++++++
 support/download/cp         | 20 +++++++++++++++-----
 support/download/cvs        | 23 +++++++++++++++--------
 support/download/git        | 24 ++++++++++++++++--------
 support/download/hg         | 24 ++++++++++++++++--------
 support/download/scp        | 20 +++++++++++++++-----
 support/download/svn        | 22 +++++++++++++++-------
 support/download/wget       | 20 +++++++++++++++-----
 9 files changed, 131 insertions(+), 53 deletions(-)
support/download/bzr +16 -7
index c157ca8..96b0844 100755
@@ -4,17 +4,26 @@
 set -e
 
 # Download helper for bzr, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: bzr repo
-#   $3: bzr revision
-#   $4: basename
-# And this environment:
+#
+# Call it as:
+#   .../bzr [-q] OUT_FILE REPO_URL REV BASENAME
+#
+# Environemnt:
 #   BZR      : the bzr command to call
 
+
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 basename="${4}"
 
-${BZR} export --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
+${BZR} export ${verbose} --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
support/download/check-hash +8 -0
index b59fd2a..4c07274 100755
@@ -10,6 +10,14 @@ set -e
 #       saved as, to be able to match it to the corresponding hashes
 #       in the .hash file
 
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  exec >/dev/null;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 h_file="${1}"
 file="${2}"
 base="${3}"
support/download/cp +15 -5
index 463fc38..1c90f15 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for cp, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: source file
-# And this environment:
+#
+# Call it as:
+#   .../cp [-q] OUT_FILE SRC_FILE
+#
+# Environment:
 #   LOCALFILES: the cp command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 source="${2}"
 
-${LOCALFILES} "${source}" "${output}"
+${LOCALFILES} ${verbose} "${source}" "${output}"
support/download/cvs +15 -8
index 56a11c2..eca6fc4 100755
@@ -4,22 +4,29 @@
 set -e
 
 # Download helper for cvs, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: cvs repo
-#   $3: cvs revision
-#   $4: package's name (eg. foobar)
-#   $5: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME
+#
+# Environment:
 #   CVS      : the cvs command to call
 
+verbose=
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-Q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 rawname="${4}"
 basename="${5}"
 
-${CVS} -z3 -d":pserver:anonymous@${repo}" \
+${CVS} ${verbose} -z3 -d":pserver:anonymous@${repo}" \
        co -d "${basename}" -r ":${rev}" -P "${rawname}"
 
 tar czf "${output}" "${basename}"
support/download/git +16 -8
index 5d36ca4..0a0e81c 100755
@@ -4,14 +4,22 @@
 set -e
 
 # Download helper for git, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: git repo
-#   $3: git cset
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../git [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
 #   GIT      : the git command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 cset="${3}"

@@ -22,7 +30,7 @@ basename="${4}"
 git_done=0
 if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
     printf "Doing shallow clone\n"
-    if ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
+    if ${GIT} clone ${verbose} --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
         git_done=1
     else
         printf "Shallow clone failed, falling back to doing a full clone\n"

@@ -30,7 +38,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
 fi
 if [ ${git_done} -eq 0 ]; then
     printf "Doing full clone\n"
-    ${GIT} clone --bare "${repo}" "${basename}"
+    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
 fi
 
 GIT_DIR="${basename}" \
support/download/hg +16 -8
index 66bd2ed..0d212c0 100755
@@ -4,21 +4,29 @@
 set -e
 
 # Download helper for hg, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: hg repo
-#   $3: hg cset
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../hg [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
 #   HG       : the hg command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 cset="${3}"
 basename="${4}"
 
-${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
+${HG} clone ${verbose} --noupdate --rev "${cset}" "${repo}" "${basename}"
 
-${HG} archive --repository "${basename}" --type tgz \
+${HG} archive ${verbose} --repository "${basename}" --type tgz \
               --prefix "${basename}" --rev "${cset}" \
               "${output}"
support/download/scp +15 -5
index f3e92f3..189e5a4 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for scp, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: URL
-# And this environment:
+#
+# Call it as:
+#   .../scp [-q] OUT_FILE SRC_URL
+#
+# Environment:
 #   SCP       : the scp command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 url="${2}"
 
-${SCP} "${url}" "${output}"
+${SCP} ${verbose} "${url}" "${output}"
support/download/svn +15 -7
index a960f7d..45fc4bf 100755
@@ -4,19 +4,27 @@
 set -e
 
 # Download helper for svn, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: svn repo
-#   $3: svn revision
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../svn [-q] OUT_FILE REPO_URL REV BASNAME
+#
+# Environment:
 #   SVN      : the svn command to call
 
+verbose=
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 basename="${4}"
 
-${SVN} export "${repo}@${rev}" "${basename}"
+${SVN} export ${verbose} "${repo}@${rev}" "${basename}"
 
 tar czf "${output}" "${basename}"
support/download/wget +15 -5
index 6b73726..e7e1d38 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for wget, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: URL
-# And this environment:
+#
+# Call it as:
+#   .../wget [-q] OUT_FILE URL
+#
+# Environment:
 #   WGET     : the wget command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 url="${2}"
 
-${WGET} -O "${output}" "${url}"
+${WGET} ${verbose} -O "${output}" "${url}"
Signature · 1 line
1.9.1
#4 lore raw

PATCH v9 2/4 support/download: support -q in all download backends

Quoted 39 lines · Yann E. MORIN

Add an option flag to all backends, as well as the check-hash script, so as to silence download helpers when the user wants a silent build.

Additionaly, make the default verbose.

Inspired by Fabio's patch on git/svn.

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Fabio Porcedda <[email protected]>

---
Note: cvs, cp and scp are not tested, because we do not have in-tree packages using those transport methods.

---
 support/download/bzr        | 23 ++++++++++++++++-------
 support/download/check-hash |  8 ++++++++
 support/download/cp         | 20 +++++++++++++++-----
 support/download/cvs        | 23 +++++++++++++++--------
 support/download/git        | 24 ++++++++++++++++--------
 support/download/hg         | 24 ++++++++++++++++--------
 support/download/scp        | 20 +++++++++++++++-----
 support/download/svn        | 22 +++++++++++++++-------
 support/download/wget       | 20 +++++++++++++++-----
 9 files changed, 131 insertions(+), 53 deletions(-)
support/download/bzr +0 -6
index c157ca8..96b0844 100755
@@ -4,17 +4,26 @@
 set -e
 
 # Download helper for bzr, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: bzr repo
-#   $3: bzr revision
-#   $4: basename
-# And this environment:

support/download/bzr line 16

Quoted 10 lines
support/download/bzr +8 -0
+#
+# Call it as:
+#   .../bzr [-q] OUT_FILE REPO_URL REV BASENAME
+#
+# Environemnt:
 #   BZR      : the bzr command to call
 
+
+verbose=-v
+while getopts :q OPT; do

The colon should be removed:

:q -> q

because with the colon if there is a wrong option it exit without giving any error message, but without the colon it gives the following error:
./support/download/bzr: illegal option -- t

This is valid for all the backends.

BR

Signature · 1 line
Fabio Porcedda
Quoted 309 lines
support/download/bzr +8 -1
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 basename="${4}"
 
-${BZR} export --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
+${BZR} export ${verbose} --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
support/download/check-hash +8 -0
index b59fd2a..4c07274 100755
@@ -10,6 +10,14 @@ set -e
 #       saved as, to be able to match it to the corresponding hashes
 #       in the .hash file
 
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  exec >/dev/null;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 h_file="${1}"
 file="${2}"
 base="${3}"
support/download/cp +15 -5
index 463fc38..1c90f15 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for cp, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: source file
-# And this environment:
+#
+# Call it as:
+#   .../cp [-q] OUT_FILE SRC_FILE
+#
+# Environment:
 #   LOCALFILES: the cp command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 source="${2}"
 
-${LOCALFILES} "${source}" "${output}"
+${LOCALFILES} ${verbose} "${source}" "${output}"
support/download/cvs +15 -8
index 56a11c2..eca6fc4 100755
@@ -4,22 +4,29 @@
 set -e
 
 # Download helper for cvs, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: cvs repo
-#   $3: cvs revision
-#   $4: package's name (eg. foobar)
-#   $5: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME
+#
+# Environment:
 #   CVS      : the cvs command to call
 
+verbose=
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-Q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 rawname="${4}"
 basename="${5}"
 
-${CVS} -z3 -d":pserver:anonymous@${repo}" \
+${CVS} ${verbose} -z3 -d":pserver:anonymous@${repo}" \
        co -d "${basename}" -r ":${rev}" -P "${rawname}"
 
 tar czf "${output}" "${basename}"
support/download/git +16 -8
index 5d36ca4..0a0e81c 100755
@@ -4,14 +4,22 @@
 set -e
 
 # Download helper for git, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: git repo
-#   $3: git cset
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../git [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
 #   GIT      : the git command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 cset="${3}"

@@ -22,7 +30,7 @@ basename="${4}"
 git_done=0
 if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
     printf "Doing shallow clone\n"
-    if ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
+    if ${GIT} clone ${verbose} --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
         git_done=1
     else
         printf "Shallow clone failed, falling back to doing a full clone\n"

@@ -30,7 +38,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
 fi
 if [ ${git_done} -eq 0 ]; then
     printf "Doing full clone\n"
-    ${GIT} clone --bare "${repo}" "${basename}"
+    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
 fi
 
 GIT_DIR="${basename}" \
support/download/hg +16 -8
index 66bd2ed..0d212c0 100755
@@ -4,21 +4,29 @@
 set -e
 
 # Download helper for hg, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: hg repo
-#   $3: hg cset
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../hg [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
 #   HG       : the hg command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 cset="${3}"
 basename="${4}"
 
-${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
+${HG} clone ${verbose} --noupdate --rev "${cset}" "${repo}" "${basename}"
 
-${HG} archive --repository "${basename}" --type tgz \
+${HG} archive ${verbose} --repository "${basename}" --type tgz \
               --prefix "${basename}" --rev "${cset}" \
               "${output}"
support/download/scp +15 -5
index f3e92f3..189e5a4 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for scp, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: URL
-# And this environment:
+#
+# Call it as:
+#   .../scp [-q] OUT_FILE SRC_URL
+#
+# Environment:
 #   SCP       : the scp command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 url="${2}"
 
-${SCP} "${url}" "${output}"
+${SCP} ${verbose} "${url}" "${output}"
support/download/svn +15 -7
index a960f7d..45fc4bf 100755
@@ -4,19 +4,27 @@
 set -e
 
 # Download helper for svn, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: svn repo
-#   $3: svn revision
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../svn [-q] OUT_FILE REPO_URL REV BASNAME
+#
+# Environment:
 #   SVN      : the svn command to call
 
+verbose=
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 basename="${4}"
 
-${SVN} export "${repo}@${rev}" "${basename}"
+${SVN} export ${verbose} "${repo}@${rev}" "${basename}"
 
 tar czf "${output}" "${basename}"
support/download/wget +15 -6
index 6b73726..e7e1d38 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for wget, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: URL
-# And this environment:
+#
+# Call it as:
+#   .../wget [-q] OUT_FILE URL
+#
+# Environment:
 #   WGET     : the wget command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 url="${2}"
 
-${WGET} -O "${output}" "${url}"
+${WGET} ${verbose} -O "${output}" "${url}"
--

1.9.1

author
#5 lore raw

PATCH v9 2/4 support/download: support -q in all download backends

Fabio, All,

On 2015-01-02 16:01 +0100, Fabio Porcedda spake thusly:

Quoted 53 lines · Yann E. MORIN
Quoted 39 lines

Add an option flag to all backends, as well as the check-hash script, so as to silence download helpers when the user wants a silent build.

Additionaly, make the default verbose.

Inspired by Fabio's patch on git/svn.

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Fabio Porcedda <[email protected]>

---
Note: cvs, cp and scp are not tested, because we do not have in-tree packages using those transport methods.

---
 support/download/bzr        | 23 ++++++++++++++++-------
 support/download/check-hash |  8 ++++++++
 support/download/cp         | 20 +++++++++++++++-----
 support/download/cvs        | 23 +++++++++++++++--------
 support/download/git        | 24 ++++++++++++++++--------
 support/download/hg         | 24 ++++++++++++++++--------
 support/download/scp        | 20 +++++++++++++++-----
 support/download/svn        | 22 +++++++++++++++-------
 support/download/wget       | 20 +++++++++++++++-----
 9 files changed, 131 insertions(+), 53 deletions(-)
support/download/bzr +0 -6
index c157ca8..96b0844 100755
@@ -4,17 +4,26 @@
 set -e
 
 # Download helper for bzr, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: bzr repo
-#   $3: bzr revision
-#   $4: basename
-# And this environment:

support/download/bzr line 16

Quoted 10 lines
support/download/bzr +8 -0
+#
+# Call it as:
+#   .../bzr [-q] OUT_FILE REPO_URL REV BASENAME
+#
+# Environemnt:
 #   BZR      : the bzr command to call
 
+
+verbose=-v
+while getopts :q OPT; do

The colon should be removed:

Quoted 8 lines

:q -> q

because with the colon if there is a wrong option it exit without giving any error message, but without the colon it gives the following error:
./support/download/bzr: illegal option -- t

This is valid for all the backends.

Yes, that's expected behaviour, and the exit is handled...

Quoted 3 lines

support/download/bzr line 19

Quoted 3 lines
support/download/bzr +3 -0
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;

... there. The backends are *not* meant to be called directly, so them failing silently when improper option is pased is perfectly acceptable IMHO.

But we could add a printf as well if you really want. ;-) I'll do that before respinning...

Regards,
Yann E. MORIN.

Quoted 306 lines
Quoted 306 lines
support/download/bzr +5 -1
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 basename="${4}"
 
-${BZR} export --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
+${BZR} export ${verbose} --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
support/download/check-hash +8 -0
index b59fd2a..4c07274 100755
@@ -10,6 +10,14 @@ set -e
 #       saved as, to be able to match it to the corresponding hashes
 #       in the .hash file
 
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  exec >/dev/null;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 h_file="${1}"
 file="${2}"
 base="${3}"
support/download/cp +15 -5
index 463fc38..1c90f15 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for cp, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: source file
-# And this environment:
+#
+# Call it as:
+#   .../cp [-q] OUT_FILE SRC_FILE
+#
+# Environment:
 #   LOCALFILES: the cp command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 source="${2}"
 
-${LOCALFILES} "${source}" "${output}"
+${LOCALFILES} ${verbose} "${source}" "${output}"
support/download/cvs +15 -8
index 56a11c2..eca6fc4 100755
@@ -4,22 +4,29 @@
 set -e
 
 # Download helper for cvs, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: cvs repo
-#   $3: cvs revision
-#   $4: package's name (eg. foobar)
-#   $5: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME
+#
+# Environment:
 #   CVS      : the cvs command to call
 
+verbose=
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-Q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 rawname="${4}"
 basename="${5}"
 
-${CVS} -z3 -d":pserver:anonymous@${repo}" \
+${CVS} ${verbose} -z3 -d":pserver:anonymous@${repo}" \
        co -d "${basename}" -r ":${rev}" -P "${rawname}"
 
 tar czf "${output}" "${basename}"
support/download/git +16 -8
index 5d36ca4..0a0e81c 100755
@@ -4,14 +4,22 @@
 set -e
 
 # Download helper for git, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: git repo
-#   $3: git cset
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../git [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
 #   GIT      : the git command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 cset="${3}"

@@ -22,7 +30,7 @@ basename="${4}"
 git_done=0
 if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
     printf "Doing shallow clone\n"
-    if ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
+    if ${GIT} clone ${verbose} --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
         git_done=1
     else
         printf "Shallow clone failed, falling back to doing a full clone\n"

@@ -30,7 +38,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
 fi
 if [ ${git_done} -eq 0 ]; then
     printf "Doing full clone\n"
-    ${GIT} clone --bare "${repo}" "${basename}"
+    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
 fi
 
 GIT_DIR="${basename}" \
support/download/hg +16 -8
index 66bd2ed..0d212c0 100755
@@ -4,21 +4,29 @@
 set -e
 
 # Download helper for hg, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: hg repo
-#   $3: hg cset
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../hg [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
 #   HG       : the hg command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 cset="${3}"
 basename="${4}"
 
-${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
+${HG} clone ${verbose} --noupdate --rev "${cset}" "${repo}" "${basename}"
 
-${HG} archive --repository "${basename}" --type tgz \
+${HG} archive ${verbose} --repository "${basename}" --type tgz \
               --prefix "${basename}" --rev "${cset}" \
               "${output}"
support/download/scp +15 -5
index f3e92f3..189e5a4 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for scp, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: URL
-# And this environment:
+#
+# Call it as:
+#   .../scp [-q] OUT_FILE SRC_URL
+#
+# Environment:
 #   SCP       : the scp command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 url="${2}"
 
-${SCP} "${url}" "${output}"
+${SCP} ${verbose} "${url}" "${output}"
support/download/svn +15 -7
index a960f7d..45fc4bf 100755
@@ -4,19 +4,27 @@
 set -e
 
 # Download helper for svn, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: svn repo
-#   $3: svn revision
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../svn [-q] OUT_FILE REPO_URL REV BASNAME
+#
+# Environment:
 #   SVN      : the svn command to call
 
+verbose=
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 basename="${4}"
 
-${SVN} export "${repo}@${rev}" "${basename}"
+${SVN} export ${verbose} "${repo}@${rev}" "${basename}"
 
 tar czf "${output}" "${basename}"
support/download/wget +15 -6
index 6b73726..e7e1d38 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for wget, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: URL
-# And this environment:
+#
+# Call it as:
+#   .../wget [-q] OUT_FILE URL
+#
+# Environment:
 #   WGET     : the wget command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 url="${2}"
 
-${WGET} -O "${output}" "${url}"
+${WGET} ${verbose} -O "${output}" "${url}"
--

1.9.1

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
author
#6 lore raw

PATCH v9 3/4 pkg-download: silence downloads if make is silent

In reply to Yann E. MORIN

From: Fabio Porcedda <[email protected]>

If it is a silent build (make -s -> QUIET=-q) silence all downloads, as well as the check-hash. This is done in two ways:
- redirect stdout to /dev/null
- pass the -q flag downward to backends

Change a printf to use the trace functions.

Signed-off-by Fabio Porcedda <[email protected]>
[yann.morin.1998 at free.fr: consign stdout to oblivion]
Signed-off-by Yann E. MORIN <[email protected]>
---
 package/pkg-download.mk     |  8 ++++++++
 support/download/dl-wrapper | 13 +++++++------
 2 files changed, 15 insertions(+), 6 deletions(-)
package/pkg-download.mk +8 -0
index ba72fc1..9c782fe 100644
@@ -88,6 +88,7 @@ define DOWNLOAD_GIT
 	$(EXTRA_ENV) $(DL_WRAPPER) -b git \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \

@@ -109,6 +110,7 @@ define DOWNLOAD_BZR
 	$(EXTRA_ENV) $(DL_WRAPPER) -b bzr \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \

@@ -127,6 +129,7 @@ define DOWNLOAD_CVS
 	$(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
 		$($(PKG)_DL_VERSION) \

@@ -147,6 +150,7 @@ define DOWNLOAD_SVN
 	$(EXTRA_ENV) $(DL_WRAPPER) -b svn \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \

@@ -168,6 +172,7 @@ define DOWNLOAD_SCP
 	$(EXTRA_ENV) $(DL_WRAPPER) -b scp \
 		-o $(DL_DIR)/$(2) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		'$(call stripurischeme,$(call qstrip,$(1)))'
 endef

@@ -185,6 +190,7 @@ define DOWNLOAD_HG
 	$(EXTRA_ENV) $(DL_WRAPPER) -b hg \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \

@@ -206,6 +212,7 @@ define DOWNLOAD_WGET
 	$(EXTRA_ENV) $(DL_WRAPPER) -b wget \
 		-o $(DL_DIR)/$(2) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		'$(call qstrip,$(1))'
 endef

@@ -222,6 +229,7 @@ define DOWNLOAD_LOCALFILES
 	$(EXTRA_ENV) $(DL_WRAPPER) -b cp \
 		-o $(DL_DIR)/$(2) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$(call stripurischeme,$(call qstrip,$(1)))
 endef
support/download/dl-wrapper +7 -6
index cced8f6..ffe111d 100755
@@ -21,15 +21,16 @@ set -e
 
 main() {
     local OPT OPTARG
-    local backend output hfile
+    local backend output hfile quiet
 
     # Parse our options; anything after '--' is for the backend
-    while getopts :hb:o:H: OPT; do
+    while getopts :hb:o:H:q OPT; do
         case "${OPT}" in
         h)  help; exit 0;;
         b)  backend="${OPTARG}";;
         o)  output="${OPTARG}";;
         H)  hfile="${OPTARG}";;
+        q)  quiet="-q"; exec >/dev/null;;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
         esac

@@ -49,11 +50,11 @@ main() {
 
     # If the output file already exists, do not download it again
     if [ -e "${output}" ]; then
-        if support/download/check-hash "${hfile}" "${output}" "${output##*/}"; then
+        if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
             exit 0
         fi
         rm -f "${output}"
-        printf "Re-downloading '%s'...\n" "${output##*/}"
+        warn "Re-downloading '%s'...\n" "${output##*/}"
     fi
 
     # tmpd is a temporary directory in which backends may store intermediate

@@ -75,7 +76,7 @@ main() {
     # If the backend fails, we can just remove the temporary directory to
     # remove all the cruft it may have left behind. Then we just exit in
     # error too.
-    if ! "${OLDPWD}/support/download/${backend}" "${tmpf}" "${@}"; then
+    if ! "${OLDPWD}/support/download/${backend}" ${quiet} "${tmpf}" "${@}"; then
         rm -rf "${tmpd}"
         exit 1
     fi

@@ -85,7 +86,7 @@ main() {
 
     # Check if the downloaded file is sane, and matches the stored hashes
     # for that file
-    if ! support/download/check-hash "${hfile}" "${tmpf}" "${output##*/}"; then
+    if ! support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
         rm -rf "${tmpd}"
         exit 1
     fi
Signature · 1 line
1.9.1
#7 lore raw

PATCH v9 3/4 pkg-download: silence downloads if make is silent

Dear Yann E. MORIN,

Quoted 6 lines · Yann E. MORIN

From: Fabio Porcedda <[email protected]>

If it is a silent build (make -s -> QUIET=-q) silence all downloads, as well as the check-hash. This is done in two ways:
- redirect stdout to /dev/null
- pass the -q flag downward to backends

Why do we need to do both?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
author
#8 lore raw

PATCH v9 3/4 pkg-download: silence downloads if make is silent

Thomas, All,

On 2015-01-01 22:56 +0100, Thomas Petazzoni spake thusly:

Quoted 9 lines · Yann E. MORIN
Quoted 6 lines

From: Fabio Porcedda <[email protected]>

If it is a silent build (make -s -> QUIET=-q) silence all downloads, as well as the check-hash. This is done in two ways:
- redirect stdout to /dev/null
- pass the -q flag downward to backends

Why do we need to do both?

Redirecting stdout is a further solution for those downloaders that still want to output something on stdout, even when told to be quiet.

I could not test CVS, CP and SCP because we do use them in in-tree packages, so it is just an additional precaution.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#9 lore raw

PATCH v9 3/4 pkg-download: silence downloads if make is silent

Dear Yann E. MORIN,

Quoted 5 lines · Yann E. MORIN

Redirecting stdout is a further solution for those downloaders that still want to output something on stdout, even when told to be quiet.

I could not test CVS, CP and SCP because we do use them in in-tree packages, so it is just an additional precaution.

Well, do we want to take "additional precautions" ? While I originally found the redirect solution nice when you proposed it, the fact that it doesn't work with certain backends makes it not very useful. So I would tend to defer the work of being quiet to each individual backend, and not attempt to do anything specific at the wrapper level.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
author
#10 lore raw

PATCH v9 3/4 pkg-download: silence downloads if make is silent

Thomas, All,

On 2015-01-02 16:36 +0100, Thomas Petazzoni spake thusly:

Quoted 8 lines · Yann E. MORIN
Quoted 6 lines

Redirecting stdout is a further solution for those downloaders that still want to output something on stdout, even when told to be quiet.

I could not test CVS, CP and SCP because we do use them in in-tree packages, so it is just an additional precaution.

Quoted 6 lines

Well, do we want to take "additional precautions" ? While I originally found the redirect solution nice when you proposed it, the fact that it doesn't work with certain backends makes it not very useful. So I would tend to defer the work of being quiet to each individual backend, and not attempt to do anything specific at the wrapper level.

OK, will do.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
author
#11 lore raw

PATCH v9 4/4 package/pkg-download: do not hard-code QUIET

In reply to Yann E. MORIN

Do not hard-code QUIET in our download commands, since it is handled in the backends.

Suggested by Fabio.

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Fabio Porcedda <[email protected]>
---
 package/pkg-download.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
package/pkg-download.mk +4 -4
index 9c782fe..5e74519 100644
@@ -8,14 +8,14 @@
 ################################################################################
 
 # Download method commands
-export WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
+export WGET := $(call qstrip,$(BR2_WGET))
 export SVN := $(call qstrip,$(BR2_SVN))
 export CVS := $(call qstrip,$(BR2_CVS))
 export BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
-export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
-export SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
-SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
+export HG := $(call qstrip,$(BR2_HG))
+export SCP := $(call qstrip,$(BR2_SCP))
+SSH := $(call qstrip,$(BR2_SSH))
 export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
 DL_WRAPPER = support/download/dl-wrapper
Signature · 1 line
1.9.1