Skip to contentSkip to conversation

Improve silent builds (branch yem/fabio/silent)

PATCH New Yann E. MORIN 2 participants 5 messages Jan 2, 2015 – Jan 4, 2015

lore
Share & tools
lore
#0 lore raw

v10 cover Improve silent builds (branch yem/fabio/silent)

Hello All!

This patch set improves silent builds.

Silent builds are also nice when top-level parallel make is being used to reduce the output clutter.

Some downloaders, like git or wget, insist on reporting progress on stderr instead of stdout, so redirecting stdout is not enough, we need to pass the quiet information all the way down to the beackends, so they know they have to be silent. Sigh... :-(

Changes v9 -> v10:
- do not redirect stdout at all (Thomas)
- on unknown options, print an error message in backends (Fabio)

Changes v8 -> v9:
- propagate to all download backends (Fabio)

Changes v7 -> v8:
- simplify by just consigning stdout to oblivion (Yann)

Regards,
Yann E. MORIN.

The following changes since commit 81cbf0bf84bb62d1b40f9ea4e30296b5ef495556:

libevent: add optional dependency on OpenSSL (2015-01-02 16:37:48 +0100)

are available in the git repository at:

git://git.busybox.net/~ymorin/git/buildroot yem/fabio/silent

for you to fetch changes up to 9b12978224418fe368d83e1ca7988619a18e9b74:

package/pkg-download: do not hard-code QUIET (2015-01-02 16:51:53 +0100)

----------------------------------------------------------------
Fabio Porcedda (1):
pkg-download: silence downloads if make is silent

Yann E. MORIN (2):
support/download: support -q in all download backends package/pkg-download: do not hard-code QUIET

 package/pkg-download.mk     | 16 ++++++++++++----
 support/download/bzr        | 23 ++++++++++++++++-------
 support/download/check-hash |  8 ++++++++
 support/download/cp         | 20 +++++++++++++++-----
 support/download/cvs        | 23 +++++++++++++++--------
 support/download/dl-wrapper | 13 +++++++------
 support/download/git        | 24 ++++++++++++++++--------
 support/download/hg         | 24 ++++++++++++++++--------
 support/download/scp        | 20 +++++++++++++++-----
 support/download/svn        | 22 +++++++++++++++-------
 support/download/wget       | 20 +++++++++++++++-----
 11 files changed, 150 insertions(+), 63 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH v10 1/3 support/download: support -q in all download backends

Add an option flag to all backends, as well as the check-hash script, so as to silence download helpers when the user wants a silent build.

Additionaly, make the default be verbose.

Inspired by Fabio's patch on git/svn.

---
Note: cvs, cp and scp are not tested, because we do not have in-tree packages using those transport methods.

---
 support/download/bzr        | 23 ++++++++++++++++-------
 support/download/check-hash |  8 ++++++++
 support/download/cp         | 20 +++++++++++++++-----
 support/download/cvs        | 23 +++++++++++++++--------
 support/download/git        | 24 ++++++++++++++++--------
 support/download/hg         | 24 ++++++++++++++++--------
 support/download/scp        | 20 +++++++++++++++-----
 support/download/svn        | 22 +++++++++++++++-------
 support/download/wget       | 20 +++++++++++++++-----
 9 files changed, 131 insertions(+), 53 deletions(-)
support/download/bzr +16 -7
index c157ca8..5819ac7 100755
@@ -4,17 +4,26 @@
 set -e
 
 # Download helper for bzr, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: bzr repo
-#   $3: bzr revision
-#   $4: basename
-# And this environment:
+#
+# Call it as:
+#   .../bzr [-q] OUT_FILE REPO_URL REV BASENAME
+#
+# Environemnt:
 #   BZR      : the bzr command to call
 
+
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 basename="${4}"
 
-${BZR} export --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
+${BZR} export ${verbose} --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
support/download/check-hash +8 -0
index b59fd2a..4c07274 100755
@@ -10,6 +10,14 @@ set -e
 #       saved as, to be able to match it to the corresponding hashes
 #       in the .hash file
 
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  exec >/dev/null;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 h_file="${1}"
 file="${2}"
 base="${3}"
support/download/cp +15 -5
index 463fc38..132cad7 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for cp, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: source file
-# And this environment:
+#
+# Call it as:
+#   .../cp [-q] OUT_FILE SRC_FILE
+#
+# Environment:
 #   LOCALFILES: the cp command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 source="${2}"
 
-${LOCALFILES} "${source}" "${output}"
+${LOCALFILES} ${verbose} "${source}" "${output}"
support/download/cvs +15 -8
index 56a11c2..2c3a666 100755
@@ -4,22 +4,29 @@
 set -e
 
 # Download helper for cvs, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: cvs repo
-#   $3: cvs revision
-#   $4: package's name (eg. foobar)
-#   $5: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME
+#
+# Environment:
 #   CVS      : the cvs command to call
 
+verbose=
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-Q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 rawname="${4}"
 basename="${5}"
 
-${CVS} -z3 -d":pserver:anonymous@${repo}" \
+${CVS} ${verbose} -z3 -d":pserver:anonymous@${repo}" \
        co -d "${basename}" -r ":${rev}" -P "${rawname}"
 
 tar czf "${output}" "${basename}"
support/download/git +16 -8
index 5d36ca4..7f1801f 100755
@@ -4,14 +4,22 @@
 set -e
 
 # Download helper for git, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: git repo
-#   $3: git cset
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../git [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
 #   GIT      : the git command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 cset="${3}"
@@ -22,7 +30,7 @@ basename="${4}"
 git_done=0
 if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
     printf "Doing shallow clone\n"
-    if ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
+    if ${GIT} clone ${verbose} --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
         git_done=1
     else
         printf "Shallow clone failed, falling back to doing a full clone\n"
@@ -30,7 +38,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
 fi
 if [ ${git_done} -eq 0 ]; then
     printf "Doing full clone\n"
-    ${GIT} clone --bare "${repo}" "${basename}"
+    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
 fi
 
 GIT_DIR="${basename}" \
support/download/hg +16 -8
index 66bd2ed..f6880fa 100755
@@ -4,21 +4,29 @@
 set -e
 
 # Download helper for hg, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: hg repo
-#   $3: hg cset
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../hg [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
 #   HG       : the hg command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 cset="${3}"
 basename="${4}"
 
-${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
+${HG} clone ${verbose} --noupdate --rev "${cset}" "${repo}" "${basename}"
 
-${HG} archive --repository "${basename}" --type tgz \
+${HG} archive ${verbose} --repository "${basename}" --type tgz \
               --prefix "${basename}" --rev "${cset}" \
               "${output}"
support/download/scp +15 -5
index f3e92f3..3d6876f 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for scp, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: URL
-# And this environment:
+#
+# Call it as:
+#   .../scp [-q] OUT_FILE SRC_URL
+#
+# Environment:
 #   SCP       : the scp command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 url="${2}"
 
-${SCP} "${url}" "${output}"
+${SCP} ${verbose} "${url}" "${output}"
support/download/svn +15 -7
index a960f7d..558bca0 100755
@@ -4,19 +4,27 @@
 set -e
 
 # Download helper for svn, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: svn repo
-#   $3: svn revision
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../svn [-q] OUT_FILE REPO_URL REV BASNAME
+#
+# Environment:
 #   SVN      : the svn command to call
 
+verbose=
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 basename="${4}"
 
-${SVN} export "${repo}@${rev}" "${basename}"
+${SVN} export ${verbose} "${repo}@${rev}" "${basename}"
 
 tar czf "${output}" "${basename}"
support/download/wget +15 -5
index 6b73726..7eb21a4 100755
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for wget, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: URL
-# And this environment:
+#
+# Call it as:
+#   .../wget [-q] OUT_FILE URL
+#
+# Environment:
 #   WGET     : the wget command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 url="${2}"
 
-${WGET} -O "${output}" "${url}"
+${WGET} ${verbose} -O "${output}" "${url}"
Signature · 1 line
1.9.1
#2 lore raw

PATCH v10 2/3 pkg-download: silence downloads if make is silent

In reply to Yann E. MORIN

From: Fabio Porcedda <[email protected]>

If doing a silent build (make -s -> QUIET=-q), silence all downloads, by passing the -q flag downward to backends as well as to check-hash.

Change a printf to use the trace functions.

---
 package/pkg-download.mk     |  8 ++++++++
 support/download/dl-wrapper | 13 +++++++------
 2 files changed, 15 insertions(+), 6 deletions(-)
package/pkg-download.mk +8 -0
index ba72fc1..9c782fe 100644
@@ -88,6 +88,7 @@ define DOWNLOAD_GIT
 	$(EXTRA_ENV) $(DL_WRAPPER) -b git \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \
@@ -109,6 +110,7 @@ define DOWNLOAD_BZR
 	$(EXTRA_ENV) $(DL_WRAPPER) -b bzr \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \
@@ -127,6 +129,7 @@ define DOWNLOAD_CVS
 	$(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
 		$($(PKG)_DL_VERSION) \
@@ -147,6 +150,7 @@ define DOWNLOAD_SVN
 	$(EXTRA_ENV) $(DL_WRAPPER) -b svn \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \
@@ -168,6 +172,7 @@ define DOWNLOAD_SCP
 	$(EXTRA_ENV) $(DL_WRAPPER) -b scp \
 		-o $(DL_DIR)/$(2) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		'$(call stripurischeme,$(call qstrip,$(1)))'
 endef
@@ -185,6 +190,7 @@ define DOWNLOAD_HG
 	$(EXTRA_ENV) $(DL_WRAPPER) -b hg \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \
@@ -206,6 +212,7 @@ define DOWNLOAD_WGET
 	$(EXTRA_ENV) $(DL_WRAPPER) -b wget \
 		-o $(DL_DIR)/$(2) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		'$(call qstrip,$(1))'
 endef
@@ -222,6 +229,7 @@ define DOWNLOAD_LOCALFILES
 	$(EXTRA_ENV) $(DL_WRAPPER) -b cp \
 		-o $(DL_DIR)/$(2) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$(call stripurischeme,$(call qstrip,$(1)))
 endef
support/download/dl-wrapper +7 -6
index cced8f6..3b30840 100755
@@ -21,15 +21,16 @@ set -e
 
 main() {
     local OPT OPTARG
-    local backend output hfile
+    local backend output hfile quiet
 
     # Parse our options; anything after '--' is for the backend
-    while getopts :hb:o:H: OPT; do
+    while getopts :hb:o:H:q OPT; do
         case "${OPT}" in
         h)  help; exit 0;;
         b)  backend="${OPTARG}";;
         o)  output="${OPTARG}";;
         H)  hfile="${OPTARG}";;
+        q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
         esac
@@ -49,11 +50,11 @@ main() {
 
     # If the output file already exists, do not download it again
     if [ -e "${output}" ]; then
-        if support/download/check-hash "${hfile}" "${output}" "${output##*/}"; then
+        if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
             exit 0
         fi
         rm -f "${output}"
-        printf "Re-downloading '%s'...\n" "${output##*/}"
+        warn "Re-downloading '%s'...\n" "${output##*/}"
     fi
 
     # tmpd is a temporary directory in which backends may store intermediate
@@ -75,7 +76,7 @@ main() {
     # If the backend fails, we can just remove the temporary directory to
     # remove all the cruft it may have left behind. Then we just exit in
     # error too.
-    if ! "${OLDPWD}/support/download/${backend}" "${tmpf}" "${@}"; then
+    if ! "${OLDPWD}/support/download/${backend}" ${quiet} "${tmpf}" "${@}"; then
         rm -rf "${tmpd}"
         exit 1
     fi
@@ -85,7 +86,7 @@ main() {
 
     # Check if the downloaded file is sane, and matches the stored hashes
     # for that file
-    if ! support/download/check-hash "${hfile}" "${tmpf}" "${output##*/}"; then
+    if ! support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
         rm -rf "${tmpd}"
         exit 1
     fi
Signature · 1 line
1.9.1
#3 lore raw

PATCH v10 3/3 package/pkg-download: do not hard-code QUIET

In reply to Yann E. MORIN

Do not hard-code QUIET in our download commands, since it is handled in the backends.

Suggested by Fabio.

---
 package/pkg-download.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
package/pkg-download.mk +4 -4
index 9c782fe..5e74519 100644
@@ -8,14 +8,14 @@
 ################################################################################
 
 # Download method commands
-export WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
+export WGET := $(call qstrip,$(BR2_WGET))
 export SVN := $(call qstrip,$(BR2_SVN))
 export CVS := $(call qstrip,$(BR2_CVS))
 export BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
-export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
-export SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
-SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
+export HG := $(call qstrip,$(BR2_HG))
+export SCP := $(call qstrip,$(BR2_SCP))
+SSH := $(call qstrip,$(BR2_SSH))
 export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
 DL_WRAPPER = support/download/dl-wrapper
Signature · 1 line
1.9.1
#4 lore raw

v10 cover Improve silent builds (branch yem/fabio/silent)

In reply to Yann E. MORIN

Dear Yann E. MORIN,

Quoted 7 lines · Yann E. MORIN

Silent builds are also nice when top-level parallel make is being used to reduce the output clutter.

Some downloaders, like git or wget, insist on reporting progress on stderr instead of stdout, so redirecting stdout is not enough, we need to pass the quiet information all the way down to the beackends, so they know they have to be silent. Sigh... :-(

Series applied, after fixing a minor typo in the first patch. Thanks to you and Fabio for working on this!

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com