Improve silent builds (branch yem/fabio/silent)
v10 cover Improve silent builds (branch yem/fabio/silent)
Hello All!
This patch set improves silent builds.
Silent builds are also nice when top-level parallel make is being used to reduce the output clutter.
Some downloaders, like git or wget, insist on reporting progress on stderr instead of stdout, so redirecting stdout is not enough, we need to pass the quiet information all the way down to the beackends, so they know they have to be silent. Sigh... :-(
Changes v9 -> v10:
- do not redirect stdout at all (Thomas)
- on unknown options, print an error message in backends (Fabio)
Changes v8 -> v9:
- propagate to all download backends (Fabio)
Changes v7 -> v8:
- simplify by just consigning stdout to oblivion (Yann)
Regards,
Yann E. MORIN.
The following changes since commit 81cbf0bf84bb62d1b40f9ea4e30296b5ef495556:
libevent: add optional dependency on OpenSSL (2015-01-02 16:37:48 +0100)
are available in the git repository at:
git://git.busybox.net/~ymorin/git/buildroot yem/fabio/silent
for you to fetch changes up to 9b12978224418fe368d83e1ca7988619a18e9b74:
package/pkg-download: do not hard-code QUIET (2015-01-02 16:51:53 +0100)
----------------------------------------------------------------
Fabio Porcedda (1):
pkg-download: silence downloads if make is silent
Yann E. MORIN (2):
support/download: support -q in all download backends package/pkg-download: do not hard-code QUIET
package/pkg-download.mk | 16 ++++++++++++---- support/download/bzr | 23 ++++++++++++++++------- support/download/check-hash | 8 ++++++++ support/download/cp | 20 +++++++++++++++----- support/download/cvs | 23 +++++++++++++++-------- support/download/dl-wrapper | 13 +++++++------ support/download/git | 24 ++++++++++++++++-------- support/download/hg | 24 ++++++++++++++++-------- support/download/scp | 20 +++++++++++++++----- support/download/svn | 22 +++++++++++++++------- support/download/wget | 20 +++++++++++++++----- 11 files changed, 150 insertions(+), 63 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v10 1/3 support/download: support -q in all download backends
Add an option flag to all backends, as well as the check-hash script, so as to silence download helpers when the user wants a silent build.
Additionaly, make the default be verbose.
Inspired by Fabio's patch on git/svn.
---
Note: cvs, cp and scp are not tested, because we do not have in-tree packages using those transport methods.
--- support/download/bzr | 23 ++++++++++++++++------- support/download/check-hash | 8 ++++++++ support/download/cp | 20 +++++++++++++++----- support/download/cvs | 23 +++++++++++++++-------- support/download/git | 24 ++++++++++++++++-------- support/download/hg | 24 ++++++++++++++++-------- support/download/scp | 20 +++++++++++++++----- support/download/svn | 22 +++++++++++++++------- support/download/wget | 20 +++++++++++++++----- 9 files changed, 131 insertions(+), 53 deletions(-)
support/download/bzr +16 -7
@@ -4,17 +4,26 @@ set -e # Download helper for bzr, to be called from the download wrapper script -# Expected arguments: -# $1: output file -# $2: bzr repo -# $3: bzr revision -# $4: basename -# And this environment: +# +# Call it as: +# .../bzr [-q] OUT_FILE REPO_URL REV BASENAME +# +# Environemnt: # BZR : the bzr command to call + +verbose=-v +while getopts :q OPT; do + case "${OPT}" in + q) verbose=-q;; + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; + esac +done +shift $((OPTIND-1)) + output="${1}" repo="${2}" rev="${3}" basename="${4}" -${BZR} export --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}" +${BZR} export ${verbose} --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
support/download/check-hash +8 -0
@@ -10,6 +10,14 @@ set -e # saved as, to be able to match it to the corresponding hashes # in the .hash file +while getopts :q OPT; do + case "${OPT}" in + q) exec >/dev/null;; + \?) exit 1;; + esac +done +shift $((OPTIND-1)) + h_file="${1}" file="${2}" base="${3}"
support/download/cp +15 -5
@@ -4,13 +4,23 @@ set -e # Download helper for cp, to be called from the download wrapper script -# Expected arguments: -# $1: output file -# $2: source file -# And this environment: +# +# Call it as: +# .../cp [-q] OUT_FILE SRC_FILE +# +# Environment: # LOCALFILES: the cp command to call +verbose=-v +while getopts :q OPT; do + case "${OPT}" in + q) verbose=;; + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; + esac +done +shift $((OPTIND-1)) + output="${1}" source="${2}" -${LOCALFILES} "${source}" "${output}" +${LOCALFILES} ${verbose} "${source}" "${output}"
support/download/cvs +15 -8
@@ -4,22 +4,29 @@ set -e # Download helper for cvs, to be called from the download wrapper script -# Expected arguments: -# $1: output file -# $2: cvs repo -# $3: cvs revision -# $4: package's name (eg. foobar) -# $5: package's basename (eg. foobar-1.2.3) -# And this environment: +# +# Call it as: +# .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME +# +# Environment: # CVS : the cvs command to call +verbose= +while getopts :q OPT; do + case "${OPT}" in + q) verbose=-Q;; + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; + esac +done +shift $((OPTIND-1)) + output="${1}" repo="${2}" rev="${3}" rawname="${4}" basename="${5}" -${CVS} -z3 -d":pserver:anonymous@${repo}" \ +${CVS} ${verbose} -z3 -d":pserver:anonymous@${repo}" \ co -d "${basename}" -r ":${rev}" -P "${rawname}" tar czf "${output}" "${basename}"
support/download/git +16 -8
@@ -4,14 +4,22 @@ set -e # Download helper for git, to be called from the download wrapper script -# Expected arguments: -# $1: output file -# $2: git repo -# $3: git cset -# $4: package's basename (eg. foobar-1.2.3) -# And this environment: +# +# Call it as: +# .../git [-q] OUT_FILE REPO_URL CSET BASENAME +# +# Environment: # GIT : the git command to call +verbose=-v +while getopts :q OPT; do + case "${OPT}" in + q) verbose=-q;; + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; + esac +done +shift $((OPTIND-1)) + output="${1}" repo="${2}" cset="${3}"
@@ -22,7 +30,7 @@ basename="${4}" git_done=0 if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then printf "Doing shallow clone\n" - if ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then + if ${GIT} clone ${verbose} --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then git_done=1 else printf "Shallow clone failed, falling back to doing a full clone\n"
@@ -30,7 +38,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then fi if [ ${git_done} -eq 0 ]; then printf "Doing full clone\n" - ${GIT} clone --bare "${repo}" "${basename}" + ${GIT} clone ${verbose} --bare "${repo}" "${basename}" fi GIT_DIR="${basename}" \
support/download/hg +16 -8
@@ -4,21 +4,29 @@ set -e # Download helper for hg, to be called from the download wrapper script -# Expected arguments: -# $1: output file -# $2: hg repo -# $3: hg cset -# $4: package's basename (eg. foobar-1.2.3) -# And this environment: +# +# Call it as: +# .../hg [-q] OUT_FILE REPO_URL CSET BASENAME +# +# Environment: # HG : the hg command to call +verbose=-v +while getopts :q OPT; do + case "${OPT}" in + q) verbose=-q;; + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; + esac +done +shift $((OPTIND-1)) + output="${1}" repo="${2}" cset="${3}" basename="${4}" -${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}" +${HG} clone ${verbose} --noupdate --rev "${cset}" "${repo}" "${basename}" -${HG} archive --repository "${basename}" --type tgz \ +${HG} archive ${verbose} --repository "${basename}" --type tgz \ --prefix "${basename}" --rev "${cset}" \ "${output}"
support/download/scp +15 -5
@@ -4,13 +4,23 @@ set -e # Download helper for scp, to be called from the download wrapper script -# Expected arguments: -# $1: output file -# $2: URL -# And this environment: +# +# Call it as: +# .../scp [-q] OUT_FILE SRC_URL +# +# Environment: # SCP : the scp command to call +verbose=-v +while getopts :q OPT; do + case "${OPT}" in + q) verbose=-q;; + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; + esac +done +shift $((OPTIND-1)) + output="${1}" url="${2}" -${SCP} "${url}" "${output}" +${SCP} ${verbose} "${url}" "${output}"
support/download/svn +15 -7
@@ -4,19 +4,27 @@ set -e # Download helper for svn, to be called from the download wrapper script -# Expected arguments: -# $1: output file -# $2: svn repo -# $3: svn revision -# $4: package's basename (eg. foobar-1.2.3) -# And this environment: +# +# Call it as: +# .../svn [-q] OUT_FILE REPO_URL REV BASNAME +# +# Environment: # SVN : the svn command to call +verbose= +while getopts :q OPT; do + case "${OPT}" in + q) verbose=-q;; + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; + esac +done +shift $((OPTIND-1)) + output="${1}" repo="${2}" rev="${3}" basename="${4}" -${SVN} export "${repo}@${rev}" "${basename}" +${SVN} export ${verbose} "${repo}@${rev}" "${basename}" tar czf "${output}" "${basename}"
support/download/wget +15 -5
@@ -4,13 +4,23 @@ set -e # Download helper for wget, to be called from the download wrapper script -# Expected arguments: -# $1: output file -# $2: URL -# And this environment: +# +# Call it as: +# .../wget [-q] OUT_FILE URL +# +# Environment: # WGET : the wget command to call +verbose=-v +while getopts :q OPT; do + case "${OPT}" in + q) verbose=-q;; + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; + esac +done +shift $((OPTIND-1)) + output="${1}" url="${2}" -${WGET} -O "${output}" "${url}" +${WGET} ${verbose} -O "${output}" "${url}"
Signature · 1 line
1.9.1
PATCH v10 2/3 pkg-download: silence downloads if make is silent
In reply to Yann E. MORINFrom: Fabio Porcedda <[email protected]>
If doing a silent build (make -s -> QUIET=-q), silence all downloads, by passing the -q flag downward to backends as well as to check-hash.
Change a printf to use the trace functions.
--- package/pkg-download.mk | 8 ++++++++ support/download/dl-wrapper | 13 +++++++------ 2 files changed, 15 insertions(+), 6 deletions(-)
package/pkg-download.mk +8 -0
@@ -88,6 +88,7 @@ define DOWNLOAD_GIT $(EXTRA_ENV) $(DL_WRAPPER) -b git \ -o $(DL_DIR)/$($(PKG)_SOURCE) \ -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \ + $(QUIET) \ -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \
@@ -109,6 +110,7 @@ define DOWNLOAD_BZR $(EXTRA_ENV) $(DL_WRAPPER) -b bzr \ -o $(DL_DIR)/$($(PKG)_SOURCE) \ -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \ + $(QUIET) \ -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \
@@ -127,6 +129,7 @@ define DOWNLOAD_CVS $(EXTRA_ENV) $(DL_WRAPPER) -b cvs \ -o $(DL_DIR)/$($(PKG)_SOURCE) \ -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \ + $(QUIET) \ -- \ $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \ $($(PKG)_DL_VERSION) \
@@ -147,6 +150,7 @@ define DOWNLOAD_SVN $(EXTRA_ENV) $(DL_WRAPPER) -b svn \ -o $(DL_DIR)/$($(PKG)_SOURCE) \ -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \ + $(QUIET) \ -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \
@@ -168,6 +172,7 @@ define DOWNLOAD_SCP $(EXTRA_ENV) $(DL_WRAPPER) -b scp \ -o $(DL_DIR)/$(2) \ -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \ + $(QUIET) \ -- \ '$(call stripurischeme,$(call qstrip,$(1)))' endef
@@ -185,6 +190,7 @@ define DOWNLOAD_HG $(EXTRA_ENV) $(DL_WRAPPER) -b hg \ -o $(DL_DIR)/$($(PKG)_SOURCE) \ -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \ + $(QUIET) \ -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \
@@ -206,6 +212,7 @@ define DOWNLOAD_WGET $(EXTRA_ENV) $(DL_WRAPPER) -b wget \ -o $(DL_DIR)/$(2) \ -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \ + $(QUIET) \ -- \ '$(call qstrip,$(1))' endef
@@ -222,6 +229,7 @@ define DOWNLOAD_LOCALFILES $(EXTRA_ENV) $(DL_WRAPPER) -b cp \ -o $(DL_DIR)/$(2) \ -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \ + $(QUIET) \ -- \ $(call stripurischeme,$(call qstrip,$(1))) endef
support/download/dl-wrapper +7 -6
@@ -21,15 +21,16 @@ set -e main() { local OPT OPTARG - local backend output hfile + local backend output hfile quiet # Parse our options; anything after '--' is for the backend - while getopts :hb:o:H: OPT; do + while getopts :hb:o:H:q OPT; do case "${OPT}" in h) help; exit 0;; b) backend="${OPTARG}";; o) output="${OPTARG}";; H) hfile="${OPTARG}";; + q) quiet="-q";; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";; esac
@@ -49,11 +50,11 @@ main() { # If the output file already exists, do not download it again if [ -e "${output}" ]; then - if support/download/check-hash "${hfile}" "${output}" "${output##*/}"; then + if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then exit 0 fi rm -f "${output}" - printf "Re-downloading '%s'...\n" "${output##*/}" + warn "Re-downloading '%s'...\n" "${output##*/}" fi # tmpd is a temporary directory in which backends may store intermediate
@@ -75,7 +76,7 @@ main() { # If the backend fails, we can just remove the temporary directory to # remove all the cruft it may have left behind. Then we just exit in # error too. - if ! "${OLDPWD}/support/download/${backend}" "${tmpf}" "${@}"; then + if ! "${OLDPWD}/support/download/${backend}" ${quiet} "${tmpf}" "${@}"; then rm -rf "${tmpd}" exit 1 fi
@@ -85,7 +86,7 @@ main() { # Check if the downloaded file is sane, and matches the stored hashes # for that file - if ! support/download/check-hash "${hfile}" "${tmpf}" "${output##*/}"; then + if ! support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then rm -rf "${tmpd}" exit 1 fi
Signature · 1 line
1.9.1
PATCH v10 3/3 package/pkg-download: do not hard-code QUIET
In reply to Yann E. MORINDo not hard-code QUIET in our download commands, since it is handled in the backends.
Suggested by Fabio.
--- package/pkg-download.mk | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
package/pkg-download.mk +4 -4
@@ -8,14 +8,14 @@ ################################################################################ # Download method commands -export WGET := $(call qstrip,$(BR2_WGET)) $(QUIET) +export WGET := $(call qstrip,$(BR2_WGET)) export SVN := $(call qstrip,$(BR2_SVN)) export CVS := $(call qstrip,$(BR2_CVS)) export BZR := $(call qstrip,$(BR2_BZR)) export GIT := $(call qstrip,$(BR2_GIT)) -export HG := $(call qstrip,$(BR2_HG)) $(QUIET) -export SCP := $(call qstrip,$(BR2_SCP)) $(QUIET) -SSH := $(call qstrip,$(BR2_SSH)) $(QUIET) +export HG := $(call qstrip,$(BR2_HG)) +export SCP := $(call qstrip,$(BR2_SCP)) +SSH := $(call qstrip,$(BR2_SSH)) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) DL_WRAPPER = support/download/dl-wrapper
Signature · 1 line
1.9.1
v10 cover Improve silent builds (branch yem/fabio/silent)
In reply to Yann E. MORINDear Yann E. MORIN,
Quoted 7 lines · Yann E. MORIN
Silent builds are also nice when top-level parallel make is being used to reduce the output clutter.
Some downloaders, like git or wget, insist on reporting progress on stderr instead of stdout, so redirecting stdout is not enough, we need to pass the quiet information all the way down to the beackends, so they know they have to be silent. Sigh... :-(
Series applied, after fixing a minor typo in the first patch. Thanks to you and Fabio for working on this!
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com