support/download: reintroduce 'source-check' target
PATCH v3 1/7 support/download: reintroduce 'source-check' target
From: Thomas De Schampheleire <[email protected]>
Commit bf28a165d992681a85b43a886005e669b3cb579b removed the source-check target to allow easier refactoring of the download infrastructure. It was partly based on the fact that no-one really used this target.
However, it turns out that there _are_ good uses for it. In a work environment where many people are submitting changes to a Buildroot repository, one cannot always blindly trust every change. A typical case of human error is when bumping the version of a package but forgetting to upload the tarball to the internal BR2_PRIMARY_SITE or (in case of a repository) pushing the new changesets to the repository.
If a user cannot directly push to the Buildroot repository but needs to queue their change via an automatic validation system, that validation system can use 'make source-check' on the relevant defconfigs to protect against such errors. A full download would also be possible but takes longer and unnecessarily uses internal network bandwidth.
With that use case in mind, this commit reintroduces the 'source-check' target, but embedded in the current situation with a dl-wrapper. The changes to the wrapper are minimal (not considering additional indentation). A new option '-C' means 'check only' and will be passed to the download backends. If the backend supports the option, no download will happen. If it does not, then the backend will actually perform a download as a means of checking that the source exists (a form of graceful degradation). In neither case, though, hash checking is performed (as the standard case is without download thus without file to calculate hashes on).
Subsequent commits will actually implement -C in the backends.
--- Makefile | 7 +++ package/pkg-download.mk | 6 +++ package/pkg-generic.mk | 14 ++++- support/download/dl-wrapper | 100 ++++++++++++++++++++---------------- 4 files changed, 82 insertions(+), 45 deletions(-)
v3: (feedback Arnout Vandecappelle)
- remove tmpd at the end of dl-wrapper
- remove duplication between DOWNLOAD and SOURCE_CHECK macros
v2:
- add a trailing dot on all usage texts like for other options
- split off the reintroduction of BR2_SSH to a separate patch
- fix the source-check for scp: it seems there were some broken things in v1
- drop incomplete implementations for git/svn/bzr/cvs so it falls back to actual download.
Makefile +7 -0
@@ -131,6 +131,7 @@ noconfig_targets := menuconfig nconfig gconfig xconfig config oldconfig randconf # (default target is to build), or when MAKECMDGOALS contains # something else than one of the nobuild_targets. nobuild_targets := source %-source \ + source-check %-source-check %-all-source-check \ legal-info %-legal-info external-deps _external-deps \ clean distclean help show-targets graph-depends \ %-graph-depends %-show-depends %-show-version \
@@ -818,6 +819,9 @@ target-post-image: $(TARGETS_ROOTFS) target-finalize staging-finalize .PHONY: source source: $(foreach p,$(PACKAGES),$(p)-all-source) +.PHONY: source-check +source-check: $(foreach p,$(PACKAGES),$(p)-all-source-check) + .PHONY: _external-deps external-deps _external-deps: $(foreach p,$(PACKAGES),$(p)-all-external-deps) external-deps:
@@ -1103,6 +1107,8 @@ help: @echo ' <pkg>-dirclean - Remove <pkg> build directory' @echo ' <pkg>-reconfigure - Restart the build from the configure step' @echo ' <pkg>-rebuild - Restart the build from the build step' + @echo ' <pkg>-source-check - Check package for valid download URLs' + @echo ' <pkg>-all-source-check - Check package and its dependencies for valid download URLs' $(foreach p,$(HELP_PACKAGES), \ @echo $(sep) \ @echo '$($(p)_NAME):' $(sep) \
@@ -1122,6 +1128,7 @@ help: @echo @echo 'Miscellaneous:' @echo ' source - download all sources needed for offline-build' + @echo ' source-check - check selected packages for valid download URLs' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' @echo ' printvars - dump all the internal variables'
package/pkg-download.mk +6 -0
@@ -70,6 +70,7 @@ export BR_NO_CHECK_HASH_FOR = # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set # # Argument 1 is the source location +# Argument 2 (optionally) provides extra arguments to pass to DL_WRAPPER # ################################################################################
@@ -92,6 +93,7 @@ endif define DOWNLOAD $(Q)mkdir -p $($(PKG)_DL_DIR) $(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \ + $(2) \ -c '$($(PKG)_DL_VERSION)' \ -d '$($(PKG)_DL_DIR)' \ -D '$(DL_DIR)' \
@@ -106,3 +108,7 @@ define DOWNLOAD -- \ $($(PKG)_DL_OPTS) endef + +define SOURCE_CHECK + $(call DOWNLOAD,$(1),-C) +endef
package/pkg-generic.mk +13 -1
@@ -788,6 +788,10 @@ $(1)-legal-source: $$($(2)_TARGET_ACTUAL_SOURCE) endif # actual sources != sources endif # actual sources != "" +$(1)-source-check: PKG=$(2) +$(1)-source-check: + $$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep)) + $(1)-external-deps: @for p in $$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS) ; do \ echo `basename $$$$p` ; \
@@ -812,6 +816,9 @@ $(1)-rsync: $$($(2)_TARGET_RSYNC) $(1)-source: $(1)-legal-source: +$(1)-source-check: + test -d $$($(2)_OVERRIDE_SRCDIR) + $(1)-external-deps: @echo "file://$$($(2)_OVERRIDE_SRCDIR)" endif
@@ -846,6 +853,9 @@ $(1)-graph-rdepends: graph-depends-requirements $(1)-all-source: $(1)-source $(1)-all-source: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source) +$(1)-all-source-check: $(1)-source-check +$(1)-all-source-check: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source-check) + $(1)-all-external-deps: $(1)-external-deps $(1)-all-external-deps: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-external-deps)
@@ -1044,6 +1054,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-all-external-deps \ $(1)-all-legal-info \ $(1)-all-source \ + $(1)-all-source-check \ $(1)-build \ $(1)-clean-for-rebuild \ $(1)-clean-for-reconfigure \
@@ -1068,7 +1079,8 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-rsync \ $(1)-show-depends \ $(1)-show-version \ - $(1)-source + $(1)-source \ + $(1)-source-check ifneq ($$($(2)_SOURCE),) ifeq ($$($(2)_SITE),)
support/download/dl-wrapper +56 -44
142 lines not on this page · raw message
Signature · 1 line
2.19.2
PATCH v3 2/7 support/download/hg: implement source-check
From: Thomas De Schampheleire <[email protected]>
Note: the implementation is different (better) than what used to be in Buildroot before source-check was removed.
The original implementation:
hg incoming --force -l1 <URL>
would only verify that the repository exists, not that the requested revision is present.
An already better implementation is:
hg incoming --force -l1 -r <revision> <URL>
but compared to the next solution it has a large resource consumption on the local machine. In the background, the full repository is first downloaded.
The implemented solution is:
hg identify -r <revision> <URL>
which operates directly on the remote repository.
--- support/download/hg | 7 +++++++ 1 file changed, 7 insertions(+)
v3: no changes
support/download/hg +7 -0
@@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the changeset exists in the remote repository. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET.
@@ -19,6 +20,7 @@ verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;; o) output="${OPTARG}";; u) uri="${OPTARG}";; c) cset="${OPTARG}";;
@@ -36,6 +38,11 @@ _hg() { eval ${HG} "${@}" } +if [ -n "${checkonly}" ]; then + _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null + exit ${?} +fi + _hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'" _hg archive ${verbose} --repository "'${basename}'" --type tgz \
Signature · 1 line
2.19.2
PATCH v3 2/7 support/download/hg: implement source-check
Thomas, All,
On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:
Quoted 39 lines
From: Thomas De Schampheleire <[email protected]>
Note: the implementation is different (better) than what used to be in Buildroot before source-check was removed.
The original implementation:
hg incoming --force -l1 <URL>would only verify that the repository exists, not that the requested revision is present.
An already better implementation is:
hg incoming --force -l1 -r <revision> <URL>but compared to the next solution it has a large resource consumption on the local machine. In the background, the full repository is first downloaded.
The implemented solution is:
hg identify -r <revision> <URL>
which operates directly on the remote repository.Signed-off-by Thomas De Schampheleire <[email protected]>--- support/download/hg | 7 +++++++ 1 file changed, 7 insertions(+)v3: no changes
support/download/hg +1
-0@@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the changeset exists in the remote repository. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET.
support/download/hg verbose= · line 23
Quoted 4 lines
support/download/hg +1
-0@@ -19,6 +20,7 @@ verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;;
I've come to like an alternate solution to handle boolean options:
checkonly=false
while getopts ...; do
case "${OPT}" in
C) checkonly=true;;
esac
doneWhich allows to write nicer conditions:
if ${checkonly}; then
...
fisupport/download/hg _hg() { · line 43
Quoted 10 lines
support/download/hg +3
-0o) output="${OPTARG}";; u) uri="${OPTARG}";; c) cset="${OPTARG}";;@@ -36,6 +38,11 @@ _hg() { eval ${HG} "${@}" } +if [ -n "${checkonly}" ]; then + _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null + exit ${?}
Incorrect use of exit. The script has a 'set -e' at the beginning, so a command that exits in error will cause the script to abort. So, if the script reaches the exit clause, it means the call to _hg did not fail. Ditto the other backends, of course.
Also, I think it always makes sense to check that the revision exists, even if doing the actual download: it allows for a fastpath even in case the user wants to do the actual download anyway:
if ! _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null; then
printf 'error blabla no such revision blabla\n'
exit 1
fi
if ${checkonly}; then exit 0; fiNote that we did use to have a similar test with git, but it was nt reliable (i.e. old git servers and/or configuration of git server that did not behave when ls-remote was fed some forms of refs, like a sha1 or susch)
Regards,
Yann E. MORIN.
Quoted 8 lines
support/download/hg +2
-0+fi + _hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'" _hg archive ${verbose} --repository "'${basename}'" --type tgz \Signature · 1 line
2.19.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 2/7 support/download/hg: implement source-check
El s?b., 9 feb. 2019 a las 22:53, Yann E. MORIN
(<[email protected]>) escribi?:
Quoted 57 lines
Thomas, All,
On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:
Quoted 39 lines
From: Thomas De Schampheleire <[email protected]>
Note: the implementation is different (better) than what used to be in Buildroot before source-check was removed.
The original implementation:
hg incoming --force -l1 <URL>would only verify that the repository exists, not that the requested revision is present.
An already better implementation is:
hg incoming --force -l1 -r <revision> <URL>but compared to the next solution it has a large resource consumption on the local machine. In the background, the full repository is first downloaded.
The implemented solution is:
hg identify -r <revision> <URL>
which operates directly on the remote repository.Signed-off-by Thomas De Schampheleire <[email protected]>--- support/download/hg | 7 +++++++ 1 file changed, 7 insertions(+)v3: no changes
support/download/hg +1
-0@@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the changeset exists in the remote repository. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET.support/download/hg verbose= · line 23
Quoted 4 lines
support/download/hg +1
-0@@ -19,6 +20,7 @@ verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;;I've come to like an alternate solution to handle boolean options:
checkonly=false while getopts ...; do case "${OPT}" in C) checkonly=true;; esac done
Quoted 5 lines
Which allows to write nicer conditions:
if ${checkonly}; then ... fi
Ok, will use this
Quoted 8 lines
Quoted 8 lines
support/download/hg +0
-0o) output="${OPTARG}";; u) uri="${OPTARG}";; c) cset="${OPTARG}";;@@ -36,6 +38,11 @@ _hg() { eval ${HG} "${@}" }
Quoted 8 lines
support/download/hg _hg() { · line 43
Quoted 3 lines
support/download/hg +3
-0+if [ -n "${checkonly}" ]; then + _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null + exit ${?}Incorrect use of exit. The script has a 'set -e' at the beginning, so a command that exits in error will cause the script to abort. So, if the script reaches the exit clause, it means the call to
_hgdid not fail. Ditto the other backends, of course.
Ok, I will fix it.
Note that there is the danger that someone removes the 'set -e' in the future, which would break the plain 'exit'. Something to be careful about...
Quoted 11 lines
Also, I think it always makes sense to check that the revision exists, even if doing the actual download: it allows for a fastpath even in case the user wants to do the actual download anyway:
if ! _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null; then printf 'error blabla no such revision blabla\n' exit 1 fi if ${checkonly}; then exit 0; fi
Ok, will do.
Thanks,
Thomas
PATCH v3 3/7 support/download/wget: implement source-check
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
--- support/download/wget | 7 +++++++ 1 file changed, 7 insertions(+)
v3: no changes
support/download/wget +7 -0
@@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the file exists remotely. # -o FILE Save into file FILE. # -f FILENAME The filename of the tarball to get at URL # -u URL Download file at URL.
@@ -19,6 +20,7 @@ verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;; o) output="${OPTARG}";; f) filename="${OPTARG}";; u) url="${OPTARG}";;
@@ -40,4 +42,9 @@ _wget() { # mirror [ -n "${encode}" ] && filename=${filename//\?/%3F} +if [ -n "${checkonly}" ]; then + _wget --spider ${verbose} "${@}" "'${url}/${filename}'" + exit ${?} +fi + _wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
Signature · 1 line
2.19.2
PATCH v3 4/7 support/download/file: implement source-check
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
--- support/download/file | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
v3: no changes
support/download/file +11 -1
@@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the source file exists. # -o FILE Copy to file FILE. # -f FILE Copy from basename file FILE. # -u DIR Copy from FILE in DIR.
@@ -23,6 +24,7 @@ verbose=-v while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=;; + C) checkonly=1;; o) output="${OPTARG}";; f) file="${OPTARG}";; u) dir="${OPTARG}";;
@@ -39,4 +41,12 @@ _localfiles() { eval ${LOCALFILES} "${@}" } -_localfiles ${verbose} "'${dir##file://}/${file}'" "'${output}'" +# Remove any scheme prefix +dir="${dir##file://}" + +if [ -n "${checkonly}" ]; then + test -e "'${dir}/${file}'" + exit ${?} +fi + +_localfiles ${verbose} "'${dir}/${file}'" "'${output}'"
Signature · 1 line
2.19.2
PATCH v3 5/7 Config.in: reintroduce BR2_SSH
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
The BR2_SSH command was removed in commit db9473bf6cd7bd12aa1f9faad0a917c973c33827 ("core/download: drop the SSH command") but will be needed again to support 'source-check' for the scp download backend.
--- Config.in | 4 ++++ package/pkg-download.mk | 1 + 2 files changed, 5 insertions(+)
v3: no changes
Config.in +4 -0
@@ -136,6 +136,10 @@ config BR2_SCP string "Secure copy (scp) command" default "scp" +config BR2_SSH + string "Secure shell (ssh) command" + default "ssh" + config BR2_HG string "Mercurial (hg) command" default "hg"
package/pkg-download.mk +1 -0
@@ -15,6 +15,7 @@ export BZR := $(call qstrip,$(BR2_BZR)) export GIT := $(call qstrip,$(BR2_GIT)) export HG := $(call qstrip,$(BR2_HG)) export SCP := $(call qstrip,$(BR2_SCP)) +export SSH := $(call qstrip,$(BR2_SSH)) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) DL_WRAPPER = support/download/dl-wrapper
Signature · 1 line
2.19.2
PATCH v3 6/7 support/download/scp: implement source-check
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
--- support/download/scp | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
v3: no changes
support/download/scp +14 -0
@@ -7,17 +7,20 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the file exists remotely. # -o FILE Copy to local file FILE. # -f FILE Copy from remote file FILE. # -u URI Download file at URI. # # Environment: # SCP : the scp command to call +# SSH : the ssh command to use for checkonly verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;; o) output="${OPTARG}";; f) filename="${OPTARG}";; u) uri="${OPTARG}";;
@@ -33,8 +36,19 @@ shift $((OPTIND-1)) # Get rid of our options _scp() { eval ${SCP} "${@}" } +_ssh() { + eval ${SSH} "${@}" +} # Remove any scheme prefix uri="${uri##scp://}" +if [ -n "${checkonly}" ]; then + # uri now looks like: foo.example.org:some/directory + domain="${uri%%:*}" + path="${uri#*:}/${filename}" + _ssh ${verbose} "${@}" "'${domain}'" ls "'${path}'" > /dev/null + exit ${?} +fi + _scp ${verbose} "${@}" "'${uri}/${filename}'" "'${output}'"
Signature · 1 line
2.19.2
PATCH v3 6/7 support/download/scp: implement source-check
Thomas, All,
On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:
Quoted 40 lines
From: Thomas De Schampheleire <[email protected]>
Signed-off-by Thomas De Schampheleire <[email protected]>--- support/download/scp | 14 ++++++++++++++ 1 file changed, 14 insertions(+)v3: no changes
support/download/scp +5
-0@@ -7,17 +7,20 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the file exists remotely. # -o FILE Copy to local file FILE. # -f FILE Copy from remote file FILE. # -u URI Download file at URI. # # Environment: # SCP : the scp command to call +# SSH : the ssh command to use for checkonly verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;; o) output="${OPTARG}";; f) filename="${OPTARG}";; u) uri="${OPTARG}";;@@ -33,8 +36,19 @@ shift $((OPTIND-1)) # Get rid of our options _scp() { eval ${SCP} "${@}" } +_ssh() { + eval ${SSH} "${@}"
support/download/scp shift $((OPTIND-1)) # Get rid of our options · line 50
Quoted 10 lines
support/download/scp +6
-0+} # Remove any scheme prefix uri="${uri##scp://}" +if [ -n "${checkonly}" ]; then + # uri now looks like: foo.example.org:some/directory + domain="${uri%%:*}" + path="${uri#*:}/${filename}" + _ssh ${verbose} "${@}" "'${domain}'" ls "'${path}'" > /dev/null
I was going to reply to the previous thread, but you were too fast to respin, so here's my proposal to avoid ls:
/usr/bin/env [ -f "'${path}'" ]It is almost impossible to have a system that lacks 'env' or that have it in another location, as POSIX mandates env to exist, and IIRC, it even mandates it to be /usr/bin/env (of is it FHS? at least, scripts in Buildroot use "#!/usr/bin/env bash").
You may have to carefully quote the [ and ], to avoid the eval from doing a nasty, weel, evaluation...
Regards,
Yann E. MORIN.
Quoted 7 lines
support/download/scp +3
-0+ exit ${?} +fi + _scp ${verbose} "${@}" "'${uri}/${filename}'" "'${output}'"Signature · 1 line
2.19.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 6/7 support/download/scp: implement source-check
El s?b., 9 feb. 2019 a las 23:09, Yann E. MORIN
(<[email protected]>) escribi?:
Quoted 60 lines
Thomas, All,
On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:
Quoted 40 lines
From: Thomas De Schampheleire <[email protected]>
Signed-off-by Thomas De Schampheleire <[email protected]>--- support/download/scp | 14 ++++++++++++++ 1 file changed, 14 insertions(+)v3: no changes
support/download/scp +5
-0@@ -7,17 +7,20 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the file exists remotely. # -o FILE Copy to local file FILE. # -f FILE Copy from remote file FILE. # -u URI Download file at URI. # # Environment: # SCP : the scp command to call +# SSH : the ssh command to use for checkonly verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;; o) output="${OPTARG}";; f) filename="${OPTARG}";; u) uri="${OPTARG}";;@@ -33,8 +36,19 @@ shift $((OPTIND-1)) # Get rid of our options _scp() { eval ${SCP} "${@}" } +_ssh() { + eval ${SSH} "${@}"support/download/scp shift $((OPTIND-1)) # Get rid of our options · line 50
Quoted 10 lines
support/download/scp +6
-0+} # Remove any scheme prefix uri="${uri##scp://}" +if [ -n "${checkonly}" ]; then + # uri now looks like: foo.example.org:some/directory + domain="${uri%%:*}" + path="${uri#*:}/${filename}" + _ssh ${verbose} "${@}" "'${domain}'" ls "'${path}'" > /dev/nullI was going to reply to the previous thread, but you were too fast to respin, so here's my proposal to avoid ls:
/usr/bin/env [ -f "'${path}'" ]
Quoted 7 lines
It is almost impossible to have a system that lacks 'env' or that have it in another location, as POSIX mandates env to exist, and IIRC, it even mandates it to be
/usr/bin/env(of is it FHS? at least, scripts in Buildroot use "#!/usr/bin/env bash").You may have to carefully quote the [ and ], to avoid the eval from doing a nasty, weel, evaluation...
I would use 'test' rather than [ ] to avoid any such problem.
Thanks for the suggestion, I will test it.
/Thomas
PATCH v3 6/7 support/download/scp: implement source-check
El vie., 15 feb. 2019 a las 20:15, Thomas De Schampheleire (<[email protected]>) escribi?:
Quoted 68 lines
El s?b., 9 feb. 2019 a las 23:09, Yann E. MORIN
(<[email protected]>) escribi?:Quoted 65 lines
Thomas, All,
On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:
Quoted 40 lines
From: Thomas De Schampheleire <[email protected]>
Signed-off-by Thomas De Schampheleire <[email protected]>--- support/download/scp | 14 ++++++++++++++ 1 file changed, 14 insertions(+)v3: no changes
support/download/scp +5
-0@@ -7,17 +7,20 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the file exists remotely. # -o FILE Copy to local file FILE. # -f FILE Copy from remote file FILE. # -u URI Download file at URI. # # Environment: # SCP : the scp command to call +# SSH : the ssh command to use for checkonly verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;; o) output="${OPTARG}";; f) filename="${OPTARG}";; u) uri="${OPTARG}";;@@ -33,8 +36,19 @@ shift $((OPTIND-1)) # Get rid of our options _scp() { eval ${SCP} "${@}" } +_ssh() { + eval ${SSH} "${@}"support/download/scp shift $((OPTIND-1)) # Get rid of our options · line 50
Quoted 10 lines
support/download/scp +6
-0+} # Remove any scheme prefix uri="${uri##scp://}" +if [ -n "${checkonly}" ]; then + # uri now looks like: foo.example.org:some/directory + domain="${uri%%:*}" + path="${uri#*:}/${filename}" + _ssh ${verbose} "${@}" "'${domain}'" ls "'${path}'" > /dev/nullI was going to reply to the previous thread, but you were too fast to respin, so here's my proposal to avoid ls:
/usr/bin/env [ -f "'${path}'" ]It is almost impossible to have a system that lacks 'env' or that have it in another location, as POSIX mandates env to exist, and IIRC, it even mandates it to be
/usr/bin/env(of is it FHS? at least, scripts in Buildroot use "#!/usr/bin/env bash").
Quoted 7 lines
Quoted 2 lines
You may have to carefully quote the [ and ], to avoid the eval from doing a nasty, weel, evaluation...
I would use 'test' rather than [ ] to avoid any such problem.
Thanks for the suggestion, I will test it.
Actually, I don't think the 'env' is needed, a simple 'test -f' will do, my reasoning below.
Arnout is right that 'ls' is not guaranteed to exist, it is not a mandatory POSIX utility but an 'XSI' extension.
http://pubs.opengroup.org/onlinepubs/009695399/utilities/ls.html
On the contrary, 'test' is a mandatory POSIX utility (i.e. there are no specific codes like XSI mentioned on following page)
http://pubs.opengroup.org/onlinepubs/000095399/utilities/test.html
So 'test' will certainly exist. We are not sure if it is in /bin/ or /usr/bin/ or another place, though.
By calling it without absolute path, we assume that it is in the standard PATH.
One could argue that '/usr/bin/env test' is better, but if 'test' is _not_ in the standard PATH, then 'env' will also not find it. So, I don't see how 'env' brings an advantage.
/Thomas
PATCH v3 7/7 support/download/{bzr, cvs, git, svn}: highlight unimplemented source-check
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
For bzr, cvs, git and svn, there is no method of checking whether the revision exists remotely, without actually cloning the repository. Therefore, there is no actual implementation for '-C' (checkonly) for these download helpers. The script will fall back to the normal download logic, which would effectively fail if the revision does not exist.
For completeness, mention this unimplemented feature in the usage text.
--- support/download/bzr | 2 ++ support/download/cvs | 2 ++ support/download/git | 2 ++ support/download/svn | 2 ++ 4 files changed, 8 insertions(+)
v3: no changes
support/download/bzr +2 -0
@@ -7,6 +7,8 @@ set -e # # Options: # -q Be quiet +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET.
support/download/cvs +2 -0
@@ -7,6 +7,8 @@ set -e # # Options: # -q Be quiet +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -o FILE Generate archive in FILE. # -u URI Checkout from repository at URI. # -c REV Use revision REV.
support/download/git +2 -0
@@ -7,6 +7,8 @@ set -E # # Options: # -q Be quiet. +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -r Clone and archive sub-modules. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI.
support/download/svn +2 -0
@@ -7,6 +7,8 @@ set -e # # Options: # -q Be quiet. +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -o FILE Generate archive in FILE. # -u URI Checkout from repository at URI. # -c REV Use revision REV.
Signature · 1 line
2.19.2
PATCH v3 7/7 support/download/{bzr, cvs, git, svn}: highlight unimplemented source-check
Thomas, All,
On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:
Quoted 3 lines
From: Thomas De Schampheleire <[email protected]>
For bzr, cvs, git and svn, there is no method of checking whether the
Acutally, for svn this is realtively trivial:
svn ls -r ${rev} ${remote}We have two packages in Buildroot that use svn, fis and open2300, so you cn use them to test:
$ svn ls -r 2892 http://svn.chezphil.org/utils/trunk Redirecting to URL 'https://svn.chezphil.org/utils/trunk': Makefile apexctl.cc [...]
Regards,
Yann E. MORIN.
Quoted 72 lines
revision exists remotely, without actually cloning the repository. Therefore, there is no actual implementation for '-C' (checkonly) for these download helpers. The script will fall back to the normal download logic, which would effectively fail if the revision does not exist.
For completeness, mention this unimplemented feature in the usage text.
Signed-off-by Thomas De Schampheleire <[email protected]>--- support/download/bzr | 2 ++ support/download/cvs | 2 ++ support/download/git | 2 ++ support/download/svn | 2 ++ 4 files changed, 8 insertions(+)v3: no changes
support/download/bzr +2
-0@@ -7,6 +7,8 @@ set -e # # Options: # -q Be quiet +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET.support/download/cvs +2
-0@@ -7,6 +7,8 @@ set -e # # Options: # -q Be quiet +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -o FILE Generate archive in FILE. # -u URI Checkout from repository at URI. # -c REV Use revision REV.support/download/git +2
-0@@ -7,6 +7,8 @@ set -E # # Options: # -q Be quiet. +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -r Clone and archive sub-modules. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI.support/download/svn +2
-0@@ -7,6 +7,8 @@ set -e # # Options: # -q Be quiet. +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -o FILE Generate archive in FILE. # -u URI Checkout from repository at URI. # -c REV Use revision REV.Signature · 1 line
2.19.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 7/7 support/download/{bzr, cvs, git, svn}: highlight unimplemented source-check
El s?b., 9 feb. 2019 a las 23:13, Yann E. MORIN
(<[email protected]>) escribi?:
Quoted 12 lines
Thomas, All,
On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:
Quoted 3 lines
From: Thomas De Schampheleire <[email protected]>
For bzr, cvs, git and svn, there is no method of checking whether the
Acutally, for svn this is realtively trivial:
svn ls -r ${rev} ${remote}
Quoted 9 lines
We have two packages in Buildroot that use svn, fis and open2300, so you cn use them to test:
$ svn ls -r 2892 http://svn.chezphil.org/utils/trunk Redirecting to URL 'https://svn.chezphil.org/utils/trunk': Makefile apexctl.cc [...]
Thanks a lot, will double-check at my end and adapt.
/Thomas