Skip to contentSkip to conversation

support/download: reintroduce 'source-check' target

PATCH Superseded Thomas De Schampheleire 2 participants 14 messages Feb 9, 2019 – Feb 15, 2019 Kbuild

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/7 support/download: reintroduce 'source-check' target

From: Thomas De Schampheleire <[email protected]>

Commit bf28a165d992681a85b43a886005e669b3cb579b removed the source-check target to allow easier refactoring of the download infrastructure. It was partly based on the fact that no-one really used this target.

However, it turns out that there _are_ good uses for it. In a work environment where many people are submitting changes to a Buildroot repository, one cannot always blindly trust every change. A typical case of human error is when bumping the version of a package but forgetting to upload the tarball to the internal BR2_PRIMARY_SITE or (in case of a repository) pushing the new changesets to the repository.
If a user cannot directly push to the Buildroot repository but needs to queue their change via an automatic validation system, that validation system can use 'make source-check' on the relevant defconfigs to protect against such errors. A full download would also be possible but takes longer and unnecessarily uses internal network bandwidth.

With that use case in mind, this commit reintroduces the 'source-check' target, but embedded in the current situation with a dl-wrapper. The changes to the wrapper are minimal (not considering additional indentation). A new option '-C' means 'check only' and will be passed to the download backends. If the backend supports the option, no download will happen. If it does not, then the backend will actually perform a download as a means of checking that the source exists (a form of graceful degradation). In neither case, though, hash checking is performed (as the standard case is without download thus without file to calculate hashes on).

Subsequent commits will actually implement -C in the backends.

---
 Makefile                    |   7 +++
 package/pkg-download.mk     |   6 +++
 package/pkg-generic.mk      |  14 ++++-
 support/download/dl-wrapper | 100 ++++++++++++++++++++----------------
 4 files changed, 82 insertions(+), 45 deletions(-)

v3: (feedback Arnout Vandecappelle)
- remove tmpd at the end of dl-wrapper
- remove duplication between DOWNLOAD and SOURCE_CHECK macros

v2:
- add a trailing dot on all usage texts like for other options
- split off the reintroduction of BR2_SSH to a separate patch
- fix the source-check for scp: it seems there were some broken things in v1
- drop incomplete implementations for git/svn/bzr/cvs so it falls back to actual download.

Makefile +7 -0
index 0d2659c46e..9ae047bfc6 100644
@@ -131,6 +131,7 @@ noconfig_targets := menuconfig nconfig gconfig xconfig config oldconfig randconf
 # (default target is to build), or when MAKECMDGOALS contains
 # something else than one of the nobuild_targets.
 nobuild_targets := source %-source \
+	source-check %-source-check %-all-source-check \
 	legal-info %-legal-info external-deps _external-deps \
 	clean distclean help show-targets graph-depends \
 	%-graph-depends %-show-depends %-show-version \
@@ -818,6 +819,9 @@ target-post-image: $(TARGETS_ROOTFS) target-finalize staging-finalize
 .PHONY: source
 source: $(foreach p,$(PACKAGES),$(p)-all-source)
 
+.PHONY: source-check
+source-check: $(foreach p,$(PACKAGES),$(p)-all-source-check)
+
 .PHONY: _external-deps external-deps
 _external-deps: $(foreach p,$(PACKAGES),$(p)-all-external-deps)
 external-deps:
@@ -1103,6 +1107,8 @@ help:
 	@echo '  <pkg>-dirclean         - Remove <pkg> build directory'
 	@echo '  <pkg>-reconfigure      - Restart the build from the configure step'
 	@echo '  <pkg>-rebuild          - Restart the build from the build step'
+	@echo '  <pkg>-source-check     - Check package for valid download URLs'
+	@echo '  <pkg>-all-source-check - Check package and its dependencies for valid download URLs'
 	$(foreach p,$(HELP_PACKAGES), \
 		@echo $(sep) \
 		@echo '$($(p)_NAME):' $(sep) \
@@ -1122,6 +1128,7 @@ help:
 	@echo
 	@echo 'Miscellaneous:'
 	@echo '  source                 - download all sources needed for offline-build'
+	@echo '  source-check           - check selected packages for valid download URLs'
 	@echo '  external-deps          - list external packages used'
 	@echo '  legal-info             - generate info about license compliance'
 	@echo '  printvars              - dump all the internal variables'
package/pkg-download.mk +6 -0
index 7cd87c38ff..5384fc2af4 100644
@@ -70,6 +70,7 @@ export BR_NO_CHECK_HASH_FOR =
 # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
 #
 # Argument 1 is the source location
+# Argument 2 (optionally) provides extra arguments to pass to DL_WRAPPER
 #
 ################################################################################
 
@@ -92,6 +93,7 @@ endif
 define DOWNLOAD
 	$(Q)mkdir -p $($(PKG)_DL_DIR)
 	$(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
+		$(2) \
 		-c '$($(PKG)_DL_VERSION)' \
 		-d '$($(PKG)_DL_DIR)' \
 		-D '$(DL_DIR)' \
@@ -106,3 +108,7 @@ define DOWNLOAD
 		-- \
 		$($(PKG)_DL_OPTS)
 endef
+
+define SOURCE_CHECK
+	$(call DOWNLOAD,$(1),-C)
+endef
package/pkg-generic.mk +13 -1
index 6168b40e89..bc57956867 100644
@@ -788,6 +788,10 @@ $(1)-legal-source:	$$($(2)_TARGET_ACTUAL_SOURCE)
 endif # actual sources != sources
 endif # actual sources != ""
 
+$(1)-source-check: PKG=$(2)
+$(1)-source-check:
+	$$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep))
+
 $(1)-external-deps:
 	@for p in $$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS) ; do \
 		echo `basename $$$$p` ; \
@@ -812,6 +816,9 @@ $(1)-rsync:		$$($(2)_TARGET_RSYNC)
 $(1)-source:
 $(1)-legal-source:
 
+$(1)-source-check:
+	test -d $$($(2)_OVERRIDE_SRCDIR)
+
 $(1)-external-deps:
 	@echo "file://$$($(2)_OVERRIDE_SRCDIR)"
 endif
@@ -846,6 +853,9 @@ $(1)-graph-rdepends: graph-depends-requirements
 $(1)-all-source:	$(1)-source
 $(1)-all-source:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source)
 
+$(1)-all-source-check:	$(1)-source-check
+$(1)-all-source-check:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source-check)
+
 $(1)-all-external-deps:	$(1)-external-deps
 $(1)-all-external-deps:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-external-deps)
 
@@ -1044,6 +1054,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
 	$(1)-all-external-deps \
 	$(1)-all-legal-info \
 	$(1)-all-source \
+	$(1)-all-source-check \
 	$(1)-build \
 	$(1)-clean-for-rebuild \
 	$(1)-clean-for-reconfigure \
@@ -1068,7 +1079,8 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
 	$(1)-rsync \
 	$(1)-show-depends \
 	$(1)-show-version \
-	$(1)-source
+	$(1)-source \
+	$(1)-source-check
 
 ifneq ($$($(2)_SOURCE),)
 ifeq ($$($(2)_SITE),)
support/download/dl-wrapper +56 -44

142 lines not on this page · raw message

Signature · 1 line
2.19.2
#1 lore raw

PATCH v3 2/7 support/download/hg: implement source-check

From: Thomas De Schampheleire <[email protected]>

Note: the implementation is different (better) than what used to be in Buildroot before source-check was removed.

The original implementation:

    hg incoming --force -l1 <URL>

would only verify that the repository exists, not that the requested revision is present.

An already better implementation is:

    hg incoming --force -l1 -r <revision> <URL>

but compared to the next solution it has a large resource consumption on the local machine. In the background, the full repository is first downloaded.

The implemented solution is:
hg identify -r <revision> <URL>
which operates directly on the remote repository.

---
 support/download/hg | 7 +++++++
 1 file changed, 7 insertions(+)

v3: no changes

support/download/hg +7 -0
index efb515fca5..ed8dcfbcff 100755
@@ -7,6 +7,7 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the changeset exists in the remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Clone from repository at URI.
 #   -c CSET     Use changeset (or revision) CSET.
@@ -19,6 +20,7 @@ verbose=
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=1;;
     o)  output="${OPTARG}";;
     u)  uri="${OPTARG}";;
     c)  cset="${OPTARG}";;
@@ -36,6 +38,11 @@ _hg() {
     eval ${HG} "${@}"
 }
 
+if [ -n "${checkonly}" ]; then
+    _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null
+    exit ${?}
+fi
+
 _hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'"
 
 _hg archive ${verbose} --repository "'${basename}'" --type tgz \
Signature · 1 line
2.19.2
#2 lore raw

PATCH v3 2/7 support/download/hg: implement source-check

Thomas, All,

On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:

Quoted 39 lines

From: Thomas De Schampheleire <[email protected]>

Note: the implementation is different (better) than what used to be in Buildroot before source-check was removed.

The original implementation:

    hg incoming --force -l1 <URL>

would only verify that the repository exists, not that the requested revision is present.

An already better implementation is:

    hg incoming --force -l1 -r <revision> <URL>

but compared to the next solution it has a large resource consumption on the local machine. In the background, the full repository is first downloaded.

The implemented solution is:
hg identify -r <revision> <URL>
which operates directly on the remote repository.

---
 support/download/hg | 7 +++++++
 1 file changed, 7 insertions(+)

v3: no changes

support/download/hg +1 -0
index efb515fca5..ed8dcfbcff 100755
@@ -7,6 +7,7 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the changeset exists in the remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Clone from repository at URI.
 #   -c CSET     Use changeset (or revision) CSET.

support/download/hg verbose= · line 23

Quoted 4 lines
support/download/hg +1 -0
@@ -19,6 +20,7 @@ verbose=
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=1;;

I've come to like an alternate solution to handle boolean options:

    checkonly=false
    while getopts ...; do
        case "${OPT}" in
            C)  checkonly=true;;
        esac
    done

Which allows to write nicer conditions:

    if ${checkonly}; then
        ...
    fi

support/download/hg _hg() { · line 43

Quoted 10 lines
support/download/hg +3 -0
     o)  output="${OPTARG}";;
     u)  uri="${OPTARG}";;
     c)  cset="${OPTARG}";;
@@ -36,6 +38,11 @@ _hg() {
     eval ${HG} "${@}"
 }
 
+if [ -n "${checkonly}" ]; then
+    _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null
+    exit ${?}

Incorrect use of exit. The script has a 'set -e' at the beginning, so a command that exits in error will cause the script to abort. So, if the script reaches the exit clause, it means the call to _hg did not fail. Ditto the other backends, of course.

Also, I think it always makes sense to check that the revision exists, even if doing the actual download: it allows for a fastpath even in case the user wants to do the actual download anyway:

    if ! _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null; then
        printf 'error blabla no such revision blabla\n'
        exit 1
    fi
    if ${checkonly}; then exit 0; fi

Note that we did use to have a similar test with git, but it was nt reliable (i.e. old git servers and/or configuration of git server that did not behave when ls-remote was fed some forms of refs, like a sha1 or susch)

Regards,
Yann E. MORIN.

Quoted 8 lines
support/download/hg +2 -0
+fi
+
 _hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'"
 
 _hg archive ${verbose} --repository "'${basename}'" --type tgz \
Signature · 1 line
2.19.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#3 lore raw

PATCH v3 2/7 support/download/hg: implement source-check

El s?b., 9 feb. 2019 a las 22:53, Yann E. MORIN
(<[email protected]>) escribi?:

Quoted 57 lines

Thomas, All,

On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:

Quoted 39 lines

From: Thomas De Schampheleire <[email protected]>

Note: the implementation is different (better) than what used to be in Buildroot before source-check was removed.

The original implementation:

    hg incoming --force -l1 <URL>

would only verify that the repository exists, not that the requested revision is present.

An already better implementation is:

    hg incoming --force -l1 -r <revision> <URL>

but compared to the next solution it has a large resource consumption on the local machine. In the background, the full repository is first downloaded.

The implemented solution is:
hg identify -r <revision> <URL>
which operates directly on the remote repository.

---
 support/download/hg | 7 +++++++
 1 file changed, 7 insertions(+)

v3: no changes

support/download/hg +1 -0
index efb515fca5..ed8dcfbcff 100755
@@ -7,6 +7,7 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the changeset exists in the remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Clone from repository at URI.
 #   -c CSET     Use changeset (or revision) CSET.

support/download/hg verbose= · line 23

Quoted 4 lines
support/download/hg +1 -0
@@ -19,6 +20,7 @@ verbose=
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=1;;

I've come to like an alternate solution to handle boolean options:

    checkonly=false
    while getopts ...; do
        case "${OPT}" in
            C)  checkonly=true;;
        esac
    done
Quoted 5 lines

Which allows to write nicer conditions:

    if ${checkonly}; then
        ...
    fi

Ok, will use this

Quoted 8 lines
Quoted 8 lines
support/download/hg +0 -0
     o)  output="${OPTARG}";;
     u)  uri="${OPTARG}";;
     c)  cset="${OPTARG}";;
@@ -36,6 +38,11 @@ _hg() {
     eval ${HG} "${@}"
 }
 
Quoted 8 lines

support/download/hg _hg() { · line 43

Quoted 3 lines
support/download/hg +3 -0
+if [ -n "${checkonly}" ]; then
+    _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null
+    exit ${?}

Incorrect use of exit. The script has a 'set -e' at the beginning, so a command that exits in error will cause the script to abort. So, if the script reaches the exit clause, it means the call to _hg did not fail. Ditto the other backends, of course.

Ok, I will fix it.

Note that there is the danger that someone removes the 'set -e' in the future, which would break the plain 'exit'. Something to be careful about...

Quoted 11 lines

Also, I think it always makes sense to check that the revision exists, even if doing the actual download: it allows for a fastpath even in case the user wants to do the actual download anyway:

    if ! _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null; then
        printf 'error blabla no such revision blabla\n'
        exit 1
    fi
    if ${checkonly}; then exit 0; fi

Ok, will do.

Thanks,
Thomas

#4 lore raw

PATCH v3 3/7 support/download/wget: implement source-check

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

---
 support/download/wget | 7 +++++++
 1 file changed, 7 insertions(+)

v3: no changes

support/download/wget +7 -0
index c69e6071aa..7f631ebb61 100755
@@ -7,6 +7,7 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the file exists remotely.
 #   -o FILE     Save into file FILE.
 #   -f FILENAME The filename of the tarball to get at URL
 #   -u URL      Download file at URL.
@@ -19,6 +20,7 @@ verbose=
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=1;;
     o)  output="${OPTARG}";;
     f)  filename="${OPTARG}";;
     u)  url="${OPTARG}";;
@@ -40,4 +42,9 @@ _wget() {
 # mirror
 [ -n "${encode}" ] && filename=${filename//\?/%3F}
 
+if [ -n "${checkonly}" ]; then
+    _wget --spider ${verbose} "${@}" "'${url}/${filename}'"
+    exit ${?}
+fi
+
 _wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
Signature · 1 line
2.19.2
#5 lore raw

PATCH v3 4/7 support/download/file: implement source-check

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

---
 support/download/file | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

v3: no changes

support/download/file +11 -1
index e52fcf2c8c..bf3c428cbe 100755
@@ -7,6 +7,7 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the source file exists.
 #   -o FILE     Copy to file FILE.
 #   -f FILE     Copy from basename file FILE.
 #   -u DIR      Copy from FILE in DIR.
@@ -23,6 +24,7 @@ verbose=-v
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=;;
+    C)  checkonly=1;;
     o)  output="${OPTARG}";;
     f)  file="${OPTARG}";;
     u)  dir="${OPTARG}";;
@@ -39,4 +41,12 @@ _localfiles() {
     eval ${LOCALFILES} "${@}"
 }
 
-_localfiles ${verbose} "'${dir##file://}/${file}'" "'${output}'"
+# Remove any scheme prefix
+dir="${dir##file://}"
+
+if [ -n "${checkonly}" ]; then
+    test -e "'${dir}/${file}'"
+    exit ${?}
+fi
+
+_localfiles ${verbose} "'${dir}/${file}'" "'${output}'"
Signature · 1 line
2.19.2
#6 lore raw

PATCH v3 5/7 Config.in: reintroduce BR2_SSH

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

The BR2_SSH command was removed in commit db9473bf6cd7bd12aa1f9faad0a917c973c33827 ("core/download: drop the SSH command") but will be needed again to support 'source-check' for the scp download backend.

---
 Config.in               | 4 ++++
 package/pkg-download.mk | 1 +
 2 files changed, 5 insertions(+)

v3: no changes

Config.in +4 -0
index d58d8dc04a..91d56907fe 100644
@@ -136,6 +136,10 @@ config BR2_SCP
 	string "Secure copy (scp) command"
 	default "scp"
 
+config BR2_SSH
+	string "Secure shell (ssh) command"
+	default "ssh"
+
 config BR2_HG
 	string "Mercurial (hg) command"
 	default "hg"
package/pkg-download.mk +1 -0
index 5384fc2af4..17d1ca99ee 100644
@@ -15,6 +15,7 @@ export BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
 export HG := $(call qstrip,$(BR2_HG))
 export SCP := $(call qstrip,$(BR2_SCP))
+export SSH := $(call qstrip,$(BR2_SSH))
 export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
 DL_WRAPPER = support/download/dl-wrapper
Signature · 1 line
2.19.2
#7 lore raw

PATCH v3 6/7 support/download/scp: implement source-check

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

---
 support/download/scp | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

v3: no changes

support/download/scp +14 -0
index 80cf495c4e..d81952956c 100755
@@ -7,17 +7,20 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the file exists remotely.
 #   -o FILE     Copy to local file FILE.
 #   -f FILE     Copy from remote file FILE.
 #   -u URI      Download file at URI.
 #
 # Environment:
 #   SCP       : the scp command to call
+#   SSH       : the ssh command to use for checkonly
 
 verbose=
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=1;;
     o)  output="${OPTARG}";;
     f)  filename="${OPTARG}";;
     u)  uri="${OPTARG}";;
@@ -33,8 +36,19 @@ shift $((OPTIND-1)) # Get rid of our options
 _scp() {
     eval ${SCP} "${@}"
 }
+_ssh() {
+    eval ${SSH} "${@}"
+}
 
 # Remove any scheme prefix
 uri="${uri##scp://}"
 
+if [ -n "${checkonly}" ]; then
+    # uri now looks like:  foo.example.org:some/directory
+    domain="${uri%%:*}"
+    path="${uri#*:}/${filename}"
+    _ssh ${verbose} "${@}" "'${domain}'" ls "'${path}'" > /dev/null
+    exit ${?}
+fi
+
 _scp ${verbose} "${@}" "'${uri}/${filename}'" "'${output}'"
Signature · 1 line
2.19.2
#8 lore raw

PATCH v3 6/7 support/download/scp: implement source-check

Thomas, All,

On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:

Quoted 40 lines

From: Thomas De Schampheleire <[email protected]>

---
 support/download/scp | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

v3: no changes

support/download/scp +5 -0
index 80cf495c4e..d81952956c 100755
@@ -7,17 +7,20 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the file exists remotely.
 #   -o FILE     Copy to local file FILE.
 #   -f FILE     Copy from remote file FILE.
 #   -u URI      Download file at URI.
 #
 # Environment:
 #   SCP       : the scp command to call
+#   SSH       : the ssh command to use for checkonly
 
 verbose=
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=1;;
     o)  output="${OPTARG}";;
     f)  filename="${OPTARG}";;
     u)  uri="${OPTARG}";;
@@ -33,8 +36,19 @@ shift $((OPTIND-1)) # Get rid of our options
 _scp() {
     eval ${SCP} "${@}"
 }
+_ssh() {
+    eval ${SSH} "${@}"

support/download/scp shift $((OPTIND-1)) # Get rid of our options · line 50

Quoted 10 lines
support/download/scp +6 -0
+}
 
 # Remove any scheme prefix
 uri="${uri##scp://}"
 
+if [ -n "${checkonly}" ]; then
+    # uri now looks like:  foo.example.org:some/directory
+    domain="${uri%%:*}"
+    path="${uri#*:}/${filename}"
+    _ssh ${verbose} "${@}" "'${domain}'" ls "'${path}'" > /dev/null

I was going to reply to the previous thread, but you were too fast to respin, so here's my proposal to avoid ls:

    /usr/bin/env [ -f "'${path}'" ]

It is almost impossible to have a system that lacks 'env' or that have it in another location, as POSIX mandates env to exist, and IIRC, it even mandates it to be /usr/bin/env (of is it FHS? at least, scripts in Buildroot use "#!/usr/bin/env bash").

You may have to carefully quote the [ and ], to avoid the eval from doing a nasty, weel, evaluation...

Regards,
Yann E. MORIN.

Quoted 7 lines
support/download/scp +3 -0
+    exit ${?}
+fi
+
 _scp ${verbose} "${@}" "'${uri}/${filename}'" "'${output}'"
Signature · 1 line
2.19.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#9 lore raw

PATCH v3 6/7 support/download/scp: implement source-check

El s?b., 9 feb. 2019 a las 23:09, Yann E. MORIN
(<[email protected]>) escribi?:

Quoted 60 lines

Thomas, All,

On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:

Quoted 40 lines

From: Thomas De Schampheleire <[email protected]>

---
 support/download/scp | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

v3: no changes

support/download/scp +5 -0
index 80cf495c4e..d81952956c 100755
@@ -7,17 +7,20 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the file exists remotely.
 #   -o FILE     Copy to local file FILE.
 #   -f FILE     Copy from remote file FILE.
 #   -u URI      Download file at URI.
 #
 # Environment:
 #   SCP       : the scp command to call
+#   SSH       : the ssh command to use for checkonly
 
 verbose=
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=1;;
     o)  output="${OPTARG}";;
     f)  filename="${OPTARG}";;
     u)  uri="${OPTARG}";;
@@ -33,8 +36,19 @@ shift $((OPTIND-1)) # Get rid of our options
 _scp() {
     eval ${SCP} "${@}"
 }
+_ssh() {
+    eval ${SSH} "${@}"

support/download/scp shift $((OPTIND-1)) # Get rid of our options · line 50

Quoted 10 lines
support/download/scp +6 -0
+}
 
 # Remove any scheme prefix
 uri="${uri##scp://}"
 
+if [ -n "${checkonly}" ]; then
+    # uri now looks like:  foo.example.org:some/directory
+    domain="${uri%%:*}"
+    path="${uri#*:}/${filename}"
+    _ssh ${verbose} "${@}" "'${domain}'" ls "'${path}'" > /dev/null

I was going to reply to the previous thread, but you were too fast to respin, so here's my proposal to avoid ls:

    /usr/bin/env [ -f "'${path}'" ]
Quoted 7 lines

It is almost impossible to have a system that lacks 'env' or that have it in another location, as POSIX mandates env to exist, and IIRC, it even mandates it to be /usr/bin/env (of is it FHS? at least, scripts in Buildroot use "#!/usr/bin/env bash").

You may have to carefully quote the [ and ], to avoid the eval from doing a nasty, weel, evaluation...

I would use 'test' rather than [ ] to avoid any such problem.

Thanks for the suggestion, I will test it.

/Thomas

#10 lore raw

PATCH v3 6/7 support/download/scp: implement source-check

El vie., 15 feb. 2019 a las 20:15, Thomas De Schampheleire (<[email protected]>) escribi?:

Quoted 68 lines

El s?b., 9 feb. 2019 a las 23:09, Yann E. MORIN
(<[email protected]>) escribi?:

Quoted 65 lines

Thomas, All,

On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:

Quoted 40 lines

From: Thomas De Schampheleire <[email protected]>

---
 support/download/scp | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

v3: no changes

support/download/scp +5 -0
index 80cf495c4e..d81952956c 100755
@@ -7,17 +7,20 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the file exists remotely.
 #   -o FILE     Copy to local file FILE.
 #   -f FILE     Copy from remote file FILE.
 #   -u URI      Download file at URI.
 #
 # Environment:
 #   SCP       : the scp command to call
+#   SSH       : the ssh command to use for checkonly
 
 verbose=
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=1;;
     o)  output="${OPTARG}";;
     f)  filename="${OPTARG}";;
     u)  uri="${OPTARG}";;
@@ -33,8 +36,19 @@ shift $((OPTIND-1)) # Get rid of our options
 _scp() {
     eval ${SCP} "${@}"
 }
+_ssh() {
+    eval ${SSH} "${@}"

support/download/scp shift $((OPTIND-1)) # Get rid of our options · line 50

Quoted 10 lines
support/download/scp +6 -0
+}
 
 # Remove any scheme prefix
 uri="${uri##scp://}"
 
+if [ -n "${checkonly}" ]; then
+    # uri now looks like:  foo.example.org:some/directory
+    domain="${uri%%:*}"
+    path="${uri#*:}/${filename}"
+    _ssh ${verbose} "${@}" "'${domain}'" ls "'${path}'" > /dev/null

I was going to reply to the previous thread, but you were too fast to respin, so here's my proposal to avoid ls:

    /usr/bin/env [ -f "'${path}'" ]

It is almost impossible to have a system that lacks 'env' or that have it in another location, as POSIX mandates env to exist, and IIRC, it even mandates it to be /usr/bin/env (of is it FHS? at least, scripts in Buildroot use "#!/usr/bin/env bash").

Quoted 7 lines
Quoted 2 lines

You may have to carefully quote the [ and ], to avoid the eval from doing a nasty, weel, evaluation...

I would use 'test' rather than [ ] to avoid any such problem.

Thanks for the suggestion, I will test it.

Actually, I don't think the 'env' is needed, a simple 'test -f' will do, my reasoning below.

Arnout is right that 'ls' is not guaranteed to exist, it is not a mandatory POSIX utility but an 'XSI' extension.
http://pubs.opengroup.org/onlinepubs/009695399/utilities/ls.html

On the contrary, 'test' is a mandatory POSIX utility (i.e. there are no specific codes like XSI mentioned on following page)
http://pubs.opengroup.org/onlinepubs/000095399/utilities/test.html

So 'test' will certainly exist. We are not sure if it is in /bin/ or /usr/bin/ or another place, though.
By calling it without absolute path, we assume that it is in the standard PATH.

One could argue that '/usr/bin/env test' is better, but if 'test' is _not_ in the standard PATH, then 'env' will also not find it. So, I don't see how 'env' brings an advantage.

/Thomas

#11 lore raw

PATCH v3 7/7 support/download/{bzr, cvs, git, svn}: highlight unimplemented source-check

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

For bzr, cvs, git and svn, there is no method of checking whether the revision exists remotely, without actually cloning the repository. Therefore, there is no actual implementation for '-C' (checkonly) for these download helpers. The script will fall back to the normal download logic, which would effectively fail if the revision does not exist.

For completeness, mention this unimplemented feature in the usage text.

---
 support/download/bzr | 2 ++
 support/download/cvs | 2 ++
 support/download/git | 2 ++
 support/download/svn | 2 ++
 4 files changed, 8 insertions(+)

v3: no changes

support/download/bzr +2 -0
index 5289a421cd..278d8778af 100755
@@ -7,6 +7,8 @@ set -e
 #
 # Options:
 #   -q          Be quiet
+#   -C          (unimplemented) Only check that the revision exists in the
+#               remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Clone from repository at URI.
 #   -c CSET     Use changeset (or revision) CSET.
support/download/cvs +2 -0
index 9d0dc3cb3a..d9a586293a 100755
@@ -7,6 +7,8 @@ set -e
 #
 # Options:
 #   -q          Be quiet
+#   -C          (unimplemented) Only check that the revision exists in the
+#               remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Checkout from repository at URI.
 #   -c REV      Use revision REV.
support/download/git +2 -0
index 17ca04eb98..e9c02d6712 100755
@@ -7,6 +7,8 @@ set -E
 #
 # Options:
 #   -q          Be quiet.
+#   -C          (unimplemented) Only check that the revision exists in the
+#               remote repository.
 #   -r          Clone and archive sub-modules.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Clone from repository at URI.
support/download/svn +2 -0
index 542b25c0a2..5fff064683 100755
@@ -7,6 +7,8 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          (unimplemented) Only check that the revision exists in the
+#               remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Checkout from repository at URI.
 #   -c REV      Use revision REV.
Signature · 1 line
2.19.2
#12 lore raw

PATCH v3 7/7 support/download/{bzr, cvs, git, svn}: highlight unimplemented source-check

Thomas, All,

On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:

Quoted 3 lines

From: Thomas De Schampheleire <[email protected]>

For bzr, cvs, git and svn, there is no method of checking whether the

Acutally, for svn this is realtively trivial:

    svn ls -r ${rev} ${remote}

We have two packages in Buildroot that use svn, fis and open2300, so you cn use them to test:

    $ svn ls -r 2892 http://svn.chezphil.org/utils/trunk
    Redirecting to URL 'https://svn.chezphil.org/utils/trunk':
    Makefile
    apexctl.cc
    [...]

Regards,
Yann E. MORIN.

Quoted 72 lines

revision exists remotely, without actually cloning the repository. Therefore, there is no actual implementation for '-C' (checkonly) for these download helpers. The script will fall back to the normal download logic, which would effectively fail if the revision does not exist.

For completeness, mention this unimplemented feature in the usage text.

---
 support/download/bzr | 2 ++
 support/download/cvs | 2 ++
 support/download/git | 2 ++
 support/download/svn | 2 ++
 4 files changed, 8 insertions(+)

v3: no changes

support/download/bzr +2 -0
index 5289a421cd..278d8778af 100755
@@ -7,6 +7,8 @@ set -e
 #
 # Options:
 #   -q          Be quiet
+#   -C          (unimplemented) Only check that the revision exists in the
+#               remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Clone from repository at URI.
 #   -c CSET     Use changeset (or revision) CSET.
support/download/cvs +2 -0
index 9d0dc3cb3a..d9a586293a 100755
@@ -7,6 +7,8 @@ set -e
 #
 # Options:
 #   -q          Be quiet
+#   -C          (unimplemented) Only check that the revision exists in the
+#               remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Checkout from repository at URI.
 #   -c REV      Use revision REV.
support/download/git +2 -0
index 17ca04eb98..e9c02d6712 100755
@@ -7,6 +7,8 @@ set -E
 #
 # Options:
 #   -q          Be quiet.
+#   -C          (unimplemented) Only check that the revision exists in the
+#               remote repository.
 #   -r          Clone and archive sub-modules.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Clone from repository at URI.
support/download/svn +2 -0
index 542b25c0a2..5fff064683 100755
@@ -7,6 +7,8 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          (unimplemented) Only check that the revision exists in the
+#               remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Checkout from repository at URI.
 #   -c REV      Use revision REV.
Signature · 1 line
2.19.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#13 lore raw

PATCH v3 7/7 support/download/{bzr, cvs, git, svn}: highlight unimplemented source-check

El s?b., 9 feb. 2019 a las 23:13, Yann E. MORIN
(<[email protected]>) escribi?:

Quoted 12 lines

Thomas, All,

On 2019-02-09 21:23 +0100, Thomas De Schampheleire spake thusly:

Quoted 3 lines

From: Thomas De Schampheleire <[email protected]>

For bzr, cvs, git and svn, there is no method of checking whether the

Acutally, for svn this is realtively trivial:

    svn ls -r ${rev} ${remote}
Quoted 9 lines

We have two packages in Buildroot that use svn, fis and open2300, so you cn use them to test:

    $ svn ls -r 2892 http://svn.chezphil.org/utils/trunk
    Redirecting to URL 'https://svn.chezphil.org/utils/trunk':
    Makefile
    apexctl.cc
    [...]

Thanks a lot, will double-check at my end and adapt.

/Thomas