support/download: reintroduce 'source-check' target
PATCH v5 1/9 support/download: reintroduce 'source-check' target
From: Thomas De Schampheleire <[email protected]>
Commit bf28a165d992681a85b43a886005e669b3cb579b removed the source-check target to allow easier refactoring of the download infrastructure. It was partly based on the fact that no-one really used this target.
However, it turns out that there _are_ good uses for it. In a work environment where many people are submitting changes to a Buildroot repository, one cannot always blindly trust every change. A typical case of human error is when bumping the version of a package but forgetting to upload the tarball to the internal BR2_PRIMARY_SITE or (in case of a repository) pushing the new changesets to the repository.
If a user cannot directly push to the Buildroot repository but needs to queue their change via an automatic validation system, that validation system can use 'make source-check' on the relevant defconfigs to protect against such errors. A full download would also be possible but takes longer and unnecessarily uses internal network bandwidth.
With that use case in mind, this commit reintroduces the 'source-check' target, but embedded in the current situation with a dl-wrapper. The changes to the wrapper are minimal (not considering additional indentation). A new option '-C' means 'check only' and will be passed to the download backends. If the backend supports the option, no download will happen. If it does not, then the backend will actually perform a download as a means of checking that the source exists (a form of graceful degradation). In neither case, though, hash checking is performed (as the standard case is without download thus without file to calculate hashes on).
Subsequent commits will actually implement -C in the backends.
--- Makefile | 7 +++ package/pkg-download.mk | 6 +++ package/pkg-generic.mk | 14 ++++- support/download/dl-wrapper | 100 ++++++++++++++++++++---------------- 4 files changed, 82 insertions(+), 45 deletions(-)
Makefile +7 -0
@@ -131,6 +131,7 @@ noconfig_targets := menuconfig nconfig gconfig xconfig config oldconfig randconf # (default target is to build), or when MAKECMDGOALS contains # something else than one of the nobuild_targets. nobuild_targets := source %-source \ + source-check %-source-check %-all-source-check \ legal-info %-legal-info external-deps _external-deps \ clean distclean help show-targets graph-depends \ %-graph-depends %-show-depends %-show-version \
@@ -818,6 +819,9 @@ target-post-image: $(TARGETS_ROOTFS) target-finalize staging-finalize .PHONY: source source: $(foreach p,$(PACKAGES),$(p)-all-source) +.PHONY: source-check +source-check: $(foreach p,$(PACKAGES),$(p)-all-source-check) + .PHONY: _external-deps external-deps _external-deps: $(foreach p,$(PACKAGES),$(p)-all-external-deps) external-deps:
@@ -1103,6 +1107,8 @@ help: @echo ' <pkg>-dirclean - Remove <pkg> build directory' @echo ' <pkg>-reconfigure - Restart the build from the configure step' @echo ' <pkg>-rebuild - Restart the build from the build step' + @echo ' <pkg>-source-check - Check package for valid download URLs' + @echo ' <pkg>-all-source-check - Check package and its dependencies for valid download URLs' $(foreach p,$(HELP_PACKAGES), \ @echo $(sep) \ @echo '$($(p)_NAME):' $(sep) \
@@ -1122,6 +1128,7 @@ help: @echo @echo 'Miscellaneous:' @echo ' source - download all sources needed for offline-build' + @echo ' source-check - check selected packages for valid download URLs' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' @echo ' printvars - dump all the internal variables'
package/pkg-download.mk +6 -0
@@ -70,6 +70,7 @@ export BR_NO_CHECK_HASH_FOR = # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set # # Argument 1 is the source location +# Argument 2 (optionally) provides extra arguments to pass to DL_WRAPPER # ################################################################################
@@ -92,6 +93,7 @@ endif define DOWNLOAD $(Q)mkdir -p $($(PKG)_DL_DIR) $(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \ + $(2) \ -c '$($(PKG)_DL_VERSION)' \ -d '$($(PKG)_DL_DIR)' \ -D '$(DL_DIR)' \
@@ -106,3 +108,7 @@ define DOWNLOAD -- \ $($(PKG)_DL_OPTS) endef + +define SOURCE_CHECK + $(call DOWNLOAD,$(1),-C) +endef
package/pkg-generic.mk +13 -1
@@ -788,6 +788,10 @@ $(1)-legal-source: $$($(2)_TARGET_ACTUAL_SOURCE) endif # actual sources != sources endif # actual sources != "" +$(1)-source-check: PKG=$(2) +$(1)-source-check: + $$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep)) + $(1)-external-deps: @for p in $$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS) ; do \ echo `basename $$$$p` ; \
@@ -812,6 +816,9 @@ $(1)-rsync: $$($(2)_TARGET_RSYNC) $(1)-source: $(1)-legal-source: +$(1)-source-check: + test -d $$($(2)_OVERRIDE_SRCDIR) + $(1)-external-deps: @echo "file://$$($(2)_OVERRIDE_SRCDIR)" endif
@@ -846,6 +853,9 @@ $(1)-graph-rdepends: graph-depends-requirements $(1)-all-source: $(1)-source $(1)-all-source: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source) +$(1)-all-source-check: $(1)-source-check +$(1)-all-source-check: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source-check) + $(1)-all-external-deps: $(1)-external-deps $(1)-all-external-deps: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-external-deps)
@@ -1044,6 +1054,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-all-external-deps \ $(1)-all-legal-info \ $(1)-all-source \ + $(1)-all-source-check \ $(1)-build \ $(1)-clean-for-rebuild \ $(1)-clean-for-reconfigure \
@@ -1068,7 +1079,8 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-rsync \ $(1)-show-depends \ $(1)-show-version \ - $(1)-source + $(1)-source \ + $(1)-source-check ifneq ($$($(2)_SOURCE),) ifeq ($$($(2)_SITE),)
support/download/dl-wrapper +56 -44
142 lines not on this page · raw message
Signature · 1 line
2.19.2
PATCH v5 2/9 support/download/hg: implement source-check
From: Thomas De Schampheleire <[email protected]>
Note: the implementation is different (better) than what used to be in Buildroot before source-check was removed.
The original implementation:
hg incoming --force -l1 <URL>
would only verify that the repository exists, not that the requested revision is present.
An already better implementation is:
hg incoming --force -l1 -r <revision> <URL>
but compared to the next solution it has a large resource consumption on the local machine. In the background, the full repository is first downloaded.
The implemented solution is:
hg identify -r <revision> <URL>
which operates directly on the remote repository.
--- support/download/hg | 12 ++++++++++++ 1 file changed, 12 insertions(+)
v5: no changes
v4: (feedback Yann E. Morin)
- use true/false as values to 'checkonly'
- always check revision, even if no source-check
- replace incorrect 'exit $?' by explicit 'exit 0'
v3: no changes
support/download/hg +12 -0
@@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the changeset exists in the remote repository. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET.
@@ -16,9 +17,11 @@ set -e # HG : the hg command to call verbose= +checkonly=false while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=true;; o) output="${OPTARG}";; u) uri="${OPTARG}";; c) cset="${OPTARG}";;
@@ -36,6 +39,15 @@ _hg() { eval ${HG} "${@}" } +if ! _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null; then + printf "Commit '%s' does not exist in this repository\n." "${cset}" + exit 1 +fi + +if ${checkonly}; then + exit 0 +fi + _hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'" _hg archive ${verbose} --repository "'${basename}'" --type tgz \
Signature · 1 line
2.19.2
PATCH v5 3/9 support/download/wget: implement source-check
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
--- support/download/wget | 8 ++++++++ 1 file changed, 8 insertions(+)
v5: no changes
v4: (feedback Yann E. Morin)
- use true/false as values to 'checkonly'
- replace incorrect 'exit $?' by explicit 'exit 0'
v3: no changes
support/download/wget +8 -0
@@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the file exists remotely. # -o FILE Save into file FILE. # -f FILENAME The filename of the tarball to get at URL # -u URL Download file at URL.
@@ -16,9 +17,11 @@ set -e # WGET : the wget command to call verbose= +checkonly=false while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=true;; o) output="${OPTARG}";; f) filename="${OPTARG}";; u) url="${OPTARG}";;
@@ -40,4 +43,9 @@ _wget() { # mirror [ -n "${encode}" ] && filename=${filename//\?/%3F} +if ${checkonly}; then + _wget --spider ${verbose} "${@}" "'${url}/${filename}'" + exit 0 +fi + _wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
Signature · 1 line
2.19.2
PATCH v5 4/9 support/download/file: implement source-check
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
--- support/download/file | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
v5: no changes
v4:
- use true/false as values to 'checkonly' (Yann E. Morin)
- replace incorrect 'exit $?' by explicit 'exit 0' (Yann E. Morin)
- replace 'test -e' by 'test -f' which is more specific
v3: no changes
support/download/file +12 -1
@@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the source file exists. # -o FILE Copy to file FILE. # -f FILE Copy from basename file FILE. # -u DIR Copy from FILE in DIR.
@@ -20,9 +21,11 @@ set -e # Make 'cp' verbose by default, so it behaves a bit like the others. verbose=-v +checkonly=false while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=;; + C) checkonly=true;; o) output="${OPTARG}";; f) file="${OPTARG}";; u) dir="${OPTARG}";;
@@ -39,4 +42,12 @@ _localfiles() { eval ${LOCALFILES} "${@}" } -_localfiles ${verbose} "'${dir##file://}/${file}'" "'${output}'" +# Remove any scheme prefix +dir="${dir##file://}" + +if ${checkonly}; then + test -f "'${dir}/${file}'" + exit 0 +fi + +_localfiles ${verbose} "'${dir}/${file}'" "'${output}'"
Signature · 1 line
2.19.2
PATCH v5 5/9 Revert "core/download: drop the SSH command"
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
This reverts commit db9473bf6cd7bd12aa1f9faad0a917c973c33827.
The SSH command will again be needed to support source-check for the 'scp' download backend.
--- Config.in | 4 ++++ package/pkg-download.mk | 1 + 2 files changed, 5 insertions(+)
v5: use 'git revert' instead
v4: no changes
v3: no changes
Config.in +4 -0
@@ -136,6 +136,10 @@ config BR2_SCP string "Secure copy (scp) command" default "scp" +config BR2_SSH + string "Secure shell (ssh) command" + default "ssh" + config BR2_HG string "Mercurial (hg) command" default "hg"
package/pkg-download.mk +1 -0
@@ -15,6 +15,7 @@ export BZR := $(call qstrip,$(BR2_BZR)) export GIT := $(call qstrip,$(BR2_GIT)) export HG := $(call qstrip,$(BR2_HG)) export SCP := $(call qstrip,$(BR2_SCP)) +SSH := $(call qstrip,$(BR2_SSH)) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) DL_WRAPPER = support/download/dl-wrapper
Signature · 1 line
2.19.2
PATCH v5 6/9 package/pkg-download: export 'SSH' for use in the download backends
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
Originally, the tools defined in pkg-download.mk were used from Makefiles and were directly available. Due to the introduction of the download infrastructure in shell scripts, these variables now need to be exported in order for the scripts to see them.
--- package/pkg-download.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
v5: new patch
package/pkg-download.mk +1 -1
@@ -15,7 +15,7 @@ export BZR := $(call qstrip,$(BR2_BZR)) export GIT := $(call qstrip,$(BR2_GIT)) export HG := $(call qstrip,$(BR2_HG)) export SCP := $(call qstrip,$(BR2_SCP)) -SSH := $(call qstrip,$(BR2_SSH)) +export SSH := $(call qstrip,$(BR2_SSH)) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) DL_WRAPPER = support/download/dl-wrapper
Signature · 1 line
2.19.2
PATCH v5 7/9 support/download/scp: implement source-check
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
We use 'test -f' rather than 'ls' because 'test' is a mandatory POSIX utility while 'ls' is not.
--- support/download/scp | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
v5: no changes
v4: (feedback Yann E. Morin)
- use true/false as values to 'checkonly'
- replace incorrect 'exit $?' by explicit 'exit 0'
- use 'test -f' instead of 'ls' which is more certain to be present (POSIX mandatory utility)
v3: no changes
support/download/scp +15 -0
@@ -7,17 +7,21 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the file exists remotely. # -o FILE Copy to local file FILE. # -f FILE Copy from remote file FILE. # -u URI Download file at URI. # # Environment: # SCP : the scp command to call +# SSH : the ssh command to use for checkonly verbose= +checkonly=false while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=true;; o) output="${OPTARG}";; f) filename="${OPTARG}";; u) uri="${OPTARG}";;
@@ -33,8 +37,19 @@ shift $((OPTIND-1)) # Get rid of our options _scp() { eval ${SCP} "${@}" } +_ssh() { + eval ${SSH} "${@}" +} # Remove any scheme prefix uri="${uri##scp://}" +if ${checkonly}; then + # uri now looks like: foo.example.org:some/directory + domain="${uri%%:*}" + path="${uri#*:}/${filename}" + _ssh ${verbose} "${@}" "'${domain}'" test -f "'${path}'" > /dev/null + exit 0 +fi + _scp ${verbose} "${@}" "'${uri}/${filename}'" "'${output}'"
Signature · 1 line
2.19.2
PATCH v5 8/9 support/download/svn: implement source-check
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
--- support/download/svn | 12 ++++++++++++ 1 file changed, 12 insertions(+)
v5: no changes
v4: new patch
support/download/svn +12 -0
@@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the revision exists in the remote repository. # -o FILE Generate archive in FILE. # -u URI Checkout from repository at URI. # -c REV Use revision REV.
@@ -16,9 +17,11 @@ set -e # SVN : the svn command to call verbose= +checkonly=false while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=true;; o) output="${OPTARG}";; u) uri="${OPTARG}";; c) rev="${OPTARG}";;
@@ -36,6 +39,15 @@ _svn() { eval ${SVN} "${@}" } +if ! _svn ls ${verbose} -r "'${rev}'" "'${uri}'" > /dev/null; then + printf "Revision '%s' does not exist in this repository\n." "${rev}" + exit 1 +fi + +if ${checkonly}; then + exit 0 +fi + _svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'" tar czf "${output}" "${basename}"
Signature · 1 line
2.19.2
PATCH v5 9/9 support/download/{bzr, cvs, git}: highlight unimplemented source-check
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
For bzr, cvs and git, there is no method of checking whether the revision exists remotely, without actually cloning the repository. Therefore, there is no actual implementation for '-C' (checkonly) for these download helpers. The script will fall back to the normal download logic, which would effectively fail if the revision does not exist.
For completeness, mention this unimplemented feature in the usage text.
--- support/download/bzr | 2 ++ support/download/cvs | 2 ++ support/download/git | 2 ++ 3 files changed, 6 insertions(+)
v5: no changes
v4: remove 'svn' which now has an actual implementation
v3: no changes
support/download/bzr +2 -0
@@ -7,6 +7,8 @@ set -e # # Options: # -q Be quiet +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET.
support/download/cvs +2 -0
@@ -7,6 +7,8 @@ set -e # # Options: # -q Be quiet +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -o FILE Generate archive in FILE. # -u URI Checkout from repository at URI. # -c REV Use revision REV.
support/download/git +2 -0
@@ -7,6 +7,8 @@ set -E # # Options: # -q Be quiet. +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -r Clone and archive sub-modules. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI.
Signature · 1 line
2.19.2
PATCH v5 1/9 support/download: reintroduce 'source-check' target
In reply to Thomas De SchampheleireThomas, All,
On 2019-02-19 11:38 +0100, Thomas De Schampheleire spake thusly:
Quoted 18 lines
From: Thomas De Schampheleire <[email protected]>
Commit
bf28a165d992681a85b43a886005e669b3cb579bremoved the source-check target to allow easier refactoring of the download infrastructure. It was partly based on the fact that no-one really used this target.However, it turns out that there _are_ good uses for it. In a work environment where many people are submitting changes to a Buildroot repository, one cannot always blindly trust every change. A typical case of human error is when bumping the version of a package but forgetting to upload the tarball to the internal
BR2_PRIMARY_SITEor (in case of a repository) pushing the new changesets to the repository.
If a user cannot directly push to the Buildroot repository but needs to queue their change via an automatic validation system, that validation system can use 'make source-check' on the relevant defconfigs to protect against such errors. A full download would also be possible but takes longer and unnecessarily uses internal network bandwidth.
Quoted 9 lines
With that use case in mind, this commit reintroduces the 'source-check' target, but embedded in the current situation with a dl-wrapper. The changes to the wrapper are minimal (not considering additional indentation). A new option '-C' means 'check only' and will be passed to the download backends. If the backend supports the option, no download will happen. If it does not, then the backend will actually perform a download as a means of checking that the source exists (a form of graceful degradation). In neither case, though, hash checking is performed (as the standard case is without download thus without file to calculate hashes on).
So, that last part made me wonder: in the wget case, if the HEAD succeeds, how does that guarantee that the resource does actually correspond to what you expected?
What if upstream moved the tarball away, but the server is now replying garbaged 404-that-is-not-404 so HEAD succeeds?
Or what if the server does not accept HEAD requests?
Or, in the case of git, what if the tag still exists but has re-tagged?
In short, source-check does not guarantee that what exist "there" is what you do expect to get.
As discussed with Thomas P., it all depends on the semantics we want to give to source-check. But the one implemented by your series does not seem that interesting in the end. Yes, it makes your remote-worker-with- a-slow-connection maybe a bit more bearable.
It also means that all it says is that, right at the time of source-check, here is actually "something" named as you expect, but there is no guarantee that a 'make source' later will be able to get that "something" anyway.
So, I am still not entirely convinced by the usefulness of source-check.
Regards,
Yann E. MORIN.
Quoted 285 lines
Subsequent commits will actually implement -C in the backends.
Signed-off-by Thomas De Schampheleire <[email protected]>--- Makefile | 7 +++ package/pkg-download.mk | 6 +++ package/pkg-generic.mk | 14 ++++- support/download/dl-wrapper | 100 ++++++++++++++++++++---------------- 4 files changed, 82 insertions(+), 45 deletions(-)Makefile +7
-0@@ -131,6 +131,7 @@ noconfig_targets := menuconfig nconfig gconfig xconfig config oldconfig randconf # (default target is to build), or when MAKECMDGOALS contains # something else than one of the nobuild_targets. nobuild_targets := source %-source \ + source-check %-source-check %-all-source-check \ legal-info %-legal-info external-deps _external-deps \ clean distclean help show-targets graph-depends \ %-graph-depends %-show-depends %-show-version \@@ -818,6 +819,9 @@ target-post-image: $(TARGETS_ROOTFS) target-finalize staging-finalize .PHONY: source source: $(foreach p,$(PACKAGES),$(p)-all-source) +.PHONY: source-check +source-check: $(foreach p,$(PACKAGES),$(p)-all-source-check) + .PHONY: _external-deps external-deps _external-deps: $(foreach p,$(PACKAGES),$(p)-all-external-deps) external-deps:@@ -1103,6 +1107,8 @@ help: @echo ' <pkg>-dirclean - Remove <pkg> build directory' @echo ' <pkg>-reconfigure - Restart the build from the configure step' @echo ' <pkg>-rebuild - Restart the build from the build step' + @echo ' <pkg>-source-check - Check package for valid download URLs' + @echo ' <pkg>-all-source-check - Check package and its dependencies for valid download URLs' $(foreach p,$(HELP_PACKAGES), \ @echo $(sep) \ @echo '$($(p)_NAME):' $(sep) \@@ -1122,6 +1128,7 @@ help: @echo @echo 'Miscellaneous:' @echo ' source - download all sources needed for offline-build' + @echo ' source-check - check selected packages for valid download URLs' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' @echo ' printvars - dump all the internal variables'package/pkg-download.mk +6
-0@@ -70,6 +70,7 @@ export BR_NO_CHECK_HASH_FOR = # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set # # Argument 1 is the source location +# Argument 2 (optionally) provides extra arguments to pass to DL_WRAPPER # ################################################################################@@ -92,6 +93,7 @@ endif define DOWNLOAD $(Q)mkdir -p $($(PKG)_DL_DIR) $(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \ + $(2) \ -c '$($(PKG)_DL_VERSION)' \ -d '$($(PKG)_DL_DIR)' \ -D '$(DL_DIR)' \@@ -106,3 +108,7 @@ define DOWNLOAD -- \ $($(PKG)_DL_OPTS) endef + +define SOURCE_CHECK + $(call DOWNLOAD,$(1),-C) +endefpackage/pkg-generic.mk +13
-1@@ -788,6 +788,10 @@ $(1)-legal-source: $$($(2)_TARGET_ACTUAL_SOURCE) endif # actual sources != sources endif # actual sources != "" +$(1)-source-check: PKG=$(2) +$(1)-source-check: + $$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep)) + $(1)-external-deps: @for p in $$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS) ; do \ echo `basename $$$$p` ; \@@ -812,6 +816,9 @@ $(1)-rsync: $$($(2)_TARGET_RSYNC) $(1)-source: $(1)-legal-source: +$(1)-source-check: + test -d $$($(2)_OVERRIDE_SRCDIR) + $(1)-external-deps: @echo "file://$$($(2)_OVERRIDE_SRCDIR)" endif@@ -846,6 +853,9 @@ $(1)-graph-rdepends: graph-depends-requirements $(1)-all-source: $(1)-source $(1)-all-source: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source) +$(1)-all-source-check: $(1)-source-check +$(1)-all-source-check: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source-check) + $(1)-all-external-deps: $(1)-external-deps $(1)-all-external-deps: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-external-deps)@@ -1044,6 +1054,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-all-external-deps \ $(1)-all-legal-info \ $(1)-all-source \ + $(1)-all-source-check \ $(1)-build \ $(1)-clean-for-rebuild \ $(1)-clean-for-reconfigure \@@ -1068,7 +1079,8 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-rsync \ $(1)-show-depends \ $(1)-show-version \ - $(1)-source + $(1)-source \ + $(1)-source-check ifneq ($$($(2)_SOURCE),) ifeq ($$($(2)_SITE),)support/download/dl-wrapper +56
-44@@ -17,7 +17,7 @@ # We want to catch any unexpected failure, and exit immediately. set -e -export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e" +export BR_BACKEND_DL_GETOPTS=":hc:Cd:o:n:N:H:ru:qf:e" main() { local OPT OPTARG@@ -25,9 +25,10 @@ main() { local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + while getopts ":c:Cd:D:o:n:N:H:rf:u:q" OPT; do case "${OPT}" in c) cset="${OPTARG}";; + C) checkonly=-C;; d) dl_dir="${OPTARG}";; D) old_dl_dir="${OPTARG}";; o) output="${OPTARG}";;@@ -46,38 +47,40 @@ main() { # Forget our options, and keep only those for the backend shift $((OPTIND-1)) - if [ -z "${output}" ]; then - error "no output specified, use -o\n" - fi + if [ -z "${checkonly}" ]; then + if [ -z "${output}" ]; then + error "no output specified, use -o\n" + fi - # Legacy handling: check if the file already exists in the global - # download directory. If it does, hard-link it. If it turns out it - # was an incorrect download, we'd still check it below anyway. - # If we can neither link nor copy, fallback to doing a download. - # NOTE! This is not atomic, is subject to TOCTTOU, but the whole - # dl-wrapper runs under an flock, so we're safe. - if [ ! -e "${output}" -a -e "${old_dl_dir}/${filename}" ]; then - ln "${old_dl_dir}/${filename}" "${output}" || \ - cp "${old_dl_dir}/${filename}" "${output}" || \ - true - fi + # Legacy handling: check if the file already exists in the global + # download directory. If it does, hard-link it. If it turns out it + # was an incorrect download, we'd still check it below anyway. + # If we can neither link nor copy, fallback to doing a download. + # NOTE! This is not atomic, is subject to TOCTTOU, but the whole + # dl-wrapper runs under an flock, so we're safe. + if [ ! -e "${output}" -a -e "${old_dl_dir}/${filename}" ]; then + ln "${old_dl_dir}/${filename}" "${output}" || \ + cp "${old_dl_dir}/${filename}" "${output}" || \ + true + fi - # If the output file already exists and: - # - there's no .hash file: do not download it again and exit promptly - # - matches all its hashes: do not download it again and exit promptly - # - fails at least one of its hashes: force a re-download - # - there's no hash (but a .hash file): consider it a hard error - if [ -e "${output}" ]; then - if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then - exit 0 - elif [ ${?} -ne 2 ]; then - # Do not remove the file, otherwise it might get re-downloaded - # from a later location (i.e. primary -> upstream -> mirror). - # Do not print a message, check-hash already did. - exit 1 + # If the output file already exists and: + # - there's no .hash file: do not download it again and exit promptly + # - matches all its hashes: do not download it again and exit promptly + # - fails at least one of its hashes: force a re-download + # - there's no hash (but a .hash file): consider it a hard error + if [ -e "${output}" ]; then + if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then + exit 0 + elif [ ${?} -ne 2 ]; then + # Do not remove the file, otherwise it might get re-downloaded + # from a later location (i.e. primary -> upstream -> mirror). + # Do not print a message, check-hash already did. + exit 1 + fi + rm -f "${output}" + warn "Re-downloading '%s'...\n" "${output##*/}" fi - rm -f "${output}" - warn "Re-downloading '%s'...\n" "${output##*/}" fi # Look through all the uris that we were given to download the package@@ -127,7 +130,7 @@ main() { -f "${filename}" \ -u "${uri}" \ -o "${tmpf}" \ - ${quiet} ${recurse} -- "${@}" + ${quiet} ${recurse} ${checkonly} -- "${@}" then # cd back to keep path coherence cd "${OLDPWD}"@@ -138,19 +141,21 @@ main() { # cd back to free the temp-dir, so we can remove it later cd "${OLDPWD}" - # Check if the downloaded file is sane, and matches the stored hashes - # for that file - if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then - rc=0 - else - if [ ${?} -ne 3 ]; then - rm -rf "${tmpd}" - continue + if [ -z "${checkonly}" ]; then + # Check if the downloaded file is sane, and matches the stored hashes + # for that file + if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then + rc=0 + else + if [ ${?} -ne 3 ]; then + rm -rf "${tmpd}" + continue + fi + + # the hash file exists and there was no hash to check the file + # against + rc=1 fi - - # the hash file exists and there was no hash to check the file - # against - rc=1 fi download_and_check=1 break@@ -163,6 +168,13 @@ main() { exit 1 fi + # If we only need to check the presence of sources, stop here. + # No need to handle output files. + if [ -n "${checkonly}" ]; then + rm -rf "${tmpd}" + exit 0 + fi + # tmp_output is in the same directory as the final output, so we can # later move it atomically. tmp_output="$(mktemp "${output}.XXXXXX")"Signature · 1 line
2.19.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v5 1/9 support/download: reintroduce 'source-check' target
Hello,
Quoted 11 lines · On Sun, 17 Mar 2019 15:32:15 +0100 "Yann E. MORIN" <[email protected]>…
As discussed with Thomas P., it all depends on the semantics we want to give to source-check. But the one implemented by your series does not seem that interesting in the end. Yes, it makes your remote-worker-with- a-slow-connection maybe a bit more bearable.
It also means that all it says is that, right at the time of source-check, here is actually "something" named as you expect, but there is no guarantee that a 'make source' later will be able to get that "something" anyway.
So, I am still not entirely convinced by the usefulness of source-check.
Just to add to Yann's explanations, here are the two aspects that makes the proposed semantic of "source-check" a bit weak:
- It only says "something" was available at the specified download location, but not that this something matches the hash we have in the package hash file. Hence a successful "make source-check" does not guarantee a successful "make source".
- It only says at the time of "make source-check" that "something" was available at the specified download location, but 2 minutes later the file could be removed upstream, the tag be dropped, etc. Hence a successful "make source-check" does not guarantee a successful "make source".
So, a "make source-check" does not provide you much guarantees. It only tells you that at the time of the source-check, "something" was available at the specified download location, but you don't know if that "something" is correct and that it will still be around when you'll do your build.
If we are willing to accept this very weak semantic of source-check, then fair enough, but we (and our users) have to be aware of the very limited guarantees that source-check provides.
What do you think?
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v5 1/9 support/download: reintroduce 'source-check' target
Quoted 37 lines · Thomas Petazzoni
Hello,
Quoted 11 lines · On Sun, 17 Mar 2019 15:32:15 +0100 "Yann E. MORIN" <[email protected]>…
As discussed with Thomas P., it all depends on the semantics we want to give to source-check. But the one implemented by your series does not seem that interesting in the end. Yes, it makes your remote-worker-with- a-slow-connection maybe a bit more bearable.
It also means that all it says is that, right at the time of source-check, here is actually "something" named as you expect, but there is no guarantee that a 'make source' later will be able to get that "something" anyway.
So, I am still not entirely convinced by the usefulness of source-check.
Just to add to Yann's explanations, here are the two aspects that makes the proposed semantic of "source-check" a bit weak:
- It only says "something" was available at the specified download location, but not that this something matches the hash we have in the package hash file. Hence a successful "make source-check" does not guarantee a successful "make source".
- It only says at the time of "make source-check" that "something" was available at the specified download location, but 2 minutes later the file could be removed upstream, the tag be dropped, etc. Hence a successful "make source-check" does not guarantee a successful "make source".
So, a "make source-check" does not provide you much guarantees. It only tells you that at the time of the source-check, "something" was available at the specified download location, but you don't know if that "something" is correct and that it will still be around when you'll do your build.
Quoted 3 lines
If we are willing to accept this very weak semantic of source-check, then fair enough, but we (and our users) have to be aware of the very limited guarantees that source-check provides.
For the use case of Thomas DS, this weak semantic is perfectly fine. What it
tries to protect against is the common mistake that you add or bump a package and forget to upload the source to PRIMARY_SITE. As explained by Thomas DS, doing a full download is eventually still needed, but takes much longer:
source-check may take in the order of 1 minute while the full download takes 10 minutes. That makes all the difference for immediate feedback to the developer, and it doesn't significantly slow down a good pipeline.
Indeed, it does not protect against uploading the wrong tarball with the
correct name. It also doesn't protect against forgetting to update the hash file. It also doesn't protect against the package failing to build. It also doesn't protect against bugs in the code which become only apparent at run time. But all of these are checked later in CI, and the source check captures an important (and likely) class of mistakes early on.
It is true that this is a very narrow use case. However, I think it passes two
tests for acceptance:
* It does not make Buildroot (much) more complex.
* There is no way to do this with scripting outside of Buildroot.
Regarding that second point: in fact there would be a way, if we would instead
have a command to print everything that would be downloaded. That would be very similar to patch 1/9, but would remove the need for the other patches in the series. The advantage of this approach is that there could be other uses for printing the list of sources. But that would be back to the drawing board for Thomas DS, so I doubt he's enthusiastic about that option :-) Plus, we have no actual example of an additional use case.
Regards,
Arnout
PATCH v5 1/9 support/download: reintroduce 'source-check' target
El mi?., 27 mar. 2019 a las 14:46, Arnout Vandecappelle (<[email protected]>) escribi?:
Quoted 53 lines · Thomas Petazzoni
Quoted 37 lines
Hello,
Quoted 11 lines · On Sun, 17 Mar 2019 15:32:15 +0100 "Yann E. MORIN" <[email protected]>…
As discussed with Thomas P., it all depends on the semantics we want to give to source-check. But the one implemented by your series does not seem that interesting in the end. Yes, it makes your remote-worker-with- a-slow-connection maybe a bit more bearable.
It also means that all it says is that, right at the time of source-check, here is actually "something" named as you expect, but there is no guarantee that a 'make source' later will be able to get that "something" anyway.
So, I am still not entirely convinced by the usefulness of source-check.
Just to add to Yann's explanations, here are the two aspects that makes the proposed semantic of "source-check" a bit weak:
- It only says "something" was available at the specified download location, but not that this something matches the hash we have in the package hash file. Hence a successful "make source-check" does not guarantee a successful "make source".
- It only says at the time of "make source-check" that "something" was available at the specified download location, but 2 minutes later the file could be removed upstream, the tag be dropped, etc. Hence a successful "make source-check" does not guarantee a successful "make source".
So, a "make source-check" does not provide you much guarantees. It only tells you that at the time of the source-check, "something" was available at the specified download location, but you don't know if that "something" is correct and that it will still be around when you'll do your build.
Quoted 3 lines
If we are willing to accept this very weak semantic of source-check, then fair enough, but we (and our users) have to be aware of the very limited guarantees that source-check provides.
For the use case of Thomas DS, this weak semantic is perfectly fine. What it
tries to protect against is the common mistake that you add or bump a package and forget to upload the source toPRIMARY_SITE. As explained by Thomas DS, doing a full download is eventually still needed, but takes much longer:
source-check may take in the order of 1 minute while the full download takes 10 minutes. That makes all the difference for immediate feedback to the developer, and it doesn't significantly slow down a good pipeline.
Quoted 6 lines
Indeed, it does not protect against uploading the wrong tarball with the
correct name. It also doesn't protect against forgetting to update the hash file. It also doesn't protect against the package failing to build. It also doesn't protect against bugs in the code which become only apparent at run time. But all of these are checked later in CI, and the source check captures an important (and likely) class of mistakes early on.
Thanks for writing this out. It conveys my thoughts very well.
Quoted 8 lines
It is true that this is a very narrow use case. However, I think it passes two
tests for acceptance:* It does not make Buildroot (much) more complex.
* There is no way to do this with scripting outside of Buildroot.
Quoted 7 lines
Regarding that second point: in fact there would be a way, if we would instead
have a command to print everything that would be downloaded. That would be very similar to patch 1/9, but would remove the need for the other patches in the series. The advantage of this approach is that there could be other uses for printing the list of sources. But that would be back to the drawing board for Thomas DS, so I doubt he's enthusiastic about that option :-) Plus, we have no actual example of an additional use case.
I'm not against exploring this, if you think it is better.
Question is what exactly you would print, e.g. all possible URLs, the main URL, only the one via PRIMARY_SITE (which are in fact two possible ones, either with or without the package directory). And how to describe the 'site method'?
For my own use case, everything is expected on the primary site via scp, except our own Mercurial repositories (which are checked via 'hg' or with an optimization via a special scp URL).
I would thus need a list of tarball names to check via scp, and a list of repositories+revisions.
One could envision an output format with one package per line, each line listing the different possible URLs where the package could be obtained, space-separated.
The external script could then have a logic that checks that at least one URL for each line can be accessed. The URLs could be those that the dl-wrapper receives, i.e. with prefixes like git|git+ssh:// that identify the method.
In the general case, the external script would need to duplicate some logic that Buildroot is already handling. I.e. how to process git, wget, etc. URLs. If I'm the only user of this feature, it's fine with me.
Best regards,
Thomas
Quoted 4 lines
Regards,
Arnout
PATCH v5 1/9 support/download: reintroduce 'source-check' target
In reply to Arnout VandecappelleHello,
Thanks a lot for sharing your feedback in this discussion, it's very useful.
Quoted 8 lines · On Wed, 27 Mar 2019 14:46:32 +0100 Arnout Vandecappelle <[email protected]> wrote:
For the use case of Thomas DS, this weak semantic is perfectly fine. What it
tries to protect against is the common mistake that you add or bump a package and forget to upload the source toPRIMARY_SITE. As explained by Thomas DS, doing a full download is eventually still needed, but takes much longer:
source-check may take in the order of 1 minute while the full download takes 10 minutes. That makes all the difference for immediate feedback to the developer, and it doesn't significantly slow down a good pipeline.
Quoted 6 lines
Indeed, it does not protect against uploading the wrong tarball with the
correct name. It also doesn't protect against forgetting to update the hash file. It also doesn't protect against the package failing to build. It also doesn't protect against bugs in the code which become only apparent at run time. But all of these are checked later in CI, and the source check captures an important (and likely) class of mistakes early on.
Interesting perspective indeed :-)
Quoted 6 lines
It is true that this is a very narrow use case. However, I think it passes two
tests for acceptance:* It does not make Buildroot (much) more complex.
* There is no way to do this with scripting outside of Buildroot.
True.
Quoted 7 lines
Regarding that second point: in fact there would be a way, if we would instead
have a command to print everything that would be downloaded. That would be very similar to patch 1/9, but would remove the need for the other patches in the series. The advantage of this approach is that there could be other uses for printing the list of sources. But that would be back to the drawing board for Thomas DS, so I doubt he's enthusiastic about that option :-) Plus, we have no actual example of an additional use case.
We already have "make external-deps" but it only prints the file names, not the full URL.
Does Thomas really want to check the BR2_PRIMARY_SITE or the original Mercurial repository ? If he wants to check BR2_PRIMARY_SITE (which contains only tarballs), then he could run "make external-deps" and checks that the tarballs are here. But I suppose that's not what Thomas
wants: he really wants to check the upstream Mercurial repository, no?
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v5 1/9 support/download: reintroduce 'source-check' target
Quoted 42 lines · Thomas Petazzoni
Hello,
Thanks a lot for sharing your feedback in this discussion, it's very useful.
Quoted 8 lines · On Wed, 27 Mar 2019 14:46:32 +0100 Arnout Vandecappelle <[email protected]> wrote:
For the use case of Thomas DS, this weak semantic is perfectly fine. What it
tries to protect against is the common mistake that you add or bump a package and forget to upload the source toPRIMARY_SITE. As explained by Thomas DS, doing a full download is eventually still needed, but takes much longer:
source-check may take in the order of 1 minute while the full download takes 10 minutes. That makes all the difference for immediate feedback to the developer, and it doesn't significantly slow down a good pipeline.Quoted 6 lines
Indeed, it does not protect against uploading the wrong tarball with the
correct name. It also doesn't protect against forgetting to update the hash file. It also doesn't protect against the package failing to build. It also doesn't protect against bugs in the code which become only apparent at run time. But all of these are checked later in CI, and the source check captures an important (and likely) class of mistakes early on.Interesting perspective indeed :-)
Quoted 6 lines
It is true that this is a very narrow use case. However, I think it passes two
tests for acceptance:* It does not make Buildroot (much) more complex.
* There is no way to do this with scripting outside of Buildroot.
True.
Quoted 7 lines
Regarding that second point: in fact there would be a way, if we would instead
have a command to print everything that would be downloaded. That would be very similar to patch 1/9, but would remove the need for the other patches in the series. The advantage of this approach is that there could be other uses for printing the list of sources. But that would be back to the drawing board for Thomas DS, so I doubt he's enthusiastic about that option :-) Plus, we have no actual example of an additional use case.
Quoted 2 lines
We already have "make external-deps" but it only prints the file names, not the full URL.
True! So, we could extend external-deps with printing the URL (similar to how
printvars uses RAW_VARS maybe, to not break existing scripts using external-deps). We could have variables to ask for primary, upstream or secondary.
Regards,
Arnout
Quoted 10 lines
Does Thomas really want to check the
BR2_PRIMARY_SITEor the original Mercurial repository ? If he wants to checkBR2_PRIMARY_SITE(which contains only tarballs), then he could run "make external-deps" and checks that the tarballs are here. But I suppose that's not what Thomas
wants: he really wants to check the upstream Mercurial repository, no?Best regards,
Thomas
PATCH v5 1/9 support/download: reintroduce 'source-check' target
Hi Thomas DS,
Quoted 3 lines · Arnout Vandecappelle
On 27/03/2019 17:35, Thomas Petazzoni wrote:
[snip]
Quoted 6 lines
Quoted 2 lines
We already have "make external-deps" but it only prints the file names, not the full URL.
True! So, we could extend external-deps with printing the URL (similar to how
printvars usesRAW_VARSmaybe, to not break existing scripts using external-deps). We could have variables to ask for primary, upstream or secondary.
With Yann's show-info series on its way in, can we consider this series as
Rejected?
Thanks for the patches anyway, it was the push needed to get Yann to implement
the Right Thing (tm) :-)
Regards,
Arnout
PATCH v5 1/9 support/download: reintroduce 'source-check' target
Hi Arnout,
Quoted 14 lines · On Sat, Apr 13, 2019, 17:24 Arnout Vandecappelle <[email protected]> wrote:
Hi Thomas DS,
Quoted 3 lines · Arnout Vandecappelle
On 27/03/2019 17:35, Thomas Petazzoni wrote:
[snip]
Quoted 4 lines
Quoted 2 lines
We already have "make external-deps" but it only prints the file names, not the full URL.
True! So, we could extend external-deps with printing the URL (similar
to how
Quoted 2 lines
printvars uses
RAW_VARSmaybe, to not break existing scripts using external-deps). We could have variables to ask for primary, upstream or
Quoted 9 lines
secondary.
With Yann's show-info series on its way in, can we consider this series as
Rejected?Thanks for the patches anyway, it was the push needed to get Yann to
implement
the Right Thing (tm) :-)
Yes, I agree to drop these patches and use show-info instead.
Thanks all for the discussion and alternative solution.
Best regards
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190413/c5fd7242/attachment.html>