Skip to contentSkip to conversation

support/download: reintroduce 'source-check' target

PATCH New Thomas De Schampheleire 4 participants 17 messages Feb 19, 2019 – Apr 13, 2019 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/9 support/download: reintroduce 'source-check' target

From: Thomas De Schampheleire <[email protected]>

Commit bf28a165d992681a85b43a886005e669b3cb579b removed the source-check target to allow easier refactoring of the download infrastructure. It was partly based on the fact that no-one really used this target.

However, it turns out that there _are_ good uses for it. In a work environment where many people are submitting changes to a Buildroot repository, one cannot always blindly trust every change. A typical case of human error is when bumping the version of a package but forgetting to upload the tarball to the internal BR2_PRIMARY_SITE or (in case of a repository) pushing the new changesets to the repository.
If a user cannot directly push to the Buildroot repository but needs to queue their change via an automatic validation system, that validation system can use 'make source-check' on the relevant defconfigs to protect against such errors. A full download would also be possible but takes longer and unnecessarily uses internal network bandwidth.

With that use case in mind, this commit reintroduces the 'source-check' target, but embedded in the current situation with a dl-wrapper. The changes to the wrapper are minimal (not considering additional indentation). A new option '-C' means 'check only' and will be passed to the download backends. If the backend supports the option, no download will happen. If it does not, then the backend will actually perform a download as a means of checking that the source exists (a form of graceful degradation). In neither case, though, hash checking is performed (as the standard case is without download thus without file to calculate hashes on).

Subsequent commits will actually implement -C in the backends.

---
 Makefile                    |   7 +++
 package/pkg-download.mk     |   6 +++
 package/pkg-generic.mk      |  14 ++++-
 support/download/dl-wrapper | 100 ++++++++++++++++++++----------------
 4 files changed, 82 insertions(+), 45 deletions(-)
Makefile +7 -0
index f736ecfb3e..de1ffa9d0e 100644
@@ -131,6 +131,7 @@ noconfig_targets := menuconfig nconfig gconfig xconfig config oldconfig randconf
 # (default target is to build), or when MAKECMDGOALS contains
 # something else than one of the nobuild_targets.
 nobuild_targets := source %-source \
+	source-check %-source-check %-all-source-check \
 	legal-info %-legal-info external-deps _external-deps \
 	clean distclean help show-targets graph-depends \
 	%-graph-depends %-show-depends %-show-version \
@@ -818,6 +819,9 @@ target-post-image: $(TARGETS_ROOTFS) target-finalize staging-finalize
 .PHONY: source
 source: $(foreach p,$(PACKAGES),$(p)-all-source)
 
+.PHONY: source-check
+source-check: $(foreach p,$(PACKAGES),$(p)-all-source-check)
+
 .PHONY: _external-deps external-deps
 _external-deps: $(foreach p,$(PACKAGES),$(p)-all-external-deps)
 external-deps:
@@ -1103,6 +1107,8 @@ help:
 	@echo '  <pkg>-dirclean         - Remove <pkg> build directory'
 	@echo '  <pkg>-reconfigure      - Restart the build from the configure step'
 	@echo '  <pkg>-rebuild          - Restart the build from the build step'
+	@echo '  <pkg>-source-check     - Check package for valid download URLs'
+	@echo '  <pkg>-all-source-check - Check package and its dependencies for valid download URLs'
 	$(foreach p,$(HELP_PACKAGES), \
 		@echo $(sep) \
 		@echo '$($(p)_NAME):' $(sep) \
@@ -1122,6 +1128,7 @@ help:
 	@echo
 	@echo 'Miscellaneous:'
 	@echo '  source                 - download all sources needed for offline-build'
+	@echo '  source-check           - check selected packages for valid download URLs'
 	@echo '  external-deps          - list external packages used'
 	@echo '  legal-info             - generate info about license compliance'
 	@echo '  printvars              - dump all the internal variables'
package/pkg-download.mk +6 -0
index 7cd87c38ff..5384fc2af4 100644
@@ -70,6 +70,7 @@ export BR_NO_CHECK_HASH_FOR =
 # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
 #
 # Argument 1 is the source location
+# Argument 2 (optionally) provides extra arguments to pass to DL_WRAPPER
 #
 ################################################################################
 
@@ -92,6 +93,7 @@ endif
 define DOWNLOAD
 	$(Q)mkdir -p $($(PKG)_DL_DIR)
 	$(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
+		$(2) \
 		-c '$($(PKG)_DL_VERSION)' \
 		-d '$($(PKG)_DL_DIR)' \
 		-D '$(DL_DIR)' \
@@ -106,3 +108,7 @@ define DOWNLOAD
 		-- \
 		$($(PKG)_DL_OPTS)
 endef
+
+define SOURCE_CHECK
+	$(call DOWNLOAD,$(1),-C)
+endef
package/pkg-generic.mk +13 -1
index 6168b40e89..bc57956867 100644
@@ -788,6 +788,10 @@ $(1)-legal-source:	$$($(2)_TARGET_ACTUAL_SOURCE)
 endif # actual sources != sources
 endif # actual sources != ""
 
+$(1)-source-check: PKG=$(2)
+$(1)-source-check:
+	$$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep))
+
 $(1)-external-deps:
 	@for p in $$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS) ; do \
 		echo `basename $$$$p` ; \
@@ -812,6 +816,9 @@ $(1)-rsync:		$$($(2)_TARGET_RSYNC)
 $(1)-source:
 $(1)-legal-source:
 
+$(1)-source-check:
+	test -d $$($(2)_OVERRIDE_SRCDIR)
+
 $(1)-external-deps:
 	@echo "file://$$($(2)_OVERRIDE_SRCDIR)"
 endif
@@ -846,6 +853,9 @@ $(1)-graph-rdepends: graph-depends-requirements
 $(1)-all-source:	$(1)-source
 $(1)-all-source:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source)
 
+$(1)-all-source-check:	$(1)-source-check
+$(1)-all-source-check:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source-check)
+
 $(1)-all-external-deps:	$(1)-external-deps
 $(1)-all-external-deps:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-external-deps)
 
@@ -1044,6 +1054,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
 	$(1)-all-external-deps \
 	$(1)-all-legal-info \
 	$(1)-all-source \
+	$(1)-all-source-check \
 	$(1)-build \
 	$(1)-clean-for-rebuild \
 	$(1)-clean-for-reconfigure \
@@ -1068,7 +1079,8 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
 	$(1)-rsync \
 	$(1)-show-depends \
 	$(1)-show-version \
-	$(1)-source
+	$(1)-source \
+	$(1)-source-check
 
 ifneq ($$($(2)_SOURCE),)
 ifeq ($$($(2)_SITE),)
support/download/dl-wrapper +56 -44

142 lines not on this page · raw message

Signature · 1 line
2.19.2
#1 lore raw

PATCH v5 2/9 support/download/hg: implement source-check

From: Thomas De Schampheleire <[email protected]>

Note: the implementation is different (better) than what used to be in Buildroot before source-check was removed.

The original implementation:

    hg incoming --force -l1 <URL>

would only verify that the repository exists, not that the requested revision is present.

An already better implementation is:

    hg incoming --force -l1 -r <revision> <URL>

but compared to the next solution it has a large resource consumption on the local machine. In the background, the full repository is first downloaded.

The implemented solution is:
hg identify -r <revision> <URL>
which operates directly on the remote repository.

---
 support/download/hg | 12 ++++++++++++
 1 file changed, 12 insertions(+)

v5: no changes

v4: (feedback Yann E. Morin)
- use true/false as values to 'checkonly'
- always check revision, even if no source-check
- replace incorrect 'exit $?' by explicit 'exit 0'

v3: no changes

support/download/hg +12 -0
index efb515fca5..8f7d1a5a90 100755
@@ -7,6 +7,7 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the changeset exists in the remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Clone from repository at URI.
 #   -c CSET     Use changeset (or revision) CSET.
@@ -16,9 +17,11 @@ set -e
 #   HG       : the hg command to call
 
 verbose=
+checkonly=false
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=true;;
     o)  output="${OPTARG}";;
     u)  uri="${OPTARG}";;
     c)  cset="${OPTARG}";;
@@ -36,6 +39,15 @@ _hg() {
     eval ${HG} "${@}"
 }
 
+if ! _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null; then
+    printf "Commit '%s' does not exist in this repository\n." "${cset}"
+    exit 1
+fi
+
+if ${checkonly}; then
+    exit 0
+fi
+
 _hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'"
 
 _hg archive ${verbose} --repository "'${basename}'" --type tgz \
Signature · 1 line
2.19.2
#2 lore raw

PATCH v5 3/9 support/download/wget: implement source-check

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

---
 support/download/wget | 8 ++++++++
 1 file changed, 8 insertions(+)

v5: no changes

v4: (feedback Yann E. Morin)
- use true/false as values to 'checkonly'
- replace incorrect 'exit $?' by explicit 'exit 0'

v3: no changes

support/download/wget +8 -0
index c69e6071aa..277eab6660 100755
@@ -7,6 +7,7 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the file exists remotely.
 #   -o FILE     Save into file FILE.
 #   -f FILENAME The filename of the tarball to get at URL
 #   -u URL      Download file at URL.
@@ -16,9 +17,11 @@ set -e
 #   WGET     : the wget command to call
 
 verbose=
+checkonly=false
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=true;;
     o)  output="${OPTARG}";;
     f)  filename="${OPTARG}";;
     u)  url="${OPTARG}";;
@@ -40,4 +43,9 @@ _wget() {
 # mirror
 [ -n "${encode}" ] && filename=${filename//\?/%3F}
 
+if ${checkonly}; then
+    _wget --spider ${verbose} "${@}" "'${url}/${filename}'"
+    exit 0
+fi
+
 _wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
Signature · 1 line
2.19.2
#3 lore raw

PATCH v5 4/9 support/download/file: implement source-check

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

---
 support/download/file | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

v5: no changes

v4:
- use true/false as values to 'checkonly' (Yann E. Morin)
- replace incorrect 'exit $?' by explicit 'exit 0' (Yann E. Morin)
- replace 'test -e' by 'test -f' which is more specific

v3: no changes

support/download/file +12 -1
index e52fcf2c8c..059a8dde4c 100755
@@ -7,6 +7,7 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the source file exists.
 #   -o FILE     Copy to file FILE.
 #   -f FILE     Copy from basename file FILE.
 #   -u DIR      Copy from FILE in DIR.
@@ -20,9 +21,11 @@ set -e
 # Make 'cp' verbose by default, so it behaves a bit like the others.
 verbose=-v
 
+checkonly=false
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=;;
+    C)  checkonly=true;;
     o)  output="${OPTARG}";;
     f)  file="${OPTARG}";;
     u)  dir="${OPTARG}";;
@@ -39,4 +42,12 @@ _localfiles() {
     eval ${LOCALFILES} "${@}"
 }
 
-_localfiles ${verbose} "'${dir##file://}/${file}'" "'${output}'"
+# Remove any scheme prefix
+dir="${dir##file://}"
+
+if ${checkonly}; then
+    test -f "'${dir}/${file}'"
+    exit 0
+fi
+
+_localfiles ${verbose} "'${dir}/${file}'" "'${output}'"
Signature · 1 line
2.19.2
#4 lore raw

PATCH v5 5/9 Revert "core/download: drop the SSH command"

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

This reverts commit db9473bf6cd7bd12aa1f9faad0a917c973c33827.

The SSH command will again be needed to support source-check for the 'scp' download backend.

---
 Config.in               | 4 ++++
 package/pkg-download.mk | 1 +
 2 files changed, 5 insertions(+)

v5: use 'git revert' instead
v4: no changes
v3: no changes

Config.in +4 -0
index d58d8dc04a..91d56907fe 100644
@@ -136,6 +136,10 @@ config BR2_SCP
 	string "Secure copy (scp) command"
 	default "scp"
 
+config BR2_SSH
+	string "Secure shell (ssh) command"
+	default "ssh"
+
 config BR2_HG
 	string "Mercurial (hg) command"
 	default "hg"
package/pkg-download.mk +1 -0
index 5384fc2af4..dfc4460c6a 100644
@@ -15,6 +15,7 @@ export BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
 export HG := $(call qstrip,$(BR2_HG))
 export SCP := $(call qstrip,$(BR2_SCP))
+SSH := $(call qstrip,$(BR2_SSH))
 export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
 DL_WRAPPER = support/download/dl-wrapper
Signature · 1 line
2.19.2
#5 lore raw

PATCH v5 6/9 package/pkg-download: export 'SSH' for use in the download backends

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

Originally, the tools defined in pkg-download.mk were used from Makefiles and were directly available. Due to the introduction of the download infrastructure in shell scripts, these variables now need to be exported in order for the scripts to see them.

---
 package/pkg-download.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

v5: new patch

package/pkg-download.mk +1 -1
index dfc4460c6a..17d1ca99ee 100644
@@ -15,7 +15,7 @@ export BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
 export HG := $(call qstrip,$(BR2_HG))
 export SCP := $(call qstrip,$(BR2_SCP))
-SSH := $(call qstrip,$(BR2_SSH))
+export SSH := $(call qstrip,$(BR2_SSH))
 export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
 DL_WRAPPER = support/download/dl-wrapper
Signature · 1 line
2.19.2
#6 lore raw

PATCH v5 7/9 support/download/scp: implement source-check

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

We use 'test -f' rather than 'ls' because 'test' is a mandatory POSIX utility while 'ls' is not.

---
 support/download/scp | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

v5: no changes

v4: (feedback Yann E. Morin)
- use true/false as values to 'checkonly'
- replace incorrect 'exit $?' by explicit 'exit 0'
- use 'test -f' instead of 'ls' which is more certain to be present (POSIX mandatory utility)

v3: no changes

support/download/scp +15 -0
index 80cf495c4e..52ccf06d6e 100755
@@ -7,17 +7,21 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the file exists remotely.
 #   -o FILE     Copy to local file FILE.
 #   -f FILE     Copy from remote file FILE.
 #   -u URI      Download file at URI.
 #
 # Environment:
 #   SCP       : the scp command to call
+#   SSH       : the ssh command to use for checkonly
 
 verbose=
+checkonly=false
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=true;;
     o)  output="${OPTARG}";;
     f)  filename="${OPTARG}";;
     u)  uri="${OPTARG}";;
@@ -33,8 +37,19 @@ shift $((OPTIND-1)) # Get rid of our options
 _scp() {
     eval ${SCP} "${@}"
 }
+_ssh() {
+    eval ${SSH} "${@}"
+}
 
 # Remove any scheme prefix
 uri="${uri##scp://}"
 
+if ${checkonly}; then
+    # uri now looks like:  foo.example.org:some/directory
+    domain="${uri%%:*}"
+    path="${uri#*:}/${filename}"
+    _ssh ${verbose} "${@}" "'${domain}'" test -f "'${path}'" > /dev/null
+    exit 0
+fi
+
 _scp ${verbose} "${@}" "'${uri}/${filename}'" "'${output}'"
Signature · 1 line
2.19.2
#7 lore raw

PATCH v5 8/9 support/download/svn: implement source-check

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

---
 support/download/svn | 12 ++++++++++++
 1 file changed, 12 insertions(+)

v5: no changes

v4: new patch

support/download/svn +12 -0
index 542b25c0a2..93984b6c45 100755
@@ -7,6 +7,7 @@ set -e
 #
 # Options:
 #   -q          Be quiet.
+#   -C          Only check that the revision exists in the remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Checkout from repository at URI.
 #   -c REV      Use revision REV.
@@ -16,9 +17,11 @@ set -e
 #   SVN      : the svn command to call
 
 verbose=
+checkonly=false
 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
+    C)  checkonly=true;;
     o)  output="${OPTARG}";;
     u)  uri="${OPTARG}";;
     c)  rev="${OPTARG}";;
@@ -36,6 +39,15 @@ _svn() {
     eval ${SVN} "${@}"
 }
 
+if ! _svn ls ${verbose} -r "'${rev}'" "'${uri}'" > /dev/null; then
+    printf "Revision '%s' does not exist in this repository\n." "${rev}"
+    exit 1
+fi
+
+if ${checkonly}; then
+    exit 0
+fi
+
 _svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'"
 
 tar czf "${output}" "${basename}"
Signature · 1 line
2.19.2
#8 lore raw

PATCH v5 9/9 support/download/{bzr, cvs, git}: highlight unimplemented source-check

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

For bzr, cvs and git, there is no method of checking whether the revision exists remotely, without actually cloning the repository. Therefore, there is no actual implementation for '-C' (checkonly) for these download helpers. The script will fall back to the normal download logic, which would effectively fail if the revision does not exist.

For completeness, mention this unimplemented feature in the usage text.

---
 support/download/bzr | 2 ++
 support/download/cvs | 2 ++
 support/download/git | 2 ++
 3 files changed, 6 insertions(+)

v5: no changes

v4: remove 'svn' which now has an actual implementation

v3: no changes

support/download/bzr +2 -0
index 5289a421cd..278d8778af 100755
@@ -7,6 +7,8 @@ set -e
 #
 # Options:
 #   -q          Be quiet
+#   -C          (unimplemented) Only check that the revision exists in the
+#               remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Clone from repository at URI.
 #   -c CSET     Use changeset (or revision) CSET.
support/download/cvs +2 -0
index 9d0dc3cb3a..d9a586293a 100755
@@ -7,6 +7,8 @@ set -e
 #
 # Options:
 #   -q          Be quiet
+#   -C          (unimplemented) Only check that the revision exists in the
+#               remote repository.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Checkout from repository at URI.
 #   -c REV      Use revision REV.
support/download/git +2 -0
index 17ca04eb98..e9c02d6712 100755
@@ -7,6 +7,8 @@ set -E
 #
 # Options:
 #   -q          Be quiet.
+#   -C          (unimplemented) Only check that the revision exists in the
+#               remote repository.
 #   -r          Clone and archive sub-modules.
 #   -o FILE     Generate archive in FILE.
 #   -u URI      Clone from repository at URI.
Signature · 1 line
2.19.2
#9 lore raw

PATCH v5 1/9 support/download: reintroduce 'source-check' target

In reply to Thomas De Schampheleire

Thomas, All,

On 2019-02-19 11:38 +0100, Thomas De Schampheleire spake thusly:

Quoted 18 lines

From: Thomas De Schampheleire <[email protected]>

Commit bf28a165d992681a85b43a886005e669b3cb579b removed the source-check target to allow easier refactoring of the download infrastructure. It was partly based on the fact that no-one really used this target.

However, it turns out that there _are_ good uses for it. In a work environment where many people are submitting changes to a Buildroot repository, one cannot always blindly trust every change. A typical case of human error is when bumping the version of a package but forgetting to upload the tarball to the internal BR2_PRIMARY_SITE or (in case of a repository) pushing the new changesets to the repository.
If a user cannot directly push to the Buildroot repository but needs to queue their change via an automatic validation system, that validation system can use 'make source-check' on the relevant defconfigs to protect against such errors. A full download would also be possible but takes longer and unnecessarily uses internal network bandwidth.

Quoted 9 lines

With that use case in mind, this commit reintroduces the 'source-check' target, but embedded in the current situation with a dl-wrapper. The changes to the wrapper are minimal (not considering additional indentation). A new option '-C' means 'check only' and will be passed to the download backends. If the backend supports the option, no download will happen. If it does not, then the backend will actually perform a download as a means of checking that the source exists (a form of graceful degradation). In neither case, though, hash checking is performed (as the standard case is without download thus without file to calculate hashes on).

So, that last part made me wonder: in the wget case, if the HEAD succeeds, how does that guarantee that the resource does actually correspond to what you expected?

What if upstream moved the tarball away, but the server is now replying garbaged 404-that-is-not-404 so HEAD succeeds?

Or what if the server does not accept HEAD requests?

Or, in the case of git, what if the tag still exists but has re-tagged?

In short, source-check does not guarantee that what exist "there" is what you do expect to get.

As discussed with Thomas P., it all depends on the semantics we want to give to source-check. But the one implemented by your series does not seem that interesting in the end. Yes, it makes your remote-worker-with- a-slow-connection maybe a bit more bearable.

It also means that all it says is that, right at the time of source-check, here is actually "something" named as you expect, but there is no guarantee that a 'make source' later will be able to get that "something" anyway.

So, I am still not entirely convinced by the usefulness of source-check.

Regards,
Yann E. MORIN.

Quoted 285 lines

Subsequent commits will actually implement -C in the backends.

---
 Makefile                    |   7 +++
 package/pkg-download.mk     |   6 +++
 package/pkg-generic.mk      |  14 ++++-
 support/download/dl-wrapper | 100 ++++++++++++++++++++----------------
 4 files changed, 82 insertions(+), 45 deletions(-)
Makefile +7 -0
index f736ecfb3e..de1ffa9d0e 100644
@@ -131,6 +131,7 @@ noconfig_targets := menuconfig nconfig gconfig xconfig config oldconfig randconf
 # (default target is to build), or when MAKECMDGOALS contains
 # something else than one of the nobuild_targets.
 nobuild_targets := source %-source \
+	source-check %-source-check %-all-source-check \
 	legal-info %-legal-info external-deps _external-deps \
 	clean distclean help show-targets graph-depends \
 	%-graph-depends %-show-depends %-show-version \
@@ -818,6 +819,9 @@ target-post-image: $(TARGETS_ROOTFS) target-finalize staging-finalize
 .PHONY: source
 source: $(foreach p,$(PACKAGES),$(p)-all-source)
 
+.PHONY: source-check
+source-check: $(foreach p,$(PACKAGES),$(p)-all-source-check)
+
 .PHONY: _external-deps external-deps
 _external-deps: $(foreach p,$(PACKAGES),$(p)-all-external-deps)
 external-deps:
@@ -1103,6 +1107,8 @@ help:
 	@echo '  <pkg>-dirclean         - Remove <pkg> build directory'
 	@echo '  <pkg>-reconfigure      - Restart the build from the configure step'
 	@echo '  <pkg>-rebuild          - Restart the build from the build step'
+	@echo '  <pkg>-source-check     - Check package for valid download URLs'
+	@echo '  <pkg>-all-source-check - Check package and its dependencies for valid download URLs'
 	$(foreach p,$(HELP_PACKAGES), \
 		@echo $(sep) \
 		@echo '$($(p)_NAME):' $(sep) \
@@ -1122,6 +1128,7 @@ help:
 	@echo
 	@echo 'Miscellaneous:'
 	@echo '  source                 - download all sources needed for offline-build'
+	@echo '  source-check           - check selected packages for valid download URLs'
 	@echo '  external-deps          - list external packages used'
 	@echo '  legal-info             - generate info about license compliance'
 	@echo '  printvars              - dump all the internal variables'
package/pkg-download.mk +6 -0
index 7cd87c38ff..5384fc2af4 100644
@@ -70,6 +70,7 @@ export BR_NO_CHECK_HASH_FOR =
 # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
 #
 # Argument 1 is the source location
+# Argument 2 (optionally) provides extra arguments to pass to DL_WRAPPER
 #
 ################################################################################
 
@@ -92,6 +93,7 @@ endif
 define DOWNLOAD
 	$(Q)mkdir -p $($(PKG)_DL_DIR)
 	$(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
+		$(2) \
 		-c '$($(PKG)_DL_VERSION)' \
 		-d '$($(PKG)_DL_DIR)' \
 		-D '$(DL_DIR)' \
@@ -106,3 +108,7 @@ define DOWNLOAD
 		-- \
 		$($(PKG)_DL_OPTS)
 endef
+
+define SOURCE_CHECK
+	$(call DOWNLOAD,$(1),-C)
+endef
package/pkg-generic.mk +13 -1
index 6168b40e89..bc57956867 100644
@@ -788,6 +788,10 @@ $(1)-legal-source:	$$($(2)_TARGET_ACTUAL_SOURCE)
 endif # actual sources != sources
 endif # actual sources != ""
 
+$(1)-source-check: PKG=$(2)
+$(1)-source-check:
+	$$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep))
+
 $(1)-external-deps:
 	@for p in $$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS) ; do \
 		echo `basename $$$$p` ; \
@@ -812,6 +816,9 @@ $(1)-rsync:		$$($(2)_TARGET_RSYNC)
 $(1)-source:
 $(1)-legal-source:
 
+$(1)-source-check:
+	test -d $$($(2)_OVERRIDE_SRCDIR)
+
 $(1)-external-deps:
 	@echo "file://$$($(2)_OVERRIDE_SRCDIR)"
 endif
@@ -846,6 +853,9 @@ $(1)-graph-rdepends: graph-depends-requirements
 $(1)-all-source:	$(1)-source
 $(1)-all-source:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source)
 
+$(1)-all-source-check:	$(1)-source-check
+$(1)-all-source-check:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source-check)
+
 $(1)-all-external-deps:	$(1)-external-deps
 $(1)-all-external-deps:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-external-deps)
 
@@ -1044,6 +1054,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
 	$(1)-all-external-deps \
 	$(1)-all-legal-info \
 	$(1)-all-source \
+	$(1)-all-source-check \
 	$(1)-build \
 	$(1)-clean-for-rebuild \
 	$(1)-clean-for-reconfigure \
@@ -1068,7 +1079,8 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
 	$(1)-rsync \
 	$(1)-show-depends \
 	$(1)-show-version \
-	$(1)-source
+	$(1)-source \
+	$(1)-source-check
 
 ifneq ($$($(2)_SOURCE),)
 ifeq ($$($(2)_SITE),)
support/download/dl-wrapper +56 -44
index 3315bd410e..920699e51d 100755
@@ -17,7 +17,7 @@
 # We want to catch any unexpected failure, and exit immediately.
 set -e
 
-export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
+export BR_BACKEND_DL_GETOPTS=":hc:Cd:o:n:N:H:ru:qf:e"
 
 main() {
     local OPT OPTARG
@@ -25,9 +25,10 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    while getopts ":c:Cd:D:o:n:N:H:rf:u:q" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
+        C)  checkonly=-C;;
         d)  dl_dir="${OPTARG}";;
         D)  old_dl_dir="${OPTARG}";;
         o)  output="${OPTARG}";;
@@ -46,38 +47,40 @@ main() {
     # Forget our options, and keep only those for the backend
     shift $((OPTIND-1))
 
-    if [ -z "${output}" ]; then
-        error "no output specified, use -o\n"
-    fi
+    if [ -z "${checkonly}" ]; then
+        if [ -z "${output}" ]; then
+            error "no output specified, use -o\n"
+        fi
 
-    # Legacy handling: check if the file already exists in the global
-    # download directory. If it does, hard-link it. If it turns out it
-    # was an incorrect download, we'd still check it below anyway.
-    # If we can neither link nor copy, fallback to doing a download.
-    # NOTE! This is not atomic, is subject to TOCTTOU, but the whole
-    # dl-wrapper runs under an flock, so we're safe.
-    if [ ! -e "${output}" -a -e "${old_dl_dir}/${filename}" ]; then
-        ln "${old_dl_dir}/${filename}" "${output}" || \
-        cp "${old_dl_dir}/${filename}" "${output}" || \
-        true
-    fi
+        # Legacy handling: check if the file already exists in the global
+        # download directory. If it does, hard-link it. If it turns out it
+        # was an incorrect download, we'd still check it below anyway.
+        # If we can neither link nor copy, fallback to doing a download.
+        # NOTE! This is not atomic, is subject to TOCTTOU, but the whole
+        # dl-wrapper runs under an flock, so we're safe.
+        if [ ! -e "${output}" -a -e "${old_dl_dir}/${filename}" ]; then
+            ln "${old_dl_dir}/${filename}" "${output}" || \
+            cp "${old_dl_dir}/${filename}" "${output}" || \
+            true
+        fi
 
-    # If the output file already exists and:
-    # - there's no .hash file: do not download it again and exit promptly
-    # - matches all its hashes: do not download it again and exit promptly
-    # - fails at least one of its hashes: force a re-download
-    # - there's no hash (but a .hash file): consider it a hard error
-    if [ -e "${output}" ]; then
-        if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
-            exit 0
-        elif [ ${?} -ne 2 ]; then
-            # Do not remove the file, otherwise it might get re-downloaded
-            # from a later location (i.e. primary -> upstream -> mirror).
-            # Do not print a message, check-hash already did.
-            exit 1
+        # If the output file already exists and:
+        # - there's no .hash file: do not download it again and exit promptly
+        # - matches all its hashes: do not download it again and exit promptly
+        # - fails at least one of its hashes: force a re-download
+        # - there's no hash (but a .hash file): consider it a hard error
+        if [ -e "${output}" ]; then
+            if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
+                exit 0
+            elif [ ${?} -ne 2 ]; then
+                # Do not remove the file, otherwise it might get re-downloaded
+                # from a later location (i.e. primary -> upstream -> mirror).
+                # Do not print a message, check-hash already did.
+                exit 1
+            fi
+            rm -f "${output}"
+            warn "Re-downloading '%s'...\n" "${output##*/}"
         fi
-        rm -f "${output}"
-        warn "Re-downloading '%s'...\n" "${output##*/}"
     fi
 
     # Look through all the uris that we were given to download the package
@@ -127,7 +130,7 @@ main() {
                 -f "${filename}" \
                 -u "${uri}" \
                 -o "${tmpf}" \
-                ${quiet} ${recurse} -- "${@}"
+                ${quiet} ${recurse} ${checkonly} -- "${@}"
         then
             # cd back to keep path coherence
             cd "${OLDPWD}"
@@ -138,19 +141,21 @@ main() {
         # cd back to free the temp-dir, so we can remove it later
         cd "${OLDPWD}"
 
-        # Check if the downloaded file is sane, and matches the stored hashes
-        # for that file
-        if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
-            rc=0
-        else
-            if [ ${?} -ne 3 ]; then
-                rm -rf "${tmpd}"
-                continue
+        if [ -z "${checkonly}" ]; then
+            # Check if the downloaded file is sane, and matches the stored hashes
+            # for that file
+            if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
+                rc=0
+            else
+                if [ ${?} -ne 3 ]; then
+                    rm -rf "${tmpd}"
+                    continue
+                fi
+
+                # the hash file exists and there was no hash to check the file
+                # against
+                rc=1
             fi
-
-            # the hash file exists and there was no hash to check the file
-            # against
-            rc=1
         fi
         download_and_check=1
         break
@@ -163,6 +168,13 @@ main() {
         exit 1
     fi
 
+    # If we only need to check the presence of sources, stop here.
+    # No need to handle output files.
+    if [ -n "${checkonly}" ]; then
+        rm -rf "${tmpd}"
+        exit 0
+    fi
+
     # tmp_output is in the same directory as the final output, so we can
     # later move it atomically.
     tmp_output="$(mktemp "${output}.XXXXXX")"
Signature · 1 line
2.19.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#10 lore raw

PATCH v5 1/9 support/download: reintroduce 'source-check' target

Hello,

Quoted 11 lines · On Sun, 17 Mar 2019 15:32:15 +0100 "Yann E. MORIN" <[email protected]>…

As discussed with Thomas P., it all depends on the semantics we want to give to source-check. But the one implemented by your series does not seem that interesting in the end. Yes, it makes your remote-worker-with- a-slow-connection maybe a bit more bearable.

It also means that all it says is that, right at the time of source-check, here is actually "something" named as you expect, but there is no guarantee that a 'make source' later will be able to get that "something" anyway.

So, I am still not entirely convinced by the usefulness of source-check.

Just to add to Yann's explanations, here are the two aspects that makes the proposed semantic of "source-check" a bit weak:

- It only says "something" was available at the specified download location, but not that this something matches the hash we have in the package hash file. Hence a successful "make source-check" does not guarantee a successful "make source".

- It only says at the time of "make source-check" that "something" was available at the specified download location, but 2 minutes later the file could be removed upstream, the tag be dropped, etc. Hence a successful "make source-check" does not guarantee a successful "make source".

So, a "make source-check" does not provide you much guarantees. It only tells you that at the time of the source-check, "something" was available at the specified download location, but you don't know if that "something" is correct and that it will still be around when you'll do your build.

If we are willing to accept this very weak semantic of source-check, then fair enough, but we (and our users) have to be aware of the very limited guarantees that source-check provides.

What do you think?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#11 lore raw

PATCH v5 1/9 support/download: reintroduce 'source-check' target

Quoted 37 lines · Thomas Petazzoni

Hello,

Quoted 11 lines · On Sun, 17 Mar 2019 15:32:15 +0100 "Yann E. MORIN" <[email protected]>…

As discussed with Thomas P., it all depends on the semantics we want to give to source-check. But the one implemented by your series does not seem that interesting in the end. Yes, it makes your remote-worker-with- a-slow-connection maybe a bit more bearable.

It also means that all it says is that, right at the time of source-check, here is actually "something" named as you expect, but there is no guarantee that a 'make source' later will be able to get that "something" anyway.

So, I am still not entirely convinced by the usefulness of source-check.

Just to add to Yann's explanations, here are the two aspects that makes the proposed semantic of "source-check" a bit weak:

- It only says "something" was available at the specified download location, but not that this something matches the hash we have in the package hash file. Hence a successful "make source-check" does not guarantee a successful "make source".

- It only says at the time of "make source-check" that "something" was available at the specified download location, but 2 minutes later the file could be removed upstream, the tag be dropped, etc. Hence a successful "make source-check" does not guarantee a successful "make source".

So, a "make source-check" does not provide you much guarantees. It only tells you that at the time of the source-check, "something" was available at the specified download location, but you don't know if that "something" is correct and that it will still be around when you'll do your build.

Quoted 3 lines

If we are willing to accept this very weak semantic of source-check, then fair enough, but we (and our users) have to be aware of the very limited guarantees that source-check provides.

For the use case of Thomas DS, this weak semantic is perfectly fine. What it
tries to protect against is the common mistake that you add or bump a package and forget to upload the source to PRIMARY_SITE. As explained by Thomas DS, doing a full download is eventually still needed, but takes much longer:
source-check may take in the order of 1 minute while the full download takes 10 minutes. That makes all the difference for immediate feedback to the developer, and it doesn't significantly slow down a good pipeline.

Indeed, it does not protect against uploading the wrong tarball with the
correct name. It also doesn't protect against forgetting to update the hash file. It also doesn't protect against the package failing to build. It also doesn't protect against bugs in the code which become only apparent at run time. But all of these are checked later in CI, and the source check captures an important (and likely) class of mistakes early on.

It is true that this is a very narrow use case. However, I think it passes two
tests for acceptance:

* It does not make Buildroot (much) more complex.

* There is no way to do this with scripting outside of Buildroot.

Regarding that second point: in fact there would be a way, if we would instead
have a command to print everything that would be downloaded. That would be very similar to patch 1/9, but would remove the need for the other patches in the series. The advantage of this approach is that there could be other uses for printing the list of sources. But that would be back to the drawing board for Thomas DS, so I doubt he's enthusiastic about that option :-) Plus, we have no actual example of an additional use case.

Regards,
Arnout

#12 lore raw

PATCH v5 1/9 support/download: reintroduce 'source-check' target

El mi?., 27 mar. 2019 a las 14:46, Arnout Vandecappelle (<[email protected]>) escribi?:

Quoted 53 lines · Thomas Petazzoni
Quoted 37 lines

Hello,

Quoted 11 lines · On Sun, 17 Mar 2019 15:32:15 +0100 "Yann E. MORIN" <[email protected]>…

As discussed with Thomas P., it all depends on the semantics we want to give to source-check. But the one implemented by your series does not seem that interesting in the end. Yes, it makes your remote-worker-with- a-slow-connection maybe a bit more bearable.

It also means that all it says is that, right at the time of source-check, here is actually "something" named as you expect, but there is no guarantee that a 'make source' later will be able to get that "something" anyway.

So, I am still not entirely convinced by the usefulness of source-check.

Just to add to Yann's explanations, here are the two aspects that makes the proposed semantic of "source-check" a bit weak:

- It only says "something" was available at the specified download location, but not that this something matches the hash we have in the package hash file. Hence a successful "make source-check" does not guarantee a successful "make source".

- It only says at the time of "make source-check" that "something" was available at the specified download location, but 2 minutes later the file could be removed upstream, the tag be dropped, etc. Hence a successful "make source-check" does not guarantee a successful "make source".

So, a "make source-check" does not provide you much guarantees. It only tells you that at the time of the source-check, "something" was available at the specified download location, but you don't know if that "something" is correct and that it will still be around when you'll do your build.

Quoted 3 lines

If we are willing to accept this very weak semantic of source-check, then fair enough, but we (and our users) have to be aware of the very limited guarantees that source-check provides.

For the use case of Thomas DS, this weak semantic is perfectly fine. What it
tries to protect against is the common mistake that you add or bump a package and forget to upload the source to PRIMARY_SITE. As explained by Thomas DS, doing a full download is eventually still needed, but takes much longer:
source-check may take in the order of 1 minute while the full download takes 10 minutes. That makes all the difference for immediate feedback to the developer, and it doesn't significantly slow down a good pipeline.

Quoted 6 lines

Indeed, it does not protect against uploading the wrong tarball with the
correct name. It also doesn't protect against forgetting to update the hash file. It also doesn't protect against the package failing to build. It also doesn't protect against bugs in the code which become only apparent at run time. But all of these are checked later in CI, and the source check captures an important (and likely) class of mistakes early on.

Thanks for writing this out. It conveys my thoughts very well.

Quoted 8 lines

It is true that this is a very narrow use case. However, I think it passes two
tests for acceptance:

* It does not make Buildroot (much) more complex.

* There is no way to do this with scripting outside of Buildroot.

Quoted 7 lines

Regarding that second point: in fact there would be a way, if we would instead
have a command to print everything that would be downloaded. That would be very similar to patch 1/9, but would remove the need for the other patches in the series. The advantage of this approach is that there could be other uses for printing the list of sources. But that would be back to the drawing board for Thomas DS, so I doubt he's enthusiastic about that option :-) Plus, we have no actual example of an additional use case.

I'm not against exploring this, if you think it is better.

Question is what exactly you would print, e.g. all possible URLs, the main URL, only the one via PRIMARY_SITE (which are in fact two possible ones, either with or without the package directory). And how to describe the 'site method'?

For my own use case, everything is expected on the primary site via scp, except our own Mercurial repositories (which are checked via 'hg' or with an optimization via a special scp URL).
I would thus need a list of tarball names to check via scp, and a list of repositories+revisions.

One could envision an output format with one package per line, each line listing the different possible URLs where the package could be obtained, space-separated.
The external script could then have a logic that checks that at least one URL for each line can be accessed. The URLs could be those that the dl-wrapper receives, i.e. with prefixes like git|git+ssh:// that identify the method.

In the general case, the external script would need to duplicate some logic that Buildroot is already handling. I.e. how to process git, wget, etc. URLs. If I'm the only user of this feature, it's fine with me.

Best regards,
Thomas

Quoted 4 lines

Regards,
Arnout

#13 lore raw

PATCH v5 1/9 support/download: reintroduce 'source-check' target

In reply to Arnout Vandecappelle

Hello,

Thanks a lot for sharing your feedback in this discussion, it's very useful.

Quoted 8 lines · On Wed, 27 Mar 2019 14:46:32 +0100 Arnout Vandecappelle <[email protected]> wrote:

For the use case of Thomas DS, this weak semantic is perfectly fine. What it
tries to protect against is the common mistake that you add or bump a package and forget to upload the source to PRIMARY_SITE. As explained by Thomas DS, doing a full download is eventually still needed, but takes much longer:
source-check may take in the order of 1 minute while the full download takes 10 minutes. That makes all the difference for immediate feedback to the developer, and it doesn't significantly slow down a good pipeline.

Quoted 6 lines

Indeed, it does not protect against uploading the wrong tarball with the
correct name. It also doesn't protect against forgetting to update the hash file. It also doesn't protect against the package failing to build. It also doesn't protect against bugs in the code which become only apparent at run time. But all of these are checked later in CI, and the source check captures an important (and likely) class of mistakes early on.

Interesting perspective indeed :-)

Quoted 6 lines

It is true that this is a very narrow use case. However, I think it passes two
tests for acceptance:

* It does not make Buildroot (much) more complex.

* There is no way to do this with scripting outside of Buildroot.

True.

Quoted 7 lines

Regarding that second point: in fact there would be a way, if we would instead
have a command to print everything that would be downloaded. That would be very similar to patch 1/9, but would remove the need for the other patches in the series. The advantage of this approach is that there could be other uses for printing the list of sources. But that would be back to the drawing board for Thomas DS, so I doubt he's enthusiastic about that option :-) Plus, we have no actual example of an additional use case.

We already have "make external-deps" but it only prints the file names, not the full URL.

Does Thomas really want to check the BR2_PRIMARY_SITE or the original Mercurial repository ? If he wants to check BR2_PRIMARY_SITE (which contains only tarballs), then he could run "make external-deps" and checks that the tarballs are here. But I suppose that's not what Thomas
wants: he really wants to check the upstream Mercurial repository, no?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#14 lore raw

PATCH v5 1/9 support/download: reintroduce 'source-check' target

Quoted 42 lines · Thomas Petazzoni

Hello,

Thanks a lot for sharing your feedback in this discussion, it's very useful.

Quoted 8 lines · On Wed, 27 Mar 2019 14:46:32 +0100 Arnout Vandecappelle <[email protected]> wrote:

For the use case of Thomas DS, this weak semantic is perfectly fine. What it
tries to protect against is the common mistake that you add or bump a package and forget to upload the source to PRIMARY_SITE. As explained by Thomas DS, doing a full download is eventually still needed, but takes much longer:
source-check may take in the order of 1 minute while the full download takes 10 minutes. That makes all the difference for immediate feedback to the developer, and it doesn't significantly slow down a good pipeline.

Quoted 6 lines

Indeed, it does not protect against uploading the wrong tarball with the
correct name. It also doesn't protect against forgetting to update the hash file. It also doesn't protect against the package failing to build. It also doesn't protect against bugs in the code which become only apparent at run time. But all of these are checked later in CI, and the source check captures an important (and likely) class of mistakes early on.

Interesting perspective indeed :-)

Quoted 6 lines

It is true that this is a very narrow use case. However, I think it passes two
tests for acceptance:

* It does not make Buildroot (much) more complex.

* There is no way to do this with scripting outside of Buildroot.

True.

Quoted 7 lines

Regarding that second point: in fact there would be a way, if we would instead
have a command to print everything that would be downloaded. That would be very similar to patch 1/9, but would remove the need for the other patches in the series. The advantage of this approach is that there could be other uses for printing the list of sources. But that would be back to the drawing board for Thomas DS, so I doubt he's enthusiastic about that option :-) Plus, we have no actual example of an additional use case.

Quoted 2 lines

We already have "make external-deps" but it only prints the file names, not the full URL.

True! So, we could extend external-deps with printing the URL (similar to how
printvars uses RAW_VARS maybe, to not break existing scripts using external-deps). We could have variables to ask for primary, upstream or secondary.

Regards,
Arnout

Quoted 10 lines

Does Thomas really want to check the BR2_PRIMARY_SITE or the original Mercurial repository ? If he wants to check BR2_PRIMARY_SITE (which contains only tarballs), then he could run "make external-deps" and checks that the tarballs are here. But I suppose that's not what Thomas
wants: he really wants to check the upstream Mercurial repository, no?

Best regards,

Thomas

#15 lore raw

PATCH v5 1/9 support/download: reintroduce 'source-check' target

Hi Thomas DS,

Quoted 3 lines · Arnout Vandecappelle

On 27/03/2019 17:35, Thomas Petazzoni wrote:

[snip]

Quoted 6 lines
Quoted 2 lines

We already have "make external-deps" but it only prints the file names, not the full URL.

True! So, we could extend external-deps with printing the URL (similar to how
printvars uses RAW_VARS maybe, to not break existing scripts using external-deps). We could have variables to ask for primary, upstream or secondary.

With Yann's show-info series on its way in, can we consider this series as
Rejected?

Thanks for the patches anyway, it was the push needed to get Yann to implement
the Right Thing (tm) :-)

Regards,
Arnout

#16 lore raw

PATCH v5 1/9 support/download: reintroduce 'source-check' target

Hi Arnout,

Quoted 14 lines · On Sat, Apr 13, 2019, 17:24 Arnout Vandecappelle <[email protected]> wrote:

Hi Thomas DS,

Quoted 3 lines · Arnout Vandecappelle

On 27/03/2019 17:35, Thomas Petazzoni wrote:

[snip]

Quoted 4 lines
Quoted 2 lines

We already have "make external-deps" but it only prints the file names, not the full URL.

True! So, we could extend external-deps with printing the URL (similar

to how

Quoted 2 lines

printvars uses RAW_VARS maybe, to not break existing scripts using external-deps). We could have variables to ask for primary, upstream or

Quoted 9 lines

secondary.

With Yann's show-info series on its way in, can we consider this series as
Rejected?

Thanks for the patches anyway, it was the push needed to get Yann to
implement
the Right Thing (tm) :-)

Yes, I agree to drop these patches and use show-info instead.

Thanks all for the discussion and alternative solution.

Best regards
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190413/c5fd7242/attachment.html>