Skip to contentSkip to conversation

infra: add the transient download mechanism

PATCH New Yann E. MORIN 5 participants 18 messages Jan 15, 2020 – Apr 8, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 infra: add the transient download mechanism

NOTE: NOT TO BE COMMITTED!

This patch is a proof of concept for the transient download mechanism, that would help using branches as versions, and keep using the head of the branch each time a build started.

A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT = YES

this means that the download infra will not use any already downloaded archive, and will instead always download it as if missing.

Since the check is done in the download wrapper, we have no TOCTOU race in case two bulds would attempt the same transient download: the archive is only replaced ato,mically as usual.

So, if the package uses a branch as version, the branch's HEAD at that very moment will be redownloaded.

This stil has the drawback that two builds in parallel may get slightly different content for the same branch, and the first build could end up using the download of the second build:

build-1 build-2

download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract

Furthermore, even with a single build, it might get what it expects:

developer-1 developer-2

git push branch
trigger CI git push branch
[...]
download

In that case the build of delopper-1 would get the code of developper-2 who pushed on the same branch.

For people who are aiming at their feet, we're now providing them with a loaded gun. ;-]

Cc Thomas Petazzoni <[email protected]>
Cc Vincent Fazio <[email protected]>
Cc Michael Nazzareno Trimarchi <[email protected]>
Cc Chris Packham <[email protected]>
Cc Nicolas Carrier <[email protected]>
Cc Adam Duskett <[email protected]>
---
 docs/manual/adding-packages-generic.txt | 16 +++++++++++++---
 package/busybox/busybox.mk              |  1 +
 package/pkg-download.mk                 |  1 +
 package/pkg-generic.mk                  |  8 +-------
 support/download/dl-wrapper             |  8 +++++---
 5 files changed, 21 insertions(+), 13 deletions(-)
docs/manual/adding-packages-generic.txt +13 -3
index baa052e31c..afffb09bc8 100644
@@ -206,12 +206,14 @@ information is (assuming the package name is +libfoo+) :
   ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+
 +
 .Note:
-Using a branch name as +FOO_VERSION+ is not supported, because it does
-not and can not work as people would expect it should:
+Using a branch name as +FOO_VERSION+, although technically possible,
+is highly discouraged, because it does not and can not work as people
+would expect it should:
 +
   1. due to local caching, Buildroot will not re-fetch the repository,
      so people who expect to be able to follow the remote repository
-     would be quite surprised and disappointed;
+     would be quite surprised and disappointed (but see
+     +LIBFOO_DOWNLOAD_TRANSIENT+, later);
   2. because two builds can never be perfectly simultaneous, and because
      the remote repository may get new commits on the branch anytime,
      two users, using the same Buildroot tree and building the same
@@ -342,6 +344,14 @@ not and can not work as people would expect it should:
   submodules when they contain bundled libraries, in which case we
   prefer to use those libraries from their own package.
 
+* +LIBFOO_DOWNLOAD_TRANSIENT+ can be set to +YES+ or +NO+ (the default).
+  If set to +YES+, the download for that package will be attempted at
+  every build; any already downloaded archive is ignored as if missing.
+  This may help when a branch is specified in +LIBFOO_VERSION+, and the
+  head/tip of the branch is to be built, like a CI pipeline would need
+  for example. This still suffers from the other issues listed above
+  about using branches, though.
+
 * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components
   (directories) that tar must strip from file names on extraction.
   The tarball for most packages has one leading component named
package/busybox/busybox.mk +1 -0
index 6283bc96ea..ee17febe48 100644
@@ -9,6 +9,7 @@ BUSYBOX_SITE = http://www.busybox.net/downloads
 BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2
 BUSYBOX_LICENSE = GPL-2.0
 BUSYBOX_LICENSE_FILES = LICENSE
+BUSYBOX_DOWNLOAD_TRANSIENT = YES
 
 define BUSYBOX_HELP_CMDS
 	@echo '  busybox-menuconfig     - Run BusyBox menuconfig'
package/pkg-download.mk +1 -0
index de619ba90a..c3a98ce7b0 100644
@@ -108,6 +108,7 @@ define DOWNLOAD
 		-n '$($(2)_BASENAME_RAW)' \
 		-N '$($(2)_RAWNAME)' \
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+		$(if $($(2)_DOWNLOAD_TRANSIENT),-F) \
 		$(if $($(2)_GIT_SUBMODULES),-r) \
 		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
package/pkg-generic.mk +1 -7
index 268d999efb..7c2e9db604 100644
@@ -158,13 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded:
 	@$(call step_start,download)
 	$(call prepare-per-package-directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES))
 	$(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep))
-# Only show the download message if it isn't already downloaded
-	$(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \
-		if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \
-			$(call MESSAGE,"Downloading") ; \
-			break ; \
-		fi ; \
-	done
+	@$(call MESSAGE,"Downloading") ; \
 	$(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p),$(PKG))$(sep))
 	$(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep))
 	$(Q)mkdir -p $(@D)
support/download/dl-wrapper +5 -3
index 3315bd410e..ab22ca7e4f 100755
@@ -21,11 +21,12 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
 
 main() {
     local OPT OPTARG
-    local backend output hfile recurse quiet rc
+    local backend output hfile recurse quiet rc force
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    force=false
+    while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -38,6 +39,7 @@ main() {
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
         q)  quiet="-q";;
+        F)  force=true;;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
         esac
@@ -67,7 +69,7 @@ main() {
     # - matches all its hashes: do not download it again and exit promptly
     # - fails at least one of its hashes: force a re-download
     # - there's no hash (but a .hash file): consider it a hard error
-    if [ -e "${output}" ]; then
+    if ! ${force} && [ -e "${output}" ]; then
         if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
             exit 0
         elif [ ${?} -ne 2 ]; then
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/1 infra: add the transient download mechanism

Hi

Quoted 10 lines · Yann E. MORIN

NOTE: NOT TO BE COMMITTED!

This patch is a proof of concept for the transient download mechanism, that would help using branches as versions, and keep using the head of the branch each time a build started.

A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT = YES

Why we can not just have like
PKG_TRANSIENT_OVERRIDE_FILE="ci.mk"

ci.mk

FOO_DOWNLOAD_TRANSIENT = YES
FOO_REVISION="<TAG> e/o BRANCH"

Michael

Quoted 172 lines

this means that the download infra will not use any already downloaded archive, and will instead always download it as if missing.

Since the check is done in the download wrapper, we have no TOCTOU race in case two bulds would attempt the same transient download: the archive is only replaced ato,mically as usual.

So, if the package uses a branch as version, the branch's HEAD at that very moment will be redownloaded.

This stil has the drawback that two builds in parallel may get slightly different content for the same branch, and the first build could end up using the download of the second build:

build-1 build-2

download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract

Furthermore, even with a single build, it might get what it expects:

developer-1 developer-2

git push branch
trigger CI git push branch
[...]
download

In that case the build of delopper-1 would get the code of developper-2 who pushed on the same branch.

For people who are aiming at their feet, we're now providing them with a loaded gun. ;-]

Cc Thomas Petazzoni <[email protected]>
Cc Vincent Fazio <[email protected]>
Cc Michael Nazzareno Trimarchi <[email protected]>
Cc Chris Packham <[email protected]>
Cc Nicolas Carrier <[email protected]>
Cc Adam Duskett <[email protected]>
---
 docs/manual/adding-packages-generic.txt | 16 +++++++++++++---
 package/busybox/busybox.mk              |  1 +
 package/pkg-download.mk                 |  1 +
 package/pkg-generic.mk                  |  8 +-------
 support/download/dl-wrapper             |  8 +++++---
 5 files changed, 21 insertions(+), 13 deletions(-)
docs/manual/adding-packages-generic.txt +13 -3
index baa052e31c..afffb09bc8 100644
@@ -206,12 +206,14 @@ information is (assuming the package name is +libfoo+) :
   ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+
 +
 .Note:
-Using a branch name as +FOO_VERSION+ is not supported, because it does
-not and can not work as people would expect it should:
+Using a branch name as +FOO_VERSION+, although technically possible,
+is highly discouraged, because it does not and can not work as people
+would expect it should:
 +
   1. due to local caching, Buildroot will not re-fetch the repository,
      so people who expect to be able to follow the remote repository
-     would be quite surprised and disappointed;
+     would be quite surprised and disappointed (but see
+     +LIBFOO_DOWNLOAD_TRANSIENT+, later);
   2. because two builds can never be perfectly simultaneous, and because
      the remote repository may get new commits on the branch anytime,
      two users, using the same Buildroot tree and building the same
@@ -342,6 +344,14 @@ not and can not work as people would expect it should:
   submodules when they contain bundled libraries, in which case we
   prefer to use those libraries from their own package.
 
+* +LIBFOO_DOWNLOAD_TRANSIENT+ can be set to +YES+ or +NO+ (the default).
+  If set to +YES+, the download for that package will be attempted at
+  every build; any already downloaded archive is ignored as if missing.
+  This may help when a branch is specified in +LIBFOO_VERSION+, and the
+  head/tip of the branch is to be built, like a CI pipeline would need
+  for example. This still suffers from the other issues listed above
+  about using branches, though.
+
 * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components
   (directories) that tar must strip from file names on extraction.
   The tarball for most packages has one leading component named
package/busybox/busybox.mk +1 -0
index 6283bc96ea..ee17febe48 100644
@@ -9,6 +9,7 @@ BUSYBOX_SITE = http://www.busybox.net/downloads
 BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2
 BUSYBOX_LICENSE = GPL-2.0
 BUSYBOX_LICENSE_FILES = LICENSE
+BUSYBOX_DOWNLOAD_TRANSIENT = YES
 
 define BUSYBOX_HELP_CMDS
        @echo '  busybox-menuconfig     - Run BusyBox menuconfig'
package/pkg-download.mk +1 -0
index de619ba90a..c3a98ce7b0 100644
@@ -108,6 +108,7 @@ define DOWNLOAD
                -n '$($(2)_BASENAME_RAW)' \
                -N '$($(2)_RAWNAME)' \
                -o '$($(2)_DL_DIR)/$(notdir $(1))' \
+               $(if $($(2)_DOWNLOAD_TRANSIENT),-F) \
                $(if $($(2)_GIT_SUBMODULES),-r) \
                $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
                $(QUIET) \
package/pkg-generic.mk +1 -7
index 268d999efb..7c2e9db604 100644
@@ -158,13 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded:
        @$(call step_start,download)
        $(call prepare-per-package-directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES))
        $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep))
-# Only show the download message if it isn't already downloaded
-       $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \
-               if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \
-                       $(call MESSAGE,"Downloading") ; \
-                       break ; \
-               fi ; \
-       done
+       @$(call MESSAGE,"Downloading") ; \
        $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p),$(PKG))$(sep))
        $(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep))
        $(Q)mkdir -p $(@D)
support/download/dl-wrapper +5 -4
index 3315bd410e..ab22ca7e4f 100755
@@ -21,11 +21,12 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
 
 main() {
     local OPT OPTARG
-    local backend output hfile recurse quiet rc
+    local backend output hfile recurse quiet rc force
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    force=false
+    while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -38,6 +39,7 @@ main() {
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
         q)  quiet="-q";;
+        F)  force=true;;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
         esac
@@ -67,7 +69,7 @@ main() {
     # - matches all its hashes: do not download it again and exit promptly
     # - fails at least one of its hashes: force a re-download
     # - there's no hash (but a .hash file): consider it a hard error
-    if [ -e "${output}" ]; then
+    if ! ${force} && [ -e "${output}" ]; then
         if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
             exit 0
         elif [ ${?} -ne 2 ]; then
--

2.20.1

Signature · 4 lines
| Michael Nazzareno Trimarchi                     Amarula Solutions BV |
| COO  -  Founder                                      Cruquiuskade 47 |
| +31(0)851119172                                 Amsterdam 1018 AM NL |
|                  [`as] http://www.amarulasolutions.com               |
#2 lore raw

PATCH 1/1 infra: add the transient download mechanism

Quoted 9 lines · Michael Nazzareno Trimarchi

Hi

Quoted 6 lines · Yann E. MORIN

NOTE: NOT TO BE COMMITTED!

This patch is a proof of concept for the transient download mechanism, that would help using branches as versions, and keep using the head of the branch each time a build started.

Quoted 5 lines
Quoted 3 lines

A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT = YES

Why we can not just have like
PKG_TRANSIENT_OVERRIDE_FILE="ci.mk"

You can use BR2_PACKAGE_OVERRIDE_FILE for this.

Quoted 5 lines

ci.mk

FOO_DOWNLOAD_TRANSIENT = YES
FOO_REVISION="<TAG> e/o BRANCH"

... but not for this, unfortunately.

There would be two ways to implement this after all:

1. implement the idea which has floated around already a couple of times to delay evaluation of most of inner-generic-package (and other infras) until later, and include the override file only after the package .mk files;

2. create new override options FOO_OVERRIDE_VERSION, ... and use those instead of the package options - similar to OVERRIDE_SRCDIR.

Both are totally orthogonal to the TRANSIENT idea, of course.

Regards,
Arnout

#3 lore raw

PATCH 1/1 infra: add the transient download mechanism

In reply to Yann E. MORIN

Hello Yann,
One small remark, to be able to reproduce the build, on top of this mechanism, there should be:
* a way to store the actual VCS revision used for the build (to at least know what was just built!)
* a way to force the build to re-use a particular set of revisions

That's features not easy to get right, but which are already handled by tools like google's repo (or partly, by git submodules).

That's part of why I don't think that this kind of feature is a good thing to integrate into buildroot. I prefer to delegate it to tools meant for that (using a mix of local packages and OVERRIDE). And like said before, I think it's preferable to have a optional feature to prevent branches from being used, making the build fail.

Quoted 217 lines · Yann E. MORIN

NOTE: NOT TO BE COMMITTED!

This patch is a proof of concept for the transient download mechanism,
that would help using branches as versions, and keep using the head of
the branch each time a build started.

A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT = YES

this means that the download infra will not use any already downloaded
archive, and will instead always download it as if missing.

Since the check is done in the download wrapper, we have no TOCTOU race
in case two bulds would attempt the same transient download: the archive
is only replaced ato,mically as usual.

So, if the package uses a branch as version, the branch's HEAD at that
very moment will be redownloaded.

This stil has the drawback that two builds in parallel may get slightly
different content for the same branch, and the first build could end up
using the download of the second build:

build-1 build-2

download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract

Furthermore, even with a single build, it might get what it expects:

developer-1 developer-2

git push branch
trigger CI git push branch
[...]
download

In that case the build of delopper-1 would get the code of developper-2
who pushed on the same branch.

For people who are aiming at their feet, we're now providing them with
a loaded gun. ;-]

Cc Thomas Petazzoni <[email protected]>
Cc Vincent Fazio <[email protected]>
Cc Michael Nazzareno Trimarchi <[email protected]>
Cc Chris Packham <[email protected]>
Cc Nicolas Carrier <[email protected]>
Cc Adam Duskett <[email protected]>
---
 docs/manual/adding-packages-generic.txt | 16 +++++++++++++---
 package/busybox/busybox.mk              |  1 +
 package/pkg-download.mk                 |  1 +
 package/pkg-generic.mk                  |  8 +-------
 support/download/dl-wrapper             |  8 +++++---
 5 files changed, 21 insertions(+), 13 deletions(-)
docs/manual/adding-packages-generic.txt +1 -1
index baa052e31c..afffb09bc8 100644
@@ -206,12 +206,14 @@ information is (assuming the package name is
+libfoo+) :
   ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+
 +
 .Note:
-Using a branch name as +FOO_VERSION+ is not supported, because it

does

docs/manual/adding-packages-generic.txt +2 -1
-not and can not work as people would expect it should:
+Using a branch name as +FOO_VERSION+, although technically possible,
+is highly discouraged, because it does not and can not work as

people
+would expect it should:
+
1. due to local caching, Buildroot will not re-fetch the
repository,
so people who expect to be able to follow the remote repository

docs/manual/adding-packages-generic.txt +2 -1
-     would be quite surprised and disappointed;
+     would be quite surprised and disappointed (but see
+     +LIBFOO_DOWNLOAD_TRANSIENT+, later);
   2. because two builds can never be perfectly simultaneous, and

because
the remote repository may get new commits on the branch
anytime,
two users, using the same Buildroot tree and building the same

docs/manual/adding-packages-generic.txt +0 -0
@@ -342,6 +344,14 @@ not and can not work as people would expect it

should:
submodules when they contain bundled libraries, in which case we prefer to use those libraries from their own package.

+* +LIBFOO_DOWNLOAD_TRANSIENT+ can be set to +YES+ or +NO+ (the default).
+ If set to +YES+, the download for that package will be attempted at
+ every build; any already downloaded archive is ignored as if missing.
+ This may help when a branch is specified in +LIBFOO_VERSION+, and the
+ head/tip of the branch is to be built, like a CI pipeline would need

docs/manual/adding-packages-generic.txt +3 -0
+  for example. This still suffers from the other issues listed above
+  about using branches, though.
+
 * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components
   (directories) that tar must strip from file names on extraction.
   The tarball for most packages has one leading component named
package/busybox/busybox.mk +1 -0
index 6283bc96ea..ee17febe48 100644
@@ -9,6 +9,7 @@ BUSYBOX_SITE = http://www.busybox.net/downloads
 BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2
 BUSYBOX_LICENSE = GPL-2.0
 BUSYBOX_LICENSE_FILES = LICENSE
+BUSYBOX_DOWNLOAD_TRANSIENT = YES
 
 define BUSYBOX_HELP_CMDS
        @echo '  busybox-menuconfig     - Run BusyBox menuconfig'
package/pkg-download.mk +1 -0
index de619ba90a..c3a98ce7b0 100644
@@ -108,6 +108,7 @@ define DOWNLOAD
                -n '$($(2)_BASENAME_RAW)' \
                -N '$($(2)_RAWNAME)' \
                -o '$($(2)_DL_DIR)/$(notdir $(1))' \
+               $(if $($(2)_DOWNLOAD_TRANSIENT),-F) \
                $(if $($(2)_GIT_SUBMODULES),-r) \
                $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u
 $(uri)) \
                $(QUIET) \
package/pkg-generic.mk +1 -7
index 268d999efb..7c2e9db604 100644
@@ -158,13 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded:
        @$(call step_start,download)
        $(call prepare-per-package-
 directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES))
        $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call
 $(hook))$(sep))
-# Only show the download message if it isn't already downloaded
-       $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \
-               if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \
-                       $(call MESSAGE,"Downloading") ; \
-                       break ; \
-               fi ; \
-       done
+       @$(call MESSAGE,"Downloading") ; \
        $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call

DOWNLOAD,$(p),$(PKG))$(sep))

        $(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call
$(hook))$(sep))
        $(Q)mkdir -p $(@D)
support/download/dl- renamed from support/download/dl-wrapper +0 -0

wrapper
index 3315bd410e..ab22ca7e4f 100755

support/download/dl-wrapper +0 -0
@@ -21,11 +21,12 @@ export

BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"

main() {
local OPT OPTARG

support/download/dl-wrapper +4 -2
-    local backend output hfile recurse quiet rc
+    local backend output hfile recurse quiet rc force
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    force=false
+    while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -38,6 +39,7 @@ main() {
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
         q)  quiet="-q";;
+        F)  force=true;;
         :)  error "option '%s' expects a mandatory argument\n"
 "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;

esac

support/download/dl-wrapper +0 -0
@@ -67,7 +69,7 @@ main() {
     # - matches all its hashes: do not download it again and exit

promptly
# - fails at least one of its hashes: force a re-download # - there's no hash (but a .hash file): consider it a hard error

support/download/dl-wrapper +1 -1
-    if [ -e "${output}" ]; then
+    if ! ${force} && [ -e "${output}" ]; then
         if support/download/check-hash ${quiet} "${hfile}"

"${output}" "${output##*/}"; then

             exit 0
         elif [ ${?} -ne 2 ]; then

--
2.20.1

ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.

#4 lore raw

PATCH 1/1 infra: add the transient download mechanism

Hello,

Quoted 4 lines · On Thu, 16 Jan 2020 09:11:32 +0000 Nicolas Carrier…

One small remark, to be able to reproduce the build, on top of this mechanism, there should be:
* a way to store the actual VCS revision used for the build (to at least know what was just built!)

That seems quite easy to do.

Quoted 1 line

* a way to force the build to re-use a particular set of revisions

I don't see how that can be done however, and how that would work from a user interface point of view. Sure the first build could store the exact Git commit used for building each transient package, and store that "somewhere", and then re-use that for the next build. But how does the user decide whether what he wants is to re-do the build with the same commits or not ? Where is that "somewhere" ?

Quoted 3 lines

That's part of why I don't think that this kind of feature is a good thing to integrate into buildroot. I prefer to delegate it to tools meant for that (using a mix of local packages and OVERRIDE).

Different people/companies have different work flows. We have people telling us that using local packages and OVERRIDE for their CI is not very convenient/practical.

Quoted 2 lines

And like said before, I think it's preferable to have a optional feature to prevent branches from being used, making the build fail.

The feature proposed by Yann *is* optional, as it's conditional on having a specific variable set in the package .mk file.

However, one question that is raised by Yann's patch is whether we want to forbid using branches when TRANSIENT_DOWNLOAD is NO.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

PATCH 1/1 infra: add the transient download mechanism

Hi

Quoted 22 lines · Thomas Petazzoni

Hello,

Quoted 4 lines · On Thu, 16 Jan 2020 09:11:32 +0000 Nicolas Carrier…

One small remark, to be able to reproduce the build, on top of this mechanism, there should be:
* a way to store the actual VCS revision used for the build (to at least know what was just built!)

That seems quite easy to do.

Quoted 1 line

* a way to force the build to re-use a particular set of revisions

I don't see how that can be done however, and how that would work from a user interface point of view. Sure the first build could store the exact Git commit used for building each transient package, and store that "somewhere", and then re-use that for the next build. But how does the user decide whether what he wants is to re-do the build with the same commits or not ? Where is that "somewhere" ?

Quoted 7 lines
Quoted 3 lines

That's part of why I don't think that this kind of feature is a good thing to integrate into buildroot. I prefer to delegate it to tools meant for that (using a mix of local packages and OVERRIDE).

Different people/companies have different work flows. We have people telling us that using local packages and OVERRIDE for their CI is not very convenient/practical.

We are using a similar setup of Nicolas. Right now we have
CI and reproducible build using external tools. What I would like to solve with buildroot is:

- make the build reproducible
- implement topic build on custom patch using CI
- make tagging and branching easy

All of this I have already with repo + buildroot

Michael

Quoted 17 lines
Quoted 3 lines

And like said before, I think it's preferable to have a optional feature to prevent branches from being used, making the build fail.

The feature proposed by Yann *is* optional, as it's conditional on having a specific variable set in the package .mk file.

However, one question that is raised by Yann's patch is whether we want to forbid using branches when TRANSIENT_DOWNLOAD is NO.

Best regards,

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Signature · 4 lines
| Michael Nazzareno Trimarchi                     Amarula Solutions BV |
| COO  -  Founder                                      Cruquiuskade 47 |
| +31(0)851119172                                 Amsterdam 1018 AM NL |
|                  [`as] http://www.amarulasolutions.com               |
#6 lore raw

PATCH 1/1 infra: add the transient download mechanism

Quoted 7 lines · On Thu, 16 Jan 2020 10:56:53 +0100 Michael Nazzareno Trimarchi…

We are using a similar setup of Nicolas. Right now we have
CI and reproducible build using external tools. What I would like to solve with buildroot is:

- make the build reproducible
- implement topic build on custom patch using CI
- make tagging and branching easy

Could you give more details, because these items are quite vague/fuzzy, and I don't understand from a very practical point of view what you mean. Could you describe some example workflow, what would be your expectation of Buildroot's behavior, etc.

Thanks,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#7 lore raw

PATCH 1/1 infra: add the transient download mechanism

Hi

Quoted 8 lines · Thomas Petazzoni
Quoted 4 lines · On Thu, 16 Jan 2020 10:56:53 +0100 Michael Nazzareno Trimarchi…

We are using a similar setup of Nicolas. Right now we have
CI and reproducible build using external tools. What I would like to solve with buildroot is:

Quoted 9 lines
Quoted 3 lines

- make the build reproducible
- implement topic build on custom patch using CI
- make tagging and branching easy

Could you give more details, because these items are quite vague/fuzzy, and I don't understand from a very practical point of view what you mean. Could you describe some example workflow, what would be your expectation of Buildroot's behavior, etc.

Suppose you have this manifest

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remote name="amarula-gitea"

           fetch=".."
           review="https://gerrit-review.amarulasolutions.com" />

<remote name="prj"

           fetch="ssh://server1/home/gitrepos/"
           revision="master" />

<default revision="prj2"

           remote="amarula-gitea"
           sync-j="4"
           sync-c="true" />

<project path="app1" name="smile" remote="prj" /> <project path="buildroot" name="prj/buildroot" /> <project path="linux" name="prj/linux" /> <project path="build" name="prj/build" >
<copyfile src="root.mk" dest="Makefile" />
</project>
<project path="library/lib1" name="prj/lib1" /> <project path="u-boot" name="prj/u-boot" />
</manifest>

This one describe the project and we have override part for the project in the manifest. For each version we can tag and each repo and create a snapshot manifest of the project. Manifest will have revison set to revision="refs/tags/<some tag>"

If I need to emit a release starting from some tag I can branch it and apply patch on top of this branch and everything
can be stored as a separed manifest. Using jenkins trigger on topic upload on gerrit we can easily build a version with some patchset applied and if we want to re-build some version we can just use the right tagged manifest

Michael

Quoted 7 lines

Thanks,

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Signature · 4 lines
| Michael Nazzareno Trimarchi                     Amarula Solutions BV |
| COO  -  Founder                                      Cruquiuskade 47 |
| +31(0)851119172                                 Amsterdam 1018 AM NL |
|                  [`as] http://www.amarulasolutions.com               |
#8 lore raw

PATCH 1/1 infra: add the transient download mechanism

Michael, All,

On 2020-01-16 11:15 +0100, Michael Nazzareno Trimarchi spake thusly:

Quoted 6 lines
Quoted 4 lines · Thomas Petazzoni
Quoted 4 lines

We are using a similar setup of Nicolas. Right now we have
CI and reproducible build using external tools. What I would like to solve with buildroot is:

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

- make the build reproducible
- implement topic build on custom patch using CI
- make tagging and branching easy

Could you give more details, because these items are quite vague/fuzzy, and I don't understand from a very practical point of view what you mean. Could you describe some example workflow, what would be your expectation of Buildroot's behavior, etc.

I have to admit that, like Thomas, I do not completely understand what you're doing or trying to achieve...

Quoted 1 line

Suppose you have this manifest

[--SNIP--]

Quoted 11 lines

This one describe the project and we have override part for the project in the manifest. For each version we can tag and each repo and create a snapshot manifest of the project. Manifest will have revison set to revision="refs/tags/<some tag>"

If I need to emit a release starting from some tag I can branch it and apply patch on top of this branch and everything
can be stored as a separed manifest. Using jenkins trigger on topic upload on gerrit we can easily build a version with some patchset applied and if we want to re-build some version we can just use the right tagged manifest

I don't understand the problem that using repo solves.

repo's manifest identifies packages, where they are from, and what version to use for each. That is exactly what the _VERSION and _SITE do in a .mk file to begin with...

Using repo and a manifest only seem interesting when one wants to have their packages in the same directory structure as Buildroot, and use _SITE_METHOD = local. But in that case, a repo manifest is very akin to git submodules.

And in that case, there is no need to be able to uses branches as _VERSION, since the packages sources jsut 'follow' the branch buildroot is on, and that repo checked out...

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#9 lore raw

PATCH 1/1 infra: add the transient download mechanism

Hi

Quoted 35 lines · Yann E. MORIN

Michael, All,

On 2020-01-16 11:15 +0100, Michael Nazzareno Trimarchi spake thusly:

Quoted 6 lines
Quoted 4 lines · Thomas Petazzoni
Quoted 4 lines

We are using a similar setup of Nicolas. Right now we have
CI and reproducible build using external tools. What I would like to solve with buildroot is:

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

- make the build reproducible
- implement topic build on custom patch using CI
- make tagging and branching easy

Could you give more details, because these items are quite vague/fuzzy, and I don't understand from a very practical point of view what you mean. Could you describe some example workflow, what would be your expectation of Buildroot's behavior, etc.

I have to admit that, like Thomas, I do not completely understand what you're doing or trying to achieve...

Quoted 1 line

Suppose you have this manifest

[--SNIP--]

Quoted 11 lines

This one describe the project and we have override part for the project in the manifest. For each version we can tag and each repo and create a snapshot manifest of the project. Manifest will have revison set to revision="refs/tags/<some tag>"

If I need to emit a release starting from some tag I can branch it and apply patch on top of this branch and everything
can be stored as a separed manifest. Using jenkins trigger on topic upload on gerrit we can easily build a version with some patchset applied and if we want to re-build some version we can just use the right tagged manifest

Quoted 5 lines

I don't understand the problem that using repo solves.

repo's manifest identifies packages, where they are from, and what version to use for each. That is exactly what the _VERSION and _SITE do in a .mk file to begin with...

Yes, definitely you can manage in the same way but having one point as a manifest file where I can manage only project I change during the product life avoid
me to change those when I create a specific tag. repo let me to tag in automatic way and let buildroot to be tagged only on project that I download from outside. I'm not saying here that the direction you are taken is wrong but not all the companies will use it.
BTW my way to write english is terrible, I like having a desk and discuss face to face. Make more sense to me to discuss in Fossdem if you will be there

Michael

Quoted 13 lines

Using repo and a manifest only seem interesting when one wants to have their packages in the same directory structure as Buildroot, and use _SITE_METHOD = local. But in that case, a repo manifest is very akin to git submodules.

And in that case, there is no need to be able to uses branches as _VERSION, since the packages sources jsut 'follow' the branch buildroot is on, and that repo checked out...

Regards,
Yann E. MORIN.

Quoted 7 lines

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

--

| Michael Nazzareno Trimarchi                     Amarula Solutions BV |
| COO  -  Founder                                      Cruquiuskade 47 |
| +31(0)851119172                                 Amsterdam 1018 AM NL |
|                  [`as] http://www.amarulasolutions.com               |
#10 lore raw

PATCH 1/1 infra: add the transient download mechanism

In reply to Thomas Petazzoni

Hello,

Quoted 4 lines

However, one question that is raised by Yann's patch is whether we want
to forbid using branches when TRANSIENT_DOWNLOAD is NO.

Just to be sure I understand, when you say "when TRANSIENT_DOWNLOAD is NO.", it'd be the same as if TRANSIENT_DOWNLOAD download is not defined at all, right?
If it's the case, then that seems to me like a good idea, because it renders explicit the risk taken by the user when using a branch (which can be done unknowingly ATM).

Quoted 10 lines

Best regards,

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.

#11 lore raw

PATCH 1/1 infra: add the transient download mechanism

Quoted 3 lines · On Thu, 16 Jan 2020 10:01:09 +0000 Nicolas Carrier…

Just to be sure I understand, when you say "when TRANSIENT_DOWNLOAD is NO.", it'd be the same as if TRANSIENT_DOWNLOAD download is not defined at all, right?

Yes.

There is maybe a little bit of an issue in Yann's current implementation as he does:

	$(if $($(2)_DOWNLOAD_TRANSIENT),-F) \

so in fact, -F would be passed as soon as <pkg>_DOWNLOAD_TRANSIENT is non-empty, so <pkg>_DOWNLOAD_TRANSIENT = NO would in fact pass -F.

But this bug already exists for <pkg>_GIT_SUBMODULES:

	$(if $($(2)_GIT_SUBMODULES),-r) \

And it is trivial to fix.

Quoted 3 lines

If it's the case, then that seems to me like a good idea, because it renders explicit the risk taken by the user when using a branch (which can be done unknowingly ATM).

Absolutely.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#12 lore raw

PATCH 1/1 infra: add the transient download mechanism

Thomas, All,

On 2020-01-16 11:05 +0100, Thomas Petazzoni spake thusly:

Quoted 12 lines
Quoted 3 lines · On Thu, 16 Jan 2020 10:01:09 +0000 Nicolas Carrier…

Just to be sure I understand, when you say "when TRANSIENT_DOWNLOAD is NO.", it'd be the same as if TRANSIENT_DOWNLOAD download is not defined at all, right?

Yes.

There is maybe a little bit of an issue in Yann's current implementation as he does:

Quoted 8 lines
	$(if $($(2)_DOWNLOAD_TRANSIENT),-F) \

so in fact, -F would be passed as soon as <pkg>_DOWNLOAD_TRANSIENT is non-empty, so <pkg>_DOWNLOAD_TRANSIENT = NO would in fact pass -F.

But this bug already exists for <pkg>_GIT_SUBMODULES:

	$(if $($(2)_GIT_SUBMODULES),-r) \

Yeah, I just mirrored for transient wghat we already had for git submodules.

Quoted 1 line

And it is trivial to fix.

If ever needed...

    $(if $(filter YES,$($(2)_DOWNLOAD_TRANSIENT)),-F)

(note that there is no equality test in GNU make, but there is one simple one described in GMSL).

Quoted 3 lines
Quoted 3 lines

If it's the case, then that seems to me like a good idea, because it renders explicit the risk taken by the user when using a branch (which can be done unknowingly ATM).

Yup.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#13 lore raw

PATCH 1/1 infra: add the transient download mechanism

In reply to Nicolas Carrier

Nicolas, All,

On 2020-01-16 09:11 +0000, Nicolas Carrier spake thusly:

Quoted 9 lines

Hello Yann,
One small remark, to be able to reproduce the build, on top of this mechanism, there should be:
* a way to store the actual VCS revision used for the build (to at least know what was just built!)
* a way to force the build to re-use a particular set of revisions

That's features not easy to get right, but which are already handled by tools like google's repo (or partly, by git submodules).

Well, that's already the case with Buildroot: setting _VERSION is what guarantees you can rebuild the same thing again and again.

I don't know much of repo, but from what I grabbed, it uses a manifest to identify packages, where they are, what changesset to use (and some less interesting stuff). This is exactly what one has in Buildroot's .mk files. The manifest is just builtin to the buildsystem.

Quoted 3 lines

That's part of why I don't think that this kind of feature is a good thing to integrate into buildroot. I prefer to delegate it to tools meant for that (using a mix of local packages and OVERRIDE).

You don't need to convince me. But apparently, some people like to aim at their feet with a loaded gun, just in case.. ;-)

On a more serious note, as Thomas explained, some people have expressed the need for using a branch, and that a build would alkways pick the latest version of that branch.

Of course, that means non-reproducibility.

I am even afraid that those people do not realise that they also need to update the _VERSION in their packages once they want to tag and release versions. And this has virtually no overhead in practice (damn... copy-pasting a sha1 is not that complex. It would even be possible to have a script that does it automatically...).

Quoted 2 lines

And like said before, I think it's preferable to have a optional feature to prevent branches from being used, making the build fail.

Yeah, that was one of the comment by Thomas, and it is a good extension to the transient mehcanism: either one uses a stable ref, or one needs to explicitly identify it as transient.

Thanks for the feedback.

Regards,
Yann E. MORIN.

Quoted 219 lines · Yann E. MORIN
Quoted 217 lines

NOTE: NOT TO BE COMMITTED!

This patch is a proof of concept for the transient download mechanism,
that would help using branches as versions, and keep using the head of
the branch each time a build started.

A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT = YES

this means that the download infra will not use any already downloaded
archive, and will instead always download it as if missing.

Since the check is done in the download wrapper, we have no TOCTOU race
in case two bulds would attempt the same transient download: the archive
is only replaced ato,mically as usual.

So, if the package uses a branch as version, the branch's HEAD at that
very moment will be redownloaded.

This stil has the drawback that two builds in parallel may get slightly
different content for the same branch, and the first build could end up
using the download of the second build:

build-1 build-2

download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract

Furthermore, even with a single build, it might get what it expects:

developer-1 developer-2

git push branch
trigger CI git push branch
[...]
download

In that case the build of delopper-1 would get the code of developper-2
who pushed on the same branch.

For people who are aiming at their feet, we're now providing them with
a loaded gun. ;-]

Cc Thomas Petazzoni <[email protected]>
Cc Vincent Fazio <[email protected]>
Cc Michael Nazzareno Trimarchi <[email protected]>
Cc Chris Packham <[email protected]>
Cc Nicolas Carrier <[email protected]>
Cc Adam Duskett <[email protected]>
---
 docs/manual/adding-packages-generic.txt | 16 +++++++++++++---
 package/busybox/busybox.mk              |  1 +
 package/pkg-download.mk                 |  1 +
 package/pkg-generic.mk                  |  8 +-------
 support/download/dl-wrapper             |  8 +++++---
 5 files changed, 21 insertions(+), 13 deletions(-)
docs/manual/adding-packages-generic.txt +1 -1
index baa052e31c..afffb09bc8 100644
@@ -206,12 +206,14 @@ information is (assuming the package name is
+libfoo+) :
   ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+
 +
 .Note:
-Using a branch name as +FOO_VERSION+ is not supported, because it

does

docs/manual/adding-packages-generic.txt +2 -1
-not and can not work as people would expect it should:
+Using a branch name as +FOO_VERSION+, although technically possible,
+is highly discouraged, because it does not and can not work as

people
+would expect it should:
+
1. due to local caching, Buildroot will not re-fetch the
repository,
so people who expect to be able to follow the remote repository

docs/manual/adding-packages-generic.txt +2 -1
-     would be quite surprised and disappointed;
+     would be quite surprised and disappointed (but see
+     +LIBFOO_DOWNLOAD_TRANSIENT+, later);
   2. because two builds can never be perfectly simultaneous, and

because
the remote repository may get new commits on the branch
anytime,
two users, using the same Buildroot tree and building the same

docs/manual/adding-packages-generic.txt +0 -0
@@ -342,6 +344,14 @@ not and can not work as people would expect it

should:
submodules when they contain bundled libraries, in which case we prefer to use those libraries from their own package.

+* +LIBFOO_DOWNLOAD_TRANSIENT+ can be set to +YES+ or +NO+ (the default).
+ If set to +YES+, the download for that package will be attempted at
+ every build; any already downloaded archive is ignored as if missing.
+ This may help when a branch is specified in +LIBFOO_VERSION+, and the
+ head/tip of the branch is to be built, like a CI pipeline would need

docs/manual/adding-packages-generic.txt +3 -0
+  for example. This still suffers from the other issues listed above
+  about using branches, though.
+
 * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components
   (directories) that tar must strip from file names on extraction.
   The tarball for most packages has one leading component named
package/busybox/busybox.mk +1 -0
index 6283bc96ea..ee17febe48 100644
@@ -9,6 +9,7 @@ BUSYBOX_SITE = http://www.busybox.net/downloads
 BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2
 BUSYBOX_LICENSE = GPL-2.0
 BUSYBOX_LICENSE_FILES = LICENSE
+BUSYBOX_DOWNLOAD_TRANSIENT = YES
 
 define BUSYBOX_HELP_CMDS
        @echo '  busybox-menuconfig     - Run BusyBox menuconfig'
package/pkg-download.mk +1 -0
index de619ba90a..c3a98ce7b0 100644
@@ -108,6 +108,7 @@ define DOWNLOAD
                -n '$($(2)_BASENAME_RAW)' \
                -N '$($(2)_RAWNAME)' \
                -o '$($(2)_DL_DIR)/$(notdir $(1))' \
+               $(if $($(2)_DOWNLOAD_TRANSIENT),-F) \
                $(if $($(2)_GIT_SUBMODULES),-r) \
                $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u
 $(uri)) \
                $(QUIET) \
package/pkg-generic.mk +1 -7
index 268d999efb..7c2e9db604 100644
@@ -158,13 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded:
        @$(call step_start,download)
        $(call prepare-per-package-
 directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES))
        $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call
 $(hook))$(sep))
-# Only show the download message if it isn't already downloaded
-       $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \
-               if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \
-                       $(call MESSAGE,"Downloading") ; \
-                       break ; \
-               fi ; \
-       done
+       @$(call MESSAGE,"Downloading") ; \
        $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call

DOWNLOAD,$(p),$(PKG))$(sep))

        $(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call
$(hook))$(sep))
        $(Q)mkdir -p $(@D)
support/download/dl- renamed from support/download/dl-wrapper +0 -0

wrapper
index 3315bd410e..ab22ca7e4f 100755

support/download/dl-wrapper +0 -0
@@ -21,11 +21,12 @@ export

BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"

main() {
local OPT OPTARG

support/download/dl-wrapper +4 -2
-    local backend output hfile recurse quiet rc
+    local backend output hfile recurse quiet rc force
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    force=false
+    while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -38,6 +39,7 @@ main() {
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
         q)  quiet="-q";;
+        F)  force=true;;
         :)  error "option '%s' expects a mandatory argument\n"
 "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;

esac

support/download/dl-wrapper +0 -0
@@ -67,7 +69,7 @@ main() {
     # - matches all its hashes: do not download it again and exit

promptly
# - fails at least one of its hashes: force a re-download # - there's no hash (but a .hash file): consider it a hard error

support/download/dl-wrapper +1 -1
-    if [ -e "${output}" ]; then
+    if ! ${force} && [ -e "${output}" ]; then
         if support/download/check-hash ${quiet} "${hfile}"

"${output}" "${output##*/}"; then

             exit 0
         elif [ ${?} -ne 2 ]; then

--
2.20.1

ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#14 lore raw

PATCH 1/1 infra: add the transient download mechanism

Quoted 13 lines · Yann E. MORIN

Nicolas, All,

On 2020-01-16 09:11 +0000, Nicolas Carrier spake thusly:

Quoted 9 lines

Hello Yann,
One small remark, to be able to reproduce the build, on top of this mechanism, there should be:
* a way to store the actual VCS revision used for the build (to at least know what was just built!)
* a way to force the build to re-use a particular set of revisions

That's features not easy to get right, but which are already handled by tools like google's repo (or partly, by git submodules).

Quoted 7 lines

Well, that's already the case with Buildroot: setting _VERSION is what guarantees you can rebuild the same thing again and again.

I don't know much of repo, but from what I grabbed, it uses a manifest to identify packages, where they are, what changesset to use (and some less interesting stuff). This is exactly what one has in Buildroot's .mk files. The manifest is just builtin to the buildsystem.

The difference is that repo doesn't need the changeset to use. If no changeset
is specified, then 'repo update' will do a pull of all branches. So the transient download mechanism pretty much approximates that.

That said, there are clear advantages to using repo during development compared
to the transient download mechanism:

- possibility to branch or tag several trees in one shot;
- easy to fixate the manifest from a specific situation;
- no need to switch to yet another mechanism (OVERRIDE_SRCDIR isntead of transient) when you want to make modifications.

So for a serious project, repo is a clear win.

However, for small projects, it's a bit overkill. For a smaller project, you'd
typically have only a few trees to manage (buildroot, external, linux, application) and most of them are not terribly active, so the features of repo don't bring a lot of added value. And managing the manifest file is definitely more complicated than just specifying the linux git in your defconfig.

Bottom line: even though repo is the better tool to manage the trees both for
development and release, the transient mechanism is a good quick win.

Quoted 19 lines
Quoted 3 lines

That's part of why I don't think that this kind of feature is a good thing to integrate into buildroot. I prefer to delegate it to tools meant for that (using a mix of local packages and OVERRIDE).

You don't need to convince me. But apparently, some people like to aim at their feet with a loaded gun, just in case.. ;-)

On a more serious note, as Thomas explained, some people have expressed the need for using a branch, and that a build would alkways pick the latest version of that branch.

Of course, that means non-reproducibility.

I am even afraid that those people do not realise that they also need to update the _VERSION in their packages once they want to tag and release versions. And this has virtually no overhead in practice (damn... copy-pasting a sha1 is not that complex. It would even be possible to have a script that does it automatically...).

Quoted 6 lines
Quoted 2 lines

And like said before, I think it's preferable to have a optional feature to prevent branches from being used, making the build fail.

Yeah, that was one of the comment by Thomas, and it is a good extension to the transient mehcanism: either one uses a stable ref, or one needs to explicitly identify it as transient.

We already have patches for that in patchwork.

With those, we'd need to propagate the -F option down to the git helper, so it
can avoid erroring out.

Regards,
Arnout

Quoted 226 lines

Thanks for the feedback.

Regards,
Yann E. MORIN.

Quoted 219 lines · Yann E. MORIN
Quoted 217 lines

NOTE: NOT TO BE COMMITTED!

This patch is a proof of concept for the transient download mechanism,
that would help using branches as versions, and keep using the head of
the branch each time a build started.

A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT = YES

this means that the download infra will not use any already downloaded
archive, and will instead always download it as if missing.

Since the check is done in the download wrapper, we have no TOCTOU race
in case two bulds would attempt the same transient download: the archive
is only replaced ato,mically as usual.

So, if the package uses a branch as version, the branch's HEAD at that
very moment will be redownloaded.

This stil has the drawback that two builds in parallel may get slightly
different content for the same branch, and the first build could end up
using the download of the second build:

build-1 build-2

download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract

Furthermore, even with a single build, it might get what it expects:

developer-1 developer-2

git push branch
trigger CI git push branch
[...]
download

In that case the build of delopper-1 would get the code of developper-2
who pushed on the same branch.

For people who are aiming at their feet, we're now providing them with
a loaded gun. ;-]

Cc Thomas Petazzoni <[email protected]>
Cc Vincent Fazio <[email protected]>
Cc Michael Nazzareno Trimarchi <[email protected]>
Cc Chris Packham <[email protected]>
Cc Nicolas Carrier <[email protected]>
Cc Adam Duskett <[email protected]>
---
 docs/manual/adding-packages-generic.txt | 16 +++++++++++++---
 package/busybox/busybox.mk              |  1 +
 package/pkg-download.mk                 |  1 +
 package/pkg-generic.mk                  |  8 +-------
 support/download/dl-wrapper             |  8 +++++---
 5 files changed, 21 insertions(+), 13 deletions(-)
docs/manual/adding-packages-generic.txt +1 -1
index baa052e31c..afffb09bc8 100644
@@ -206,12 +206,14 @@ information is (assuming the package name is
+libfoo+) :
   ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+
 +
 .Note:
-Using a branch name as +FOO_VERSION+ is not supported, because it

does

docs/manual/adding-packages-generic.txt +2 -1
-not and can not work as people would expect it should:
+Using a branch name as +FOO_VERSION+, although technically possible,
+is highly discouraged, because it does not and can not work as

people
+would expect it should:
+
1. due to local caching, Buildroot will not re-fetch the
repository,
so people who expect to be able to follow the remote repository

docs/manual/adding-packages-generic.txt +2 -1
-     would be quite surprised and disappointed;
+     would be quite surprised and disappointed (but see
+     +LIBFOO_DOWNLOAD_TRANSIENT+, later);
   2. because two builds can never be perfectly simultaneous, and

because
the remote repository may get new commits on the branch
anytime,
two users, using the same Buildroot tree and building the same

docs/manual/adding-packages-generic.txt +0 -0
@@ -342,6 +344,14 @@ not and can not work as people would expect it

should:
submodules when they contain bundled libraries, in which case we prefer to use those libraries from their own package.

+* +LIBFOO_DOWNLOAD_TRANSIENT+ can be set to +YES+ or +NO+ (the default).
+ If set to +YES+, the download for that package will be attempted at
+ every build; any already downloaded archive is ignored as if missing.
+ This may help when a branch is specified in +LIBFOO_VERSION+, and the
+ head/tip of the branch is to be built, like a CI pipeline would need

docs/manual/adding-packages-generic.txt +3 -0
+  for example. This still suffers from the other issues listed above
+  about using branches, though.
+
 * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components
   (directories) that tar must strip from file names on extraction.
   The tarball for most packages has one leading component named
package/busybox/busybox.mk +1 -0
index 6283bc96ea..ee17febe48 100644
@@ -9,6 +9,7 @@ BUSYBOX_SITE = http://www.busybox.net/downloads
 BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2
 BUSYBOX_LICENSE = GPL-2.0
 BUSYBOX_LICENSE_FILES = LICENSE
+BUSYBOX_DOWNLOAD_TRANSIENT = YES
 
 define BUSYBOX_HELP_CMDS
        @echo '  busybox-menuconfig     - Run BusyBox menuconfig'
package/pkg-download.mk +1 -0
index de619ba90a..c3a98ce7b0 100644
@@ -108,6 +108,7 @@ define DOWNLOAD
                -n '$($(2)_BASENAME_RAW)' \
                -N '$($(2)_RAWNAME)' \
                -o '$($(2)_DL_DIR)/$(notdir $(1))' \
+               $(if $($(2)_DOWNLOAD_TRANSIENT),-F) \
                $(if $($(2)_GIT_SUBMODULES),-r) \
                $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u
 $(uri)) \
                $(QUIET) \
package/pkg-generic.mk +1 -7
index 268d999efb..7c2e9db604 100644
@@ -158,13 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded:
        @$(call step_start,download)
        $(call prepare-per-package-
 directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES))
        $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call
 $(hook))$(sep))
-# Only show the download message if it isn't already downloaded
-       $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \
-               if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \
-                       $(call MESSAGE,"Downloading") ; \
-                       break ; \
-               fi ; \
-       done
+       @$(call MESSAGE,"Downloading") ; \
        $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call

DOWNLOAD,$(p),$(PKG))$(sep))

        $(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call
$(hook))$(sep))
        $(Q)mkdir -p $(@D)
support/download/dl- renamed from support/download/dl-wrapper +0 -0

wrapper
index 3315bd410e..ab22ca7e4f 100755

support/download/dl-wrapper +0 -0
@@ -21,11 +21,12 @@ export

BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"

main() {
local OPT OPTARG

support/download/dl-wrapper +4 -2
-    local backend output hfile recurse quiet rc
+    local backend output hfile recurse quiet rc force
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    force=false
+    while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -38,6 +39,7 @@ main() {
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
         q)  quiet="-q";;
+        F)  force=true;;
         :)  error "option '%s' expects a mandatory argument\n"
 "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;

esac

support/download/dl-wrapper +0 -0
@@ -67,7 +69,7 @@ main() {
     # - matches all its hashes: do not download it again and exit

promptly
# - fails at least one of its hashes: force a re-download # - there's no hash (but a .hash file): consider it a hard error

support/download/dl-wrapper +1 -1
-    if [ -e "${output}" ]; then
+    if ! ${force} && [ -e "${output}" ]; then
         if support/download/check-hash ${quiet} "${hfile}"

"${output}" "${output##*/}"; then

             exit 0
         elif [ ${?} -ne 2 ]; then

--
2.20.1

ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.

#15 lore raw

PATCH 1/1 infra: add the transient download mechanism

In reply to Yann E. MORIN

Hello,

Quoted 5 lines · On Wed, 15 Jan 2020 21:37:53 +0100 "Yann E. MORIN" <[email protected]>…

NOTE: NOT TO BE COMMITTED!

This patch is a proof of concept for the transient download mechanism, that would help using branches as versions, and keep using the head of the branch each time a build started.

So what is the status of this patch? I see Yann you marked it as to be ignored by patchwork, so we don't have it in patchwork.

I think there's generally been some positive feedback, with Arnout saying:

""
Bottom line: even though repo is the better tool to manage the trees both for
development and release, the transient mechanism is a good quick win. ""

Quoted 9 lines

A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT = YES

this means that the download infra will not use any already downloaded archive, and will instead always download it as if missing.

Since the check is done in the download wrapper, we have no TOCTOU race in case two bulds would attempt the same transient download: the archive is only replaced ato,mically as usual.

Typo: atomically

Quoted 7 lines

So, if the package uses a branch as version, the branch's HEAD at that very moment will be redownloaded.

This stil has the drawback that two builds in parallel may get slightly different content for the same branch, and the first build could end up using the download of the second build:

Quoted 8 lines

build-1 build-2

download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract

That is not a drawback, that is exactly what you expect with this feature.

Quoted 11 lines

Furthermore, even with a single build, it might get what it expects:

developer-1 developer-2

git push branch
trigger CI git push branch
[...]
download

In that case the build of delopper-1 would get the code of developper-2 who pushed on the same branch.

Ditto here: this is exactly what we expect for a transient download feature. If developer-1 wants to test only his own feature, then he has to trigger CI on a branch that only he has control over.

There is however one drawback:

* You run a build (1), which downloads package A from a Git repo, and places the tarball in global DL_DIR.

* You run a different build (2), which downloads the same package A from the Git repo. Due to the transient download mechanism being enabled for package A, it re-downloads it, and places an updated tarball in the global DL_DIR.

* Back to build (1), you run "make legal-info", which picks up the updated tarball, which was generated by build (2).

So, in build (1), your image uses the "previous" version of package A, while your legal-info contains the "new" version of package A.

To solve this, one solution would be to use a per-build download directory for the transient packages, rather than the global DL_DIR.

But I think this could be a separate improvement. Indeed, the transient download mechanism is clearly oriented towards development/CI workflow, where consistency of legal-info may not be a primary concern. For a final release build, you would not use TRANSIENT_DOWNLOAD = YES.

For example, custom packages that make use of TRANSIENT_DOWNLOAD could be implemented like this:

ifeq ($(BR2_DEVELOPER_MODE),y)
<pkg>_TRANSIENT_DOWNLOAD = YES
<pkg>_VERSION = some-branch
else
<pkg>_VERSION = some-tag
endif

So, while in development mode, users and CI use BR2_DEVELOPER_MODE=y, but for final releases, BR2_DEVELOPER_MODE is disabled, which ensures we have a fully reproducible build, and consistent legal-info output.

Yann, do you think you could resubmit your patch, for actual merging ? Of course, we don't need the busybox.mk change.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#16 lore raw

PATCH 1/1 infra: add the transient download mechanism

Thomas, All,

On 2020-04-08 08:39 +0200, Thomas Petazzoni spake thusly:

Quoted 9 lines
Quoted 3 lines · On Wed, 15 Jan 2020 21:37:53 +0100 "Yann E. MORIN" <[email protected]>…

This patch is a proof of concept for the transient download mechanism, that would help using branches as versions, and keep using the head of the branch each time a build started.

So what is the status of this patch? I see Yann you marked it as to be ignored by patchwork, so we don't have it in patchwork.

Quoted 7 lines

I think there's generally been some positive feedback, with Arnout saying:

""
Bottom line: even though repo is the better tool to manage the trees both for
development and release, the transient mechanism is a good quick win. ""

I will address all the questions and comments in the review, and will respin it for proper consideration.

[--SNIP--]

Quoted 13 lines
Quoted 4 lines

This stil has the drawback that two builds in parallel may get slightly different content for the same branch, and the first build could end up using the download of the second build:

Quoted 8 lines

build-1 build-2

download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract

Quoted 2 lines

That is not a drawback, that is exactly what you expect with this feature.

Sorry, the two builds are supposed to be running in parallel... For build 1, it would get the download from build 2, which is not what build 1 expects.

But drawback is not the correct word. This is a pitfall.

Quoted 9 lines
Quoted 9 lines

Furthermore, even with a single build, it might get what it expects:

developer-1 developer-2

git push branch
trigger CI git push branch
[...]
download

Quoted 6 lines
Quoted 2 lines

In that case the build of delopper-1 would get the code of developper-2 who pushed on the same branch.

Ditto here: this is exactly what we expect for a transient download feature. If developer-1 wants to test only his own feature, then he has to trigger CI on a branch that only he has control over.

Yes, this is not a drawback, but a pitfall.

Quoted 5 lines

There is however one drawback:

* You run a build (1), which downloads package A from a Git repo, and places the tarball in global DL_DIR.

Quoted 7 lines

* You run a different build (2), which downloads the same package A from the Git repo. Due to the transient download mechanism being enabled for package A, it re-downloads it, and places an updated tarball in the global DL_DIR.

* Back to build (1), you run "make legal-info", which picks up the updated tarball, which was generated by build (2).

This is not a drawback; this is another pitfall.

Quoted 5 lines

So, in build (1), your image uses the "previous" version of package A, while your legal-info contains the "new" version of package A.

To solve this, one solution would be to use a per-build download directory for the transient packages, rather than the global DL_DIR.

Meh... Or rather, make legal-info impossible for a package that is transient, like when it is 'local'. This is the best solution.

Quoted 8 lines

But I think this could be a separate improvement. Indeed, the transient download mechanism is clearly oriented towards development/CI workflow, where consistency of legal-info may not be a primary concern. For a final release build, you would not use TRANSIENT_DOWNLOAD = YES.

For example, custom packages that make use of TRANSIENT_DOWNLOAD could be implemented like this:

Quoted 6 lines

ifeq ($(BR2_DEVELOPER_MODE),y)
<pkg>_TRANSIENT_DOWNLOAD = YES
<pkg>_VERSION = some-branch
else
<pkg>_VERSION = some-tag
endif

The more I think about it, the less proud I am to have come up with this transient idea. Using unstable references is complete madness to begin with and is so flawed by design... We should not even provide such a feature. But we can't stop people from being... people... :-[

Quoted 5 lines

So, while in development mode, users and CI use BR2_DEVELOPER_MODE=y, but for final releases, BR2_DEVELOPER_MODE is disabled, which ensures we have a fully reproducible build, and consistent legal-info output.

Yann, do you think you could resubmit your patch, for actual merging ?

Yes I will.

Quoted 1 line

Of course, we don't need the busybox.mk change.

We don't? How comne" ;-)

Quoted 7 lines

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#17 lore raw

PATCH 1/1 infra: add the transient download mechanism

Hello,

Quoted 9 lines · On Wed, 8 Apr 2020 21:27:53 +0200 "Yann E. MORIN" <[email protected]>…
Quoted 9 lines
Quoted 8 lines

build-1 build-2

download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract

Quoted 8 lines
Quoted 2 lines

That is not a drawback, that is exactly what you expect with this feature.

Sorry, the two builds are supposed to be running in parallel... For build 1, it would get the download from build 2, which is not what build 1 expects.

But drawback is not the correct word. This is a pitfall.

Again, I think that what you're describing is exactly what we expect: if you want to a branch, then you are guaranteed that you will do a build with a state of the branch that is at least the one that was present at the time you started the build. But it can be a later state of that branch, if it gets updated in the mean time.

Quoted 8 lines
Quoted 5 lines

So, in build (1), your image uses the "previous" version of package A, while your legal-info contains the "new" version of package A.

To solve this, one solution would be to use a per-build download directory for the transient packages, rather than the global DL_DIR.

Meh... Or rather, make legal-info impossible for a package that is transient, like when it is 'local'. This is the best solution.

Sounds good to me indeed.

Quoted 15 lines
Quoted 8 lines

But I think this could be a separate improvement. Indeed, the transient download mechanism is clearly oriented towards development/CI workflow, where consistency of legal-info may not be a primary concern. For a final release build, you would not use TRANSIENT_DOWNLOAD = YES.

For example, custom packages that make use of TRANSIENT_DOWNLOAD could be implemented like this:

Quoted 6 lines

ifeq ($(BR2_DEVELOPER_MODE),y)
<pkg>_TRANSIENT_DOWNLOAD = YES
<pkg>_VERSION = some-branch
else
<pkg>_VERSION = some-tag
endif

Quoted 4 lines

The more I think about it, the less proud I am to have come up with this transient idea. Using unstable references is complete madness to begin with and is so flawed by design... We should not even provide such a feature. But we can't stop people from being... people... :-[

I think it's something that a number of people want, and that makes sense for a number of use-cases.

It's all about moving away from the idea that Buildroot is only the "final" integration tool. It is also used during application development, it is also used to do CI, etc. And that requires to relax a bit the very strict reproducibility requirements that we enforce for the "final integration" use case.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com