infra: add the transient download mechanism
PATCH 1/1 infra: add the transient download mechanism
NOTE: NOT TO BE COMMITTED!
This patch is a proof of concept for the transient download mechanism, that would help using branches as versions, and keep using the head of the branch each time a build started.
A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT = YES
this means that the download infra will not use any already downloaded archive, and will instead always download it as if missing.
Since the check is done in the download wrapper, we have no TOCTOU race in case two bulds would attempt the same transient download: the archive is only replaced ato,mically as usual.
So, if the package uses a branch as version, the branch's HEAD at that very moment will be redownloaded.
This stil has the drawback that two builds in parallel may get slightly different content for the same branch, and the first build could end up using the download of the second build:
build-1 build-2
download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract
Furthermore, even with a single build, it might get what it expects:
developer-1 developer-2
git push branch
trigger CI git push branch
[...]
download
In that case the build of delopper-1 would get the code of developper-2 who pushed on the same branch.
For people who are aiming at their feet, we're now providing them with a loaded gun. ;-]
--- docs/manual/adding-packages-generic.txt | 16 +++++++++++++--- package/busybox/busybox.mk | 1 + package/pkg-download.mk | 1 + package/pkg-generic.mk | 8 +------- support/download/dl-wrapper | 8 +++++--- 5 files changed, 21 insertions(+), 13 deletions(-)
docs/manual/adding-packages-generic.txt +13 -3
@@ -206,12 +206,14 @@ information is (assuming the package name is +libfoo+) : ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+ + .Note: -Using a branch name as +FOO_VERSION+ is not supported, because it does -not and can not work as people would expect it should: +Using a branch name as +FOO_VERSION+, although technically possible, +is highly discouraged, because it does not and can not work as people +would expect it should: + 1. due to local caching, Buildroot will not re-fetch the repository, so people who expect to be able to follow the remote repository - would be quite surprised and disappointed; + would be quite surprised and disappointed (but see + +LIBFOO_DOWNLOAD_TRANSIENT+, later); 2. because two builds can never be perfectly simultaneous, and because the remote repository may get new commits on the branch anytime, two users, using the same Buildroot tree and building the same
@@ -342,6 +344,14 @@ not and can not work as people would expect it should: submodules when they contain bundled libraries, in which case we prefer to use those libraries from their own package. +* +LIBFOO_DOWNLOAD_TRANSIENT+ can be set to +YES+ or +NO+ (the default). + If set to +YES+, the download for that package will be attempted at + every build; any already downloaded archive is ignored as if missing. + This may help when a branch is specified in +LIBFOO_VERSION+, and the + head/tip of the branch is to be built, like a CI pipeline would need + for example. This still suffers from the other issues listed above + about using branches, though. + * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components (directories) that tar must strip from file names on extraction. The tarball for most packages has one leading component named
package/busybox/busybox.mk +1 -0
@@ -9,6 +9,7 @@ BUSYBOX_SITE = http://www.busybox.net/downloads BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2 BUSYBOX_LICENSE = GPL-2.0 BUSYBOX_LICENSE_FILES = LICENSE +BUSYBOX_DOWNLOAD_TRANSIENT = YES define BUSYBOX_HELP_CMDS @echo ' busybox-menuconfig - Run BusyBox menuconfig'
package/pkg-download.mk +1 -0
@@ -108,6 +108,7 @@ define DOWNLOAD -n '$($(2)_BASENAME_RAW)' \ -N '$($(2)_RAWNAME)' \ -o '$($(2)_DL_DIR)/$(notdir $(1))' \ + $(if $($(2)_DOWNLOAD_TRANSIENT),-F) \ $(if $($(2)_GIT_SUBMODULES),-r) \ $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \ $(QUIET) \
package/pkg-generic.mk +1 -7
@@ -158,13 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded: @$(call step_start,download) $(call prepare-per-package-directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) -# Only show the download message if it isn't already downloaded - $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \ - if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \ - $(call MESSAGE,"Downloading") ; \ - break ; \ - fi ; \ - done + @$(call MESSAGE,"Downloading") ; \ $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p),$(PKG))$(sep)) $(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) $(Q)mkdir -p $(@D)
support/download/dl-wrapper +5 -3
@@ -21,11 +21,12 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e" main() { local OPT OPTARG - local backend output hfile recurse quiet rc + local backend output hfile recurse quiet rc force local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + force=false + while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do case "${OPT}" in c) cset="${OPTARG}";; d) dl_dir="${OPTARG}";;
@@ -38,6 +39,7 @@ main() { f) filename="${OPTARG}";; u) uris+=( "${OPTARG}" );; q) quiet="-q";; + F) force=true;; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";; esac
@@ -67,7 +69,7 @@ main() { # - matches all its hashes: do not download it again and exit promptly # - fails at least one of its hashes: force a re-download # - there's no hash (but a .hash file): consider it a hard error - if [ -e "${output}" ]; then + if ! ${force} && [ -e "${output}" ]; then if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then exit 0 elif [ ${?} -ne 2 ]; then
Signature · 1 line
2.20.1
PATCH 1/1 infra: add the transient download mechanism
Hi
Quoted 10 lines · Yann E. MORIN
NOTE: NOT TO BE COMMITTED!
This patch is a proof of concept for the transient download mechanism, that would help using branches as versions, and keep using the head of the branch each time a build started.
A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT= YES
Why we can not just have like
PKG_TRANSIENT_OVERRIDE_FILE="ci.mk"
ci.mk
FOO_DOWNLOAD_TRANSIENT = YES
FOO_REVISION="<TAG> e/o BRANCH"
Michael
Quoted 172 lines
this means that the download infra will not use any already downloaded archive, and will instead always download it as if missing.
Since the check is done in the download wrapper, we have no TOCTOU race in case two bulds would attempt the same transient download: the archive is only replaced ato,mically as usual.
So, if the package uses a branch as version, the branch's HEAD at that very moment will be redownloaded.
This stil has the drawback that two builds in parallel may get slightly different content for the same branch, and the first build could end up using the download of the second build:
build-1 build-2
download
| download
| |
save to dl-dir |
[...] save to dl-dir
extractFurthermore, even with a single build, it might get what it expects:
developer-1 developer-2
git push branch
trigger CI git push branch
[...]
downloadIn that case the build of delopper-1 would get the code of developper-2 who pushed on the same branch.
For people who are aiming at their feet, we're now providing them with a loaded gun. ;-]
Signed-off-by Yann E. MORIN <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Vincent Fazio <[email protected]>Cc Michael Nazzareno Trimarchi <[email protected]>Cc Chris Packham <[email protected]>Cc Nicolas Carrier <[email protected]>Cc Adam Duskett <[email protected]>--- docs/manual/adding-packages-generic.txt | 16 +++++++++++++--- package/busybox/busybox.mk | 1 + package/pkg-download.mk | 1 + package/pkg-generic.mk | 8 +------- support/download/dl-wrapper | 8 +++++--- 5 files changed, 21 insertions(+), 13 deletions(-)docs/manual/adding-packages-generic.txt +13
-3@@ -206,12 +206,14 @@ information is (assuming the package name is +libfoo+) : ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+ + .Note: -Using a branch name as +FOO_VERSION+ is not supported, because it does -not and can not work as people would expect it should: +Using a branch name as +FOO_VERSION+, although technically possible, +is highly discouraged, because it does not and can not work as people +would expect it should: + 1. due to local caching, Buildroot will not re-fetch the repository, so people who expect to be able to follow the remote repository - would be quite surprised and disappointed; + would be quite surprised and disappointed (but see + +LIBFOO_DOWNLOAD_TRANSIENT+, later); 2. because two builds can never be perfectly simultaneous, and because the remote repository may get new commits on the branch anytime, two users, using the same Buildroot tree and building the same@@ -342,6 +344,14 @@ not and can not work as people would expect it should: submodules when they contain bundled libraries, in which case we prefer to use those libraries from their own package. +* +LIBFOO_DOWNLOAD_TRANSIENT+ can be set to +YES+ or +NO+ (the default). + If set to +YES+, the download for that package will be attempted at + every build; any already downloaded archive is ignored as if missing. + This may help when a branch is specified in +LIBFOO_VERSION+, and the + head/tip of the branch is to be built, like a CI pipeline would need + for example. This still suffers from the other issues listed above + about using branches, though. + * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components (directories) that tar must strip from file names on extraction. The tarball for most packages has one leading component namedpackage/busybox/busybox.mk +1
-0@@ -9,6 +9,7 @@ BUSYBOX_SITE = http://www.busybox.net/downloads BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2 BUSYBOX_LICENSE = GPL-2.0 BUSYBOX_LICENSE_FILES = LICENSE +BUSYBOX_DOWNLOAD_TRANSIENT = YES define BUSYBOX_HELP_CMDS @echo ' busybox-menuconfig - Run BusyBox menuconfig'package/pkg-download.mk +1
-0@@ -108,6 +108,7 @@ define DOWNLOAD -n '$($(2)_BASENAME_RAW)' \ -N '$($(2)_RAWNAME)' \ -o '$($(2)_DL_DIR)/$(notdir $(1))' \ + $(if $($(2)_DOWNLOAD_TRANSIENT),-F) \ $(if $($(2)_GIT_SUBMODULES),-r) \ $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \ $(QUIET) \package/pkg-generic.mk +1
-7@@ -158,13 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded: @$(call step_start,download) $(call prepare-per-package-directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) -# Only show the download message if it isn't already downloaded - $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \ - if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \ - $(call MESSAGE,"Downloading") ; \ - break ; \ - fi ; \ - done + @$(call MESSAGE,"Downloading") ; \ $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p),$(PKG))$(sep)) $(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) $(Q)mkdir -p $(@D)support/download/dl-wrapper +5
-4@@ -21,11 +21,12 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e" main() { local OPT OPTARG - local backend output hfile recurse quiet rc + local backend output hfile recurse quiet rc force local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + force=false + while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do case "${OPT}" in c) cset="${OPTARG}";; d) dl_dir="${OPTARG}";;@@ -38,6 +39,7 @@ main() { f) filename="${OPTARG}";; u) uris+=( "${OPTARG}" );; q) quiet="-q";; + F) force=true;; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";; esac@@ -67,7 +69,7 @@ main() { # - matches all its hashes: do not download it again and exit promptly # - fails at least one of its hashes: force a re-download # - there's no hash (but a .hash file): consider it a hard error - if [ -e "${output}" ]; then + if ! ${force} && [ -e "${output}" ]; then if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then exit 0 elif [ ${?} -ne 2 ]; then --2.20.1
Signature · 4 lines
| Michael Nazzareno Trimarchi Amarula Solutions BV | | COO - Founder Cruquiuskade 47 | | +31(0)851119172 Amsterdam 1018 AM NL | | [`as] http://www.amarulasolutions.com |
PATCH 1/1 infra: add the transient download mechanism
Quoted 9 lines · Michael Nazzareno Trimarchi
Hi
Quoted 6 lines · Yann E. MORIN
NOTE: NOT TO BE COMMITTED!
This patch is a proof of concept for the transient download mechanism, that would help using branches as versions, and keep using the head of the branch each time a build started.
Quoted 5 lines
Quoted 3 lines
A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT= YESWhy we can not just have like
PKG_TRANSIENT_OVERRIDE_FILE="ci.mk"
You can use BR2_PACKAGE_OVERRIDE_FILE for this.
Quoted 5 lines
ci.mk
FOO_DOWNLOAD_TRANSIENT= YES
FOO_REVISION="<TAG> e/o BRANCH"
... but not for this, unfortunately.
There would be two ways to implement this after all:
1. implement the idea which has floated around already a couple of times to delay evaluation of most of inner-generic-package (and other infras) until later, and include the override file only after the package .mk files;
2. create new override options FOO_OVERRIDE_VERSION, ... and use those instead of the package options - similar to OVERRIDE_SRCDIR.
Both are totally orthogonal to the TRANSIENT idea, of course.
Regards,
Arnout
PATCH 1/1 infra: add the transient download mechanism
In reply to Yann E. MORINHello Yann,
One small remark, to be able to reproduce the build, on top of this mechanism, there should be:
* a way to store the actual VCS revision used for the build (to at least know what was just built!)
* a way to force the build to re-use a particular set of revisions
That's features not easy to get right, but which are already handled by tools like google's repo (or partly, by git submodules).
That's part of why I don't think that this kind of feature is a good thing to integrate into buildroot. I prefer to delegate it to tools meant for that (using a mix of local packages and OVERRIDE). And like said before, I think it's preferable to have a optional feature to prevent branches from being used, making the build fail.
Quoted 217 lines · Yann E. MORIN
NOTE: NOT TO BE COMMITTED!
This patch is a proof of concept for the transient download mechanism,
that would help using branches as versions, and keep using the head of
the branch each time a build started.A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT= YESthis means that the download infra will not use any already downloaded
archive, and will instead always download it as if missing.Since the check is done in the download wrapper, we have no TOCTOU race
in case two bulds would attempt the same transient download: the archive
is only replaced ato,mically as usual.So, if the package uses a branch as version, the branch's HEAD at that
very moment will be redownloaded.This stil has the drawback that two builds in parallel may get slightly
different content for the same branch, and the first build could end up
using the download of the second build:build-1 build-2
download
| download
| |
save to dl-dir |
[...] save to dl-dir
extractFurthermore, even with a single build, it might get what it expects:
developer-1 developer-2
git push branch
trigger CI git push branch
[...]
downloadIn that case the build of delopper-1 would get the code of developper-2
who pushed on the same branch.For people who are aiming at their feet, we're now providing them with
a loaded gun. ;-]Signed-off-by Yann E. MORIN <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Vincent Fazio <[email protected]>Cc Michael Nazzareno Trimarchi <[email protected]>Cc Chris Packham <[email protected]>Cc Nicolas Carrier <[email protected]>Cc Adam Duskett <[email protected]>--- docs/manual/adding-packages-generic.txt | 16 +++++++++++++--- package/busybox/busybox.mk | 1 + package/pkg-download.mk | 1 + package/pkg-generic.mk | 8 +------- support/download/dl-wrapper | 8 +++++--- 5 files changed, 21 insertions(+), 13 deletions(-)docs/manual/adding-packages-generic.txt +1
-1@@ -206,12 +206,14 @@ information is (assuming the package name is +libfoo+) : ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+ + .Note: -Using a branch name as +FOO_VERSION+ is not supported, because itdoes
docs/manual/adding-packages-generic.txt +2
-1-not and can not work as people would expect it should: +Using a branch name as +FOO_VERSION+, although technically possible, +is highly discouraged, because it does not and can not work aspeople
+would expect it should:
+
1. due to local caching, Buildroot will not re-fetch the
repository,
so people who expect to be able to follow the remote repositorydocs/manual/adding-packages-generic.txt +2
-1- would be quite surprised and disappointed; + would be quite surprised and disappointed (but see + +LIBFOO_DOWNLOAD_TRANSIENT+, later); 2. because two builds can never be perfectly simultaneous, andbecause
the remote repository may get new commits on the branch
anytime,
two users, using the same Buildroot tree and building the samedocs/manual/adding-packages-generic.txt +0
-0@@ -342,6 +344,14 @@ not and can not work as people would expect itshould:
submodules when they contain bundled libraries, in which case we prefer to use those libraries from their own package.+* +LIBFOO_DOWNLOAD_TRANSIENT+ can be set to +YES+ or +NO+ (the default).
+ If set to +YES+, the download for that package will be attempted at
+ every build; any already downloaded archive is ignored as if missing.
+ This may help when a branch is specified in +LIBFOO_VERSION+, and the
+ head/tip of the branch is to be built, like a CI pipeline would needdocs/manual/adding-packages-generic.txt +3
-0+ for example. This still suffers from the other issues listed above + about using branches, though. + * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components (directories) that tar must strip from file names on extraction. The tarball for most packages has one leading component namedpackage/busybox/busybox.mk +1
-0@@ -9,6 +9,7 @@ BUSYBOX_SITE = http://www.busybox.net/downloads BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2 BUSYBOX_LICENSE = GPL-2.0 BUSYBOX_LICENSE_FILES = LICENSE +BUSYBOX_DOWNLOAD_TRANSIENT = YES define BUSYBOX_HELP_CMDS @echo ' busybox-menuconfig - Run BusyBox menuconfig'package/pkg-download.mk +1
-0@@ -108,6 +108,7 @@ define DOWNLOAD -n '$($(2)_BASENAME_RAW)' \ -N '$($(2)_RAWNAME)' \ -o '$($(2)_DL_DIR)/$(notdir $(1))' \ + $(if $($(2)_DOWNLOAD_TRANSIENT),-F) \ $(if $($(2)_GIT_SUBMODULES),-r) \ $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \$(QUIET) \package/pkg-generic.mk +1
-7@@ -158,13 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded: @$(call step_start,download) $(call prepare-per-package- directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) -# Only show the download message if it isn't already downloaded - $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \ - if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \ - $(call MESSAGE,"Downloading") ; \ - break ; \ - fi ; \ - done + @$(call MESSAGE,"Downloading") ; \ $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(callDOWNLOAD,$(p),$(PKG))$(sep))
$(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) $(Q)mkdir -p $(@D)support/download/dl- renamed from support/download/dl-wrapper +0
-0wrapper
index3315bd410e..ab22ca7e4f100755support/download/dl-wrapper +0
-0@@ -21,11 +21,12 @@ exportBR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
main(){
local OPT OPTARGsupport/download/dl-wrapper +4
-2- local backend output hfile recurse quiet rc + local backend output hfile recurse quiet rc force local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + force=false + while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do case "${OPT}" in c) cset="${OPTARG}";; d) dl_dir="${OPTARG}";;@@ -38,6 +39,7 @@ main() { f) filename="${OPTARG}";; u) uris+=( "${OPTARG}" );; q) quiet="-q";; + F) force=true;; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";;esac
support/download/dl-wrapper +0
-0@@ -67,7 +69,7 @@ main() { # - matches all its hashes: do not download it again and exitpromptly
# - fails at least one of its hashes: force a re-download # - there's no hash (but a .hash file): consider it a hard errorsupport/download/dl-wrapper +1
-1- if [ -e "${output}" ]; then + if ! ${force} && [ -e "${output}" ]; then if support/download/check-hash ${quiet} "${hfile}""${output}" "${output##*/}"; then
exit 0 elif [ ${?} -ne 2 ]; then--
2.20.1ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.
PATCH 1/1 infra: add the transient download mechanism
Hello,
Quoted 4 lines · On Thu, 16 Jan 2020 09:11:32 +0000 Nicolas Carrier…
One small remark, to be able to reproduce the build, on top of this mechanism, there should be:
* a way to store the actual VCS revision used for the build (to at least know what was just built!)
That seems quite easy to do.
Quoted 1 line
* a way to force the build to re-use a particular set of revisions
I don't see how that can be done however, and how that would work from a user interface point of view. Sure the first build could store the exact Git commit used for building each transient package, and store that "somewhere", and then re-use that for the next build. But how does the user decide whether what he wants is to re-do the build with the same commits or not ? Where is that "somewhere" ?
Quoted 3 lines
That's part of why I don't think that this kind of feature is a good thing to integrate into buildroot. I prefer to delegate it to tools meant for that (using a mix of local packages and OVERRIDE).
Different people/companies have different work flows. We have people telling us that using local packages and OVERRIDE for their CI is not very convenient/practical.
Quoted 2 lines
And like said before, I think it's preferable to have a optional feature to prevent branches from being used, making the build fail.
The feature proposed by Yann *is* optional, as it's conditional on having a specific variable set in the package .mk file.
However, one question that is raised by Yann's patch is whether we want to forbid using branches when TRANSIENT_DOWNLOAD is NO.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/1 infra: add the transient download mechanism
Hi
Quoted 22 lines · Thomas Petazzoni
Hello,
Quoted 4 lines · On Thu, 16 Jan 2020 09:11:32 +0000 Nicolas Carrier…
One small remark, to be able to reproduce the build, on top of this mechanism, there should be:
* a way to store the actual VCS revision used for the build (to at least know what was just built!)That seems quite easy to do.
Quoted 1 line
* a way to force the build to re-use a particular set of revisions
I don't see how that can be done however, and how that would work from a user interface point of view. Sure the first build could store the exact Git commit used for building each transient package, and store that "somewhere", and then re-use that for the next build. But how does the user decide whether what he wants is to re-do the build with the same commits or not ? Where is that "somewhere" ?
Quoted 7 lines
Quoted 3 lines
That's part of why I don't think that this kind of feature is a good thing to integrate into buildroot. I prefer to delegate it to tools meant for that (using a mix of local packages and OVERRIDE).
Different people/companies have different work flows. We have people telling us that using local packages and OVERRIDE for their CI is not very convenient/practical.
We are using a similar setup of Nicolas. Right now we have
CI and reproducible build using external tools. What I would like to solve with buildroot is:
- make the build reproducible
- implement topic build on custom patch using CI
- make tagging and branching easy
All of this I have already with repo + buildroot
Michael
Quoted 17 lines
Quoted 3 lines
And like said before, I think it's preferable to have a optional feature to prevent branches from being used, making the build fail.
The feature proposed by Yann *is* optional, as it's conditional on having a specific variable set in the package .mk file.
However, one question that is raised by Yann's patch is whether we want to forbid using branches when
TRANSIENT_DOWNLOADis NO.Best regards,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 4 lines
| Michael Nazzareno Trimarchi Amarula Solutions BV | | COO - Founder Cruquiuskade 47 | | +31(0)851119172 Amsterdam 1018 AM NL | | [`as] http://www.amarulasolutions.com |
PATCH 1/1 infra: add the transient download mechanism
Quoted 7 lines · On Thu, 16 Jan 2020 10:56:53 +0100 Michael Nazzareno Trimarchi…
We are using a similar setup of Nicolas. Right now we have
CI and reproducible build using external tools. What I would like to solve with buildroot is:- make the build reproducible
- implement topic build on custom patch using CI
- make tagging and branching easy
Could you give more details, because these items are quite vague/fuzzy, and I don't understand from a very practical point of view what you mean. Could you describe some example workflow, what would be your expectation of Buildroot's behavior, etc.
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/1 infra: add the transient download mechanism
Hi
Quoted 8 lines · Thomas Petazzoni
Quoted 4 lines · On Thu, 16 Jan 2020 10:56:53 +0100 Michael Nazzareno Trimarchi…
We are using a similar setup of Nicolas. Right now we have
CI and reproducible build using external tools. What I would like to solve with buildroot is:
Quoted 9 lines
Quoted 3 lines
- make the build reproducible
- implement topic build on custom patch using CI
- make tagging and branching easyCould you give more details, because these items are quite vague/fuzzy, and I don't understand from a very practical point of view what you mean. Could you describe some example workflow, what would be your expectation of Buildroot's behavior, etc.
Suppose you have this manifest
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remote name="amarula-gitea"
fetch=".."
review="https://gerrit-review.amarulasolutions.com" /><remote name="prj"
fetch="ssh://server1/home/gitrepos/"
revision="master" /><default revision="prj2"
remote="amarula-gitea"
sync-j="4"
sync-c="true" /> <project path="app1" name="smile" remote="prj" /> <project path="buildroot" name="prj/buildroot" /> <project path="linux" name="prj/linux" /> <project path="build" name="prj/build" >
<copyfile src="root.mk" dest="Makefile" />
</project>
<project path="library/lib1" name="prj/lib1" /> <project path="u-boot" name="prj/u-boot" />
</manifest>
This one describe the project and we have override part for the project in the manifest. For each version we can tag and each repo and create a snapshot manifest of the project. Manifest will have revison set to revision="refs/tags/<some tag>"
If I need to emit a release starting from some tag I can branch it and apply patch on top of this branch and everything
can be stored as a separed manifest. Using jenkins trigger on topic upload on gerrit we can easily build a version with some patchset applied and if we want to re-build some version we can just use the right tagged manifest
Michael
Quoted 7 lines
Thanks,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 4 lines
| Michael Nazzareno Trimarchi Amarula Solutions BV | | COO - Founder Cruquiuskade 47 | | +31(0)851119172 Amsterdam 1018 AM NL | | [`as] http://www.amarulasolutions.com |
PATCH 1/1 infra: add the transient download mechanism
Michael, All,
On 2020-01-16 11:15 +0100, Michael Nazzareno Trimarchi spake thusly:
Quoted 6 lines
Quoted 4 lines · Thomas Petazzoni
Quoted 4 lines
We are using a similar setup of Nicolas. Right now we have
CI and reproducible build using external tools. What I would like to solve with buildroot is:
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
- make the build reproducible
- implement topic build on custom patch using CI
- make tagging and branching easyCould you give more details, because these items are quite vague/fuzzy, and I don't understand from a very practical point of view what you mean. Could you describe some example workflow, what would be your expectation of Buildroot's behavior, etc.
I have to admit that, like Thomas, I do not completely understand what you're doing or trying to achieve...
Quoted 1 line
Suppose you have this manifest
[--SNIP--]
Quoted 11 lines
This one describe the project and we have override part for the project in the manifest. For each version we can tag and each repo and create a snapshot manifest of the project. Manifest will have revison set to revision="refs/tags/<some tag>"
If I need to emit a release starting from some tag I can branch it and apply patch on top of this branch and everything
can be stored as a separed manifest. Using jenkins trigger on topic upload on gerrit we can easily build a version with some patchset applied and if we want to re-build some version we can just use the right tagged manifest
I don't understand the problem that using repo solves.
repo's manifest identifies packages, where they are from, and what version to use for each. That is exactly what the _VERSION and _SITE do in a .mk file to begin with...
Using repo and a manifest only seem interesting when one wants to have their packages in the same directory structure as Buildroot, and use _SITE_METHOD = local. But in that case, a repo manifest is very akin to git submodules.
And in that case, there is no need to be able to uses branches as _VERSION, since the packages sources jsut 'follow' the branch buildroot is on, and that repo checked out...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 infra: add the transient download mechanism
Hi
Quoted 35 lines · Yann E. MORIN
Michael, All,
On 2020-01-16 11:15 +0100, Michael Nazzareno Trimarchi spake thusly:
Quoted 6 lines
Quoted 4 lines · Thomas Petazzoni
Quoted 4 lines
We are using a similar setup of Nicolas. Right now we have
CI and reproducible build using external tools. What I would like to solve with buildroot is:Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
- make the build reproducible
- implement topic build on custom patch using CI
- make tagging and branching easyCould you give more details, because these items are quite vague/fuzzy, and I don't understand from a very practical point of view what you mean. Could you describe some example workflow, what would be your expectation of Buildroot's behavior, etc.
I have to admit that, like Thomas, I do not completely understand what you're doing or trying to achieve...
Quoted 1 line
Suppose you have this manifest
[--SNIP--]
Quoted 11 lines
This one describe the project and we have override part for the project in the manifest. For each version we can tag and each repo and create a snapshot manifest of the project. Manifest will have revison set to revision="refs/tags/<some tag>"
If I need to emit a release starting from some tag I can branch it and apply patch on top of this branch and everything
can be stored as a separed manifest. Using jenkins trigger on topic upload on gerrit we can easily build a version with some patchset applied and if we want to re-build some version we can just use the right tagged manifest
Quoted 5 lines
I don't understand the problem that using repo solves.
repo's manifest identifies packages, where they are from, and what version to use for each. That is exactly what the
_VERSIONand_SITEdo in a .mk file to begin with...
Yes, definitely you can manage in the same way but having one point as a manifest file where I can manage only project I change during the product life avoid
me to change those when I create a specific tag. repo let me to tag in automatic way and let buildroot to be tagged only on project that I download from outside. I'm not saying here that the direction you are taken is wrong but not all the companies will use it.
BTW my way to write english is terrible, I like having a desk and discuss face to face. Make more sense to me to discuss in Fossdem if you will be there
Michael
Quoted 13 lines
Using repo and a manifest only seem interesting when one wants to have their packages in the same directory structure as Buildroot, and use
_SITE_METHOD= local. But in that case, a repo manifest is very akin to git submodules.And in that case, there is no need to be able to uses branches as
_VERSION, since the packages sources jsut 'follow' the branch buildroot is on, and that repo checked out...Regards,
Yann E. MORIN.
Quoted 7 lines
--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
--
| Michael Nazzareno Trimarchi Amarula Solutions BV | | COO - Founder Cruquiuskade 47 | | +31(0)851119172 Amsterdam 1018 AM NL | | [`as] http://www.amarulasolutions.com |
PATCH 1/1 infra: add the transient download mechanism
In reply to Thomas PetazzoniHello,
Quoted 4 lines
However, one question that is raised by Yann's patch is whether we want
to forbid using branches whenTRANSIENT_DOWNLOADis NO.
Just to be sure I understand, when you say "when TRANSIENT_DOWNLOAD is NO.", it'd be the same as if TRANSIENT_DOWNLOAD download is not defined at all, right?
If it's the case, then that seems to me like a good idea, because it renders explicit the risk taken by the user when using a branch (which can be done unknowingly ATM).
Quoted 10 lines
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.
PATCH 1/1 infra: add the transient download mechanism
Quoted 3 lines · On Thu, 16 Jan 2020 10:01:09 +0000 Nicolas Carrier…
Just to be sure I understand, when you say "when
TRANSIENT_DOWNLOADis NO.", it'd be the same as ifTRANSIENT_DOWNLOADdownload is not defined at all, right?
Yes.
There is maybe a little bit of an issue in Yann's current implementation as he does:
$(if $($(2)_DOWNLOAD_TRANSIENT),-F) \
so in fact, -F would be passed as soon as <pkg>_DOWNLOAD_TRANSIENT is non-empty, so <pkg>_DOWNLOAD_TRANSIENT = NO would in fact pass -F.
But this bug already exists for <pkg>_GIT_SUBMODULES:
$(if $($(2)_GIT_SUBMODULES),-r) \
And it is trivial to fix.
Quoted 3 lines
If it's the case, then that seems to me like a good idea, because it renders explicit the risk taken by the user when using a branch (which can be done unknowingly ATM).
Absolutely.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/1 infra: add the transient download mechanism
Thomas, All,
On 2020-01-16 11:05 +0100, Thomas Petazzoni spake thusly:
Quoted 12 lines
Quoted 3 lines · On Thu, 16 Jan 2020 10:01:09 +0000 Nicolas Carrier…
Just to be sure I understand, when you say "when
TRANSIENT_DOWNLOADis NO.", it'd be the same as ifTRANSIENT_DOWNLOADdownload is not defined at all, right?Yes.
There is maybe a little bit of an issue in Yann's current implementation as he does:
Quoted 8 lines
$(if $($(2)_DOWNLOAD_TRANSIENT),-F) \so in fact, -F would be passed as soon as <pkg>_DOWNLOAD_TRANSIENT is non-empty, so <pkg>_DOWNLOAD_TRANSIENT = NO would in fact pass -F.
But this bug already exists for <pkg>_GIT_SUBMODULES:
$(if $($(2)_GIT_SUBMODULES),-r) \
Yeah, I just mirrored for transient wghat we already had for git submodules.
Quoted 1 line
And it is trivial to fix.
If ever needed...
$(if $(filter YES,$($(2)_DOWNLOAD_TRANSIENT)),-F)
(note that there is no equality test in GNU make, but there is one simple one described in GMSL).
Quoted 3 lines
Quoted 3 lines
If it's the case, then that seems to me like a good idea, because it renders explicit the risk taken by the user when using a branch (which can be done unknowingly ATM).
Yup.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 infra: add the transient download mechanism
In reply to Nicolas CarrierNicolas, All,
On 2020-01-16 09:11 +0000, Nicolas Carrier spake thusly:
Quoted 9 lines
Hello Yann,
One small remark, to be able to reproduce the build, on top of this mechanism, there should be:
* a way to store the actual VCS revision used for the build (to at least know what was just built!)
* a way to force the build to re-use a particular set of revisionsThat's features not easy to get right, but which are already handled by tools like google's repo (or partly, by git submodules).
Well, that's already the case with Buildroot: setting _VERSION is what guarantees you can rebuild the same thing again and again.
I don't know much of repo, but from what I grabbed, it uses a manifest to identify packages, where they are, what changesset to use (and some less interesting stuff). This is exactly what one has in Buildroot's .mk files. The manifest is just builtin to the buildsystem.
Quoted 3 lines
That's part of why I don't think that this kind of feature is a good thing to integrate into buildroot. I prefer to delegate it to tools meant for that (using a mix of local packages and OVERRIDE).
You don't need to convince me. But apparently, some people like to aim at their feet with a loaded gun, just in case.. ;-)
On a more serious note, as Thomas explained, some people have expressed the need for using a branch, and that a build would alkways pick the latest version of that branch.
Of course, that means non-reproducibility.
I am even afraid that those people do not realise that they also need to update the _VERSION in their packages once they want to tag and release versions. And this has virtually no overhead in practice (damn... copy-pasting a sha1 is not that complex. It would even be possible to have a script that does it automatically...).
Quoted 2 lines
And like said before, I think it's preferable to have a optional feature to prevent branches from being used, making the build fail.
Yeah, that was one of the comment by Thomas, and it is a good extension to the transient mehcanism: either one uses a stable ref, or one needs to explicitly identify it as transient.
Thanks for the feedback.
Regards,
Yann E. MORIN.
Quoted 219 lines · Yann E. MORIN
Quoted 217 lines
NOTE: NOT TO BE COMMITTED!
This patch is a proof of concept for the transient download mechanism,
that would help using branches as versions, and keep using the head of
the branch each time a build started.A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT= YESthis means that the download infra will not use any already downloaded
archive, and will instead always download it as if missing.Since the check is done in the download wrapper, we have no TOCTOU race
in case two bulds would attempt the same transient download: the archive
is only replaced ato,mically as usual.So, if the package uses a branch as version, the branch's HEAD at that
very moment will be redownloaded.This stil has the drawback that two builds in parallel may get slightly
different content for the same branch, and the first build could end up
using the download of the second build:build-1 build-2
download
| download
| |
save to dl-dir |
[...] save to dl-dir
extractFurthermore, even with a single build, it might get what it expects:
developer-1 developer-2
git push branch
trigger CI git push branch
[...]
downloadIn that case the build of delopper-1 would get the code of developper-2
who pushed on the same branch.For people who are aiming at their feet, we're now providing them with
a loaded gun. ;-]Signed-off-by Yann E. MORIN <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Vincent Fazio <[email protected]>Cc Michael Nazzareno Trimarchi <[email protected]>Cc Chris Packham <[email protected]>Cc Nicolas Carrier <[email protected]>Cc Adam Duskett <[email protected]>--- docs/manual/adding-packages-generic.txt | 16 +++++++++++++--- package/busybox/busybox.mk | 1 + package/pkg-download.mk | 1 + package/pkg-generic.mk | 8 +------- support/download/dl-wrapper | 8 +++++--- 5 files changed, 21 insertions(+), 13 deletions(-)docs/manual/adding-packages-generic.txt +1
-1@@ -206,12 +206,14 @@ information is (assuming the package name is +libfoo+) : ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+ + .Note: -Using a branch name as +FOO_VERSION+ is not supported, because itdoes
docs/manual/adding-packages-generic.txt +2
-1-not and can not work as people would expect it should: +Using a branch name as +FOO_VERSION+, although technically possible, +is highly discouraged, because it does not and can not work aspeople
+would expect it should:
+
1. due to local caching, Buildroot will not re-fetch the
repository,
so people who expect to be able to follow the remote repositorydocs/manual/adding-packages-generic.txt +2
-1- would be quite surprised and disappointed; + would be quite surprised and disappointed (but see + +LIBFOO_DOWNLOAD_TRANSIENT+, later); 2. because two builds can never be perfectly simultaneous, andbecause
the remote repository may get new commits on the branch
anytime,
two users, using the same Buildroot tree and building the samedocs/manual/adding-packages-generic.txt +0
-0@@ -342,6 +344,14 @@ not and can not work as people would expect itshould:
submodules when they contain bundled libraries, in which case we prefer to use those libraries from their own package.+* +LIBFOO_DOWNLOAD_TRANSIENT+ can be set to +YES+ or +NO+ (the default).
+ If set to +YES+, the download for that package will be attempted at
+ every build; any already downloaded archive is ignored as if missing.
+ This may help when a branch is specified in +LIBFOO_VERSION+, and the
+ head/tip of the branch is to be built, like a CI pipeline would needdocs/manual/adding-packages-generic.txt +3
-0+ for example. This still suffers from the other issues listed above + about using branches, though. + * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components (directories) that tar must strip from file names on extraction. The tarball for most packages has one leading component namedpackage/busybox/busybox.mk +1
-0@@ -9,6 +9,7 @@ BUSYBOX_SITE = http://www.busybox.net/downloads BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2 BUSYBOX_LICENSE = GPL-2.0 BUSYBOX_LICENSE_FILES = LICENSE +BUSYBOX_DOWNLOAD_TRANSIENT = YES define BUSYBOX_HELP_CMDS @echo ' busybox-menuconfig - Run BusyBox menuconfig'package/pkg-download.mk +1
-0@@ -108,6 +108,7 @@ define DOWNLOAD -n '$($(2)_BASENAME_RAW)' \ -N '$($(2)_RAWNAME)' \ -o '$($(2)_DL_DIR)/$(notdir $(1))' \ + $(if $($(2)_DOWNLOAD_TRANSIENT),-F) \ $(if $($(2)_GIT_SUBMODULES),-r) \ $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \$(QUIET) \package/pkg-generic.mk +1
-7@@ -158,13 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded: @$(call step_start,download) $(call prepare-per-package- directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) -# Only show the download message if it isn't already downloaded - $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \ - if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \ - $(call MESSAGE,"Downloading") ; \ - break ; \ - fi ; \ - done + @$(call MESSAGE,"Downloading") ; \ $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(callDOWNLOAD,$(p),$(PKG))$(sep))
$(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) $(Q)mkdir -p $(@D)support/download/dl- renamed from support/download/dl-wrapper +0
-0wrapper
index3315bd410e..ab22ca7e4f100755support/download/dl-wrapper +0
-0@@ -21,11 +21,12 @@ exportBR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
main(){
local OPT OPTARGsupport/download/dl-wrapper +4
-2- local backend output hfile recurse quiet rc + local backend output hfile recurse quiet rc force local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + force=false + while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do case "${OPT}" in c) cset="${OPTARG}";; d) dl_dir="${OPTARG}";;@@ -38,6 +39,7 @@ main() { f) filename="${OPTARG}";; u) uris+=( "${OPTARG}" );; q) quiet="-q";; + F) force=true;; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";;esac
support/download/dl-wrapper +0
-0@@ -67,7 +69,7 @@ main() { # - matches all its hashes: do not download it again and exitpromptly
# - fails at least one of its hashes: force a re-download # - there's no hash (but a .hash file): consider it a hard errorsupport/download/dl-wrapper +1
-1- if [ -e "${output}" ]; then + if ! ${force} && [ -e "${output}" ]; then if support/download/check-hash ${quiet} "${hfile}""${output}" "${output##*/}"; then
exit 0 elif [ ${?} -ne 2 ]; then--
2.20.1ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 infra: add the transient download mechanism
Quoted 13 lines · Yann E. MORIN
Nicolas, All,
On 2020-01-16 09:11 +0000, Nicolas Carrier spake thusly:
Quoted 9 lines
Hello Yann,
One small remark, to be able to reproduce the build, on top of this mechanism, there should be:
* a way to store the actual VCS revision used for the build (to at least know what was just built!)
* a way to force the build to re-use a particular set of revisionsThat's features not easy to get right, but which are already handled by tools like google's repo (or partly, by git submodules).
Quoted 7 lines
Well, that's already the case with Buildroot: setting
_VERSIONis what guarantees you can rebuild the same thing again and again.I don't know much of repo, but from what I grabbed, it uses a manifest to identify packages, where they are, what changesset to use (and some less interesting stuff). This is exactly what one has in Buildroot's .mk files. The manifest is just builtin to the buildsystem.
The difference is that repo doesn't need the changeset to use. If no changeset
is specified, then 'repo update' will do a pull of all branches. So the transient download mechanism pretty much approximates that.
That said, there are clear advantages to using repo during development compared
to the transient download mechanism:
- possibility to branch or tag several trees in one shot;
- easy to fixate the manifest from a specific situation;
- no need to switch to yet another mechanism (OVERRIDE_SRCDIR isntead of transient) when you want to make modifications.
So for a serious project, repo is a clear win.
However, for small projects, it's a bit overkill. For a smaller project, you'd
typically have only a few trees to manage (buildroot, external, linux, application) and most of them are not terribly active, so the features of repo don't bring a lot of added value. And managing the manifest file is definitely more complicated than just specifying the linux git in your defconfig.
Bottom line: even though repo is the better tool to manage the trees both for
development and release, the transient mechanism is a good quick win.
Quoted 19 lines
Quoted 3 lines
That's part of why I don't think that this kind of feature is a good thing to integrate into buildroot. I prefer to delegate it to tools meant for that (using a mix of local packages and OVERRIDE).
You don't need to convince me. But apparently, some people like to aim at their feet with a loaded gun, just in case.. ;-)
On a more serious note, as Thomas explained, some people have expressed the need for using a branch, and that a build would alkways pick the latest version of that branch.
Of course, that means non-reproducibility.
I am even afraid that those people do not realise that they also need to update the
_VERSIONin their packages once they want to tag and release versions. And this has virtually no overhead in practice (damn... copy-pasting a sha1 is not that complex. It would even be possible to have a script that does it automatically...).
Quoted 6 lines
Quoted 2 lines
And like said before, I think it's preferable to have a optional feature to prevent branches from being used, making the build fail.
Yeah, that was one of the comment by Thomas, and it is a good extension to the transient mehcanism: either one uses a stable ref, or one needs to explicitly identify it as transient.
We already have patches for that in patchwork.
With those, we'd need to propagate the -F option down to the git helper, so it
can avoid erroring out.
Regards,
Arnout
Quoted 226 lines
Thanks for the feedback.
Regards,
Yann E. MORIN.Quoted 219 lines · Yann E. MORIN
Quoted 217 lines
NOTE: NOT TO BE COMMITTED!
This patch is a proof of concept for the transient download mechanism,
that would help using branches as versions, and keep using the head of
the branch each time a build started.A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT= YESthis means that the download infra will not use any already downloaded
archive, and will instead always download it as if missing.Since the check is done in the download wrapper, we have no TOCTOU race
in case two bulds would attempt the same transient download: the archive
is only replaced ato,mically as usual.So, if the package uses a branch as version, the branch's HEAD at that
very moment will be redownloaded.This stil has the drawback that two builds in parallel may get slightly
different content for the same branch, and the first build could end up
using the download of the second build:build-1 build-2
download
| download
| |
save to dl-dir |
[...] save to dl-dir
extractFurthermore, even with a single build, it might get what it expects:
developer-1 developer-2
git push branch
trigger CI git push branch
[...]
downloadIn that case the build of delopper-1 would get the code of developper-2
who pushed on the same branch.For people who are aiming at their feet, we're now providing them with
a loaded gun. ;-]Signed-off-by Yann E. MORIN <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Vincent Fazio <[email protected]>Cc Michael Nazzareno Trimarchi <[email protected]>Cc Chris Packham <[email protected]>Cc Nicolas Carrier <[email protected]>Cc Adam Duskett <[email protected]>--- docs/manual/adding-packages-generic.txt | 16 +++++++++++++--- package/busybox/busybox.mk | 1 + package/pkg-download.mk | 1 + package/pkg-generic.mk | 8 +------- support/download/dl-wrapper | 8 +++++--- 5 files changed, 21 insertions(+), 13 deletions(-)docs/manual/adding-packages-generic.txt +1
-1@@ -206,12 +206,14 @@ information is (assuming the package name is +libfoo+) : ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+ + .Note: -Using a branch name as +FOO_VERSION+ is not supported, because itdoes
docs/manual/adding-packages-generic.txt +2
-1-not and can not work as people would expect it should: +Using a branch name as +FOO_VERSION+, although technically possible, +is highly discouraged, because it does not and can not work aspeople
+would expect it should:
+
1. due to local caching, Buildroot will not re-fetch the
repository,
so people who expect to be able to follow the remote repositorydocs/manual/adding-packages-generic.txt +2
-1- would be quite surprised and disappointed; + would be quite surprised and disappointed (but see + +LIBFOO_DOWNLOAD_TRANSIENT+, later); 2. because two builds can never be perfectly simultaneous, andbecause
the remote repository may get new commits on the branch
anytime,
two users, using the same Buildroot tree and building the samedocs/manual/adding-packages-generic.txt +0
-0@@ -342,6 +344,14 @@ not and can not work as people would expect itshould:
submodules when they contain bundled libraries, in which case we prefer to use those libraries from their own package.+* +LIBFOO_DOWNLOAD_TRANSIENT+ can be set to +YES+ or +NO+ (the default).
+ If set to +YES+, the download for that package will be attempted at
+ every build; any already downloaded archive is ignored as if missing.
+ This may help when a branch is specified in +LIBFOO_VERSION+, and the
+ head/tip of the branch is to be built, like a CI pipeline would needdocs/manual/adding-packages-generic.txt +3
-0+ for example. This still suffers from the other issues listed above + about using branches, though. + * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components (directories) that tar must strip from file names on extraction. The tarball for most packages has one leading component namedpackage/busybox/busybox.mk +1
-0@@ -9,6 +9,7 @@ BUSYBOX_SITE = http://www.busybox.net/downloads BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2 BUSYBOX_LICENSE = GPL-2.0 BUSYBOX_LICENSE_FILES = LICENSE +BUSYBOX_DOWNLOAD_TRANSIENT = YES define BUSYBOX_HELP_CMDS @echo ' busybox-menuconfig - Run BusyBox menuconfig'package/pkg-download.mk +1
-0@@ -108,6 +108,7 @@ define DOWNLOAD -n '$($(2)_BASENAME_RAW)' \ -N '$($(2)_RAWNAME)' \ -o '$($(2)_DL_DIR)/$(notdir $(1))' \ + $(if $($(2)_DOWNLOAD_TRANSIENT),-F) \ $(if $($(2)_GIT_SUBMODULES),-r) \ $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \$(QUIET) \package/pkg-generic.mk +1
-7@@ -158,13 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded: @$(call step_start,download) $(call prepare-per-package- directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) -# Only show the download message if it isn't already downloaded - $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \ - if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \ - $(call MESSAGE,"Downloading") ; \ - break ; \ - fi ; \ - done + @$(call MESSAGE,"Downloading") ; \ $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(callDOWNLOAD,$(p),$(PKG))$(sep))
$(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) $(Q)mkdir -p $(@D)support/download/dl- renamed from support/download/dl-wrapper +0
-0wrapper
index3315bd410e..ab22ca7e4f100755support/download/dl-wrapper +0
-0@@ -21,11 +21,12 @@ exportBR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
main(){
local OPT OPTARGsupport/download/dl-wrapper +4
-2- local backend output hfile recurse quiet rc + local backend output hfile recurse quiet rc force local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + force=false + while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do case "${OPT}" in c) cset="${OPTARG}";; d) dl_dir="${OPTARG}";;@@ -38,6 +39,7 @@ main() { f) filename="${OPTARG}";; u) uris+=( "${OPTARG}" );; q) quiet="-q";; + F) force=true;; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";;esac
support/download/dl-wrapper +0
-0@@ -67,7 +69,7 @@ main() { # - matches all its hashes: do not download it again and exitpromptly
# - fails at least one of its hashes: force a re-download # - there's no hash (but a .hash file): consider it a hard errorsupport/download/dl-wrapper +1
-1- if [ -e "${output}" ]; then + if ! ${force} && [ -e "${output}" ]; then if support/download/check-hash ${quiet} "${hfile}""${output}" "${output##*/}"; then
exit 0 elif [ ${?} -ne 2 ]; then--
2.20.1ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.
PATCH 1/1 infra: add the transient download mechanism
In reply to Yann E. MORINHello,
Quoted 5 lines · On Wed, 15 Jan 2020 21:37:53 +0100 "Yann E. MORIN" <[email protected]>…
NOTE: NOT TO BE COMMITTED!
This patch is a proof of concept for the transient download mechanism, that would help using branches as versions, and keep using the head of the branch each time a build started.
So what is the status of this patch? I see Yann you marked it as to be ignored by patchwork, so we don't have it in patchwork.
I think there's generally been some positive feedback, with Arnout saying:
""
Bottom line: even though repo is the better tool to manage the trees both for
development and release, the transient mechanism is a good quick win. ""
Quoted 9 lines
A package declares its download as transient with:
FOO_DOWNLOAD_TRANSIENT= YESthis means that the download infra will not use any already downloaded archive, and will instead always download it as if missing.
Since the check is done in the download wrapper, we have no TOCTOU race in case two bulds would attempt the same transient download: the archive is only replaced ato,mically as usual.
Typo: atomically
Quoted 7 lines
So, if the package uses a branch as version, the branch's HEAD at that very moment will be redownloaded.
This stil has the drawback that two builds in parallel may get slightly different content for the same branch, and the first build could end up using the download of the second build:
Quoted 8 lines
build-1 build-2
download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract
That is not a drawback, that is exactly what you expect with this feature.
Quoted 11 lines
Furthermore, even with a single build, it might get what it expects:
developer-1 developer-2
git push branch
trigger CI git push branch
[...]
downloadIn that case the build of delopper-1 would get the code of developper-2 who pushed on the same branch.
Ditto here: this is exactly what we expect for a transient download feature. If developer-1 wants to test only his own feature, then he has to trigger CI on a branch that only he has control over.
There is however one drawback:
* You run a build (1), which downloads package A from a Git repo, and places the tarball in global DL_DIR.
* You run a different build (2), which downloads the same package A from the Git repo. Due to the transient download mechanism being enabled for package A, it re-downloads it, and places an updated tarball in the global DL_DIR.
* Back to build (1), you run "make legal-info", which picks up the updated tarball, which was generated by build (2).
So, in build (1), your image uses the "previous" version of package A, while your legal-info contains the "new" version of package A.
To solve this, one solution would be to use a per-build download directory for the transient packages, rather than the global DL_DIR.
But I think this could be a separate improvement. Indeed, the transient download mechanism is clearly oriented towards development/CI workflow, where consistency of legal-info may not be a primary concern. For a final release build, you would not use TRANSIENT_DOWNLOAD = YES.
For example, custom packages that make use of TRANSIENT_DOWNLOAD could be implemented like this:
ifeq ($(BR2_DEVELOPER_MODE),y)
<pkg>_TRANSIENT_DOWNLOAD = YES
<pkg>_VERSION = some-branch
else
<pkg>_VERSION = some-tag
endif
So, while in development mode, users and CI use BR2_DEVELOPER_MODE=y, but for final releases, BR2_DEVELOPER_MODE is disabled, which ensures we have a fully reproducible build, and consistent legal-info output.
Yann, do you think you could resubmit your patch, for actual merging ? Of course, we don't need the busybox.mk change.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/1 infra: add the transient download mechanism
Thomas, All,
On 2020-04-08 08:39 +0200, Thomas Petazzoni spake thusly:
Quoted 9 lines
Quoted 3 lines · On Wed, 15 Jan 2020 21:37:53 +0100 "Yann E. MORIN" <[email protected]>…
This patch is a proof of concept for the transient download mechanism, that would help using branches as versions, and keep using the head of the branch each time a build started.
So what is the status of this patch? I see Yann you marked it as to be ignored by patchwork, so we don't have it in patchwork.
Quoted 7 lines
I think there's generally been some positive feedback, with Arnout saying:
""
Bottom line: even though repo is the better tool to manage the trees both for
development and release, the transient mechanism is a good quick win. ""
I will address all the questions and comments in the review, and will respin it for proper consideration.
[--SNIP--]
Quoted 13 lines
Quoted 4 lines
This stil has the drawback that two builds in parallel may get slightly different content for the same branch, and the first build could end up using the download of the second build:
Quoted 8 lines
build-1 build-2
download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract
Quoted 2 lines
That is not a drawback, that is exactly what you expect with this feature.
Sorry, the two builds are supposed to be running in parallel... For build 1, it would get the download from build 2, which is not what build 1 expects.
But drawback is not the correct word. This is a pitfall.
Quoted 9 lines
Quoted 9 lines
Furthermore, even with a single build, it might get what it expects:
developer-1 developer-2
git push branch
trigger CI git push branch
[...]
download
Quoted 6 lines
Quoted 2 lines
In that case the build of delopper-1 would get the code of developper-2 who pushed on the same branch.
Ditto here: this is exactly what we expect for a transient download feature. If developer-1 wants to test only his own feature, then he has to trigger CI on a branch that only he has control over.
Yes, this is not a drawback, but a pitfall.
Quoted 5 lines
There is however one drawback:
* You run a build (1), which downloads package A from a Git repo, and places the tarball in global
DL_DIR.
Quoted 7 lines
* You run a different build (2), which downloads the same package A from the Git repo. Due to the transient download mechanism being enabled for package A, it re-downloads it, and places an updated tarball in the global
DL_DIR.* Back to build (1), you run "make legal-info", which picks up the updated tarball, which was generated by build (2).
This is not a drawback; this is another pitfall.
Quoted 5 lines
So, in build (1), your image uses the "previous" version of package A, while your legal-info contains the "new" version of package A.
To solve this, one solution would be to use a per-build download directory for the transient packages, rather than the global
DL_DIR.
Meh... Or rather, make legal-info impossible for a package that is transient, like when it is 'local'. This is the best solution.
Quoted 8 lines
But I think this could be a separate improvement. Indeed, the transient download mechanism is clearly oriented towards development/CI workflow, where consistency of legal-info may not be a primary concern. For a final release build, you would not use
TRANSIENT_DOWNLOAD= YES.For example, custom packages that make use of
TRANSIENT_DOWNLOADcould be implemented like this:
Quoted 6 lines
ifeq ($(BR2_DEVELOPER_MODE),y)
<pkg>_TRANSIENT_DOWNLOAD = YES
<pkg>_VERSION = some-branch
else
<pkg>_VERSION = some-tag
endif
The more I think about it, the less proud I am to have come up with this transient idea. Using unstable references is complete madness to begin with and is so flawed by design... We should not even provide such a feature. But we can't stop people from being... people... :-[
Quoted 5 lines
So, while in development mode, users and CI use BR2_DEVELOPER_MODE=y, but for final releases,
BR2_DEVELOPER_MODEis disabled, which ensures we have a fully reproducible build, and consistent legal-info output.Yann, do you think you could resubmit your patch, for actual merging ?
Yes I will.
Quoted 1 line
Of course, we don't need the busybox.mk change.
We don't? How comne" ;-)
Quoted 7 lines
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 infra: add the transient download mechanism
Hello,
Quoted 9 lines · On Wed, 8 Apr 2020 21:27:53 +0200 "Yann E. MORIN" <[email protected]>…
Quoted 9 lines
Quoted 8 lines
build-1 build-2
download
| download
| |
save to dl-dir |
[...] save to dl-dir
extract
Quoted 8 lines
Quoted 2 lines
That is not a drawback, that is exactly what you expect with this feature.
Sorry, the two builds are supposed to be running in parallel... For build 1, it would get the download from build 2, which is not what build 1 expects.
But drawback is not the correct word. This is a pitfall.
Again, I think that what you're describing is exactly what we expect: if you want to a branch, then you are guaranteed that you will do a build with a state of the branch that is at least the one that was present at the time you started the build. But it can be a later state of that branch, if it gets updated in the mean time.
Quoted 8 lines
Quoted 5 lines
So, in build (1), your image uses the "previous" version of package A, while your legal-info contains the "new" version of package A.
To solve this, one solution would be to use a per-build download directory for the transient packages, rather than the global
DL_DIR.Meh... Or rather, make legal-info impossible for a package that is transient, like when it is 'local'. This is the best solution.
Sounds good to me indeed.
Quoted 15 lines
Quoted 8 lines
But I think this could be a separate improvement. Indeed, the transient download mechanism is clearly oriented towards development/CI workflow, where consistency of legal-info may not be a primary concern. For a final release build, you would not use
TRANSIENT_DOWNLOAD= YES.For example, custom packages that make use of
TRANSIENT_DOWNLOADcould be implemented like this:Quoted 6 lines
ifeq ($(BR2_DEVELOPER_MODE),y)
<pkg>_TRANSIENT_DOWNLOAD = YES
<pkg>_VERSION = some-branch
else
<pkg>_VERSION = some-tag
endif
Quoted 4 lines
The more I think about it, the less proud I am to have come up with this transient idea. Using unstable references is complete madness to begin with and is so flawed by design... We should not even provide such a feature. But we can't stop people from being... people... :-[
I think it's something that a number of people want, and that makes sense for a number of use-cases.
It's all about moving away from the idea that Buildroot is only the "final" integration tool. It is also used during application development, it is also used to do CI, etc. And that requires to relax a bit the very strict reproducibility requirements that we enforce for the "final integration" use case.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com