core/download: add support for git sub-modules (branch yem/git)
v2 cover core/download: add support for git sub-modules (branch yem/git)
Hello All!
This little series adds support for packages coming from a git clone, with git sub-modules.
The commit logs are pretty detailed, but roughly, it goes as thus;
- we can't get sub-modules in a bare clone, so we no longer use bare
clones, but use full clones (with a working copy);
- git-archive does not support submodules, so we generate archives
manually;
- we check out the working copy to the correct cset, then we retrieve
the submodules;
- we expose a new package variable to request sub-modules.
Changes v1 -> v2:
- properly accept -r in the download wrapper (Aleksandar)
- split the non-bare-git and manual-archive changes in two separate patches, as they are not semantically related
- some of my usual typoes fixed...
Regards,
Yann E. MORIN.
The following changes since commit 2cd5abf7f9edd3c0a609e9626f12f71e2a6e4e7c:
alsa-utils: bump to version 1.1.1 (2016-04-01 03:57:54 +0200)
are available in the git repository at:
git://git.busybox.net/~ymorin/git/buildroot yem/git
for you to fetch changes up to 8373904cdf1fee9724d6834b56efaa706a92fc31:
core/pkg-infra: download git submodules if the package wants them (2016-04-01 22:12:08 +0200)
----------------------------------------------------------------
Yann E. MORIN (4):
support/download/git: do not use bare clones support/download/git: do not use git archive, handle it manually support/download/git: add support for submodules core/pkg-infra: download git submodules if the package wants them
package/pkg-download.mk | 1 + package/pkg-generic.mk | 8 ++++++++ support/download/dl-wrapper | 7 ++++--- support/download/git | 46 +++++++++++++++++++++++++++++++++++++++------ 4 files changed, 53 insertions(+), 9 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 1/4 support/download/git: do not use bare clones
Currently, we are using bare clones, so as to minimise the disk usage, most notably for largeish repositories such as the one for the Linux kernel, which can go beyond the 1GiB barrier.
However, this precludes updating (and thus using) the submodules, if any, of the repositories, as a working copy is required to use submodules (becaue we need to know the list of submodules, where to find them, where to clone them, what cset to checkout, and all those is dependent upon the checked out cset of the father repository).
Switch to using /plain/ clones with a working copy.
This means that the extra refs used by some forges (like pull-requests for Github, or changes for gerrit...) are no longer fetched as part of the clone, because git does not offer to do a mirror clone when there is a working copy.
Instead, we have to fetch those special refs by hand. Since there is no easy solution to know whether the cset the user asked for is such a special ref or not, we just try to always fetch the cset requested by the user; if this fails, we assume that this is not a special ref (most probably, it is a sha1) and we defer the check to the archive creation, which would fail if the requested cset is missing anyway.
--- support/download/git | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
support/download/git +18 -3
@@ -41,7 +41,7 @@ _git() { git_done=0 if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then printf "Doing shallow clone\n" - if _git clone ${verbose} --depth 1 -b "'${cset}'" --bare "'${repo}'" "'${basename}'"; then + if _git clone ${verbose} --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then git_done=1 else printf "Shallow clone failed, falling back to doing a full clone\n"
@@ -49,10 +49,25 @@ if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then fi if [ ${git_done} -eq 0 ]; then printf "Doing full clone\n" - _git clone ${verbose} --mirror "'${repo}'" "'${basename}'" + _git clone ${verbose} "'${repo}'" "'${basename}'" +fi + +pushd "${basename}" >/dev/null + +# Try to get the special refs exposed by some forges (pull-requests for +# github, changes for gerrit...). There is no easy way to know whether +# the cset the user passed us is such a special ref or a tag or a sha1 +# or whatever else. We'll eventually fail at checking out that cset, +# below, if there is an issue anyway. Since most of the cset we're gonna +# have to clone are not such special refs, consign the output to oblivion +# so as not to alarm unsuspecting users, but still trace it as a warning. +if ! _git fetch "'${cset}:${cset}'" >/dev/null 2>&1; then + printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" fi -GIT_DIR="${basename}" \ _git archive --prefix="'${basename}/'" -o "'${output}.tmp'" --format=tar "'${cset}'" +# Not really required, but here for consistency +popd >/dev/null + gzip -n <"${output}.tmp" >"${output}"
Signature · 1 line
1.9.1
PATCH v2 1/4 support/download/git: do not use bare clones
Yann,
Quoted 17 lines · Yann E. MORIN
Currently, we are using bare clones, so as to minimise the disk usage, most notably for largeish repositories such as the one for the Linux kernel, which can go beyond the 1GiB barrier.
However, this precludes updating (and thus using) the submodules, if any, of the repositories, as a working copy is required to use submodules (becaue we need to know the list of submodules, where to find them, where to clone them, what cset to checkout, and all those is dependent upon the checked out cset of the father repository).
Switch to using /plain/ clones with a working copy.
This means that the extra refs used by some forges (like pull-requests for Github, or changes for gerrit...) are no longer fetched as part of the clone, because git does not offer to do a mirror clone when there is a working copy.
Quoted 8 lines
Instead, we have to fetch those special refs by hand. Since there is no easy solution to know whether the cset the user asked for is such a special ref or not, we just try to always fetch the cset requested by the user; if this fails, we assume that this is not a special ref (most probably, it is a sha1) and we defer the check to the archive creation, which would fail if the requested cset is missing anyway.
Signed-off-by "Yann E. MORIN" <[email protected]>
Quoted 52 lines
--- support/download/git | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)support/download/git +18
-4@@ -41,7 +41,7 @@ _git() { git_done=0 if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then printf "Doing shallow clone\n" - if _git clone ${verbose} --depth 1 -b "'${cset}'" --bare "'${repo}'" "'${basename}'"; then + if _git clone ${verbose} --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then git_done=1 else printf "Shallow clone failed, falling back to doing a full clone\n"@@ -49,10 +49,25 @@ if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then fi if [ ${git_done} -eq 0 ]; then printf "Doing full clone\n" - _git clone ${verbose} --mirror "'${repo}'" "'${basename}'" + _git clone ${verbose} "'${repo}'" "'${basename}'" +fi + +pushd "${basename}" >/dev/null + +# Try to get the special refs exposed by some forges (pull-requests for +# github, changes for gerrit...). There is no easy way to know whether +# the cset the user passed us is such a special ref or a tag or a sha1 +# or whatever else. We'll eventually fail at checking out that cset, +# below, if there is an issue anyway. Since most of the cset we're gonna +# have to clone are not such special refs, consign the output to oblivion +# so as not to alarm unsuspecting users, but still trace it as a warning. +if ! _git fetch "'${cset}:${cset}'" >/dev/null 2>&1; then + printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" fi -GIT_DIR="${basename}" \ _git archive --prefix="'${basename}/'" -o "'${output}.tmp'" --format=tar "'${cset}'" +# Not really required, but here for consistency +popd >/dev/null + gzip -n <"${output}.tmp" >"${output}" --1.9.1
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 2 lines
Thanks, Matt
PATCH v2 2/4 support/download/git: do not use git archive, handle it manually
In reply to Yann E. MORINWe currently use git-archive to generate the tarball. This is all handy and dandy, but git-archive does not support submodules. In the follow-up patch, we're going to handle submodules, so we would not be able to use git-archive.
Instead, we manually generate the archive:
- extract the tree to the requested cset,
- get the date of the commit to store in the archive,
- store only numeric owners,
- store owner and group as 0 (zero, although any arbitrary value would have been fine, as long as it's a constant),
- sort the files to store in the archive.
We also get rid of the .git directory, because there is no reason to keep it in the context of Buildroot. Some people would love to keep it so as to speed up later downloads when updating a package, but that is not really doable. For example:
- use current Buildroot
- it would need foo-12345, so do a clone and keep the .git in the generated tarball
- update Buildroot
- it would need foo-98765
For that second clone, how could we know we would have to first extract foo-12345 ? So, the .git in the archive is pretty much useless for Buildroot.
--- support/download/git | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
support/download/git +10 -2
@@ -65,9 +65,17 @@ if ! _git fetch "'${cset}:${cset}'" >/dev/null 2>&1; then printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" fi -_git archive --prefix="'${basename}/'" -o "'${output}.tmp'" --format=tar "'${cset}'" +# Checkout the required changeset. +_git checkout -q "'${cset}'" + +# Get date of commit to generate a reproducible archive. +date="$( _git show --no-patch --pretty=format:%cD )" + +# We do not need the .git dir to generate the tarball +rm -rf .git -# Not really required, but here for consistency popd >/dev/null +tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" \ + -T <(find "${basename}" -not -type d |sort) >"${output}.tmp" gzip -n <"${output}.tmp" >"${output}"
Signature · 1 line
1.9.1
PATCH v2 2/4 support/download/git: do not use git archive, handle it manually
Yann,
Quoted 26 lines · Yann E. MORIN
We currently use git-archive to generate the tarball. This is all handy and dandy, but git-archive does not support submodules. In the follow-up patch, we're going to handle submodules, so we would not be able to use git-archive.
Instead, we manually generate the archive:
- extract the tree to the requested cset,
- get the date of the commit to store in the archive,
- store only numeric owners,
- store owner and group as 0 (zero, although any arbitrary value would have been fine, as long as it's a constant),
- sort the files to store in the archive.We also get rid of the .git directory, because there is no reason to keep it in the context of Buildroot. Some people would love to keep it so as to speed up later downloads when updating a package, but that is not really doable. For example:
- use current Buildroot
- it would need foo-12345, so do a clone and keep the .git in the generated tarball
- update Buildroot
- it would need foo-98765
For that second clone, how could we know we would have to first extract foo-12345 ? So, the .git in the archive is pretty much useless for Buildroot.
Quoted 1 line
Signed-off-by "Yann E. MORIN" <[email protected]>
Quoted 35 lines
--- support/download/git | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)support/download/git +10
-3@@ -65,9 +65,17 @@ if ! _git fetch "'${cset}:${cset}'" >/dev/null 2>&1; then printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" fi -_git archive --prefix="'${basename}/'" -o "'${output}.tmp'" --format=tar "'${cset}'" +# Checkout the required changeset. +_git checkout -q "'${cset}'" + +# Get date of commit to generate a reproducible archive. +date="$( _git show --no-patch --pretty=format:%cD )" + +# We do not need the .git dir to generate the tarball +rm -rf .git -# Not really required, but here for consistency popd >/dev/null +tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" \ + -T <(find "${basename}" -not -type d |sort) >"${output}.tmp" gzip -n <"${output}.tmp" >"${output}" --1.9.1
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 2 lines
Thanks, Matt
PATCH v2 3/4 support/download/git: add support for submodules
In reply to Yann E. MORINSome git repositories may be split into a master repository and submodules. Up until now, we did not have support for submodules, because we were using bare clones, in which it is not possible to update the list of submodules.
Now that we are using plain clones with a working copy, we can retrieve the submdoules.
Add an option to the git downlaod helper to kick the update of submodules, so that they are only fetched for those packages that require them. Also document the existing -q option at the same time.
Submodules have a .git file at their root, which contains the path to the real .git directory of the master repository. Since we remove it, there is no point in keeping those .git files either.
Note: this is currently unused, but will be enabled with the follow-up patch , that adds the necessary parts in the pkg-generic and pkg-download infrastructures.
--- support/download/git | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
support/download/git +17 -6
@@ -6,15 +6,20 @@ set -e # Download helper for git, to be called from the download wrapper script # # Call it as: -# .../git [-q] OUT_FILE REPO_URL CSET BASENAME +# .../git [-q] [-r] OUT_FILE REPO_URL CSET BASENAME +# +# -q Be quiet. +# -r Clone and archive sub-modules. # # Environment: # GIT : the git command to call verbose= -while getopts :q OPT; do +recurse=0 +while getopts :qr OPT; do case "${OPT}" in q) verbose=-q; exec >/dev/null;; + r) recurse=1;; \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; esac done
@@ -61,18 +66,24 @@ pushd "${basename}" >/dev/null # below, if there is an issue anyway. Since most of the cset we're gonna # have to clone are not such special refs, consign the output to oblivion # so as not to alarm unsuspecting users, but still trace it as a warning. -if ! _git fetch "'${cset}:${cset}'" >/dev/null 2>&1; then +if ! _git fetch origin "'${cset}'" >/dev/null 2>&1; then printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" fi -# Checkout the required changeset. +# Checkout the required changeset, so that we can update the required +# submodules. _git checkout -q "'${cset}'" # Get date of commit to generate a reproducible archive. date="$( _git show --no-patch --pretty=format:%cD )" -# We do not need the .git dir to generate the tarball -rm -rf .git +# There might be submodules, so fetch them. +if [ ${recurse} -eq 1 ]; then + _git submodule update --init --recursive +fi + +# We do not need the .git dir and files to generate the tarball +find . \( -name .git -o -name .gitmodules \) -exec rm -rf {} + popd >/dev/null
Signature · 1 line
1.9.1
PATCH v2 3/4 support/download/git: add support for submodules
Yann,
Quoted 16 lines · Yann E. MORIN
Some git repositories may be split into a master repository and submodules. Up until now, we did not have support for submodules, because we were using bare clones, in which it is not possible to update the list of submodules.
Now that we are using plain clones with a working copy, we can retrieve the submdoules.
Add an option to the git downlaod helper to kick the update of submodules, so that they are only fetched for those packages that require them. Also document the existing -q option at the same time.
Submodules have a .git file at their root, which contains the path to the real .git directory of the master repository. Since we remove it, there is no point in keeping those .git files either.
Quoted 5 lines
Note: this is currently unused, but will be enabled with the follow-up patch , that adds the necessary parts in the pkg-generic and pkg-download infrastructures.
Signed-off-by "Yann E. MORIN" <[email protected]>
Quoted 67 lines
--- support/download/git | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)support/download/git +17
-7@@ -6,15 +6,20 @@ set -e # Download helper for git, to be called from the download wrapper script # # Call it as: -# .../git [-q] OUT_FILE REPO_URL CSET BASENAME +# .../git [-q] [-r] OUT_FILE REPO_URL CSET BASENAME +# +# -q Be quiet. +# -r Clone and archive sub-modules. # # Environment: # GIT : the git command to call verbose= -while getopts :q OPT; do +recurse=0 +while getopts :qr OPT; do case "${OPT}" in q) verbose=-q; exec >/dev/null;; + r) recurse=1;; \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; esac done@@ -61,18 +66,24 @@ pushd "${basename}" >/dev/null # below, if there is an issue anyway. Since most of the cset we're gonna # have to clone are not such special refs, consign the output to oblivion # so as not to alarm unsuspecting users, but still trace it as a warning. -if ! _git fetch "'${cset}:${cset}'" >/dev/null 2>&1; then +if ! _git fetch origin "'${cset}'" >/dev/null 2>&1; then printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" fi -# Checkout the required changeset. +# Checkout the required changeset, so that we can update the required +# submodules. _git checkout -q "'${cset}'" # Get date of commit to generate a reproducible archive. date="$( _git show --no-patch --pretty=format:%cD )" -# We do not need the .git dir to generate the tarball -rm -rf .git +# There might be submodules, so fetch them. +if [ ${recurse} -eq 1 ]; then + _git submodule update --init --recursive +fi + +# We do not need the .git dir and files to generate the tarball +find . \( -name .git -o -name .gitmodules \) -exec rm -rf {} + popd >/dev/null --1.9.1
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 2 lines
Thanks, Matt
PATCH v2 4/4 core/pkg-infra: download git submodules if the package wants them
In reply to Yann E. MORINAdd a new package variable that packages can set to specify that they need git submodules.
Only accept this option if the download method is git, as we can not get submodules via an http download (via wget).
---
Changes v1 -> v2:
- properly accept the -r in the download wrapper (Aleksandar)
--- package/pkg-download.mk | 1 + package/pkg-generic.mk | 8 ++++++++ support/download/dl-wrapper | 7 ++++--- 3 files changed, 13 insertions(+), 3 deletions(-)
package/pkg-download.mk +1 -0
@@ -76,6 +76,7 @@ export BR_NO_CHECK_HASH_FOR = define DOWNLOAD_GIT $(EXTRA_ENV) $(DL_WRAPPER) -b git \ -o $(DL_DIR)/$($(PKG)_SOURCE) \ + $(if $($(PKG)_GIT_SUBMODULES),-r) \ $(QUIET) \ -- \ $($(PKG)_SITE) \
package/pkg-generic.mk +8 -0
@@ -453,6 +453,14 @@ ifndef $(2)_SITE_METHOD endif endif +# Do not accept to download git submodule if not using the git method +ifneq ($$($(2)_GIT_SUBMODULES),) + ifneq ($$($(2)_SITE_METHOD),git) + $$(error $(2) declares having git sub-modules, but does not use the \ + 'git' method (uses '$$($(2)_SITE_METHOD)' instead)) + endif +endif + ifeq ($$($(2)_SITE_METHOD),local) ifeq ($$($(2)_OVERRIDE_SRCDIR),) $(2)_OVERRIDE_SRCDIR = $$($(2)_SITE)
support/download/dl-wrapper +4 -3
@@ -21,15 +21,16 @@ set -e main() { local OPT OPTARG - local backend output hfile quiet + local backend output hfile recurse quiet # Parse our options; anything after '--' is for the backend - while getopts :hb:o:H:q OPT; do + while getopts :hb:o:H:rq OPT; do case "${OPT}" in h) help; exit 0;; b) backend="${OPTARG}";; o) output="${OPTARG}";; H) hfile="${OPTARG}";; + r) recurse="-r";; q) quiet="-q";; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";;
@@ -82,7 +83,7 @@ main() { # If the backend fails, we can just remove the temporary directory to # remove all the cruft it may have left behind. Then we just exit in # error too. - if ! "${OLDPWD}/support/download/${backend}" ${quiet} "${tmpf}" "${@}"; then + if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then rm -rf "${tmpd}" exit 1 fi
Signature · 1 line
1.9.1
PATCH v2 4/4 core/pkg-infra: download git submodules if the package wants them
All,
Quoted 8 lines · Yann E. MORIN
Add a new package variable that packages can set to specify that they need git submodules.
Only accept this option if the download method is git, as we can not get submodules via an http download (via wget).
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Aleksandar Simeonov <[email protected]>
Quoted 80 lines
---
Changes v1 -> v2:
- properly accept the -r in the download wrapper (Aleksandar)--- package/pkg-download.mk | 1 + package/pkg-generic.mk | 8 ++++++++ support/download/dl-wrapper | 7 ++++--- 3 files changed, 13 insertions(+), 3 deletions(-)package/pkg-download.mk +1
-0@@ -76,6 +76,7 @@ export BR_NO_CHECK_HASH_FOR = define DOWNLOAD_GIT $(EXTRA_ENV) $(DL_WRAPPER) -b git \ -o $(DL_DIR)/$($(PKG)_SOURCE) \ + $(if $($(PKG)_GIT_SUBMODULES),-r) \ $(QUIET) \ -- \ $($(PKG)_SITE) \package/pkg-generic.mk +8
-0@@ -453,6 +453,14 @@ ifndef $(2)_SITE_METHOD endif endif +# Do not accept to download git submodule if not using the git method +ifneq ($$($(2)_GIT_SUBMODULES),) + ifneq ($$($(2)_SITE_METHOD),git) + $$(error $(2) declares having git sub-modules, but does not use the \ + 'git' method (uses '$$($(2)_SITE_METHOD)' instead)) + endif +endif + ifeq ($$($(2)_SITE_METHOD),local) ifeq ($$($(2)_OVERRIDE_SRCDIR),) $(2)_OVERRIDE_SRCDIR = $$($(2)_SITE)support/download/dl-wrapper +4
-4@@ -21,15 +21,16 @@ set -e main() { local OPT OPTARG - local backend output hfile quiet + local backend output hfile recurse quiet # Parse our options; anything after '--' is for the backend - while getopts :hb:o:H:q OPT; do + while getopts :hb:o:H:rq OPT; do case "${OPT}" in h) help; exit 0;; b) backend="${OPTARG}";; o) output="${OPTARG}";; H) hfile="${OPTARG}";; + r) recurse="-r";; q) quiet="-q";; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";;@@ -82,7 +83,7 @@ main() { # If the backend fails, we can just remove the temporary directory to # remove all the cruft it may have left behind. Then we just exit in # error too. - if ! "${OLDPWD}/support/download/${backend}" ${quiet} "${tmpf}" "${@}"; then + if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then rm -rf "${tmpd}" exit 1 fi --1.9.1
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 2 lines
Thanks, Matt
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH v2 4/4 core/pkg-infra: download git submodules if the package wants them
Alex,
Quoted 6 lines · Aleksandar Simeonov
Hi Yann, Mathew,
Have you checked that this patch does not affect the download of packages not using GIT? For me it breaks the download of anything using WGET, for example dosfstools:
Yeah, but to be sure, I just did a clean and explicitly built dosfstools along with a complete build. Would you mind doing a savedefconfig and pastebin your defconifg? Also what hash did you add the patchset on top of (I used 2cd5abf)?
Thanks,
Matt
PATCH v2 4/4 core/pkg-infra: download git submodules if the package wants them
In reply to a message not in the archiveAleksandar, All,
On 2016-04-04 12:38 +0200, Aleksandar Simeonov spake thusly:
Quoted 3 lines
Have you checked that this patch does not affect the download of packages not using GIT? For me it breaks the download of anything using WGET, for example dosfstools:
Yes, I just retried right now, and I can at least download packages using:
- wget (gcc, binutils... and dosfstools)
- mercurial (eigen)
- git with no submodules (opkg)
-git with sub-modules (a local package to test submodules)
and they all downloaded fine, especially dosfstools:
Quoted 1 line
Quoted 1 line
Quoted 1 line
dosfstools 3.0.28 Downloading
--2016-04-04 22:59:53-- https://github.com/dosfstools/dosfstools/releases/download/v3.0.28/dosfstools-3.0.28.tar.xz Connecting to 127.0.0.1:8080... connected. Proxy request sent, awaiting response... 302 Found Location: https://github-cloud.s3.amazonaws.com/releases/26462150/579f8e7e-fb7e-11e4-88ca-bc3c6ebe56b6.xz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20160404%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160404T205953Z&X-Amz-Expires=300&X-Amz-Signature=20455f9da857c4f99c4fc5dd74a12238c53958c728cf469bdc80cb0e7c69e272&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Ddosfstools-3.0.28.tar.xz&response-content-type=application%2Foctet-stream [following] --2016-04-04 22:59:53-- https://github-cloud.s3.amazonaws.com/releases/26462150/579f8e7e-fb7e-11e4-88ca-bc3c6ebe56b6.xz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20160404%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160404T205953Z&X-Amz-Expires=300&X-Amz-Signature=20455f9da857c4f99c4fc5dd74a12238c53958c728cf469bdc80cb0e7c69e272&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Ddosfstools-3.0.28.tar.xz&response-content-type=application%2Foctet-stream Connecting to 127.0.0.1:8080... connected. Proxy request sent, awaiting response... 200 OK Length: 82980 (81K) [application/octet-stream] Saving to: ?/home/ymorin/dev/buildroot/O/build/.dosfstools-3.0.28.tar.xz.wMTT4G/output? 100%[================================================================>] 82,980 228KB/s in 0.4s 2016-04-04 22:59:55 (228 KB/s) - ?/home/ymorin/dev/buildroot/O/build/.dosfstools-3.0.28.tar.xz.wMTT4G/output? saved [82980/82980] dosfstools-3.0.28.tar.xz: OK (sha256: ee95913044ecf2719b63ea11212917649709a6e53209a72d622135aaa8517ee2)
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
dosfstools 3.0.28 Downloading
BACKEND=wget
We have nothing in Buildroot that outputs this "BACKEND" string.
Quoted 11 lines
--2016-04-04 12:28:06-- http://package/dosfstools//dosfstools.hash
Resolving package (package)... failed: Name or service not known.
wget: unable to resolve host address ?package?
BACKEND=wget
--2016-04-04 12:28:06-- http://package/dosfstools//dosfstools.hash
Resolving package (package)... failed: Name or service not known.
wget: unable to resolve host address ?package?
make[1]: *** [/home/alex/workspace/BR2_merge/buildroot/output/build/dosfstools-3.0.28/.stamp_downloaded] Error 1
make: *** [_all] Error 2
Quoted 2 lines
I will try to find out what is wrong, but I strongly suspect it is the ${recurse} option we add in the command.
${recurse} should only be set for a git download. And even if it was passed to the wget backend, you'd have a failure in the backend, because it does not know the -r option.
However, what you report is probably due to somthing else. Have you applied on top of local changes? Can you reproduce the error without this patchset?
Regards,
Yann E. MORIN.
Quoted 103 lines
Cheers
Alex
Quoted 95 lines · Matthew Weber
All,
Quoted 8 lines · Yann E. MORIN
Add a new package variable that packages can set to specify that they need git submodules.
Only accept this option if the download method is git, as we can not get submodules via an http download (via wget).
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Aleksandar Simeonov <[email protected]>Tested-by Matt Weber <[email protected]>Reviewed-by Matt Weber <[email protected]>Quoted 79 lines
---
Changes v1 -> v2:
- properly accept the -r in the download wrapper (Aleksandar)--- package/pkg-download.mk | 1 + package/pkg-generic.mk | 8 ++++++++ support/download/dl-wrapper | 7 ++++--- 3 files changed, 13 insertions(+), 3 deletions(-)package/pkg-download.mk +1
-0@@ -76,6 +76,7 @@ export BR_NO_CHECK_HASH_FOR = define DOWNLOAD_GIT $(EXTRA_ENV) $(DL_WRAPPER) -b git \ -o $(DL_DIR)/$($(PKG)_SOURCE) \ + $(if $($(PKG)_GIT_SUBMODULES),-r) \ $(QUIET) \ -- \ $($(PKG)_SITE) \package/pkg-generic.mk +8
-0@@ -453,6 +453,14 @@ ifndef $(2)_SITE_METHOD endif endif +# Do not accept to download git submodule if not using the git method +ifneq ($$($(2)_GIT_SUBMODULES),) + ifneq ($$($(2)_SITE_METHOD),git) + $$(error $(2) declares having git sub-modules, but does not use the \ + 'git' method (uses '$$($(2)_SITE_METHOD)' instead)) + endif +endif + ifeq ($$($(2)_SITE_METHOD),local) ifeq ($$($(2)_OVERRIDE_SRCDIR),) $(2)_OVERRIDE_SRCDIR = $$($(2)_SITE)support/download/dl-wrapper +4
-4@@ -21,15 +21,16 @@ set -e main() { local OPT OPTARG - local backend output hfile quiet + local backend output hfile recurse quiet # Parse our options; anything after '--' is for the backend - while getopts :hb:o:H:q OPT; do + while getopts :hb:o:H:rq OPT; do case "${OPT}" in h) help; exit 0;; b) backend="${OPTARG}";; o) output="${OPTARG}";; H) hfile="${OPTARG}";; + r) recurse="-r";; q) quiet="-q";; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";;@@ -82,7 +83,7 @@ main() { # If the backend fails, we can just remove the temporary directory to # remove all the cruft it may have left behind. Then we just exit in # error too. - if ! "${OLDPWD}/support/download/${backend}" ${quiet} "${tmpf}" "${@}"; then + if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then rm -rf "${tmpd}" exit 1 fi --1.9.1
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 4/4 core/pkg-infra: download git submodules if the package wants them
Hi Yann,
Quoted 6 lines · Yann E. MORIN
Aleksandar, All,
On 2016-04-04 12:38 +0200, Aleksandar Simeonov spake thusly:
Quoted 3 lines
Have you checked that this patch does not affect the download of packages not using GIT? For me it breaks the download of anything using WGET, for example dosfstools:
Quoted 10 lines
Yes, I just retried right now, and I can at least download packages using:
- wget (gcc, binutils... and dosfstools)
- mercurial (eigen)
- git with no submodules (opkg)
-git with sub-modules (a local package to test submodules)and they all downloaded fine, especially dosfstools:
It was failing on all packages-not only on dosfstools. But it was my fault, I was applying your v2 fixes not in the correct way, so it was failing on whatever package comes next to download. I have re-downloaded them from the link you have posted, and applied them on top of 2cd5abf7f9edd3c0a609e9626f12f71e2a6e4e7c as Mathew has explained, and now all works fine.
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
dosfstools 3.0.28 Downloading
BACKEND=wget
We have nothing in Buildroot that outputs this "BACKEND" string.
This was a debug message I have added
Quoted 3 lines
However, what you report is probably due to somthing else. Have you applied on top of local changes? Can you reproduce the error without this patchset? Regards, Yann E. MORIN.
No, without the patchset works perfectly, With it-as well :) Sorry about the noise, I was not a member of the maillist, so my reply did go only to Mathew Weber. Now should be fine.
Great work, thank you all!
Cheers,
Alex
PATCH v2 4/4 core/pkg-infra: download git submodules if the package wants them
In reply to Yann E. MORINQuoted 8 lines · Yann E. MORIN
Add a new package variable that packages can set to specify that they need git submodules.
Only accept this option if the download method is git, as we can not get submodules via an http download (via wget).
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Aleksandar Simeonov <[email protected]>
Tested on some local packages. We already had ugly local patches to support submodules.
v2 cover core/download: add support for git sub-modules (branch yem/git)
In reply to Yann E. MORINAll,
Quoted 10 lines · Yann E. MORIN
Hello All!
This little series adds support for packages coming from a git clone, with git sub-modules.
The commit logs are pretty detailed, but roughly, it goes as thus;
- we can't get sub-modules in a bare clone, so we no longer use bare
clones, but use full clones (with a working copy);
Quoted 7 lines
- git-archive does not support submodules, so we generate archives
manually;- we check out the working copy to the correct cset, then we retrieve
the submodules;- we expose a new package variable to request sub-modules.
I took the refpolicy package (pending merge) and reworked the patch to use the git submodule feature. I had it download/extract a specific tag and then I observed the submodule checkout. Originally the refpolicy package used two packages to achieve this same functionality. I appreciate the patches!