scripts: use git archive in archive-source
PATCH 1/1 scripts: use git archive in archive-source
Use git archive to create tarballs of qemu and submodules instead of cloning the repository and the submodules. This is a order of magnitude faster because it doesn't fetch the submodules from the internet each time the script runs.
--- scripts/archive-source.sh | 46 ++++++++++------------------------------------ 1 file changed, 10 insertions(+), 36 deletions(-)
scripts/archive-source.sh +10 -36
@@ -19,8 +19,7 @@ if test $# -lt 1; then fi tar_file=$(realpath "$1") -list_file="${tar_file}.list" -vroot_dir="${tar_file}.vroot" +sub_file=$(mktemp "${tar_file%.tar}.sub.XXXXXXXX.tar") # We want a predictable list of submodules for builds, that is # independent of what the developer currently has initialized
@@ -28,7 +27,7 @@ vroot_dir="${tar_file}.vroot" # different to the host OS. submodules="dtc ui/keycodemapdb tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3" -trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15 +trap "status=$?; rm -rf \"$sub_file\" ; exit \$status" 0 1 2 3 15 if git diff-index --quiet HEAD -- &>/dev/null then
@@ -36,38 +35,13 @@ then else HEAD=$(git stash create) fi -git clone --shared . "$vroot_dir" -test $? -ne 0 && error "failed to clone into '$vroot_dir'" - -cd "$vroot_dir" -test $? -ne 0 && error "failed to change into '$vroot_dir'" - -git checkout $HEAD -test $? -ne 0 && error "failed to checkout $HEAD revision" - +git archive --format tar $HEAD > "$tar_file" +test $? -ne 0 && error "failed to archive qemu" for sm in $submodules; do - git submodule update --init $sm - test $? -ne 0 && error "failed to init submodule $sm" + git submodule update --init "$sm" + test $? -ne 0 && error "failed to update submodule $sm" + (cd $sm; git archive --format tar --prefix "$sm/" HEAD) > "$sub_file" + test $? -ne 0 && error "failed to archive submodule $sm" + tar --concatenate --file "$tar_file" "$sub_file" + test $? -ne 0 && error "failed append submodule $sm to $tar_file" done - -if test -n "$submodules"; then - { - git ls-files || error "git ls-files failed" - for sm in $submodules; do - (cd $sm; git ls-files) | sed "s:^:$sm/:" - if test "${PIPESTATUS[*]}" != "0 0"; then - error "git ls-files in submodule $sm failed" - fi - done - } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file" -else - git ls-files > "$list_file" -fi - -if test $? -ne 0; then - error "failed to generate list file" -fi - -tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file" - -exit 0
Signature · 1 line
2.9.3
Quoted 4 lines · Gerd Hoffmann
Use git archive to create tarballs of qemu and submodules instead of cloning the repository and the submodules. This is a order of magnitude faster because it doesn't fetch the submodules from the internet each time the script runs.
There's another patch addressing the same problem here, which takes the approach of 'git clone' on the submodules to avoid pulling them from the internet each time:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06588.html
Quoted 37 lines
Signed-off-by Gerd Hoffmann <[email protected]>--- scripts/archive-source.sh | 46 ++++++++++------------------------------------ 1 file changed, 10 insertions(+), 36 deletions(-)scripts/archive-source.sh +2
-8@@ -19,8 +19,7 @@ if test $# -lt 1; then fi tar_file=$(realpath "$1") -list_file="${tar_file}.list" -vroot_dir="${tar_file}.vroot" +sub_file=$(mktemp "${tar_file%.tar}.sub.XXXXXXXX.tar") # We want a predictable list of submodules for builds, that is # independent of what the developer currently has initialized@@ -28,7 +27,7 @@ vroot_dir="${tar_file}.vroot" # different to the host OS. submodules="dtc ui/keycodemapdb tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3" -trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15 +trap "status=$?; rm -rf \"$sub_file\" ; exit \$status" 0 1 2 3 15 if git diff-index --quiet HEAD -- &>/dev/null then@@ -36,38 +35,13 @@ then else HEAD=$(git stash create) fi -git clone --shared . "$vroot_dir" -test $? -ne 0 && error "failed to clone into '$vroot_dir'" - -cd "$vroot_dir" -test $? -ne 0 && error "failed to change into '$vroot_dir'"
scripts/archive-source.sh then · line 41
Quoted 10 lines
scripts/archive-source.sh +3
-6- -git checkout $HEAD -test $? -ne 0 && error "failed to checkout $HEAD revision" - +git archive --format tar $HEAD > "$tar_file" +test $? -ne 0 && error "failed to archive qemu" for sm in $submodules; do - git submodule update --init $sm - test $? -ne 0 && error "failed to init submodule $sm" + git submodule update --init "$sm"
The current archive-source.sh code is careful to not touch the developer's current git checkout, because it runs submodule update in a clone'd tree. The earlier patch hunk gets rid of the clone, so now submodule update is touching the developer's master git checkout. IMHO this is not acceptable - running tests (which use archive-source.sh) must never alter the developer's git checkout status in any way.
The alternate patch mentioned above does that.
Even if we keep the git clone part, we could still usefully switch to running git archive, instead of re-inventing it via git ls-files + tar.
Quoted 32 lines
scripts/archive-source.sh +5
-22+ test $? -ne 0 && error "failed to update submodule $sm" + (cd $sm; git archive --format tar --prefix "$sm/" HEAD) > "$sub_file" + test $? -ne 0 && error "failed to archive submodule $sm" + tar --concatenate --file "$tar_file" "$sub_file" + test $? -ne 0 && error "failed append submodule $sm to $tar_file" done - -if test -n "$submodules"; then - { - git ls-files || error "git ls-files failed" - for sm in $submodules; do - (cd $sm; git ls-files) | sed "s:^:$sm/:" - if test "${PIPESTATUS[*]}" != "0 0"; then - error "git ls-files in submodule $sm failed" - fi - done - } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file" -else - git ls-files > "$list_file" -fi - -if test $? -ne 0; then - error "failed to generate list file" -fi - -tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file" - -exit 0Signature · 1 line
2.9.3
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi,
Quoted 7 lines
scripts/archive-source.sh then · line 41
Quoted 6 lines
scripts/archive-source.sh +3
-2+git archive --format tar $HEAD > "$tar_file" +test $? -ne 0 && error "failed to archive qemu" for sm in $submodules; do - git submodule update --init $sm - test $? -ne 0 && error "failed to init submodule $sm" + git submodule update --init "$sm"
Quoted 7 lines
The current
archive-source.shcode is careful to not touch the developer's current git checkout, because it runs submodule update in a clone'd tree. The earlier patch hunk gets rid of the clone, so now submodule update is touching the developer's master git checkout. IMHO this is not acceptable - running tests (which usearchive-source.sh) must never alter the developer's git checkout status in any way.
Can easily be fixed, the sha1 can be pulled out of "git submodule status" instead of updating the submodules & use HEAD.
The only question is what to do best on uninitialized submodules. Options are:
* fetch the submodule into the developer tree (once).
* clone the submodule into a temporary directory (each run).
* throw an error.
cheers,
Gerd
Quoted 17 lines · Gerd Hoffmann
Hi,
Quoted 7 lines
scripts/archive-source.sh then · line 41
Quoted 6 lines
scripts/archive-source.sh +3
-2+git archive --format tar $HEAD > "$tar_file" +test $? -ne 0 && error "failed to archive qemu" for sm in $submodules; do - git submodule update --init $sm - test $? -ne 0 && error "failed to init submodule $sm" + git submodule update --init "$sm"Quoted 7 lines
The current
archive-source.shcode is careful to not touch the developer's current git checkout, because it runs submodule update in a clone'd tree. The earlier patch hunk gets rid of the clone, so now submodule update is touching the developer's master git checkout. IMHO this is not acceptable - running tests (which usearchive-source.sh) must never alter the developer's git checkout status in any way.
Quoted 6 lines
Can easily be fixed, the sha1 can be pulled out of "git submodule status" instead of updating the submodules & use HEAD.
The only question is what to do best on uninitialized submodules. Options are:
* fetch the submodule into the developer tree (once).
As above, we shouldn't ever touch the developer's git state.
Quoted 1 line
* clone the submodule into a temporary directory (each run).
IMHO this is the only option
Quoted 1 line
* throw an error.
Definitely don't want this, as an uninitialized submodule is going to be a common case for the 'dtc' module when people have the suitable version already on their host, so this would break the CI scripts.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series seems to have some coding style problems. See output below for more information:
Subject: [Qemu-devel] [PATCH] scripts: use git archive in archive-source
Type: series
Message-id: [email protected]
=== TEST SCRIPT BEGIN ===
#!/bin/bash
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===
Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384 Switched to a new branch 'test'42fe2deeca scripts: use git archive in archive-source
=== OUTPUT BEGIN ===
ERROR: code indent should never use tabs
#56: FILE: scripts/archive-source.sh:41:
+^Igit submodule update --init "$sm"$
ERROR: code indent should never use tabs
#57: FILE: scripts/archive-source.sh:42:
+^Itest $? -ne 0 && error "failed to update submodule $sm"$
ERROR: code indent should never use tabs
#58: FILE: scripts/archive-source.sh:43:
+^I(cd $sm; git archive --format tar --prefix "$sm/" HEAD) > "$sub_file"$
ERROR: code indent should never use tabs
#59: FILE: scripts/archive-source.sh:44:
+^Itest $? -ne 0 && error "failed to archive submodule $sm"$
ERROR: code indent should never use tabs #60: FILE: scripts/archive-source.sh:45:
+^Itar --concatenate --file "$tar_file" "$sub_file"$
ERROR: code indent should never use tabs
#61: FILE: scripts/archive-source.sh:46:
+^Itest $? -ne 0 && error "failed append submodule $sm to $tar_file"$
total: 6 errors, 0 warnings, 41 lines checked
Commit 42fe2deecad0 (scripts: use git archive in archive-source) has style problems, please review. If any of these errors are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===
Test command exited with code: 1
The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to [email protected]