Skip to contentSkip to conversation

scripts: use git archive in archive-source

PATCH Superseded Gerd Hoffmann 3 participants 5 messages Jan 30, 2019 – Jan 31, 2019

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scripts: use git archive in archive-source

Use git archive to create tarballs of qemu and submodules instead of cloning the repository and the submodules. This is a order of magnitude faster because it doesn't fetch the submodules from the internet each time the script runs.

---
 scripts/archive-source.sh | 46 ++++++++++------------------------------------
 1 file changed, 10 insertions(+), 36 deletions(-)
scripts/archive-source.sh +10 -36
index 6eed2a29bd..beae968f80 100755
@@ -19,8 +19,7 @@ if test $# -lt 1; then
 fi
 
 tar_file=$(realpath "$1")
-list_file="${tar_file}.list"
-vroot_dir="${tar_file}.vroot"
+sub_file=$(mktemp "${tar_file%.tar}.sub.XXXXXXXX.tar")
 
 # We want a predictable list of submodules for builds, that is
 # independent of what the developer currently has initialized
@@ -28,7 +27,7 @@ vroot_dir="${tar_file}.vroot"
 # different to the host OS.
 submodules="dtc ui/keycodemapdb tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
 
-trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15
+trap "status=$?; rm -rf \"$sub_file\" ; exit \$status" 0 1 2 3 15
 
 if git diff-index --quiet HEAD -- &>/dev/null
 then
@@ -36,38 +35,13 @@ then
 else
     HEAD=$(git stash create)
 fi
-git clone --shared . "$vroot_dir"
-test $? -ne 0 && error "failed to clone into '$vroot_dir'"
-
-cd "$vroot_dir"
-test $? -ne 0 && error "failed to change into '$vroot_dir'"
-
-git checkout $HEAD
-test $? -ne 0 && error "failed to checkout $HEAD revision"
-
+git archive --format tar $HEAD > "$tar_file"
+test $? -ne 0 && error "failed to archive qemu"
 for sm in $submodules; do
-    git submodule update --init $sm
-    test $? -ne 0 && error "failed to init submodule $sm"
+	git submodule update --init "$sm"
+	test $? -ne 0 && error "failed to update submodule $sm"
+	(cd $sm; git archive --format tar --prefix "$sm/" HEAD) > "$sub_file"
+	test $? -ne 0 && error "failed to archive submodule $sm"
+	tar --concatenate --file "$tar_file" "$sub_file"
+	test $? -ne 0 && error "failed append submodule $sm to $tar_file"
 done
-
-if test -n "$submodules"; then
-    {
-        git ls-files || error "git ls-files failed"
-        for sm in $submodules; do
-            (cd $sm; git ls-files) | sed "s:^:$sm/:"
-            if test "${PIPESTATUS[*]}" != "0 0"; then
-                error "git ls-files in submodule $sm failed"
-            fi
-        done
-    } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file"
-else
-    git ls-files > "$list_file"
-fi
-
-if test $? -ne 0; then
-    error "failed to generate list file"
-fi
-
-tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
-
-exit 0
Signature · 1 line
2.9.3
#1 lore raw
Quoted 4 lines · Gerd Hoffmann

Use git archive to create tarballs of qemu and submodules instead of cloning the repository and the submodules. This is a order of magnitude faster because it doesn't fetch the submodules from the internet each time the script runs.

There's another patch addressing the same problem here, which takes the approach of 'git clone' on the submodules to avoid pulling them from the internet each time:

https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06588.html

Quoted 37 lines
---
 scripts/archive-source.sh | 46 ++++++++++------------------------------------
 1 file changed, 10 insertions(+), 36 deletions(-)
scripts/archive-source.sh +2 -8
index 6eed2a29bd..beae968f80 100755
@@ -19,8 +19,7 @@ if test $# -lt 1; then
 fi
 
 tar_file=$(realpath "$1")
-list_file="${tar_file}.list"
-vroot_dir="${tar_file}.vroot"
+sub_file=$(mktemp "${tar_file%.tar}.sub.XXXXXXXX.tar")
 
 # We want a predictable list of submodules for builds, that is
 # independent of what the developer currently has initialized
@@ -28,7 +27,7 @@ vroot_dir="${tar_file}.vroot"
 # different to the host OS.
 submodules="dtc ui/keycodemapdb tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
 
-trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15
+trap "status=$?; rm -rf \"$sub_file\" ; exit \$status" 0 1 2 3 15
 
 if git diff-index --quiet HEAD -- &>/dev/null
 then
@@ -36,38 +35,13 @@ then
 else
     HEAD=$(git stash create)
 fi
-git clone --shared . "$vroot_dir"
-test $? -ne 0 && error "failed to clone into '$vroot_dir'"
-
-cd "$vroot_dir"
-test $? -ne 0 && error "failed to change into '$vroot_dir'"

scripts/archive-source.sh then · line 41

Quoted 10 lines
scripts/archive-source.sh +3 -6
-
-git checkout $HEAD
-test $? -ne 0 && error "failed to checkout $HEAD revision"
-
+git archive --format tar $HEAD > "$tar_file"
+test $? -ne 0 && error "failed to archive qemu"
 for sm in $submodules; do
-    git submodule update --init $sm
-    test $? -ne 0 && error "failed to init submodule $sm"
+	git submodule update --init "$sm"

The current archive-source.sh code is careful to not touch the developer's current git checkout, because it runs submodule update in a clone'd tree. The earlier patch hunk gets rid of the clone, so now submodule update is touching the developer's master git checkout. IMHO this is not acceptable - running tests (which use archive-source.sh) must never alter the developer's git checkout status in any way.

The alternate patch mentioned above does that.

Even if we keep the git clone part, we could still usefully switch to running git archive, instead of re-inventing it via git ls-files + tar.

Quoted 32 lines
scripts/archive-source.sh +5 -22
+	test $? -ne 0 && error "failed to update submodule $sm"
+	(cd $sm; git archive --format tar --prefix "$sm/" HEAD) > "$sub_file"
+	test $? -ne 0 && error "failed to archive submodule $sm"
+	tar --concatenate --file "$tar_file" "$sub_file"
+	test $? -ne 0 && error "failed append submodule $sm to $tar_file"
 done
-
-if test -n "$submodules"; then
-    {
-        git ls-files || error "git ls-files failed"
-        for sm in $submodules; do
-            (cd $sm; git ls-files) | sed "s:^:$sm/:"
-            if test "${PIPESTATUS[*]}" != "0 0"; then
-                error "git ls-files in submodule $sm failed"
-            fi
-        done
-    } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file"
-else
-    git ls-files > "$list_file"
-fi
-
-if test $? -ne 0; then
-    error "failed to generate list file"
-fi
-
-tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
-
-exit 0
Signature · 1 line
2.9.3

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#2 lore raw

Hi,

Quoted 7 lines

scripts/archive-source.sh then · line 41

Quoted 6 lines
scripts/archive-source.sh +3 -2
+git archive --format tar $HEAD > "$tar_file"
+test $? -ne 0 && error "failed to archive qemu"
 for sm in $submodules; do
-    git submodule update --init $sm
-    test $? -ne 0 && error "failed to init submodule $sm"
+	git submodule update --init "$sm"
Quoted 7 lines

The current archive-source.sh code is careful to not touch the developer's current git checkout, because it runs submodule update in a clone'd tree. The earlier patch hunk gets rid of the clone, so now submodule update is touching the developer's master git checkout. IMHO this is not acceptable - running tests (which use archive-source.sh) must never alter the developer's git checkout status in any way.

Can easily be fixed, the sha1 can be pulled out of "git submodule status" instead of updating the submodules & use HEAD.

The only question is what to do best on uninitialized submodules. Options are:
* fetch the submodule into the developer tree (once).
* clone the submodule into a temporary directory (each run).
* throw an error.

cheers,
Gerd

#3 lore raw
Quoted 17 lines · Gerd Hoffmann

Hi,

Quoted 7 lines

scripts/archive-source.sh then · line 41

Quoted 6 lines
scripts/archive-source.sh +3 -2
+git archive --format tar $HEAD > "$tar_file"
+test $? -ne 0 && error "failed to archive qemu"
 for sm in $submodules; do
-    git submodule update --init $sm
-    test $? -ne 0 && error "failed to init submodule $sm"
+	git submodule update --init "$sm"
Quoted 7 lines

The current archive-source.sh code is careful to not touch the developer's current git checkout, because it runs submodule update in a clone'd tree. The earlier patch hunk gets rid of the clone, so now submodule update is touching the developer's master git checkout. IMHO this is not acceptable - running tests (which use archive-source.sh) must never alter the developer's git checkout status in any way.

Quoted 6 lines

Can easily be fixed, the sha1 can be pulled out of "git submodule status" instead of updating the submodules & use HEAD.

The only question is what to do best on uninitialized submodules. Options are:
* fetch the submodule into the developer tree (once).

As above, we shouldn't ever touch the developer's git state.

Quoted 1 line

* clone the submodule into a temporary directory (each run).

IMHO this is the only option

Quoted 1 line

* throw an error.

Definitely don't want this, as an uninitialized submodule is going to be a common case for the 'dtc' module when people have the suitable version already on their host, so this would break the CI scripts.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw
In reply to Gerd Hoffmann
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series seems to have some coding style problems. See output below for more information:

Subject: [Qemu-devel] [PATCH] scripts: use git archive in archive-source
Type: series
Message-id: [email protected]

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384 Switched to a new branch 'test'
42fe2deeca scripts: use git archive in archive-source

=== OUTPUT BEGIN ===
ERROR: code indent should never use tabs
#56: FILE: scripts/archive-source.sh:41:
+^Igit submodule update --init "$sm"$

ERROR: code indent should never use tabs
#57: FILE: scripts/archive-source.sh:42:
+^Itest $? -ne 0 && error "failed to update submodule $sm"$

ERROR: code indent should never use tabs
#58: FILE: scripts/archive-source.sh:43:
+^I(cd $sm; git archive --format tar --prefix "$sm/" HEAD) > "$sub_file"$

ERROR: code indent should never use tabs
#59: FILE: scripts/archive-source.sh:44:
+^Itest $? -ne 0 && error "failed to archive submodule $sm"$

ERROR: code indent should never use tabs #60: FILE: scripts/archive-source.sh:45:
+^Itar --concatenate --file "$tar_file" "$sub_file"$

ERROR: code indent should never use tabs
#61: FILE: scripts/archive-source.sh:46:
+^Itest $? -ne 0 && error "failed append submodule $sm to $tar_file"$

total: 6 errors, 0 warnings, 41 lines checked

Commit 42fe2deecad0 (scripts: use git archive in archive-source) has style problems, please review. If any of these errors are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1

The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to [email protected]