scripts: use git archive in archive-source
PATCH v2 1/1 scripts: use git archive in archive-source
Use git archive to create tarballs of qemu and submodules instead of cloning the repository and the submodules. This is a order of magnitude faster because it doesn't fetch the submodules from the internet each time the script runs.
---
Notes:
v2: don't run git submodule update (except for uninitialized submodules)
scripts/archive-source.sh | 58 ++++++++++++++++++----------------------------- 1 file changed, 22 insertions(+), 36 deletions(-)
scripts/archive-source.sh +22 -36
@@ -19,8 +19,7 @@ if test $# -lt 1; then fi tar_file=$(realpath "$1") -list_file="${tar_file}.list" -vroot_dir="${tar_file}.vroot" +sub_file=$(mktemp "${tar_file%.tar}.sub.XXXXXXXX.tar") # We want a predictable list of submodules for builds, that is # independent of what the developer currently has initialized
@@ -28,7 +27,7 @@ vroot_dir="${tar_file}.vroot" # different to the host OS. submodules="dtc ui/keycodemapdb tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3" -trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15 +trap "status=$?; rm -rf \"$sub_file\" ; exit \$status" 0 1 2 3 15 if git diff-index --quiet HEAD -- &>/dev/null then
@@ -36,38 +35,25 @@ then else HEAD=$(git stash create) fi -git clone --shared . "$vroot_dir" -test $? -ne 0 && error "failed to clone into '$vroot_dir'" - -cd "$vroot_dir" -test $? -ne 0 && error "failed to change into '$vroot_dir'" - -git checkout $HEAD -test $? -ne 0 && error "failed to checkout $HEAD revision" - +git archive --format tar $HEAD > "$tar_file" +test $? -ne 0 && error "failed to archive qemu" for sm in $submodules; do - git submodule update --init $sm - test $? -ne 0 && error "failed to init submodule $sm" + status="$(git submodule status "$sm")" + smhash="${status# }" + smhash="${smhash#+}" + smhash="${smhash#-}" + smhash="${smhash%% *}" + case "$status" in + -*) + echo "NOTICE: fetching submodule $sm" + git submodule update --init "$sm" + ;; + +*) + echo "WARNING: submodule $sm is out of sync" + ;; + esac + (cd $sm; git archive --format tar --prefix "$sm/" $smhash) > "$sub_file" + test $? -ne 0 && error "failed to archive submodule $sm ($smhash)" + tar --concatenate --file "$tar_file" "$sub_file" + test $? -ne 0 && error "failed append submodule $sm to $tar_file" done - -if test -n "$submodules"; then - { - git ls-files || error "git ls-files failed" - for sm in $submodules; do - (cd $sm; git ls-files) | sed "s:^:$sm/:" - if test "${PIPESTATUS[*]}" != "0 0"; then - error "git ls-files in submodule $sm failed" - fi - done - } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file" -else - git ls-files > "$list_file" -fi - -if test $? -ne 0; then - error "failed to generate list file" -fi - -tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file" - -exit 0
Signature · 1 line
2.9.3
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series seems to have some coding style problems. See output below for more information:
Subject: [Qemu-devel] [PATCH v2] scripts: use git archive in archive-source
Message-id: [email protected]
Type: series
=== TEST SCRIPT BEGIN ===
#!/bin/bash
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===
Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Submodule 'capstone' (https://git.qemu.org/git/capstone.git) registered for path 'capstone'
Submodule 'dtc' (https://git.qemu.org/git/dtc.git) registered for path 'dtc'
Submodule 'roms/QemuMacDrivers' (https://git.qemu.org/git/QemuMacDrivers.git) registered for path 'roms/QemuMacDrivers' Submodule 'roms/SLOF' (https://git.qemu.org/git/SLOF.git) registered for path 'roms/SLOF'
Submodule 'roms/ipxe' (https://git.qemu.org/git/ipxe.git) registered for path 'roms/ipxe'
Submodule 'roms/openbios' (https://git.qemu.org/git/openbios.git) registered for path 'roms/openbios'
Submodule 'roms/openhackware' (https://git.qemu.org/git/openhackware.git) registered for path 'roms/openhackware' Submodule 'roms/qemu-palcode' (https://git.qemu.org/git/qemu-palcode.git) registered for path 'roms/qemu-palcode' Submodule 'roms/seabios' (https://git.qemu.org/git/seabios.git/) registered for path 'roms/seabios'
Submodule 'roms/seabios-hppa' (https://github.com/hdeller/seabios-hppa.git) registered for path 'roms/seabios-hppa' Submodule 'roms/sgabios' (https://git.qemu.org/git/sgabios.git) registered for path 'roms/sgabios'
Submodule 'roms/skiboot' (https://git.qemu.org/git/skiboot.git) registered for path 'roms/skiboot'
Submodule 'roms/u-boot' (https://git.qemu.org/git/u-boot.git) registered for path 'roms/u-boot'
Submodule 'roms/u-boot-sam460ex' (https://git.qemu.org/git/u-boot-sam460ex.git) registered for path 'roms/u-boot-sam460ex' Submodule 'tests/fp/berkeley-softfloat-3' (https://github.com/cota/berkeley-softfloat-3) registered for path 'tests/fp/berkeley-softfloat-3' Submodule 'tests/fp/berkeley-testfloat-3' (https://github.com/cota/berkeley-testfloat-3) registered for path 'tests/fp/berkeley-testfloat-3' Submodule 'ui/keycodemapdb' (https://git.qemu.org/git/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into 'capstone'...
Submodule path 'capstone': checked out '22ead3e0bfdb87516656453336160e0a37b066bf'
Cloning into 'dtc'...
Submodule path 'dtc': checked out '88f18909db731a627456f26d779445f84e449536'
Cloning into 'roms/QemuMacDrivers'...
Submodule path 'roms/QemuMacDrivers': checked out 'd4e7d7ac663fcb55f1b93575445fcbca372f17a7'
Cloning into 'roms/SLOF'...
Submodule path 'roms/SLOF': checked out '9b7ab2fa020341dee8bf9df6c9cf40003e0136df'
Cloning into 'roms/ipxe'...
Submodule path 'roms/ipxe': checked out 'de4565cbe76ea9f7913a01f331be3ee901bb6e17'
Cloning into 'roms/openbios'...
Submodule path 'roms/openbios': checked out '441a84d3a642a10b948369c63f32367e8ff6395b'
Cloning into 'roms/openhackware'...
Submodule path 'roms/openhackware': checked out 'c559da7c8eec5e45ef1f67978827af6f0b9546f5'
Cloning into 'roms/qemu-palcode'...
Submodule path 'roms/qemu-palcode': checked out '51c237d7e20d05100eacadee2f61abc17e6bc097'
Cloning into 'roms/seabios'...
Submodule path 'roms/seabios': checked out 'a698c8995ffb2838296ec284fe3c4ad33dfca307'
Cloning into 'roms/seabios-hppa'...
Submodule path 'roms/seabios-hppa': checked out '1ef99a01572c2581c30e16e6fe69e9ea2ef92ce0'
Cloning into 'roms/sgabios'...
Submodule path 'roms/sgabios': checked out 'cbaee52287e5f32373181cff50a00b6c4ac9015a'
Cloning into 'roms/skiboot'...
Submodule path 'roms/skiboot': checked out 'e0ee24c27a172bcf482f6f2bc905e6211c134bcc'
Cloning into 'roms/u-boot'...
Submodule path 'roms/u-boot': checked out 'd85ca029f257b53a96da6c2fb421e78a003a9943'
Cloning into 'roms/u-boot-sam460ex'...
Submodule path 'roms/u-boot-sam460ex': checked out '60b3916f33e617a815973c5a6df77055b2e3a588'
Cloning into 'tests/fp/berkeley-softfloat-3'...
Submodule path 'tests/fp/berkeley-softfloat-3': checked out 'b64af41c3276f97f0e181920400ee056b9c88037'
Cloning into 'tests/fp/berkeley-testfloat-3'...
Submodule path 'tests/fp/berkeley-testfloat-3': checked out '5a59dcec19327396a011a17fd924aed4fec416b3'
Cloning into 'ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out '6b3d716e2b6472eb7189d3220552280ef3d832ce'
Switched to a new branch 'test'
abfc980 scripts: use git archive in archive-source
=== OUTPUT BEGIN ===
ERROR: code indent should never use tabs
#56: FILE: scripts/archive-source.sh:41:
+^Istatus="$(git submodule status "$sm")"$
ERROR: code indent should never use tabs
#57: FILE: scripts/archive-source.sh:42:
+^Ismhash="${status# }"$
ERROR: code indent should never use tabs
#58: FILE: scripts/archive-source.sh:43:
+^Ismhash="${smhash#+}"$
ERROR: code indent should never use tabs
#59: FILE: scripts/archive-source.sh:44:
+^Ismhash="${smhash#-}"$
ERROR: code indent should never use tabs
#60: FILE: scripts/archive-source.sh:45:
+^Ismhash="${smhash%% *}"$
ERROR: code indent should never use tabs
#61: FILE: scripts/archive-source.sh:46:
+^Icase "$status" in$
ERROR: code indent should never use tabs
#62: FILE: scripts/archive-source.sh:47:
+^I -*)$
ERROR: code indent should never use tabs
#63: FILE: scripts/archive-source.sh:48:
+^I^Iecho "NOTICE: fetching submodule $sm"$
ERROR: code indent should never use tabs
#64: FILE: scripts/archive-source.sh:49:
+^I^Igit submodule update --init "$sm"$
ERROR: code indent should never use tabs
#65: FILE: scripts/archive-source.sh:50:
+^I^I;;$
ERROR: code indent should never use tabs
#66: FILE: scripts/archive-source.sh:51:
+^I +*)$
ERROR: code indent should never use tabs #67: FILE: scripts/archive-source.sh:52:
+^I^Iecho "WARNING: submodule $sm is out of sync"$
ERROR: code indent should never use tabs
#68: FILE: scripts/archive-source.sh:53:
+^I^I;;$
ERROR: code indent should never use tabs
#69: FILE: scripts/archive-source.sh:54:
+^Iesac$
ERROR: code indent should never use tabs
#70: FILE: scripts/archive-source.sh:55:
+^I(cd $sm; git archive --format tar --prefix "$sm/" $smhash) > "$sub_file"$
ERROR: code indent should never use tabs
#71: FILE: scripts/archive-source.sh:56:
+^Itest $? -ne 0 && error "failed to archive submodule $sm ($smhash)"$
ERROR: code indent should never use tabs #72: FILE: scripts/archive-source.sh:57:
+^Itar --concatenate --file "$tar_file" "$sub_file"$
ERROR: code indent should never use tabs
#73: FILE: scripts/archive-source.sh:58:
+^Itest $? -ne 0 && error "failed append submodule $sm to $tar_file"$
total: 18 errors, 0 warnings, 53 lines checked
Commit abfc980abb39 (scripts: use git archive in archive-source) has style problems, please review. If any of these errors are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===
Test command exited with code: 1
The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to [email protected]