Performance test framework
cover Performance test framework
Hi,
This is the first shot at a usable performance test suite. It's another angle than the refperf work of Michael Haggerty, however he helped shape this in #git-devel (thanks!).
There's a big README, but if you just want to dive in, here's me comparing the grep performance of my POC "pack reading in parallel" branch against Junio's next:
$ cd t/perf
$ ./run origin/next t/sha1_file-parallel p7810-grep.sh
[snip no-op compilation]
=== Running 1 tests in build/7a6d658aa3c9016dd04ff3515cbf15edca6562a4 ===
perf 1 - grep worktree, cheap regex: 1 2 3 4 5 ok
perf 2 - grep worktree, expensive regex: 1 2 3 4 5 ok
perf 3 - grep --cached, cheap regex: 1 2 3 4 5 ok
perf 4 - grep --cached, expensive regex: 1 2 3 4 5 ok
# passed all 4 test(s)
1..4
GIT_VERSION = 1.7.8.GIT
* new build flags or prefix
* new link flags
GEN common-cmds.h
CC hex.o
CC kwset.o
[snip rest of compilation]
=== Running 1 tests in build/dd2bf650b382f5aca727b7d93a48598fb1a2f7d9 ===
perf 1 - grep worktree, cheap regex: 1 2 3 4 5 ok
perf 2 - grep worktree, expensive regex: 1 2 3 4 5 ok
perf 3 - grep --cached, cheap regex: 1 2 3 4 5 ok
perf 4 - grep --cached, expensive regex: 1 2 3 4 5 ok
# passed all 4 test(s)
1..4
Test origin/next t/sha1_file-parallel
----------------------------------------------------------------------------------
7810.1: grep worktree, cheap regex 0.16(0.16+0.35) 0.16(0.15+0.36) +0.0%
7810.2: grep worktree, expensive regex 7.83(29.68+0.39) 7.95(29.98+0.39) +1.5%
7810.3: grep --cached, cheap regex 3.12(3.11+0.24) 1.11(3.46+0.18) -64.4%
7810.4: grep --cached, expensive regex 9.43(30.53+0.28) 8.89(32.99+0.22) -5.7%Note in particular that neither of the two branches contains the perf work.
Have fun!
Thomas Rast (4):
Move the user-facing test library to test-lib-functions.sh
test-lib: allow testing another git build tree Introduce a performance testing framework
Add a performance test for git-grep
Makefile | 26 ++- t/Makefile | 5 +- t/perf/.gitignore | 2 + t/perf/Makefile | 15 + t/perf/README | 146 ++++++++++ t/perf/aggregate.perl | 166 ++++++++++++ t/perf/min_time.perl | 21 ++ t/perf/p0000-perf-lib-sanity.sh | 41 +++ t/perf/p0001-rev-list.sh | 17 ++ t/perf/p7810-grep.sh | 23 ++ t/perf/perf-lib.sh | 199 ++++++++++++++ t/perf/run | 82 ++++++ t/test-lib-functions.sh | 528 ++++++++++++++++++++++++++++++++++++ t/test-lib.sh | 561 +++------------------------------------ 14 files changed, 1300 insertions(+), 532 deletions(-) create mode 100644 t/perf/.gitignore create mode 100644 t/perf/Makefile create mode 100644 t/perf/README create mode 100755 t/perf/aggregate.perl create mode 100755 t/perf/min_time.perl create mode 100755 t/perf/p0000-perf-lib-sanity.sh create mode 100755 t/perf/p0001-rev-list.sh create mode 100755 t/perf/p7810-grep.sh create mode 100644 t/perf/perf-lib.sh create mode 100755 t/perf/run create mode 100644 t/test-lib-functions.sh
Signature · 1 line
1.7.8.304.ge42e4
RFC PATCH 1/4 Move the user-facing test library to test-lib-functions.sh
This just moves all the user-facing functions to a separate file and sources that instead.
--- t/test-lib-functions.sh | 528 ++++++++++++++++++++++++++++++++++++++++++++++ t/test-lib.sh | 530 +---------------------------------------------- 2 files changed, 533 insertions(+), 525 deletions(-) create mode 100644 t/test-lib-functions.sh
t/test-lib-functions.sh new file +528 -0
@@ -0,0 +1,528 @@ +# The semantics of the editor variables are that of invoking +# sh -c "$EDITOR \"$@\"" files ... +# +# If our trash directory contains shell metacharacters, they will be +# interpreted if we just set $EDITOR directly, so do a little dance with +# environment variables to work around this. +# +# In particular, quoting isn't enough, as the path may contain the same quote +# that we're using. +test_set_editor () { + FAKE_EDITOR="$1" + export FAKE_EDITOR + EDITOR='"$FAKE_EDITOR"' + export EDITOR +} + +test_decode_color () { + awk ' + function name(n) { + if (n == 0) return "RESET"; + if (n == 1) return "BOLD"; + if (n == 30) return "BLACK"; + if (n == 31) return "RED"; + if (n == 32) return "GREEN"; + if (n == 33) return "YELLOW"; + if (n == 34) return "BLUE"; + if (n == 35) return "MAGENTA"; + if (n == 36) return "CYAN"; + if (n == 37) return "WHITE"; + if (n == 40) return "BLACK"; + if (n == 41) return "BRED"; + if (n == 42) return "BGREEN"; + if (n == 43) return "BYELLOW"; + if (n == 44) return "BBLUE"; + if (n == 45) return "BMAGENTA"; + if (n == 46) return "BCYAN"; + if (n == 47) return "BWHITE"; + } + { + while (match($0, /\033\[[0-9;]*m/) != 0) { + printf "%s<", substr($0, 1, RSTART-1); + codes = substr($0, RSTART+2, RLENGTH-3); + if (length(codes) == 0) + printf "%s", name(0) + else { + n = split(codes, ary, ";"); + sep = ""; + for (i = 1; i <= n; i++) { + printf "%s%s", sep, name(ary[i]); + sep = ";" + } + } + printf ">"; + $0 = substr($0, RSTART + RLENGTH, length($0) - RSTART - RLENGTH + 1); + } + print + } + ' +} + +nul_to_q () { + perl -pe 'y/\000/Q/' +} + +q_to_nul () { + perl -pe 'y/Q/\000/' +} + +q_to_cr () { + tr Q '\015' +} + +q_to_tab () { + tr Q '\011' +} + +append_cr () { + sed -e 's/$/Q/' | tr Q '\015' +} + +remove_cr () { + tr '\015' Q | sed -e 's/Q$//' +} + +# In some bourne shell implementations, the "unset" builtin returns +# nonzero status when a variable to be unset was not set in the first +# place. +# +# Use sane_unset when that should not be considered an error. + +sane_unset () { + unset "$@" + return 0 +} + +test_tick () { + if test -z "${test_tick+set}" + then + test_tick=1112911993 + else + test_tick=$(($test_tick + 60)) + fi + GIT_COMMITTER_DATE="$test_tick -0700" + GIT_AUTHOR_DATE="$test_tick -0700" + export GIT_COMMITTER_DATE GIT_AUTHOR_DATE +} + +# Call test_commit with the arguments "<message> [<file> [<contents>]]" +# +# This will commit a file with the given contents and the given commit +# message. It will also add a tag with <message> as name. +# +# Both <file> and <contents> default to <message>. + +test_commit () { + file=${2:-"$1.t"} + echo "${3-$1}" > "$file" && + git add "$file" && + test_tick && + git commit -m "$1" && + git tag "$1" +} + +# Call test_merge with the arguments "<message> <commit>", where <commit> +# can be a tag pointing to the commit-to-merge. + +test_merge () { + test_tick && + git merge -m "$1" "$2" && + git tag "$1" +} + +# This function helps systems where core.filemode=false is set. +# Use it instead of plain 'chmod +x' to set or unset the executable bit +# of a file in the working directory and add it to the index. + +test_chmod () { + chmod "$@" && + git update-index --add "--chmod=$@" +} + +# Unset a configuration variable, but don't fail if it doesn't exist. +test_unconfig () { + git config --unset-all "$@" + config_status=$? + case "$config_status" in + 5) # ok, nothing to unset + config_status=0 + ;; + esac + return $config_status +} + +# Set git config, automatically unsetting it after the test is over. +test_config () { + test_when_finished "test_unconfig '$1'" && + git config "$@" +} + +test_config_global () { + test_when_finished "test_unconfig --global '$1'" && + git config --global "$@" +} + +# Use test_set_prereq to tell that a particular prerequisite is available. +# The prerequisite can later be checked for in two ways: +# +# - Explicitly using test_have_prereq. +# +# - Implicitly by specifying the prerequisite tag in the calls to +# test_expect_{success,failure,code}. +# +# The single parameter is the prerequisite tag (a simple word, in all +# capital letters by convention). + +test_set_prereq () { + satisfied="$satisfied$1 " +} +satisfied=" " + +test_have_prereq () { + # prerequisites can be concatenated with ',' + save_IFS=$IFS + IFS=, + set -- $* + IFS=$save_IFS + + total_prereq=0 + ok_prereq=0 + missing_prereq= + + for prerequisite + do + total_prereq=$(($total_prereq + 1)) + case $satisfied in + *" $prerequisite "*) + ok_prereq=$(($ok_prereq + 1)) + ;; + *) + # Keep a list of missing prerequisites + if test -z "$missing_prereq" + then + missing_prereq=$prerequisite + else + missing_prereq="$prerequisite,$missing_prereq" + fi + esac + done + + test $total_prereq = $ok_prereq +} + +test_declared_prereq () { + case ",$test_prereq," in + *,$1,*) + return 0 + ;; + esac + return 1 +} + +test_expect_failure () { + test "$#" = 3 && { test_prereq=$1; shift; } || test_prereq= + test "$#" = 2 || + error "bug in the test script: not 2 or 3 parameters to test-expect-failure" + export test_prereq + if ! test_skip "$@" + then + say >&3 "checking known breakage: $2" + if test_run_ "$2" expecting_failure + then + test_known_broken_ok_ "$1" + else + test_known_broken_failure_ "$1" + fi + fi + echo >&3 "" +} + +test_expect_success () { + test "$#" = 3 && { test_prereq=$1; shift; } || test_prereq= + test "$#" = 2 || + error "bug in the test script: not 2 or 3 parameters to test-expect-success" + export test_prereq + if ! test_skip "$@" + then + say >&3 "expecting success: $2" + if test_run_ "$2" + then + test_ok_ "$1" + else + test_failure_ "$@" + fi + fi + echo >&3 "" +} + +# test_external runs external test scripts that provide continuous +# test output about their progress, and succeeds/fails on +# zero/non-zero exit code. It outputs the test output on stdout even +# in non-verbose mode, and announces the external script with "# run +# <n>: ..." before running it. When providing relative paths, keep in +# mind that all scripts run in "trash directory". +# Usage: test_external description command arguments... +# Example: test_external 'Perl API' perl ../path/to/test.pl +test_external () { + test "$#" = 4 && { test_prereq=$1; shift; } || test_prereq= + test "$#" = 3 || + error >&5 "bug in the test script: not 3 or 4 parameters to test_external" + descr="$1" + shift + export test_prereq + if ! test_skip "$descr" "$@" + then + # Announce the script to reduce confusion about the + # test output that follows. + say_color "" "# run $test_count: $descr ($*)" + # Export TEST_DIRECTORY, TRASH_DIRECTORY and GIT_TEST_LONG + # to be able to use them in script + export TEST_DIRECTORY TRASH_DIRECTORY GIT_TEST_LONG + # Run command; redirect its stderr to &4 as in + # test_run_, but keep its stdout on our stdout even in + # non-verbose mode. + "$@" 2>&4 + if [ "$?" = 0 ] + then + if test $test_external_has_tap -eq 0; then + test_ok_ "$descr" + else + say_color "" "# test_external test $descr was ok" + test_success=$(($test_success + 1)) + fi + else + if test $test_external_has_tap -eq 0; then + test_failure_ "$descr" "$@" + else + say_color error "# test_external test $descr failed: $@" + test_failure=$(($test_failure + 1)) + fi + fi + fi +} + +# Like test_external, but in addition tests that the command generated +# no output on stderr. +test_external_without_stderr () { + # The temporary file has no (and must have no) security + # implications. + tmp=${TMPDIR:-/tmp} + stderr="$tmp/git-external-stderr.$$.tmp" + test_external "$@" 4> "$stderr" + [ -f "$stderr" ] || error "Internal error: $stderr disappeared." + descr="no stderr: $1" + shift + say >&3 "# expecting no stderr from previous command" + if [ ! -s "$stderr" ]; then + rm "$stderr" + + if test $test_external_has_tap -eq 0; then + test_ok_ "$descr" + else + say_color "" "# test_external_without_stderr test $descr was ok" + test_success=$(($test_success + 1)) + fi + else + if [ "$verbose" = t ]; then + output=`echo; echo "# Stderr is:"; cat "$stderr"` + else + output= + fi + # rm first in case test_failure exits. + rm "$stderr" + if test $test_external_has_tap -eq 0; then + test_failure_ "$descr" "$@" "$output" + else + say_color error "# test_external_without_stderr test $descr failed: $@: $output" + test_failure=$(($test_failure + 1)) + fi + fi +} + +# debugging-friendly alternatives to "test [-f|-d|-e]" +# The commands test the existence or non-existence of $1. $2 can be +# given to provide a more precise diagnosis. +test_path_is_file () { + if ! [ -f "$1" ] + then + echo "File $1 doesn't exist. $*" + false + fi +} + +test_path_is_dir () { + if ! [ -d "$1" ] + then + echo "Directory $1 doesn't exist. $*" + false + fi +} + +test_path_is_missing () { + if [ -e "$1" ] + then + echo "Path exists:" + ls -ld "$1" + if [ $# -ge 1 ]; then + echo "$*" + fi + false + fi +} + +# test_line_count checks that a file has the number of lines it +# ought to. For example: +# +# test_expect_success 'produce exactly one line of output' ' +# do something >output && +# test_line_count = 1 output +# ' +# +# is like "test $(wc -l <output) = 1" except that it passes the +# output through when the number of lines is wrong. + +test_line_count () { + if test $# != 3 + then + error "bug in the test script: not 3 parameters to test_line_count" + elif ! test $(wc -l <"$3") "$1" "$2" + then + echo "test_line_count: line count for $3 !$1 $2" + cat "$3" + return 1 + fi +} + +# This is not among top-level (test_expect_success | test_expect_failure) +# but is a prefix that can be used in the test script, like: +# +# test_expect_success 'complain and die' ' +# do something && +# do something else && +# test_must_fail git checkout ../outerspace +# ' +# +# Writing this as "! git checkout ../outerspace" is wrong, because +# the failure could be due to a segv. We want a controlled failure. + +test_must_fail () { + "$@" + exit_code=$? + if test $exit_code = 0; then + echo >&2 "test_must_fail: command succeeded: $*" + return 1 + elif test $exit_code -gt 129 -a $exit_code -le 192; then + echo >&2 "test_must_fail: died by signal: $*" + return 1 + elif test $exit_code = 127; then + echo >&2 "test_must_fail: command not found: $*" + return 1 + fi + return 0 +} + +# Similar to test_must_fail, but tolerates success, too. This is +# meant to be used in contexts like: +# +# test_expect_success 'some command works without configuration' ' +# test_might_fail git config --unset all.configuration && +# do something +# ' +# +# Writing "git config --unset all.configuration || :" would be wrong, +# because we want to notice if it fails due to segv. + +test_might_fail () { + "$@" + exit_code=$? + if test $exit_code -gt 129 -a $exit_code -le 192; then + echo >&2 "test_might_fail: died by signal: $*" + return 1 + elif test $exit_code = 127; then + echo >&2 "test_might_fail: command not found: $*" + return 1 + fi + return 0 +} + +# Similar to test_must_fail and test_might_fail, but check that a +# given command exited with a given exit code. Meant to be used as: +# +# test_expect_success 'Merge with d/f conflicts' ' +# test_expect_code 1 git merge "merge msg" B master +# ' + +test_expect_code () { + want_code=$1 + shift + "$@" + exit_code=$? + if test $exit_code = $want_code + then + return 0 + fi + + echo >&2 "test_expect_code: command exited with $exit_code, we wanted $want_code $*" + return 1 +} + +# test_cmp is a helper function to compare actual and expected output. +# You can use it like: +# +# test_expect_success 'foo works' ' +# echo expected >expected && +# foo >actual && +# test_cmp expected actual +# ' +# +# This could be written as either "cmp" or "diff -u", but: +# - cmp's output is not nearly as easy to read as diff -u +# - not all diff versions understand "-u" + +test_cmp() { + $GIT_TEST_CMP "$@" +} + +# This function can be used to schedule some commands to be run +# unconditionally at the end of the test to restore sanity: +# +# test_expect_success 'test core.capslock' ' +# git config core.capslock true && +# test_when_finished "git config --unset core.capslock" && +# hello world +# ' +# +# That would be roughly equivalent to +# +# test_expect_success 'test core.capslock' ' +# git config core.capslock true && +# hello world +# git config --unset core.capslock +# ' +# +# except that the greeting and config --unset must both succeed for +# the test to pass. +# +# Note that under --immediate mode, no clean-up is done to help diagnose +# what went wrong. + +test_when_finished () { + test_cleanup="{ $* + } && (exit \"\$eval_ret\"); eval_ret=\$?; $test_cleanup" +} + +# Most tests can use the created repository, but some may need to create more. +# Usage: test_create_repo <directory> +test_create_repo () { + test "$#" = 1 || + error "bug in the test script: not 1 parameter to test-create-repo" + repo="$1" + mkdir -p "$repo" + ( + cd "$repo" || error "Cannot setup test environment" + "$GIT_EXEC_PATH/git-init" "--template=$GIT_BUILD_DIR/templates/blt/" >&3 2>&4 || + error "cannot run git init -- have you built things yet?" + mv .git/hooks .git/hooks-disabled + ) || exit +} +
t/test-lib.sh +5 -525
548 lines not on this page · raw message
Signature · 1 line
1.7.8.304.ge42e4
RFC PATCH 2/4 test-lib: allow testing another git build tree
In reply to Thomas RastThe perf-lib work wants this feature, so we may as well do it for test-lib in general. You can now say
GIT_BUILD_DIR=/another/git/tree
make test # or any other test
and it will run the tests of the current tree against the binaries of the other tree.
--- Makefile | 7 ++++++- t/test-lib.sh | 19 ++++++++++++++++--- 2 files changed, 22 insertions(+), 4 deletions(-)
Makefile +6 -1
@@ -2215,6 +2215,12 @@ GIT-BUILD-OPTIONS: FORCE @echo USE_LIBPCRE=\''$(subst ','\'',$(subst ','\'',$(USE_LIBPCRE)))'\' >>$@ @echo NO_PERL=\''$(subst ','\'',$(subst ','\'',$(NO_PERL)))'\' >>$@ @echo NO_PYTHON=\''$(subst ','\'',$(subst ','\'',$(NO_PYTHON)))'\' >>$@ + @echo NO_UNIX_SOCKETS=\''$(subst ','\'',$(subst ','\'',$(NO_UNIX_SOCKETS)))'\' >>$@ + +GIT-TEST-OPTIONS: FORCE +ifdef GIT_TEST_OPTS + @echo GIT_TEST_OPTS=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_OPTS)))'\' >$@ +endif ifdef GIT_TEST_CMP @echo GIT_TEST_CMP=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_CMP)))'\' >>$@ endif
@@ -2222,7 +2228,6 @@ ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT @echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@ endif @echo GETTEXT_POISON=\''$(subst ','\'',$(subst ','\'',$(GETTEXT_POISON)))'\' >>$@ - @echo NO_UNIX_SOCKETS=\''$(subst ','\'',$(subst ','\'',$(NO_UNIX_SOCKETS)))'\' >>$@ ### Detect Tck/Tk interpreter path changes ifndef NO_TCLTK
t/test-lib.sh +16 -3
@@ -54,6 +54,7 @@ unset $(perl -e ' .*_TEST PROVE VALGRIND + BUILD_DIR )); my @vars = grep(/^GIT_/ && !/^GIT_($ok)/o, @env); print join("\n", @vars);
@@ -318,7 +319,7 @@ test_done () { GIT_EXIT_OK=t if test -z "$HARNESS_ACTIVE"; then - test_results_dir="$TEST_DIRECTORY/test-results" + test_results_dir="$TEST_OUTPUT_DIRECTORY/test-results" mkdir -p "$test_results_dir" test_results_path="$test_results_dir/${0%.sh}-$$.counts"
@@ -379,7 +380,18 @@ then # itself. TEST_DIRECTORY=$(pwd) fi -GIT_BUILD_DIR="$TEST_DIRECTORY"/.. +if test -z "$TEST_OUTPUT_DIRECTORY" +then + # Similarly, override this to store the test-results subdir + # elsewhere + TEST_OUTPUT_DIRECTORY=$TEST_DIRECTORY +fi +if test -z "$GIT_BUILD_DIR" +then + # Similarly, override this to test a different git build tree + # than the one you are running the tests from + GIT_BUILD_DIR="$TEST_DIRECTORY"/.. +fi if test -n "$valgrind" then
@@ -477,6 +489,7 @@ GIT_ATTR_NOSYSTEM=1 export PATH GIT_EXEC_PATH GIT_TEMPLATE_DIR GIT_CONFIG_NOSYSTEM GIT_ATTR_NOSYSTEM . "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS +. "$TEST_DIRECTORY"/../GIT-TEST-OPTIONS if test -z "$GIT_TEST_CMP" then
@@ -514,7 +527,7 @@ test="trash directory.$(basename "$0" .sh)" test -n "$root" && test="$root/$test" case "$test" in /*) TRASH_DIRECTORY="$test" ;; - *) TRASH_DIRECTORY="$TEST_DIRECTORY/$test" ;; + *) TRASH_DIRECTORY="$TEST_OUTPUT_DIRECTORY/$test" ;; esac test ! -z "$debug" || remove_trash=$TRASH_DIRECTORY rm -fr "$test" || {
Signature · 1 line
1.7.8.304.ge42e4
Re: [RFC PATCH 2/4] test-lib: allow testing another git build tree
Quoted 2 lines · Thomas Rast
The perf-lib work wants this feature, so we may as well do it for test-lib in general.
How is this different from what GIT_TEST_INSTALLED already gives us (other than "needs more diskspace to keep another source tree fully built", that is)?
Re: [RFC PATCH 2/4] test-lib: allow testing another git build tree
Quoted 7 lines · On Wednesday 14 December 2011 19:07:35 Junio C Hamano wrote:
Quoted 2 lines · Thomas Rast
The perf-lib work wants this feature, so we may as well do it for test-lib in general.
How is this different from what
GIT_TEST_INSTALLEDalready gives us (other than "needs more diskspace to keep another source tree fully built", that is)?
I was scared away by the note that it would use (among others) perl libs from the current build tree. Upon investigation I also see that the test-* situation is still not satisfactory. Some (like test-chmtime) are used by the tests for a vital task, and if they ever have to be fixed, we would want to use the fixed version in any "test an old git" run. OTOH, others (e.g., test-dump-cache-tree) are linked with the rest of the code and serve to test an otherwise not accessible part of it, and testing an old git should use them from the tested tree.
The disk space argument is moot IMO: for sane perf testing you need the extra build tree anyway because you cannot checkout another version in the current tree. Otherwise the scripts may change and/or disappear from under themselves.
An optimization might be to have the run script only use a single build tree and several install trees. However, while such a built tree takes just over 100MB of space in my tests, everything installed $PREFIX/libexec/git-core is also already 65MB here. So the latter scheme would only amortize itself if you had at least 3 trees tested simultaneously.
Signature · 2 lines
Thomas Rast
trast@{inf,student}.ethz.chRFC PATCH 3/4 Introduce a performance testing framework
In reply to Thomas RastThis introduces a performance testing framework under t/perf/. It tries to be as close to the test-lib.sh infrastructure as possible, and thus should be easy to get used to for git developers.
The following points were considered for the implementation:
1. You usually want to compare arbitrary revisions/build trees against each other. They may not have the performance test under consideration, or even the perf-lib.sh infrastructure.
To cope with this, the 'run' script lets you specify arbitrary build dirs and revisions. It even automatically builds the revisions if it doesn't have them at hand yet.
2. Usually you would not want to run all tests. It would take too long anyway. The 'run' script lets you specify which tests to run; or you can also do it manually. There is a Makefile for discoverability and 'make clean', but it is not meant for real-world use.
3. Creating test repos from scratch in every test is extremely time-consuming, and shipping or downloading such large/weird repos is out of the question.
We leave this decision to the user. Two different sizes of test repos can be configured, and the scripts just copy one or more of those (using hardlinks for the object store). By default it tries to use the build tree's git.git repository.
This is fairly fast and versatile. Using a copy instead of a clone preserves many properties that the user may want to test for, such as lots of loose objects, unpacked refs, etc.
--- Makefile | 19 ++++- t/Makefile | 5 +- t/perf/.gitignore | 2 + t/perf/Makefile | 15 +++ t/perf/README | 146 ++++++++++++++++++++++++++++ t/perf/aggregate.perl | 166 ++++++++++++++++++++++++++++++++ t/perf/min_time.perl | 21 ++++ t/perf/p0000-perf-lib-sanity.sh | 41 ++++++++ t/perf/p0001-rev-list.sh | 17 ++++ t/perf/perf-lib.sh | 199 +++++++++++++++++++++++++++++++++++++++ t/perf/run | 82 ++++++++++++++++ t/test-lib.sh | 12 ++- 12 files changed, 722 insertions(+), 3 deletions(-) create mode 100644 t/perf/.gitignore create mode 100644 t/perf/Makefile create mode 100644 t/perf/README create mode 100755 t/perf/aggregate.perl create mode 100755 t/perf/min_time.perl create mode 100755 t/perf/p0000-perf-lib-sanity.sh create mode 100755 t/perf/p0001-rev-list.sh create mode 100644 t/perf/perf-lib.sh create mode 100755 t/perf/run
Makefile +18 -1
@@ -1769,7 +1769,7 @@ export DIFF TAR INSTALL DESTDIR SHELL_PATH SHELL = $(SHELL_PATH) -all:: shell_compatibility_test $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) GIT-BUILD-OPTIONS +all:: shell_compatibility_test $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) GIT-BUILD-OPTIONS GIT-TEST-OPTIONS ifneq (,$X) $(QUIET_BUILT_IN)$(foreach p,$(patsubst %$X,%,$(filter %$X,$(ALL_PROGRAMS) $(BUILT_INS) git$X)), test -d '$p' -o '$p' -ef '$p$X' || $(RM) '$p';) endif
@@ -2228,6 +2228,18 @@ ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT @echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@ endif @echo GETTEXT_POISON=\''$(subst ','\'',$(subst ','\'',$(GETTEXT_POISON)))'\' >>$@ +ifdef GIT_PERF_REPEAT_COUNT + @echo GIT_PERF_REPEAT_COUNT=\''$(subst ','\'',$(subst ','\'',$(GIT_PERF_REPEAT_COUNT)))'\' >>$@ +endif +ifdef GIT_PERF_REPO + @echo GIT_PERF_REPO=\''$(subst ','\'',$(subst ','\'',$(GIT_PERF_REPO)))'\' >>$@ +endif +ifdef GIT_PERF_LARGE_REPO + @echo GIT_PERF_LARGE_REPO=\''$(subst ','\'',$(subst ','\'',$(GIT_PERF_LARGE_REPO)))'\' >>$@ +endif +ifdef GIT_PERF_MAKE_OPTS + @echo GIT_PERF_MAKE_OPTS=\''$(subst ','\'',$(subst ','\'',$(GIT_PERF_MAKE_OPTS)))'\' >>$@ +endif ### Detect Tck/Tk interpreter path changes ifndef NO_TCLTK
@@ -2263,6 +2275,11 @@ export NO_SVN_TESTS test: all $(MAKE) -C t/ all +perf: all + $(MAKE) -C t/perf/ all + +.PHONY: test perf + test-ctype$X: ctype.o test-date$X: date.o ctype.o
t/Makefile +4 -1
@@ -73,6 +73,9 @@ gitweb-test: valgrind: $(MAKE) GIT_TEST_OPTS="$(GIT_TEST_OPTS) --valgrind" +perf: + $(MAKE) -C perf/ all + # Smoke testing targets -include ../GIT-VERSION-FILE uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo unknown')
@@ -111,4 +114,4 @@ smoke_report: smoke http://smoke.git.nix.is/app/projects/process_add_report/1 \ | grep -v ^Redirecting -.PHONY: pre-clean $(T) aggregate-results clean valgrind smoke smoke_report +.PHONY: pre-clean $(T) aggregate-results clean valgrind smoke smoke_report perf
t/perf/.gitignore new file +2 -0
@@ -0,0 +1,2 @@ +build/ +test-results/
t/perf/Makefile new file +15 -0
@@ -0,0 +1,15 @@ +-include ../../config.mak +export GIT_TEST_OPTIONS + +all: perf + +perf: pre-clean + ./run + +pre-clean: + rm -rf test-results + +clean: + rm -rf build "trash directory".* test-results + +.PHONY: all perf pre-clean clean
t/perf/README new file +146 -0
146 lines not on this page · raw message
t/perf/aggregate.perl new file +166 -0
166 lines not on this page · raw message
t/perf/min_time.perl new file +21 -0
@@ -0,0 +1,21 @@ +#!/usr/bin/perl + +my $minrt = 1e100; +my $min; + +while (<>) { + # [h:]m:s.xx U.xx S.xx + /^(?:(\d+):)?(\d+):(\d+(?:\.\d+)?) (\d+(?:\.\d+)?) (\d+(?:\.\d+)?)$/ + or die "bad input line: $_"; + my $rt = ((defined $1 ? $1 : 0.0)*60+$2)*60+$3; + if ($rt < $minrt) { + $min = $_; + $minrt = $rt; + } +} + +if (!defined $min) { + die "no input found"; +} + +print $min;
t/perf/p0000-perf-lib-sanity.sh new file +41 -0
41 lines not on this page · raw message
t/perf/p0001-rev-list.sh new file +17 -0
@@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Tests history walking performance" + +. ./perf-lib.sh + +test_perf_default_repo + +test_perf 'rev-list --all' ' + git rev-list --all >/dev/null +' + +test_perf 'rev-list --all --objects' ' + git rev-list --all --objects >/dev/null +' + +test_done
t/perf/perf-lib.sh new file +199 -0
199 lines not on this page · raw message
t/perf/run new file +82 -0
82 lines not on this page · raw message
t/test-lib.sh +11 -1
@@ -55,6 +55,7 @@ unset $(perl -e ' PROVE VALGRIND BUILD_DIR + PERF_AGGREGATING_LATER )); my @vars = grep(/^GIT_/ && !/^GIT_($ok)/o, @env); print join("\n", @vars);
@@ -313,6 +314,9 @@ test_skip () { esac } +# stub; perf-lib overrides it +test_at_end_hook_ () { + : } test_done () {
@@ -358,6 +362,8 @@ test_done () { cd "$(dirname "$remove_trash")" && rm -rf "$(basename "$remove_trash")" + test_at_end_hook_ + exit 0 ;; *)
@@ -539,7 +545,11 @@ rm -fr "$test" || { HOME="$TRASH_DIRECTORY" export HOME -test_create_repo "$test" +if test -z "$TEST_NO_CREATE_REPO"; then + test_create_repo "$test" +else + mkdir -p "$test" +fi # Use -P to resolve symlinks in our working directory so that the cwd # in subprocesses like git equals our $PWD (for pathname comparisons). cd -P "$test" || exit 1
Signature · 1 line
1.7.8.304.ge42e4
Re: [RFC PATCH 3/4] Introduce a performance testing framework
This patch actually needs the below fixup on top. Old files from previous runs had fooled my testing...
t/perf/perf-lib.sh +1 -1
@@ -154,7 +154,7 @@ test_perf () { then base=$(basename "$0" .sh) echo "$test_count" >>"$perf_results_dir"/$base.subtests - echo "$1" >"$perf_results_dir"/$base.descr + echo "$1" >"$perf_results_dir"/$base.$test_count.descr if test -z "$verbose"; then echo -n "perf $test_count - $1:" else
Signature · 2 lines
Thomas Rast
trast@{inf,student}.ethz.chRFC PATCH 4/4 Add a performance test for git-grep
In reply to Thomas RastThe only catch is that we don't really know what our repo contains, so we have to ignore any possible "not found" status from git-grep.
--- t/perf/p7810-grep.sh | 23 +++++++++++++++++++++++ 1 files changed, 23 insertions(+), 0 deletions(-) create mode 100755 t/perf/p7810-grep.sh
t/perf/p7810-grep.sh new file +23 -0
@@ -0,0 +1,23 @@ +#!/bin/sh + +test_description="git-grep performance in various modes" + +. ./perf-lib.sh + +test_perf_large_repo +test_checkout_worktree + +test_perf 'grep worktree, cheap regex' ' + git grep some_nonexistent_string || : +' +test_perf 'grep worktree, expensive regex' ' + git grep "^.* *some_nonexistent_string$" || : +' +test_perf 'grep --cached, cheap regex' ' + git grep --cached some_nonexistent_string || : +' +test_perf 'grep --cached, expensive regex' ' + git grep --cached "^.* *some_nonexistent_string$" || : +' + +test_done
Signature · 1 line
1.7.8.304.ge42e4
Re: [RFC PATCH 2/4] test-lib: allow testing another git build tree
In reply to Thomas RastQuoted 9 lines · Thomas Rast
Quoted 7 lines · On Wednesday 14 December 2011 19:07:35 Junio C Hamano wrote:
Quoted 2 lines · Thomas Rast
The perf-lib work wants this feature, so we may as well do it for test-lib in general.
How is this different from what
GIT_TEST_INSTALLEDalready gives us (other than "needs more diskspace to keep another source tree fully built", that is)?
Quoted 9 lines
I was scared away by the note that it would use (among others) perl libs from the current build tree. Upon investigation I also see that the test-* situation is still not satisfactory. Some (like test-chmtime) are used by the tests for a vital task, and if they ever have to be fixed, we would want to use the fixed version in any "test an old git" run. OTOH, others (e.g., test-dump-cache-tree) are linked with the rest of the code and serve to test an otherwise not accessible part of it, and testing an old git should use them from the tested tree.
Okay, so GIT_BUILD_DIR variant should now use everything from the tree being tested for consistency to improve the situation, right?
Ehh, not quite, if you worry about an old one having broken test-chmtime, you actively do not want that consistency; how is that addressed?
Not that I think we should address such issues---these standalone programs tend to be small and simple, so always using them from the tree being tested is a much better policy than using them from the tree you are running the tests from.
Quoted 1 line
The disk space argument is moot IMO:
I was not really worried about "space" (disks are cheap) but more about mental burden of having to keep multiple build trees in the source area. Keeping multiple installed versions is already necessary to be able to quickly respond to "I am using Git version X and this does not work".