t/perf/p3400: speed up setup using fast-import
RFC PATCH 1/1 t/perf/p3400: speed up setup using fast-import
The setup phase in 't/perf/p3400-rebase.sh' generates 100 commits to simulate a noisy history. It currently uses a shell loop that invokes 'git add', 'git commit', 'test_seq', and 'sort' in each iteration. This makes the whole process slow.
Optimize the setup by using 'git fast-import' to generate the commit history in a single stream. Additionally, pre-compute the forward and reversed file contents to avoid repetitive execution of 'seq' and 'sort'.
Performance enhancement:
Before: 29.045s
After: 18.081s
Measured on Lenovo Yoga 2020, Ubuntu 24.04.
--- t/perf/p3400-rebase.sh | 53 +++++++++++++++++++++++++++++------------- 1 file changed, 37 insertions(+), 16 deletions(-)
t/perf/p3400-rebase.sh +37 -16
@@ -9,25 +9,46 @@ test_expect_success 'setup rebasing on top of a lot of changes' ' git checkout -f -B base && git checkout -B to-rebase && git checkout -B upstream && - for i in $(test_seq 100) - do - # simulate huge diffs - echo change$i >unrelated-file$i && - test_seq 1000 >>unrelated-file$i && - git add unrelated-file$i && - test_tick && - git commit -m commit$i unrelated-file$i && - echo change$i >unrelated-file$i && - test_seq 1000 | sort -nr >>unrelated-file$i && - git add unrelated-file$i && - test_tick && - git commit -m commit$i-reverse unrelated-file$i || - return 1 - done && + + test_seq 1000 >content_fwd && + test_seq 1000 | sort -nr >content_rev && + + ( + for i in $(test_seq 100) + do + echo "commit refs/heads/upstream" && + echo "committer WGYDY <[email protected]> $i +0000" && + echo "data <<EOF" && + echo "commit$i" && + echo "EOF" && + + if test "$i" = 1; then + echo "from refs/heads/upstream^0" + fi && + + echo "M 100644 inline unrelated-file$i" && + echo "data <<EOF" && + echo "change$i" && + cat content_fwd && + echo "EOF" && + + echo "commit refs/heads/upstream" && + echo "committer WGYDY <[email protected]> $i +0000" && + echo "data <<EOF" && + echo "commit$i-reversed" && + echo "EOF" && + echo "M 100644 inline unrelated-file$i" && + echo "data <<EOF" && + echo "change$i" && + cat content_rev && + echo "EOF" || return 1 + done + ) | git fast-import && + + git checkout -f upstream && git checkout to-rebase && test_commit our-patch interesting-file ' - test_perf 'rebase on top of a lot of unrelated changes' ' git rebase --onto upstream HEAD^ && git rebase --onto base HEAD^
Signature · 1 line
2.43.0
I noticed that while the setup phase is much faster (as shown in the commit message), the actual rebase performance test regressed slightly by about 3 seconds (from ~2s to ~5s). I don't quite understand what led to this outcome.
Additionally, I haven't delved deeply into the technical details of git fast-import,
and I'm not entirely sure if my approach is reasonable and compliant with the specifications.
I would greatly appreciate any guidance from those knowledgeable in these area!
Regards,
Yuchen
Tian Yuchen <[email protected]> writes:
Quoted 10 lines
I noticed that while the setup phase is much faster (as shown in the commit message), the actual rebase performance test regressed slightly by about 3 seconds (from ~2s to ~5s). I don't quite understand what led to this outcome.
Additionally, I haven't delved deeply into the technical details of
git fast-import,
and I'm not entirely sure if my approach is reasonable and compliant with the specifications.
Quoted 5 lines
I would greatly appreciate any guidance from those knowledgeable in these area!
Regards,
Yuchen
I highly suspect that git fast-import generate the packfile that is NOT delta compressed. According to git fast-import documentation:
Quoted 3 lines
...For this reason it is strongly recommended that users repack the repository with 'git repack -a -d' after fast-import completes, allowing Git to reorganize the packfiles for faster data access...
Could this be the reason why it's even slower than handling “loose objects”? I'm working
on it currently.
Regards,
Yuchen
Quoted 3 lines
...For this reason it is strongly recommended that users repack the repository with 'git repack -a -d' after fast-import completes, allowing Git to reorganize the packfiles for faster data access...
After adding 'git repack -a -d', the time does changed:
original(no change) patch v1 NEW
real 29.845 18.081 22.231
rebase 13.34 14.94 12.78
It seems that the 'git repack -a -d' line does reduce runtime. However, the total duration
has seen a noticeable performance regression. I'm still trying to figure it out.
With regards,
Yuchen
PATCH v2 1/1 t/perf/p3400: speed up setup using fast-import
In reply to Tian YuchenThe setup phase in 't/perf/p3400-rebase.sh' generates 100 commits to simulate a noisy history. It currently uses a shell loop that invokes 'git add', 'git commit', 'test_seq', and 'sort' in each iteration. This incurs significant overhead due to repeated process spawning.
Optimize the setup by using 'git fast-import' to generate the commit history in a single stream. Additionally, pre-compute the forward and reversed file contents to avoid repetitive execution of 'seq' and 'sort'.
To ensure the test measures rebase performance against a consistent object layout (rather than the suboptimal pack/loose objects created by the raw import), perform a full repack (git repack -a -d) at the end of the setup.
This reduces the setup time significantly while maintaining the validity of the subsequent performance tests.
Performance enhancement:
Real Rebase Before: 29.045s 13.34s After: 22.231s 12.78s
Measured on Lenovo Yoga 2020, Ubuntu 24.04.
---
Changes since v1:
- Added 'git repack -a -d' at the end of the setup phase.
- This fixes a performance regression observed in the subsequent rebase test, ensuring the object layout is normalized before testing.
- Updated setup timing in the commit message to reflect the repack overhead.
t/perf/p3400-rebase.sh | 54 +++++++++++++++++++++++++++++------------- 1 file changed, 38 insertions(+), 16 deletions(-)
t/perf/p3400-rebase.sh +38 -16
@@ -9,25 +9,47 @@ test_expect_success 'setup rebasing on top of a lot of changes' ' git checkout -f -B base && git checkout -B to-rebase && git checkout -B upstream && - for i in $(test_seq 100) - do - # simulate huge diffs - echo change$i >unrelated-file$i && - test_seq 1000 >>unrelated-file$i && - git add unrelated-file$i && - test_tick && - git commit -m commit$i unrelated-file$i && - echo change$i >unrelated-file$i && - test_seq 1000 | sort -nr >>unrelated-file$i && - git add unrelated-file$i && - test_tick && - git commit -m commit$i-reverse unrelated-file$i || - return 1 - done && + + test_seq 1000 >content_fwd && + test_seq 1000 | sort -nr >content_rev && + + ( + for i in $(test_seq 100) + do + echo "commit refs/heads/upstream" && + echo "committer WGYDY <[email protected]> $i +0000" && + echo "data <<EOF" && + echo "commit$i" && + echo "EOF" && + + if test "$i" = 1; then + echo "from refs/heads/upstream^0" + fi && + + echo "M 100644 inline unrelated-file$i" && + echo "data <<EOF" && + echo "change$i" && + cat content_fwd && + echo "EOF" && + + echo "commit refs/heads/upstream" && + echo "committer WGYDY <[email protected]> $i +0000" && + echo "data <<EOF" && + echo "commit$i-reversed" && + echo "EOF" && + echo "M 100644 inline unrelated-file$i" && + echo "data <<EOF" && + echo "change$i" && + cat content_rev && + echo "EOF" || return 1 + done + ) | git fast-import && + + git repack -a -d && + git checkout -f upstream && git checkout to-rebase && test_commit our-patch interesting-file ' - test_perf 'rebase on top of a lot of unrelated changes' ' git rebase --onto upstream HEAD^ && git rebase --onto base HEAD^
Signature · 1 line
2.43.0
Am 28.01.26 um 17:07 schrieb Tian Yuchen:
t/perf/p3400-rebase.sh test_expect_success 'setup rebasing on top of a lot of changes' ' · line 14
Quoted 2 lines
t/perf/p3400-rebase.sh +2
-0+ test_seq 1000 >content_fwd && + test_seq 1000 | sort -nr >content_rev &&
Remove another pipe with
sort -nr content_fwd >content_rev &&
t/perf/p3400-rebase.sh test_expect_success 'setup rebasing on top of a lot of changes' ' · line 44
Quoted 7 lines
t/perf/p3400-rebase.sh +7
-0+ + ( + for i in $(test_seq 100) + do + ... + cat content_rev && + echo "EOF" || return 1
This is an incorrect use of return outside a function. This must be || exit 1 (or just || exit) to actually break out of the loop.
Of course, the error code is ignored, because it is in the upstream of a pipe, so in order to fail the complete command, it would be necessary that the original error left an incorrect fast-import stream such that the down-stream git fast-import fails. Otherwise, the entire command can complete successfully in an unexpected way.
Maybe instead of a pipe, dump the stream into a temporary file, so that the loop's exit code can be captured?
t/perf/p3400-rebase.sh test_expect_success 'setup rebasing on top of a lot of changes' ' · line 46
Quoted 2 lines
t/perf/p3400-rebase.sh +2
-0+ done + ) | git fast-import &&
-- Hannes
Am 30.01.26 um 07:41 schrieb Johannes Sixt:
Quoted 11 lines
Am 28.01.26 um 17:07 schrieb Tian Yuchen:
t/perf/p3400-rebase.sh test_expect_success 'setup rebasing on top of a lot of changes' ' · line 44
Quoted 7 lines
t/perf/p3400-rebase.sh +7
-0+ + ( + for i in $(test_seq 100) + do + ... + cat content_rev && + echo "EOF" || return 1This is an incorrect use of
returnoutside a function. This must be|| exit 1(or just|| exit) to actually break out of the loop.
I am wrong here, because my earlier tests were too narrow. This use of return actually does work as intended, because it happens inside a function. Neither the subshell nor being in the upstream of a pipe negates this fact.
The rest of the comment still stands, though.
Quoted 11 lines
Of course, the error code is ignored, because it is in the upstream of a pipe, so in order to fail the complete command, it would be necessary that the original error left an incorrect fast-import stream such that the down-stream
git fast-importfails. Otherwise, the entire command can complete successfully in an unexpected way.Maybe instead of a pipe, dump the stream into a temporary file, so that the loop's exit code can be captured?
t/perf/p3400-rebase.sh test_expect_success 'setup rebasing on top of a lot of changes' ' · line 46
Quoted 2 lines
t/perf/p3400-rebase.sh +2
-0+ done + ) | git fast-import &&
-- Hannes
Quoted 7 lines · Johannes Sixt
Am 28.01.26 um 17:07 schrieb Tian Yuchen:
t/perf/p3400-rebase.sh test_expect_success 'setup rebasing on top of a lot of changes' ' · line 14
Quoted 2 lines
t/perf/p3400-rebase.sh +2
-0+ test_seq 1000 >content_fwd && + test_seq 1000 | sort -nr >content_rev &&Remove another pipe with
sort -nr content_fwd >content_rev &&
I agree with the outcome (i.e., we do not run test_seq twice) but not necessarily with the motivation (i.e., we do not assume test_seq may fail and care about its failure code, so no need to avoid pipes for the sake of avoiding pipes).
Quoted 9 lines
t/perf/p3400-rebase.sh test_expect_success 'setup rebasing on top of a lot of changes' ' · line 44
Quoted 6 lines
t/perf/p3400-rebase.sh +6
-0+ ( + for i in $(test_seq 100) + do + ... + cat content_rev && + echo "EOF" || return 1This is an incorrect use of
returnoutside a function. This must be|| exit 1(or just|| exit) to actually break out of the loop.
Good.
Quoted 8 lines
Of course, the error code is ignored, because it is in the upstream of a pipe, so in order to fail the complete command, it would be necessary that the original error left an incorrect fast-import stream such that the down-stream
git fast-importfails. Otherwise, the entire command can complete successfully in an unexpected way.Maybe instead of a pipe, dump the stream into a temporary file, so that the loop's exit code can be captured?
I like this suggestion. A temporary file, unless it is excessively large, has an additional advantage that it allows you to inspect after "git fast-import" dies on it.
Quoted 3 lines
t/perf/p3400-rebase.sh test_expect_success 'setup rebasing on top of a lot of changes' ' · line 46
Quoted 2 lines
t/perf/p3400-rebase.sh +2
-0+ done + ) | git fast-import &&-- Hannes
A message that is not in the archive (sent off-list or not yet received): [email protected]
Am 30.01.26 um 17:39 schrieb Tian Yuchen:
Quoted 1 line
I will send V4 shortly.
In this case, please also undo the unnecessary removal of a blank line at the end of the patch.
-- Hannes
Quoted 14 lines · Tian Yuchen
The setup phase in '
t/perf/p3400-rebase.sh' generates 100 commits to simulate a noisy history. It currently uses a shell loop that invokes 'git add', 'git commit', 'test_seq', and 'sort' in each iteration. This incurs significant overhead due to repeated process spawning.Optimize the setup by using 'git fast-import' to generate the commit history in a single stream. Additionally, pre-compute the forward and reversed file contents to avoid repetitive execution of 'seq' and 'sort'.
To ensure the test measures rebase performance against a consistent object layout (rather than the suboptimal pack/loose objects created by the raw import), perform a full repack (
git repack -a -d) at the end of the setup.
Quoted 7 lines
This reduces the setup time significantly while maintaining the validity of the subsequent performance tests.
Performance enhancement:
Real Rebase Before: 29.045s 13.34s After: 22.231s 12.78s
That's a nice speedup in the test setup
Quoted 22 lines
t/perf/p3400-rebase.sh +0
-14@@ -9,25 +9,47 @@ test_expect_success 'setup rebasing on top of a lot of changes' ' git checkout -f -B base && git checkout -B to-rebase && git checkout -B upstream && - for i in $(test_seq 100) - do - # simulate huge diffs - echo change$i >unrelated-file$i && - test_seq 1000 >>unrelated-file$i && - git add unrelated-file$i && - test_tick && - git commit -m commit$i unrelated-file$i && - echo change$i >unrelated-file$i && - test_seq 1000 | sort -nr >>unrelated-file$i && - git add unrelated-file$i && - test_tick && - git commit -m commit$i-reverse unrelated-file$i || - return 1
t/perf/p3400-rebase.sh test_expect_success 'setup rebasing on top of a lot of changes' ' · line 20
Quoted 10 lines
t/perf/p3400-rebase.sh +9
-1- done && + + test_seq 1000 >content_fwd && + test_seq 1000 | sort -nr >content_rev && + + ( + for i in $(test_seq 100) + do + echo "commit refs/heads/upstream" && + echo "committer WGYDY <[email protected]> $i +0000" &&
You can keep the same author and committer as the original with
test_tick && echo "author $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL> $GIT_AUTHOR_DATE" && echo "committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL>
$GIT_COMMITTER_DATE" &&
here and below
Quoted 8 lines
t/perf/p3400-rebase.sh +8
-0+ echo "data <<EOF" && + echo "commit$i" && + echo "EOF" && + + if test "$i" = 1; then + echo "from refs/heads/upstream^0" + fi && +
t/perf/p3400-rebase.sh test_expect_success 'setup rebasing on top of a lot of changes' ' · line 38
Quoted 10 lines
t/perf/p3400-rebase.sh +10
-0+ echo "M 100644 inline unrelated-file$i" && + echo "data <<EOF" && + echo "change$i" && + cat content_fwd && + echo "EOF" && + + echo "commit refs/heads/upstream" && + echo "committer WGYDY <[email protected]> $i +0000" && + echo "data <<EOF" && + echo "commit$i-reversed" &&
The commit message in the original is "commit$i-reverse", not "reversed"
t/perf/p3400-rebase.sh test_expect_success 'setup rebasing on top of a lot of changes' ' · line 45
Quoted 7 lines
t/perf/p3400-rebase.sh +7
-0+ echo "EOF" && + echo "M 100644 inline unrelated-file$i" && + echo "data <<EOF" && + echo "change$i" && + cat content_rev && + echo "EOF" || return 1 + done
As Johannes pointed out we'll ignore the any failure above. We can address that by adding "echo done" here and adding "--done" to "git fast-import" below. That will cause "git fast-import" to fail because if there is an error in the loop as the last line of input to fast-import will not be "done"
Thanks
Phillip
Quoted 11 lines
t/perf/p3400-rebase.sh +4
-1+ ) | git fast-import && + + git repack -a -d && + git checkout -f upstream && git checkout to-rebase && test_commit our-patch interesting-file ' - test_perf 'rebase on top of a lot of unrelated changes' ' git rebase --onto upstream HEAD^ && git rebase --onto base HEAD^
Hi Johannes, Hi Phillip,
Thanks a lot for the detailed review and the great suggestions!
@Johannes: I will adopt the 'sort -nr content_fwd' optimization to save a process.
@Phillip: Your point about the pipe swallowing the exit code is very insightful. I will
implement the '--done' protocol to ensure any failure in the loop
causes the test to fail.
I will also update the author/committer generation to use test_tick and the standard
environment variables, and fix the "reversed" typo.
I will prepare and send a v3 patch shortly incorporating these changes.
Thanks,
Tian Yuchen
PATCH v3 1/1 t/perf/p3400: speed up setup using fast-import
In reply to Tian YuchenThe setup phase in 't/perf/p3400-rebase.sh' generates 100 commits to simulate a noisy history. It currently uses a shell loop that invokes 'git add', 'git commit', 'test_seq', and 'sort' in each iteration. This incurs significant overhead due to repeated process spawning.
Optimize the setup by using 'git fast-import' to generate the commit history in a single stream. Additionally, pre-compute the forward and reversed file contents to avoid repetitive execution of 'seq' and 'sort'.
To ensure the test measures rebase performance against a consistent object layout (rather than the suboptimal pack/loose objects created by the raw import), perform a full repack (git repack -a -d) at the end of the setup.
This reduces the setup time significantly while maintaining the validity of the subsequent performance tests.
Performance enhancement (Average value of 5 tests):
Real Rebase Before: 29.045s 13.34s After: 22.431s 12.98s
Measured on Lenovo Yoga 2020, Ubuntu 24.04.
---
Changes since v2:
- Optimized content_rev generation by sorting content_fwd directly (Johannes Sixt).
- Used test_tick and standard $GIT_COMMITTER_* variables for consistency (Phillip Wood).
- Fixed typo "reversed" -> "reverse" to match original test (Phillip Wood).
- Added "done" command and used git fast-import --done to prevent errors in the loop being ignored by the pipe (Phillip Wood).
t/perf/p3400-rebase.sh | 56 ++++++++++++++++++++++++++++++------------ 1 file changed, 40 insertions(+), 16 deletions(-)
t/perf/p3400-rebase.sh +40 -16
@@ -9,25 +9,49 @@ test_expect_success 'setup rebasing on top of a lot of changes' ' git checkout -f -B base && git checkout -B to-rebase && git checkout -B upstream && - for i in $(test_seq 100) - do - # simulate huge diffs - echo change$i >unrelated-file$i && - test_seq 1000 >>unrelated-file$i && - git add unrelated-file$i && - test_tick && - git commit -m commit$i unrelated-file$i && - echo change$i >unrelated-file$i && - test_seq 1000 | sort -nr >>unrelated-file$i && - git add unrelated-file$i && - test_tick && - git commit -m commit$i-reverse unrelated-file$i || - return 1 - done && + + test_seq 1000 >content_fwd && + sort -nr content_fwd >content_rev && + + ( + for i in $(test_seq 100) + do + test_tick && + echo "commit refs/heads/upstream" && + echo "committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE" && + echo "data <<EOF" && + echo "commit$i" && + echo "EOF" && + + if test "$i" = 1; then + echo "from refs/heads/upstream^0" + fi && + + echo "M 100644 inline unrelated-file$i" && + echo "data <<EOF" && + echo "change$i" && + cat content_fwd && + echo "EOF" && + + echo "commit refs/heads/upstream" && + echo "committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE" && + echo "data <<EOF" && + echo "commit$i-reverse" && + echo "EOF" && + echo "M 100644 inline unrelated-file$i" && + echo "data <<EOF" && + echo "change$i" && + cat content_rev && + echo "EOF" || return 1 + done && + echo "done" + ) | git fast-import --done && + + git repack -a -d && + git checkout -f upstream && git checkout to-rebase && test_commit our-patch interesting-file ' - test_perf 'rebase on top of a lot of unrelated changes' ' git rebase --onto upstream HEAD^ && git rebase --onto base HEAD^
Signature · 1 line
2.43.0
PATCH v4 1/1 t/perf/p3400: speed up setup using fast-import
The setup phase in 't/perf/p3400-rebase.sh' generates 100 commits to simulate a noisy history. It currently uses a shell loop that invokes 'git add', 'git commit', 'test_seq', and 'sort' in each iteration. This incurs significant overhead due to repeated process spawning.
Optimize the setup by using 'git fast-import' to generate the commit history. Additionally, pre-compute the forward and reversed file contents to avoid repetitive execution of 'seq' and 'sort'.
To ensure the test measures rebase performance against a consistent object layout (rather than the suboptimal pack/loose objects created by the raw import), perform a full repack (git repack -a -d) at the end of the setup.
This reduces the setup time significantly while maintaining the validity of the subsequent performance tests.
Performance enhancement (Average value of 5 tests):
Real Rebase Before: 29.045s 13.34s After: 21.989s 12.84s
Measured on Lenovo Yoga 2020, Ubuntu 24.04.
---
Changes since v3:
- Refactored the fast-import generation to write to a temporary file (fast_import_stream) instead of using a pipe. This allows for better debugging if fast-import fails, as suggested by Johannes Sixt and endorsed by Junio C Hamano.
- Switched from return 1 to exit 1 inside the subshell to correctly propagate exit codes.
- Undid the unnecessary removal of a blank line.
t/perf/p3400-rebase.sh | 53 ++++++++++++++++++++++++++++++------------ 1 file changed, 38 insertions(+), 15 deletions(-)
t/perf/p3400-rebase.sh +38 -15
@@ -9,21 +9,44 @@ test_expect_success 'setup rebasing on top of a lot of changes' ' git checkout -f -B base && git checkout -B to-rebase && git checkout -B upstream && - for i in $(test_seq 100) - do - # simulate huge diffs - echo change$i >unrelated-file$i && - test_seq 1000 >>unrelated-file$i && - git add unrelated-file$i && - test_tick && - git commit -m commit$i unrelated-file$i && - echo change$i >unrelated-file$i && - test_seq 1000 | sort -nr >>unrelated-file$i && - git add unrelated-file$i && - test_tick && - git commit -m commit$i-reverse unrelated-file$i || - return 1 - done && + test_seq 1000 >content_fwd && + sort -nr content_fwd >content_rev && + ( + for i in $(test_seq 100) + do + test_tick && + echo "commit refs/heads/upstream" && + echo "committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE" && + echo "data <<EOF" && + echo "commit$i" && + echo "EOF" && + + if test "$i" = 1; then + echo "from refs/heads/upstream^0" + fi && + + echo "M 100644 inline unrelated-file$i" && + echo "data <<EOF" && + echo "change$i" && + cat content_fwd && + echo "EOF" && + + echo "commit refs/heads/upstream" && + echo "committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE" && + echo "data <<EOF" && + echo "commit$i-reverse" && + echo "EOF" && + echo "M 100644 inline unrelated-file$i" && + echo "data <<EOF" && + echo "change$i" && + cat content_rev && + echo "EOF" || exit 1 + done + ) >fast_import_stream && + + git fast-import <fast_import_stream && + git repack -a -d && + git checkout -f upstream && git checkout to-rebase && test_commit our-patch interesting-file '
Signature · 1 line
2.43.0
Quoted 5 lines · Tian Yuchen
Changes since v2:
- Optimizedcontent_revgeneration by sortingcontent_fwddirectly (Johannes Sixt).
- Usedtest_tickand standard$GIT_COMMITTER_*variables for consistency (Phillip Wood).
- Fixed typo "reversed" -> "reverse" to match original test (Phillip Wood).
- Added "done" command and usedgit fast-import --doneto prevent errors in the loop being ignored by the pipe (Phillip Wood).
Thanks. Will replace.