unpack-objects: unpack large object in stream
Re: [RFC PATCH] object-file.c: batched disk flushes for stream_loose_object()
In reply to Han XinHi,
Quoted 7 lines · Han Xin
Neeraj Singh[1] pointed out that if batch fsync is enabled, we should still call
prepare_loose_object_bulk_checkin()to potentially create the bulk checkin objdir.1. https://lore.kernel.org/git/[email protected]/
Signed-off-by Han Xin <[email protected]>
I like a good commit message that is concise and yet has all the necessary information. Well done!
Quoted 10 lines
--- object-file.c | 3 +++ t/t5351-unpack-large-objects.sh | 15 ++++++++++++++- 2 files changed, 17 insertions(+), 1 deletion(-)object-file.c +0
-0
object-file.c int stream_loose_object(struct input_stream *in_stream, size_t len, · line 2136
Quoted 6 lines
object-file.c +3
-0@@ -2131,6 +2131,9 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,char hdr[MAX_HEADER_LEN ];int hdrlen; +if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) +prepare_loose_object_bulk_checkin (); +
Makes sense.
Quoted 8 lines
object-file.c +0
-0/* Since oid is not determined, save tmp file to odb path. */ strbuf_addf (&filename,"%s/" ,get_object_directory ()); hdrlen =format_object_header (hdr,sizeof (hdr),OBJ_BLOB , len);t/t5351-unpack-large-objects.sh +0
-0
t/t5351-unpack-large-objects.sh test_expect_success "create large objects (1.5 MB) and PACK" ' · line 24
Quoted 8 lines
t/t5351-unpack-large-objects.sh +4
-1@@ -18,7 +18,10 @@ test_expect_success "create large objects (1.5 MB) and PACK" ' test_commit --append foo big-blob && test-tool genrandom bar 1500000 >big-blob && test_commit --append bar big-blob && - PACK=$(echo HEAD | git pack-objects --revs pack) + PACK=$(echo HEAD | git pack-objects --revs pack) && + git verify-pack -v pack-$PACK.pack | + grep -E "commit|tree|blob" | + sed -n -e "s/^\([0-9a-f]*\).*/\1/p" >obj-list
Here, I would recommend avoiding the pipe, to ensure that we would catch problems in the verify-pack invocation, and I think we can avoid the grep altogether:
git verify-pack -v pack-$PACK.pack >out && sed -n 's/^\([0-9a-f][0-9a-f]*\).*\(commit\|tree\|blob\)/\1/p' \ <out >obj-list
t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 55
Quoted 12 lines
t/t5351-unpack-large-objects.sh +5
-0' test_expect_success 'set memory limitation to 1MB' '@@ -45,6 +48,16 @@ test_expect_success 'unpack big object in stream' ' test_dir_is_empty dest.git/objects/pack ' +BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch' + +test_expect_success 'unpack big object in stream (core.fsyncmethod=batch)' ' + prepare_dest 1m && + git $BATCH_CONFIGURATION -C dest.git unpack-objects <pack-$PACK.pack &&
I think the canonical way would be to use test_config core.fsync ..., but the presented way works, too.
t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 57
Quoted 2 lines
t/t5351-unpack-large-objects.sh +2
-0+ test_dir_is_empty dest.git/objects/pack && + git -C dest.git cat-file --batch-check="%(objectname)" <obj-list >current &&
Good. The --batch-check="%(objectname)" part forces cat-file to read the actual object.
t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 59
Quoted 2 lines
t/t5351-unpack-large-objects.sh +2
-0+ cmp obj-list current +'
My main question about this test case is whether it _actually_ verifies that the batch-mode fsync()ing took place.
I kind of had expected to see Trace2 enabled and a grep for fsync/hardware-flush. Do you think that would still make sense to add?
Thank you for working on the fsync() aspects of Git! Dscho
Quoted 8 lines
t/t5351-unpack-large-objects.sh +1
-1+ test_expect_success 'do not unpack existing large objects' ' prepare_dest 1m && git -C dest.git index-pack --stdin <pack-$PACK.pack && --2.36.1
Re: [RFC PATCH] object-file.c: batched disk flushes for stream_loose_object()
Quoted 56 lines · Johannes Schindelin
Hi,
Quoted 7 lines · Han Xin
Neeraj Singh[1] pointed out that if batch fsync is enabled, we should still call
prepare_loose_object_bulk_checkin()to potentially create the bulk checkin objdir.1. https://lore.kernel.org/git/[email protected]/
Signed-off-by Han Xin <[email protected]>I like a good commit message that is concise and yet has all the necessary information. Well done!
Quoted 10 lines
--- object-file.c | 3 +++ t/t5351-unpack-large-objects.sh | 15 ++++++++++++++- 2 files changed, 17 insertions(+), 1 deletion(-)object-file.c +0
-0object-file.c int stream_loose_object(struct input_stream *in_stream, size_t len, · line 2136
Quoted 6 lines
object-file.c +3
-0@@ -2131,6 +2131,9 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,char hdr[MAX_HEADER_LEN ];int hdrlen; +if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) +prepare_loose_object_bulk_checkin (); +Makes sense.
Quoted 8 lines
object-file.c +0
-0/* Since oid is not determined, save tmp file to odb path. */ strbuf_addf (&filename,"%s/" ,get_object_directory ()); hdrlen =format_object_header (hdr,sizeof (hdr),OBJ_BLOB , len);t/t5351-unpack-large-objects.sh +0
-0t/t5351-unpack-large-objects.sh test_expect_success "create large objects (1.5 MB) and PACK" ' · line 24
Quoted 8 lines
t/t5351-unpack-large-objects.sh +4
-1@@ -18,7 +18,10 @@ test_expect_success "create large objects (1.5 MB) and PACK" ' test_commit --append foo big-blob && test-tool genrandom bar 1500000 >big-blob && test_commit --append bar big-blob && - PACK=$(echo HEAD | git pack-objects --revs pack) + PACK=$(echo HEAD | git pack-objects --revs pack) && + git verify-pack -v pack-$PACK.pack | + grep -E "commit|tree|blob" | + sed -n -e "s/^\([0-9a-f]*\).*/\1/p" >obj-listHere, I would recommend avoiding the pipe, to ensure that we would catch problems in the
verify-packinvocation, and I think we can avoid thegrepaltogether:
Quoted 4 lines
git verify-pack -v pack-$PACK.pack >out &&
sed -n 's/^\([0-9a-f][0-9a-f]*\).*\(commit\|tree\|blob\)/\1/p' \
<out >obj-listGood suggestion. I will take it.
Thanks.
-Han Xin
Quoted 25 lines
t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 55
Quoted 12 lines
t/t5351-unpack-large-objects.sh +5
-0' test_expect_success 'set memory limitation to 1MB' '@@ -45,6 +48,16 @@ test_expect_success 'unpack big object in stream' ' test_dir_is_empty dest.git/objects/pack ' +BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch' + +test_expect_success 'unpack big object in stream (core.fsyncmethod=batch)' ' + prepare_dest 1m && + git $BATCH_CONFIGURATION -C dest.git unpack-objects <pack-$PACK.pack &&I think the canonical way would be to use
test_config core.fsync ..., but the presented way works, too.t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 57
Quoted 2 lines
t/t5351-unpack-large-objects.sh +2
-0+ test_dir_is_empty dest.git/objects/pack && + git -C dest.git cat-file --batch-check="%(objectname)" <obj-list >current &&Good. The
--batch-check="%(objectname)"part forcescat-fileto read the actual object.t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 59
Quoted 2 lines
t/t5351-unpack-large-objects.sh +2
-0+ cmp obj-list current +'
Quoted 9 lines
My main question about this test case is whether it _actually_ verifies that the batch-mode
fsync()ing took place.I kind of had expected to see Trace2 enabled and a
grepforfsync/hardware-flush. Do you think that would still make sense to add?Thank you for working on the
fsync()aspects of Git! Dscho
More rigorous inspection should be adopted.
Thanks.
-Han Xin
Quoted 8 lines
Quoted 8 lines
t/t5351-unpack-large-objects.sh +1
-1+ test_expect_success 'do not unpack existing large objects' ' prepare_dest 1m && git -C dest.git index-pack --stdin <pack-$PACK.pack && --2.36.1
PATCH v13 6/7 core doc: modernize core.bigFileThreshold documentation
In reply to Ævar Arnfjörð BjarmasonThe core.bigFileThreshold documentation has been largely unchanged since 5eef828bc03 (fast-import: Stream very large blobs directly to pack, 2010-02-01).
But since then this setting has been expanded to affect a lot more than that description indicated. Most notably in how "git diff" treats them, see 6bf3b813486 (diff --stat: mark any file larger than core.bigfilethreshold binary, 2014-08-16).
In addition to that, numerous commands and APIs make use of a streaming mode for files above this threshold.
So let's attempt to summarize 12 years of changes in behavior, which can be seen with:
git log --oneline -Gbig_file_thre 5eef828bc03.. -- '*.c'
To do that turn this into a bullet-point list. The summary Han Xin produced in [1] helped a lot, but is a bit too detailed for documentation aimed at users. Let's instead summarize how user-observable behavior differs, and generally describe how we tend to stream these files in various commands.
1. https://lore.kernel.org/git/[email protected]/
--- Documentation/config/core.txt | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-)
Documentation/config/core.txt +24 -9
@@ -444,17 +444,32 @@ You probably do not need to adjust this value. Common unit suffixes of 'k', 'm', or 'g' are supported. core.bigFileThreshold:: - Files larger than this size are stored deflated, without - attempting delta compression. Storing large files without - delta compression avoids excessive memory usage, at the - slight expense of increased disk usage. Additionally files - larger than this size are always treated as binary. + The size of files considered "big", which as discussed below + changes the behavior of numerous git commands, as well as how + such files are stored within the repository. The default is + 512 MiB. Common unit suffixes of 'k', 'm', or 'g' are + supported. + -Default is 512 MiB on all platforms. This should be reasonable -for most projects as source code and other text files can still -be delta compressed, but larger binary media files won't be. +Files above the configured limit will be: + -Common unit suffixes of 'k', 'm', or 'g' are supported. +* Stored deflated, without attempting delta compression. ++ +The default limit is primarily set with this use-case in mind. With it +most projects will have their source code and other text files delta +compressed, but not larger binary media files. ++ +Storing large files without delta compression avoids excessive memory +usage, at the slight expense of increased disk usage. ++ +* Will be treated as if though they were labeled "binary" (see + linkgit:gitattributes[5]). This means that e.g. linkgit:git-log[1] + and linkgit:git-diff[1] will not diffs for files above this limit. ++ +* Will be generally be streamed when written, which avoids excessive +memory usage, at the cost of some fixed overhead. Commands that make +use of this include linkgit:git-archive[1], +linkgit:git-fast-import[1], linkgit:git-index-pack[1] and +linkgit:git-fsck[1]. core.excludesFile:: Specifies the pathname to the file that contains patterns to
Signature · 1 line
2.36.1.1124.g52838f02905
Re: [PATCH v13 6/7] core doc: modernize core.bigFileThreshold documentation
Quoted 10 lines · Ævar Arnfjörð Bjarmason
So let's attempt to summarize 12 years of changes in behavior, which can be seen with:
git log --oneline -Gbig_file_thre 5eef828bc03.. -- '*.c'To do that turn this into a bullet-point list. The summary Han Xin produced in [1] helped a lot, but is a bit too detailed for documentation aimed at users. Let's instead summarize how user-observable behavior differs, and generally describe how we tend to stream these files in various commands.
Nicely studied. Very much appreciated.
Quoted 9 lines
core.bigFileThreshold::
Documentation/config/core.txt +3
-5- Files larger than this size are stored deflated, without - attempting delta compression. Storing large files without - delta compression avoids excessive memory usage, at the - slight expense of increased disk usage. Additionally files - larger than this size are always treated as binary. + The size of files considered "big", which as discussed below + changes the behavior of numerous git commands, as well as how + such files are stored within the repository. The default is
Documentation/config/core.txt You probably do not need to adjust this value. · line 455
Quoted 10 lines
Documentation/config/core.txt +4
-4+ 512 MiB. Common unit suffixes of 'k', 'm', or 'g' are + supported. + -Default is 512 MiB on all platforms. This should be reasonable -for most projects as source code and other text files can still -be delta compressed, but larger binary media files won't be. +Files above the configured limit will be: + -Common unit suffixes of 'k', 'm', or 'g' are supported. +* Stored deflated, without attempting delta compression.
"even in packfiles" (with or without "even") is better be there in the sentence---loose objects are always stored deflated anyway.
Documentation/config/core.txt You probably do not need to adjust this value.
Quoted 10 lines
Documentation/config/core.txt +9
-0+The default limit is primarily set with this use-case in mind. With it +most projects will have their source code and other text files delta +compressed, but not larger binary media files. ++ +Storing large files without delta compression avoids excessive memory +usage, at the slight expense of increased disk usage. +* Will be treated as if though they were labeled "binary" (see + linkgit:gitattributes[5]). This means that e.g. linkgit:git-log[1] + and linkgit:git-diff[1] will not diffs for files above this limit.
Good. You can lose three words "This means that" and the sentence means the same thing, so lose them (I always recommend people to reread the sentence when they write "This means that" with an eye to rewrite it better---it often is a sign that either the previous sentence is insufficiently clear, in which case it can be discarded and description after the three words can be enhanced to a better result).
Documentation/config/core.txt You probably do not need to adjust this value. · line 472
Quoted 5 lines
Documentation/config/core.txt +5
-0+* Will be generally be streamed when written, which avoids excessive +memory usage, at the cost of some fixed overhead. Commands that make +use of this include linkgit:git-archive[1], +linkgit:git-fast-import[1], linkgit:git-index-pack[1] and +linkgit:git-fsck[1].
Nice. And this series adds unpack-objects to the mix.
Quoted 2 lines
core.excludesFile::
Specifies the pathname to the file that contains patterns to
Excellent.
Thanks.
PATCH v13 7/7 unpack-objects: use stream_loose_object() to unpack large objects
In reply to Ævar Arnfjörð BjarmasonFrom: Han Xin <[email protected]>
Make use of the stream_loose_object() function introduced in the preceding commit to unpack large objects. Before this we'd need to malloc() the size of the blob before unpacking it, which could cause OOM with very large blobs.
We could use the new streaming interface to unpack all blobs, but doing so would be much slower, as demonstrated e.g. with this benchmark using git-hyperfine[0]:
rm -rf /tmp/scalar.git && git clone --bare https://github.com/Microsoft/scalar.git /tmp/scalar.git && mv /tmp/scalar.git/objects/pack/*.pack /tmp/scalar.git/my.pack && git hyperfine \ -r 2 --warmup 1 \ -L rev origin/master,HEAD -L v "10,512,1k,1m" \ -s 'make' \ -p 'git init --bare dest.git' \ -c 'rm -rf dest.git' \ './git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/scalar.git/my.pack'
Here we'll perform worse with lower core.bigFileThreshold settings with this change in terms of speed, but we're getting lower memory use in return:
Summary
'./git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' ran
1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.02 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.02 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.09 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.10 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.11 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD'
A better benchmark to demonstrate the benefits of that this one, which creates an artificial repo with a 1, 25, 50, 75 and 100MB blob:
rm -rf /tmp/repo &&
git init /tmp/repo &&
(
cd /tmp/repo &&
for i in 1 25 50 75 100
do
dd if=/dev/urandom of=blob.$i count=$(($i*1024)) bs=1024
done &&
git add blob.* &&
git commit -mblobs &&
git gc &&
PACK=$(echo .git/objects/pack/pack-*.pack) &&
cp "$PACK" my.pack
) &&
git hyperfine \
--show-output \
-L rev origin/master,HEAD -L v "512,50m,100m" \
-s 'make' \
-p 'git init --bare dest.git' \
-c 'rm -rf dest.git' \
'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum'Using this test we'll always use >100MB of memory on origin/master (around ~105MB), but max out at e.g. ~55MB if we set core.bigFileThreshold=50m.
The relevant "Maximum resident set size" lines were manually added below the relevant benchmark:
'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master' ran
Maximum resident set size (kbytes): 107080
1.02 ± 0.78 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 106968
1.09 ± 0.79 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 107032
1.42 ± 1.07 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 107072
1.83 ± 1.02 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 55704
2.16 ± 1.19 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 4564
This shows that if you have enough memory this new streaming method is slower the lower you set the streaming threshold, but the benefit is more bounded memory use.
An earlier version of this patch introduced a new "core.bigFileStreamingThreshold" instead of re-using the existing "core.bigFileThreshold" variable[1]. As noted in a detailed overview of its users in [2] using it has several different meanings.
Still, we consider it good enough to simply re-use it. While it's possible that someone might want to e.g. consider objects "small" for the purposes of diffing but "big" for the purposes of writing them such use-cases are probably too obscure to worry about. We can always split up "core.bigFileThreshold" in the future if there's a need for that.
0. https://github.com/avar/git-hyperfine/
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
--- Documentation/config/core.txt | 4 +- builtin/unpack-objects.c | 69 ++++++++++++++++++++++++++++++++- t/t5351-unpack-large-objects.sh | 26 +++++++++++-- 3 files changed, 93 insertions(+), 6 deletions(-)
Documentation/config/core.txt +2 -2
@@ -468,8 +468,8 @@ usage, at the slight expense of increased disk usage. * Will be generally be streamed when written, which avoids excessive memory usage, at the cost of some fixed overhead. Commands that make use of this include linkgit:git-archive[1], -linkgit:git-fast-import[1], linkgit:git-index-pack[1] and -linkgit:git-fsck[1]. +linkgit:git-fast-import[1], linkgit:git-index-pack[1], +linkgit:git-unpack-objects[1] and linkgit:git-fsck[1]. core.excludesFile:: Specifies the pathname to the file that contains patterns to
builtin/unpack-objects.c +68 -1
83 lines not on this page · raw message
t/t5351-unpack-large-objects.sh +23 -3
@@ -9,7 +9,8 @@ test_description='git unpack-objects with large objects' prepare_dest () { test_when_finished "rm -rf dest.git" && - git init --bare dest.git + git init --bare dest.git && + git -C dest.git config core.bigFileThreshold "$1" } test_expect_success "create large objects (1.5 MB) and PACK" '
@@ -26,16 +27,35 @@ test_expect_success 'set memory limitation to 1MB' ' ' test_expect_success 'unpack-objects failed under memory limitation' ' - prepare_dest && + prepare_dest 2m && test_must_fail git -C dest.git unpack-objects <pack-$PACK.pack 2>err && grep "fatal: attempting to allocate" err ' test_expect_success 'unpack-objects works with memory limitation in dry-run mode' ' - prepare_dest && + prepare_dest 2m && git -C dest.git unpack-objects -n <pack-$PACK.pack && test_stdout_line_count = 0 find dest.git/objects -type f && test_dir_is_empty dest.git/objects/pack ' +test_expect_success 'unpack big object in stream' ' + prepare_dest 1m && + git -C dest.git unpack-objects <pack-$PACK.pack && + test_dir_is_empty dest.git/objects/pack +' + +test_expect_success 'do not unpack existing large objects' ' + prepare_dest 1m && + git -C dest.git index-pack --stdin <pack-$PACK.pack && + git -C dest.git unpack-objects <pack-$PACK.pack && + + # The destination came up with the exact same pack... + DEST_PACK=$(echo dest.git/objects/pack/pack-*.pack) && + test_cmp pack-$PACK.pack $DEST_PACK && + + # ...and wrote no loose objects + test_stdout_line_count = 0 find dest.git/objects -type f ! -name "pack-*" +' + test_done
Signature · 1 line
2.36.1.1124.g52838f02905
v14 cover unpack-objects: support streaming blobs to disk
In reply to Ævar Arnfjörð BjarmasonThis series makes "unpack-objects" capable of streaming large objects to disk.
As 7/7 shows streaming e.g. a 100MB blob now uses ~5MB of memory instead of ~105MB. This streaming method is slower if you've got memory to handle the blobs in-core, but if you don't it allows you to unpack objects at all, as you might otherwise OOM.
Changes since v13:
* Make the error checking in the loop of get_data() the same way as we do in the non dry-run mode.
* Add batched disk flushes for stream_loose_object(). This is pointed out by Neeraj Singh[1].
* Minor typo/grammar/comment etc. fixes throughout.
1. https://lore.kernel.org/git/[email protected]/
Han Xin (4):
unpack-objects: low memory footprint for get_data() in dry_run mode object-file.c: refactor write_loose_object() to several steps object-file.c: add "stream_loose_object()" to handle large object
unpack-objects: use stream_loose_object() to unpack large objects
Ævar Arnfjörð Bjarmason (3):
object-file.c: do fsync() and close() before post-write die() object-file.c: factor out deflate part of write_loose_object() core doc: modernize core.bigFileThreshold documentation
Documentation/config/core.txt | 33 +++-- builtin/unpack-objects.c | 106 ++++++++++++-- object-file.c | 240 +++++++++++++++++++++++++++----- object-store.h | 8 ++ t/t5351-unpack-large-objects.sh | 76 ++++++++++ 5 files changed, 408 insertions(+), 55 deletions(-) create mode 100755 t/t5351-unpack-large-objects.sh
Range-diff against v13:
1: 6703df6350 ! 1: bf600a2fa8 unpack-objects: low memory footprint for get_data() in dry_run mode
@@ Commit message
Because in dry_run mode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can
- allocate a smaller buffer and reuse it as zstream output. Therefore,
- in dry_run mode, "get_data()" will release the allocated buffer and
- return NULL instead of returning garbage data.
+ allocate a smaller buffer and use it as zstream output. Make the function
+ return NULL in the dry-run mode, as no callers use the returned buffer.
The "find [...]objects/?? -type f | wc -l" test idiom being used here is adapted from the same "find" use added to another test in
@@ builtin/unpack-objects.c: static void use(int bytes)
}
+/*
-+ * Decompress zstream from stdin and return specific size of data. ++ * Decompress zstream from the standard input into a newly ++ * allocated buffer of specified size and return the buffer.
+ * The caller is responsible to free the returned buffer.
+ *
+ * But for dry_run mode, "get_data()" is only used to check the
@@ builtin/unpack-objects.c: static void *get_data(unsigned long size)
+ if (dry_run) {
+ /* reuse the buffer in dry_run mode */
+ stream.next_out = buf;
-+ stream.avail_out = bufsize;
++ stream.avail_out = bufsize > size - stream.total_out ?
++ size - stream.total_out :
++ bufsize;
+ }
}
git_inflate_end(&stream);
2: 6e289d25c1 = 2: a327f484f7 object-file.c: do fsync() and close() before post-write die()
3: 46f9def06c ! 3: 9bc8002282 object-file.c: refactor write_loose_object() to several steps
@@ object-file.c: static int create_tmpfile(struct strbuf *tmp, const char *filenam
+ *
+ * - End the compression of zlib stream.
+ * - Get the calculated oid to "oid".
-+ * - fsync() and close() the "fd"
+ */
+static int end_loose_object_common(git_hash_ctx *c, git_zstream *stream,
+ struct object_id *oid)
4: 5a95ebede6 = 4: 7c73815f18 object-file.c: factor out deflate part of write_loose_object()
5: 26847541aa ! 5: 28a9588f9c object-file.c: add "stream_loose_object()" to handle large object @@ Commit message
path.
"freshen_packed_object()" or "freshen_loose_object()" will be called
- inside "stream_loose_object()" after obtaining the "oid".
+ inside "stream_loose_object()" after obtaining the "oid". After the
+ temporary file is written, we wants to mark the object to recent and we
+ may find that where indeed is already the object. We should remove the
+ temporary and do not leave a new copy of the object.
@@ object-file.c: static int freshen_packed_object(const struct object_id *oid)
+ char hdr[MAX_HEADER_LEN];
+ int hdrlen;
+
++ if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
++ prepare_loose_object_bulk_checkin();
++
+ /* Since oid is not determined, save tmp file to odb path. */
+ strbuf_addf(&filename, "%s/", get_object_directory());
+ hdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, len);
@@ object-file.c: static int freshen_packed_object(const struct object_id *oid)
+ die(_("write stream object %ld != %"PRIuMAX), stream.total_in,
+ (uintmax_t)len + hdrlen);
+
-+ /* Common steps for write_loose_object and stream_loose_object to
++ /*
++ * Common steps for write_loose_object and stream_loose_object to
+ * end writing loose oject:
+ *
+ * - End the compression of zlib stream.
6: eb962b60b9 ! 6: dea5c4172b core doc: modernize core.bigFileThreshold documentation
@@ Documentation/config/core.txt: You probably do not need to adjust this value.
+Files above the configured limit will be:
+
-Common unit suffixes of 'k', 'm', or 'g' are supported.
-+* Stored deflated, without attempting delta compression.
++* Stored deflated in packfiles, without attempting delta compression.
++
+The default limit is primarily set with this use-case in mind. With it +most projects will have their source code and other text files delta
@@ Documentation/config/core.txt: You probably do not need to adjust this value.
+usage, at the slight expense of increased disk usage.
++
+* Will be treated as if though they were labeled "binary" (see
-+ linkgit:gitattributes[5]). This means that e.g. linkgit:git-log[1]
-+ and linkgit:git-diff[1] will not diffs for files above this limit.
++ linkgit:gitattributes[5]). e.g. linkgit:git-log[1] and
++ linkgit:git-diff[1] will not diffs for files above this limit.
++
+* Will be generally be streamed when written, which avoids excessive +memory usage, at the cost of some fixed overhead. Commands that make
7: 88a2754fcb ! 7: d236230a4c unpack-objects: use stream_loose_object() to unpack large objects
@@ t/t5351-unpack-large-objects.sh: test_description='git unpack-objects with large
}
test_expect_success "create large objects (1.5 MB) and PACK" '
+@@ t/t5351-unpack-large-objects.sh: test_expect_success "create large objects (1.5 MB) and PACK" '
+ test_commit --append foo big-blob &&
+ test-tool genrandom bar 1500000 >big-blob &&
+ test_commit --append bar big-blob &&
+- PACK=$(echo HEAD | git pack-objects --revs pack)
++ PACK=$(echo HEAD | git pack-objects --revs pack) &&
++ git verify-pack -v pack-$PACK.pack >out &&
++ sed -n -e "s/^\([0-9a-f][0-9a-f]*\).*\(commit\|tree\|blob\).*/\1/p" \
++ <out >obj-list
+ '
+
+ test_expect_success 'set memory limitation to 1MB' '
@@ t/t5351-unpack-large-objects.sh: test_expect_success 'set memory limitation to 1MB' '
'
@@ t/t5351-unpack-large-objects.sh: test_expect_success 'set memory limitation to 1
+ test_dir_is_empty dest.git/objects/pack
+'
+
++BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch'
++
++test_expect_success 'unpack big object in stream (core.fsyncmethod=batch)' '
++ prepare_dest 1m &&
++ GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \
++ git -C dest.git $BATCH_CONFIGURATION unpack-objects <pack-$PACK.pack &&
++ grep fsync/hardware-flush trace2.txt &&
++ test_dir_is_empty dest.git/objects/pack &&
++ git -C dest.git cat-file --batch-check="%(objectname)" <obj-list >current &&
++ cmp obj-list current
++'
++
+test_expect_success 'do not unpack existing large objects' '
+ prepare_dest 1m &&
+ git -C dest.git index-pack --stdin <pack-$PACK.pack &&Signature · 1 line
2.36.1
PATCH v14 1/7 unpack-objects: low memory footprint for get_data() in dry_run mode
From: Han Xin <[email protected]>
As the name implies, "get_data(size)" will allocate and return a given amount of memory. Allocating memory for a large blob object may cause the system to run out of memory. Before preparing to replace calling of "get_data()" to unpack large blob objects in latter commits, refactor "get_data()" to reduce memory footprint for dry_run mode.
Because in dry_run mode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can allocate a smaller buffer and use it as zstream output. Make the function return NULL in the dry-run mode, as no callers use the returned buffer.
The "find [...]objects/?? -type f | wc -l" test idiom being used here is adapted from the same "find" use added to another test in d9545c7f465 (fast-import: implement unpack limit, 2016-04-25).
--- builtin/unpack-objects.c | 37 ++++++++++++++++++++--------- t/t5351-unpack-large-objects.sh | 41 +++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 11 deletions(-) create mode 100755 t/t5351-unpack-large-objects.sh
builtin/unpack-objects.c +26 -11
@@ -97,15 +97,27 @@ static void use(int bytes)display_throughput (progress, consumed_bytes); } +/* +* Decompress zstream from the standard input into a newly +* allocated buffer of specified size and return the buffer. +* The caller is responsible to free the returned buffer. +* +* But for dry_run mode, "get_data()" is only used to check the +* integrity of data, and the returned buffer is not used at all. +* Therefore, in dry_run mode, "get_data()" will release the small +* allocated buffer which is reused to hold temporary zstream output +* and return NULL instead of returning garbage data. +*/ static void *get_data (unsigned long size) {git_zstream stream; -void *buf =xmallocz (size); +unsigned long bufsize = dry_run && size >8192 ?8192 : size; +void *buf =xmallocz (bufsize);memset (&stream,0 ,sizeof (stream)); stream.next_out = buf; - stream.avail_out = size; + stream.avail_out = bufsize; stream.next_in =fill (1 ); stream.avail_in = len;git_inflate_init (&stream);
@@ -125,8 +137,17 @@ static void *get_data(unsigned long size) } stream.next_in =fill (1 ); stream.avail_in = len; +if (dry_run) { +/* reuse the buffer in dry_run mode */ + stream.next_out = buf; + stream.avail_out = bufsize > size - stream.total_out ? + size - stream.total_out : + bufsize; + } }git_inflate_end (&stream); +if (dry_run) +FREE_AND_NULL (buf);return buf; }
@@ -326,10 +347,8 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size, {void *buf =get_data (size); -if (!dry_run && buf) +if (buf)write_object (nr, type, buf, size); -else -free (buf); } static int resolve_against_held(unsigned nr, const struct object_id *base,
@@ -359,10 +378,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,oidread (&base_oid,fill (the_hash_algo->rawsz));use (the_hash_algo->rawsz); delta_data =get_data (delta_size); -if (dry_run || !delta_data) { -free (delta_data); +if (!delta_data)return ; - }if (has_object_file (&base_oid)) ;/* Ok we have this one */ else if (resolve_against_held(nr, &base_oid,
@@ -398,10 +415,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,die ("offset value out of bound for delta base object" ); delta_data =get_data (delta_size); -if (dry_run || !delta_data) { -free (delta_data); +if (!delta_data)return ; - } lo =0 ; hi = nr;while (lo < hi) {
t/t5351-unpack-large-objects.sh new file +41 -0
41 lines not on this page · raw message
Signature · 1 line
2.36.1
PATCH v14 2/7 object-file.c: do fsync() and close() before post-write die()
In reply to Han XinFrom: Ævar Arnfjörð Bjarmason <[email protected]>
Change write_loose_object() to do an fsync() and close() before the oideq() sanity check at the end. This change re-joins code that was split up by the die() sanity check added in 748af44c63e (sha1_file: be paranoid when creating loose objects, 2010-02-21).
I don't think that this change matters in itself, if we called die() it was possible that our data wouldn't fully make it to disk, but in any case we were writing data that we'd consider corrupted. It's possible that a subsequent "git fsck" will be less confused now.
The real reason to make this change is that in a subsequent commit we'll split this code in write_loose_object() into a utility function, all its callers will want the preceding sanity checks, but not the "oideq" check. By moving the close_loose_object() earlier it'll be easier to reason about the introduction of the utility function.
--- object-file.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
object-file.c +2 -2
@@ -2012,12 +2012,12 @@ static int write_loose_object(const struct object_id *oid, char *hdr,die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), ret); the_hash_algo->final_oid_fn (¶no_oid, &c); +close_loose_object (fd, tmp_file.buf); +if (!oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid)); -close_loose_object (fd, tmp_file.buf); -if (mtime) {struct utimbuf utb; utb.actime = mtime;
Signature · 1 line
2.36.1
Re: [PATCH v14 2/7] object-file.c: do fsync() and close() before post-write die()
Am 10.06.22 um 16:46 schrieb Han Xin:
Quoted 11 lines
From: Ævar Arnfjörð Bjarmason <[email protected]>
Change
write_loose_object()to do anfsync()andclose()before theoideq()sanity check at the end. This change re-joins code that was split up by thedie()sanity check added in748af44c63e(sha1_file: be paranoid when creating loose objects, 2010-02-21).I don't think that this change matters in itself, if we called
die()it was possible that our data wouldn't fully make it to disk, but in any case we were writing data that we'd consider corrupted. It's possible that a subsequent "git fsck" will be less confused now.
This is done before renaming the file, so git fsck is going to see (at most) a tmp_obj_?????? file, which it ignores in either case, right?
Quoted 5 lines
The real reason to make this change is that in a subsequent commit we'll split this code in
write_loose_object()into a utility function, all its callers will want the preceding sanity checks, but not the "oideq" check. By moving theclose_loose_object()earlier it'll be easier to reason about the introduction of the utility function.
This sounds like a patch would move the close_loose_object() call to some other place, but that's not the case. The sequence below (starting from the close_loose_object() call) is still present after applying the whole series, so it seems this patch is not necessary.
Quoted 25 lines
Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- object-file.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)object-file.c +2
-2@@ -2012,12 +2012,12 @@ static int write_loose_object(const struct object_id *oid, char *hdr,die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), ret); the_hash_algo->final_oid_fn (¶no_oid, &c); +close_loose_object (fd, tmp_file.buf); +if (!oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid)); -close_loose_object (fd, tmp_file.buf); -if (mtime) {struct utimbuf utb; utb.actime = mtime;
Re: [PATCH v14 2/7] object-file.c: do fsync() and close() before post-write die()
Quoted 8 lines · René Scharfe
Am 10.06.22 um 16:46 schrieb Han Xin:
Quoted 7 lines
From: Ævar Arnfjörð Bjarmason <[email protected]>
Change
write_loose_object()to do anfsync()andclose()before theoideq()sanity check at the end. This change re-joins code that was split up by thedie()sanity check added in748af44c63e(sha1_file: be paranoid when creating loose objects, 2010-02-21).
Quoted 7 lines
Quoted 4 lines
I don't think that this change matters in itself, if we called
die()it was possible that our data wouldn't fully make it to disk, but in any case we were writing data that we'd consider corrupted. It's possible that a subsequent "git fsck" will be less confused now.This is done before renaming the file, so git fsck is going to see (at most) a
tmp_obj_?????? file, which it ignores in either case, right?
Yes, I thought I pointed that out in my review on the previous round, but I missed that it was still here in this round X-<.
Thanks for noticing.
Re: [PATCH v14 2/7] object-file.c: do fsync() and close() before post-write die()
Quoted 16 lines · Junio C Hamano
Quoted 13 lines · René Scharfe
Am 10.06.22 um 16:46 schrieb Han Xin:
Quoted 11 lines
From: Ævar Arnfjörð Bjarmason <[email protected]>
Change
write_loose_object()to do anfsync()andclose()before theoideq()sanity check at the end. This change re-joins code that was split up by thedie()sanity check added in748af44c63e(sha1_file: be paranoid when creating loose objects, 2010-02-21).I don't think that this change matters in itself, if we called
die()it was possible that our data wouldn't fully make it to disk, but in any case we were writing data that we'd consider corrupted. It's possible that a subsequent "git fsck" will be less confused now.
Quoted 8 lines
Quoted 2 lines
This is done before renaming the file, so git fsck is going to see (at most) a
tmp_obj_?????? file, which it ignores in either case, right?Yes, I thought I pointed that out in my review on the previous round, but I missed that it was still here in this round X-<.
Thanks for noticing.
Yes, agree with both of you, I'll be removing this patch in the next series.
This patch was first introduced in v10[1], close_loose_object() was moved to end_loose_object_common(), but it was put back in v12[2]. It is indeed no longer necessary now.
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
Thank you both.
-Han Xin
-Han Xin
PATCH v14 3/7 object-file.c: refactor write_loose_object() to several steps
In reply to Han XinFrom: Han Xin <[email protected]>
When writing a large blob using "write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in later commit to resolve this issue.
Before introducing that streaming function, do some refactoring on "write_loose_object()" to reuse code for both versions.
Rewrite "write_loose_object()" as follows:
1. Figure out a path for the (temp) object file. This step is only used in "write_loose_object()".
2. Move common steps for starting to write loose objects into a new function "start_loose_object_common()".
3. Compress data.
4. Move common steps for ending zlib stream into a new function
"end_loose_object_common()".
5. Close fd and finalize the object file.
--- object-file.c | 101 +++++++++++++++++++++++++++++++++++++------------- 1 file changed, 75 insertions(+), 26 deletions(-)
object-file.c +75 -26
@@ -1951,6 +1951,74 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)return fd; } +/** +* Common steps for loose object writers to start writing loose +* objects: +* +* - Create tmpfile for the loose object. +* - Setup zlib stream for compression. +* - Start to feed header to zlib stream. +* +* Returns a "fd", which should later be provided to +* end_loose_object_common(). +*/ +static int start_loose_object_common (struct strbuf *tmp_file, +const char *filename,unsigned flags, +git_zstream *stream, +unsigned char *buf,size_t buflen, +git_hash_ctx *c, +char *hdr,int hdrlen) +{ +int fd; + + fd =create_tmpfile (tmp_file, filename); +if (fd <0 ) { +if (flags &HASH_SILENT ) +return -1 ; +else if (errno ==EACCES ) +return error (_ ("insufficient permission for adding " +"an object to repository database %s" ), +get_object_directory ()); +else +return error_errno ( +_ ("unable to create temporary file" )); + } + +/* Setup zlib stream for compression */ +git_deflate_init (stream, zlib_compression_level); + stream->next_out = buf; + stream->avail_out = buflen; + the_hash_algo->init_fn (c); + +/* Start to feed header to zlib stream */ + stream->next_in = (unsigned char *)hdr; + stream->avail_in = hdrlen; +while (git_deflate (stream,0 ) ==Z_OK ) + ;/* nothing */ + the_hash_algo->update_fn (c, hdr, hdrlen); + +return fd; +} + +/** +* Common steps for loose object writers to end writing loose objects: +* +* - End the compression of zlib stream. +* - Get the calculated oid to "oid". +*/ +static int end_loose_object_common (git_hash_ctx *c,git_zstream *stream, +struct object_id *oid) +{ +int ret; + + ret =git_deflate_end_gently (stream); +if (ret !=Z_OK ) +return ret; + the_hash_algo->final_oid_fn (oid, c); + +return Z_OK ; +} +static int write_loose_object (const struct object_id *oid,char *hdr,int hdrlen,const void *buf,unsigned long len,time_t mtime,unsigned flags)
@@ -1968,28 +2036,11 @@ static int write_loose_object(const struct object_id *oid, char *hdr,loose_object_path (the_repository, &filename, oid); - fd =create_tmpfile (&tmp_file, filename.buf); -if (fd <0 ) { -if (flags &HASH_SILENT ) -return -1 ; -else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); -else -return error_errno (_ ("unable to create temporary file" )); - } - -/* Set it up */ -git_deflate_init (&stream, zlib_compression_level); - stream.next_out = compressed; - stream.avail_out =sizeof (compressed); - the_hash_algo->init_fn (&c); - -/* First header.. */ - stream.next_in = (unsigned char *)hdr; - stream.avail_in = hdrlen; -while (git_deflate (&stream,0 ) ==Z_OK ) - ;/* nothing */ - the_hash_algo->update_fn (&c, hdr, hdrlen); + fd =start_loose_object_common (&tmp_file, filename.buf, flags, + &stream, compressed,sizeof (compressed), + &c, hdr, hdrlen); +if (fd <0 ) +return -1 ;/* Then the data itself.. */ stream.next_in = (void *)buf;
@@ -2007,11 +2058,9 @@ static int write_loose_object(const struct object_id *oid, char *hdr,if (ret !=Z_STREAM_END )die (_ ("unable to deflate new object %s (%d)" ),oid_to_hex (oid), ret); - ret =git_deflate_end_gently (&stream); + ret =end_loose_object_common (&c, &stream, ¶no_oid);if (ret !=Z_OK ) -die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), - ret); - the_hash_algo->final_oid_fn (¶no_oid, &c); +die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), ret);close_loose_object (fd, tmp_file.buf);if (!oideq (oid, ¶no_oid))
Signature · 1 line
2.36.1
PATCH v14 4/7 object-file.c: factor out deflate part of write_loose_object()
In reply to Han XinFrom: Ævar Arnfjörð Bjarmason <[email protected]>
Split out the part of write_loose_object() that deals with calling git_deflate() into a utility function, a subsequent commit will introduce another function that'll make use of it.
--- object-file.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-)
object-file.c +25 -6
@@ -2000,6 +2000,28 @@ static int start_loose_object_common(struct strbuf *tmp_file,return fd; } +/** +* Common steps for the inner git_deflate() loop for writing loose +* objects. Returns what git_deflate() returns. +*/ +static int write_loose_object_common (git_hash_ctx *c, +git_zstream *stream,const int flush, +unsigned char *in0,const int fd, +unsigned char *compressed, +const size_t compressed_len) +{ +int ret; + + ret =git_deflate (stream, flush ?Z_FINISH :0 ); + the_hash_algo->update_fn (c, in0, stream->next_in - in0); +if (write_buffer (fd, compressed, stream->next_out - compressed) <0 ) +die (_ ("unable to write loose object file" )); + stream->next_out = compressed; + stream->avail_out = compressed_len; + +return ret; +} +/** * Common steps for loose object writers to end writing loose objects: *
@@ -2047,12 +2069,9 @@ static int write_loose_object(const struct object_id *oid, char *hdr, stream.avail_in = len;do {unsigned char *in0 = stream.next_in; - ret =git_deflate (&stream,Z_FINISH ); - the_hash_algo->update_fn (&c, in0, stream.next_in - in0); -if (write_buffer (fd, compressed, stream.next_out - compressed) <0 ) -die (_ ("unable to write loose object file" )); - stream.next_out = compressed; - stream.avail_out =sizeof (compressed); + + ret =write_loose_object_common (&c, &stream,1 , in0, fd, + compressed,sizeof (compressed)); }while (ret ==Z_OK );if (ret !=Z_STREAM_END )
Signature · 1 line
2.36.1
PATCH v14 5/7 object-file.c: add "stream_loose_object()" to handle large object
In reply to Han XinFrom: Han Xin <[email protected]>
If we want unpack and write a loose object using "write_loose_object", we have to feed it with a buffer with the same size of the object, which will consume lots of memory and may cause OOM. This can be improved by feeding data to "stream_loose_object()" in a stream.
Add a new function "stream_loose_object()", which is a stream version of "write_loose_object()" but with a low memory footprint. We will use this function to unpack large blob object in later commit.
Another difference with "write_loose_object()" is that we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. In "write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object.
Still, we need to save the temporary file we're preparing somewhere. We'll do that in the top-level ".git/objects/" directory (or whatever "GIT_OBJECT_DIRECTORY" is set to). Once we've streamed it we'll know the OID, and will move it to its canonical path.
"freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid". After the temporary file is written, we wants to mark the object to recent and we may find that where indeed is already the object. We should remove the temporary and do not leave a new copy of the object.
--- object-file.c | 104 +++++++++++++++++++++++++++++++++++++++++++++++++ object-store.h | 8 ++++ 2 files changed, 112 insertions(+)
object-file.c +104 -0
@@ -2118,6 +2118,110 @@ static int freshen_packed_object(const struct object_id *oid)return 1 ; } +int stream_loose_object (struct input_stream *in_stream,size_t len, +struct object_id *oid) +{ +int fd, ret, err =0 , flush =0 ; +unsigned char compressed[4096 ]; +git_zstream stream; +git_hash_ctx c; +struct strbuf tmp_file =STRBUF_INIT ; +struct strbuf filename =STRBUF_INIT ; +int dirlen; +char hdr[MAX_HEADER_LEN ]; +int hdrlen; + +if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) +prepare_loose_object_bulk_checkin (); + +/* Since oid is not determined, save tmp file to odb path. */ +strbuf_addf (&filename,"%s/" ,get_object_directory ()); + hdrlen =format_object_header (hdr,sizeof (hdr),OBJ_BLOB , len); + +/* +* Common steps for write_loose_object and stream_loose_object to +* start writing loose objects: +* +* - Create tmpfile for the loose object. +* - Setup zlib stream for compression. +* - Start to feed header to zlib stream. +*/ + fd =start_loose_object_common (&tmp_file, filename.buf,0 , + &stream, compressed,sizeof (compressed), + &c, hdr, hdrlen); +if (fd <0 ) { + err =-1 ; + goto cleanup; + } + +/* Then the data itself.. */ +do { +unsigned char *in0 = stream.next_in; + +if (!stream.avail_in && !in_stream->is_finished) { +const void *in = in_stream->read (in_stream, &stream.avail_in); + stream.next_in = (void *)in; + in0 = (unsigned char *)in; +/* All data has been read. */ +if (in_stream->is_finished) + flush =1 ; + } + ret =write_loose_object_common (&c, &stream, flush, in0, fd, + compressed,sizeof (compressed)); +/* +* Unlike write_loose_object(), we do not have the entire +* buffer. If we get Z_BUF_ERROR due to too few input bytes, +* then we'll replenish them in the next input_stream->read() +* call when we loop. +*/ + }while (ret ==Z_OK || ret ==Z_BUF_ERROR ); + +if (stream.total_in != len + hdrlen) +die (_ ("write stream object %ld != %" PRIuMAX), stream.total_in, + (uintmax_t )len + hdrlen); + +/* +* Common steps for write_loose_object and stream_loose_object to +* end writing loose oject: +* +* - End the compression of zlib stream. +* - Get the calculated oid. +*/ +if (ret !=Z_STREAM_END ) +die (_ ("unable to stream deflate new object (%d)" ), ret); + ret =end_loose_object_common (&c, &stream, oid); +if (ret !=Z_OK ) +die (_ ("deflateEnd on stream object failed (%d)" ), ret); +close_loose_object (fd, tmp_file.buf); + +if (freshen_packed_object (oid) ||freshen_loose_object (oid)) { +unlink_or_warn (tmp_file.buf); + goto cleanup; + } + +loose_object_path (the_repository, &filename, oid); + +/* We finally know the object path, and create the missing dir. */ + dirlen =directory_size (filename.buf); +if (dirlen) { +struct strbuf dir =STRBUF_INIT ; +strbuf_add (&dir, filename.buf, dirlen); + +if (mkdir_in_gitdir (dir.buf) && errno !=EEXIST ) { + err =error_errno (_ ("unable to create directory %s" ), dir.buf); +strbuf_release (&dir); + goto cleanup; + } +strbuf_release (&dir); + } + + err =finalize_object_file (tmp_file.buf, filename.buf); +cleanup: +strbuf_release (&tmp_file); +strbuf_release (&filename); +return err; +} +int write_object_file_flags (const void *buf,unsigned long len,enum object_type type,struct object_id *oid,unsigned flags)
object-store.h +8 -0
@@ -46,6 +46,12 @@ struct object_directory {char *path; }; +struct input_stream { +const void *(*read)(struct input_stream *,unsigned long *len); +void *data; +int is_finished; +}; + KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq)
@@ -269,6 +275,8 @@ static inline int write_object_file(const void *buf, unsigned long len,int write_object_file_literally (const void *buf,unsigned long len,const char *type,struct object_id *oid,unsigned flags); +int stream_loose_object (struct input_stream *in_stream,size_t len, +struct object_id *oid);/* * Add an object file to the in-memory object store, without writing it
Signature · 1 line
2.36.1
PATCH v14 6/7 core doc: modernize core.bigFileThreshold documentation
In reply to Han XinFrom: Ævar Arnfjörð Bjarmason <[email protected]>
The core.bigFileThreshold documentation has been largely unchanged since 5eef828bc03 (fast-import: Stream very large blobs directly to pack, 2010-02-01).
But since then this setting has been expanded to affect a lot more than that description indicated. Most notably in how "git diff" treats them, see 6bf3b813486 (diff --stat: mark any file larger than core.bigfilethreshold binary, 2014-08-16).
In addition to that, numerous commands and APIs make use of a streaming mode for files above this threshold.
So let's attempt to summarize 12 years of changes in behavior, which can be seen with:
git log --oneline -Gbig_file_thre 5eef828bc03.. -- '*.c'
To do that turn this into a bullet-point list. The summary Han Xin produced in [1] helped a lot, but is a bit too detailed for documentation aimed at users. Let's instead summarize how user-observable behavior differs, and generally describe how we tend to stream these files in various commands.
1. https://lore.kernel.org/git/[email protected]/
--- Documentation/config/core.txt | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-)
Documentation/config/core.txt +24 -9
@@ -444,17 +444,32 @@ You probably do not need to adjust this value. Common unit suffixes of 'k', 'm', or 'g' are supported. core.bigFileThreshold:: - Files larger than this size are stored deflated, without - attempting delta compression. Storing large files without - delta compression avoids excessive memory usage, at the - slight expense of increased disk usage. Additionally files - larger than this size are always treated as binary. + The size of files considered "big", which as discussed below + changes the behavior of numerous git commands, as well as how + such files are stored within the repository. The default is + 512 MiB. Common unit suffixes of 'k', 'm', or 'g' are + supported. + -Default is 512 MiB on all platforms. This should be reasonable -for most projects as source code and other text files can still -be delta compressed, but larger binary media files won't be. +Files above the configured limit will be: + -Common unit suffixes of 'k', 'm', or 'g' are supported. +* Stored deflated in packfiles, without attempting delta compression. ++ +The default limit is primarily set with this use-case in mind. With it +most projects will have their source code and other text files delta +compressed, but not larger binary media files. ++ +Storing large files without delta compression avoids excessive memory +usage, at the slight expense of increased disk usage. ++ +* Will be treated as if though they were labeled "binary" (see + linkgit:gitattributes[5]). e.g. linkgit:git-log[1] and + linkgit:git-diff[1] will not diffs for files above this limit. ++ +* Will be generally be streamed when written, which avoids excessive +memory usage, at the cost of some fixed overhead. Commands that make +use of this include linkgit:git-archive[1], +linkgit:git-fast-import[1], linkgit:git-index-pack[1] and +linkgit:git-fsck[1]. core.excludesFile:: Specifies the pathname to the file that contains patterns to
Signature · 1 line
2.36.1
Re: [PATCH v14 6/7] core doc: modernize core.bigFileThreshold documentation
Documentation/config/core.txt You probably do not need to adjust this value. · line 457
Quoted 13 lines · Han Xin
core.bigFileThreshold::
Documentation/config/core.txt +9
-0+ The size of files considered "big", which as discussed below + changes the behavior of numerous git commands, as well as how + such files are stored within the repository. The default is + 512 MiB. Common unit suffixes of 'k', 'm', or 'g' are + supported. + +Files above the configured limit will be: + +* Stored deflated in packfiles, without attempting delta compression. ++ +The default limit is primarily set with this use-case in mind. With it
"With it" -> "With it,"
Documentation/config/core.txt You probably do not need to adjust this value. · line 462
Quoted 5 lines
Documentation/config/core.txt +5
-0+most projects will have their source code and other text files delta +compressed, but not larger binary media files. + +Storing large files without delta compression avoids excessive memory +usage, at the slight expense of increased disk usage.
Makes sense.
Quoted 1 line
+* Will be treated as if though they were labeled "binary" (see
"as if though" -> "as if"
Documentation/config/core.txt You probably do not need to adjust this value. · line 466
Quoted 2 lines
Documentation/config/core.txt +2
-0+ linkgit:gitattributes[5]). e.g. linkgit:git-log[1] and + linkgit:git-diff[1] will not diffs for files above this limit.
"will not diffs" -ECANNOTPARSE. "will not compute diffs", probably?
Quoted 2 lines
++
+* Will be generally be streamed when written, which avoids excessive
"be generally be" -> "generally be"
Quoted 7 lines
Documentation/config/core.txt +4
-0+memory usage, at the cost of some fixed overhead. Commands that make +use of this include linkgit:git-archive[1], +linkgit:git-fast-import[1], linkgit:git-index-pack[1] and +linkgit:git-fsck[1]. core.excludesFile:: Specifies the pathname to the file that contains patterns to
PATCH v14 7/7 unpack-objects: use stream_loose_object() to unpack large objects
In reply to Han XinFrom: Han Xin <[email protected]>
Make use of the stream_loose_object() function introduced in the preceding commit to unpack large objects. Before this we'd need to malloc() the size of the blob before unpacking it, which could cause OOM with very large blobs.
We could use the new streaming interface to unpack all blobs, but doing so would be much slower, as demonstrated e.g. with this benchmark using git-hyperfine[0]:
rm -rf /tmp/scalar.git && git clone --bare https://github.com/Microsoft/scalar.git /tmp/scalar.git && mv /tmp/scalar.git/objects/pack/*.pack /tmp/scalar.git/my.pack && git hyperfine \ -r 2 --warmup 1 \ -L rev origin/master,HEAD -L v "10,512,1k,1m" \ -s 'make' \ -p 'git init --bare dest.git' \ -c 'rm -rf dest.git' \ './git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/scalar.git/my.pack'
Here we'll perform worse with lower core.bigFileThreshold settings with this change in terms of speed, but we're getting lower memory use in return:
Summary
'./git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' ran
1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.02 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.02 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.09 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.10 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.11 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD'
A better benchmark to demonstrate the benefits of that this one, which creates an artificial repo with a 1, 25, 50, 75 and 100MB blob:
rm -rf /tmp/repo &&
git init /tmp/repo &&
(
cd /tmp/repo &&
for i in 1 25 50 75 100
do
dd if=/dev/urandom of=blob.$i count=$(($i*1024)) bs=1024
done &&
git add blob.* &&
git commit -mblobs &&
git gc &&
PACK=$(echo .git/objects/pack/pack-*.pack) &&
cp "$PACK" my.pack
) &&
git hyperfine \
--show-output \
-L rev origin/master,HEAD -L v "512,50m,100m" \
-s 'make' \
-p 'git init --bare dest.git' \
-c 'rm -rf dest.git' \
'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum'Using this test we'll always use >100MB of memory on origin/master (around ~105MB), but max out at e.g. ~55MB if we set core.bigFileThreshold=50m.
The relevant "Maximum resident set size" lines were manually added below the relevant benchmark:
'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master' ran
Maximum resident set size (kbytes): 107080
1.02 ± 0.78 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 106968
1.09 ± 0.79 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 107032
1.42 ± 1.07 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 107072
1.83 ± 1.02 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 55704
2.16 ± 1.19 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 4564
This shows that if you have enough memory this new streaming method is slower the lower you set the streaming threshold, but the benefit is more bounded memory use.
An earlier version of this patch introduced a new "core.bigFileStreamingThreshold" instead of re-using the existing "core.bigFileThreshold" variable[1]. As noted in a detailed overview of its users in [2] using it has several different meanings.
Still, we consider it good enough to simply re-use it. While it's possible that someone might want to e.g. consider objects "small" for the purposes of diffing but "big" for the purposes of writing them such use-cases are probably too obscure to worry about. We can always split up "core.bigFileThreshold" in the future if there's a need for that.
0. https://github.com/avar/git-hyperfine/
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
--- Documentation/config/core.txt | 4 +- builtin/unpack-objects.c | 69 ++++++++++++++++++++++++++++++++- t/t5351-unpack-large-objects.sh | 43 ++++++++++++++++++-- 3 files changed, 109 insertions(+), 7 deletions(-)
Documentation/config/core.txt +2 -2
@@ -468,8 +468,8 @@ usage, at the slight expense of increased disk usage. * Will be generally be streamed when written, which avoids excessive memory usage, at the cost of some fixed overhead. Commands that make use of this include linkgit:git-archive[1], -linkgit:git-fast-import[1], linkgit:git-index-pack[1] and -linkgit:git-fsck[1]. +linkgit:git-fast-import[1], linkgit:git-index-pack[1], +linkgit:git-unpack-objects[1] and linkgit:git-fsck[1]. core.excludesFile:: Specifies the pathname to the file that contains patterns to
builtin/unpack-objects.c +68 -1
83 lines not on this page · raw message
t/t5351-unpack-large-objects.sh +39 -4
69 lines not on this page · raw message
Signature · 1 line
2.36.1
v15 cover unpack-objects: support streaming blobs to disk
In reply to Han XinThis series makes "unpack-objects" capable of streaming large objects to disk.
As 6/6 shows streaming e.g. a 100MB blob now uses ~5MB of memory instead of ~105MB. This streaming method is slower if you've got memory to handle the blobs in-core, but if you don't it allows you to unpack objects at all, as you might otherwise OOM.
Changes since v14:
* Remove "object-file.c: do fsync() and close() before post-write die()" as it's not necessary anymore. It was first introduced in v10 and was no longer in the utility function end_loose_object_common() since v12. We can see the discussion[1].
* Minor grammar/comment etc. fixes throughout.
1. https://lore.kernel.org/git/[email protected]/
Han Xin (4):
unpack-objects: low memory footprint for get_data() in dry_run mode object-file.c: refactor write_loose_object() to several steps object-file.c: add "stream_loose_object()" to handle large object
unpack-objects: use stream_loose_object() to unpack large objects
Ævar Arnfjörð Bjarmason (2):
object-file.c: factor out deflate part of write_loose_object() core doc: modernize core.bigFileThreshold documentation
Documentation/config/core.txt | 33 +++-- builtin/unpack-objects.c | 106 +++++++++++++-- object-file.c | 233 ++++++++++++++++++++++++++++---- object-store.h | 8 ++ t/t5351-unpack-large-objects.sh | 76 +++++++++++ 5 files changed, 405 insertions(+), 51 deletions(-) create mode 100755 t/t5351-unpack-large-objects.sh
Range-diff against v14:
1: bf600a2fa8 ! 1: 9a776f717d unpack-objects: low memory footprint for get_data() in dry_run mode
@@ Commit message
d9545c7f465 (fast-import: implement unpack limit, 2016-04-25).
- Signed-off-by: Han Xin <[email protected]>
+ Signed-off-by: Han Xin <[email protected]>
## builtin/unpack-objects.c ##
2: a327f484f7 < -: ---------- object-file.c: do fsync() and close() before post-write die() 3: 9bc8002282 ! 2: a1e090d338 object-file.c: refactor write_loose_object() to several steps
@@ Commit message
- Signed-off-by: Han Xin <[email protected]>
+ Signed-off-by: Han Xin <[email protected]>
## object-file.c ##
@@ object-file.c: static int write_loose_object(const struct object_id *oid, char *
- ret = git_deflate_end_gently(&stream);
+ ret = end_loose_object_common(&c, &stream, ¶no_oid);
if (ret != Z_OK)
-- die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
-- ret);
+ die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
+ ret);
- the_hash_algo->final_oid_fn(¶no_oid, &c);
-+ die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid), ret);
- close_loose_object(fd, tmp_file.buf);
-
if (!oideq(oid, ¶no_oid))
+ die(_("confused by unstable object source data for %s"),
+ oid_to_hex(oid));
4: 7c73815f18 = 3: 0ddf912d47 object-file.c: factor out deflate part of write_loose_object() 5: 28a9588f9c ! 4: f9e51d3c68 object-file.c: add "stream_loose_object()" to handle large object
@@ Commit message
- Signed-off-by: Han Xin <[email protected]>
+ Signed-off-by: Han Xin <[email protected]>
## object-file.c ##
6: dea5c4172b ! 5: 61ae1c1632 core doc: modernize core.bigFileThreshold documentation
@@ Documentation/config/core.txt: You probably do not need to adjust this value.
-Common unit suffixes of 'k', 'm', or 'g' are supported.
+* Stored deflated in packfiles, without attempting delta compression.
++
-+The default limit is primarily set with this use-case in mind. With it ++The default limit is primarily set with this use-case in mind. With it,
+most projects will have their source code and other text files delta +compressed, but not larger binary media files.
++
+Storing large files without delta compression avoids excessive memory +usage, at the slight expense of increased disk usage.
++
-+* Will be treated as if though they were labeled "binary" (see
++* Will be treated as if they were labeled "binary" (see
+ linkgit:gitattributes[5]). e.g. linkgit:git-log[1] and
-+ linkgit:git-diff[1] will not diffs for files above this limit.
++ linkgit:git-diff[1] will not compute diffs for files above this limit.
++
-+* Will be generally be streamed when written, which avoids excessive
++* Will generally be streamed when written, which avoids excessive
+memory usage, at the cost of some fixed overhead. Commands that make
+use of this include linkgit:git-archive[1],
+linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
7: d236230a4c ! 6: 5a4782d746 unpack-objects: use stream_loose_object() to unpack large objects
@@ Commit message
- Signed-off-by: Han Xin <[email protected]>
+ Signed-off-by: Han Xin <[email protected]>
## Documentation/config/core.txt ##
@@ Documentation/config/core.txt: usage, at the slight expense of increased disk usage.
- * Will be generally be streamed when written, which avoids excessive
+ * Will generally be streamed when written, which avoids excessive memory usage, at the cost of some fixed overhead. Commands that make use of this include linkgit:git-archive[1], -linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
Signature · 1 line
2.36.1
PATCH v15 1/6 unpack-objects: low memory footprint for get_data() in dry_run mode
From: Han Xin <[email protected]>
As the name implies, "get_data(size)" will allocate and return a given amount of memory. Allocating memory for a large blob object may cause the system to run out of memory. Before preparing to replace calling of "get_data()" to unpack large blob objects in latter commits, refactor "get_data()" to reduce memory footprint for dry_run mode.
Because in dry_run mode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can allocate a smaller buffer and use it as zstream output. Make the function return NULL in the dry-run mode, as no callers use the returned buffer.
The "find [...]objects/?? -type f | wc -l" test idiom being used here is adapted from the same "find" use added to another test in d9545c7f465 (fast-import: implement unpack limit, 2016-04-25).
--- builtin/unpack-objects.c | 37 ++++++++++++++++++++--------- t/t5351-unpack-large-objects.sh | 41 +++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 11 deletions(-) create mode 100755 t/t5351-unpack-large-objects.sh
builtin/unpack-objects.c +26 -11
@@ -97,15 +97,27 @@ static void use(int bytes)display_throughput (progress, consumed_bytes); } +/* +* Decompress zstream from the standard input into a newly +* allocated buffer of specified size and return the buffer. +* The caller is responsible to free the returned buffer. +* +* But for dry_run mode, "get_data()" is only used to check the +* integrity of data, and the returned buffer is not used at all. +* Therefore, in dry_run mode, "get_data()" will release the small +* allocated buffer which is reused to hold temporary zstream output +* and return NULL instead of returning garbage data. +*/ static void *get_data (unsigned long size) {git_zstream stream; -void *buf =xmallocz (size); +unsigned long bufsize = dry_run && size >8192 ?8192 : size; +void *buf =xmallocz (bufsize);memset (&stream,0 ,sizeof (stream)); stream.next_out = buf; - stream.avail_out = size; + stream.avail_out = bufsize; stream.next_in =fill (1 ); stream.avail_in = len;git_inflate_init (&stream);
@@ -125,8 +137,17 @@ static void *get_data(unsigned long size) } stream.next_in =fill (1 ); stream.avail_in = len; +if (dry_run) { +/* reuse the buffer in dry_run mode */ + stream.next_out = buf; + stream.avail_out = bufsize > size - stream.total_out ? + size - stream.total_out : + bufsize; + } }git_inflate_end (&stream); +if (dry_run) +FREE_AND_NULL (buf);return buf; }
@@ -326,10 +347,8 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size, {void *buf =get_data (size); -if (!dry_run && buf) +if (buf)write_object (nr, type, buf, size); -else -free (buf); } static int resolve_against_held(unsigned nr, const struct object_id *base,
@@ -359,10 +378,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,oidread (&base_oid,fill (the_hash_algo->rawsz));use (the_hash_algo->rawsz); delta_data =get_data (delta_size); -if (dry_run || !delta_data) { -free (delta_data); +if (!delta_data)return ; - }if (has_object_file (&base_oid)) ;/* Ok we have this one */ else if (resolve_against_held(nr, &base_oid,
@@ -398,10 +415,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,die ("offset value out of bound for delta base object" ); delta_data =get_data (delta_size); -if (dry_run || !delta_data) { -free (delta_data); +if (!delta_data)return ; - } lo =0 ; hi = nr;while (lo < hi) {
t/t5351-unpack-large-objects.sh new file +41 -0
41 lines not on this page · raw message
Signature · 1 line
2.36.1
PATCH v15 2/6 object-file.c: refactor write_loose_object() to several steps
In reply to Han XinFrom: Han Xin <[email protected]>
When writing a large blob using "write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in later commit to resolve this issue.
Before introducing that streaming function, do some refactoring on "write_loose_object()" to reuse code for both versions.
Rewrite "write_loose_object()" as follows:
1. Figure out a path for the (temp) object file. This step is only used in "write_loose_object()".
2. Move common steps for starting to write loose objects into a new function "start_loose_object_common()".
3. Compress data.
4. Move common steps for ending zlib stream into a new function
"end_loose_object_common()".
5. Close fd and finalize the object file.
--- object-file.c | 98 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 74 insertions(+), 24 deletions(-)
object-file.c +74 -24
@@ -1951,6 +1951,74 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)return fd; } +/** +* Common steps for loose object writers to start writing loose +* objects: +* +* - Create tmpfile for the loose object. +* - Setup zlib stream for compression. +* - Start to feed header to zlib stream. +* +* Returns a "fd", which should later be provided to +* end_loose_object_common(). +*/ +static int start_loose_object_common (struct strbuf *tmp_file, +const char *filename,unsigned flags, +git_zstream *stream, +unsigned char *buf,size_t buflen, +git_hash_ctx *c, +char *hdr,int hdrlen) +{ +int fd; + + fd =create_tmpfile (tmp_file, filename); +if (fd <0 ) { +if (flags &HASH_SILENT ) +return -1 ; +else if (errno ==EACCES ) +return error (_ ("insufficient permission for adding " +"an object to repository database %s" ), +get_object_directory ()); +else +return error_errno ( +_ ("unable to create temporary file" )); + } + +/* Setup zlib stream for compression */ +git_deflate_init (stream, zlib_compression_level); + stream->next_out = buf; + stream->avail_out = buflen; + the_hash_algo->init_fn (c); + +/* Start to feed header to zlib stream */ + stream->next_in = (unsigned char *)hdr; + stream->avail_in = hdrlen; +while (git_deflate (stream,0 ) ==Z_OK ) + ;/* nothing */ + the_hash_algo->update_fn (c, hdr, hdrlen); + +return fd; +} + +/** +* Common steps for loose object writers to end writing loose objects: +* +* - End the compression of zlib stream. +* - Get the calculated oid to "oid". +*/ +static int end_loose_object_common (git_hash_ctx *c,git_zstream *stream, +struct object_id *oid) +{ +int ret; + + ret =git_deflate_end_gently (stream); +if (ret !=Z_OK ) +return ret; + the_hash_algo->final_oid_fn (oid, c); + +return Z_OK ; +} +static int write_loose_object (const struct object_id *oid,char *hdr,int hdrlen,const void *buf,unsigned long len,time_t mtime,unsigned flags)
@@ -1968,28 +2036,11 @@ static int write_loose_object(const struct object_id *oid, char *hdr,loose_object_path (the_repository, &filename, oid); - fd =create_tmpfile (&tmp_file, filename.buf); -if (fd <0 ) { -if (flags &HASH_SILENT ) -return -1 ; -else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); -else -return error_errno (_ ("unable to create temporary file" )); - } - -/* Set it up */ -git_deflate_init (&stream, zlib_compression_level); - stream.next_out = compressed; - stream.avail_out =sizeof (compressed); - the_hash_algo->init_fn (&c); - -/* First header.. */ - stream.next_in = (unsigned char *)hdr; - stream.avail_in = hdrlen; -while (git_deflate (&stream,0 ) ==Z_OK ) - ;/* nothing */ - the_hash_algo->update_fn (&c, hdr, hdrlen); + fd =start_loose_object_common (&tmp_file, filename.buf, flags, + &stream, compressed,sizeof (compressed), + &c, hdr, hdrlen); +if (fd <0 ) +return -1 ;/* Then the data itself.. */ stream.next_in = (void *)buf;
@@ -2007,11 +2058,10 @@ static int write_loose_object(const struct object_id *oid, char *hdr,if (ret !=Z_STREAM_END )die (_ ("unable to deflate new object %s (%d)" ),oid_to_hex (oid), ret); - ret =git_deflate_end_gently (&stream); + ret =end_loose_object_common (&c, &stream, ¶no_oid);if (ret !=Z_OK )die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), ret); - the_hash_algo->final_oid_fn (¶no_oid, &c);if (!oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));
Signature · 1 line
2.36.1
PATCH v15 3/6 object-file.c: factor out deflate part of write_loose_object()
In reply to Han XinFrom: Ævar Arnfjörð Bjarmason <[email protected]>
Split out the part of write_loose_object() that deals with calling git_deflate() into a utility function, a subsequent commit will introduce another function that'll make use of it.
--- object-file.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-)
object-file.c +25 -6
@@ -2000,6 +2000,28 @@ static int start_loose_object_common(struct strbuf *tmp_file,return fd; } +/** +* Common steps for the inner git_deflate() loop for writing loose +* objects. Returns what git_deflate() returns. +*/ +static int write_loose_object_common (git_hash_ctx *c, +git_zstream *stream,const int flush, +unsigned char *in0,const int fd, +unsigned char *compressed, +const size_t compressed_len) +{ +int ret; + + ret =git_deflate (stream, flush ?Z_FINISH :0 ); + the_hash_algo->update_fn (c, in0, stream->next_in - in0); +if (write_buffer (fd, compressed, stream->next_out - compressed) <0 ) +die (_ ("unable to write loose object file" )); + stream->next_out = compressed; + stream->avail_out = compressed_len; + +return ret; +} +/** * Common steps for loose object writers to end writing loose objects: *
@@ -2047,12 +2069,9 @@ static int write_loose_object(const struct object_id *oid, char *hdr, stream.avail_in = len;do {unsigned char *in0 = stream.next_in; - ret =git_deflate (&stream,Z_FINISH ); - the_hash_algo->update_fn (&c, in0, stream.next_in - in0); -if (write_buffer (fd, compressed, stream.next_out - compressed) <0 ) -die (_ ("unable to write loose object file" )); - stream.next_out = compressed; - stream.avail_out =sizeof (compressed); + + ret =write_loose_object_common (&c, &stream,1 , in0, fd, + compressed,sizeof (compressed)); }while (ret ==Z_OK );if (ret !=Z_STREAM_END )
Signature · 1 line
2.36.1
PATCH v15 4/6 object-file.c: add "stream_loose_object()" to handle large object
In reply to Han XinFrom: Han Xin <[email protected]>
If we want unpack and write a loose object using "write_loose_object", we have to feed it with a buffer with the same size of the object, which will consume lots of memory and may cause OOM. This can be improved by feeding data to "stream_loose_object()" in a stream.
Add a new function "stream_loose_object()", which is a stream version of "write_loose_object()" but with a low memory footprint. We will use this function to unpack large blob object in later commit.
Another difference with "write_loose_object()" is that we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. In "write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object.
Still, we need to save the temporary file we're preparing somewhere. We'll do that in the top-level ".git/objects/" directory (or whatever "GIT_OBJECT_DIRECTORY" is set to). Once we've streamed it we'll know the OID, and will move it to its canonical path.
"freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid". After the temporary file is written, we wants to mark the object to recent and we may find that where indeed is already the object. We should remove the temporary and do not leave a new copy of the object.
--- object-file.c | 104 +++++++++++++++++++++++++++++++++++++++++++++++++ object-store.h | 8 ++++ 2 files changed, 112 insertions(+)
object-file.c +104 -0
@@ -2119,6 +2119,110 @@ static int freshen_packed_object(const struct object_id *oid)return 1 ; } +int stream_loose_object (struct input_stream *in_stream,size_t len, +struct object_id *oid) +{ +int fd, ret, err =0 , flush =0 ; +unsigned char compressed[4096 ]; +git_zstream stream; +git_hash_ctx c; +struct strbuf tmp_file =STRBUF_INIT ; +struct strbuf filename =STRBUF_INIT ; +int dirlen; +char hdr[MAX_HEADER_LEN ]; +int hdrlen; + +if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) +prepare_loose_object_bulk_checkin (); + +/* Since oid is not determined, save tmp file to odb path. */ +strbuf_addf (&filename,"%s/" ,get_object_directory ()); + hdrlen =format_object_header (hdr,sizeof (hdr),OBJ_BLOB , len); + +/* +* Common steps for write_loose_object and stream_loose_object to +* start writing loose objects: +* +* - Create tmpfile for the loose object. +* - Setup zlib stream for compression. +* - Start to feed header to zlib stream. +*/ + fd =start_loose_object_common (&tmp_file, filename.buf,0 , + &stream, compressed,sizeof (compressed), + &c, hdr, hdrlen); +if (fd <0 ) { + err =-1 ; + goto cleanup; + } + +/* Then the data itself.. */ +do { +unsigned char *in0 = stream.next_in; + +if (!stream.avail_in && !in_stream->is_finished) { +const void *in = in_stream->read (in_stream, &stream.avail_in); + stream.next_in = (void *)in; + in0 = (unsigned char *)in; +/* All data has been read. */ +if (in_stream->is_finished) + flush =1 ; + } + ret =write_loose_object_common (&c, &stream, flush, in0, fd, + compressed,sizeof (compressed)); +/* +* Unlike write_loose_object(), we do not have the entire +* buffer. If we get Z_BUF_ERROR due to too few input bytes, +* then we'll replenish them in the next input_stream->read() +* call when we loop. +*/ + }while (ret ==Z_OK || ret ==Z_BUF_ERROR ); + +if (stream.total_in != len + hdrlen) +die (_ ("write stream object %ld != %" PRIuMAX), stream.total_in, + (uintmax_t )len + hdrlen); + +/* +* Common steps for write_loose_object and stream_loose_object to +* end writing loose oject: +* +* - End the compression of zlib stream. +* - Get the calculated oid. +*/ +if (ret !=Z_STREAM_END ) +die (_ ("unable to stream deflate new object (%d)" ), ret); + ret =end_loose_object_common (&c, &stream, oid); +if (ret !=Z_OK ) +die (_ ("deflateEnd on stream object failed (%d)" ), ret); +close_loose_object (fd, tmp_file.buf); + +if (freshen_packed_object (oid) ||freshen_loose_object (oid)) { +unlink_or_warn (tmp_file.buf); + goto cleanup; + } + +loose_object_path (the_repository, &filename, oid); + +/* We finally know the object path, and create the missing dir. */ + dirlen =directory_size (filename.buf); +if (dirlen) { +struct strbuf dir =STRBUF_INIT ; +strbuf_add (&dir, filename.buf, dirlen); + +if (mkdir_in_gitdir (dir.buf) && errno !=EEXIST ) { + err =error_errno (_ ("unable to create directory %s" ), dir.buf); +strbuf_release (&dir); + goto cleanup; + } +strbuf_release (&dir); + } + + err =finalize_object_file (tmp_file.buf, filename.buf); +cleanup: +strbuf_release (&tmp_file); +strbuf_release (&filename); +return err; +} +int write_object_file_flags (const void *buf,unsigned long len,enum object_type type,struct object_id *oid,unsigned flags)
object-store.h +8 -0
@@ -46,6 +46,12 @@ struct object_directory {char *path; }; +struct input_stream { +const void *(*read)(struct input_stream *,unsigned long *len); +void *data; +int is_finished; +}; + KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq)
@@ -269,6 +275,8 @@ static inline int write_object_file(const void *buf, unsigned long len,int write_object_file_literally (const void *buf,unsigned long len,const char *type,struct object_id *oid,unsigned flags); +int stream_loose_object (struct input_stream *in_stream,size_t len, +struct object_id *oid);/* * Add an object file to the in-memory object store, without writing it
Signature · 1 line
2.36.1
PATCH v15 5/6 core doc: modernize core.bigFileThreshold documentation
In reply to Han XinFrom: Ævar Arnfjörð Bjarmason <[email protected]>
The core.bigFileThreshold documentation has been largely unchanged since 5eef828bc03 (fast-import: Stream very large blobs directly to pack, 2010-02-01).
But since then this setting has been expanded to affect a lot more than that description indicated. Most notably in how "git diff" treats them, see 6bf3b813486 (diff --stat: mark any file larger than core.bigfilethreshold binary, 2014-08-16).
In addition to that, numerous commands and APIs make use of a streaming mode for files above this threshold.
So let's attempt to summarize 12 years of changes in behavior, which can be seen with:
git log --oneline -Gbig_file_thre 5eef828bc03.. -- '*.c'
To do that turn this into a bullet-point list. The summary Han Xin produced in [1] helped a lot, but is a bit too detailed for documentation aimed at users. Let's instead summarize how user-observable behavior differs, and generally describe how we tend to stream these files in various commands.
1. https://lore.kernel.org/git/[email protected]/
--- Documentation/config/core.txt | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-)
Documentation/config/core.txt +24 -9
@@ -444,17 +444,32 @@ You probably do not need to adjust this value. Common unit suffixes of 'k', 'm', or 'g' are supported. core.bigFileThreshold:: - Files larger than this size are stored deflated, without - attempting delta compression. Storing large files without - delta compression avoids excessive memory usage, at the - slight expense of increased disk usage. Additionally files - larger than this size are always treated as binary. + The size of files considered "big", which as discussed below + changes the behavior of numerous git commands, as well as how + such files are stored within the repository. The default is + 512 MiB. Common unit suffixes of 'k', 'm', or 'g' are + supported. + -Default is 512 MiB on all platforms. This should be reasonable -for most projects as source code and other text files can still -be delta compressed, but larger binary media files won't be. +Files above the configured limit will be: + -Common unit suffixes of 'k', 'm', or 'g' are supported. +* Stored deflated in packfiles, without attempting delta compression. ++ +The default limit is primarily set with this use-case in mind. With it, +most projects will have their source code and other text files delta +compressed, but not larger binary media files. ++ +Storing large files without delta compression avoids excessive memory +usage, at the slight expense of increased disk usage. ++ +* Will be treated as if they were labeled "binary" (see + linkgit:gitattributes[5]). e.g. linkgit:git-log[1] and + linkgit:git-diff[1] will not compute diffs for files above this limit. ++ +* Will generally be streamed when written, which avoids excessive +memory usage, at the cost of some fixed overhead. Commands that make +use of this include linkgit:git-archive[1], +linkgit:git-fast-import[1], linkgit:git-index-pack[1] and +linkgit:git-fsck[1]. core.excludesFile:: Specifies the pathname to the file that contains patterns to
Signature · 1 line
2.36.1
PATCH v15 6/6 unpack-objects: use stream_loose_object() to unpack large objects
In reply to Han XinFrom: Han Xin <[email protected]>
Make use of the stream_loose_object() function introduced in the preceding commit to unpack large objects. Before this we'd need to malloc() the size of the blob before unpacking it, which could cause OOM with very large blobs.
We could use the new streaming interface to unpack all blobs, but doing so would be much slower, as demonstrated e.g. with this benchmark using git-hyperfine[0]:
rm -rf /tmp/scalar.git && git clone --bare https://github.com/Microsoft/scalar.git /tmp/scalar.git && mv /tmp/scalar.git/objects/pack/*.pack /tmp/scalar.git/my.pack && git hyperfine \ -r 2 --warmup 1 \ -L rev origin/master,HEAD -L v "10,512,1k,1m" \ -s 'make' \ -p 'git init --bare dest.git' \ -c 'rm -rf dest.git' \ './git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/scalar.git/my.pack'
Here we'll perform worse with lower core.bigFileThreshold settings with this change in terms of speed, but we're getting lower memory use in return:
Summary
'./git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' ran
1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.02 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.02 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.09 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.10 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.11 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD'
A better benchmark to demonstrate the benefits of that this one, which creates an artificial repo with a 1, 25, 50, 75 and 100MB blob:
rm -rf /tmp/repo &&
git init /tmp/repo &&
(
cd /tmp/repo &&
for i in 1 25 50 75 100
do
dd if=/dev/urandom of=blob.$i count=$(($i*1024)) bs=1024
done &&
git add blob.* &&
git commit -mblobs &&
git gc &&
PACK=$(echo .git/objects/pack/pack-*.pack) &&
cp "$PACK" my.pack
) &&
git hyperfine \
--show-output \
-L rev origin/master,HEAD -L v "512,50m,100m" \
-s 'make' \
-p 'git init --bare dest.git' \
-c 'rm -rf dest.git' \
'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum'Using this test we'll always use >100MB of memory on origin/master (around ~105MB), but max out at e.g. ~55MB if we set core.bigFileThreshold=50m.
The relevant "Maximum resident set size" lines were manually added below the relevant benchmark:
'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master' ran
Maximum resident set size (kbytes): 107080
1.02 ± 0.78 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 106968
1.09 ± 0.79 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 107032
1.42 ± 1.07 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 107072
1.83 ± 1.02 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 55704
2.16 ± 1.19 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 4564
This shows that if you have enough memory this new streaming method is slower the lower you set the streaming threshold, but the benefit is more bounded memory use.
An earlier version of this patch introduced a new "core.bigFileStreamingThreshold" instead of re-using the existing "core.bigFileThreshold" variable[1]. As noted in a detailed overview of its users in [2] using it has several different meanings.
Still, we consider it good enough to simply re-use it. While it's possible that someone might want to e.g. consider objects "small" for the purposes of diffing but "big" for the purposes of writing them such use-cases are probably too obscure to worry about. We can always split up "core.bigFileThreshold" in the future if there's a need for that.
0. https://github.com/avar/git-hyperfine/
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
--- Documentation/config/core.txt | 4 +- builtin/unpack-objects.c | 69 ++++++++++++++++++++++++++++++++- t/t5351-unpack-large-objects.sh | 43 ++++++++++++++++++-- 3 files changed, 109 insertions(+), 7 deletions(-)
Documentation/config/core.txt +2 -2
@@ -468,8 +468,8 @@ usage, at the slight expense of increased disk usage. * Will generally be streamed when written, which avoids excessive memory usage, at the cost of some fixed overhead. Commands that make use of this include linkgit:git-archive[1], -linkgit:git-fast-import[1], linkgit:git-index-pack[1] and -linkgit:git-fsck[1]. +linkgit:git-fast-import[1], linkgit:git-index-pack[1], +linkgit:git-unpack-objects[1] and linkgit:git-fsck[1]. core.excludesFile:: Specifies the pathname to the file that contains patterns to
builtin/unpack-objects.c +68 -1
83 lines not on this page · raw message
t/t5351-unpack-large-objects.sh +39 -4
69 lines not on this page · raw message
Signature · 1 line
2.36.1
Re: [PATCH v15 6/6] unpack-objects: use stream_loose_object() to unpack large objects
t/t5351-unpack-large-objects.sh test_expect_success 'set memory limitation to 1MB' ' · line 61
Quoted 11 lines · Han Xin
t/t5351-unpack-large-objects.sh +11
-0+BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch' + +test_expect_success 'unpack big object in stream (core.fsyncmethod=batch)' ' + prepare_dest 1m && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \ + git -C dest.git $BATCH_CONFIGURATION unpack-objects <pack-$PACK.pack && + grep fsync/hardware-flush trace2.txt && + test_dir_is_empty dest.git/objects/pack && + git -C dest.git cat-file --batch-check="%(objectname)" <obj-list >current && + cmp obj-list current +'
This test without any prerequisite expects that "hardware-flush" will always appear in the trace, but is that reasonable? Don't we need either
(1) some sort of prerequisite to make sure this test piece runs only on platforms that will use hardware-flush, or
(2) loosen grep pattern to look for just "fsync/", or
(3) something else?
It will become even worse when we queue Ævar's "trace2 squelch" patch on top, as we will stop emitting trace entries for that did not trigger.
cover unpack-objects: low memory footprint for get_data() in dry_run mode
In reply to Ævar Arnfjörð BjarmasonThis patch teaches "git unpack-objects" to use a lower memory footprint for "get_data()" in dry-run mode since the returned data is not used.
This patch is separeted from "[PATCH v12 0/8] unpack-objects: support streaming blobs to disk"[1] because it has less impact and less controversy on existing ones.
1. https://lore.kernel.org/git/[email protected]/
Han Xin (1):
unpack-objects: low memory footprint for get_data() in dry_run mode
builtin/unpack-objects.c | 34 ++++++++++++++++++--------- t/t5351-unpack-large-objects.sh | 41 +++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+), 11 deletions(-) create mode 100755 t/t5351-unpack-large-objects.sh
Signature · 1 line
2.36.1
PATCH 1/1 unpack-objects: low memory footprint for get_data() in dry_run mode
From: Han Xin <[email protected]>
As the name implies, "get_data(size)" will allocate and return a given amount of memory. Allocating memory for a large blob object may cause the system to run out of memory. Before preparing to replace calling of "get_data()" to unpack large blob objects in latter commits, refactor "get_data()" to reduce memory footprint for dry_run mode.
Because in dry_run mode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can allocate a smaller buffer and reuse it as zstream output. Therefore, in dry_run mode, "get_data()" will release the allocated buffer and return NULL instead of returning garbage data.
The "find [...]objects/?? -type f | wc -l" test idiom being used here is adapted from the same "find" use added to another test in d9545c7f465 (fast-import: implement unpack limit, 2016-04-25).
--- builtin/unpack-objects.c | 34 ++++++++++++++++++--------- t/t5351-unpack-large-objects.sh | 41 +++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+), 11 deletions(-) create mode 100755 t/t5351-unpack-large-objects.sh
builtin/unpack-objects.c +23 -11
@@ -96,15 +96,26 @@ static void use(int bytes)display_throughput (progress, consumed_bytes); } +/* +* Decompress zstream from stdin and return specific size of data. +* The caller is responsible to free the returned buffer. +* +* But for dry_run mode, "get_data()" is only used to check the +* integrity of data, and the returned buffer is not used at all. +* Therefore, in dry_run mode, "get_data()" will release the small +* allocated buffer which is reused to hold temporary zstream output +* and return NULL instead of returning garbage data. +*/ static void *get_data (unsigned long size) {git_zstream stream; -void *buf =xmallocz (size); +unsigned long bufsize = dry_run && size >8192 ?8192 : size; +void *buf =xmallocz (bufsize);memset (&stream,0 ,sizeof (stream)); stream.next_out = buf; - stream.avail_out = size; + stream.avail_out = bufsize; stream.next_in =fill (1 ); stream.avail_in = len;git_inflate_init (&stream);
@@ -124,8 +135,15 @@ static void *get_data(unsigned long size) } stream.next_in =fill (1 ); stream.avail_in = len; +if (dry_run) { +/* reuse the buffer in dry_run mode */ + stream.next_out = buf; + stream.avail_out = bufsize; + } }git_inflate_end (&stream); +if (dry_run) +FREE_AND_NULL (buf);return buf; }
@@ -325,10 +343,8 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size, {void *buf =get_data (size); -if (!dry_run && buf) +if (buf)write_object (nr, type, buf, size); -else -free (buf); } static int resolve_against_held(unsigned nr, const struct object_id *base,
@@ -358,10 +374,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,oidread (&base_oid,fill (the_hash_algo->rawsz));use (the_hash_algo->rawsz); delta_data =get_data (delta_size); -if (dry_run || !delta_data) { -free (delta_data); +if (!delta_data)return ; - }if (has_object_file (&base_oid)) ;/* Ok we have this one */ else if (resolve_against_held(nr, &base_oid,
@@ -397,10 +411,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,die ("offset value out of bound for delta base object" ); delta_data =get_data (delta_size); -if (dry_run || !delta_data) { -free (delta_data); +if (!delta_data)return ; - } lo =0 ; hi = nr;while (lo < hi) {
t/t5351-unpack-large-objects.sh new file +41 -0
41 lines not on this page · raw message
Signature · 1 line
2.36.1
PATCH v9 2/5 object-file.c: refactor write_loose_object() to several steps
In reply to Han XinFrom: Han Xin <[email protected]>
When writing a large blob using "write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in latter commit to resolve this issue.
Before introducing a stream vesion function for writing loose object, do some refactoring on "write_loose_object()" to reuse code for both versions.
Rewrite "write_loose_object()" as follows:
1. Figure out a path for the (temp) object file. This step is only used in "write_loose_object()".
2. Move common steps for starting to write loose objects into a new function "start_loose_object_common()".
3. Compress data.
4. Move common steps for ending zlib stream into a new funciton
"end_loose_object_common()".
5. Close fd and finalize the object file.
--- object-file.c | 105 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 75 insertions(+), 30 deletions(-)
object-file.c +75 -30
@@ -1857,6 +1857,59 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)return fd; } +static int start_loose_object_common (struct strbuf *tmp_file, +const char *filename,unsigned flags, +git_zstream *stream, +unsigned char *buf,size_t buflen, +git_hash_ctx *c, +char *hdr,int hdrlen) +{ +int fd; + + fd =create_tmpfile (tmp_file, filename); +if (fd <0 ) { +if (flags &HASH_SILENT ) +return -1 ; +else if (errno ==EACCES ) +return error (_ ("insufficient permission for adding " +"an object to repository database %s" ), +get_object_directory ()); +else +return error_errno ( +_ ("unable to create temporary file" )); + } + +/* Setup zlib stream for compression */ +git_deflate_init (stream, zlib_compression_level); + stream->next_out = buf; + stream->avail_out = buflen; + the_hash_algo->init_fn (c); + +/* Start to feed header to zlib stream */ + stream->next_in = (unsigned char *)hdr; + stream->avail_in = hdrlen; +while (git_deflate (stream,0 ) ==Z_OK ) + ;/* nothing */ + the_hash_algo->update_fn (c, hdr, hdrlen); + +return fd; +} + +static void end_loose_object_common (int ret,git_hash_ctx *c, +git_zstream *stream, +struct object_id *parano_oid, +const struct object_id *expected_oid, +const char *die_msg1_fmt, +const char *die_msg2_fmt) +{ +if (ret !=Z_STREAM_END ) +die (_ (die_msg1_fmt), ret, expected_oid); + ret =git_deflate_end_gently (stream); +if (ret !=Z_OK ) +die (_ (die_msg2_fmt), ret, expected_oid); + the_hash_algo->final_oid_fn (parano_oid, c); +} +static int write_loose_object (const struct object_id *oid,char *hdr,int hdrlen,const void *buf,unsigned long len,time_t mtime,unsigned flags)
@@ -1871,28 +1924,18 @@ static int write_loose_object(const struct object_id *oid, char *hdr,loose_object_path (the_repository, &filename, oid); - fd =create_tmpfile (&tmp_file, filename.buf); -if (fd <0 ) { -if (flags &HASH_SILENT ) -return -1 ; -else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); -else -return error_errno (_ ("unable to create temporary file" )); - } - -/* Set it up */ -git_deflate_init (&stream, zlib_compression_level); - stream.next_out = compressed; - stream.avail_out =sizeof (compressed); - the_hash_algo->init_fn (&c); - -/* First header.. */ - stream.next_in = (unsigned char *)hdr; - stream.avail_in = hdrlen; -while (git_deflate (&stream,0 ) ==Z_OK ) - ;/* nothing */ - the_hash_algo->update_fn (&c, hdr, hdrlen); +/* Common steps for write_loose_object and stream_loose_object to +* start writing loose oject: +* +* - Create tmpfile for the loose object. +* - Setup zlib stream for compression. +* - Start to feed header to zlib stream. +*/ + fd =start_loose_object_common (&tmp_file, filename.buf, flags, + &stream, compressed,sizeof (compressed), + &c, hdr, hdrlen); +if (fd <0 ) +return -1 ;/* Then the data itself.. */ stream.next_in = (void *)buf;
@@ -1907,14 +1950,16 @@ static int write_loose_object(const struct object_id *oid, char *hdr, stream.avail_out =sizeof (compressed); }while (ret ==Z_OK ); -if (ret !=Z_STREAM_END ) -die (_ ("unable to deflate new object %s (%d)" ),oid_to_hex (oid), - ret); - ret =git_deflate_end_gently (&stream); -if (ret !=Z_OK ) -die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), - ret); - the_hash_algo->final_oid_fn (¶no_oid, &c); +/* Common steps for write_loose_object and stream_loose_object to +* end writing loose oject: +* +* - End the compression of zlib stream. +* - Get the calculated oid to "parano_oid". +*/ +end_loose_object_common (ret, &c, &stream, ¶no_oid, oid, +N_ ("unable to deflate new object %s (%d)" ), +N_ ("deflateEnd on object %s failed (%d)" )); +if (!oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
PATCH v9 3/5 object-file.c: add "stream_loose_object()" to handle large object
In reply to Han XinFrom: Han Xin <[email protected]>
If we want unpack and write a loose object using "write_loose_object", we have to feed it with a buffer with the same size of the object, which will consume lots of memory and may cause OOM. This can be improved by feeding data to "stream_loose_object()" in a stream.
Add a new function "stream_loose_object()", which is a stream version of "write_loose_object()" but with a low memory footprint. We will use this function to unpack large blob object in latter commit.
Another difference with "write_loose_object()" is that we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. In "write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
"freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid".
--- object-file.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++++ object-store.h | 9 +++++ 2 files changed, 110 insertions(+)
object-file.c +101 -0
@@ -1996,6 +1996,107 @@ static int freshen_packed_object(const struct object_id *oid)return 1 ; } +int stream_loose_object (struct input_stream *in_stream,size_t len, +struct object_id *oid) +{ +int fd, ret, err =0 , flush =0 ; +unsigned char compressed[4096 ]; +git_zstream stream; +git_hash_ctx c; +struct strbuf tmp_file =STRBUF_INIT ; +struct strbuf filename =STRBUF_INIT ; +int dirlen; +char hdr[MAX_HEADER_LEN ]; +int hdrlen; + +/* Since oid is not determined, save tmp file to odb path. */ +strbuf_addf (&filename,"%s/" ,get_object_directory ()); + hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX,type_name (OBJ_BLOB ), len) +1 ; + +/* Common steps for write_loose_object and stream_loose_object to +* start writing loose oject: +* +* - Create tmpfile for the loose object. +* - Setup zlib stream for compression. +* - Start to feed header to zlib stream. +*/ + fd =start_loose_object_common (&tmp_file, filename.buf,0 , + &stream, compressed,sizeof (compressed), + &c, hdr, hdrlen); +if (fd <0 ) { + err =-1 ; + goto cleanup; + } + +/* Then the data itself.. */ +do { +unsigned char *in0 = stream.next_in; +if (!stream.avail_in && !in_stream->is_finished) { +const void *in = in_stream->read (in_stream, &stream.avail_in); + stream.next_in = (void *)in; + in0 = (unsigned char *)in; +/* All data has been read. */ +if (in_stream->is_finished) + flush =Z_FINISH ; + } + ret =git_deflate (&stream, flush); + the_hash_algo->update_fn (&c, in0, stream.next_in - in0); +if (write_buffer (fd, compressed, stream.next_out - compressed) <0 ) +die (_ ("unable to write loose object file" )); + stream.next_out = compressed; + stream.avail_out =sizeof (compressed); +/* +* Unlike write_loose_object(), we do not have the entire +* buffer. If we get Z_BUF_ERROR due to too few input bytes, +* then we'll replenish them in the next input_stream->read() +* call when we loop. +*/ + }while (ret ==Z_OK || ret ==Z_BUF_ERROR ); + +if (stream.total_in != len + hdrlen) +die (_ ("write stream object %ld != %" PRIuMAX), stream.total_in, + (uintmax_t )len + hdrlen); + +/* Common steps for write_loose_object and stream_loose_object to +* end writing loose oject: +* +* - End the compression of zlib stream. +* - Get the calculated oid. +*/ +end_loose_object_common (ret, &c, &stream, oid,NULL , +N_ ("unable to stream deflate new object (%d)" ), +N_ ("deflateEnd on stream object failed (%d)" )); + +close_loose_object (fd); + +if (freshen_packed_object (oid) ||freshen_loose_object (oid)) { +unlink_or_warn (tmp_file.buf); + goto cleanup; + } + +loose_object_path (the_repository, &filename, oid); + +/* We finally know the object path, and create the missing dir. */ + dirlen =directory_size (filename.buf); +if (dirlen) { +struct strbuf dir =STRBUF_INIT ; +strbuf_add (&dir, filename.buf, dirlen); + +if (mkdir_in_gitdir (dir.buf) && errno !=EEXIST ) { + err =error_errno (_ ("unable to create directory %s" ), dir.buf); +strbuf_release (&dir); + goto cleanup; + } +strbuf_release (&dir); + } + + err =finalize_object_file (tmp_file.buf, filename.buf); +cleanup: +strbuf_release (&tmp_file); +strbuf_release (&filename); +return err; +} +int write_object_file_flags (const void *buf,unsigned long len,const char *type,struct object_id *oid,unsigned flags)
object-store.h +9 -0
@@ -34,6 +34,12 @@ struct object_directory {char *path; }; +struct input_stream { +const void *(*read)(struct input_stream *,unsigned long *len); +void *data; +int is_finished; +}; + KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq)
@@ -232,6 +238,9 @@ static inline int write_object_file(const void *buf, unsigned long len,return write_object_file_flags (buf, len, type, oid,0 ); } +int stream_loose_object (struct input_stream *in_stream,size_t len, +struct object_id *oid); +int hash_object_file_literally (const void *buf,unsigned long len,const char *type,struct object_id *oid,unsigned flags);
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
PATCH v9 4/5 unpack-objects: unpack_non_delta_entry() read data in a stream
In reply to Han XinFrom: Han Xin <[email protected]>
We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.
By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()". However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty.
$ hyperfine \
--setup \
'if ! test -d scalar.git; then git clone --bare
https://github.com/microsoft/scalar.git;
cp scalar.git/objects/pack/*.pack small.pack; fi' \
--prepare 'rm -rf dest.git && git init --bare dest.git' \
... Summary
'./git -C dest.git -c core.bigFileThreshold=512m
unpack-objects <small.pack' in 'origin/master'
1.01 ± 0.04 times faster than './git -C dest.git
-c core.bigFileThreshold=512m unpack-objects
<small.pack' in 'HEAD~1'
1.01 ± 0.04 times faster than './git -C dest.git
-c core.bigFileThreshold=512m unpack-objects
<small.pack' in 'HEAD~0'
1.03 ± 0.10 times faster than './git -C dest.git
-c core.bigFileThreshold=16k unpack-objects
<small.pack' in 'origin/master'
1.02 ± 0.07 times faster than './git -C dest.git
-c core.bigFileThreshold=16k unpack-objects
<small.pack' in 'HEAD~0'
1.10 ± 0.04 times faster than './git -C dest.git
-c core.bigFileThreshold=16k unpack-objects
<small.pack' in 'HEAD~1'Therefore, only unpack objects larger than the "core.bigFileThreshold" in zstream. Until now, the config variable has been used in the following cases, and our new case belongs to the packfile category.
* Archive:
+ archive.c: write_entry(): write large blob entries to archive in stream.
* Loose objects:
+ object-file.c: index_fd(): when hashing large files in worktree, read files in a stream, and create one packfile per large blob if want to save files to git object store.
+ object-file.c: read_loose_object(): when checking loose objects using "git-fsck", do not read full content of large loose objects.
* Packfile:
+ fast-import.c: parse_and_store_blob(): streaming large blob from foreign source to packfile.
+ index-pack.c: check_collison(): read and check large blob in stream.
+ index-pack.c: unpack_entry_data(): do not return the entire contents of the big blob from packfile, but uses a fixed buf to perform some integrity checks on the object.
+ pack-check.c: verify_packfile(): used by "git-fsck" and will call check_object_signature() to check large blob in pack with the streaming interface.
+ pack-objects.c: get_object_details(): set "no_try_delta" for large blobs when counting objects.
+ pack-objects.c: write_no_reuse_object(): streaming large blob to pack.
+ unpack-objects.c: unpack_non_delta_entry(): unpack large blob in stream from packfile.
* Others:
+ diff.c: diff_populate_filespec(): treat large blob file as binary.
+ streaming.c: istream_source(): as a helper of "open_istream()" to select proper streaming interface to read large blob from packfile.
--- builtin/unpack-objects.c | 71 ++++++++++++++++++++++++++++++++- t/t5328-unpack-large-objects.sh | 23 +++++++++-- 2 files changed, 90 insertions(+), 4 deletions(-)
builtin/unpack-objects.c +70 -1
@@ -343,11 +343,80 @@ static void added_object(unsigned nr, enum object_type type, } } +struct input_zstream_data { +git_zstream *zstream; +unsigned char buf[8192 ]; +int status; +}; + +static const void *feed_input_zstream (struct input_stream *in_stream, +unsigned long *readlen) +{ +struct input_zstream_data *data = in_stream->data; +git_zstream *zstream = data->zstream; +void *in =fill (1 ); + +if (in_stream->is_finished) { + *readlen =0 ; +return NULL ; + } + + zstream->next_out = data->buf; + zstream->avail_out =sizeof (data->buf); + zstream->next_in = in; + zstream->avail_in = len; + + data->status =git_inflate (zstream,0 ); + + in_stream->is_finished = data->status !=Z_OK ; +use (len - zstream->avail_in); + *readlen =sizeof (data->buf) - zstream->avail_out; + +return data->buf; +} + +static void write_stream_blob (unsigned nr,size_t size) +{ +git_zstream zstream = {0 }; +struct input_zstream_data data = {0 }; +struct input_stream in_stream = { + .read = feed_input_zstream, + .data = &data, + }; + + data.zstream = &zstream; +git_inflate_init (&zstream); + +if (stream_loose_object (&in_stream, size, &obj_list[nr].oid)) +die (_ ("failed to write object in stream" )); + +if (data.status !=Z_STREAM_END ) +die (_ ("inflate returned (%d)" ), data.status); +git_inflate_end (&zstream); + +if (strict) { +struct blob *blob = +lookup_blob (the_repository, &obj_list[nr].oid); +if (blob) + blob->object.flags |=FLAG_WRITTEN ; +else +die (_ ("invalid blob object from stream" )); + } + obj_list[nr].obj =NULL ; +} +static void unpack_non_delta_entry (enum object_type type,unsigned long size,unsigned nr) { -void *buf =get_data (size); +void *buf; + +/* Write large blob in stream without allocating full buffer. */ +if (!dry_run && type ==OBJ_BLOB && size > big_file_threshold) { +write_stream_blob (nr, size); +return ; + } + buf =get_data (size);if (buf)write_object (nr, type, buf, size); }
t/t5328-unpack-large-objects.sh +20 -3
@@ -9,7 +9,11 @@ test_description='git unpack-objects with large objects' prepare_dest () { test_when_finished "rm -rf dest.git" && - git init --bare dest.git + git init --bare dest.git && + if test -n "$1" + then + git -C dest.git config core.bigFileThreshold $1 + fi } test_no_loose () {
@@ -33,16 +37,29 @@ test_expect_success 'set memory limitation to 1MB' ' ' test_expect_success 'unpack-objects failed under memory limitation' ' - prepare_dest && + prepare_dest 2m && test_must_fail git -C dest.git unpack-objects <test-$PACK.pack 2>err && grep "fatal: attempting to allocate" err ' test_expect_success 'unpack-objects works with memory limitation in dry-run mode' ' - prepare_dest && + prepare_dest 2m && git -C dest.git unpack-objects -n <test-$PACK.pack && test_no_loose && test_dir_is_empty dest.git/objects/pack ' +test_expect_success 'unpack big object in stream' ' + prepare_dest 1m && + git -C dest.git unpack-objects <test-$PACK.pack && + test_dir_is_empty dest.git/objects/pack +' + +test_expect_success 'do not unpack existing large objects' ' + prepare_dest 1m && + git -C dest.git index-pack --stdin <test-$PACK.pack && + git -C dest.git unpack-objects <test-$PACK.pack && + test_no_loose +' + test_done
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
PATCH v9 5/5 object-file API: add a format_object_header() function
In reply to Han XinFrom: Ævar Arnfjörð Bjarmason <[email protected]>
Add a convenience function to wrap the xsnprintf() command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.
All except one caller of it had a valid "enum object_type" for us, it's only write_object_file_prepare() which might need to deal with "git hash-object --literally" and a potential garbage type. Let's have the primary API use an "enum object_type", and define an *_extended() function that can take an arbitrary "const char *" for the type.
See [1] for the discussion that prompted this patch, i.e. new code in object-file.c that wanted to copy/paste the xsnprintf() invocation.
1. https://lore.kernel.org/git/[email protected]/
--- builtin/index-pack.c | 3 +-- bulk-checkin.c | 4 ++-- cache.h | 21 +++++++++++++++++++++ http-push.c | 2 +- object-file.c | 16 ++++++++++++---- 5 files changed, 37 insertions(+), 9 deletions(-)
builtin/index-pack.c +1 -2
@@ -449,8 +449,7 @@ static void *unpack_entry_data(off_t offset, unsigned long size,int hdrlen;if (!is_delta_type (type)) { - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX, -type_name (type),(uintmax_t )size) +1 ; + hdrlen =format_object_header (hdr,sizeof (hdr), type, size); the_hash_algo->init_fn (&c); the_hash_algo->update_fn (&c, hdr, hdrlen); }else
bulk-checkin.c +2 -2
@@ -220,8 +220,8 @@ static int deflate_to_pack(struct bulk_checkin_state *state,if (seekback == (off_t) -1 )return error ("cannot find the current offset" ); - header_len =xsnprintf ((char *)obuf,sizeof (obuf),"%s %" PRIuMAX, -type_name (type), (uintmax_t )size) +1 ; + header_len =format_object_header ((char *)obuf,sizeof (obuf), + type, size); the_hash_algo->init_fn (&ctx); the_hash_algo->update_fn (&ctx, obuf, header_len);
cache.h +21 -0
@@ -1310,6 +1310,27 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,unsigned long bufsiz, struct strbuf *hdrbuf); +/** +* format_object_header() is a thin wrapper around s xsnprintf() that +* writes the initial "<type> <obj-len>" part of the loose object +* header. It returns the size that snprintf() returns + 1. +* +* The format_object_header_extended() function allows for writing a +* type_name that's not one of the "enum object_type" types. This is +* used for "git hash-object --literally". Pass in a OBJ_NONE as the +* type, and a non-NULL "type_str" to do that. +* +* format_object_header() is a convenience wrapper for +* format_object_header_extended(). +*/ +int format_object_header_extended (char *str,size_t size,enum object_type type, +const char *type_str,size_t objsize); +static inline int format_object_header (char *str,size_t size, +enum object_type type,size_t objsize) +{ +return format_object_header_extended (str, size, type,NULL , objsize); +} +/** * parse_loose_header() parses the starting "<type> <len>\0" of an * object. If it doesn't follow that format -1 is returned. To check
http-push.c +1 -1
@@ -363,7 +363,7 @@ static void start_put(struct transfer_request *request)git_zstream stream; unpacked =read_object_file (&request->obj->oid, &type, &len); - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; + hdrlen =format_object_header (hdr,sizeof (hdr), type, len);/* Set it up */ git_deflate_init (&stream, zlib_compression_level);
object-file.c +12 -4
@@ -1006,6 +1006,14 @@ void *xmmap(void *start, size_t length,return ret; } +int format_object_header_extended (char *str,size_t size,enum object_type type, +const char *typestr,size_t objsize) +{ +const char *s = type ==OBJ_NONE ? typestr :type_name (type); + +return xsnprintf (str, size,"%s %" PRIuMAX, s, (uintmax_t )objsize) +1 ; +} +/* * With an in-core object data in "map", rehash it to make sure the * object name actually matches "oid" to detect object corruption.
@@ -1034,7 +1042,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,return -1 ;/* Generate the header */ - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (obj_type), (uintmax_t )size) +1 ; + hdrlen =format_object_header (hdr,sizeof (hdr), obj_type, size);/* Sha1.. */ r->hash_algo->init_fn (&c);
@@ -1734,7 +1742,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,git_hash_ctx c;/* Generate the header */ - *hdrlen =xsnprintf (hdr, *hdrlen,"%s %" PRIuMAX , type, (uintmax_t )len)+1 ; + *hdrlen =format_object_header_extended (hdr, *hdrlen,OBJ_NONE , type, len);/* Sha1.. */ algo->init_fn (&c);
@@ -2011,7 +2019,7 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,/* Since oid is not determined, save tmp file to odb path. */ strbuf_addf (&filename,"%s/" ,get_object_directory ()); - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX,type_name (OBJ_BLOB ), len) +1 ; + hdrlen =format_object_header (hdr,sizeof (hdr),OBJ_BLOB , len);/* Common steps for write_loose_object and stream_loose_object to * start writing loose oject:
@@ -2152,7 +2160,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime) buf =read_object (the_repository, oid, &type, &len);if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid)); - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; + hdrlen =format_object_header (hdr,sizeof (hdr), type, len); ret =write_loose_object (oid, hdr, hdrlen, buf, len, mtime,0 );free (buf);
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
PATCH v8 1/6 unpack-objects: low memory footprint for get_data() in dry_run mode
In reply to Han XinFrom: Han Xin <[email protected]>
As the name implies, "get_data(size)" will allocate and return a given size of memory. Allocating memory for a large blob object may cause the system to run out of memory. Before preparing to replace calling of "get_data()" to unpack large blob objects in latter commits, refactor "get_data()" to reduce memory footprint for dry_run mode.
Because in dry_run mode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can allocate a smaller buffer and reuse it as zstream output. Therefore, in dry_run mode, "get_data()" will release the allocated buffer and return NULL instead of returning garbage data.
--- builtin/unpack-objects.c | 39 ++++++++++++++++++------- t/t5329-unpack-large-objects.sh | 52 +++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+), 11 deletions(-) create mode 100755 t/t5329-unpack-large-objects.sh
builtin/unpack-objects.c +28 -11
@@ -96,15 +96,31 @@ static void use(int bytes)display_throughput (progress, consumed_bytes); } +/* +* Decompress zstream from stdin and return specific size of data. +* The caller is responsible to free the returned buffer. +* +* But for dry_run mode, "get_data()" is only used to check the +* integrity of data, and the returned buffer is not used at all. +* Therefore, in dry_run mode, "get_data()" will release the small +* allocated buffer which is reused to hold temporary zstream output +* and return NULL instead of returning garbage data. +*/ static void *get_data (unsigned long size) {git_zstream stream; -void *buf =xmallocz (size); +unsigned long bufsize; +void *buf;memset (&stream,0 ,sizeof (stream)); +if (dry_run && size >8192 ) + bufsize =8192 ; +else + bufsize = size; + buf =xmallocz (bufsize); stream.next_out = buf; - stream.avail_out = size; + stream.avail_out = bufsize; stream.next_in =fill (1 ); stream.avail_in = len;git_inflate_init (&stream);
@@ -124,8 +140,15 @@ static void *get_data(unsigned long size) } stream.next_in =fill (1 ); stream.avail_in = len; +if (dry_run) { +/* reuse the buffer in dry_run mode */ + stream.next_out = buf; + stream.avail_out = bufsize; + } }git_inflate_end (&stream); +if (dry_run) +FREE_AND_NULL (buf);return buf; }
@@ -325,10 +348,8 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size, {void *buf =get_data (size); -if (!dry_run && buf) +if (buf)write_object (nr, type, buf, size); -else -free (buf); } static int resolve_against_held(unsigned nr, const struct object_id *base,
@@ -358,10 +379,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,oidread (&base_oid,fill (the_hash_algo->rawsz));use (the_hash_algo->rawsz); delta_data =get_data (delta_size); -if (dry_run || !delta_data) { -free (delta_data); +if (!delta_data)return ; - }if (has_object_file (&base_oid)) ;/* Ok we have this one */ else if (resolve_against_held(nr, &base_oid,
@@ -397,10 +416,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,die ("offset value out of bound for delta base object" ); delta_data =get_data (delta_size); -if (dry_run || !delta_data) { -free (delta_data); +if (!delta_data)return ; - } lo =0 ; hi = nr;while (lo < hi) {
t/t5329-unpack-large-objects.sh new file +52 -0
52 lines not on this page · raw message
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
Re: [PATCH v8 1/6] unpack-objects: low memory footprint for get_data() in dry_run mode
Am 08.01.22 um 09:54 schrieb Han Xin:
Quoted 102 lines
From: Han Xin <[email protected]>
As the name implies, "
get_data(size)" will allocate and return a given size of memory. Allocating memory for a large blob object may cause the system to run out of memory. Before preparing to replace calling of "get_data()" to unpack large blob objects in latter commits, refactor "get_data()" to reduce memory footprint fordry_runmode.Because in
dry_runmode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can allocate a smaller buffer and reuse it as zstream output. Therefore, indry_runmode, "get_data()" will release the allocated buffer and return NULL instead of returning garbage data.Suggested-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- builtin/unpack-objects.c | 39 ++++++++++++++++++------- t/t5329-unpack-large-objects.sh | 52 +++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+), 11 deletions(-) create mode 100755 t/t5329-unpack-large-objects.shbuiltin/unpack-objects.c +27
-8@@ -96,15 +96,31 @@ static void use(int bytes)display_throughput (progress, consumed_bytes); } +/* +* Decompress zstream from stdin and return specific size of data. +* The caller is responsible to free the returned buffer. +* +* But for dry_run mode, "get_data()" is only used to check the +* integrity of data, and the returned buffer is not used at all. +* Therefore, in dry_run mode, "get_data()" will release the small +* allocated buffer which is reused to hold temporary zstream output +* and return NULL instead of returning garbage data. +*/ static void *get_data (unsigned long size) {git_zstream stream; -void *buf =xmallocz (size); +unsigned long bufsize; +void *buf;memset (&stream,0 ,sizeof (stream)); +if (dry_run && size >8192 ) + bufsize =8192 ; +else + bufsize = size; + buf =xmallocz (bufsize); stream.next_out = buf; - stream.avail_out = size; + stream.avail_out = bufsize; stream.next_in =fill (1 ); stream.avail_in = len;git_inflate_init (&stream);@@ -124,8 +140,15 @@ static void *get_data(unsigned long size) } stream.next_in =fill (1 ); stream.avail_in = len; +if (dry_run) { +/* reuse the buffer in dry_run mode */ + stream.next_out = buf; + stream.avail_out = bufsize; + } }git_inflate_end (&stream); +if (dry_run) +FREE_AND_NULL (buf);return buf; }@@ -325,10 +348,8 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size, {void *buf =get_data (size); -if (!dry_run && buf) +if (buf)write_object (nr, type, buf, size); -else -free (buf); } static int resolve_against_held(unsigned nr, const struct object_id *base,@@ -358,10 +379,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,oidread (&base_oid,fill (the_hash_algo->rawsz));use (the_hash_algo->rawsz); delta_data =get_data (delta_size); -if (dry_run || !delta_data) { -free (delta_data); +if (!delta_data)return ; - }if (has_object_file (&base_oid)) ;/* Ok we have this one */ else if (resolve_against_held(nr, &base_oid,@@ -397,10 +416,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,die ("offset value out of bound for delta base object" );
builtin/unpack-objects.c static void unpack_delta_entry(enum object_type type, unsigned long delta_size, · line 423
Quoted 10 lines
builtin/unpack-objects.c +1
-3delta_data =get_data (delta_size); -if (dry_run || !delta_data) { -free (delta_data); +if (!delta_data)return ; - } lo =0 ; hi = nr;while (lo < hi) {
Nice!
Quoted 19 lines
t/t5329-unpack-large-objects.sh new file +13
-0@@ -0,0 +1,52 @@ +#!/bin/sh +# +# Copyright (c) 2021 Han Xin +# + +test_description='git unpack-objects with large objects' + +. ./test-lib.sh + +prepare_dest () { + test_when_finished "rm -rf dest.git" && + git init --bare dest.git +}
t/t5329-unpack-large-objects.sh line 23
Quoted 10 lines
t/t5329-unpack-large-objects.sh new file +10
-0+ +assert_no_loose () { + glob=dest.git/objects/?? && + echo "$glob" >expect && + eval "echo $glob" >actual && + test_cmp expect actual +} + +assert_no_pack () { + rmdir dest.git/objects/pack
I would expect a function whose name starts with "assert" to have no side effects. It doesn't matter here, because it's called only at the very end, but that might change. You can use test_dir_is_empty instead of rmdir.
Quoted 29 lines
t/t5329-unpack-large-objects.sh new file +29
-0+} + +test_expect_success "create large objects (1.5 MB) and PACK" ' + test-tool genrandom foo 1500000 >big-blob && + test_commit --append foo big-blob && + test-tool genrandom bar 1500000 >big-blob && + test_commit --append bar big-blob && + PACK=$(echo HEAD | git pack-objects --revs test) +' + +test_expect_success 'set memory limitation to 1MB' ' + GIT_ALLOC_LIMIT=1m && + export GIT_ALLOC_LIMIT +' + +test_expect_success 'unpack-objects failed under memory limitation' ' + prepare_dest && + test_must_fail git -C dest.git unpack-objects <test-$PACK.pack 2>err && + grep "fatal: attempting to allocate" err +' + +test_expect_success 'unpack-objects works with memory limitation in dry-run mode' ' + prepare_dest && + git -C dest.git unpack-objects -n <test-$PACK.pack && + assert_no_loose && + assert_no_pack +' + +test_done
Re: [PATCH v8 1/6] unpack-objects: low memory footprint for get_data() in dry_run mode
Quoted 4 lines · René Scharfe
Am 08.01.22 um 09:54 schrieb Han Xin:
Quoted 2 lines
From: Han Xin <[email protected]>
Quoted 15 lines
t/t5329-unpack-large-objects.sh line 23
Quoted 9 lines
t/t5329-unpack-large-objects.sh new file +9
-0+assert_no_loose () { + glob=dest.git/objects/?? && + echo "$glob" >expect && + eval "echo $glob" >actual && + test_cmp expect actual +} + +assert_no_pack () { + rmdir dest.git/objects/packI would expect a function whose name starts with "assert" to have no side effects. It doesn't matter here, because it's called only at the very end, but that might change. You can use
test_dir_is_emptyinstead of rmdir.
*nod*
I think it would be better to rename "assert_no_loose()" to "test_no_loose()". I will remove "assert_no_pack()" and use "test_dir_is_empty()" instead.
Thanks
-Han Xin
PATCH v8 2/6 object-file.c: refactor write_loose_object() to several steps
In reply to Han XinFrom: Han Xin <[email protected]>
When writing a large blob using "write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in latter commit to resolve this issue.
Before introducing a stream vesion function for writing loose object, do some refactoring on "write_loose_object()" to reuse code for both versions.
Rewrite "write_loose_object()" as follows:
1. Figure out a path for the (temp) object file. This step is only used in "write_loose_object()".
2. Move common steps for starting to write loose objects into a new function "start_loose_object_common()".
3. Compress data.
4. Move common steps for ending zlib stream into a new funciton
"end_loose_object_common()".
5. Close fd and finalize the object file.
--- object-file.c | 149 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 105 insertions(+), 44 deletions(-)
object-file.c +105 -44
@@ -1743,6 +1743,25 @@ static void write_object_file_prepare(const struct git_hash_algo *algo, algo->final_oid_fn (oid, &c); } +/* +* Move the just written object with proper mtime into its final resting place. +*/ +static int finalize_object_file_with_mtime (const char *tmpfile, +const char *filename, +time_t mtime, +unsigned flags) +{ +struct utimbuf utb; + +if (mtime) { + utb.actime = mtime; + utb.modtime = mtime; +if (utime (tmpfile, &utb) <0 && !(flags &HASH_SILENT )) +warning_errno (_ ("failed utime() on %s" ), tmpfile); + } +return finalize_object_file (tmpfile, filename); +} +/* * Move the just written object into its final resting place. */
@@ -1828,7 +1847,8 @@ static inline int directory_size(const char *filename)* We want to avoid cross-directory filename renames, because those * can have problems on various filesystems (FAT, NFS, Coda). */ -static int create_tmpfile (struct strbuf *tmp,const char *filename) +static int create_tmpfile (struct strbuf *tmp,const char *filename, +unsigned flags) {int fd, dirlen =directory_size (filename);
@@ -1836,7 +1856,9 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)strbuf_add (tmp, filename, dirlen);strbuf_addstr (tmp,"tmp_obj_XXXXXX" ); fd =git_mkstemp_mode (tmp->buf,0444 ); -if (fd <0 && dirlen && errno ==ENOENT ) { + do { + if (fd >= 0 || !dirlen || errno != ENOENT) + break; /* * Make sure the directory exists; note that the contents * of the buffer are undefined after mkstemp returns an
@@ -1846,17 +1868,72 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)strbuf_reset (tmp);strbuf_add (tmp, filename, dirlen -1 );if (mkdir (tmp->buf,0777 ) && errno !=EEXIST ) -return -1 ; +break ;if (adjust_shared_perm (tmp->buf)) -return -1 ; +break ;/* Try again */ strbuf_addstr (tmp,"/tmp_obj_XXXXXX" ); fd =git_mkstemp_mode (tmp->buf,0444 ); + }while (0 ); + +if (fd <0 && !(flags &HASH_SILENT )) { +if (errno ==EACCES ) +return error (_ ("insufficient permission for adding an " +"object to repository database %s" ), +get_object_directory ()); +else +return error_errno (_ ("unable to create temporary file" )); } +return fd; } +static int start_loose_object_common (struct strbuf *tmp_file, +const char *filename,unsigned flags, +git_zstream *stream, +unsigned char *buf,size_t buflen, +git_hash_ctx *c, +enum object_type type,size_t len, +char *hdr,int hdrlen) +{ +int fd; + + fd =create_tmpfile (tmp_file, filename, flags); +if (fd <0 ) +return -1 ; + +/* Setup zlib stream for compression */ +git_deflate_init (stream, zlib_compression_level); + stream->next_out = buf; + stream->avail_out = buflen; + the_hash_algo->init_fn (c); + +/* Start to feed header to zlib stream */ + stream->next_in = (unsigned char *)hdr; + stream->avail_in = hdrlen; +while (git_deflate (stream,0 ) ==Z_OK ) + ;/* nothing */ + the_hash_algo->update_fn (c, hdr, hdrlen); + +return fd; +} + +static void end_loose_object_common (int ret,git_hash_ctx *c, +git_zstream *stream, +struct object_id *parano_oid, +const struct object_id *expected_oid, +const char *die_msg1_fmt, +const char *die_msg2_fmt) +{ +if (ret !=Z_STREAM_END ) +die (_ (die_msg1_fmt), ret, expected_oid); + ret =git_deflate_end_gently (stream); +if (ret !=Z_OK ) +die (_ (die_msg2_fmt), ret, expected_oid); + the_hash_algo->final_oid_fn (parano_oid, c); +} +static int write_loose_object (const struct object_id *oid,char *hdr,int hdrlen,const void *buf,unsigned long len,time_t mtime,unsigned flags)
@@ -1871,28 +1948,18 @@ static int write_loose_object(const struct object_id *oid, char *hdr,loose_object_path (the_repository, &filename, oid); - fd =create_tmpfile (&tmp_file, filename.buf); -if (fd <0 ) { -if (flags &HASH_SILENT ) -return -1 ; -else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); -else -return error_errno (_ ("unable to create temporary file" )); - } - -/* Set it up */ -git_deflate_init (&stream, zlib_compression_level); - stream.next_out = compressed; - stream.avail_out =sizeof (compressed); - the_hash_algo->init_fn (&c); - -/* First header.. */ - stream.next_in = (unsigned char *)hdr; - stream.avail_in = hdrlen; -while (git_deflate (&stream,0 ) ==Z_OK ) - ;/* nothing */ - the_hash_algo->update_fn (&c, hdr, hdrlen); +/* Common steps for write_loose_object and stream_loose_object to +* start writing loose oject: +* +* - Create tmpfile for the loose object. +* - Setup zlib stream for compression. +* - Start to feed header to zlib stream. +*/ + fd =start_loose_object_common (&tmp_file, filename.buf, flags, + &stream, compressed,sizeof (compressed), + &c,OBJ_NONE ,0 , hdr, hdrlen); +if (fd <0 ) +return -1 ;/* Then the data itself.. */ stream.next_in = (void *)buf;
@@ -1907,30 +1974,24 @@ static int write_loose_object(const struct object_id *oid, char *hdr, stream.avail_out =sizeof (compressed); }while (ret ==Z_OK ); -if (ret !=Z_STREAM_END ) -die (_ ("unable to deflate new object %s (%d)" ),oid_to_hex (oid), - ret); - ret =git_deflate_end_gently (&stream); -if (ret !=Z_OK ) -die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), - ret); - the_hash_algo->final_oid_fn (¶no_oid, &c); +/* Common steps for write_loose_object and stream_loose_object to +* end writing loose oject: +* +* - End the compression of zlib stream. +* - Get the calculated oid to "parano_oid". +*/ +end_loose_object_common (ret, &c, &stream, ¶no_oid, oid, +N_ ("unable to deflate new object %s (%d)" ), +N_ ("deflateEnd on object %s failed (%d)" )); +if (!oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));close_loose_object (fd); -if (mtime) { -struct utimbuf utb; - utb.actime = mtime; - utb.modtime = mtime; -if (utime (tmp_file.buf, &utb) <0 && - !(flags &HASH_SILENT )) -warning_errno (_ ("failed utime() on %s" ), tmp_file.buf); - } - -return finalize_object_file (tmp_file.buf, filename.buf); +return finalize_object_file_with_mtime (tmp_file.buf, filename.buf, + mtime, flags); }static int freshen_loose_object (const struct object_id *oid)
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
Re: [PATCH v8 2/6] object-file.c: refactor write_loose_object() to several steps
Am 08.01.22 um 09:54 schrieb Han Xin:
Quoted 39 lines
From: Han Xin <[email protected]>
When writing a large blob using "
write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in latter commit to resolve this issue.Before introducing a stream vesion function for writing loose object, do some refactoring on "
write_loose_object()" to reuse code for both versions.Rewrite "
write_loose_object()" as follows:1. Figure out a path for the (temp) object file. This step is only used in "
write_loose_object()".2. Move common steps for starting to write loose objects into a new function "
start_loose_object_common()".3. Compress data.
4. Move common steps for ending zlib stream into a new funciton
"end_loose_object_common()".5. Close fd and finalize the object file.
Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 149 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 105 insertions(+), 44 deletions(-)object-file.c +0
-0
object-file.c static void write_object_file_prepare(const struct git_hash_algo *algo, · line 1752
Quoted 10 lines
object-file.c +7
-0@@ -1743,6 +1743,25 @@ static void write_object_file_prepare(const struct git_hash_algo *algo, algo->final_oid_fn (oid, &c); } +/* +* Move the just written object with proper mtime into its final resting place. +*/ +static int finalize_object_file_with_mtime (const char *tmpfile, +const char *filename, +time_t mtime, +unsigned flags)
This function is called only once after your series. Should it be used by stream_loose_object()? Probably not -- the latter doesn't have a way to force a certain modification time and its caller doesn't need one. So creating finalize_object_file_with_mtime() seems unnecessary for this series.
Quoted 16 lines
object-file.c +12
-0+{ +struct utimbuf utb; + +if (mtime) { + utb.actime = mtime; + utb.modtime = mtime; +if (utime (tmpfile, &utb) <0 && !(flags &HASH_SILENT )) +warning_errno (_ ("failed utime() on %s" ), tmpfile); + } +return finalize_object_file (tmpfile, filename); +} +/* * Move the just written object into its final resting place. */
object-file.c static inline int directory_size(const char *filename) · line 1851
Quoted 6 lines
object-file.c +2
-1@@ -1828,7 +1847,8 @@ static inline int directory_size(const char *filename)* We want to avoid cross-directory filename renames, because those * can have problems on various filesystems (FAT, NFS, Coda). */ -static int create_tmpfile (struct strbuf *tmp,const char *filename) +static int create_tmpfile (struct strbuf *tmp,const char *filename, +unsigned flags)
create_tmpfile() is not mentioned in the commit message, yet it's changed here. Hrm.
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1861
Quoted 11 lines
object-file.c +3
-1{int fd, dirlen =directory_size (filename);@@ -1836,7 +1856,9 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)strbuf_add (tmp, filename, dirlen);strbuf_addstr (tmp,"tmp_obj_XXXXXX" ); fd =git_mkstemp_mode (tmp->buf,0444 ); -if (fd <0 && dirlen && errno ==ENOENT ) { + do { + if (fd >= 0 || !dirlen || errno != ENOENT) + break;
Why turn this branch into a loop? Is this done to mkdir multiple components, e.g. with filename being "a/b/c/file" to create "a", "a/b", and "a/b/c"? It's only used for loose objects, so a fan-out directory (e.g. ".git/objects/ff") can certainly be missing, but can their parent be missing as well sometimes? If that's the point then such a fix would be worth its own patch. (Which probably would benefit from using safe_create_leading_directories()).
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1873
Quoted 12 lines
object-file.c +2
-2/* * Make sure the directory exists; note that the contents * of the buffer are undefined after mkstemp returns an @@ -1846,17 +1868,72 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)strbuf_reset (tmp);strbuf_add (tmp, filename, dirlen -1 );if (mkdir (tmp->buf,0777 ) && errno !=EEXIST ) -return -1 ; +break ;if (adjust_shared_perm (tmp->buf)) -return -1 ; +break ;
Or is it just to replace these returns with a jump to the new error reporting section?
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1877
Quoted 4 lines
object-file.c +0
-0/* Try again */ strbuf_addstr (tmp,"/tmp_obj_XXXXXX" ); fd =git_mkstemp_mode (tmp->buf,0444 );
In that case a break would be missing here.
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1887
Quoted 10 lines
object-file.c +9
-0+ }while (0 ); + +if (fd <0 && !(flags &HASH_SILENT )) { +if (errno ==EACCES ) +return error (_ ("insufficient permission for adding an " +"object to repository database %s" ), +get_object_directory ()); +else +return error_errno (_ ("unable to create temporary file" )); }
Why move this error reporting code into create_tmpfile()? This function has a single caller both before and after your series, so the code could just as well stay at its call-site, avoiding the need to add the flags parameter.
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1897
Quoted 10 lines
object-file.c +7
-0+return fd; } +static int start_loose_object_common(struct strbuf *tmp_file, + const char *filename, unsigned flags, + git_zstream *stream, + unsigned char *buf, size_t buflen, + git_hash_ctx *c, + enum object_type type, size_t len,
The parameters type and len are not used by this function and thus can be dropped.
Quoted 20 lines
object-file.c +20
-0+ char *hdr, int hdrlen) +{ + int fd; + + fd =create_tmpfile (tmp_file, filename, flags); +if (fd <0 ) +return -1 ; + +/* Setup zlib stream for compression */ +git_deflate_init (stream, zlib_compression_level); + stream->next_out = buf; + stream->avail_out = buflen; + the_hash_algo->init_fn (c); + +/* Start to feed header to zlib stream */ + stream->next_in = (unsigned char *)hdr; + stream->avail_in = hdrlen; +while (git_deflate (stream,0 ) ==Z_OK ) + ;/* nothing */ + the_hash_algo->update_fn (c, hdr, hdrlen);
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1927
Quoted 10 lines
object-file.c +10
-0+ +return fd; +} + +static void end_loose_object_common (int ret,git_hash_ctx *c, +git_zstream *stream, +struct object_id *parano_oid, +const struct object_id *expected_oid, +const char *die_msg1_fmt, +const char *die_msg2_fmt)
Hmm, the signature needs as many lines as the function body.
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1933
Quoted 6 lines
object-file.c +6
-0+{ +if (ret !=Z_STREAM_END ) +die (_ (die_msg1_fmt), ret, expected_oid); + ret =git_deflate_end_gently (stream); +if (ret !=Z_OK ) +die (_ (die_msg2_fmt), ret, expected_oid);
These format strings cannot be checked by the compiler.
Considering those two together I think I'd either unify the error messages and move their strings here (losing the ability for users to see if streaming was used) or not extract the function and duplicate its few shared lines. Just a feeling, though.
Quoted 90 lines
object-file.c +27
-40+ the_hash_algo->final_oid_fn (parano_oid, c); +} +static int write_loose_object (const struct object_id *oid,char *hdr,int hdrlen,const void *buf,unsigned long len,time_t mtime,unsigned flags)@@ -1871,28 +1948,18 @@ static int write_loose_object(const struct object_id *oid, char *hdr,loose_object_path (the_repository, &filename, oid); - fd =create_tmpfile (&tmp_file, filename.buf); -if (fd <0 ) { -if (flags &HASH_SILENT ) -return -1 ; -else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); -else -return error_errno (_ ("unable to create temporary file" )); - } - -/* Set it up */ -git_deflate_init (&stream, zlib_compression_level); - stream.next_out = compressed; - stream.avail_out =sizeof (compressed); - the_hash_algo->init_fn (&c); - -/* First header.. */ - stream.next_in = (unsigned char *)hdr; - stream.avail_in = hdrlen; -while (git_deflate (&stream,0 ) ==Z_OK ) - ;/* nothing */ - the_hash_algo->update_fn (&c, hdr, hdrlen); +/* Common steps for write_loose_object and stream_loose_object to +* start writing loose oject: +* +* - Create tmpfile for the loose object. +* - Setup zlib stream for compression. +* - Start to feed header to zlib stream. +*/ + fd =start_loose_object_common (&tmp_file, filename.buf, flags, + &stream, compressed,sizeof (compressed), + &c,OBJ_NONE ,0 , hdr, hdrlen); +if (fd <0 ) +return -1 ;/* Then the data itself.. */ stream.next_in = (void *)buf;@@ -1907,30 +1974,24 @@ static int write_loose_object(const struct object_id *oid, char *hdr, stream.avail_out =sizeof (compressed); }while (ret ==Z_OK ); -if (ret !=Z_STREAM_END ) -die (_ ("unable to deflate new object %s (%d)" ),oid_to_hex (oid), - ret); - ret =git_deflate_end_gently (&stream); -if (ret !=Z_OK ) -die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), - ret); - the_hash_algo->final_oid_fn (¶no_oid, &c); +/* Common steps for write_loose_object and stream_loose_object to +* end writing loose oject: +* +* - End the compression of zlib stream. +* - Get the calculated oid to "parano_oid". +*/ +end_loose_object_common (ret, &c, &stream, ¶no_oid, oid, +N_ ("unable to deflate new object %s (%d)" ), +N_ ("deflateEnd on object %s failed (%d)" )); +if (!oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));close_loose_object (fd); -if (mtime) { -struct utimbuf utb; - utb.actime = mtime; - utb.modtime = mtime; -if (utime (tmp_file.buf, &utb) <0 && - !(flags &HASH_SILENT )) -warning_errno (_ ("failed utime() on %s" ), tmp_file.buf); - } - -return finalize_object_file (tmp_file.buf, filename.buf); +return finalize_object_file_with_mtime (tmp_file.buf, filename.buf, + mtime, flags); }static int freshen_loose_object (const struct object_id *oid)
Re: [PATCH v8 2/6] object-file.c: refactor write_loose_object() to several steps
Quoted 44 lines · René Scharfe
Am 08.01.22 um 09:54 schrieb Han Xin:
Quoted 42 lines
From: Han Xin <[email protected]>
When writing a large blob using "
write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in latter commit to resolve this issue.Before introducing a stream vesion function for writing loose object, do some refactoring on "
write_loose_object()" to reuse code for both versions.Rewrite "
write_loose_object()" as follows:1. Figure out a path for the (temp) object file. This step is only used in "
write_loose_object()".2. Move common steps for starting to write loose objects into a new function "
start_loose_object_common()".3. Compress data.
4. Move common steps for ending zlib stream into a new funciton
"end_loose_object_common()".5. Close fd and finalize the object file.
Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 149 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 105 insertions(+), 44 deletions(-)object-file.c +0
-0@@ -1743,6 +1743,25 @@ static void write_object_file_prepare(const struct git_hash_algo *algo, algo->final_oid_fn (oid, &c); }
Quoted 14 lines
object-file.c static void write_object_file_prepare(const struct git_hash_algo *algo, · line 1752
Quoted 7 lines
object-file.c +7
-0+/* +* Move the just written object with proper mtime into its final resting place. +*/ +static int finalize_object_file_with_mtime (const char *tmpfile, +const char *filename, +time_t mtime, +unsigned flags)This function is called only once after your series. Should it be used by
stream_loose_object()? Probably not -- the latter doesn't have a way to force a certain modification time and its caller doesn't need one. So creatingfinalize_object_file_with_mtime()seems unnecessary for this series.
After accepting the suggestion by Ævar Arnfjörð Bjarmason[1] to remove finalize_object_file_with_mtime() from stream_loose_object() , it seems to be an overkill for write_loose_object() now. I'll put it back into write_loose_object() .
1. https://lore.kernel.org/git/[email protected]/
Thanks
-Han Xin
Quoted 80 lines
Quoted 16 lines
object-file.c +12
-0+{ +struct utimbuf utb; + +if (mtime) { + utb.actime = mtime; + utb.modtime = mtime; +if (utime (tmpfile, &utb) <0 && !(flags &HASH_SILENT )) +warning_errno (_ ("failed utime() on %s" ), tmpfile); + } +return finalize_object_file (tmpfile, filename); +} +/* * Move the just written object into its final resting place. */ object-file.c static inline int directory_size(const char *filename) · line 1851
Quoted 6 lines
object-file.c +2
-1@@ -1828,7 +1847,8 @@ static inline int directory_size(const char *filename)* We want to avoid cross-directory filename renames, because those * can have problems on various filesystems (FAT, NFS, Coda). */ -static int create_tmpfile (struct strbuf *tmp,const char *filename) +static int create_tmpfile (struct strbuf *tmp,const char *filename, +unsigned flags)
create_tmpfile()is not mentioned in the commit message, yet it's changed here. Hrm.object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1861
Quoted 11 lines
object-file.c +3
-1{int fd, dirlen =directory_size (filename);@@ -1836,7 +1856,9 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)strbuf_add (tmp, filename, dirlen);strbuf_addstr (tmp,"tmp_obj_XXXXXX" ); fd =git_mkstemp_mode (tmp->buf,0444 ); -if (fd <0 && dirlen && errno ==ENOENT ) { + do { + if (fd >= 0 || !dirlen || errno != ENOENT) + break;Why turn this branch into a loop? Is this done to mkdir multiple components, e.g. with filename being "a/b/c/file" to create "a", "a/b", and "a/b/c"? It's only used for loose objects, so a fan-out directory (e.g. ".git/objects/ff") can certainly be missing, but can their parent be missing as well sometimes? If that's the point then such a fix would be worth its own patch. (Which probably would benefit from using
safe_create_leading_directories()).object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1873
Quoted 12 lines
object-file.c +2
-2/* * Make sure the directory exists; note that the contents * of the buffer are undefined after mkstemp returns an @@ -1846,17 +1868,72 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)strbuf_reset (tmp);strbuf_add (tmp, filename, dirlen -1 );if (mkdir (tmp->buf,0777 ) && errno !=EEXIST ) -return -1 ; +break ;if (adjust_shared_perm (tmp->buf)) -return -1 ; +break ;Or is it just to replace these returns with a jump to the new error reporting section?
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1877
Quoted 4 lines
object-file.c +0
-0/* Try again */ strbuf_addstr (tmp,"/tmp_obj_XXXXXX" ); fd =git_mkstemp_mode (tmp->buf,0444 );In that case a break would be missing here.
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1887
Quoted 10 lines
object-file.c +9
-0+ }while (0 ); + +if (fd <0 && !(flags &HASH_SILENT )) { +if (errno ==EACCES ) +return error (_ ("insufficient permission for adding an " +"object to repository database %s" ), +get_object_directory ()); +else +return error_errno (_ ("unable to create temporary file" )); }
Quoted 5 lines
Why move this error reporting code into
create_tmpfile()? This function has a single caller both before and after your series, so the code could just as well stay at its call-site, avoiding the need to add the flags parameter.
Here is a legacy from v7, now there is no step called "Figuring out a path for the (temp) object file.", and it's only used in start_loose_object_common(). I will bring it back to what it was.
Thanks
-Han Xin
Quoted 4 lines
Quoted 4 lines
object-file.c +1
-0+return fd; }
Quoted 10 lines
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1897
Quoted 6 lines
object-file.c +6
-0+static int start_loose_object_common(struct strbuf *tmp_file, + const char *filename, unsigned flags, + git_zstream *stream, + unsigned char *buf, size_t buflen, + git_hash_ctx *c, + enum object_type type, size_t len,The parameters type and len are not used by this function and thus can be dropped.
*nod*
Quoted 137 lines
Quoted 20 lines
object-file.c +20
-0+ char *hdr, int hdrlen) +{ + int fd; + + fd =create_tmpfile (tmp_file, filename, flags); +if (fd <0 ) +return -1 ; + +/* Setup zlib stream for compression */ +git_deflate_init (stream, zlib_compression_level); + stream->next_out = buf; + stream->avail_out = buflen; + the_hash_algo->init_fn (c); + +/* Start to feed header to zlib stream */ + stream->next_in = (unsigned char *)hdr; + stream->avail_in = hdrlen; +while (git_deflate (stream,0 ) ==Z_OK ) + ;/* nothing */ + the_hash_algo->update_fn (c, hdr, hdrlen);object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1927
Quoted 10 lines
object-file.c +10
-0+ +return fd; +} + +static void end_loose_object_common (int ret,git_hash_ctx *c, +git_zstream *stream, +struct object_id *parano_oid, +const struct object_id *expected_oid, +const char *die_msg1_fmt, +const char *die_msg2_fmt)Hmm, the signature needs as many lines as the function body.
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1933
Quoted 6 lines
object-file.c +6
-0+{ +if (ret !=Z_STREAM_END ) +die (_ (die_msg1_fmt), ret, expected_oid); + ret =git_deflate_end_gently (stream); +if (ret !=Z_OK ) +die (_ (die_msg2_fmt), ret, expected_oid);These format strings cannot be checked by the compiler.
Considering those two together I think I'd either unify the error messages and move their strings here (losing the ability for users to see if streaming was used) or not extract the function and duplicate its few shared lines. Just a feeling, though.
Quoted 90 lines
object-file.c +27
-40+ the_hash_algo->final_oid_fn (parano_oid, c); +} +static int write_loose_object (const struct object_id *oid,char *hdr,int hdrlen,const void *buf,unsigned long len,time_t mtime,unsigned flags)@@ -1871,28 +1948,18 @@ static int write_loose_object(const struct object_id *oid, char *hdr,loose_object_path (the_repository, &filename, oid); - fd =create_tmpfile (&tmp_file, filename.buf); -if (fd <0 ) { -if (flags &HASH_SILENT ) -return -1 ; -else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); -else -return error_errno (_ ("unable to create temporary file" )); - } - -/* Set it up */ -git_deflate_init (&stream, zlib_compression_level); - stream.next_out = compressed; - stream.avail_out =sizeof (compressed); - the_hash_algo->init_fn (&c); - -/* First header.. */ - stream.next_in = (unsigned char *)hdr; - stream.avail_in = hdrlen; -while (git_deflate (&stream,0 ) ==Z_OK ) - ;/* nothing */ - the_hash_algo->update_fn (&c, hdr, hdrlen); +/* Common steps for write_loose_object and stream_loose_object to +* start writing loose oject: +* +* - Create tmpfile for the loose object. +* - Setup zlib stream for compression. +* - Start to feed header to zlib stream. +*/ + fd =start_loose_object_common (&tmp_file, filename.buf, flags, + &stream, compressed,sizeof (compressed), + &c,OBJ_NONE ,0 , hdr, hdrlen); +if (fd <0 ) +return -1 ;/* Then the data itself.. */ stream.next_in = (void *)buf;@@ -1907,30 +1974,24 @@ static int write_loose_object(const struct object_id *oid, char *hdr, stream.avail_out =sizeof (compressed); }while (ret ==Z_OK ); -if (ret !=Z_STREAM_END ) -die (_ ("unable to deflate new object %s (%d)" ),oid_to_hex (oid), - ret); - ret =git_deflate_end_gently (&stream); -if (ret !=Z_OK ) -die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), - ret); - the_hash_algo->final_oid_fn (¶no_oid, &c); +/* Common steps for write_loose_object and stream_loose_object to +* end writing loose oject: +* +* - End the compression of zlib stream. +* - Get the calculated oid to "parano_oid". +*/ +end_loose_object_common (ret, &c, &stream, ¶no_oid, oid, +N_ ("unable to deflate new object %s (%d)" ), +N_ ("deflateEnd on object %s failed (%d)" )); +if (!oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));close_loose_object (fd); -if (mtime) { -struct utimbuf utb; - utb.actime = mtime; - utb.modtime = mtime; -if (utime (tmp_file.buf, &utb) <0 && - !(flags &HASH_SILENT )) -warning_errno (_ ("failed utime() on %s" ), tmp_file.buf); - } - -return finalize_object_file (tmp_file.buf, filename.buf); +return finalize_object_file_with_mtime (tmp_file.buf, filename.buf, + mtime, flags); }static int freshen_loose_object (const struct object_id *oid)
PATCH v8 3/6 object-file.c: remove the slash for directory_size()
In reply to Han XinFrom: Han Xin <[email protected]>
Since "mkdir foo/" works as well as "mkdir foo", let's remove the end slash as many users of it want.
--- object-file.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
object-file.c +4 -4
@@ -1831,13 +1831,13 @@ static void close_loose_object(int fd)die_errno (_ ("error when closing loose object file" )); } -/* Size of directory component, including the ending '/' */ +/* Size of directory component, excluding the ending '/' */ static inline int directory_size (const char *filename) {const char *s =strrchr (filename,'/' );if (!s)return 0 ; -return s - filename +1 ; +return s - filename; }/*
@@ -1854,7 +1854,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,strbuf_reset (tmp);strbuf_add (tmp, filename, dirlen); -strbuf_addstr (tmp,"tmp_obj_XXXXXX" ); +strbuf_addstr (tmp,"/tmp_obj_XXXXXX" ); fd =git_mkstemp_mode (tmp->buf,0444 ); do { if (fd >= 0 || !dirlen || errno != ENOENT)
@@ -1866,7 +1866,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,* scratch. */ strbuf_reset (tmp); -strbuf_add (tmp, filename, dirlen -1 ); +strbuf_add (tmp, filename, dirlen);if (mkdir (tmp->buf,0777 ) && errno !=EEXIST )break ;if (adjust_shared_perm (tmp->buf))
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
Re: [PATCH v8 3/6] object-file.c: remove the slash for directory_size()
Am 08.01.22 um 09:54 schrieb Han Xin:
Quoted 18 lines
From: Han Xin <[email protected]>
Since "mkdir foo/" works as well as "mkdir foo", let's remove the end slash as many users of it want.
Suggested-by Ævar Arnfjörð Bjarmason <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)object-file.c +0
-0@@ -1831,13 +1831,13 @@ static void close_loose_object(int fd)die_errno (_ ("error when closing loose object file" )); }
object-file.c static void close_loose_object(int fd) · line 1840
Quoted 10 lines
object-file.c +2
-2-/* Size of directory component, including the ending '/' */ +/* Size of directory component, excluding the ending '/' */ static inline int directory_size (const char *filename) {const char *s =strrchr (filename,'/' );if (!s)return 0 ; -return s - filename +1 ; +return s - filename;
This will return zero both for "filename" and "/filename". Hmm. Since it's only used for loose object files we can assume that at least one slash is present, so this removal of functionality is not actually a problem. But I don't understand its benefit.
Quoted 13 lines
object-file.c +1
-1}/* @@ -1854,7 +1854,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,strbuf_reset (tmp);strbuf_add (tmp, filename, dirlen); -strbuf_addstr (tmp,"tmp_obj_XXXXXX" ); +strbuf_addstr (tmp,"/tmp_obj_XXXXXX" ); fd =git_mkstemp_mode (tmp->buf,0444 ); do { if (fd >= 0 || !dirlen || errno != ENOENT)
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename, · line 1870
Quoted 6 lines
object-file.c +1
-1@@ -1866,7 +1866,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,* scratch. */ strbuf_reset (tmp); -strbuf_add (tmp, filename, dirlen -1 ); +strbuf_add (tmp, filename, dirlen);if (mkdir (tmp->buf,0777 ) && errno !=EEXIST )
This code makes sure that mkdir(2) is called without the trailing slash, both with or without this patch. From the commit message above I somehow expected a change in this regard -- but again I wouldn't understand its benefit.
Is this change really needed? Is streaming unpack not possible with the original directory_size() function?
Quoted 2 lines
object-file.c +0
-0break ;if (adjust_shared_perm (tmp->buf))
Re: [PATCH v8 3/6] object-file.c: remove the slash for directory_size()
Quoted 61 lines · René Scharfe
Am 08.01.22 um 09:54 schrieb Han Xin:
Quoted 18 lines
From: Han Xin <[email protected]>
Since "mkdir foo/" works as well as "mkdir foo", let's remove the end slash as many users of it want.
Suggested-by Ævar Arnfjörð Bjarmason <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)object-file.c +0
-0@@ -1831,13 +1831,13 @@ static void close_loose_object(int fd)die_errno (_ ("error when closing loose object file" )); }object-file.c static void close_loose_object(int fd) · line 1840
Quoted 10 lines
object-file.c +2
-2-/* Size of directory component, including the ending '/' */ +/* Size of directory component, excluding the ending '/' */ static inline int directory_size (const char *filename) {const char *s =strrchr (filename,'/' );if (!s)return 0 ; -return s - filename +1 ; +return s - filename;This will return zero both for "filename" and "/filename". Hmm. Since it's only used for loose object files we can assume that at least one slash is present, so this removal of functionality is not actually a problem. But I don't understand its benefit.
Quoted 13 lines
object-file.c +1
-1}/* @@ -1854,7 +1854,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,strbuf_reset (tmp);strbuf_add (tmp, filename, dirlen); -strbuf_addstr (tmp,"tmp_obj_XXXXXX" ); +strbuf_addstr (tmp,"/tmp_obj_XXXXXX" ); fd =git_mkstemp_mode (tmp->buf,0444 ); do { if (fd >= 0 || !dirlen || errno != ENOENT)object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename, · line 1870
Quoted 6 lines
object-file.c +1
-1@@ -1866,7 +1866,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,* scratch. */ strbuf_reset (tmp); -strbuf_add (tmp, filename, dirlen -1 ); +strbuf_add (tmp, filename, dirlen);if (mkdir (tmp->buf,0777 ) && errno !=EEXIST )This code makes sure that mkdir(2) is called without the trailing slash, both with or without this patch. From the commit message above I somehow expected a change in this regard -- but again I wouldn't understand its benefit.
Quoted 3 lines
Is this change really needed? Is streaming unpack not possible with the original
directory_size()function?
*nod*
Streaming unpacking still works with the original directory_size().
This patch is more of a code cleanup that reduces the extra handling of directory size first increasing and then decreasing. I'll seriously consider if I should remove this patch, or move it to the tail.
Thanks
-Han Xin
Quoted 2 lines
Quoted 2 lines
object-file.c +0
-0break ;if (adjust_shared_perm (tmp->buf))
PATCH v8 4/6 object-file.c: add "stream_loose_object()" to handle large object
In reply to Han XinFrom: Han Xin <[email protected]>
If we want unpack and write a loose object using "write_loose_object", we have to feed it with a buffer with the same size of the object, which will consume lots of memory and may cause OOM. This can be improved by feeding data to "stream_loose_object()" in a stream.
Add a new function "stream_loose_object()", which is a stream version of "write_loose_object()" but with a low memory footprint. We will use this function to unpack large blob object in latter commit.
Another difference with "write_loose_object()" is that we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. In "write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
"freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid".
--- object-file.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++++ object-store.h | 9 +++++ 2 files changed, 110 insertions(+)
object-file.c +101 -0
@@ -2012,6 +2012,107 @@ static int freshen_packed_object(const struct object_id *oid)return 1 ; } +int stream_loose_object (struct input_stream *in_stream,size_t len, +struct object_id *oid) +{ +int fd, ret, err =0 , flush =0 ; +unsigned char compressed[4096 ]; +git_zstream stream; +git_hash_ctx c; +struct strbuf tmp_file =STRBUF_INIT ; +struct strbuf filename =STRBUF_INIT ; +int dirlen; +char hdr[MAX_HEADER_LEN ]; +int hdrlen; + +/* Since oid is not determined, save tmp file to odb path. */ +strbuf_addf (&filename,"%s/" ,get_object_directory ()); + hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX,type_name (OBJ_BLOB ), len) +1 ; + +/* Common steps for write_loose_object and stream_loose_object to +* start writing loose oject: +* +* - Create tmpfile for the loose object. +* - Setup zlib stream for compression. +* - Start to feed header to zlib stream. +*/ + fd =start_loose_object_common (&tmp_file, filename.buf,0 , + &stream, compressed,sizeof (compressed), + &c,OBJ_BLOB , len, hdr, hdrlen); +if (fd <0 ) { + err =-1 ; + goto cleanup; + } + +/* Then the data itself.. */ +do { +unsigned char *in0 = stream.next_in; +if (!stream.avail_in && !in_stream->is_finished) { +const void *in = in_stream->read (in_stream, &stream.avail_in); + stream.next_in = (void *)in; + in0 = (unsigned char *)in; +/* All data has been read. */ +if (in_stream->is_finished) + flush =Z_FINISH ; + } + ret =git_deflate (&stream, flush); + the_hash_algo->update_fn (&c, in0, stream.next_in - in0); +if (write_buffer (fd, compressed, stream.next_out - compressed) <0 ) +die (_ ("unable to write loose object file" )); + stream.next_out = compressed; + stream.avail_out =sizeof (compressed); +/* +* Unlike write_loose_object(), we do not have the entire +* buffer. If we get Z_BUF_ERROR due to too few input bytes, +* then we'll replenish them in the next input_stream->read() +* call when we loop. +*/ + }while (ret ==Z_OK || ret ==Z_BUF_ERROR ); + +if (stream.total_in != len + hdrlen) +die (_ ("write stream object %ld != %" PRIuMAX), stream.total_in, + (uintmax_t )len + hdrlen); + +/* Common steps for write_loose_object and stream_loose_object to +* end writing loose oject: +* +* - End the compression of zlib stream. +* - Get the calculated oid. +*/ +end_loose_object_common (ret, &c, &stream, oid,NULL , +N_ ("unable to stream deflate new object (%d)" ), +N_ ("deflateEnd on stream object failed (%d)" )); + +close_loose_object (fd); + +if (freshen_packed_object (oid) ||freshen_loose_object (oid)) { +unlink_or_warn (tmp_file.buf); + goto cleanup; + } + +loose_object_path (the_repository, &filename, oid); + +/* We finally know the object path, and create the missing dir. */ + dirlen =directory_size (filename.buf); +if (dirlen) { +struct strbuf dir =STRBUF_INIT ; +strbuf_add (&dir, filename.buf, dirlen); + +if (mkdir_in_gitdir (dir.buf) && errno !=EEXIST ) { + err =error_errno (_ ("unable to create directory %s" ), dir.buf); +strbuf_release (&dir); + goto cleanup; + } +strbuf_release (&dir); + } + + err =finalize_object_file (tmp_file.buf, filename.buf); +cleanup: +strbuf_release (&tmp_file); +strbuf_release (&filename); +return err; +} +int write_object_file_flags (const void *buf,unsigned long len,const char *type,struct object_id *oid,unsigned flags)
object-store.h +9 -0
@@ -34,6 +34,12 @@ struct object_directory {char *path; }; +struct input_stream { +const void *(*read)(struct input_stream *,unsigned long *len); +void *data; +int is_finished; +}; + KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq)
@@ -232,6 +238,9 @@ static inline int write_object_file(const void *buf, unsigned long len,return write_object_file_flags (buf, len, type, oid,0 ); } +int stream_loose_object (struct input_stream *in_stream,size_t len, +struct object_id *oid); +int hash_object_file_literally (const void *buf,unsigned long len,const char *type,struct object_id *oid,unsigned flags);
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
PATCH v8 5/6 unpack-objects: unpack_non_delta_entry() read data in a stream
In reply to Han XinFrom: Han Xin <[email protected]>
We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.
By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()". However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty.
$ hyperfine \
--setup \
'if ! test -d scalar.git; then git clone --bare
https://github.com/microsoft/scalar.git;
cp scalar.git/objects/pack/*.pack small.pack; fi' \
--prepare 'rm -rf dest.git && git init --bare dest.git' \
... Summary
'./git -C dest.git -c core.bigFileThreshold=512m
unpack-objects <small.pack' in 'origin/master'
1.01 ± 0.04 times faster than './git -C dest.git
-c core.bigFileThreshold=512m unpack-objects
<small.pack' in 'HEAD~1'
1.01 ± 0.04 times faster than './git -C dest.git
-c core.bigFileThreshold=512m unpack-objects
<small.pack' in 'HEAD~0'
1.03 ± 0.10 times faster than './git -C dest.git
-c core.bigFileThreshold=16k unpack-objects
<small.pack' in 'origin/master'
1.02 ± 0.07 times faster than './git -C dest.git
-c core.bigFileThreshold=16k unpack-objects
<small.pack' in 'HEAD~0'
1.10 ± 0.04 times faster than './git -C dest.git
-c core.bigFileThreshold=16k unpack-objects
<small.pack' in 'HEAD~1'Therefore, only unpack objects larger than the "core.bigFileThreshold" in zstream. Until now, the config variable has been used in the following cases, and our new case belongs to the packfile category.
* Archive:
+ archive.c: write_entry(): write large blob entries to archive in stream.
* Loose objects:
+ object-file.c: index_fd(): when hashing large files in worktree, read files in a stream, and create one packfile per large blob if want to save files to git object store.
+ object-file.c: read_loose_object(): when checking loose objects using "git-fsck", do not read full content of large loose objects.
* Packfile:
+ fast-import.c: parse_and_store_blob(): streaming large blob from foreign source to packfile.
+ index-pack.c: check_collison(): read and check large blob in stream.
+ index-pack.c: unpack_entry_data(): do not return the entire contents of the big blob from packfile, but uses a fixed buf to perform some integrity checks on the object.
+ pack-check.c: verify_packfile(): used by "git-fsck" and will call check_object_signature() to check large blob in pack with the streaming interface.
+ pack-objects.c: get_object_details(): set "no_try_delta" for large blobs when counting objects.
+ pack-objects.c: write_no_reuse_object(): streaming large blob to pack.
+ unpack-objects.c: unpack_non_delta_entry(): unpack large blob in stream from packfile.
* Others:
+ diff.c: diff_populate_filespec(): treat large blob file as binary.
+ streaming.c: istream_source(): as a helper of "open_istream()" to select proper streaming interface to read large blob from packfile.
--- builtin/unpack-objects.c | 71 ++++++++++++++++++++++++++++++++- t/t5329-unpack-large-objects.sh | 23 +++++++++-- 2 files changed, 90 insertions(+), 4 deletions(-)
builtin/unpack-objects.c +70 -1
@@ -343,11 +343,80 @@ static void added_object(unsigned nr, enum object_type type, } } +struct input_zstream_data { +git_zstream *zstream; +unsigned char buf[8192 ]; +int status; +}; + +static const void *feed_input_zstream (struct input_stream *in_stream, +unsigned long *readlen) +{ +struct input_zstream_data *data = in_stream->data; +git_zstream *zstream = data->zstream; +void *in =fill (1 ); + +if (in_stream->is_finished) { + *readlen =0 ; +return NULL ; + } + + zstream->next_out = data->buf; + zstream->avail_out =sizeof (data->buf); + zstream->next_in = in; + zstream->avail_in = len; + + data->status =git_inflate (zstream,0 ); + + in_stream->is_finished = data->status !=Z_OK ; +use (len - zstream->avail_in); + *readlen =sizeof (data->buf) - zstream->avail_out; + +return data->buf; +} + +static void write_stream_blob (unsigned nr,size_t size) +{ +git_zstream zstream = {0 }; +struct input_zstream_data data = {0 }; +struct input_stream in_stream = { + .read = feed_input_zstream, + .data = &data, + }; + + data.zstream = &zstream; +git_inflate_init (&zstream); + +if (stream_loose_object (&in_stream, size, &obj_list[nr].oid)) +die (_ ("failed to write object in stream" )); + +if (data.status !=Z_STREAM_END ) +die (_ ("inflate returned (%d)" ), data.status); +git_inflate_end (&zstream); + +if (strict) { +struct blob *blob = +lookup_blob (the_repository, &obj_list[nr].oid); +if (blob) + blob->object.flags |=FLAG_WRITTEN ; +else +die (_ ("invalid blob object from stream" )); + } + obj_list[nr].obj =NULL ; +} +static void unpack_non_delta_entry (enum object_type type,unsigned long size,unsigned nr) { -void *buf =get_data (size); +void *buf; + +/* Write large blob in stream without allocating full buffer. */ +if (!dry_run && type ==OBJ_BLOB && size > big_file_threshold) { +write_stream_blob (nr, size); +return ; + } + buf =get_data (size);if (buf)write_object (nr, type, buf, size); }
t/t5329-unpack-large-objects.sh +20 -3
@@ -9,7 +9,11 @@ test_description='git unpack-objects with large objects' prepare_dest () { test_when_finished "rm -rf dest.git" && - git init --bare dest.git + git init --bare dest.git && + if test -n "$1" + then + git -C dest.git config core.bigFileThreshold $1 + fi } assert_no_loose () {
@@ -37,16 +41,29 @@ test_expect_success 'set memory limitation to 1MB' ' ' test_expect_success 'unpack-objects failed under memory limitation' ' - prepare_dest && + prepare_dest 2m && test_must_fail git -C dest.git unpack-objects <test-$PACK.pack 2>err && grep "fatal: attempting to allocate" err ' test_expect_success 'unpack-objects works with memory limitation in dry-run mode' ' - prepare_dest && + prepare_dest 2m && git -C dest.git unpack-objects -n <test-$PACK.pack && assert_no_loose && assert_no_pack ' +test_expect_success 'unpack big object in stream' ' + prepare_dest 1m && + git -C dest.git unpack-objects <test-$PACK.pack && + assert_no_pack +' + +test_expect_success 'do not unpack existing large objects' ' + prepare_dest 1m && + git -C dest.git index-pack --stdin <test-$PACK.pack && + git -C dest.git unpack-objects <test-$PACK.pack && + assert_no_loose +' + test_done
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
PATCH v8 6/6 object-file API: add a format_object_header() function
In reply to Han XinFrom: Ævar Arnfjörð Bjarmason <[email protected]>
Add a convenience function to wrap the xsnprintf() command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.
All except one caller of it had a valid "enum object_type" for us, it's only write_object_file_prepare() which might need to deal with "git hash-object --literally" and a potential garbage type. Let's have the primary API use an "enum object_type", and define an *_extended() function that can take an arbitrary "const char *" for the type.
See [1] for the discussion that prompted this patch, i.e. new code in object-file.c that wanted to copy/paste the xsnprintf() invocation.
1. https://lore.kernel.org/git/[email protected]/
--- builtin/index-pack.c | 3 +-- bulk-checkin.c | 4 ++-- cache.h | 21 +++++++++++++++++++++ http-push.c | 2 +- object-file.c | 16 ++++++++++++---- 5 files changed, 37 insertions(+), 9 deletions(-)
builtin/index-pack.c +1 -2
@@ -449,8 +449,7 @@ static void *unpack_entry_data(off_t offset, unsigned long size,int hdrlen;if (!is_delta_type (type)) { - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX, -type_name (type),(uintmax_t )size) +1 ; + hdrlen =format_object_header (hdr,sizeof (hdr), type, size); the_hash_algo->init_fn (&c); the_hash_algo->update_fn (&c, hdr, hdrlen); }else
bulk-checkin.c +2 -2
@@ -220,8 +220,8 @@ static int deflate_to_pack(struct bulk_checkin_state *state,if (seekback == (off_t) -1 )return error ("cannot find the current offset" ); - header_len =xsnprintf ((char *)obuf,sizeof (obuf),"%s %" PRIuMAX, -type_name (type), (uintmax_t )size) +1 ; + header_len =format_object_header ((char *)obuf,sizeof (obuf), + type, size); the_hash_algo->init_fn (&ctx); the_hash_algo->update_fn (&ctx, obuf, header_len);
cache.h +21 -0
@@ -1310,6 +1310,27 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,unsigned long bufsiz, struct strbuf *hdrbuf); +/** +* format_object_header() is a thin wrapper around s xsnprintf() that +* writes the initial "<type> <obj-len>" part of the loose object +* header. It returns the size that snprintf() returns + 1. +* +* The format_object_header_extended() function allows for writing a +* type_name that's not one of the "enum object_type" types. This is +* used for "git hash-object --literally". Pass in a OBJ_NONE as the +* type, and a non-NULL "type_str" to do that. +* +* format_object_header() is a convenience wrapper for +* format_object_header_extended(). +*/ +int format_object_header_extended (char *str,size_t size,enum object_type type, +const char *type_str,size_t objsize); +static inline int format_object_header (char *str,size_t size, +enum object_type type,size_t objsize) +{ +return format_object_header_extended (str, size, type,NULL , objsize); +} +/** * parse_loose_header() parses the starting "<type> <len>\0" of an * object. If it doesn't follow that format -1 is returned. To check
http-push.c +1 -1
@@ -363,7 +363,7 @@ static void start_put(struct transfer_request *request)git_zstream stream; unpacked =read_object_file (&request->obj->oid, &type, &len); - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; + hdrlen =format_object_header (hdr,sizeof (hdr), type, len);/* Set it up */ git_deflate_init (&stream, zlib_compression_level);
object-file.c +12 -4
@@ -1006,6 +1006,14 @@ void *xmmap(void *start, size_t length,return ret; } +int format_object_header_extended (char *str,size_t size,enum object_type type, +const char *typestr,size_t objsize) +{ +const char *s = type ==OBJ_NONE ? typestr :type_name (type); + +return xsnprintf (str, size,"%s %" PRIuMAX, s, (uintmax_t )objsize) +1 ; +} +/* * With an in-core object data in "map", rehash it to make sure the * object name actually matches "oid" to detect object corruption.
@@ -1034,7 +1042,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,return -1 ;/* Generate the header */ - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (obj_type), (uintmax_t )size) +1 ; + hdrlen =format_object_header (hdr,sizeof (hdr), obj_type, size);/* Sha1.. */ r->hash_algo->init_fn (&c);
@@ -1734,7 +1742,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,git_hash_ctx c;/* Generate the header */ - *hdrlen =xsnprintf (hdr, *hdrlen,"%s %" PRIuMAX , type, (uintmax_t )len)+1 ; + *hdrlen =format_object_header_extended (hdr, *hdrlen,OBJ_NONE , type, len);/* Sha1.. */ algo->init_fn (&c);
@@ -2027,7 +2035,7 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,/* Since oid is not determined, save tmp file to odb path. */ strbuf_addf (&filename,"%s/" ,get_object_directory ()); - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX,type_name (OBJ_BLOB ), len) +1 ; + hdrlen =format_object_header (hdr,sizeof (hdr),OBJ_BLOB , len);/* Common steps for write_loose_object and stream_loose_object to * start writing loose oject:
@@ -2168,7 +2176,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime) buf =read_object (the_repository, oid, &type, &len);if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid)); - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; + hdrlen =format_object_header (hdr,sizeof (hdr), type, len); ret =write_loose_object (oid, hdr, hdrlen, buf, len, mtime,0 );free (buf);
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
PATCH v6 1/6 object-file.c: release strbuf in write_loose_object()
In reply to Han XinFrom: Han Xin <[email protected]>
Fix a strbuf leak in "write_loose_object()" sugguested by Ævar Arnfjörð Bjarmason.
--- object-file.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
object-file.c +11 -4
@@ -1874,11 +1874,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr, fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {if (flags &HASH_SILENT ) -return -1 ; + ret =-1 ;else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); + ret =error (_ ("insufficient permission for adding an " +"object to repository database %s" ), +get_object_directory ());else -return error_errno (_ ("unable to create temporary file" )); + ret =error_errno (_ ("unable to create temporary file" )); + goto cleanup; }/* Set it up */
@@ -1930,7 +1933,11 @@ static int write_loose_object(const struct object_id *oid, char *hdr,warning_errno (_ ("failed utime() on %s" ), tmp_file.buf); } -return finalize_object_file (tmp_file.buf, filename.buf); + ret =finalize_object_file (tmp_file.buf, filename.buf); +cleanup: +strbuf_release (&filename); +strbuf_release (&tmp_file); +return ret; }static int freshen_loose_object (const struct object_id *oid)
Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
Re: [PATCH v6 1/6] object-file.c: release strbuf in write_loose_object()
Am 17.12.21 um 12:26 schrieb Han Xin:
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr,
Quoted 16 lines
From: Han Xin <[email protected]>
Fix a strbuf leak in "
write_loose_object()" sugguested by Ævar Arnfjörð Bjarmason.Signed-off-by Han Xin <[email protected]>--- object-file.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)object-file.c +0
-0@@ -1874,11 +1874,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
Relevant context lines:
static struct strbuf tmp_file = STRBUF_INIT; static struct strbuf filename = STRBUF_INIT; loose_object_path(the_repository, &filename, oid);
Quoted 18 lines
object-file.c +6
-3fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {if (flags &HASH_SILENT ) -return -1 ; + ret =-1 ;else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); + ret =error (_ ("insufficient permission for adding an " +"object to repository database %s" ), +get_object_directory ());else -return error_errno (_ ("unable to create temporary file" )); + ret =error_errno (_ ("unable to create temporary file" )); + goto cleanup; }/* Set it up */
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1939
Quoted 8 lines
object-file.c +4
-1@@ -1930,7 +1933,11 @@ static int write_loose_object(const struct object_id *oid, char *hdr,warning_errno (_ ("failed utime() on %s" ), tmp_file.buf); } -return finalize_object_file (tmp_file.buf, filename.buf); + ret =finalize_object_file (tmp_file.buf, filename.buf); +cleanup: +strbuf_release (&filename); +strbuf_release (&tmp_file);
There was no leak before. Both strbufs are static and both functions they are passed to (loose_object_path() and create_tmpfile()) reset them first. So while the allocated memory was not released before, it was reused.
Not sure if making write_loose_object() allocate and release these buffers on every call has much of a performance impact. The only reason I can think of for wanting such a change is to get rid of the static buffers, to allow the function to be used by concurrent threads.
So I think either keeping the code as-is or also making the strbufs non-static would be better (but then discussing a possible performance impact in the commit message would be nice).
Quoted 4 lines
object-file.c +1
-0+return ret; }static int freshen_loose_object (const struct object_id *oid)
Re: [PATCH v6 1/6] object-file.c: release strbuf in write_loose_object()
Quoted 11 lines · René Scharfe
There was no leak before. Both strbufs are static and both functions they are passed to (
loose_object_path()andcreate_tmpfile()) reset them first. So while the allocated memory was not released before, it was reused.Not sure if making
write_loose_object()allocate and release these buffers on every call has much of a performance impact. The only reason I can think of for wanting such a change is to get rid of the static buffers, to allow the function to be used by concurrent threads.
Quoted 3 lines
So I think either keeping the code as-is or also making the strbufs non-static would be better (but then discussing a possible performance impact in the commit message would be nice).
Makes sense.
PATCH v6 2/6 object-file.c: refactor object header generation into a function
In reply to Han XinFrom: Han Xin <[email protected]>
There are 3 places where "xsnprintf" is used to generate the object header, and I originally planned to add a fourth in the latter patch.
According to Ævar Arnfjörð Bjarmason’s suggestion, although it's just one line, it's also code that's very central to git, so reafactor them into a function which will help later readability.
--- object-file.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
object-file.c +11 -3
@@ -1006,6 +1006,14 @@ void *xmmap(void *start, size_t length,return ret; } +static inline int generate_object_header (char *buf,int bufsz, +const char *type_name, +unsigned long size) +{ +return xsnprintf (buf, bufsz,"%s %" PRIuMAX, type_name, + (uintmax_t )size) +1 ; +} +/* * With an in-core object data in "map", rehash it to make sure the * object name actually matches "oid" to detect object corruption.
@@ -1034,7 +1042,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,return -1 ;/* Generate the header */ - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (obj_type), (uintmax_t )size) +1 ; + hdrlen =generate_object_header (hdr,sizeof (hdr),type_name (obj_type), size);/* Sha1.. */ r->hash_algo->init_fn (&c);
@@ -1734,7 +1742,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,git_hash_ctx c;/* Generate the header */ - *hdrlen =xsnprintf (hdr, *hdrlen,"%s %" PRIuMAX , type, (uintmax_t )len)+1 ; + *hdrlen =generate_object_header (hdr, *hdrlen, type, len);/* Sha1.. */ algo->init_fn (&c);
@@ -2013,7 +2021,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime) buf =read_object (the_repository, oid, &type, &len);if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid)); - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; + hdrlen =generate_object_header (hdr,sizeof (hdr),type_name (type), len); ret =write_loose_object (oid, hdr, hdrlen, buf, len, mtime,0 );free (buf);
Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
RFC PATCH 1/1 object-file API: add a format_loose_header() function
Add a convenience function to wrap the xsnprintf() command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.
All except one caller of it had a valid "enum object_type" for us, it's only write_object_file_prepare() which might need to deal with "git hash-object --literally" and a potential garbage type. Let's have the primary API use an "enum object_type", and define an *_extended() function that can take an arbitrary "const char *" for the type.
See [1] for the discussion that prompted this patch, i.e. new code in object-file.c that wanted to copy/paste the xsnprintf() invocation.
1. https://lore.kernel.org/git/[email protected]/
---
Quoted 11 lines · Han Xin
From: Han Xin <[email protected]>
There are 3 places where "xsnprintf" is used to generate the object header, and I originally planned to add a fourth in the latter patch.
According to Ævar Arnfjörð Bjarmason’s suggestion, although it's just one line, it's also code that's very central to git, so reafactor them into a function which will help later readability.
Signed-off-by Han Xin <[email protected]>
I came up with this after my comment on the earlier round suggesting to factor out that header formatting. I don't know if this more thorough approach is worth it or if you'd like to replace your change with this one, but just posting it here as an RFC.
builtin/index-pack.c | 3 +-- bulk-checkin.c | 4 ++-- cache.h | 21 +++++++++++++++++++++ http-push.c | 2 +- object-file.c | 14 +++++++++++--- 5 files changed, 36 insertions(+), 8 deletions(-)
builtin/index-pack.c +1 -2
@@ -449,8 +449,7 @@ static void *unpack_entry_data(off_t offset, unsigned long size,int hdrlen;if (!is_delta_type (type)) { - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX, -type_name (type),(uintmax_t )size) +1 ; + hdrlen =format_loose_header (hdr,sizeof (hdr), type, (uintmax_t )size); the_hash_algo->init_fn (&c); the_hash_algo->update_fn (&c, hdr, hdrlen); }else
bulk-checkin.c +2 -2
@@ -220,8 +220,8 @@ static int deflate_to_pack(struct bulk_checkin_state *state,if (seekback == (off_t) -1 )return error ("cannot find the current offset" ); - header_len =xsnprintf ((char *)obuf,sizeof (obuf),"%s %" PRIuMAX, -type_name (type), (uintmax_t )size) +1 ; + header_len =format_loose_header ((char *)obuf,sizeof (obuf), + type, (uintmax_t )size); the_hash_algo->init_fn (&ctx); the_hash_algo->update_fn (&ctx, obuf, header_len);
cache.h +21 -0
@@ -1309,6 +1309,27 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,unsigned long bufsiz, struct strbuf *hdrbuf); +/** +* format_loose_header() is a thin wrapper around s xsnprintf() that +* writes the initial "<type> <obj-len>" part of the loose object +* header. It returns the size that snprintf() returns + 1. +* +* The format_loose_header_extended() function allows for writing a +* type_name that's not one of the "enum object_type" types. This is +* used for "git hash-object --literally". Pass in a OBJ_NONE as the +* type, and a non-NULL "type_str" to do that. +* +* format_loose_header() is a convenience wrapper for +* format_loose_header_extended(). +*/ +int format_loose_header_extended (char *str,size_t size,enum object_type type, +const char *type_str,size_t objsize); +static inline int format_loose_header (char *str,size_t size, +enum object_type type,size_t objsize) +{ +return format_loose_header_extended (str, size, type,NULL , objsize); +} +/** * parse_loose_header() parses the starting "<type> <len>\0" of an * object. If it doesn't follow that format -1 is returned. To check
http-push.c +1 -1
@@ -363,7 +363,7 @@ static void start_put(struct transfer_request *request)git_zstream stream; unpacked =read_object_file (&request->obj->oid, &type, &len); - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; + hdrlen =format_loose_header (hdr,sizeof (hdr), type, (uintmax_t )len);/* Set it up */ git_deflate_init (&stream, zlib_compression_level);
object-file.c +11 -3
@@ -1009,6 +1009,14 @@ void *xmmap(void *start, size_t length,return ret; } +int format_loose_header_extended (char *str,size_t size,enum object_type type, +const char *typestr,size_t objsize) +{ +const char *s = type ==OBJ_NONE ? typestr :type_name (type); + +return xsnprintf (str, size,"%s %" PRIuMAX, s, (uintmax_t )objsize) +1 ; +} +/* * With an in-core object data in "map", rehash it to make sure the * object name actually matches "oid" to detect object corruption.
@@ -1037,7 +1045,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,return -1 ;/* Generate the header */ - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (obj_type), (uintmax_t )size) +1 ; + hdrlen =format_loose_header (hdr,sizeof (hdr), obj_type, size);/* Sha1.. */ r->hash_algo->init_fn (&c);
@@ -1737,7 +1745,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,git_hash_ctx c;/* Generate the header */ - *hdrlen =xsnprintf (hdr, *hdrlen,"%s %" PRIuMAX , type, (uintmax_t )len)+1 ; + *hdrlen =format_loose_header_extended (hdr, *hdrlen,OBJ_NONE , type, len);/* Sha1.. */ algo->init_fn (&c);
@@ -2009,7 +2017,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime) buf =read_object (the_repository, oid, &type, &len);if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid)); - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; + hdrlen =format_loose_header (hdr,sizeof (hdr), type, len); ret =write_loose_object (oid, hdr, hdrlen, buf, len, mtime,0 );free (buf);
Signature · 1 line
2.34.1.1119.g606023410ba
Re: [RFC PATCH] object-file API: add a format_loose_header() function
Hi Ævar,
(catching up after a week away, and noticed your patch today..)
Quoted 10 lines · Ævar Arnfjörð Bjarmason
Add a convenience function to wrap the
xsnprintf()command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.All except one caller of it had a valid "enum
object_type" for us, it's onlywrite_object_file_prepare()which might need to deal with "git hash-object --literally" and a potential garbage type. Let's have the primary API use an "enumobject_type", and define an*_extended()function that can take an arbitrary "const char *" for the type.
I recently completed a PR in the Git for Windows build that is focused on "git hash-object --literally" as a starter for LLP64 large file (>4GB) compatibility.
(https://github.com/git-for-windows/git/pull/3533), which Dscho has merged (cc'd).
I'm not sure that the extended version will work as expected across the test suite
as multiple fake object types are tried, though I only skimmed the patch.
I'd support the general thrust, but just wanted to synchronise any changes.
Philip
Quoted 154 lines
See [1] for the discussion that prompted this patch, i.e. new code in
object-file.cthat wanted to copy/paste thexsnprintf()invocation.1. https://lore.kernel.org/git/[email protected]/
Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>---
Quoted 11 lines · Han Xin
From: Han Xin <[email protected]>
There are 3 places where "xsnprintf" is used to generate the object header, and I originally planned to add a fourth in the latter patch.
According to Ævar Arnfjörð Bjarmason’s suggestion, although it's just one line, it's also code that's very central to git, so reafactor them into a function which will help later readability.
Signed-off-by Han Xin <[email protected]>I came up with this after my comment on the earlier round suggesting to factor out that header formatting. I don't know if this more thorough approach is worth it or if you'd like to replace your change with this one, but just posting it here as an RFC.
builtin/index-pack.c | 3 +-- bulk-checkin.c | 4 ++-- cache.h | 21 +++++++++++++++++++++ http-push.c | 2 +- object-file.c | 14 +++++++++++--- 5 files changed, 36 insertions(+), 8 deletions(-)builtin/index-pack.c +1
-2@@ -449,8 +449,7 @@ static void *unpack_entry_data(off_t offset, unsigned long size,int hdrlen;if (!is_delta_type (type)) { - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX, -type_name (type),(uintmax_t )size) +1 ; + hdrlen =format_loose_header (hdr,sizeof (hdr), type, (uintmax_t )size); the_hash_algo->init_fn (&c); the_hash_algo->update_fn (&c, hdr, hdrlen); }else bulk-checkin.c +2
-2@@ -220,8 +220,8 @@ static int deflate_to_pack(struct bulk_checkin_state *state,if (seekback == (off_t) -1 )return error ("cannot find the current offset" ); - header_len =xsnprintf ((char *)obuf,sizeof (obuf),"%s %" PRIuMAX, -type_name (type), (uintmax_t )size) +1 ; + header_len =format_loose_header ((char *)obuf,sizeof (obuf), + type, (uintmax_t )size); the_hash_algo->init_fn (&ctx); the_hash_algo->update_fn (&ctx, obuf, header_len);cache.h +21
-0@@ -1309,6 +1309,27 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,unsigned long bufsiz, struct strbuf *hdrbuf); +/** +* format_loose_header() is a thin wrapper around s xsnprintf() that +* writes the initial "<type> <obj-len>" part of the loose object +* header. It returns the size that snprintf() returns + 1. +* +* The format_loose_header_extended() function allows for writing a +* type_name that's not one of the "enum object_type" types. This is +* used for "git hash-object --literally". Pass in a OBJ_NONE as the +* type, and a non-NULL "type_str" to do that. +* +* format_loose_header() is a convenience wrapper for +* format_loose_header_extended(). +*/ +int format_loose_header_extended (char *str,size_t size,enum object_type type, +const char *type_str,size_t objsize); +static inline int format_loose_header (char *str,size_t size, +enum object_type type,size_t objsize) +{ +return format_loose_header_extended (str, size, type,NULL , objsize); +} +/** * parse_loose_header() parses the starting "<type> <len>\0" of an * object. If it doesn't follow that format -1 is returned. To check http-push.c +1
-1@@ -363,7 +363,7 @@ static void start_put(struct transfer_request *request)git_zstream stream; unpacked =read_object_file (&request->obj->oid, &type, &len); - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; + hdrlen =format_loose_header (hdr,sizeof (hdr), type, (uintmax_t )len);/* Set it up */ git_deflate_init (&stream, zlib_compression_level);object-file.c +11
-3@@ -1009,6 +1009,14 @@ void *xmmap(void *start, size_t length,return ret; } +int format_loose_header_extended (char *str,size_t size,enum object_type type, +const char *typestr,size_t objsize) +{ +const char *s = type ==OBJ_NONE ? typestr :type_name (type); + +return xsnprintf (str, size,"%s %" PRIuMAX, s, (uintmax_t )objsize) +1 ; +} +/* * With an in-core object data in "map", rehash it to make sure the * object name actually matches "oid" to detect object corruption. @@ -1037,7 +1045,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,return -1 ;/* Generate the header */ - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (obj_type), (uintmax_t )size) +1 ; + hdrlen =format_loose_header (hdr,sizeof (hdr), obj_type, size);/* Sha1.. */ r->hash_algo->init_fn (&c);@@ -1737,7 +1745,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,git_hash_ctx c;/* Generate the header */ - *hdrlen =xsnprintf (hdr, *hdrlen,"%s %" PRIuMAX , type, (uintmax_t )len)+1 ; + *hdrlen =format_loose_header_extended (hdr, *hdrlen,OBJ_NONE , type, len);/* Sha1.. */ algo->init_fn (&c);@@ -2009,7 +2017,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime) buf =read_object (the_repository, oid, &type, &len);if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid)); - hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; + hdrlen =format_loose_header (hdr,sizeof (hdr), type, len); ret =write_loose_object (oid, hdr, hdrlen, buf, len, mtime,0 );free (buf);
Re: [RFC PATCH] object-file API: add a format_loose_header() function
In reply to Ævar Arnfjörð BjarmasonQuoted 7 lines · Ævar Arnfjörð Bjarmason
Add a convenience function to wrap the
xsnprintf()command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.
...
+/**
+ *format_loose_header()is a thin wrapper around sxsnprintf()that
The name should have "object" somewhere in it. Not all readers can be expected to know that you meant "loose" to be an acceptable short hand for "loose object".
That nit aside, I think it is a good idea to give people a common helper function to call. I am undecided if it is a good idea to make it take enum or "const char *"; most everybody should be able to say
format_object_header(type_name(OBJ_COMMIT), ...)
just fine, so two variants might be overkill, just to allow
format_object_header(OBJ_COMMIT, ...)
and to forbid
format_object_header("connit", ...)I dunno.
Re: [RFC PATCH] object-file API: add a format_loose_header() function
Quoted 10 lines · Junio C Hamano
Quoted 7 lines · Ævar Arnfjörð Bjarmason
Add a convenience function to wrap the
xsnprintf()command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.
...
+/**
+ *format_loose_header()is a thin wrapper around sxsnprintf()that
Quoted 3 lines
The name should have "object" somewhere in it. Not all readers can be expected to know that you meant "loose" to be an acceptable short hand for "loose object".
*nod*
Quoted 9 lines
That nit aside, I think it is a good idea to give people a common helper function to call. I am undecided if it is a good idea to make it take enum or "const char *"; most everybody should be able to say
format_object_header(type_name(OBJ_COMMIT), ...)just fine, so two variants might be overkill, just to allow
Quoted 7 lines
format_object_header(OBJ_COMMIT, ...)and to forbid
format_object_header("connit", ...)I dunno.
Ultimately only a single API caller in hash-object.c really cares about something else than the enum.
I've got some patches locally to convert e.g. write_object_file() to use the enum, and it removes the need for some callers to convert enum to char *, only to have other things convert it back.
So I think for any new APIs it makes sense to work towards sidelining the hash-object.c --literally caller.
Re: [RFC PATCH] object-file API: add a format_loose_header() function
Quoted 6 lines · Ævar Arnfjörð Bjarmason
I've got some patches locally to convert e.g.
write_object_file()to use the enum, and it removes the need for some callers to convert enum to char *, only to have other things convert it back.So I think for any new APIs it makes sense to work towards sidelining the
hash-object.c--literally caller.
Your logic is backwards to argue "because I did something this way, it makes sense to do it this way"?
Re: [RFC PATCH] object-file API: add a format_loose_header() function
Quoted 11 lines · Junio C Hamano
Quoted 6 lines · Ævar Arnfjörð Bjarmason
I've got some patches locally to convert e.g.
write_object_file()to use the enum, and it removes the need for some callers to convert enum to char *, only to have other things convert it back.So I think for any new APIs it makes sense to work towards sidelining the
hash-object.c--literally caller.Your logic is backwards to argue "because I did something this way, it makes sense to do it this way"?
No, it's that if you look at the write_object_file() and hash_object_file() callers in-tree now many, including in object-file.c itself are taking an "enum object_type" only to convert it to a string, and then we'll in turn sometimes convert that to the "enum object_type" again at some lower level.
That API inconsistency dates back to at least Linus's a733cb606fe (Change pack file format. Hopefully for the last time., 2005-06-28).
I'm just pointing out that I have local patches that prove that a lot of back & forth is done for no good reason, and that this is one of the codepaths that's tangentally involved. So it makes sense in this case to make any new API take "enum object_type" as the primary interface.
Re: [RFC PATCH] object-file API: add a format_loose_header() function
In reply to Ævar Arnfjörð BjarmasonQuoted 5 lines · Ævar Arnfjörð Bjarmason
I came up with this after my comment on the earlier round suggesting to factor out that header formatting. I don't know if this more thorough approach is worth it or if you'd like to replace your change with this one, but just posting it here as an RFC.
I will take this patch and rename the function name from "format_loose_header()" to "format_object_header()".
Thanks
-Han Xin
PATCH v6 3/6 object-file.c: refactor write_loose_object() to reuse in stream version
In reply to Han XinFrom: Han Xin <[email protected]>
We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.
This can be improved by feeding data to "stream_loose_object()" in stream instead of read into the whole buf.
As this new method "stream_loose_object()" has many similarities with "write_loose_object()", we split up "write_loose_object()" into some steps:
1. Figuring out a path for the (temp) object file.
2. Creating the tempfile.
3. Setting up zlib and write header.
4. Write object data and handle errors.
5. Optionally, do someting after write, maybe force a loose object if
"mtime".
--- object-file.c | 98 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 63 insertions(+), 35 deletions(-)
object-file.c +63 -35
@@ -1751,6 +1751,25 @@ static void write_object_file_prepare(const struct git_hash_algo *algo, algo->final_oid_fn (oid, &c); } +/* +* Move the just written object with proper mtime into its final resting place. +*/ +static int finalize_object_file_with_mtime (const char *tmpfile, +const char *filename, +time_t mtime, +unsigned flags) +{ +struct utimbuf utb; + +if (mtime) { + utb.actime = mtime; + utb.modtime = mtime; +if (utime (tmpfile, &utb) <0 && !(flags &HASH_SILENT )) +warning_errno (_ ("failed utime() on %s" ), tmpfile); + } +return finalize_object_file (tmpfile, filename); +} +/* * Move the just written object into its final resting place. */
@@ -1836,7 +1855,8 @@ static inline int directory_size(const char *filename)* We want to avoid cross-directory filename renames, because those * can have problems on various filesystems (FAT, NFS, Coda). */ -static int create_tmpfile (struct strbuf *tmp,const char *filename) +static int create_tmpfile (struct strbuf *tmp,const char *filename, +unsigned flags) {int fd, dirlen =directory_size (filename);
@@ -1844,7 +1864,9 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)strbuf_add (tmp, filename, dirlen);strbuf_addstr (tmp,"tmp_obj_XXXXXX" ); fd =git_mkstemp_mode (tmp->buf,0444 ); -if (fd <0 && dirlen && errno ==ENOENT ) { + do { + if (fd >= 0 || !dirlen || errno != ENOENT) + break; /* * Make sure the directory exists; note that the contents * of the buffer are undefined after mkstemp returns an
@@ -1854,17 +1876,48 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)strbuf_reset (tmp);strbuf_add (tmp, filename, dirlen -1 );if (mkdir (tmp->buf,0777 ) && errno !=EEXIST ) -return -1 ; +break ;if (adjust_shared_perm (tmp->buf)) -return -1 ; +break ;/* Try again */ strbuf_addstr (tmp,"/tmp_obj_XXXXXX" ); fd =git_mkstemp_mode (tmp->buf,0444 ); + }while (0 ); + +if (fd <0 && !(flags &HASH_SILENT )) { +if (errno ==EACCES ) +return error (_ ("insufficient permission for adding an " +"object to repository database %s" ), +get_object_directory ()); +else +return error_errno (_ ("unable to create temporary file" )); } +return fd; } +static void setup_stream_and_header (git_zstream *stream, +unsigned char *compressed, +unsigned long compressed_size, +git_hash_ctx *c, +char *hdr, +int hdrlen) +{ +/* Set it up */ +git_deflate_init (stream, zlib_compression_level); + stream->next_out = compressed; + stream->avail_out = compressed_size; + the_hash_algo->init_fn (c); + +/* First header.. */ + stream->next_in = (unsigned char *)hdr; + stream->avail_in = hdrlen; +while (git_deflate (stream,0 ) ==Z_OK ) + ;/* nothing */ + the_hash_algo->update_fn (c, hdr, hdrlen); +} +static int write_loose_object (const struct object_id *oid,char *hdr,int hdrlen,const void *buf,unsigned long len,time_t mtime,unsigned flags)
@@ -1879,31 +1932,15 @@ static int write_loose_object(const struct object_id *oid, char *hdr,loose_object_path (the_repository, &filename, oid); - fd =create_tmpfile (&tmp_file, filename.buf); + fd =create_tmpfile (&tmp_file, filename.buf, flags);if (fd <0 ) { -if (flags &HASH_SILENT ) - ret =-1 ; -else if (errno ==EACCES ) - ret =error (_ ("insufficient permission for adding an " -"object to repository database %s" ), -get_object_directory ()); -else - ret =error_errno (_ ("unable to create temporary file" )); + ret =-1 ; goto cleanup; } -/* Set it up */ -git_deflate_init (&stream, zlib_compression_level); - stream.next_out = compressed; - stream.avail_out =sizeof (compressed); - the_hash_algo->init_fn (&c); - -/* First header.. */ - stream.next_in = (unsigned char *)hdr; - stream.avail_in = hdrlen; -while (git_deflate (&stream,0 ) ==Z_OK ) - ;/* nothing */ - the_hash_algo->update_fn (&c, hdr, hdrlen); +/* Set it up and write header */ +setup_stream_and_header (&stream, compressed,sizeof (compressed), + &c, hdr, hdrlen);/* Then the data itself.. */ stream.next_in = (void *)buf;
@@ -1932,16 +1969,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,close_loose_object (fd); -if (mtime) { -struct utimbuf utb; - utb.actime = mtime; - utb.modtime = mtime; -if (utime (tmp_file.buf, &utb) <0 && - !(flags &HASH_SILENT )) -warning_errno (_ ("failed utime() on %s" ), tmp_file.buf); - } - - ret =finalize_object_file (tmp_file.buf, filename.buf); + ret =finalize_object_file_with_mtime (tmp_file.buf, filename.buf, mtime, flags); cleanup:strbuf_release (&filename);strbuf_release (&tmp_file);
Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
PATCH v6 4/6 object-file.c: make "write_object_file_flags()" to support read in stream
In reply to Han XinFrom: Han Xin <[email protected]>
We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.
This can be improved by feeding data to "stream_loose_object()" in a stream. The input stream is implemented as an interface.
When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in a new function called "stream_loose_object()".
In "write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
We will reuse "write_object_file_flags()" in "unpack_non_delta_entry()" to read the entire data contents in stream, so a new flag "HASH_STREAM" is added. When read in stream, we needn't prepare the "oid" before "write_loose_object()", only generate the header. "freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid".
--- cache.h | 1 + object-file.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++ object-store.h | 5 +++ 3 files changed, 98 insertions(+)
cache.h +1 -0
@@ -898,6 +898,7 @@ int ie_modified(struct index_state *, const struct cache_entry *, struct stat *,#define HASH_FORMAT_CHECK 2#define HASH_RENORMALIZE 4#define HASH_SILENT 8 +#define HASH_STREAM 16int index_fd (struct index_state *istate,struct object_id *oid,int fd,struct stat *st,enum object_type type,const char *path,unsigned flags);int index_path (struct index_state *istate,struct object_id *oid,const char *path,struct stat *st,unsigned flags);
object-file.c +92 -0
104 lines not on this page · raw message
object-store.h +5 -0
@@ -34,6 +34,11 @@ struct object_directory {char *path; }; +struct input_stream { +const void *(*read)(const struct input_stream *,unsigned long *len); +void *data; +}; + KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq)
Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
Re: [PATCH v6 4/6] object-file.c: make "write_object_file_flags()" to support read in stream
Am 17.12.21 um 12:26 schrieb Han Xin:
Quoted 57 lines
From: Han Xin <[email protected]>
We used to call "
get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.This can be improved by feeding data to "
stream_loose_object()" in a stream. The input stream is implemented as an interface.When streaming a large blob object to "
write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in a new function called "stream_loose_object()".In "
write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.We will reuse "
write_object_file_flags()" in "unpack_non_delta_entry()" to read the entire data contents in stream, so a new flag "HASH_STREAM" is added. When read in stream, we needn't prepare the "oid" before "write_loose_object()", only generate the header. "freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid".Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- cache.h | 1 + object-file.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++ object-store.h | 5 +++ 3 files changed, 98 insertions(+)cache.h +1
-0@@ -898,6 +898,7 @@ int ie_modified(struct index_state *, const struct cache_entry *, struct stat *,#define HASH_FORMAT_CHECK 2#define HASH_RENORMALIZE 4#define HASH_SILENT 8 +#define HASH_STREAM 16int index_fd (struct index_state *istate,struct object_id *oid,int fd,struct stat *st,enum object_type type,const char *path,unsigned flags);int index_path (struct index_state *istate,struct object_id *oid,const char *path,struct stat *st,unsigned flags);object-file.c +1
-0@@ -1994,6 +1994,88 @@ static int freshen_packed_object(const struct object_id *oid)return 1 ; } +static int stream_loose_object(struct object_id *oid, char *hdr, int hdrlen,
object-file.c static int freshen_packed_object(const struct object_id *oid) · line 2007
Quoted 10 lines
object-file.c +10
-0+const struct input_stream *in_stream, + unsigned long len, time_t mtime, unsigned flags) +{ + int fd, ret, err = 0, flush = 0; +unsigned char compressed[4096 ]; +git_zstream stream; +git_hash_ctx c; +struct object_id parano_oid; +static struct strbuf tmp_file =STRBUF_INIT ; +static struct strbuf filename =STRBUF_INIT ;
Note these static strbufs.
Quoted 58 lines
object-file.c +58
-0+int dirlen; + +/* When oid is not determined, save tmp file to odb path. */ +strbuf_addf (&filename,"%s/" ,get_object_directory ()); + + fd =create_tmpfile (&tmp_file, filename.buf, flags); +if (fd <0 ) { + err =-1 ; + goto cleanup; + } + +/* Set it up and write header */ +setup_stream_and_header (&stream, compressed,sizeof (compressed), + &c, hdr, hdrlen); + +/* Then the data itself.. */ +do { +unsigned char *in0 = stream.next_in; +if (!stream.avail_in) { +const void *in = in_stream->read (in_stream, &stream.avail_in); + stream.next_in = (void *)in; + in0 = (unsigned char *)in; +/* All data has been read. */ +if (len + hdrlen == stream.total_in + stream.avail_in) + flush =Z_FINISH ; + } + ret =git_deflate (&stream, flush); + the_hash_algo->update_fn (&c, in0, stream.next_in - in0); +if (write_buffer (fd, compressed, stream.next_out - compressed) <0 ) +die (_ ("unable to write loose object file" )); + stream.next_out = compressed; + stream.avail_out =sizeof (compressed); + }while (ret ==Z_OK || ret ==Z_BUF_ERROR ); + +if (ret !=Z_STREAM_END ) +die (_ ("unable to deflate new object streamingly (%d)" ), ret); + ret =git_deflate_end_gently (&stream); +if (ret !=Z_OK ) +die (_ ("deflateEnd on object streamingly failed (%d)" ), ret); + the_hash_algo->final_oid_fn (¶no_oid, &c); + +close_loose_object (fd); + +oidcpy (oid, ¶no_oid); + +if (freshen_packed_object (oid) ||freshen_loose_object (oid)) { +unlink_or_warn (tmp_file.buf); + goto cleanup; + } + +loose_object_path (the_repository, &filename, oid); + +/* We finally know the object path, and create the missing dir. */ + dirlen =directory_size (filename.buf); +if (dirlen) { +struct strbuf dir =STRBUF_INIT ; +strbuf_add (&dir, filename.buf, dirlen -1 ); +
object-file.c static int freshen_packed_object(const struct object_id *oid) · line 2075
Quoted 10 lines
object-file.c +10
-0+if (mkdir_in_gitdir (dir.buf) <0 ) { + err =-1 ; + goto cleanup; + } + } + + err =finalize_object_file_with_mtime (tmp_file.buf, filename.buf, mtime, flags); +cleanup: +strbuf_release (&tmp_file); +strbuf_release (&filename);
The static strbufs are released here. That combination is strange -- why keep the variable values between calls by making them static, but throw away the allocated buffers instead of reusing them?
Given that this function is only used for huge objects I think making the strbufs non-static and releasing them is the best choice here.
Quoted 9 lines
object-file.c +3
-0+return err; +} +int write_object_file_flags (const void *buf,unsigned long len,const char *type,struct object_id *oid,unsigned flags)@@ -2001,6 +2083,16 @@ int write_object_file_flags(const void *buf, unsigned long len,char hdr[MAX_HEADER_LEN ];int hdrlen =sizeof (hdr);
object-file.c int write_object_file_flags(const void *buf, unsigned long len, · line 2094
Quoted 10 lines
object-file.c +9
-0+/* When streaming a large blob object (marked as HASH_STREAM), +* we have no chance to run "write_object_file_prepare()" to +* calculate the "oid" in advance. Call "stream_loose_object()" +* to write loose object in stream. +*/ +if (flags &HASH_STREAM ) { + hdrlen =generate_object_header (hdr, hdrlen, type, len); +return stream_loose_object (oid, hdr, hdrlen, buf, len,0 , flags); + }
So stream_loose_object() is called by passing the flag HASH_STREAM to write_object_file_flags() and passing a struct input_stream via its buf pointer. That's ... unconventional. Certainly scary. Why not export stream_loose_object() and call it directly? Demo patch below.
Quoted 10 lines
object-file.c +1
-0+/* Normally if we have it in the pack then we do not bother writing * it out into .git/objects/??/?{38} file. */ object-store.h +0
-0@@ -34,6 +34,11 @@ struct object_directory {char *path;
object-store.h struct object_directory { · line 44
Quoted 10 lines
object-store.h +5
-0}; +struct input_stream { +const void *(*read)(const struct input_stream *,unsigned long *len); +void *data; +}; + KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq)
builtin/unpack-objects.c +2 -4
@@ -375,10 +375,8 @@ static void write_stream_blob(unsigned nr, unsigned long size) data.zstream = &zstream;git_inflate_init (&zstream); -if (write_object_file_flags (&in_stream, size, -type_name (OBJ_BLOB ), - &obj_list[nr].oid, -HASH_STREAM )) +if (stream_loose_object (&in_stream, size,type_name (OBJ_BLOB ),0 ,0 , + &obj_list[nr].oid))die (_ ("failed to write object in stream" ));if (zstream.total_out != size || data.status !=Z_STREAM_END )
object-file.c +7 -13
@@ -1994,9 +1994,9 @@ static int freshen_packed_object(const struct object_id *oid)return 1 ; } -static int stream_loose_object (struct object_id *oid,char *hdr,int hdrlen, -const struct input_stream *in_stream, -unsigned long len,time_t mtime,unsigned flags) +int stream_loose_object (struct input_stream *in_stream,unsigned long len, +const char *type,time_t mtime,unsigned flags, +struct object_id *oid) {int fd, ret, err =0 , flush =0 ;unsigned char compressed[4096 ];
@@ -2006,6 +2006,10 @@ static int stream_loose_object(struct object_id *oid, char *hdr, int hdrlen,static struct strbuf tmp_file =STRBUF_INIT ;static struct strbuf filename =STRBUF_INIT ;int dirlen; +char hdr[MAX_HEADER_LEN ]; +int hdrlen =sizeof (hdr); + + hdrlen =generate_object_header (hdr, hdrlen, type, len);/* When oid is not determined, save tmp file to odb path. */ strbuf_addf (&filename,"%s/" ,get_object_directory ());
@@ -2083,16 +2087,6 @@ int write_object_file_flags(const void *buf, unsigned long len,char hdr[MAX_HEADER_LEN ];int hdrlen =sizeof (hdr); -/* When streaming a large blob object (marked as HASH_STREAM), -* we have no chance to run "write_object_file_prepare()" to -* calculate the "oid" in advance. Call "stream_loose_object()" -* to write loose object in stream. -*/ -if (flags &HASH_STREAM ) { - hdrlen =generate_object_header (hdr, hdrlen, type, len); -return stream_loose_object (oid, hdr, hdrlen, buf, len,0 , flags); - } -/* Normally if we have it in the pack then we do not bother writing * it out into .git/objects/??/?{38} file. */
object-store.h +4 -0
@@ -237,6 +237,10 @@ static inline int write_object_file(const void *buf, unsigned long len,return write_object_file_flags (buf, len, type, oid,0 ); } +int stream_loose_object (struct input_stream *in_stream,unsigned long len, +const char *type,time_t mtime,unsigned flags, +struct object_id *oid); +int hash_object_file_literally (const void *buf,unsigned long len,const char *type,struct object_id *oid,unsigned flags);
PATCH v6 5/6 unpack-objects.c: add dry_run mode for get_data()
In reply to Han XinFrom: Han Xin <[email protected]>
In dry_run mode, "get_data()" is used to verify the inflation of data, and the returned buffer will not be used at all and will be freed immediately. Even in dry_run mode, it is dangerous to allocate a full-size buffer for a large blob object. Therefore, only allocate a low memory footprint when calling "get_data()" in dry_run mode.
--- builtin/unpack-objects.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
builtin/unpack-objects.c +17 -6
@@ -96,15 +96,21 @@ static void use(int bytes)display_throughput (progress, consumed_bytes); } -static void *get_data (unsigned long size) +static void *get_data (unsigned long size,int dry_run) {git_zstream stream; -void *buf =xmallocz (size); +unsigned long bufsize; +void *buf;memset (&stream,0 ,sizeof (stream)); +if (dry_run && size >8192 ) + bufsize =8192 ; +else + bufsize = size; + buf =xmallocz (bufsize); stream.next_out = buf; - stream.avail_out = size; + stream.avail_out = bufsize; stream.next_in =fill (1 ); stream.avail_in = len;git_inflate_init (&stream);
@@ -124,6 +130,11 @@ static void *get_data(unsigned long size) } stream.next_in =fill (1 ); stream.avail_in = len; +if (dry_run) { +/* reuse the buffer in dry_run mode */ + stream.next_out = buf; + stream.avail_out = bufsize; + } }git_inflate_end (&stream);return buf;
@@ -323,7 +334,7 @@ static void added_object(unsigned nr, enum object_type type,static void unpack_non_delta_entry (enum object_type type,unsigned long size,unsigned nr) { -void *buf =get_data (size); +void *buf =get_data (size, dry_run);if (!dry_run && buf)write_object (nr, type, buf, size);
@@ -357,7 +368,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,if (type ==OBJ_REF_DELTA ) {oidread (&base_oid,fill (the_hash_algo->rawsz));use (the_hash_algo->rawsz); - delta_data =get_data (delta_size); + delta_data =get_data (delta_size, dry_run);if (dry_run || !delta_data) {free (delta_data);return ;
@@ -396,7 +407,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,if (base_offset <=0 || base_offset >= obj_list[nr].offset)die ("offset value out of bound for delta base object" ); - delta_data =get_data (delta_size); + delta_data =get_data (delta_size, dry_run);if (dry_run || !delta_data) {free (delta_data);return ;
Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
Re: [PATCH v6 5/6] unpack-objects.c: add dry_run mode for get_data()
Am 17.12.21 um 12:26 schrieb Han Xin:
Quoted 7 lines
From: Han Xin <[email protected]>
In
dry_runmode, "get_data()" is used to verify the inflation of data, and the returned buffer will not be used at all and will be freed immediately. Even indry_runmode, it is dangerous to allocate a full-size buffer for a large blob object. Therefore, only allocate a low memory footprint when calling "get_data()" indry_runmode.
Clever. Looks good to me.
For some reason I was expecting this patch to have some connection to one of the earlier ones (perhaps because get_data() was mentioned), but it is technically independent.
Quoted 75 lines
Suggested-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- builtin/unpack-objects.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)builtin/unpack-objects.c +17
-6@@ -96,15 +96,21 @@ static void use(int bytes)display_throughput (progress, consumed_bytes); } -static void *get_data (unsigned long size) +static void *get_data (unsigned long size,int dry_run) {git_zstream stream; -void *buf =xmallocz (size); +unsigned long bufsize; +void *buf;memset (&stream,0 ,sizeof (stream)); +if (dry_run && size >8192 ) + bufsize =8192 ; +else + bufsize = size; + buf =xmallocz (bufsize); stream.next_out = buf; - stream.avail_out = size; + stream.avail_out = bufsize; stream.next_in =fill (1 ); stream.avail_in = len;git_inflate_init (&stream);@@ -124,6 +130,11 @@ static void *get_data(unsigned long size) } stream.next_in =fill (1 ); stream.avail_in = len; +if (dry_run) { +/* reuse the buffer in dry_run mode */ + stream.next_out = buf; + stream.avail_out = bufsize; + } }git_inflate_end (&stream);return buf;@@ -323,7 +334,7 @@ static void added_object(unsigned nr, enum object_type type,static void unpack_non_delta_entry (enum object_type type,unsigned long size,unsigned nr) { -void *buf =get_data (size); +void *buf =get_data (size, dry_run);if (!dry_run && buf)write_object (nr, type, buf, size);@@ -357,7 +368,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,if (type ==OBJ_REF_DELTA ) {oidread (&base_oid,fill (the_hash_algo->rawsz));use (the_hash_algo->rawsz); - delta_data =get_data (delta_size); + delta_data =get_data (delta_size, dry_run);if (dry_run || !delta_data) {free (delta_data);return ;@@ -396,7 +407,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,if (base_offset <=0 || base_offset >= obj_list[nr].offset)die ("offset value out of bound for delta base object" ); - delta_data =get_data (delta_size); + delta_data =get_data (delta_size, dry_run);if (dry_run || !delta_data) {free (delta_data);return ;
PATCH v6 6/6 unpack-objects: unpack_non_delta_entry() read data in a stream
In reply to Han XinFrom: Han Xin <[email protected]>
We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.
By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".
However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "core.BigFileStreamingThreshold" in zstream. See the following benchmarks:
hyperfine \
--setup \
'if ! test -d scalar.git; then git clone --bare https://github.com/microsoft/scalar.git; cp scalar.git/objects/pack/*.pack small.pack; fi' \
--prepare 'rm -rf dest.git && git init --bare dest.git'
Summary
'./git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'origin/master'
1.01 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~1' 1.01 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~0' 1.03 ± 0.10 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'origin/master' 1.02 ± 0.07 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~0' 1.10 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~1'
--- Documentation/config/core.txt | 11 ++++ builtin/unpack-objects.c | 73 +++++++++++++++++++++++- cache.h | 1 + config.c | 5 ++ environment.c | 1 + t/t5590-unpack-non-delta-objects.sh | 87 +++++++++++++++++++++++++++++ 6 files changed, 177 insertions(+), 1 deletion(-) create mode 100755 t/t5590-unpack-non-delta-objects.sh
Documentation/config/core.txt +11 -0
@@ -424,6 +424,17 @@ be delta compressed, but larger binary media files won't be. + Common unit suffixes of 'k', 'm', or 'g' are supported. +core.bigFileStreamingThreshold:: + Files larger than this will be streamed out to a temporary + object file while being hashed, which will when be renamed + in-place to a loose object, particularly if the + `core.bigFileThreshold' setting dictates that they're always + written out as loose objects. ++ +Default is 128 MiB on all platforms. ++ +Common unit suffixes of 'k', 'm', or 'g' are supported. + core.excludesFile:: Specifies the pathname to the file that contains patterns to describe paths that are not meant to be tracked, in addition
builtin/unpack-objects.c +72 -1
83 lines not on this page · raw message
cache.h +1 -0
@@ -975,6 +975,7 @@ extern size_t packed_git_window_size;extern size_t packed_git_limit;extern size_t delta_base_cache_limit;extern unsigned long big_file_threshold; +extern unsigned long big_file_streaming_threshold;extern unsigned long pack_size_limit_cfg;/*
config.c +5 -0
@@ -1408,6 +1408,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.bigfilestreamingthreshold" )) { + big_file_streaming_threshold =git_config_ulong (var, value); +return 0 ; + } +if (!strcmp (var,"core.packedgitlimit" )) { packed_git_limit =git_config_ulong (var, value);return 0 ;
environment.c +1 -0
@@ -47,6 +47,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ;size_t delta_base_cache_limit =96 *1024 *1024 ;unsigned long big_file_threshold =512 *1024 *1024 ; +unsigned long big_file_streaming_threshold =128 *1024 *1024 ;int pager_use_color =1 ;const char *editor_program;const char *askpass_program;
t/t5590-unpack-non-delta-objects.sh new file +87 -0
87 lines not on this page · raw message
Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
PATCH v5 1/6 object-file: refactor write_loose_object() to support read from stream
In reply to Han XinFrom: Han Xin <[email protected]>
We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.
This can be improved by feeding data to "write_loose_object()" in a stream. The input stream is implemented as an interface.
In the first step, we add a new flag called "HASH_STREAM" and make a simple implementation, feeding the entire buffer in the stream to "write_loose_object()" as a refactor.
--- cache.h | 1 + object-file.c | 7 ++++++- object-store.h | 5 +++++ 3 files changed, 12 insertions(+), 1 deletion(-)
cache.h +1 -0
@@ -888,6 +888,7 @@ int ie_modified(struct index_state *, const struct cache_entry *, struct stat *,#define HASH_FORMAT_CHECK 2#define HASH_RENORMALIZE 4#define HASH_SILENT 8 +#define HASH_STREAM 16int index_fd (struct index_state *istate,struct object_id *oid,int fd,struct stat *st,enum object_type type,const char *path,unsigned flags);int index_path (struct index_state *istate,struct object_id *oid,const char *path,struct stat *st,unsigned flags);
object-file.c +6 -1
@@ -1898,7 +1898,12 @@ static int write_loose_object(const struct object_id *oid, char *hdr, the_hash_algo->update_fn (&c, hdr, hdrlen);/* Then the data itself.. */ - stream.next_in = (void *)buf; +if (flags &HASH_STREAM ) { +struct input_stream *in_stream = (struct input_stream *)buf; + stream.next_in = (void *)in_stream->read (in_stream, &len); + }else { + stream.next_in = (void *)buf; + } stream.avail_in = len; do { unsigned char *in0 = stream.next_in;
object-store.h +5 -0
@@ -34,6 +34,11 @@ struct object_directory {char *path; }; +struct input_stream { +const void *(*read)(struct input_stream *,unsigned long *len); +void *data; +}; + KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq)
Signature · 1 line
2.34.0
PATCH v5 2/6 object-file.c: handle undetermined oid in write_loose_object()
In reply to Han XinFrom: Han Xin <[email protected]>
When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".
In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
The promise that "oid" is constant in "write_loose_object()" has been removed because it will be filled after reading all stream data.
--- object-file.c | 48 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 39 insertions(+), 9 deletions(-)
object-file.c +39 -9
@@ -1860,11 +1860,11 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)return fd; } -static int write_loose_object (const struct object_id *oid,char *hdr, +static int write_loose_object (struct object_id *oid,char *hdr,int hdrlen,const void *buf,unsigned long len,time_t mtime,unsigned flags) { -int fd, ret; +int fd, ret, err =0 ;unsigned char compressed[4096 ];git_zstream stream;git_hash_ctx c;
@@ -1872,16 +1872,21 @@ static int write_loose_object(const struct object_id *oid, char *hdr,static struct strbuf tmp_file =STRBUF_INIT ;static struct strbuf filename =STRBUF_INIT ; -loose_object_path (the_repository, &filename, oid); +if (flags &HASH_STREAM ) +/* When oid is not determined, save tmp file to odb path. */ +strbuf_addf (&filename,"%s/" ,get_object_directory ()); +else +loose_object_path (the_repository, &filename, oid); fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {if (flags &HASH_SILENT ) -return -1 ; + err =-1 ;else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); + err =error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ());else -return error_errno (_ ("unable to create temporary file" )); + err =error_errno (_ ("unable to create temporary file" )); + goto cleanup; }/* Set it up */
@@ -1923,12 +1928,34 @@ static int write_loose_object(const struct object_id *oid, char *hdr,die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), ret); the_hash_algo->final_oid_fn (¶no_oid, &c); -if (!oideq (oid, ¶no_oid)) +if (!(flags &HASH_STREAM ) && !oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));close_loose_object (fd); +if (flags &HASH_STREAM ) { +int dirlen; + +oidcpy ((struct object_id *)oid, ¶no_oid); +loose_object_path (the_repository, &filename, oid); + +/* We finally know the object path, and create the missing dir. */ + dirlen =directory_size (filename.buf); +if (dirlen) { +struct strbuf dir =STRBUF_INIT ; +strbuf_add (&dir, filename.buf, dirlen -1 ); +if (mkdir (dir.buf,0777 ) && errno !=EEXIST ) + err =-1 ; +else if (adjust_shared_perm (dir.buf)) + err =-1 ; +else +strbuf_release (&dir); +if (err <0 ) + goto cleanup; + } + } +if (mtime) {struct utimbuf utb; utb.actime = mtime;
@@ -1938,7 +1965,10 @@ static int write_loose_object(const struct object_id *oid, char *hdr,warning_errno (_ ("failed utime() on %s" ), tmp_file.buf); } -return finalize_object_file (tmp_file.buf, filename.buf); + err =finalize_object_file (tmp_file.buf, filename.buf); +cleanup: +strbuf_release (&filename); +return err; }static int freshen_loose_object (const struct object_id *oid)
@@ -2015,7 +2045,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid)); hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; - ret =write_loose_object (oid, hdr, hdrlen, buf, len, mtime,0 ); + ret =write_loose_object ((struct object_id *) oid, hdr, hdrlen, buf, len, mtime,0 );free (buf);return ret;
Signature · 1 line
2.34.0
Re: [PATCH v5 2/6] object-file.c: handle undetermined oid in write_loose_object()
Quoted 68 lines · Han Xin
From: Han Xin <[email protected]>
When streaming a large blob object to "
write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
The promise that "oid" is constant in "
write_loose_object()" has been removed because it will be filled after reading all stream data.Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 48 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 39 insertions(+), 9 deletions(-)object-file.c +11
-6@@ -1860,11 +1860,11 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)return fd; } -static int write_loose_object (const struct object_id *oid,char *hdr, +static int write_loose_object (struct object_id *oid,char *hdr,int hdrlen,const void *buf,unsigned long len,time_t mtime,unsigned flags) { -int fd, ret; +int fd, ret, err =0 ;unsigned char compressed[4096 ];git_zstream stream;git_hash_ctx c;@@ -1872,16 +1872,21 @@ static int write_loose_object(const struct object_id *oid, char *hdr,static struct strbuf tmp_file =STRBUF_INIT ;static struct strbuf filename =STRBUF_INIT ; -loose_object_path (the_repository, &filename, oid); +if (flags &HASH_STREAM ) +/* When oid is not determined, save tmp file to odb path. */ +strbuf_addf (&filename,"%s/" ,get_object_directory ()); +else +loose_object_path (the_repository, &filename, oid); fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {if (flags &HASH_SILENT ) -return -1 ; + err =-1 ;else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); + err =error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ());else -return error_errno (_ ("unable to create temporary file" )); + err =error_errno (_ ("unable to create temporary file" )); + goto cleanup; }/* Set it up */
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1933
Quoted 7 lines
object-file.c +1
-1@@ -1923,12 +1928,34 @@ static int write_loose_object(const struct object_id *oid, char *hdr,die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), ret); the_hash_algo->final_oid_fn (¶no_oid, &c); -if (!oideq (oid, ¶no_oid)) +if (!(flags &HASH_STREAM ) && !oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));
Here we don't have a meaningful "const" OID anymore, but still if we die we use the "oid".
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1940
Quoted 6 lines
object-file.c +4
-0close_loose_object (fd); +if (flags &HASH_STREAM ) { +int dirlen; + +oidcpy ((struct object_id *)oid, ¶no_oid);
This cast isn't needed anymore now that you stripped the "const" off, but more on that later...
Quoted 5 lines
object-file.c +5
-0+loose_object_path (the_repository, &filename, oid); + +/* We finally know the object path, and create the missing dir. */ + dirlen =directory_size (filename.buf); +if (dirlen) {
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1955
Quoted 10 lines
object-file.c +10
-0+struct strbuf dir =STRBUF_INIT ; +strbuf_add (&dir, filename.buf, dirlen -1 ); +if (mkdir (dir.buf,0777 ) && errno !=EEXIST ) + err =-1 ; +else if (adjust_shared_perm (dir.buf)) + err =-1 ; +else +strbuf_release (&dir); +if (err <0 ) + goto cleanup;
Can't we use one of the existing utility functions for this? Testing locally I could replace this with:
diff --git a/object-file.c b/object-file.c
index 7c93db11b2d..05e1fae893d 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1952,14 +1952,11 @@ static int write_loose_object(struct object_id *oid, char *hdr,
if (dirlen) {
struct strbuf dir = STRBUF_INIT;
strbuf_add(&dir, filename.buf, dirlen - 1);
- if (mkdir(dir.buf, 0777) && errno != EEXIST)
+
+ if (mkdir_in_gitdir(dir.buf) < 0) {
err = -1;
- else if (adjust_shared_perm(dir.buf))
- err = -1;
- else
- strbuf_release(&dir);
- if (err < 0)
goto cleanup;
+ }
}
}And your tests still pass. Maybe they have a blind spot, or maybe we can just use the existing function.
Quoted 7 lines
object-file.c +3
-0+ } + } +if (mtime) {struct utimbuf utb; utb.actime = mtime;
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1972
Quoted 9 lines
object-file.c +4
-1@@ -1938,7 +1965,10 @@ static int write_loose_object(const struct object_id *oid, char *hdr,warning_errno (_ ("failed utime() on %s" ), tmp_file.buf); } -return finalize_object_file (tmp_file.buf, filename.buf); + err =finalize_object_file (tmp_file.buf, filename.buf); +cleanup: +strbuf_release (&filename); +return err; }
Reading this series is an odd mixture of of things that would really be much easier to understand if they were combined, e.g. 1/6 adding APIs that aren't used by anything, but then adding one codepath (also unused), that we then use later. Could just add it at the same time as the use and the patch would be easier to read....
...and then this, which *is* something that could be split up into an earlier cleanup step, i.e. the strbuf leak here exists before this series, fixing it is good, but splitting that up into its own patch would make this diff smaller & the actual behavior changes easier to reason about.
object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2051
Quoted 10 lines
object-file.c +1
-1static int freshen_loose_object (const struct object_id *oid)@@ -2015,7 +2045,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid)); hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; - ret =write_loose_object (oid, hdr, hdrlen, buf, len, mtime,0 ); + ret =write_loose_object ((struct object_id *) oid, hdr, hdrlen, buf, len, mtime,0 );free (buf);return ret;
...on the "more on that later", here we're casting the "oid" from const for a function that's never going to be involved in the streaming codepath.
I know I suggested the HASH_STREAM flag, but what I was really going for was "let's share more of the code?", looking at this v5 (which is already much better than v4) I think a better approach is to split up write_loose_object().
I.e. it already calls close_loose_object() and finalize_object_file() to do some of its work, but around that we have:
1. Figuring out a path for the (temp) object file
2. Creating the tempfile
3. Setting up zlib
4. Once zlib is set up inspect its state, die with a message about oid_to_hex(oid) if we failed
5. Optionally, do HASH_STREAM stuff
Maybe force a loose object if "mtime".
I think if that's split up so that each of those is its own little function what's now write_loose_object() can call those in sequence, and a new stream_loose_object() can just do #1 differentl, followed by the same #2 and #4, but do #4 differently etc.
You'll still be able to re-use the write_object_file_prepare() etc. logic.
As an example your 5/6 copy/pastes the xsnprintf() formatting of the object header. It's just one line, but it's also code that's very central to git, so I think instead of just copy/pasting it a prep step of factoring it out would make sense, and that would be a prep cleanup that would help later readability. E.g.:
diff --git a/object-file.c b/object-file.c
index eac67f6f5f9..a7dcbd929e9 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1009,6 +1009,13 @@ void *xmmap(void *start, size_t length,
return ret;
}
+static int generate_object_header(char *buf, int bufsz, const char *type_name,
+ unsigned long size)
+{
+ return xsnprintf(buf, bufsz, "%s %"PRIuMAX , type_name,
+ (uintmax_t)size) + 1;
+}
+
/*
* With an in-core object data in "map", rehash it to make sure the
* object name actually matches "oid" to detect object corruption.
@@ -1037,7 +1044,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,
return -1;
/* Generate the header */
- hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(obj_type), (uintmax_t)size) + 1;
+ hdrlen = generate_object_header(hdr, sizeof(hdr), type_name(obj_type), size);
/* Sha1.. */
r->hash_algo->init_fn(&c);
@@ -1737,7 +1744,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,
git_hash_ctx c;
/* Generate the header */
- *hdrlen = xsnprintf(hdr, *hdrlen, "%s %"PRIuMAX , type, (uintmax_t)len)+1;
+ *hdrlen = generate_object_header(hdr, *hdrlen, type, len);
/* Sha1.. */
algo->init_fn(&c);
@@ -2009,7 +2016,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
buf = read_object(the_repository, oid, &type, &len);
if (!buf)
return error(_("cannot read object for %s"), oid_to_hex(oid));
- hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
+ hdrlen = generate_object_header(hdr, sizeof(hdr), type_name(type), len);
ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
free(buf);Then in your change on top you just call that generate_object_header(), or better yet your amended write_object_file_flags() can just call a similarly amended write_object_file_prepare() directly.
PATCH v5 3/6 object-file.c: read stream in a loop in write_loose_object()
In reply to Han XinFrom: Han Xin <[email protected]>
In order to prepare the stream version of "write_loose_object()", read the input stream in a loop in "write_loose_object()", so that we can feed the contents of large blob object to "write_loose_object()" using a small fixed buffer.
--- object-file.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-)
object-file.c +15 -8
@@ -1864,7 +1864,7 @@ static int write_loose_object(struct object_id *oid, char *hdr, int hdrlen, const void *buf, unsigned long len, time_t mtime, unsigned flags) { - int fd, ret, err = 0; + int fd, ret, err = 0, flush = 0;unsigned char compressed[4096 ];git_zstream stream;git_hash_ctx c;
@@ -1903,22 +1903,29 @@ static int write_loose_object(struct object_id *oid, char *hdr, the_hash_algo->update_fn (&c, hdr, hdrlen);/* Then the data itself.. */ -if (flags &HASH_STREAM ) { -struct input_stream *in_stream = (struct input_stream *)buf; - stream.next_in = (void *)in_stream->read (in_stream, &len); - }else { +if (!(flags &HASH_STREAM )) { stream.next_in = (void *)buf; + stream.avail_in = len; + flush =Z_FINISH ; } - stream.avail_in = len;do {unsigned char *in0 = stream.next_in; - ret =git_deflate (&stream,Z_FINISH ); +if (flags &HASH_STREAM && !stream.avail_in) { +struct input_stream *in_stream = (struct input_stream *)buf; +const void *in = in_stream->read (in_stream, &stream.avail_in); + stream.next_in = (void *)in; + in0 = (unsigned char *)in; +/* All data has been read. */ +if (len + hdrlen == stream.total_in + stream.avail_in) + flush =Z_FINISH ; + } + ret =git_deflate (&stream, flush); the_hash_algo->update_fn (&c, in0, stream.next_in - in0);if (write_buffer (fd, compressed, stream.next_out - compressed) <0 )die (_ ("unable to write loose object file" )); stream.next_out = compressed; stream.avail_out =sizeof (compressed); - }while (ret ==Z_OK ); + }while (ret ==Z_OK || ret ==Z_BUF_ERROR );if (ret !=Z_STREAM_END ) die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
Signature · 1 line
2.34.0
PATCH v5 4/6 unpack-objects.c: add dry_run mode for get_data()
In reply to Han XinFrom: Han Xin <[email protected]>
In dry_run mode, "get_data()" is used to verify the inflation of data, and the returned buffer will not be used at all and will be freed immediately. Even in dry_run mode, it is dangerous to allocate a full-size buffer for a large blob object. Therefore, only allocate a low memory footprint when calling "get_data()" in dry_run mode.
--- builtin/unpack-objects.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
builtin/unpack-objects.c +12 -6
@@ -96,15 +96,16 @@ static void use(int bytes)display_throughput (progress, consumed_bytes); } -static void *get_data (unsigned long size) +static void *get_data (unsigned long size,int dry_run) {git_zstream stream; -void *buf =xmallocz (size); +unsigned long bufsize = dry_run ?8192 : size; +void *buf =xmallocz (bufsize);memset (&stream,0 ,sizeof (stream)); stream.next_out = buf; - stream.avail_out = size; + stream.avail_out = bufsize; stream.next_in =fill (1 ); stream.avail_in = len;git_inflate_init (&stream);
@@ -124,6 +125,11 @@ static void *get_data(unsigned long size) } stream.next_in =fill (1 ); stream.avail_in = len; +if (dry_run) { +/* reuse the buffer in dry_run mode */ + stream.next_out = buf; + stream.avail_out = bufsize; + } }git_inflate_end (&stream);return buf;
@@ -323,7 +329,7 @@ static void added_object(unsigned nr, enum object_type type,static void unpack_non_delta_entry (enum object_type type,unsigned long size,unsigned nr) { -void *buf =get_data (size); +void *buf =get_data (size, dry_run);if (!dry_run && buf)write_object (nr, type, buf, size);
@@ -357,7 +363,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,if (type ==OBJ_REF_DELTA ) {oidread (&base_oid,fill (the_hash_algo->rawsz));use (the_hash_algo->rawsz); - delta_data =get_data (delta_size); + delta_data =get_data (delta_size, dry_run);if (dry_run || !delta_data) {free (delta_data);return ;
@@ -396,7 +402,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,if (base_offset <=0 || base_offset >= obj_list[nr].offset)die ("offset value out of bound for delta base object" ); - delta_data =get_data (delta_size); + delta_data =get_data (delta_size, dry_run);if (dry_run || !delta_data) {free (delta_data);return ;
Signature · 1 line
2.34.0
PATCH v5 5/6 object-file.c: make "write_object_file_flags()" to support "HASH_STREAM"
In reply to Han XinFrom: Han Xin <[email protected]>
We will use "write_object_file_flags()" in "unpack_non_delta_entry()" to read the entire data contents in stream. When read in stream, we needn't prepare "oid" before "write_loose_object()", only generate the header.
--- object-file.c | 5 +++++ 1 file changed, 5 insertions(+)
object-file.c +5 -0
@@ -2002,6 +2002,11 @@ int write_object_file_flags(const void *buf, unsigned long len, {char hdr[MAX_HEADER_LEN ];int hdrlen =sizeof (hdr); +if (flags &HASH_STREAM ) { +/* Generate the header */ + hdrlen =xsnprintf (hdr, hdrlen,"%s %" PRIuMAX , type, (uintmax_t )len)+1 ; +return write_loose_object (oid, hdr, hdrlen, buf, len,0 , flags); + }/* Normally if we have it in the pack then we do not bother writing * it out into .git/objects/??/?{38} file.
Signature · 1 line
2.34.0
PATCH v5 6/6 unpack-objects: unpack_non_delta_entry() read data in a stream
In reply to Han XinFrom: Han Xin <[email protected]>
We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.
By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".
However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "core.BigFileStreamingThreshold" in zstream. See the following benchmarks:
hyperfine \
--setup \
'if ! test -d scalar.git; then git clone --bare https://github.com/microsoft/scalar.git; cp scalar.git/objects/pack/*.pack small.pack; fi' \
--prepare 'rm -rf dest.git && git init --bare dest.git'
Summary
'./git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'origin/master'
1.01 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~1' 1.01 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~0' 1.03 ± 0.10 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'origin/master' 1.02 ± 0.07 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~0' 1.10 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~1'
--- Documentation/config/core.txt | 11 +++++ builtin/unpack-objects.c | 70 ++++++++++++++++++++++++++++- cache.h | 1 + config.c | 5 +++ environment.c | 1 + t/t5590-unpack-non-delta-objects.sh | 70 +++++++++++++++++++++++++++++ 6 files changed, 157 insertions(+), 1 deletion(-) create mode 100755 t/t5590-unpack-non-delta-objects.sh
Documentation/config/core.txt +11 -0
@@ -424,6 +424,17 @@ be delta compressed, but larger binary media files won't be. + Common unit suffixes of 'k', 'm', or 'g' are supported. +core.bigFileStreamingThreshold:: + Files larger than this will be streamed out to a temporary + object file while being hashed, which will when be renamed + in-place to a loose object, particularly if the + `core.bigFileThreshold' setting dictates that they're always + written out as loose objects. ++ +Default is 128 MiB on all platforms. ++ +Common unit suffixes of 'k', 'm', or 'g' are supported. + core.excludesFile:: Specifies the pathname to the file that contains patterns to describe paths that are not meant to be tracked, in addition
builtin/unpack-objects.c +69 -1
80 lines not on this page · raw message
cache.h +1 -0
@@ -965,6 +965,7 @@ extern size_t packed_git_window_size;extern size_t packed_git_limit;extern size_t delta_base_cache_limit;extern unsigned long big_file_threshold; +extern unsigned long big_file_streaming_threshold;extern unsigned long pack_size_limit_cfg;/*
config.c +5 -0
@@ -1408,6 +1408,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.bigfilestreamingthreshold" )) { + big_file_streaming_threshold =git_config_ulong (var, value); +return 0 ; + } +if (!strcmp (var,"core.packedgitlimit" )) { packed_git_limit =git_config_ulong (var, value);return 0 ;
environment.c +1 -0
@@ -46,6 +46,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ;size_t delta_base_cache_limit =96 *1024 *1024 ;unsigned long big_file_threshold =512 *1024 *1024 ; +unsigned long big_file_streaming_threshold =128 *1024 *1024 ;int pager_use_color =1 ;const char *editor_program;const char *askpass_program;
t/t5590-unpack-non-delta-objects.sh new file +70 -0
70 lines not on this page · raw message
Signature · 1 line
2.34.0
Re: [PATCH v5 6/6] unpack-objects: unpack_non_delta_entry() read data in a stream
builtin/unpack-objects.c static void added_object(unsigned nr, enum object_type type, · line 373
Quoted 3 lines · Han Xin
From: Han Xin <[email protected]>
[...]builtin/unpack-objects.c +1
-0+if ((ret =write_object_file_flags (&in_stream, size,type_name (OBJ_BLOB ) ,&obj_list[nr].oid,HASH_STREAM )))
There's some odd code formatting here, i.e.. ") ,&" not "), &". Could also use line-wrapping at 79 characters.
PATCH v4 1/5 object-file: refactor write_loose_object() to read buffer from stream
In reply to Han XinFrom: Han Xin <[email protected]>
We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.
This can be improved by feeding data to "write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.
--- object-file.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++---- object-store.h | 6 ++++++ 2 files changed, 55 insertions(+), 4 deletions(-)
object-file.c +49 -4
@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)return fd; } +struct simple_input_stream_data { +const void *buf; +unsigned long len; +}; + +static const void *feed_simple_input_stream (struct input_stream *in_stream,unsigned long *len) +{ +struct simple_input_stream_data *data = in_stream->data; + +if (data->len ==0 ) { + *len =0 ; +return NULL ; + } + *len = data->len; + data->len =0 ; +return data->buf; +} +static int write_loose_object (const struct object_id *oid,char *hdr, -int hdrlen,const void *buf,unsigned long len, +int hdrlen,struct input_stream *in_stream,time_t mtime,unsigned flags) {int fd, ret;
@@ -1871,6 +1889,8 @@ static int write_loose_object(const struct object_id *oid, char *hdr,struct object_id parano_oid;static struct strbuf tmp_file =STRBUF_INIT ;static struct strbuf filename =STRBUF_INIT ; +const void *buf; +unsigned long len;loose_object_path (the_repository, &filename, oid);
@@ -1898,6 +1918,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr, the_hash_algo->update_fn (&c, hdr, hdrlen);/* Then the data itself.. */ + buf = in_stream->read (in_stream, &len); stream.next_in = (void *)buf; stream.avail_in = len; do {
@@ -1960,6 +1981,14 @@ int write_object_file_flags(const void *buf, unsigned long len, {char hdr[MAX_HEADER_LEN ];int hdrlen =sizeof (hdr); +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + .size = len, + };/* Normally if we have it in the pack then we do not bother writing * it out into .git/objects/??/?{38} file.
@@ -1968,7 +1997,7 @@ int write_object_file_flags(const void *buf, unsigned long len, &hdrlen);if (freshen_packed_object (oid) ||freshen_loose_object (oid))return 0 ; -return write_loose_object (oid, hdr, hdrlen, buf, len,0 , flags); +return write_loose_object (oid, hdr, hdrlen, &in_stream,0 , flags); } int hash_object_file_literally(const void *buf, unsigned long len,
@@ -1977,6 +2006,14 @@ int hash_object_file_literally(const void *buf, unsigned long len, {char *header;int hdrlen, status =0 ; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + .size = len, + };/* type string, SP, %lu of the length plus NUL must fit this */ hdrlen =strlen (type) +MAX_HEADER_LEN ;
@@ -1988,7 +2025,7 @@ int hash_object_file_literally(const void *buf, unsigned long len, goto cleanup;if (freshen_packed_object (oid) ||freshen_loose_object (oid)) goto cleanup; - status =write_loose_object (oid, header, hdrlen, buf, len,0 ,0 ); + status =write_loose_object (oid, header, hdrlen, &in_stream,0 ,0 ); cleanup:free (header);
@@ -2003,14 +2040,22 @@ int force_object_loose(const struct object_id *oid, time_t mtime)char hdr[MAX_HEADER_LEN ];int hdrlen;int ret; +struct simple_input_stream_data data; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = &data, + };if (has_loose_object (oid))return 0 ; buf =read_object (the_repository, oid, &type, &len); + in_stream.size = len;if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid)); + data.buf = buf; + data.len = len; hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; - ret =write_loose_object (oid, hdr, hdrlen, buf, len, mtime,0 ); + ret =write_loose_object (oid, hdr, hdrlen, &in_stream, mtime,0 );free (buf);return ret;
object-store.h +6 -0
@@ -34,6 +34,12 @@ struct object_directory {char *path; }; +struct input_stream { +const void *(*read)(struct input_stream *,unsigned long *len); +void *data; +size_t size; +}; + KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq)
Signature · 1 line
2.34.0
Re: [PATCH v4 1/5] object-file: refactor write_loose_object() to read buffer from stream
Quoted 23 lines · Han Xin
From: Han Xin <[email protected]>
We used to call "
get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.This can be improved by feeding data to "
write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++---- object-store.h | 6 ++++++ 2 files changed, 55 insertions(+), 4 deletions(-)object-file.c +0
-0
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1866
Quoted 7 lines
object-file.c +4
-0@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)return fd; } +struct simple_input_stream_data { +const void *buf; +unsigned long len; +};
I see why you picked "const void *buf" here, over say const char *, it's what "struct input_stream" uses.
But why not use size_t for the length, as input_stream does?
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1872
Quoted 5 lines
object-file.c +5
-0+static const void *feed_simple_input_stream (struct input_stream *in_stream,unsigned long *len) +{ +struct simple_input_stream_data *data = in_stream->data; + +if (data->len ==0 ) {
nit: if (!data->len)...
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1878
Quoted 6 lines
object-file.c +6
-0+ *len =0 ; +return NULL ; + } + *len = data->len; + data->len =0 ; +return data->buf;
But isn't the body of this functin the same as:
*len = data->len;
if (!len)
return NULL;
data->len = 0;
return data->buf;I.e. you don't need the condition for setting "*len" if it's 0, then data->len is also 0. You just want to return NULL afterwards, and not set (harmless, but no need) data->len to 0)< or return data->buf.
object-file.c int write_object_file_flags(const void *buf, unsigned long len, · line 1991
Quoted 8 lines
object-file.c +8
-0+struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + .size = len, + };
Maybe it's that I'm unused to it, but I find this a bit more readable:
@@ -2013,12 +2011,13 @@ int write_object_file_flags(const void *buf, unsigned long len,
{
char hdr[MAX_HEADER_LEN];
int hdrlen = sizeof(hdr);
+ struct simple_input_stream_data tmp = {
+ .buf = buf,
+ .len = len,
+ };
struct input_stream in_stream = {
.read = feed_simple_input_stream,
- .data = (void *)&(struct simple_input_stream_data) {
- .buf = buf,
- .len = len,
- },
+ .data = (void *)&tmp,
.size = len,
};Yes there's a temporary variable, but no denser inline casting. Also easier to strep through in a debugger (which will have the type information on "tmp".
object-file.c int hash_object_file_literally(const void *buf, unsigned long len, · line 2016
Quoted 13 lines
object-file.c +8
-0int hash_object_file_literally(const void *buf, unsigned long len,@@ -1977,6 +2006,14 @@ int hash_object_file_literally(const void *buf, unsigned long len, {char *header;int hdrlen, status =0 ; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + .size = len, + };
ditto..
Quoted 15 lines
object-file.c +1
-1/* type string, SP, %lu of the length plus NUL must fit this */ hdrlen =strlen (type) +MAX_HEADER_LEN ;@@ -1988,7 +2025,7 @@ int hash_object_file_literally(const void *buf, unsigned long len, goto cleanup;if (freshen_packed_object (oid) ||freshen_loose_object (oid)) goto cleanup; - status =write_loose_object (oid, header, hdrlen, buf, len,0 ,0 ); + status =write_loose_object (oid, header, hdrlen, &in_stream,0 ,0 ); cleanup:free (header);@@ -2003,14 +2040,22 @@ int force_object_loose(const struct object_id *oid, time_t mtime)char hdr[MAX_HEADER_LEN ];int hdrlen;int ret;
object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2052
Quoted 10 lines
object-file.c +6
-0+struct simple_input_stream_data data; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = &data, + };if (has_loose_object (oid))return 0 ; buf =read_object (the_repository, oid, &type, &len); + in_stream.size = len;
Why are we setting this here?...
object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2054
Quoted 2 lines
object-file.c +0
-0if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid));
...Insted of after this point, as we may error and never use it?
object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2056
Quoted 2 lines
object-file.c +2
-0+ data.buf = buf; + data.len = len;
Probably won't matter, just a nit...
object-file.c int force_object_loose(const struct object_id *oid, time_t mtime)
Quoted 6 lines
object-file.c +6
-0+struct input_stream { +const void *(*read)(struct input_stream *,unsigned long *len); +void *data; +size_t size; +}; +
Ah, and here's the size_t... :)
Re: [PATCH v4 1/5] object-file: refactor write_loose_object() to read buffer from stream
Quoted 38 lines · Ævar Arnfjörð Bjarmason
Quoted 23 lines · Han Xin
From: Han Xin <[email protected]>
We used to call "
get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.This can be improved by feeding data to "
write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++---- object-store.h | 6 ++++++ 2 files changed, 55 insertions(+), 4 deletions(-)object-file.c +0
-0object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1866
Quoted 7 lines
object-file.c +4
-0@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)return fd; } +struct simple_input_stream_data { +const void *buf; +unsigned long len; +};I see why you picked "const void *buf" here, over say const char *, it's what "struct
input_stream" uses.
Quoted 2 lines
But why not use
size_tfor the length, asinput_streamdoes?
Yes, "size_t" will be better here.
Quoted 8 lines
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1872
Quoted 5 lines
object-file.c +5
-0+static const void *feed_simple_input_stream (struct input_stream *in_stream,unsigned long *len) +{ +struct simple_input_stream_data *data = in_stream->data; + +if (data->len ==0 ) {nit: if (!data->len)...
Will apply.
Quoted 15 lines
object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1878
Quoted 6 lines
object-file.c +6
-0+ *len =0 ; +return NULL ; + } + *len = data->len; + data->len =0 ; +return data->buf;But isn't the body of this functin the same as:
*len = data->len; if (!len) return NULL; data->len = 0; return data->buf;
Quoted 3 lines
I.e. you don't need the condition for setting "*len" if it's 0, then
data->lenis also 0. You just want to return NULL afterwards, and not set (harmless, but no need)data->lento 0)< or returndata->buf.
Will apply.
Quoted 29 lines
object-file.c int write_object_file_flags(const void *buf, unsigned long len, · line 1991
Quoted 8 lines
object-file.c +8
-0+struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + .size = len, + };Maybe it's that I'm unused to it, but I find this a bit more readable:
@@ -2013,12 +2011,13 @@ int write_object_file_flags(const void *buf, unsigned long len,
{
char hdr[MAX_HEADER_LEN];
int hdrlen = sizeof(hdr);
+ structsimple_input_stream_datatmp = {
+ .buf = buf,
+ .len = len,
+ };
structinput_streamin_stream= {
.read =feed_simple_input_stream,
- .data = (void *)&(structsimple_input_stream_data) {
- .buf = buf,
- .len = len,
- },
+ .data = (void *)&tmp,
.size = len,
};
Quoted 4 lines
Yes there's a temporary variable, but no denser inline casting. Also easier to strep through in a debugger (which will have the type information on "tmp".
Will apply.
Quoted 37 lines
object-file.c int hash_object_file_literally(const void *buf, unsigned long len, · line 2016
Quoted 13 lines
object-file.c +8
-0int hash_object_file_literally(const void *buf, unsigned long len,@@ -1977,6 +2006,14 @@ int hash_object_file_literally(const void *buf, unsigned long len, {char *header;int hdrlen, status =0 ; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + .size = len, + };ditto..
Quoted 21 lines
object-file.c +6
-1/* type string, SP, %lu of the length plus NUL must fit this */ hdrlen =strlen (type) +MAX_HEADER_LEN ;@@ -1988,7 +2025,7 @@ int hash_object_file_literally(const void *buf, unsigned long len, goto cleanup;if (freshen_packed_object (oid) ||freshen_loose_object (oid)) goto cleanup; - status =write_loose_object (oid, header, hdrlen, buf, len,0 ,0 ); + status =write_loose_object (oid, header, hdrlen, &in_stream,0 ,0 ); cleanup:free (header);@@ -2003,14 +2040,22 @@ int force_object_loose(const struct object_id *oid, time_t mtime)char hdr[MAX_HEADER_LEN ];int hdrlen;int ret; +struct simple_input_stream_data data; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = &data, + };
Quoted 7 lines
object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2052
Quoted 4 lines
object-file.c +1
-0if (has_loose_object (oid))return 0 ; buf =read_object (the_repository, oid, &type, &len); + in_stream.size = len;Why are we setting this here?...
Yes, putting "in_stream.size=len;" here was a stupid decision.
Quoted 18 lines
object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2054
Quoted 2 lines
object-file.c +0
-0if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid));...Insted of after this point, as we may error and never use it?
object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2056
Quoted 2 lines
object-file.c +2
-0+ data.buf = buf; + data.len = len;Probably won't matter, just a nit...
object-file.c int force_object_loose(const struct object_id *oid, time_t mtime)
Quoted 6 lines
object-file.c +6
-0+struct input_stream { +const void *(*read)(struct input_stream *,unsigned long *len); +void *data; +size_t size; +}; +Ah, and here's the
size_t... :)
PATCH v4 2/5 object-file.c: handle undetermined oid in write_loose_object()
In reply to Han XinFrom: Han Xin <[email protected]>
When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".
In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
--- object-file.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)
object-file.c +28 -2
@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,const void *buf;unsigned long len; -loose_object_path (the_repository, &filename, oid); +if (is_null_oid (oid)) { +/* When oid is not determined, save tmp file to odb path. */ +strbuf_reset (&filename); +strbuf_addstr (&filename, the_repository->objects->odb->path); +strbuf_addch (&filename,'/' ); + }else { +loose_object_path (the_repository, &filename, oid); + } fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {
@@ -1939,12 +1946,31 @@ static int write_loose_object(const struct object_id *oid, char *hdr,die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), ret); the_hash_algo->final_oid_fn (¶no_oid, &c); -if (!oideq (oid, ¶no_oid)) +if (!is_null_oid (oid) && !oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));close_loose_object (fd); +if (is_null_oid (oid)) { +int dirlen; + +oidcpy ((struct object_id *)oid, ¶no_oid); +loose_object_path (the_repository, &filename, oid); + +/* We finally know the object path, and create the missing dir. */ + dirlen =directory_size (filename.buf); +if (dirlen) { +struct strbuf dir =STRBUF_INIT ; +strbuf_add (&dir, filename.buf, dirlen -1 ); +if (mkdir (dir.buf,0777 ) && errno !=EEXIST ) +return -1 ; +if (adjust_shared_perm (dir.buf)) +return -1 ; +strbuf_release (&dir); + } + } +if (mtime) {struct utimbuf utb; utb.actime = mtime;
Signature · 1 line
2.34.0
Re: [PATCH v4 2/5] object-file.c: handle undetermined oid in write_loose_object()
Quoted 45 lines · Han Xin
From: Han Xin <[email protected]>
When streaming a large blob object to "
write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)object-file.c +9
-2@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,const void *buf;unsigned long len; -loose_object_path (the_repository, &filename, oid); +if (is_null_oid (oid)) { +/* When oid is not determined, save tmp file to odb path. */ +strbuf_reset (&filename); +strbuf_addstr (&filename, the_repository->objects->odb->path); +strbuf_addch (&filename,'/' ); + }else { +loose_object_path (the_repository, &filename, oid); + } fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {@@ -1939,12 +1946,31 @@ static int write_loose_object(const struct object_id *oid, char *hdr,die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), ret); the_hash_algo->final_oid_fn (¶no_oid, &c); -if (!oideq (oid, ¶no_oid)) +if (!is_null_oid (oid) && !oideq (oid, ¶no_oid))
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1959
Quoted 10 lines
object-file.c +5
-0die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));close_loose_object (fd); +if (is_null_oid (oid)) { +int dirlen; + +oidcpy ((struct object_id *)oid, ¶no_oid); +loose_object_path (the_repository, &filename, oid);
Why are we breaking the promise that "oid" is constant here? I tested locally with the below on top, and it seems to work (at least no tests broke). Isn't it preferrable to the cast & the caller having its "oid" changed?
object-file.c +3 -2
@@ -1958,10 +1958,11 @@ int write_loose_object(const struct object_id *oid, char *hdr,close_loose_object (fd);if (is_null_oid (oid)) { +struct object_id oid2;int dirlen; -oidcpy ((struct object_id *)oid, ¶no_oid); -loose_object_path (the_repository, &filename, oid); +oidcpy (&oid2, ¶no_oid); +loose_object_path (the_repository, &filename, &oid2);/* We finally know the object path, and create the missing dir. */ dirlen =directory_size (filename.buf);
Re: [PATCH v4 2/5] object-file.c: handle undetermined oid in write_loose_object()
Quoted 73 lines · Ævar Arnfjörð Bjarmason
Quoted 45 lines · Han Xin
From: Han Xin <[email protected]>
When streaming a large blob object to "
write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)object-file.c +9
-2@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,const void *buf;unsigned long len; -loose_object_path (the_repository, &filename, oid); +if (is_null_oid (oid)) { +/* When oid is not determined, save tmp file to odb path. */ +strbuf_reset (&filename); +strbuf_addstr (&filename, the_repository->objects->odb->path); +strbuf_addch (&filename,'/' ); + }else { +loose_object_path (the_repository, &filename, oid); + } fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {@@ -1939,12 +1946,31 @@ static int write_loose_object(const struct object_id *oid, char *hdr,die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), ret); the_hash_algo->final_oid_fn (¶no_oid, &c); -if (!oideq (oid, ¶no_oid)) +if (!is_null_oid (oid) && !oideq (oid, ¶no_oid))object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1959
Quoted 10 lines
object-file.c +5
-0die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));close_loose_object (fd); +if (is_null_oid (oid)) { +int dirlen; + +oidcpy ((struct object_id *)oid, ¶no_oid); +loose_object_path (the_repository, &filename, oid);Why are we breaking the promise that "oid" is constant here? I tested locally with the below on top, and it seems to work (at least no tests broke). Isn't it preferrable to the cast & the caller having its "oid" changed?
object-file.c +0
-0@@ -1958,10 +1958,11 @@ int write_loose_object(const struct object_id *oid, char *hdr,close_loose_object (fd);if (is_null_oid (oid)) {
object-file.c int write_loose_object(const struct object_id *oid, char *hdr, · line 1968
Quoted 10 lines
object-file.c +3
-2+struct object_id oid2;int dirlen; -oidcpy ((struct object_id *)oid, ¶no_oid); -loose_object_path (the_repository, &filename, oid); +oidcpy (&oid2, ¶no_oid); +loose_object_path (the_repository, &filename, &oid2);/* We finally know the object path, and create the missing dir. */ dirlen =directory_size (filename.buf);
Maybe I should change the promise that "oid" is constant in "write_loose_object()".
The original write_object_file_flags() defines a variable "oid", and completes the calculation of the "oid" in "write_object_file_prepare()" which will be passed to "write_loose_object()".
If a null oid is maintained after calling "write_loose_object()", "--strict" will become meaningless, although it does not break existing test cases.
Re: [PATCH v4 2/5] object-file.c: handle undetermined oid in write_loose_object()
In reply to Han XinQuoted 24 lines · Han Xin
From: Han Xin <[email protected]>
When streaming a large blob object to "
write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)object-file.c +0
-0
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1897
Quoted 7 lines
object-file.c +3
-1@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,const void *buf;unsigned long len; -loose_object_path (the_repository, &filename, oid); +if (is_null_oid (oid)) { +/* When oid is not determined, save tmp file to odb path. */ +strbuf_reset (&filename);
Why re-use this & leak memory? An existing strbuf use in this function doesn't leak in the same way. Just release it as in the below patch on top (the ret v.s. err variable naming is a bit confused, maybe could do with a prep cleanup step.).
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1899
Quoted 2 lines
object-file.c +2
-0+strbuf_addstr (&filename, the_repository->objects->odb->path); +strbuf_addch (&filename,'/' );
And once we do that this could just become:
strbuf_addf($filename, "%s/", ...)
Is there's existing uses of this pattern, so mayb e not worth it, but it allows you to remove the braces on the if/else.
object-file.c +16 -10
@@ -1880,7 +1880,7 @@ int write_loose_object(const struct object_id *oid, char *hdr, int hdrlen, struct input_stream *in_stream, time_t mtime, unsigned flags) { - int fd, ret; + int fd, ret, err = 0;unsigned char compressed[4096 ];git_zstream stream;git_hash_ctx c;
@@ -1892,7 +1892,6 @@ int write_loose_object(const struct object_id *oid, char *hdr,if (is_null_oid (oid)) {/* When oid is not determined, save tmp file to odb path. */ -strbuf_reset (&filename);strbuf_addstr (&filename, the_repository->objects->odb->path);strbuf_addch (&filename,'/' ); }else {
@@ -1902,11 +1901,12 @@ int write_loose_object(const struct object_id *oid, char *hdr, fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {if (flags &HASH_SILENT ) -return -1 ; + err =-1 ;else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); + err =error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ());else -return error_errno (_ ("unable to create temporary file" )); + err =error_errno (_ ("unable to create temporary file" )); + goto cleanup; }/* Set it up */
@@ -1968,10 +1968,13 @@ int write_loose_object(const struct object_id *oid, char *hdr,struct strbuf dir =STRBUF_INIT ;strbuf_add (&dir, filename.buf, dirlen -1 );if (mkdir (dir.buf,0777 ) && errno !=EEXIST ) -return -1 ; -if (adjust_shared_perm (dir.buf)) -return -1 ; -strbuf_release (&dir); + err =-1 ; +else if (adjust_shared_perm (dir.buf)) + err =-1 ; +else +strbuf_release (&dir); +if (err <0 ) + goto cleanup; } }
@@ -1984,7 +1987,10 @@ int write_loose_object(const struct object_id *oid, char *hdr,warning_errno (_ ("failed utime() on %s" ), tmp_file.buf); } -return finalize_object_file (tmp_file.buf, filename.buf); + err =finalize_object_file (tmp_file.buf, filename.buf); +cleanup: +strbuf_release (&filename); +return err; }static int freshen_loose_object (const struct object_id *oid)
Re: [PATCH v4 2/5] object-file.c: handle undetermined oid in write_loose_object()
Quoted 108 lines · Ævar Arnfjörð Bjarmason
Quoted 24 lines · Han Xin
From: Han Xin <[email protected]>
When streaming a large blob object to "
write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)object-file.c +0
-0object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1897
Quoted 7 lines
object-file.c +3
-1@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,const void *buf;unsigned long len; -loose_object_path (the_repository, &filename, oid); +if (is_null_oid (oid)) { +/* When oid is not determined, save tmp file to odb path. */ +strbuf_reset (&filename);Why re-use this & leak memory? An existing strbuf use in this function doesn't leak in the same way. Just release it as in the below patch on top (the ret v.s. err variable naming is a bit confused, maybe could do with a prep cleanup step.).
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1899
Quoted 2 lines
object-file.c +2
-0+strbuf_addstr (&filename, the_repository->objects->odb->path); +strbuf_addch (&filename,'/' );And once we do that this could just become:
strbuf_addf($filename, "%s/", ...)Is there's existing uses of this pattern, so mayb e not worth it, but it allows you to remove the braces on the if/else.
object-file.c +12
-9@@ -1880,7 +1880,7 @@ int write_loose_object(const struct object_id *oid, char *hdr, int hdrlen, struct input_stream *in_stream, time_t mtime, unsigned flags) { - int fd, ret; + int fd, ret, err = 0;unsigned char compressed[4096 ];git_zstream stream;git_hash_ctx c;@@ -1892,7 +1892,6 @@ int write_loose_object(const struct object_id *oid, char *hdr,if (is_null_oid (oid)) {/* When oid is not determined, save tmp file to odb path. */ -strbuf_reset (&filename);strbuf_addstr (&filename, the_repository->objects->odb->path);strbuf_addch (&filename,'/' ); }else {@@ -1902,11 +1901,12 @@ int write_loose_object(const struct object_id *oid, char *hdr, fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {if (flags &HASH_SILENT ) -return -1 ; + err =-1 ;else if (errno ==EACCES ) -return error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ()); + err =error (_ ("insufficient permission for adding an object to repository database %s" ),get_object_directory ());else -return error_errno (_ ("unable to create temporary file" )); + err =error_errno (_ ("unable to create temporary file" )); + goto cleanup; }/* Set it up */ @@ -1968,10 +1968,13 @@ int write_loose_object(const struct object_id *oid, char *hdr,struct strbuf dir =STRBUF_INIT ;strbuf_add (&dir, filename.buf, dirlen -1 );if (mkdir (dir.buf,0777 ) && errno !=EEXIST ) -return -1 ; -if (adjust_shared_perm (dir.buf)) -return -1 ; -strbuf_release (&dir); + err =-1 ; +else if (adjust_shared_perm (dir.buf)) + err =-1 ; +else +strbuf_release (&dir); +if (err <0 ) + goto cleanup; } }@@ -1984,7 +1987,10 @@ int write_loose_object(const struct object_id *oid, char *hdr,warning_errno (_ ("failed utime() on %s" ), tmp_file.buf);
object-file.c int write_loose_object(const struct object_id *oid, char *hdr, · line 1996
Quoted 10 lines
object-file.c +4
-1} -return finalize_object_file (tmp_file.buf, filename.buf); + err =finalize_object_file (tmp_file.buf, filename.buf); +cleanup: +strbuf_release (&filename); +return err; }static int freshen_loose_object (const struct object_id *oid)
Yes, this will be much better. Will apply.
PATCH v4 3/5 object-file.c: read stream in a loop in write_loose_object()
In reply to Han XinFrom: Han Xin <[email protected]>
In order to prepare the stream version of "write_loose_object()", read the input stream in a loop in "write_loose_object()", so that we can feed the contents of large blob object to "write_loose_object()" using a small fixed buffer.
--- object-file.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
object-file.c +11 -6
@@ -1890,7 +1890,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,static struct strbuf tmp_file =STRBUF_INIT ;static struct strbuf filename =STRBUF_INIT ;const void *buf; -unsigned long len; +int flush =0 ;if (is_null_oid (oid)) {/* When oid is not determined, save tmp file to odb path. */
@@ -1925,18 +1925,23 @@ static int write_loose_object(const struct object_id *oid, char *hdr, the_hash_algo->update_fn (&c, hdr, hdrlen);/* Then the data itself.. */ - buf = in_stream->read (in_stream, &len); - stream.next_in = (void *)buf; - stream.avail_in = len;do {unsigned char *in0 = stream.next_in; - ret =git_deflate (&stream,Z_FINISH ); +if (!stream.avail_in) { + buf = in_stream->read (in_stream, &stream.avail_in); + stream.next_in = (void *)buf; + in0 = (unsigned char *)buf; +/* All data has been read. */ +if (in_stream->size + hdrlen == stream.total_in + stream.avail_in) + flush =Z_FINISH ; + } + ret =git_deflate (&stream, flush); the_hash_algo->update_fn (&c, in0, stream.next_in - in0);if (write_buffer (fd, compressed, stream.next_out - compressed) <0 )die (_ ("unable to write loose object file" )); stream.next_out = compressed; stream.avail_out =sizeof (compressed); - }while (ret ==Z_OK ); + }while (ret ==Z_OK || ret ==Z_BUF_ERROR );if (ret !=Z_STREAM_END ) die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
Signature · 1 line
2.34.0
PATCH v4 4/5 unpack-objects.c: add dry_run mode for get_data()
In reply to Han XinFrom: Han Xin <[email protected]>
In dry_run mode, "get_data()" is used to verify the inflation of data, and the returned buffer will not be used at all and will be freed immediately. Even in dry_run mode, it is dangerous to allocate a full-size buffer for a large blob object. Therefore, only allocate a low memory footprint when calling "get_data()" in dry_run mode.
--- builtin/unpack-objects.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
builtin/unpack-objects.c +12 -6
@@ -96,15 +96,16 @@ static void use(int bytes)display_throughput (progress, consumed_bytes); } -static void *get_data (unsigned long size) +static void *get_data (unsigned long size,int dry_run) {git_zstream stream; -void *buf =xmallocz (size); +unsigned long bufsize = dry_run ?4096 : size; +void *buf =xmallocz (bufsize);memset (&stream,0 ,sizeof (stream)); stream.next_out = buf; - stream.avail_out = size; + stream.avail_out = bufsize; stream.next_in =fill (1 ); stream.avail_in = len;git_inflate_init (&stream);
@@ -124,6 +125,11 @@ static void *get_data(unsigned long size) } stream.next_in =fill (1 ); stream.avail_in = len; +if (dry_run) { +/* reuse the buffer in dry_run mode */ + stream.next_out = buf; + stream.avail_out = bufsize; + } }git_inflate_end (&stream);return buf;
@@ -323,7 +329,7 @@ static void added_object(unsigned nr, enum object_type type,static void unpack_non_delta_entry (enum object_type type,unsigned long size,unsigned nr) { -void *buf =get_data (size); +void *buf =get_data (size, dry_run);if (!dry_run && buf)write_object (nr, type, buf, size);
@@ -357,7 +363,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,if (type ==OBJ_REF_DELTA ) {oidread (&base_oid,fill (the_hash_algo->rawsz));use (the_hash_algo->rawsz); - delta_data =get_data (delta_size); + delta_data =get_data (delta_size, dry_run);if (dry_run || !delta_data) {free (delta_data);return ;
@@ -396,7 +402,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,if (base_offset <=0 || base_offset >= obj_list[nr].offset)die ("offset value out of bound for delta base object" ); - delta_data =get_data (delta_size); + delta_data =get_data (delta_size, dry_run);if (dry_run || !delta_data) {free (delta_data);return ;
Signature · 1 line
2.34.0
Re: [PATCH v4 4/5] unpack-objects.c: add dry_run mode for get_data()
builtin/unpack-objects.c static void use(int bytes) · line 103
Quoted 2 lines · Han Xin
builtin/unpack-objects.c +2
-0+unsigned long bufsize = dry_run ?4096 : size; +void *buf =xmallocz (bufsize);
It's probably nothing, but in your CL you note that you changed another hardcoding from 4k to 8k, should this one still be 4k?
It's probably fine, just wondering...
Re: [PATCH v4 4/5] unpack-objects.c: add dry_run mode for get_data()
Quoted 11 lines · Ævar Arnfjörð Bjarmason
builtin/unpack-objects.c static void use(int bytes) · line 103
Quoted 2 lines · Han Xin
builtin/unpack-objects.c +2
-0+unsigned long bufsize = dry_run ?4096 : size; +void *buf =xmallocz (bufsize);It's probably nothing, but in your CL you note that you changed another hardcoding from 4k to 8k, should this one still be 4k?
It's probably fine, just wondering...
Yes, I think this is an omission from my work.
PATCH v4 5/5 unpack-objects: unpack_non_delta_entry() read data in a stream
In reply to Han XinFrom: Han Xin <[email protected]>
We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.
By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".
However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "big_file_threshold" in zstream. See the following benchmarks:
hyperfine \
--setup \
'if ! test -d scalar.git; then git clone --bare https://github.com/microsoft/scalar.git; cp scalar.git/objects/pack/*.pack small.pack; fi' \
--prepare 'rm -rf dest.git && git init --bare dest.git' \
-n 'old' 'git -C dest.git unpack-objects <small.pack' \
-n 'new' 'new/git -C dest.git unpack-objects <small.pack' \
-n 'new (small threshold)' \
'new/git -c core.bigfilethreshold=16k -C dest.git unpack-objects <small.pack'
Benchmark 1: old
Time (mean ± σ): 6.075 s ± 0.069 s [User: 5.047 s, System: 0.991 s]
Range (min … max): 6.018 s … 6.189 s 10 runs
Benchmark 2: new
Time (mean ± σ): 6.090 s ± 0.033 s [User: 5.075 s, System: 0.976 s]
Range (min … max): 6.030 s … 6.142 s 10 runs
Benchmark 3: new (small threshold)
Time (mean ± σ): 6.755 s ± 0.029 s [User: 5.150 s, System: 1.560 s]
Range (min … max): 6.711 s … 6.809 s 10 runs
Summary
'old' ran
1.00 ± 0.01 times faster than 'new'
1.11 ± 0.01 times faster than 'new (small threshold)'--- builtin/unpack-objects.c | 77 ++++++++++++++++++++++++++++- object-file.c | 6 +-- object-store.h | 4 ++ t/t5590-unpack-non-delta-objects.sh | 76 ++++++++++++++++++++++++++++ 4 files changed, 159 insertions(+), 4 deletions(-) create mode 100755 t/t5590-unpack-non-delta-objects.sh
builtin/unpack-objects.c +76 -1
@@ -326,11 +326,86 @@ static void added_object(unsigned nr, enum object_type type, } } +struct input_zstream_data { +git_zstream *zstream; +unsigned char buf[8192 ]; +int status; +}; + +static const void *feed_input_zstream (struct input_stream *in_stream,unsigned long *readlen) +{ +struct input_zstream_data *data = in_stream->data; +git_zstream *zstream = data->zstream; +void *in =fill (1 ); + +if (!len || data->status ==Z_STREAM_END ) { + *readlen =0 ; +return NULL ; + } + + zstream->next_out = data->buf; + zstream->avail_out =sizeof (data->buf); + zstream->next_in = in; + zstream->avail_in = len; + + data->status =git_inflate (zstream,0 ); +use (len - zstream->avail_in); + *readlen =sizeof (data->buf) - zstream->avail_out; + +return data->buf; +} + +static void write_stream_blob (unsigned nr,unsigned long size) +{ +char hdr[32 ]; +int hdrlen; +git_zstream zstream; +struct input_zstream_data data; +struct input_stream in_stream = { + .read = feed_input_zstream, + .data = &data, + .size = size, + }; +struct object_id *oid = &obj_list[nr].oid; +int ret; + +memset (&zstream,0 ,sizeof (zstream)); +memset (&data,0 ,sizeof (data)); + data.zstream = &zstream; +git_inflate_init (&zstream); + +/* Generate the header */ + hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX,type_name (OBJ_BLOB ), (uintmax_t )size) +1 ; + +if ((ret =write_loose_object (oid, hdr, hdrlen, &in_stream,0 ,0 ))) +die (_ ("failed to write object in stream %d" ), ret); + +if (zstream.total_out != size || data.status !=Z_STREAM_END ) +die (_ ("inflate returned %d" ), data.status); +git_inflate_end (&zstream); + +if (strict && !dry_run) { +struct blob *blob =lookup_blob (the_repository, oid); +if (blob) + blob->object.flags |=FLAG_WRITTEN ; +else +die ("invalid blob object from stream" ); + } + obj_list[nr].obj =NULL ; +} +static void unpack_non_delta_entry (enum object_type type,unsigned long size,unsigned nr) { -void *buf =get_data (size, dry_run); +void *buf; + +/* Write large blob in stream without allocating full buffer. */ +if (!dry_run && type ==OBJ_BLOB && size > big_file_threshold) { +write_stream_blob (nr, size); +return ; + } + buf =get_data (size, dry_run);if (!dry_run && buf)write_object (nr, type, buf, size);else
object-file.c +3 -3
@@ -1878,9 +1878,9 @@ static const void *feed_simple_input_stream(struct input_stream *in_stream, unsireturn data->buf; } -static int write_loose_object (const struct object_id *oid,char *hdr, -int hdrlen,struct input_stream *in_stream, -time_t mtime,unsigned flags) +int write_loose_object (const struct object_id *oid,char *hdr, +int hdrlen,struct input_stream *in_stream, +time_t mtime,unsigned flags) {int fd, ret;unsigned char compressed[4096 ];
object-store.h +4 -0
@@ -229,6 +229,10 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, unsigned long len, const char *type, struct object_id *oid); +int write_loose_object (const struct object_id *oid,char *hdr, +int hdrlen,struct input_stream *in_stream, +time_t mtime,unsigned flags); +int write_object_file_flags (const void *buf,unsigned long len,const char *type,struct object_id *oid,unsigned flags);
t/t5590-unpack-non-delta-objects.sh new file +76 -0
76 lines not on this page · raw message
Signature · 1 line
2.34.0
Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream
Quoted 30 lines · Han Xin
From: Han Xin <[email protected]>
We used to call "
get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.By implementing a zstream version of
input_streaminterface, we can use a small fixed buffer for "unpack_non_delta_entry()".However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "
big_file_threshold" in zstream. See the following benchmarks:hyperfine \ --setup \ 'if ! test -d scalar.git; then git clone --bare https://github.com/microsoft/scalar.git; cp scalar.git/objects/pack/*.pack small.pack; fi' \ --prepare 'rm -rf dest.git && git init --bare dest.git' \ -n 'old' 'git -C dest.git unpack-objects <small.pack' \ -n 'new' 'new/git -C dest.git unpack-objects <small.pack' \ -n 'new (small threshold)' \ 'new/git -c core.bigfilethreshold=16k -C dest.git unpack-objects <small.pack' Benchmark 1: old Time (mean ± σ): 6.075 s ± 0.069 s [User: 5.047 s, System: 0.991 s] Range (min … max): 6.018 s … 6.189 s 10 runs Benchmark 2: new Time (mean ± σ): 6.090 s ± 0.033 s [User: 5.075 s, System: 0.976 s] Range (min … max): 6.030 s … 6.142 s 10 runs
Quoted 8 lines
Benchmark 3: new (small threshold)
Time (mean ± σ): 6.755 s ± 0.029 s [User: 5.150 s, System: 1.560 s]
Range (min … max): 6.711 s … 6.809 s 10 runs
Summary
'old' ran
1.00 ± 0.01 times faster than 'new'
1.11 ± 0.01 times faster than 'new (small threshold)'So before we wrote used core.bigfilethreshold for two things (or more?):
Whether we show a diff for it (we mark it "binary") and whether it's split into a loose object.
Now it's three things, we've added a "this is a threshold when we'll stream the object" to that.
Might it make sense to squash something like this in, so we can have our cake & eat it too?
With this I get, where HEAD~0 is this change:
Summary
'./git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~0' ran
1.00 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~1' 1.00 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'origin/master' 1.01 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~0' 1.06 ± 0.14 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'origin/master' 1.20 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~1'
I.e. it's 5% slower, not 20% (haven't looked into why), but we'll not stream out 16k..128MB objects (maybe the repo has even bigger ones?)
Documentation/config/core.txt +11 -0
@@ -424,6 +424,17 @@ be delta compressed, but larger binary media files won't be. + Common unit suffixes of 'k', 'm', or 'g' are supported. +core.bigFileStreamingThreshold:: + Files larger than this will be streamed out to a temporary + object file while being hashed, which will when be renamed + in-place to a loose object, particularly if the + `core.bigFileThreshold' setting dictates that they're always + written out as loose objects. ++ +Default is 128 MiB on all platforms. ++ +Common unit suffixes of 'k', 'm', or 'g' are supported. + core.excludesFile:: Specifies the pathname to the file that contains patterns to describe paths that are not meant to be tracked, in addition
builtin/unpack-objects.c +1 -1
@@ -400,7 +400,7 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,void *buf;/* Write large blob in stream without allocating full buffer. */ -if (!dry_run && type ==OBJ_BLOB && size > big_file_threshold) { +if (!dry_run && type ==OBJ_BLOB && size > big_file_streaming_threshold) {write_stream_blob (nr, size);return ; }
cache.h +1 -0
@@ -964,6 +964,7 @@ extern size_t packed_git_window_size;extern size_t packed_git_limit;extern size_t delta_base_cache_limit;extern unsigned long big_file_threshold; +extern unsigned long big_file_streaming_threshold;extern unsigned long pack_size_limit_cfg;/*
config.c +5 -0
@@ -1408,6 +1408,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.bigfilestreamingthreshold" )) { + big_file_streaming_threshold =git_config_ulong (var, value); +return 0 ; + } +if (!strcmp (var,"core.packedgitlimit" )) { packed_git_limit =git_config_ulong (var, value);return 0 ;
environment.c +1 -0
@@ -46,6 +46,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ;size_t delta_base_cache_limit =96 *1024 *1024 ;unsigned long big_file_threshold =512 *1024 *1024 ; +unsigned long big_file_streaming_threshold =128 *1024 *1024 ;int pager_use_color =1 ;const char *editor_program;const char *askpass_program;
Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream
Quoted 134 lines · Ævar Arnfjörð Bjarmason
Quoted 30 lines · Han Xin
From: Han Xin <[email protected]>
We used to call "
get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.By implementing a zstream version of
input_streaminterface, we can use a small fixed buffer for "unpack_non_delta_entry()".However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "
big_file_threshold" in zstream. See the following benchmarks:hyperfine \ --setup \ 'if ! test -d scalar.git; then git clone --bare https://github.com/microsoft/scalar.git; cp scalar.git/objects/pack/*.pack small.pack; fi' \ --prepare 'rm -rf dest.git && git init --bare dest.git' \ -n 'old' 'git -C dest.git unpack-objects <small.pack' \ -n 'new' 'new/git -C dest.git unpack-objects <small.pack' \ -n 'new (small threshold)' \ 'new/git -c core.bigfilethreshold=16k -C dest.git unpack-objects <small.pack' Benchmark 1: old Time (mean ± σ): 6.075 s ± 0.069 s [User: 5.047 s, System: 0.991 s] Range (min … max): 6.018 s … 6.189 s 10 runs Benchmark 2: new Time (mean ± σ): 6.090 s ± 0.033 s [User: 5.075 s, System: 0.976 s] Range (min … max): 6.030 s … 6.142 s 10 runsQuoted 8 lines
Benchmark 3: new (small threshold) Time (mean ± σ): 6.755 s ± 0.029 s [User: 5.150 s, System: 1.560 s] Range (min … max): 6.711 s … 6.809 s 10 runs Summary 'old' ran 1.00 ± 0.01 times faster than 'new' 1.11 ± 0.01 times faster than 'new (small threshold)'So before we wrote used core.bigfilethreshold for two things (or more?):
Whether we show a diff for it (we mark it "binary") and whether it's split into a loose object.Now it's three things, we've added a "this is a threshold when we'll stream the object" to that.
Might it make sense to squash something like this in, so we can have our cake & eat it too?
With this I get, where HEAD~0 is this change:
Summary
'./git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~0' ran
1.00 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~1' 1.00 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'origin/master' 1.01 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~0' 1.06 ± 0.14 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'origin/master' 1.20 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~1'I.e. it's 5% slower, not 20% (haven't looked into why), but we'll not stream out 16k..128MB objects (maybe the repo has even bigger ones?)
Documentation/config/core.txt +11
-0@@ -424,6 +424,17 @@ be delta compressed, but larger binary media files won't be. + Common unit suffixes of 'k', 'm', or 'g' are supported. +core.bigFileStreamingThreshold:: + Files larger than this will be streamed out to a temporary + object file while being hashed, which will when be renamed + in-place to a loose object, particularly if the + `core.bigFileThreshold' setting dictates that they're always + written out as loose objects. ++ +Default is 128 MiB on all platforms. ++ +Common unit suffixes of 'k', 'm', or 'g' are supported. + core.excludesFile:: Specifies the pathname to the file that contains patterns to describe paths that are not meant to be tracked, in additionbuiltin/unpack-objects.c +1
-1@@ -400,7 +400,7 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,void *buf;/* Write large blob in stream without allocating full buffer. */ -if (!dry_run && type ==OBJ_BLOB && size > big_file_threshold) { +if (!dry_run && type ==OBJ_BLOB && size > big_file_streaming_threshold) {write_stream_blob (nr, size);return ; }cache.h +1
-0@@ -964,6 +964,7 @@ extern size_t packed_git_window_size;extern size_t packed_git_limit;extern size_t delta_base_cache_limit;extern unsigned long big_file_threshold; +extern unsigned long big_file_streaming_threshold;extern unsigned long pack_size_limit_cfg;/* config.c +5
-0@@ -1408,6 +1408,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.bigfilestreamingthreshold" )) { + big_file_streaming_threshold =git_config_ulong (var, value); +return 0 ; + } +if (!strcmp (var,"core.packedgitlimit" )) { packed_git_limit =git_config_ulong (var, value);return 0 ;environment.c +0
-0
environment.c size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; · line 52
Quoted 7 lines
environment.c +1
-0@@ -46,6 +46,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ;size_t delta_base_cache_limit =96 *1024 *1024 ;unsigned long big_file_threshold =512 *1024 *1024 ; +unsigned long big_file_streaming_threshold =128 *1024 *1024 ;int pager_use_color =1 ;const char *editor_program;const char *askpass_program;
I'm not sure if we need an additional "core.bigFileStreamingThreshold" here, because "core.bigFileThreshold" has been widely used in "index-pack", "read_object" and so on.
In the test case which uses "core.bigFileStreamingThreshold" instead of "core.bigFileThreshold", I found the test case execution failed because of "fsck", who tried to allocate 15MB of memory.
In the process of "fsck_loose()", "read_loose_object()" will be called, which contains the following content:
if (*oi->typep == OBJ_BLOB && *size> big_file_threshold) {
if (check_stream_oid(&stream, hdr, *size, path, expected_oid) <0)
goto out;
} else {
/* this will allocate 15MB of memory */
*contents = unpack_loose_rest(&stream, hdr, *size, expected_oid);
...
}The same case can be found in "unpack_entry_data()":
static char fixed_buf[8192];
...
if (type == OBJ_BLOB && size > big_file_threshold)
buf = fixed_buf;
else
buf = xmallocz(size);...
Although I know that setting a "core.bigfilethreshold" smaller than the default value on the server side does not help me prevent users from creating large delta objects on the client side, it can still effectively help me reduce the Memory allocation in "receive-pack".
If this is not the correct way to use "core.bigfilethreshold", maybe you can share some better solutions to me, if you want.
Thanks.
-Han Xin
Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream
In reply to Han XinQuoted 16 lines · Han Xin
t/t5590-unpack-non-delta-objects.sh new file +10
-0@@ -0,0 +1,76 @@ +#!/bin/sh +# +# Copyright (c) 2021 Han Xin +# + +test_description='Test unpack-objects when receive pack' + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME +
t/t5590-unpack-non-delta-objects.sh line 20
Quoted 10 lines
t/t5590-unpack-non-delta-objects.sh new file +10
-0+. ./test-lib.sh + +test_expect_success "create commit with big blobs (1.5 MB)" ' + test-tool genrandom foo 1500000 >big-blob && + test_commit --append foo big-blob && + test-tool genrandom bar 1500000 >big-blob && + test_commit --append bar big-blob && + ( + cd .git && + find objects/?? -type f | sort
...are thse...
t/t5590-unpack-non-delta-objects.sh line 22
Quoted 2 lines
t/t5590-unpack-non-delta-objects.sh new file +2
-0+ ) >expect && + PACK=$(echo main | git pack-objects --progress --revs test)
Is --progress needed?
t/t5590-unpack-non-delta-objects.sh line 33
Quoted 11 lines
t/t5590-unpack-non-delta-objects.sh new file +11
-0+' + +test_expect_success 'setup GIT_ALLOC_LIMIT to 1MB' ' + GIT_ALLOC_LIMIT=1m && + export GIT_ALLOC_LIMIT +' + +test_expect_success 'prepare dest repository' ' + git init --bare dest.git && + git -C dest.git config core.bigFileThreshold 2m && + git -C dest.git config receive.unpacklimit 100
I think it would be better to just (could roll this into a function):
test_when_finished "rm -rf dest.git" && git init dest.git && git -C dest.git config ...
Then you can use it with e.g. --run=3-4 and not have it error out because of skipped setup.
A lot of our tests fail like that, but in this case fixing it seems trivial.
t/t5590-unpack-non-delta-objects.sh line 38
Quoted 5 lines
t/t5590-unpack-non-delta-objects.sh new file +5
-0+' + +test_expect_success 'fail to unpack-objects: cannot allocate' ' + test_must_fail git -C dest.git unpack-objects <test-$PACK.pack 2>err && + test_i18ngrep "fatal: attempting to allocate" err &&
nit: just "grep", not "test_i18ngrep"
t/t5590-unpack-non-delta-objects.sh line 41
Quoted 3 lines
t/t5590-unpack-non-delta-objects.sh new file +3
-0+ ( + cd dest.git && + find objects/?? -type f | sort
..."find" needed over just globbing?:
obj=$(echo objects/*/*)
?
Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream
Quoted 33 lines · Ævar Arnfjörð Bjarmason
Quoted 16 lines · Han Xin
t/t5590-unpack-non-delta-objects.sh new file +10
-0@@ -0,0 +1,76 @@ +#!/bin/sh +# +# Copyright (c) 2021 Han Xin +# + +test_description='Test unpack-objects when receive pack' + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME +t/t5590-unpack-non-delta-objects.sh line 20
Quoted 10 lines
t/t5590-unpack-non-delta-objects.sh new file +10
-0+. ./test-lib.sh + +test_expect_success "create commit with big blobs (1.5 MB)" ' + test-tool genrandom foo 1500000 >big-blob && + test_commit --append foo big-blob && + test-tool genrandom bar 1500000 >big-blob && + test_commit --append bar big-blob && + ( + cd .git && + find objects/?? -type f | sort...are thse...
Quoted 5 lines
t/t5590-unpack-non-delta-objects.sh line 22
Quoted 2 lines
t/t5590-unpack-non-delta-objects.sh new file +2
-0+ ) >expect && + PACK=$(echo main | git pack-objects --progress --revs test)Is --progress needed?
"--progress" is not necessary.
Quoted 18 lines
t/t5590-unpack-non-delta-objects.sh line 33
Quoted 11 lines
t/t5590-unpack-non-delta-objects.sh new file +11
-0+' + +test_expect_success 'setup GIT_ALLOC_LIMIT to 1MB' ' + GIT_ALLOC_LIMIT=1m && + export GIT_ALLOC_LIMIT +' + +test_expect_success 'prepare dest repository' ' + git init --bare dest.git && + git -C dest.git config core.bigFileThreshold 2m && + git -C dest.git config receive.unpacklimit 100I think it would be better to just (could roll this into a function):
test_when_finished "rm -rf dest.git" && git init dest.git && git -C dest.git config ...
Quoted 7 lines
Then you can use it with e.g. --run=3-4 and not have it error out because of skipped setup.
A lot of our tests fail like that, but in this case fixing it seems trivial.
OK, I will take it.
Quoted 13 lines
t/t5590-unpack-non-delta-objects.sh line 38
Quoted 6 lines
t/t5590-unpack-non-delta-objects.sh new file +5
-0+' + +test_expect_success 'fail to unpack-objects: cannot allocate' ' + test_must_fail git -C dest.git unpack-objects <test-$PACK.pack 2>err && + test_i18ngrep "fatal: attempting to allocate" err &&nit: just "grep", not "
test_i18ngrep"t/t5590-unpack-non-delta-objects.sh line 41
Quoted 3 lines
t/t5590-unpack-non-delta-objects.sh new file +3
-0+ ( + cd dest.git && + find objects/?? -type f | sort
Quoted 5 lines
..."find" needed over just globbing?:
obj=$(echo objects/*/*)?
I tried to use "echo" instead of "find". It works well on my personal computer, but fails due to the "info/commit-graph" generated when CI on Github.
So it seems that ".git/objects/??" will be more rigorous?
Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream
In reply to Han XinQuoted 30 lines · Han Xin
From: Han Xin <[email protected]>
[..]builtin/unpack-objects.c +28
-0+static void write_stream_blob (unsigned nr,unsigned long size) +{ +char hdr[32 ]; +int hdrlen; +git_zstream zstream; +struct input_zstream_data data; +struct input_stream in_stream = { + .read = feed_input_zstream, + .data = &data, + .size = size, + }; +struct object_id *oid = &obj_list[nr].oid; +int ret; + +memset (&zstream,0 ,sizeof (zstream)); +memset (&data,0 ,sizeof (data)); + data.zstream = &zstream; +git_inflate_init (&zstream); + +/* Generate the header */ + hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX,type_name (OBJ_BLOB ), (uintmax_t )size) +1 ; + +if ((ret =write_loose_object (oid, hdr, hdrlen, &in_stream,0 ,0 ))) +die (_ ("failed to write object in stream %d" ), ret); + +if (zstream.total_out != size || data.status !=Z_STREAM_END ) +die (_ ("inflate returned %d" ), data.status); +git_inflate_end (&zstream);
builtin/unpack-objects.c static void added_object(unsigned nr, enum object_type type, · line 395
Quoted 10 lines
builtin/unpack-objects.c +10
-0+ +if (strict && !dry_run) { +struct blob *blob =lookup_blob (the_repository, oid); +if (blob) + blob->object.flags |=FLAG_WRITTEN ; +else +die ("invalid blob object from stream" ); + } + obj_list[nr].obj =NULL ; +}
Just a side-note, I think (but am not 100% sure) that these existing occurances aren't needed due to our use of CALLOC_ARRAY():
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index 4a9466295ba..00b349412c5 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -248,7 +248,6 @@ static void write_object(unsigned nr, enum object_type type,
die("failed to write object");
added_object(nr, type, buf, size);
free(buf);
- obj_list[nr].obj = NULL;
} else if (type == OBJ_BLOB) {
struct blob *blob;
if (write_object_file(buf, size, type_name(type),
@@ -262,7 +261,6 @@ static void write_object(unsigned nr, enum object_type type,
blob->object.flags |= FLAG_WRITTEN;
else
die("invalid blob object");
- obj_list[nr].obj = NULL;
} else {
struct object *obj;
int eaten;The reason I'm noting it is that the same seems to be true of your new addition here. I.e. are these assignments to NULL needed?
Anyway, the reason I started poking at this it tha this write_stream_blob() seems to duplicate much of write_object(). AFAICT only the writing part is really different, the part where we lookup_blob() after, set FLAG_WRITTEN etc. is all the same.
Why can't we call write_object() here?
The obvious answer seems to be that the call to write_object_file() isn't prepared to do the sort of streaming that you want, so instead you're bypassing it and calling write_loose_object() directly.
I haven't tried this myself, but isn't a better and cleaner approach here to not add another meaning to what is_null_oid() means, but to just add a HASH_STREAM flag that'll get passed down as "unsigned flags" to write_loose_object()? See FLAG_BITS in object.h.
Then the "obj_list[nr].obj" here could also become "obj_list[nr].obj.flags |= (1u<<12)" or whatever (but that wouldn't strictly be needed I think.
But by adding the "HASH_STREAM" flag you could I think stop duplicating the "Generate the header" etc. here and call write_object_file_flags().
I don't so much care about how it's done within unpack-objects.c, but not having another meaning to is_null_oid() in play would be really nice, and it this case it seems entirely avoidable.
Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream
Quoted 92 lines · Ævar Arnfjörð Bjarmason
Quoted 30 lines · Han Xin
From: Han Xin <[email protected]>
[..]builtin/unpack-objects.c +28
-0+static void write_stream_blob (unsigned nr,unsigned long size) +{ +char hdr[32 ]; +int hdrlen; +git_zstream zstream; +struct input_zstream_data data; +struct input_stream in_stream = { + .read = feed_input_zstream, + .data = &data, + .size = size, + }; +struct object_id *oid = &obj_list[nr].oid; +int ret; + +memset (&zstream,0 ,sizeof (zstream)); +memset (&data,0 ,sizeof (data)); + data.zstream = &zstream; +git_inflate_init (&zstream); + +/* Generate the header */ + hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX,type_name (OBJ_BLOB ), (uintmax_t )size) +1 ; + +if ((ret =write_loose_object (oid, hdr, hdrlen, &in_stream,0 ,0 ))) +die (_ ("failed to write object in stream %d" ), ret); + +if (zstream.total_out != size || data.status !=Z_STREAM_END ) +die (_ ("inflate returned %d" ), data.status); +git_inflate_end (&zstream);builtin/unpack-objects.c static void added_object(unsigned nr, enum object_type type, · line 395
Quoted 10 lines
builtin/unpack-objects.c +10
-0+ +if (strict && !dry_run) { +struct blob *blob =lookup_blob (the_repository, oid); +if (blob) + blob->object.flags |=FLAG_WRITTEN ; +else +die ("invalid blob object from stream" ); + } + obj_list[nr].obj =NULL ; +}Just a side-note, I think (but am not 100% sure) that these existing occurances aren't needed due to our use of
CALLOC_ARRAY():diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c index 4a9466295ba..00b349412c5 100644 --- a/builtin/unpack-objects.c +++ b/builtin/unpack-objects.c @@ -248,7 +248,6 @@ static void write_object(unsigned nr, enum object_type type, die("failed to write object"); added_object(nr, type, buf, size); free(buf); - obj_list[nr].obj = NULL; } else if (type == OBJ_BLOB) { struct blob *blob; if (write_object_file(buf, size, type_name(type), @@ -262,7 +261,6 @@ static void write_object(unsigned nr, enum object_type type, blob->object.flags |= FLAG_WRITTEN; else die("invalid blob object"); - obj_list[nr].obj = NULL; } else { struct object *obj; int eaten;The reason I'm noting it is that the same seems to be true of your new addition here. I.e. are these assignments to NULL needed?
Anyway, the reason I started poking at this it tha this
write_stream_blob()seems to duplicate much ofwrite_object(). AFAICT only the writing part is really different, the part where welookup_blob()after, setFLAG_WRITTENetc. is all the same.Why can't we call
write_object()here?The obvious answer seems to be that the call to
write_object_file()isn't prepared to do the sort of streaming that you want, so instead you're bypassing it and callingwrite_loose_object()directly.I haven't tried this myself, but isn't a better and cleaner approach here to not add another meaning to what
is_null_oid()means, but to just add aHASH_STREAMflag that'll get passed down as "unsigned flags" towrite_loose_object()? SeeFLAG_BITSinobject.h.Then the "obj_list[nr].obj" here could also become "obj_list[nr].obj.flags |= (1u<<12)" or whatever (but that wouldn't strictly be needed I think.
Quoted 6 lines
But by adding the "
HASH_STREAM" flag you could I think stop duplicating the "Generate the header" etc. here and callwrite_object_file_flags().I don't so much care about how it's done within
unpack-objects.c, but not having another meaning tois_null_oid()in play would be really nice, and it this case it seems entirely avoidable.
I did refactor it according to your suggestions in my next patch version. Using a HASH_STREAM tag is indeed a better way to deal with it, and it can also reduce my refactor to the original contents.
Thanks.
-Han Xin
PATCH v3 1/5 object-file: refactor write_loose_object() to read buffer from stream
In reply to Han XinFrom: Han Xin <[email protected]>
We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.
This can be improved by feeding data to "write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.
--- object-file.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++---- object-store.h | 5 +++++ 2 files changed, 51 insertions(+), 4 deletions(-)
object-file.c +46 -4
@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)return fd; } +struct simple_input_stream_data { +const void *buf; +unsigned long len; +}; + +static const void *feed_simple_input_stream (struct input_stream *in_stream,unsigned long *len) +{ +struct simple_input_stream_data *data = in_stream->data; + +if (data->len ==0 ) { + *len =0 ; +return NULL ; + } + *len = data->len; + data->len =0 ; +return data->buf; +} +static int write_loose_object (const struct object_id *oid,char *hdr, -int hdrlen,const void *buf,unsigned long len, +int hdrlen,struct input_stream *in_stream,time_t mtime,unsigned flags) {int fd, ret;
@@ -1871,6 +1889,8 @@ static int write_loose_object(const struct object_id *oid, char *hdr,struct object_id parano_oid;static struct strbuf tmp_file =STRBUF_INIT ;static struct strbuf filename =STRBUF_INIT ; +const void *buf; +unsigned long len;loose_object_path (the_repository, &filename, oid);
@@ -1898,6 +1918,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr, the_hash_algo->update_fn (&c, hdr, hdrlen);/* Then the data itself.. */ + buf = in_stream->read (in_stream, &len); stream.next_in = (void *)buf; stream.avail_in = len; do {
@@ -1960,6 +1981,13 @@ int write_object_file_flags(const void *buf, unsigned long len, {char hdr[MAX_HEADER_LEN ];int hdrlen =sizeof (hdr); +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + };/* Normally if we have it in the pack then we do not bother writing * it out into .git/objects/??/?{38} file.
@@ -1968,7 +1996,7 @@ int write_object_file_flags(const void *buf, unsigned long len, &hdrlen);if (freshen_packed_object (oid) ||freshen_loose_object (oid))return 0 ; -return write_loose_object (oid, hdr, hdrlen, buf, len,0 , flags); +return write_loose_object (oid, hdr, hdrlen, &in_stream,0 , flags); } int hash_object_file_literally(const void *buf, unsigned long len,
@@ -1977,6 +2005,13 @@ int hash_object_file_literally(const void *buf, unsigned long len, {char *header;int hdrlen, status =0 ; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + };/* type string, SP, %lu of the length plus NUL must fit this */ hdrlen =strlen (type) +MAX_HEADER_LEN ;
@@ -1988,7 +2023,7 @@ int hash_object_file_literally(const void *buf, unsigned long len, goto cleanup;if (freshen_packed_object (oid) ||freshen_loose_object (oid)) goto cleanup; - status =write_loose_object (oid, header, hdrlen, buf, len,0 ,0 ); + status =write_loose_object (oid, header, hdrlen, &in_stream,0 ,0 ); cleanup:free (header);
@@ -2003,14 +2038,21 @@ int force_object_loose(const struct object_id *oid, time_t mtime)char hdr[MAX_HEADER_LEN ];int hdrlen;int ret; +struct simple_input_stream_data data; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = &data, + };if (has_loose_object (oid))return 0 ; buf =read_object (the_repository, oid, &type, &len);if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid)); + data.buf = buf; + data.len = len; hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; - ret =write_loose_object (oid, hdr, hdrlen, buf, len, mtime,0 ); + ret =write_loose_object (oid, hdr, hdrlen, &in_stream, mtime,0 );free (buf);return ret;
object-store.h +5 -0
@@ -34,6 +34,11 @@ struct object_directory {char *path; }; +struct input_stream { +const void *(*read)(struct input_stream *,unsigned long *len); +void *data; +}; + KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq)
Signature · 1 line
2.34.0.6.g676eedc724
Re: [PATCH v3 1/5] object-file: refactor write_loose_object() to read buffer from stream
Quoted 10 lines · Han Xin
From: Han Xin <[email protected]>
We used to call "
get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.This can be improved by feeding data to "
write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.
Possibly a stupid question (not a review).
How does this compare with "struct git_istream" implemented for a few existing codepaths? It seems that the existing users are pack-objects, index-pack and archive and all of them use the interface to obtain data given an object name without having to grab everything in core at once.
If we are adding a new streaming interface to go in the opposite direction, i.e. from the working tree data to object store, I would understand it as a complementary interface (but then I suspect there is a half of it already in bulk-checkin API), but I am not sure how this new thing fits in the larger picture.
Quoted 140 lines
Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++---- object-store.h | 5 +++++ 2 files changed, 51 insertions(+), 4 deletions(-)object-file.c +46
-4@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)return fd; } +struct simple_input_stream_data { +const void *buf; +unsigned long len; +}; + +static const void *feed_simple_input_stream (struct input_stream *in_stream,unsigned long *len) +{ +struct simple_input_stream_data *data = in_stream->data; + +if (data->len ==0 ) { + *len =0 ; +return NULL ; + } + *len = data->len; + data->len =0 ; +return data->buf; +} +static int write_loose_object (const struct object_id *oid,char *hdr, -int hdrlen,const void *buf,unsigned long len, +int hdrlen,struct input_stream *in_stream,time_t mtime,unsigned flags) {int fd, ret;@@ -1871,6 +1889,8 @@ static int write_loose_object(const struct object_id *oid, char *hdr,struct object_id parano_oid;static struct strbuf tmp_file =STRBUF_INIT ;static struct strbuf filename =STRBUF_INIT ; +const void *buf; +unsigned long len;loose_object_path (the_repository, &filename, oid);@@ -1898,6 +1918,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr, the_hash_algo->update_fn (&c, hdr, hdrlen);/* Then the data itself.. */ + buf = in_stream->read (in_stream, &len); stream.next_in = (void *)buf; stream.avail_in = len; do {@@ -1960,6 +1981,13 @@ int write_object_file_flags(const void *buf, unsigned long len, {char hdr[MAX_HEADER_LEN ];int hdrlen =sizeof (hdr); +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + };/* Normally if we have it in the pack then we do not bother writing * it out into .git/objects/??/?{38} file. @@ -1968,7 +1996,7 @@ int write_object_file_flags(const void *buf, unsigned long len, &hdrlen);if (freshen_packed_object (oid) ||freshen_loose_object (oid))return 0 ; -return write_loose_object (oid, hdr, hdrlen, buf, len,0 , flags); +return write_loose_object (oid, hdr, hdrlen, &in_stream,0 , flags); } int hash_object_file_literally(const void *buf, unsigned long len,@@ -1977,6 +2005,13 @@ int hash_object_file_literally(const void *buf, unsigned long len, {char *header;int hdrlen, status =0 ; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + };/* type string, SP, %lu of the length plus NUL must fit this */ hdrlen =strlen (type) +MAX_HEADER_LEN ;@@ -1988,7 +2023,7 @@ int hash_object_file_literally(const void *buf, unsigned long len, goto cleanup;if (freshen_packed_object (oid) ||freshen_loose_object (oid)) goto cleanup; - status =write_loose_object (oid, header, hdrlen, buf, len,0 ,0 ); + status =write_loose_object (oid, header, hdrlen, &in_stream,0 ,0 ); cleanup:free (header);@@ -2003,14 +2038,21 @@ int force_object_loose(const struct object_id *oid, time_t mtime)char hdr[MAX_HEADER_LEN ];int hdrlen;int ret; +struct simple_input_stream_data data; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = &data, + };if (has_loose_object (oid))return 0 ; buf =read_object (the_repository, oid, &type, &len);if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid)); + data.buf = buf; + data.len = len; hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; - ret =write_loose_object (oid, hdr, hdrlen, buf, len, mtime,0 ); + ret =write_loose_object (oid, hdr, hdrlen, &in_stream, mtime,0 );free (buf);return ret;object-store.h +5
-0@@ -34,6 +34,11 @@ struct object_directory {char *path; }; +struct input_stream { +const void *(*read)(struct input_stream *,unsigned long *len); +void *data; +}; + KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq)
Re: [PATCH v3 1/5] object-file: refactor write_loose_object() to read buffer from stream
Quoted 22 lines · Junio C Hamano
Quoted 10 lines · Han Xin
From: Han Xin <[email protected]>
We used to call "
get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.This can be improved by feeding data to "
write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.Possibly a stupid question (not a review).
How does this compare with "struct
git_istream" implemented for a few existing codepaths? It seems that the existing users are pack-objects, index-pack and archive and all of them use the interface to obtain data given an object name without having to grab everything in core at once.
Quoted 6 lines
If we are adding a new streaming interface to go in the opposite direction, i.e. from the working tree data to object store, I would understand it as a complementary interface (but then I suspect there is a half of it already in bulk-checkin API), but I am not sure how this new thing fits in the larger picture.
Thank you for your reply.
Before starting to make this patch, I did consider whether I should reuse "struct git_istream" to solve the problem, but I found that in the process of git unpack-objects, the data comes from stdin, and we cannot get an oid in advance until the whole object data is read. Also, we can't do "lseek()“ on stdin to change the data reading position.
I compared the implementation of "bulk-checkin", and they do have some similarities.
I think the difference in the reverse implementation is that we do not always clearly know where the boundary of the target data is. For example, in the process of "unpack-objects", the "buffer" has been partially read after calling "fill()". And the "buffer" remaining after reading cannot be discarded because it is the beginning of the next object.
Perhaps "struct input_stream" can make some improvements to "index_bulk_checkin()", so that it can read from an inner buffer in addition to reading from "fd" if necessary.
Quoted 142 lines
Quoted 142 lines
Helped-by Jiang Xin <[email protected]>Signed-off-by Han Xin <[email protected]>--- object-file.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++---- object-store.h | 5 +++++ 2 files changed, 51 insertions(+), 4 deletions(-)object-file.c +46
-4@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)return fd; } +struct simple_input_stream_data { +const void *buf; +unsigned long len; +}; + +static const void *feed_simple_input_stream (struct input_stream *in_stream,unsigned long *len) +{ +struct simple_input_stream_data *data = in_stream->data; + +if (data->len ==0 ) { + *len =0 ; +return NULL ; + } + *len = data->len; + data->len =0 ; +return data->buf; +} +static int write_loose_object (const struct object_id *oid,char *hdr, -int hdrlen,const void *buf,unsigned long len, +int hdrlen,struct input_stream *in_stream,time_t mtime,unsigned flags) {int fd, ret;@@ -1871,6 +1889,8 @@ static int write_loose_object(const struct object_id *oid, char *hdr,struct object_id parano_oid;static struct strbuf tmp_file =STRBUF_INIT ;static struct strbuf filename =STRBUF_INIT ; +const void *buf; +unsigned long len;loose_object_path (the_repository, &filename, oid);@@ -1898,6 +1918,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr, the_hash_algo->update_fn (&c, hdr, hdrlen);/* Then the data itself.. */ + buf = in_stream->read (in_stream, &len); stream.next_in = (void *)buf; stream.avail_in = len; do {@@ -1960,6 +1981,13 @@ int write_object_file_flags(const void *buf, unsigned long len, {char hdr[MAX_HEADER_LEN ];int hdrlen =sizeof (hdr); +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + };/* Normally if we have it in the pack then we do not bother writing * it out into .git/objects/??/?{38} file. @@ -1968,7 +1996,7 @@ int write_object_file_flags(const void *buf, unsigned long len, &hdrlen);if (freshen_packed_object (oid) ||freshen_loose_object (oid))return 0 ; -return write_loose_object (oid, hdr, hdrlen, buf, len,0 , flags); +return write_loose_object (oid, hdr, hdrlen, &in_stream,0 , flags); } int hash_object_file_literally(const void *buf, unsigned long len,@@ -1977,6 +2005,13 @@ int hash_object_file_literally(const void *buf, unsigned long len, {char *header;int hdrlen, status =0 ; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = (void *)&(struct simple_input_stream_data ) { + .buf = buf, + .len = len, + }, + };/* type string, SP, %lu of the length plus NUL must fit this */ hdrlen =strlen (type) +MAX_HEADER_LEN ;@@ -1988,7 +2023,7 @@ int hash_object_file_literally(const void *buf, unsigned long len, goto cleanup;if (freshen_packed_object (oid) ||freshen_loose_object (oid)) goto cleanup; - status =write_loose_object (oid, header, hdrlen, buf, len,0 ,0 ); + status =write_loose_object (oid, header, hdrlen, &in_stream,0 ,0 ); cleanup:free (header);@@ -2003,14 +2038,21 @@ int force_object_loose(const struct object_id *oid, time_t mtime)char hdr[MAX_HEADER_LEN ];int hdrlen;int ret; +struct simple_input_stream_data data; +struct input_stream in_stream = { + .read = feed_simple_input_stream, + .data = &data, + };if (has_loose_object (oid))return 0 ; buf =read_object (the_repository, oid, &type, &len);if (!buf)return error (_ ("cannot read object for %s" ),oid_to_hex (oid)); + data.buf = buf; + data.len = len; hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX ,type_name (type), (uintmax_t )len) +1 ; - ret =write_loose_object (oid, hdr, hdrlen, buf, len, mtime,0 ); + ret =write_loose_object (oid, hdr, hdrlen, &in_stream, mtime,0 );free (buf);return ret;object-store.h +5
-0@@ -34,6 +34,11 @@ struct object_directory {char *path; }; +struct input_stream { +const void *(*read)(struct input_stream *,unsigned long *len); +void *data; +}; + KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq)
PATCH v3 2/5 object-file.c: handle undetermined oid in write_loose_object()
In reply to Han XinFrom: Han Xin <[email protected]>
When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".
In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
--- object-file.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)
object-file.c +28 -2
@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,const void *buf;unsigned long len; -loose_object_path (the_repository, &filename, oid); +if (is_null_oid (oid)) { +/* When oid is not determined, save tmp file to odb path. */ +strbuf_reset (&filename); +strbuf_addstr (&filename, the_repository->objects->odb->path); +strbuf_addch (&filename,'/' ); + }else { +loose_object_path (the_repository, &filename, oid); + } fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {
@@ -1939,12 +1946,31 @@ static int write_loose_object(const struct object_id *oid, char *hdr,die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), ret); the_hash_algo->final_oid_fn (¶no_oid, &c); -if (!oideq (oid, ¶no_oid)) +if (!is_null_oid (oid) && !oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));close_loose_object (fd); +if (is_null_oid (oid)) { +int dirlen; + +oidcpy ((struct object_id *)oid, ¶no_oid); +loose_object_path (the_repository, &filename, oid); + +/* We finally know the object path, and create the missing dir. */ + dirlen =directory_size (filename.buf); +if (dirlen) { +struct strbuf dir =STRBUF_INIT ; +strbuf_add (&dir, filename.buf, dirlen -1 ); +if (mkdir (dir.buf,0777 ) && errno !=EEXIST ) +return -1 ; +if (adjust_shared_perm (dir.buf)) +return -1 ; +strbuf_release (&dir); + } + } +if (mtime) {struct utimbuf utb; utb.actime = mtime;
Signature · 1 line
2.34.0.6.g676eedc724
Re: [PATCH v3 2/5] object-file.c: handle undetermined oid in write_loose_object()
Quoted 7 lines · Han Xin
From: Han Xin <[email protected]>
When streaming a large blob object to "
write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".
Quoted 5 lines
In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.
My first reaction is to not write into .git/objects/ directly, but instead make a .git/objects/tmp/ directory and write within that directory. The idea is to prevent leaving stale files in the .git/objects/ directory if the process terminates strangely (say, a power outage or segfault).
If this was an interesting idea to pursue, it does leave a question:
should we clean up the tmp/ directory when it is empty? That would require adding a check in finalize_object_file() that is probably best left unchecked (the lstat() would add a cost per loose object write that is probably too costly). I would rather leave an empty tmp/ directory than add that cost per loose object write.
I suppose another way to do it would be to register the check as an event at the end of the process, so we only check once, and that only happens if we created a loose object with this streaming method.
With all of these complications in mind, I think cleaning up the stale tmp/ directory could (at the very least) be delayed to another commit or patch series. Hopefully adding the directory is not too much complication to add here.
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1899
Quoted 6 lines
object-file.c +5
-1-loose_object_path (the_repository, &filename, oid); +if (is_null_oid (oid)) { +/* When oid is not determined, save tmp file to odb path. */ +strbuf_reset (&filename); +strbuf_addstr (&filename, the_repository->objects->odb->path); +strbuf_addch (&filename,'/' );
Here, you could instead of the strbuf_addch() do
strbuf_add(&filename, "/tmp/", 5);
if (safe_create_leading_directories(filename.buf)) {
error(_("failed to create '%s'"));
strbuf_release(&filename);
return -1;
} Quoted 26 lines
object-file.c +13
-1+ }else { +loose_object_path (the_repository, &filename, oid); + } fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {@@ -1939,12 +1946,31 @@ static int write_loose_object(const struct object_id *oid, char *hdr,die (_ ("deflateEnd on object %s failed (%d)" ),oid_to_hex (oid), ret); the_hash_algo->final_oid_fn (¶no_oid, &c); -if (!oideq (oid, ¶no_oid)) +if (!is_null_oid (oid) && !oideq (oid, ¶no_oid))die (_ ("confused by unstable object source data for %s" ),oid_to_hex (oid));close_loose_object (fd); +if (is_null_oid (oid)) { +int dirlen; + +oidcpy ((struct object_id *)oid, ¶no_oid); +loose_object_path (the_repository, &filename, oid); + +/* We finally know the object path, and create the missing dir. */ + dirlen =directory_size (filename.buf); +if (dirlen) {
object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1973
Quoted 10 lines
object-file.c +10
-0+struct strbuf dir =STRBUF_INIT ; +strbuf_add (&dir, filename.buf, dirlen -1 ); +if (mkdir (dir.buf,0777 ) && errno !=EEXIST ) +return -1 ; +if (adjust_shared_perm (dir.buf)) +return -1 ; +strbuf_release (&dir); + } + } +
Upon first reading I was asking "where is the file rename?" but it is part of finalize_object_file() which is called further down.
Thanks,
-Stolee
Re: [PATCH v3 2/5] object-file.c: handle undetermined oid in write_loose_object()
Quoted 5 lines · Derrick Stolee
My first reaction is to not write into .git/objects/ directly, but instead make a .git/objects/tmp/ directory and write within that directory. The idea is to prevent leaving stale files in the .git/objects/ directory if the process terminates strangely (say, a power outage or segfault).
Even if we know the name of the object we are writing beforehand, I do not think it is a good idea to open-write-close the final object file. The approach we already use everywhere is to write into a tmpfile/lockfile and rename it to the final name
object-file.c::write_loose_object() uses create_tmpfile() to prepare a temporary file whose name begins with "tmp_obj_", so that "gc" can recognize stale ones and remove them.
Quoted 6 lines
If this was an interesting idea to pursue, it does leave a question:
should we clean up the tmp/ directory when it is empty? That would require adding a check infinalize_object_file()that is probably best left unchecked (thelstat()would add a cost per loose object write that is probably too costly). I would rather leave an empty tmp/ directory than add that cost per loose object write.
I am not sure why we want a new tmp/ directory.
Re: [PATCH v3 2/5] object-file.c: handle undetermined oid in write_loose_object()
Quoted 8 lines · Junio C Hamano
Quoted 5 lines · Derrick Stolee
My first reaction is to not write into .git/objects/ directly, but instead make a .git/objects/tmp/ directory and write within that directory. The idea is to prevent leaving stale files in the .git/objects/ directory if the process terminates strangely (say, a power outage or segfault).
Quoted 8 lines
Even if we know the name of the object we are writing beforehand, I do not think it is a good idea to open-write-close the final object file. The approach we already use everywhere is to write into a tmpfile/lockfile and rename it to the final name
object-file.c::write_loose_object() uses
create_tmpfile()to prepare a temporary file whose name begins with "tmp_obj_", so that "gc" can recognize stale ones and remove them.
The only difference is that the tmp_obj_* file would go into the loose object directory corresponding to the first two hex characters of the OID, but that no longer happens now.
Quoted 8 lines
Quoted 6 lines
If this was an interesting idea to pursue, it does leave a question:
should we clean up the tmp/ directory when it is empty? That would require adding a check infinalize_object_file()that is probably best left unchecked (thelstat()would add a cost per loose object write that is probably too costly). I would rather leave an empty tmp/ directory than add that cost per loose object write.I am not sure why we want a new tmp/ directory.
I'm just thinking of a case where this fails repeatedly I would rather have those failed tmp_obj_* files isolated in their own directory. It's an extremely minor point, so I'm fine to drop the recommendation.
Thanks,
-Stolee
Re: [PATCH v3 2/5] object-file.c: handle undetermined oid in write_loose_object()
Quoted 15 lines · Derrick Stolee
Quoted 13 lines · Junio C Hamano
Quoted 5 lines · Derrick Stolee
My first reaction is to not write into .git/objects/ directly, but instead make a .git/objects/tmp/ directory and write within that directory. The idea is to prevent leaving stale files in the .git/objects/ directory if the process terminates strangely (say, a power outage or segfault).
Even if we know the name of the object we are writing beforehand, I do not think it is a good idea to open-write-close the final object file. The approach we already use everywhere is to write into a tmpfile/lockfile and rename it to the final name
Quoted 8 lines
Quoted 3 lines
object-file.c::write_loose_object() uses
create_tmpfile()to prepare a temporary file whose name begins with "tmp_obj_", so that "gc" can recognize stale ones and remove them.The only difference is that the tmp_obj_* file would go into the loose object directory corresponding to the first two hex characters of the OID, but that no longer happens now.
At the beginning of this patch, I did save the temporary object in a two hex characters directory of "null_oid", but this is also a very strange behavior. "Gc" will indeed clean up these tmp_obj_* files, no matter if they are in .git/objects/ or .git/objects/xx.
Thanks,
-Han Xin
PATCH v3 3/5 object-file.c: read stream in a loop in write_loose_object()
In reply to Han XinFrom: Han Xin <[email protected]>
In order to prepare the stream version of "write_loose_object()", read the input stream in a loop in "write_loose_object()", so that we can feed the contents of large blob object to "write_loose_object()" using a small fixed buffer.
--- object-file.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
object-file.c +11 -5
@@ -1890,7 +1890,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,static struct strbuf tmp_file =STRBUF_INIT ;static struct strbuf filename =STRBUF_INIT ;const void *buf; -unsigned long len; +int flush =0 ;if (is_null_oid (oid)) {/* When oid is not determined, save tmp file to odb path. */
@@ -1925,12 +1925,18 @@ static int write_loose_object(const struct object_id *oid, char *hdr, the_hash_algo->update_fn (&c, hdr, hdrlen);/* Then the data itself.. */ - buf = in_stream->read (in_stream, &len); - stream.next_in = (void *)buf; - stream.avail_in = len; do { unsigned char *in0 = stream.next_in; - ret = git_deflate(&stream, Z_FINISH); + if (!stream.avail_in) { + buf = in_stream->read(in_stream, &stream.avail_in); + if (buf) { + stream.next_in = (void *)buf; + in0 = (unsigned char *)buf; + } else { + flush = Z_FINISH; + } + } + ret = git_deflate(&stream, flush); the_hash_algo->update_fn(&c, in0, stream.next_in - in0); if (write_buffer(fd, compressed, stream.next_out - compressed) < 0) die(_("unable to write loose object file"));
Signature · 1 line
2.34.0.6.g676eedc724
PATCH v3 4/5 unpack-objects.c: add dry_run mode for get_data()
In reply to Han XinFrom: Han Xin <[email protected]>
In dry_run mode, "get_data()" is used to verify the inflation of data, and the returned buffer will not be used at all and will be freed immediately. Even in dry_run mode, it is dangerous to allocate a full-size buffer for a large blob object. Therefore, only allocate a low memory footprint when calling "get_data()" in dry_run mode.
--- builtin/unpack-objects.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
builtin/unpack-objects.c +12 -6
@@ -96,15 +96,16 @@ static void use(int bytes)display_throughput (progress, consumed_bytes); } -static void *get_data (unsigned long size) +static void *get_data (unsigned long size,int dry_run) {git_zstream stream; -void *buf =xmallocz (size); +unsigned long bufsize = dry_run ?4096 : size; +void *buf =xmallocz (bufsize);memset (&stream,0 ,sizeof (stream)); stream.next_out = buf; - stream.avail_out = size; + stream.avail_out = bufsize; stream.next_in =fill (1 ); stream.avail_in = len;git_inflate_init (&stream);
@@ -124,6 +125,11 @@ static void *get_data(unsigned long size) } stream.next_in =fill (1 ); stream.avail_in = len; +if (dry_run) { +/* reuse the buffer in dry_run mode */ + stream.next_out = buf; + stream.avail_out = bufsize; + } }git_inflate_end (&stream);return buf;
@@ -323,7 +329,7 @@ static void added_object(unsigned nr, enum object_type type,static void unpack_non_delta_entry (enum object_type type,unsigned long size,unsigned nr) { -void *buf =get_data (size); +void *buf =get_data (size, dry_run);if (!dry_run && buf)write_object (nr, type, buf, size);
@@ -357,7 +363,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,if (type ==OBJ_REF_DELTA ) {oidread (&base_oid,fill (the_hash_algo->rawsz));use (the_hash_algo->rawsz); - delta_data =get_data (delta_size); + delta_data =get_data (delta_size, dry_run);if (dry_run || !delta_data) {free (delta_data);return ;
@@ -396,7 +402,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,if (base_offset <=0 || base_offset >= obj_list[nr].offset)die ("offset value out of bound for delta base object" ); - delta_data =get_data (delta_size); + delta_data =get_data (delta_size, dry_run);if (dry_run || !delta_data) {free (delta_data);return ;
Signature · 1 line
2.34.0.6.g676eedc724
PATCH v3 5/5 unpack-objects: unpack_non_delta_entry() read data in a stream
In reply to Han XinFrom: Han Xin <[email protected]>
We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.
By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".
However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "big_file_threshold" in zstream. See the following benchmarks:
$ hyperfine \
--prepare 'rm -rf dest.git && git init --bare dest.git' \
'git -C dest.git unpack-objects <binary_320M.pack'
Benchmark 1: git -C dest.git unpack-objects <binary_320M.pack
Time (mean ± σ): 10.029 s ± 0.270 s [User: 8.265 s, System: 1.522 s]
Range (min … max): 9.786 s … 10.603 s 10 runs $ hyperfine \
--prepare 'rm -rf dest.git && git init --bare dest.git' \
'git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack'
Benchmark 1: git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack
Time (mean ± σ): 10.859 s ± 0.774 s [User: 8.813 s, System: 1.898 s]
Range (min … max): 9.884 s … 12.192 s 10 runs $ hyperfine \
--prepare 'rm -rf dest.git && git init --bare dest.git' \
'git -C dest.git unpack-objects <binary_96M.pack'
Benchmark 1: git -C dest.git unpack-objects <binary_96M.pack
Time (mean ± σ): 2.678 s ± 0.037 s [User: 2.205 s, System: 0.450 s]
Range (min … max): 2.639 s … 2.743 s 10 runs $ hyperfine \
--prepare 'rm -rf dest.git && git init --bare dest.git' \
'git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_96M.pack'
Benchmark 1: git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_96M.pack
Time (mean ± σ): 2.819 s ± 0.124 s [User: 2.216 s, System: 0.564 s]
Range (min … max): 2.679 s … 3.125 s 10 runs--- builtin/unpack-objects.c | 76 ++++++++++++++++++++++++++++- object-file.c | 6 +-- object-store.h | 4 ++ t/t5590-unpack-non-delta-objects.sh | 76 +++++++++++++++++++++++++++++ 4 files changed, 158 insertions(+), 4 deletions(-) create mode 100755 t/t5590-unpack-non-delta-objects.sh
builtin/unpack-objects.c +75 -1
@@ -326,11 +326,85 @@ static void added_object(unsigned nr, enum object_type type, } } +struct input_zstream_data { +git_zstream *zstream; +unsigned char buf[4096 ]; +int status; +}; + +static const void *feed_input_zstream (struct input_stream *in_stream,unsigned long *readlen) +{ +struct input_zstream_data *data = in_stream->data; +git_zstream *zstream = data->zstream; +void *in =fill (1 ); + +if (!len || data->status ==Z_STREAM_END ) { + *readlen =0 ; +return NULL ; + } + + zstream->next_out = data->buf; + zstream->avail_out =sizeof (data->buf); + zstream->next_in = in; + zstream->avail_in = len; + + data->status =git_inflate (zstream,0 ); +use (len - zstream->avail_in); + *readlen =sizeof (data->buf) - zstream->avail_out; + +return data->buf; +} + +static void write_stream_blob (unsigned nr,unsigned long size) +{ +char hdr[32 ]; +int hdrlen; +git_zstream zstream; +struct input_zstream_data data; +struct input_stream in_stream = { + .read = feed_input_zstream, + .data = &data, + }; +struct object_id *oid = &obj_list[nr].oid; +int ret; + +memset (&zstream,0 ,sizeof (zstream)); +memset (&data,0 ,sizeof (data)); + data.zstream = &zstream; +git_inflate_init (&zstream); + +/* Generate the header */ + hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %" PRIuMAX,type_name (OBJ_BLOB ), (uintmax_t )size) +1 ; + +if ((ret =write_loose_object (oid, hdr, hdrlen, &in_stream,0 ,0 ))) +die (_ ("failed to write object in stream %d" ), ret); + +if (zstream.total_out != size || data.status !=Z_STREAM_END ) +die (_ ("inflate returned %d" ), data.status); +git_inflate_end (&zstream); + +if (strict && !dry_run) { +struct blob *blob =lookup_blob (the_repository, oid); +if (blob) + blob->object.flags |=FLAG_WRITTEN ; +else +die ("invalid blob object from stream" ); + } + obj_list[nr].obj =NULL ; +} +static void unpack_non_delta_entry (enum object_type type,unsigned long size,unsigned nr) { -void *buf =get_data (size, dry_run); +void *buf; + +/* Write large blob in stream without allocating full buffer. */ +if (!dry_run && type ==OBJ_BLOB && size > big_file_threshold) { +write_stream_blob (nr, size); +return ; + } + buf =get_data (size, dry_run);if (!dry_run && buf)write_object (nr, type, buf, size);else
object-file.c +3 -3
@@ -1878,9 +1878,9 @@ static const void *feed_simple_input_stream(struct input_stream *in_stream, unsireturn data->buf; } -static int write_loose_object (const struct object_id *oid,char *hdr, -int hdrlen,struct input_stream *in_stream, -time_t mtime,unsigned flags) +int write_loose_object (const struct object_id *oid,char *hdr, +int hdrlen,struct input_stream *in_stream, +time_t mtime,unsigned flags) {int fd, ret;unsigned char compressed[4096 ];
object-store.h +4 -0
@@ -228,6 +228,10 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, unsigned long len, const char *type, struct object_id *oid); +int write_loose_object (const struct object_id *oid,char *hdr, +int hdrlen,struct input_stream *in_stream, +time_t mtime,unsigned flags); +int write_object_file_flags (const void *buf,unsigned long len,const char *type,struct object_id *oid,unsigned flags);
t/t5590-unpack-non-delta-objects.sh new file +76 -0
76 lines not on this page · raw message
Signature · 1 line
2.34.0.6.g676eedc724
Re: [PATCH v3 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream
Quoted 21 lines · Han Xin
From: Han Xin <[email protected]>
We used to call "
get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.By implementing a zstream version of
input_streaminterface, we can use a small fixed buffer for "unpack_non_delta_entry()".However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "
big_file_threshold" in zstream. See the following benchmarks:$ hyperfine \ --prepare 'rm -rf dest.git && git init --bare dest.git' \ 'git -C dest.git unpack-objects <binary_320M.pack' Benchmark 1: git -C dest.git unpack-objects <binary_320M.pack Time (mean ± σ): 10.029 s ± 0.270 s [User: 8.265 s, System: 1.522 s] Range (min … max): 9.786 s … 10.603 s 10 runs
Quoted 6 lines
$ hyperfine \
--prepare 'rm -rf dest.git && git init --bare dest.git' \
'git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack'
Benchmark 1: git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack
Time (mean ± σ): 10.859 s ± 0.774 s [User: 8.813 s, System: 1.898 s]
Range (min … max): 9.884 s … 12.192 s 10 runsIt seems that you want us to compare this pair of results, and hyperfine can assist with that by including multiple benchmarks (with labels, using '-n') as follows:
$ hyperfine \
--prepare 'rm -rf dest.git && git init --bare dest.git' \
-n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \
-n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \
-n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'Benchmark 1: old
Time (mean ± σ): 20.835 s ± 0.058 s [User: 14.510 s, System: 6.284 s]
Range (min … max): 20.741 s … 20.909 s 10 runs
Benchmark 2: new
Time (mean ± σ): 26.515 s ± 0.072 s [User: 19.783 s, System: 6.696 s]
Range (min … max): 26.419 s … 26.611 s 10 runs
Benchmark 3: new (small threshold)
Time (mean ± σ): 26.523 s ± 0.101 s [User: 19.805 s, System: 6.680 s]
Range (min … max): 26.416 s … 26.739 s 10 runs
Summary
'old' ran
1.27 ± 0.00 times faster than 'new'
1.27 ± 0.01 times faster than 'new (small threshold)'
(Here, 'old' is testing a compiled version of the latest 'master' branch, while 'new' has your patches applied on top.)
Notice from this example I had a pack with many small objects (mostly commits and trees) and I see that this change introduces significant overhead to this case.
It would be nice to understand this overhead and fix it before taking this change any further.
Thanks,
-Stolee
Re: [PATCH v3 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream
Quoted 64 lines · Derrick Stolee
Quoted 21 lines · Han Xin
From: Han Xin <[email protected]>
We used to call "
get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.By implementing a zstream version of
input_streaminterface, we can use a small fixed buffer for "unpack_non_delta_entry()".However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "
big_file_threshold" in zstream. See the following benchmarks:$ hyperfine \ --prepare 'rm -rf dest.git && git init --bare dest.git' \ 'git -C dest.git unpack-objects <binary_320M.pack' Benchmark 1: git -C dest.git unpack-objects <binary_320M.pack Time (mean ± σ): 10.029 s ± 0.270 s [User: 8.265 s, System: 1.522 s] Range (min … max): 9.786 s … 10.603 s 10 runsQuoted 6 lines
$ hyperfine \ --prepare 'rm -rf dest.git && git init --bare dest.git' \ 'git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack' Benchmark 1: git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack Time (mean ± σ): 10.859 s ± 0.774 s [User: 8.813 s, System: 1.898 s] Range (min … max): 9.884 s … 12.192 s 10 runsIt seems that you want us to compare this pair of results, and hyperfine can assist with that by including multiple benchmarks (with labels, using '-n') as follows:
$ hyperfine \ --prepare 'rm -rf dest.git && git init --bare dest.git' \ -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \ -n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \ -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'Benchmark 1: old
Time (mean ± σ): 20.835 s ± 0.058 s [User: 14.510 s, System: 6.284 s]
Range (min … max): 20.741 s … 20.909 s 10 runsBenchmark 2: new
Time (mean ± σ): 26.515 s ± 0.072 s [User: 19.783 s, System: 6.696 s]
Range (min … max): 26.419 s … 26.611 s 10 runsBenchmark 3: new (small threshold)
Time (mean ± σ): 26.523 s ± 0.101 s [User: 19.805 s, System: 6.680 s]
Range (min … max): 26.416 s … 26.739 s 10 runsSummary
'old' ran
1.27 ± 0.00 times faster than 'new'
1.27 ± 0.01 times faster than 'new (small threshold)'(Here, 'old' is testing a compiled version of the latest 'master' branch, while 'new' has your patches applied on top.)
Notice from this example I had a pack with many small objects (mostly commits and trees) and I see that this change introduces significant overhead to this case.
Quoted 5 lines
It would be nice to understand this overhead and fix it before taking this change any further.
Thanks,
-Stolee
Can you show me the specific information of the repository you tested, so that I can analyze it further.
I test this repository, but did not meet the problem:
Unpacking objects: 100% (18345/18345), 43.15 MiB
hyperfine \
--prepare 'rm -rf dest.git && git init --bare dest.git' \
-n 'old' 'git -C dest.git unpack-objects <big.pack' \
-n 'new' 'new/git -C dest.git unpack-objects <big.pack' \
-n 'new (small threshold)' 'new/git -ccore.bigfilethreshold=64k -C dest.git unpack-objects <big.pack' Benchmark 1: old
Time (mean ± σ): 17.403 s ± 0.880 s [User: 4.996 s, System: 11.803 s]
Range (min … max): 15.911 s … 19.368 s 10 runs
Benchmark 2: new
Time (mean ± σ): 17.788 s ± 0.199 s [User: 5.054 s, System: 12.257 s]
Range (min … max): 17.420 s … 18.195 s 10 runs
Benchmark 3: new (small threshold)
Time (mean ± σ): 18.433 s ± 0.711 s [User: 4.982 s, System: 12.338 s]
Range (min … max): 17.518 s … 19.775 s 10 runs
Summary
'old' ran
1.02 ± 0.05 times faster than 'new'
1.06 ± 0.07 times faster than 'new (small threshold)'
Thanks,
- Han Xin
Re: [PATCH v3 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream
Quoted 31 lines · Han Xin
Quoted 30 lines · Derrick Stolee
$ hyperfine \ --prepare 'rm -rf dest.git && git init --bare dest.git' \ -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \ -n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \ -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'Benchmark 1: old
Time (mean ± σ): 20.835 s ± 0.058 s [User: 14.510 s, System: 6.284 s]
Range (min … max): 20.741 s … 20.909 s 10 runsBenchmark 2: new
Time (mean ± σ): 26.515 s ± 0.072 s [User: 19.783 s, System: 6.696 s]
Range (min … max): 26.419 s … 26.611 s 10 runsBenchmark 3: new (small threshold)
Time (mean ± σ): 26.523 s ± 0.101 s [User: 19.805 s, System: 6.680 s]
Range (min … max): 26.416 s … 26.739 s 10 runsSummary
'old' ran
1.27 ± 0.00 times faster than 'new'
1.27 ± 0.01 times faster than 'new (small threshold)'(Here, 'old' is testing a compiled version of the latest 'master' branch, while 'new' has your patches applied on top.)
Notice from this example I had a pack with many small objects (mostly commits and trees) and I see that this change introduces significant overhead to this case.
Quoted 8 lines
Quoted 5 lines
It would be nice to understand this overhead and fix it before taking this change any further.
Thanks,
-StoleeCan you show me the specific information of the repository you tested, so that I can analyze it further.
I used a pack-file from an internal repo. It happened to be using partial clone, so here is a repro with the git/git repository after cloning this way:
$ git clone --no-checkout --filter=blob:none https://github.com/git/git
(copy the large .pack from git/.git/objects/pack/ to big.pack)
$ hyperfine \ --prepare 'rm -rf dest.git && git init --bare dest.git' \ -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \ -n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \ -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'
Benchmark 1: old
Time (mean ± σ): 82.748 s ± 0.445 s [User: 50.512 s, System: 32.049 s]
Range (min … max): 82.042 s … 83.587 s 10 runs
Benchmark 2: new
Time (mean ± σ): 101.644 s ± 0.524 s [User: 67.470 s, System: 34.047 s]
Range (min … max): 100.866 s … 102.633 s 10 runs
Benchmark 3: new (small threshold)
Time (mean ± σ): 101.093 s ± 0.269 s [User: 67.404 s, System: 33.559 s]
Range (min … max): 100.639 s … 101.375 s 10 runs
Summary
'old' ran
1.22 ± 0.01 times faster than 'new (small threshold)' 1.23 ± 0.01 times faster than 'new'
I'm also able to repro this with a smaller repo (microsoft/scalar) so the tests complete much faster:
$ hyperfine \
--prepare 'rm -rf dest.git && git init --bare dest.git' \
-n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <small.pack' \
-n 'new' '~/_git/git/git -C dest.git unpack-objects <small.pack' \
-n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <small.pack'Benchmark 1: old
Time (mean ± σ): 3.295 s ± 0.023 s [User: 1.063 s, System: 2.228 s]
Range (min … max): 3.269 s … 3.351 s 10 runs
Benchmark 2: new
Time (mean ± σ): 3.592 s ± 0.105 s [User: 1.261 s, System: 2.328 s]
Range (min … max): 3.378 s … 3.679 s 10 runs
Benchmark 3: new (small threshold)
Time (mean ± σ): 3.584 s ± 0.144 s [User: 1.241 s, System: 2.339 s]
Range (min … max): 3.359 s … 3.747 s 10 runs
Summary
'old' ran
1.09 ± 0.04 times faster than 'new (small threshold)' 1.09 ± 0.03 times faster than 'new'
It's not the same relative overhead, but still significant.
These pack-files contain (mostly) small objects, no large blobs. I know that's not the target of your efforts, but it would be good to avoid a regression here.
Thanks,
-Stolee
"git hyperfine" (was: [PATCH v3 5/5] unpack-objects[...])
I hadn't sent a shameless plug for my "git hyperfine" script to the list, perhaps this is a good time. It's just a thin shellscript wrapper around "hyperfine" that I wrote the other day, which...
Quoted 23 lines · Derrick Stolee
[...]
I used a pack-file from an internal repo. It happened to be using partial clone, so here is a repro with the git/git repository after cloning this way:$ git clone --no-checkout --filter=blob:none https://github.com/git/git(copy the large .pack from git/.git/objects/pack/ to big.pack)
$ hyperfine \ --prepare 'rm -rf dest.git && git init --bare dest.git' \ -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \ -n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \ -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'Benchmark 1: old
Time (mean ± σ): 82.748 s ± 0.445 s [User: 50.512 s, System: 32.049 s]
Range (min … max): 82.042 s … 83.587 s 10 runsBenchmark 2: new
Time (mean ± σ): 101.644 s ± 0.524 s [User: 67.470 s, System: 34.047 s]
Range (min … max): 100.866 s … 102.633 s 10 runs
Quoted 8 lines
Benchmark 3: new (small threshold)
Time (mean ± σ): 101.093 s ± 0.269 s [User: 67.404 s, System: 33.559 s]
Range (min … max): 100.639 s … 101.375 s 10 runsSummary
'old' ran
1.22 ± 0.01 times faster than 'new (small threshold)' 1.23 ± 0.01 times faster than 'new'
...adds enough sugar around "hyperfine" itself to do this as e.g. (the "-s" is a feature I submitted to hyperfine itself, it's not in a release yet[1], but in this case you could also use "-p"):
git hyperfine -L rev v2.20.0,origin/master \
-s 'if ! test -d redis.git; then git clone --bare --filter=blob:none https://github.com/redis/redis; fi && make' \
-p 'rm -rf dest.git; git init --bare dest.git' \
'./git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)'The sugar being that for each named "rev" parameter it'll set up "git worktree" for you, so under the hood each of those is chdir-ing to the respective revision of:
$ git worktree list
[...]
/run/user/1001/git-hyperfine/origin/master abe6bb39053 (detached HEAD)
/run/user/1001/git-hyperfine/v2.33.0 225bc32a989 (detached HEAD)That they're named revisions and not git-rev-parse'd is intentional, since you'll benefit from faster incremental "make" (even if using "ccache"). I'm typically benchmarking HEAD~1,HEAD~0.
The output will then use those "rev" parameters, and be e.g.:
Benchmark 1: ./git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'v2.20.0
Time (mean ± σ): 6.678 s ± 0.046 s [User: 4.525 s, System: 2.117 s]
Range (min … max): 6.619 s … 6.765 s 10 runs
Benchmark 2: ./git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'origin/master
Time (mean ± σ): 6.756 s ± 0.074 s [User: 4.586 s, System: 2.134 s]
Range (min … max): 6.691 s … 6.941 s 10 runs
Summary
'./git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'v2.20.0' ran
1.01 ± 0.01 times faster than './git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'origin/master'I think if you're routinely benchmarking N different git versions you'll find it handy, it also has configurable hook support (using git config), so e.g. it's easy to copy your config.mak in-place in the worktrees. E.g. my config is:
$ git -P config --get-regexp '^hyperfine'
hyperfine.run-dir $XDG_RUNTIME_DIR/git-hyperfine
hyperfine.xargs-options -r
hyperfine.hook.setup ~/g/git.meta/config.mak.shIt's hosted at https://github.com/avar/git-hyperfine/ and
https://gitlab.com/avar/git-hyperfine/; It's implemented in (portable) POSIX shell script.
There's surely some bugs in it, one known one is that unlike hyperfine it doesn't accept there being spaces in the parameters to -L, because I'm screwing up some quoting-within-quoting in the (shellscript) implementation (suggestions for that particular one most welcome).
I hacked it up after this suggestion from Jeff King[2] of moving t/perf over to it.
I haven't done any of that legwork, but I think a wrapper like "git-hyperfine" that prepares worktrees for the N revisions we're benchmarking is a good direction to go in.
We don't use git-worktrees in t/perf, but probably could for most/all tests. In any case it would be easy to have the script setup the revs to be benchmarked in some hookable custom manner to have it do exactly what t/perf/run is doing now.
1. https://github.com/sharkdp/hyperfine/commit/017d55a
2. https://lore.kernel.org/git/[email protected]/
Re: [PATCH v3 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream
In reply to Derrick StoleeQuoted 58 lines · Derrick Stolee
I used a pack-file from an internal repo. It happened to be using partial clone, so here is a repro with the git/git repository after cloning this way:
$ git clone --no-checkout --filter=blob:none https://github.com/git/git(copy the large .pack from git/.git/objects/pack/ to big.pack)
$ hyperfine \ --prepare 'rm -rf dest.git && git init --bare dest.git' \ -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \ -n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \ -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'Benchmark 1: old
Time (mean ± σ): 82.748 s ± 0.445 s [User: 50.512 s, System: 32.049 s]
Range (min … max): 82.042 s … 83.587 s 10 runsBenchmark 2: new
Time (mean ± σ): 101.644 s ± 0.524 s [User: 67.470 s, System: 34.047 s]
Range (min … max): 100.866 s … 102.633 s 10 runsBenchmark 3: new (small threshold)
Time (mean ± σ): 101.093 s ± 0.269 s [User: 67.404 s, System: 33.559 s]
Range (min … max): 100.639 s … 101.375 s 10 runsSummary
'old' ran
1.22 ± 0.01 times faster than 'new (small threshold)' 1.23 ± 0.01 times faster than 'new'I'm also able to repro this with a smaller repo (microsoft/scalar) so the tests complete much faster:
$ hyperfine \ --prepare 'rm -rf dest.git && git init --bare dest.git' \ -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <small.pack' \ -n 'new' '~/_git/git/git -C dest.git unpack-objects <small.pack' \ -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <small.pack'Benchmark 1: old
Time (mean ± σ): 3.295 s ± 0.023 s [User: 1.063 s, System: 2.228 s]
Range (min … max): 3.269 s … 3.351 s 10 runsBenchmark 2: new
Time (mean ± σ): 3.592 s ± 0.105 s [User: 1.261 s, System: 2.328 s]
Range (min … max): 3.378 s … 3.679 s 10 runsBenchmark 3: new (small threshold)
Time (mean ± σ): 3.584 s ± 0.144 s [User: 1.241 s, System: 2.339 s]
Range (min … max): 3.359 s … 3.747 s 10 runsSummary
'old' ran
1.09 ± 0.04 times faster than 'new (small threshold)' 1.09 ± 0.03 times faster than 'new'
Quoted 8 lines
It's not the same relative overhead, but still significant.
These pack-files contain (mostly) small objects, no large blobs. I know that's not the target of your efforts, but it would be good to avoid a regression here.
Thanks,
-Stolee
With your help, I did catch this performance problem, which was introduced in this patch:
https://lore.kernel.org/git/[email protected]/
This patch changes the original data reading ino to stream reading, but its problem is that even for the original reading of the whole object data, it still generates an additional git_deflate() and subsequent transfer.
I will fix it in a follow-up patch.
Thanks,
-Han Xin
Re: [PATCH v3 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream
Quoted 8 lines · Han Xin
Quoted 6 lines · Derrick Stolee
These pack-files contain (mostly) small objects, no large blobs. I know that's not the target of your efforts, but it would be good to avoid a regression here.
Thanks,
-Stolee
Quoted 7 lines
With your help, I did catch this performance problem, which was introduced in this patch:
https://lore.kernel.org/git/[email protected]/This patch changes the original data reading ino to stream reading, but its problem is that even for the original reading of the whole object data, it still generates an additional
git_deflate()and subsequent transfer.
I'm glad you found it!
Quoted 1 line
I will fix it in a follow-up patch.
Looking forward to it.
Thanks,
-Stolee