Skip to contentSkip to conversation

unpack-objects: unpack large object in stream

PATCH Superseded Han Xin 10 participants 211 messages Oct 9, 2021 – Jul 1, 2022 Documentation Page 2 of 2

lore
Share & tools
lore
#107 lore raw

Re: [RFC PATCH] object-file.c: batched disk flushes for stream_loose_object()

In reply to Han Xin

Hi,

Quoted 7 lines · Han Xin

Neeraj Singh[1] pointed out that if batch fsync is enabled, we should still call prepare_loose_object_bulk_checkin() to potentially create the bulk checkin objdir.

1. https://lore.kernel.org/git/[email protected]/

I like a good commit message that is concise and yet has all the necessary information. Well done!

Quoted 10 lines
---
 object-file.c                   |  3 +++
 t/t5351-unpack-large-objects.sh | 15 ++++++++++++++-
 2 files changed, 17 insertions(+), 1 deletion(-)
object-file.c +0 -0
index 2dd828b45b..3a1be74775 100644

object-file.c int stream_loose_object(struct input_stream *in_stream, size_t len, · line 2136

Quoted 6 lines
object-file.c +3 -0
@@ -2131,6 +2131,9 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen;
 
+	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
+		prepare_loose_object_bulk_checkin();
+

Makes sense.

Quoted 8 lines
object-file.c +0 -0
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", get_object_directory());
 	hdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, len);
t/t5351-unpack-large-objects.sh +0 -0
index 461ca060b2..a66a51f7df 100755

t/t5351-unpack-large-objects.sh test_expect_success "create large objects (1.5 MB) and PACK" ' · line 24

Quoted 8 lines
t/t5351-unpack-large-objects.sh +4 -1
@@ -18,7 +18,10 @@ test_expect_success "create large objects (1.5 MB) and PACK" '
 	test_commit --append foo big-blob &&
 	test-tool genrandom bar 1500000 >big-blob &&
 	test_commit --append bar big-blob &&
-	PACK=$(echo HEAD | git pack-objects --revs pack)
+	PACK=$(echo HEAD | git pack-objects --revs pack) &&
+	git verify-pack -v pack-$PACK.pack |
+	    grep -E "commit|tree|blob" |
+		sed -n -e "s/^\([0-9a-f]*\).*/\1/p" >obj-list

Here, I would recommend avoiding the pipe, to ensure that we would catch problems in the verify-pack invocation, and I think we can avoid the grep altogether:

	git verify-pack -v pack-$PACK.pack >out &&
	sed -n 's/^\([0-9a-f][0-9a-f]*\).*\(commit\|tree\|blob\)/\1/p' \
		<out >obj-list

t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 55

Quoted 12 lines
t/t5351-unpack-large-objects.sh +5 -0
 '
 
 test_expect_success 'set memory limitation to 1MB' '
@@ -45,6 +48,16 @@ test_expect_success 'unpack big object in stream' '
 	test_dir_is_empty dest.git/objects/pack
 '
 
+BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch'
+
+test_expect_success 'unpack big object in stream (core.fsyncmethod=batch)' '
+	prepare_dest 1m &&
+	git $BATCH_CONFIGURATION -C dest.git unpack-objects <pack-$PACK.pack &&

I think the canonical way would be to use test_config core.fsync ..., but the presented way works, too.

t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 57

Quoted 2 lines
t/t5351-unpack-large-objects.sh +2 -0
+	test_dir_is_empty dest.git/objects/pack &&
+	git -C dest.git cat-file --batch-check="%(objectname)" <obj-list >current &&

Good. The --batch-check="%(objectname)" part forces cat-file to read the actual object.

t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 59

Quoted 2 lines
t/t5351-unpack-large-objects.sh +2 -0
+	cmp obj-list current
+'

My main question about this test case is whether it _actually_ verifies that the batch-mode fsync()ing took place.

I kind of had expected to see Trace2 enabled and a grep for fsync/hardware-flush. Do you think that would still make sense to add?

Thank you for working on the fsync() aspects of Git! Dscho

Quoted 8 lines
t/t5351-unpack-large-objects.sh +1 -1
+
 test_expect_success 'do not unpack existing large objects' '
 	prepare_dest 1m &&
 	git -C dest.git index-pack --stdin <pack-$PACK.pack &&
--

2.36.1

author
#108 lore raw

Re: [RFC PATCH] object-file.c: batched disk flushes for stream_loose_object()

Quoted 56 lines · Johannes Schindelin

Hi,

Quoted 7 lines · Han Xin

Neeraj Singh[1] pointed out that if batch fsync is enabled, we should still call prepare_loose_object_bulk_checkin() to potentially create the bulk checkin objdir.

1. https://lore.kernel.org/git/[email protected]/

I like a good commit message that is concise and yet has all the necessary information. Well done!

Quoted 10 lines
---
 object-file.c                   |  3 +++
 t/t5351-unpack-large-objects.sh | 15 ++++++++++++++-
 2 files changed, 17 insertions(+), 1 deletion(-)
object-file.c +0 -0
index 2dd828b45b..3a1be74775 100644

object-file.c int stream_loose_object(struct input_stream *in_stream, size_t len, · line 2136

Quoted 6 lines
object-file.c +3 -0
@@ -2131,6 +2131,9 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
      char hdr[MAX_HEADER_LEN];
      int hdrlen;
 
+     if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
+             prepare_loose_object_bulk_checkin();
+

Makes sense.

Quoted 8 lines
object-file.c +0 -0
      /* Since oid is not determined, save tmp file to odb path. */
      strbuf_addf(&filename, "%s/", get_object_directory());
      hdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, len);
t/t5351-unpack-large-objects.sh +0 -0
index 461ca060b2..a66a51f7df 100755

t/t5351-unpack-large-objects.sh test_expect_success "create large objects (1.5 MB) and PACK" ' · line 24

Quoted 8 lines
t/t5351-unpack-large-objects.sh +4 -1
@@ -18,7 +18,10 @@ test_expect_success "create large objects (1.5 MB) and PACK" '
      test_commit --append foo big-blob &&
      test-tool genrandom bar 1500000 >big-blob &&
      test_commit --append bar big-blob &&
-     PACK=$(echo HEAD | git pack-objects --revs pack)
+     PACK=$(echo HEAD | git pack-objects --revs pack) &&
+     git verify-pack -v pack-$PACK.pack |
+         grep -E "commit|tree|blob" |
+             sed -n -e "s/^\([0-9a-f]*\).*/\1/p" >obj-list

Here, I would recommend avoiding the pipe, to ensure that we would catch problems in the verify-pack invocation, and I think we can avoid the grep altogether:

Quoted 4 lines
        git verify-pack -v pack-$PACK.pack >out &&
        sed -n 's/^\([0-9a-f][0-9a-f]*\).*\(commit\|tree\|blob\)/\1/p' \
                <out >obj-list

Good suggestion. I will take it.

Thanks.
-Han Xin

Quoted 25 lines

t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 55

Quoted 12 lines
t/t5351-unpack-large-objects.sh +5 -0
 '
 
 test_expect_success 'set memory limitation to 1MB' '
@@ -45,6 +48,16 @@ test_expect_success 'unpack big object in stream' '
      test_dir_is_empty dest.git/objects/pack
 '
 
+BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch'
+
+test_expect_success 'unpack big object in stream (core.fsyncmethod=batch)' '
+     prepare_dest 1m &&
+     git $BATCH_CONFIGURATION -C dest.git unpack-objects <pack-$PACK.pack &&

I think the canonical way would be to use test_config core.fsync ..., but the presented way works, too.

t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 57

Quoted 2 lines
t/t5351-unpack-large-objects.sh +2 -0
+     test_dir_is_empty dest.git/objects/pack &&
+     git -C dest.git cat-file --batch-check="%(objectname)" <obj-list >current &&

Good. The --batch-check="%(objectname)" part forces cat-file to read the actual object.

t/t5351-unpack-large-objects.sh test_expect_success 'unpack big object in stream' ' · line 59

Quoted 2 lines
t/t5351-unpack-large-objects.sh +2 -0
+     cmp obj-list current
+'
Quoted 9 lines

My main question about this test case is whether it _actually_ verifies that the batch-mode fsync()ing took place.

I kind of had expected to see Trace2 enabled and a grep for fsync/hardware-flush. Do you think that would still make sense to add?

Thank you for working on the fsync() aspects of Git! Dscho

More rigorous inspection should be adopted.

Thanks.
-Han Xin

Quoted 8 lines
Quoted 8 lines
t/t5351-unpack-large-objects.sh +1 -1
+
 test_expect_success 'do not unpack existing large objects' '
      prepare_dest 1m &&
      git -C dest.git index-pack --stdin <pack-$PACK.pack &&
--

2.36.1

#109 lore raw

PATCH v13 6/7 core doc: modernize core.bigFileThreshold documentation

In reply to Ævar Arnfjörð Bjarmason

The core.bigFileThreshold documentation has been largely unchanged since 5eef828bc03 (fast-import: Stream very large blobs directly to pack, 2010-02-01).

But since then this setting has been expanded to affect a lot more than that description indicated. Most notably in how "git diff" treats them, see 6bf3b813486 (diff --stat: mark any file larger than core.bigfilethreshold binary, 2014-08-16).

In addition to that, numerous commands and APIs make use of a streaming mode for files above this threshold.

So let's attempt to summarize 12 years of changes in behavior, which can be seen with:

    git log --oneline -Gbig_file_thre 5eef828bc03.. -- '*.c'

To do that turn this into a bullet-point list. The summary Han Xin produced in [1] helped a lot, but is a bit too detailed for documentation aimed at users. Let's instead summarize how user-observable behavior differs, and generally describe how we tend to stream these files in various commands.

1. https://lore.kernel.org/git/[email protected]/

---
 Documentation/config/core.txt | 33 ++++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 9 deletions(-)
Documentation/config/core.txt +24 -9
index 41e330f3069..ff6ae6bb647 100644
@@ -444,17 +444,32 @@ You probably do not need to adjust this value.
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
 core.bigFileThreshold::
-	Files larger than this size are stored deflated, without
-	attempting delta compression.  Storing large files without
-	delta compression avoids excessive memory usage, at the
-	slight expense of increased disk usage. Additionally files
-	larger than this size are always treated as binary.
+	The size of files considered "big", which as discussed below
+	changes the behavior of numerous git commands, as well as how
+	such files are stored within the repository. The default is
+	512 MiB. Common unit suffixes of 'k', 'm', or 'g' are
+	supported.
 +
-Default is 512 MiB on all platforms.  This should be reasonable
-for most projects as source code and other text files can still
-be delta compressed, but larger binary media files won't be.
+Files above the configured limit will be:
 +
-Common unit suffixes of 'k', 'm', or 'g' are supported.
+* Stored deflated, without attempting delta compression.
++
+The default limit is primarily set with this use-case in mind. With it
+most projects will have their source code and other text files delta
+compressed, but not larger binary media files.
++
+Storing large files without delta compression avoids excessive memory
+usage, at the slight expense of increased disk usage.
++
+* Will be treated as if though they were labeled "binary" (see
+  linkgit:gitattributes[5]). This means that e.g. linkgit:git-log[1]
+  and linkgit:git-diff[1] will not diffs for files above this limit.
++
+* Will be generally be streamed when written, which avoids excessive
+memory usage, at the cost of some fixed overhead. Commands that make
+use of this include linkgit:git-archive[1],
+linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
+linkgit:git-fsck[1].
 
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
Signature · 1 line
2.36.1.1124.g52838f02905
#110 lore raw

Re: [PATCH v13 6/7] core doc: modernize core.bigFileThreshold documentation

Quoted 10 lines · Ævar Arnfjörð Bjarmason

So let's attempt to summarize 12 years of changes in behavior, which can be seen with:

    git log --oneline -Gbig_file_thre 5eef828bc03.. -- '*.c'

To do that turn this into a bullet-point list. The summary Han Xin produced in [1] helped a lot, but is a bit too detailed for documentation aimed at users. Let's instead summarize how user-observable behavior differs, and generally describe how we tend to stream these files in various commands.

Nicely studied. Very much appreciated.

Quoted 9 lines

core.bigFileThreshold::

Documentation/config/core.txt +3 -5
-	Files larger than this size are stored deflated, without
-	attempting delta compression.  Storing large files without
-	delta compression avoids excessive memory usage, at the
-	slight expense of increased disk usage. Additionally files
-	larger than this size are always treated as binary.
+	The size of files considered "big", which as discussed below
+	changes the behavior of numerous git commands, as well as how
+	such files are stored within the repository. The default is

Documentation/config/core.txt You probably do not need to adjust this value. · line 455

Quoted 10 lines
Documentation/config/core.txt +4 -4
+	512 MiB. Common unit suffixes of 'k', 'm', or 'g' are
+	supported.
 +
-Default is 512 MiB on all platforms.  This should be reasonable
-for most projects as source code and other text files can still
-be delta compressed, but larger binary media files won't be.
+Files above the configured limit will be:
 +
-Common unit suffixes of 'k', 'm', or 'g' are supported.
+* Stored deflated, without attempting delta compression.

"even in packfiles" (with or without "even") is better be there in the sentence---loose objects are always stored deflated anyway.

Documentation/config/core.txt You probably do not need to adjust this value.

Quoted 10 lines
Documentation/config/core.txt +9 -0
+The default limit is primarily set with this use-case in mind. With it
+most projects will have their source code and other text files delta
+compressed, but not larger binary media files.
++
+Storing large files without delta compression avoids excessive memory
+usage, at the slight expense of increased disk usage.
 
+* Will be treated as if though they were labeled "binary" (see
+  linkgit:gitattributes[5]). This means that e.g. linkgit:git-log[1]
+  and linkgit:git-diff[1] will not diffs for files above this limit.

Good. You can lose three words "This means that" and the sentence means the same thing, so lose them (I always recommend people to reread the sentence when they write "This means that" with an eye to rewrite it better---it often is a sign that either the previous sentence is insufficiently clear, in which case it can be discarded and description after the three words can be enhanced to a better result).

Documentation/config/core.txt You probably do not need to adjust this value. · line 472

Quoted 5 lines
Documentation/config/core.txt +5 -0
+* Will be generally be streamed when written, which avoids excessive
+memory usage, at the cost of some fixed overhead. Commands that make
+use of this include linkgit:git-archive[1],
+linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
+linkgit:git-fsck[1].

Nice. And this series adds unpack-objects to the mix.

Quoted 2 lines

core.excludesFile::
Specifies the pathname to the file that contains patterns to

Excellent.

Thanks.

#111 lore raw

PATCH v13 7/7 unpack-objects: use stream_loose_object() to unpack large objects

In reply to Ævar Arnfjörð Bjarmason

From: Han Xin <[email protected]>

Make use of the stream_loose_object() function introduced in the preceding commit to unpack large objects. Before this we'd need to malloc() the size of the blob before unpacking it, which could cause OOM with very large blobs.

We could use the new streaming interface to unpack all blobs, but doing so would be much slower, as demonstrated e.g. with this benchmark using git-hyperfine[0]:

	rm -rf /tmp/scalar.git &&
	git clone --bare https://github.com/Microsoft/scalar.git /tmp/scalar.git &&
	mv /tmp/scalar.git/objects/pack/*.pack /tmp/scalar.git/my.pack &&
	git hyperfine \
		-r 2 --warmup 1 \
		-L rev origin/master,HEAD -L v "10,512,1k,1m" \
		-s 'make' \
		-p 'git init --bare dest.git' \
		-c 'rm -rf dest.git' \
		'./git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/scalar.git/my.pack'

Here we'll perform worse with lower core.bigFileThreshold settings with this change in terms of speed, but we're getting lower memory use in return:

Summary
'./git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' ran
1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.02 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.02 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.09 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.10 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.11 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD'

A better benchmark to demonstrate the benefits of that this one, which creates an artificial repo with a 1, 25, 50, 75 and 100MB blob:

	rm -rf /tmp/repo &&
	git init /tmp/repo &&
	(
		cd /tmp/repo &&
		for i in 1 25 50 75 100
		do
			dd if=/dev/urandom of=blob.$i count=$(($i*1024)) bs=1024
		done &&
		git add blob.* &&
		git commit -mblobs &&
		git gc &&
		PACK=$(echo .git/objects/pack/pack-*.pack) &&
		cp "$PACK" my.pack
	) &&
	git hyperfine \
		--show-output \
		-L rev origin/master,HEAD -L v "512,50m,100m" \
		-s 'make' \
		-p 'git init --bare dest.git' \
		-c 'rm -rf dest.git' \
		'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum'

Using this test we'll always use >100MB of memory on origin/master (around ~105MB), but max out at e.g. ~55MB if we set core.bigFileThreshold=50m.

The relevant "Maximum resident set size" lines were manually added below the relevant benchmark:

'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master' ran
Maximum resident set size (kbytes): 107080
1.02 ± 0.78 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 106968
1.09 ± 0.79 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 107032
1.42 ± 1.07 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 107072
1.83 ± 1.02 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 55704
2.16 ± 1.19 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 4564

This shows that if you have enough memory this new streaming method is slower the lower you set the streaming threshold, but the benefit is more bounded memory use.

An earlier version of this patch introduced a new "core.bigFileStreamingThreshold" instead of re-using the existing "core.bigFileThreshold" variable[1]. As noted in a detailed overview of its users in [2] using it has several different meanings.

Still, we consider it good enough to simply re-use it. While it's possible that someone might want to e.g. consider objects "small" for the purposes of diffing but "big" for the purposes of writing them such use-cases are probably too obscure to worry about. We can always split up "core.bigFileThreshold" in the future if there's a need for that.

0. https://github.com/avar/git-hyperfine/
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/

---
 Documentation/config/core.txt   |  4 +-
 builtin/unpack-objects.c        | 69 ++++++++++++++++++++++++++++++++-
 t/t5351-unpack-large-objects.sh | 26 +++++++++++--
 3 files changed, 93 insertions(+), 6 deletions(-)
Documentation/config/core.txt +2 -2
index ff6ae6bb647..b97bc7e3e55 100644
@@ -468,8 +468,8 @@ usage, at the slight expense of increased disk usage.
 * Will be generally be streamed when written, which avoids excessive
 memory usage, at the cost of some fixed overhead. Commands that make
 use of this include linkgit:git-archive[1],
-linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
-linkgit:git-fsck[1].
+linkgit:git-fast-import[1], linkgit:git-index-pack[1],
+linkgit:git-unpack-objects[1] and linkgit:git-fsck[1].
 
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
builtin/unpack-objects.c +68 -1

83 lines not on this page · raw message

t/t5351-unpack-large-objects.sh +23 -3
index 8d84313221c..461ca060b2b 100755
@@ -9,7 +9,8 @@ test_description='git unpack-objects with large objects'
 
 prepare_dest () {
 	test_when_finished "rm -rf dest.git" &&
-	git init --bare dest.git
+	git init --bare dest.git &&
+	git -C dest.git config core.bigFileThreshold "$1"
 }
 
 test_expect_success "create large objects (1.5 MB) and PACK" '
@@ -26,16 +27,35 @@ test_expect_success 'set memory limitation to 1MB' '
 '
 
 test_expect_success 'unpack-objects failed under memory limitation' '
-	prepare_dest &&
+	prepare_dest 2m &&
 	test_must_fail git -C dest.git unpack-objects <pack-$PACK.pack 2>err &&
 	grep "fatal: attempting to allocate" err
 '
 
 test_expect_success 'unpack-objects works with memory limitation in dry-run mode' '
-	prepare_dest &&
+	prepare_dest 2m &&
 	git -C dest.git unpack-objects -n <pack-$PACK.pack &&
 	test_stdout_line_count = 0 find dest.git/objects -type f &&
 	test_dir_is_empty dest.git/objects/pack
 '
 
+test_expect_success 'unpack big object in stream' '
+	prepare_dest 1m &&
+	git -C dest.git unpack-objects <pack-$PACK.pack &&
+	test_dir_is_empty dest.git/objects/pack
+'
+
+test_expect_success 'do not unpack existing large objects' '
+	prepare_dest 1m &&
+	git -C dest.git index-pack --stdin <pack-$PACK.pack &&
+	git -C dest.git unpack-objects <pack-$PACK.pack &&
+
+	# The destination came up with the exact same pack...
+	DEST_PACK=$(echo dest.git/objects/pack/pack-*.pack) &&
+	test_cmp pack-$PACK.pack $DEST_PACK &&
+
+	# ...and wrote no loose objects
+	test_stdout_line_count = 0 find dest.git/objects -type f ! -name "pack-*"
+'
+
 test_done
Signature · 1 line
2.36.1.1124.g52838f02905
author
#112 lore raw

v14 cover unpack-objects: support streaming blobs to disk

In reply to Ævar Arnfjörð Bjarmason

This series makes "unpack-objects" capable of streaming large objects to disk.

As 7/7 shows streaming e.g. a 100MB blob now uses ~5MB of memory instead of ~105MB. This streaming method is slower if you've got memory to handle the blobs in-core, but if you don't it allows you to unpack objects at all, as you might otherwise OOM.

Changes since v13:

* Make the error checking in the loop of get_data() the same way as we do in the non dry-run mode.

* Add batched disk flushes for stream_loose_object(). This is pointed out by Neeraj Singh[1].

* Minor typo/grammar/comment etc. fixes throughout.

1. https://lore.kernel.org/git/[email protected]/

Han Xin (4):
unpack-objects: low memory footprint for get_data() in dry_run mode object-file.c: refactor write_loose_object() to several steps object-file.c: add "stream_loose_object()" to handle large object
unpack-objects: use stream_loose_object() to unpack large objects

Ævar Arnfjörð Bjarmason (3):
object-file.c: do fsync() and close() before post-write die() object-file.c: factor out deflate part of write_loose_object() core doc: modernize core.bigFileThreshold documentation

 Documentation/config/core.txt   |  33 +++--
 builtin/unpack-objects.c        | 106 ++++++++++++--
 object-file.c                   | 240 +++++++++++++++++++++++++++-----
 object-store.h                  |   8 ++
 t/t5351-unpack-large-objects.sh |  76 ++++++++++
 5 files changed, 408 insertions(+), 55 deletions(-)
 create mode 100755 t/t5351-unpack-large-objects.sh

Range-diff against v13:
1: 6703df6350 ! 1: bf600a2fa8 unpack-objects: low memory footprint for get_data() in dry_run mode
@@ Commit message

Because in dry_run mode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can
- allocate a smaller buffer and reuse it as zstream output. Therefore,
- in dry_run mode, "get_data()" will release the allocated buffer and
- return NULL instead of returning garbage data.
+ allocate a smaller buffer and use it as zstream output. Make the function
+ return NULL in the dry-run mode, as no callers use the returned buffer.

The "find [...]objects/?? -type f | wc -l" test idiom being used here is adapted from the same "find" use added to another test in
@@ builtin/unpack-objects.c: static void use(int bytes)
}

+/*
-+ * Decompress zstream from stdin and return specific size of data. ++ * Decompress zstream from the standard input into a newly ++ * allocated buffer of specified size and return the buffer.
+ * The caller is responsible to free the returned buffer.
+ *
+ * But for dry_run mode, "get_data()" is only used to check the
@@ builtin/unpack-objects.c: static void *get_data(unsigned long size)
+ if (dry_run) {
+ /* reuse the buffer in dry_run mode */
+ stream.next_out = buf;
-+ stream.avail_out = bufsize;
++ stream.avail_out = bufsize > size - stream.total_out ?
++ size - stream.total_out :
++ bufsize;
+ }
}
git_inflate_end(&stream);

2:  6e289d25c1 = 2:  a327f484f7 object-file.c: do fsync() and close() before post-write die()
3:  46f9def06c ! 3:  9bc8002282 object-file.c: refactor write_loose_object() to several steps
    @@ object-file.c: static int create_tmpfile(struct strbuf *tmp, const char *filenam
     + *
     + * - End the compression of zlib stream.
     + * - Get the calculated oid to "oid".
    -+ * - fsync() and close() the "fd"
     + */
     +static int end_loose_object_common(git_hash_ctx *c, git_zstream *stream,
     +				   struct object_id *oid)
4:  5a95ebede6 = 4:  7c73815f18 object-file.c: factor out deflate part of write_loose_object()
5:  26847541aa ! 5:  28a9588f9c object-file.c: add "stream_loose_object()" to handle large object

@@ Commit message
path.

"freshen_packed_object()" or "freshen_loose_object()" will be called
- inside "stream_loose_object()" after obtaining the "oid".
+ inside "stream_loose_object()" after obtaining the "oid". After the
+ temporary file is written, we wants to mark the object to recent and we
+ may find that where indeed is already the object. We should remove the
+ temporary and do not leave a new copy of the object.

@@ object-file.c: static int freshen_packed_object(const struct object_id *oid)
+ char hdr[MAX_HEADER_LEN];
+ int hdrlen;
+
++ if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
++ prepare_loose_object_bulk_checkin();
++
+ /* Since oid is not determined, save tmp file to odb path. */
+ strbuf_addf(&filename, "%s/", get_object_directory());
+ hdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, len);
@@ object-file.c: static int freshen_packed_object(const struct object_id *oid)
+ die(_("write stream object %ld != %"PRIuMAX), stream.total_in,
+ (uintmax_t)len + hdrlen);
+
-+ /* Common steps for write_loose_object and stream_loose_object to
++ /*
++ * Common steps for write_loose_object and stream_loose_object to
+ * end writing loose oject:
+ *
+ * - End the compression of zlib stream.
6: eb962b60b9 ! 6: dea5c4172b core doc: modernize core.bigFileThreshold documentation
@@ Documentation/config/core.txt: You probably do not need to adjust this value.
+Files above the configured limit will be:
+
-Common unit suffixes of 'k', 'm', or 'g' are supported.
-+* Stored deflated, without attempting delta compression.
++* Stored deflated in packfiles, without attempting delta compression.
++
+The default limit is primarily set with this use-case in mind. With it +most projects will have their source code and other text files delta
@@ Documentation/config/core.txt: You probably do not need to adjust this value.
+usage, at the slight expense of increased disk usage.
++
+* Will be treated as if though they were labeled "binary" (see
-+ linkgit:gitattributes[5]). This means that e.g. linkgit:git-log[1]
-+ and linkgit:git-diff[1] will not diffs for files above this limit.
++ linkgit:gitattributes[5]). e.g. linkgit:git-log[1] and
++ linkgit:git-diff[1] will not diffs for files above this limit.
++
+* Will be generally be streamed when written, which avoids excessive +memory usage, at the cost of some fixed overhead. Commands that make
7: 88a2754fcb ! 7: d236230a4c unpack-objects: use stream_loose_object() to unpack large objects

    @@ t/t5351-unpack-large-objects.sh: test_description='git unpack-objects with large
      }
      
      test_expect_success "create large objects (1.5 MB) and PACK" '
    +@@ t/t5351-unpack-large-objects.sh: test_expect_success "create large objects (1.5 MB) and PACK" '
    + 	test_commit --append foo big-blob &&
    + 	test-tool genrandom bar 1500000 >big-blob &&
    + 	test_commit --append bar big-blob &&
    +-	PACK=$(echo HEAD | git pack-objects --revs pack)
    ++	PACK=$(echo HEAD | git pack-objects --revs pack) &&
    ++	git verify-pack -v pack-$PACK.pack >out &&
    ++	sed -n -e "s/^\([0-9a-f][0-9a-f]*\).*\(commit\|tree\|blob\).*/\1/p" \
    ++		<out >obj-list
    + '
    + 
    + test_expect_success 'set memory limitation to 1MB' '
     @@ t/t5351-unpack-large-objects.sh: test_expect_success 'set memory limitation to 1MB' '
      '
      
    @@ t/t5351-unpack-large-objects.sh: test_expect_success 'set memory limitation to 1
     +	test_dir_is_empty dest.git/objects/pack
     +'
     +
    ++BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch'
    ++
    ++test_expect_success 'unpack big object in stream (core.fsyncmethod=batch)' '
    ++	prepare_dest 1m &&
    ++	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \
    ++		git -C dest.git $BATCH_CONFIGURATION unpack-objects <pack-$PACK.pack &&
    ++	grep fsync/hardware-flush trace2.txt &&
    ++	test_dir_is_empty dest.git/objects/pack &&
    ++	git -C dest.git cat-file --batch-check="%(objectname)" <obj-list >current &&
    ++	cmp obj-list current
    ++'
    ++
     +test_expect_success 'do not unpack existing large objects' '
     +	prepare_dest 1m &&
     +	git -C dest.git index-pack --stdin <pack-$PACK.pack &&
Signature · 1 line
2.36.1
author
#113 lore raw

PATCH v14 1/7 unpack-objects: low memory footprint for get_data() in dry_run mode

From: Han Xin <[email protected]>

As the name implies, "get_data(size)" will allocate and return a given amount of memory. Allocating memory for a large blob object may cause the system to run out of memory. Before preparing to replace calling of "get_data()" to unpack large blob objects in latter commits, refactor "get_data()" to reduce memory footprint for dry_run mode.

Because in dry_run mode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can allocate a smaller buffer and use it as zstream output. Make the function return NULL in the dry-run mode, as no callers use the returned buffer.

The "find [...]objects/?? -type f | wc -l" test idiom being used here is adapted from the same "find" use added to another test in d9545c7f465 (fast-import: implement unpack limit, 2016-04-25).

---
 builtin/unpack-objects.c        | 37 ++++++++++++++++++++---------
 t/t5351-unpack-large-objects.sh | 41 +++++++++++++++++++++++++++++++++
 2 files changed, 67 insertions(+), 11 deletions(-)
 create mode 100755 t/t5351-unpack-large-objects.sh
builtin/unpack-objects.c +26 -11
index 56d05e2725..32e8b47059 100644
@@ -97,15 +97,27 @@ static void use(int bytes)
 	display_throughput(progress, consumed_bytes);
 }
 
+/*
+ * Decompress zstream from the standard input into a newly
+ * allocated buffer of specified size and return the buffer.
+ * The caller is responsible to free the returned buffer.
+ *
+ * But for dry_run mode, "get_data()" is only used to check the
+ * integrity of data, and the returned buffer is not used at all.
+ * Therefore, in dry_run mode, "get_data()" will release the small
+ * allocated buffer which is reused to hold temporary zstream output
+ * and return NULL instead of returning garbage data.
+ */
 static void *get_data(unsigned long size)
 {
 	git_zstream stream;
-	void *buf = xmallocz(size);
+	unsigned long bufsize = dry_run && size > 8192 ? 8192 : size;
+	void *buf = xmallocz(bufsize);
 
 	memset(&stream, 0, sizeof(stream));
 
 	stream.next_out = buf;
-	stream.avail_out = size;
+	stream.avail_out = bufsize;
 	stream.next_in = fill(1);
 	stream.avail_in = len;
 	git_inflate_init(&stream);
@@ -125,8 +137,17 @@ static void *get_data(unsigned long size)
 		}
 		stream.next_in = fill(1);
 		stream.avail_in = len;
+		if (dry_run) {
+			/* reuse the buffer in dry_run mode */
+			stream.next_out = buf;
+			stream.avail_out = bufsize > size - stream.total_out ?
+						   size - stream.total_out :
+						   bufsize;
+		}
 	}
 	git_inflate_end(&stream);
+	if (dry_run)
+		FREE_AND_NULL(buf);
 	return buf;
 }
 
@@ -326,10 +347,8 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 {
 	void *buf = get_data(size);
 
-	if (!dry_run && buf)
+	if (buf)
 		write_object(nr, type, buf, size);
-	else
-		free(buf);
 }
 
 static int resolve_against_held(unsigned nr, const struct object_id *base,
@@ -359,10 +378,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 		oidread(&base_oid, fill(the_hash_algo->rawsz));
 		use(the_hash_algo->rawsz);
 		delta_data = get_data(delta_size);
-		if (dry_run || !delta_data) {
-			free(delta_data);
+		if (!delta_data)
 			return;
-		}
 		if (has_object_file(&base_oid))
 			; /* Ok we have this one */
 		else if (resolve_against_held(nr, &base_oid,
@@ -398,10 +415,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 			die("offset value out of bound for delta base object");
 
 		delta_data = get_data(delta_size);
-		if (dry_run || !delta_data) {
-			free(delta_data);
+		if (!delta_data)
 			return;
-		}
 		lo = 0;
 		hi = nr;
 		while (lo < hi) {
t/t5351-unpack-large-objects.sh new file +41 -0

41 lines not on this page · raw message

Signature · 1 line
2.36.1
author
#114 lore raw

PATCH v14 2/7 object-file.c: do fsync() and close() before post-write die()

In reply to Han Xin

From: Ævar Arnfjörð Bjarmason <[email protected]>

Change write_loose_object() to do an fsync() and close() before the oideq() sanity check at the end. This change re-joins code that was split up by the die() sanity check added in 748af44c63e (sha1_file: be paranoid when creating loose objects, 2010-02-21).

I don't think that this change matters in itself, if we called die() it was possible that our data wouldn't fully make it to disk, but in any case we were writing data that we'd consider corrupted. It's possible that a subsequent "git fsck" will be less confused now.

The real reason to make this change is that in a subsequent commit we'll split this code in write_loose_object() into a utility function, all its callers will want the preceding sanity checks, but not the "oideq" check. By moving the close_loose_object() earlier it'll be easier to reason about the introduction of the utility function.

---
 object-file.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
object-file.c +2 -2
index 79eb8339b6..e4a83012ba 100644
@@ -2012,12 +2012,12 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
 		    ret);
 	the_hash_algo->final_oid_fn(&parano_oid, &c);
+	close_loose_object(fd, tmp_file.buf);
+
 	if (!oideq(oid, &parano_oid))
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));
 
-	close_loose_object(fd, tmp_file.buf);
-
 	if (mtime) {
 		struct utimbuf utb;
 		utb.actime = mtime;
Signature · 1 line
2.36.1
#115 lore raw

Re: [PATCH v14 2/7] object-file.c: do fsync() and close() before post-write die()

Am 10.06.22 um 16:46 schrieb Han Xin:

Quoted 11 lines

From: Ævar Arnfjörð Bjarmason <[email protected]>

Change write_loose_object() to do an fsync() and close() before the oideq() sanity check at the end. This change re-joins code that was split up by the die() sanity check added in 748af44c63e (sha1_file: be paranoid when creating loose objects, 2010-02-21).

I don't think that this change matters in itself, if we called die() it was possible that our data wouldn't fully make it to disk, but in any case we were writing data that we'd consider corrupted. It's possible that a subsequent "git fsck" will be less confused now.

This is done before renaming the file, so git fsck is going to see (at most) a tmp_obj_?????? file, which it ignores in either case, right?

Quoted 5 lines

The real reason to make this change is that in a subsequent commit we'll split this code in write_loose_object() into a utility function, all its callers will want the preceding sanity checks, but not the "oideq" check. By moving the close_loose_object() earlier it'll be easier to reason about the introduction of the utility function.

This sounds like a patch would move the close_loose_object() call to some other place, but that's not the case. The sequence below (starting from the close_loose_object() call) is still present after applying the whole series, so it seems this patch is not necessary.

Quoted 25 lines
---
 object-file.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
object-file.c +2 -2
index 79eb8339b6..e4a83012ba 100644
@@ -2012,12 +2012,12 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
 		    ret);
 	the_hash_algo->final_oid_fn(&parano_oid, &c);
+	close_loose_object(fd, tmp_file.buf);
+
 	if (!oideq(oid, &parano_oid))
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));
 
-	close_loose_object(fd, tmp_file.buf);
-
 	if (mtime) {
 		struct utimbuf utb;
 		utb.actime = mtime;
#116 lore raw

Re: [PATCH v14 2/7] object-file.c: do fsync() and close() before post-write die()

Quoted 8 lines · René Scharfe

Am 10.06.22 um 16:46 schrieb Han Xin:

Quoted 7 lines

From: Ævar Arnfjörð Bjarmason <[email protected]>

Change write_loose_object() to do an fsync() and close() before the oideq() sanity check at the end. This change re-joins code that was split up by the die() sanity check added in 748af44c63e (sha1_file: be paranoid when creating loose objects, 2010-02-21).

Quoted 7 lines
Quoted 4 lines

I don't think that this change matters in itself, if we called die() it was possible that our data wouldn't fully make it to disk, but in any case we were writing data that we'd consider corrupted. It's possible that a subsequent "git fsck" will be less confused now.

This is done before renaming the file, so git fsck is going to see (at most) a tmp_obj_?????? file, which it ignores in either case, right?

Yes, I thought I pointed that out in my review on the previous round, but I missed that it was still here in this round X-<.

Thanks for noticing.

author
#117 lore raw

Re: [PATCH v14 2/7] object-file.c: do fsync() and close() before post-write die()

Quoted 16 lines · Junio C Hamano
Quoted 13 lines · René Scharfe

Am 10.06.22 um 16:46 schrieb Han Xin:

Quoted 11 lines

From: Ævar Arnfjörð Bjarmason <[email protected]>

Change write_loose_object() to do an fsync() and close() before the oideq() sanity check at the end. This change re-joins code that was split up by the die() sanity check added in 748af44c63e (sha1_file: be paranoid when creating loose objects, 2010-02-21).

I don't think that this change matters in itself, if we called die() it was possible that our data wouldn't fully make it to disk, but in any case we were writing data that we'd consider corrupted. It's possible that a subsequent "git fsck" will be less confused now.

Quoted 8 lines
Quoted 2 lines

This is done before renaming the file, so git fsck is going to see (at most) a tmp_obj_?????? file, which it ignores in either case, right?

Yes, I thought I pointed that out in my review on the previous round, but I missed that it was still here in this round X-<.

Thanks for noticing.

Yes, agree with both of you, I'll be removing this patch in the next series.

This patch was first introduced in v10[1], close_loose_object() was moved to end_loose_object_common(), but it was put back in v12[2]. It is indeed no longer necessary now.

1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/

Thank you both.
-Han Xin

-Han Xin

author
#118 lore raw

PATCH v14 3/7 object-file.c: refactor write_loose_object() to several steps

In reply to Han Xin

From: Han Xin <[email protected]>

When writing a large blob using "write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in later commit to resolve this issue.

Before introducing that streaming function, do some refactoring on "write_loose_object()" to reuse code for both versions.

Rewrite "write_loose_object()" as follows:

1. Figure out a path for the (temp) object file. This step is only used in "write_loose_object()".

2. Move common steps for starting to write loose objects into a new function "start_loose_object_common()".

3. Compress data.

4. Move common steps for ending zlib stream into a new function

    "end_loose_object_common()".

5. Close fd and finalize the object file.

---
 object-file.c | 101 +++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 75 insertions(+), 26 deletions(-)
object-file.c +75 -26
index e4a83012ba..f4d7f8c109 100644
@@ -1951,6 +1951,74 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	return fd;
 }
 
+/**
+ * Common steps for loose object writers to start writing loose
+ * objects:
+ *
+ * - Create tmpfile for the loose object.
+ * - Setup zlib stream for compression.
+ * - Start to feed header to zlib stream.
+ *
+ * Returns a "fd", which should later be provided to
+ * end_loose_object_common().
+ */
+static int start_loose_object_common(struct strbuf *tmp_file,
+				     const char *filename, unsigned flags,
+				     git_zstream *stream,
+				     unsigned char *buf, size_t buflen,
+				     git_hash_ctx *c,
+				     char *hdr, int hdrlen)
+{
+	int fd;
+
+	fd = create_tmpfile(tmp_file, filename);
+	if (fd < 0) {
+		if (flags & HASH_SILENT)
+			return -1;
+		else if (errno == EACCES)
+			return error(_("insufficient permission for adding "
+				       "an object to repository database %s"),
+				     get_object_directory());
+		else
+			return error_errno(
+				_("unable to create temporary file"));
+	}
+
+	/*  Setup zlib stream for compression */
+	git_deflate_init(stream, zlib_compression_level);
+	stream->next_out = buf;
+	stream->avail_out = buflen;
+	the_hash_algo->init_fn(c);
+
+	/*  Start to feed header to zlib stream */
+	stream->next_in = (unsigned char *)hdr;
+	stream->avail_in = hdrlen;
+	while (git_deflate(stream, 0) == Z_OK)
+		; /* nothing */
+	the_hash_algo->update_fn(c, hdr, hdrlen);
+
+	return fd;
+}
+
+/**
+ * Common steps for loose object writers to end writing loose objects:
+ *
+ * - End the compression of zlib stream.
+ * - Get the calculated oid to "oid".
+ */
+static int end_loose_object_common(git_hash_ctx *c, git_zstream *stream,
+				   struct object_id *oid)
+{
+	int ret;
+
+	ret = git_deflate_end_gently(stream);
+	if (ret != Z_OK)
+		return ret;
+	the_hash_algo->final_oid_fn(oid, c);
+
+	return Z_OK;
+}
+
 static int write_loose_object(const struct object_id *oid, char *hdr,
 			      int hdrlen, const void *buf, unsigned long len,
 			      time_t mtime, unsigned flags)
@@ -1968,28 +2036,11 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 
 	loose_object_path(the_repository, &filename, oid);
 
-	fd = create_tmpfile(&tmp_file, filename.buf);
-	if (fd < 0) {
-		if (flags & HASH_SILENT)
-			return -1;
-		else if (errno == EACCES)
-			return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
-		else
-			return error_errno(_("unable to create temporary file"));
-	}
-
-	/* Set it up */
-	git_deflate_init(&stream, zlib_compression_level);
-	stream.next_out = compressed;
-	stream.avail_out = sizeof(compressed);
-	the_hash_algo->init_fn(&c);
-
-	/* First header.. */
-	stream.next_in = (unsigned char *)hdr;
-	stream.avail_in = hdrlen;
-	while (git_deflate(&stream, 0) == Z_OK)
-		; /* nothing */
-	the_hash_algo->update_fn(&c, hdr, hdrlen);
+	fd = start_loose_object_common(&tmp_file, filename.buf, flags,
+				       &stream, compressed, sizeof(compressed),
+				       &c, hdr, hdrlen);
+	if (fd < 0)
+		return -1;
 
 	/* Then the data itself.. */
 	stream.next_in = (void *)buf;
@@ -2007,11 +2058,9 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	if (ret != Z_STREAM_END)
 		die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
 		    ret);
-	ret = git_deflate_end_gently(&stream);
+	ret = end_loose_object_common(&c, &stream, &parano_oid);
 	if (ret != Z_OK)
-		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
-		    ret);
-	the_hash_algo->final_oid_fn(&parano_oid, &c);
+		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid), ret);
 	close_loose_object(fd, tmp_file.buf);
 
 	if (!oideq(oid, &parano_oid))
Signature · 1 line
2.36.1
author
#119 lore raw

PATCH v14 4/7 object-file.c: factor out deflate part of write_loose_object()

In reply to Han Xin

From: Ævar Arnfjörð Bjarmason <[email protected]>

Split out the part of write_loose_object() that deals with calling git_deflate() into a utility function, a subsequent commit will introduce another function that'll make use of it.

---
 object-file.c | 31 +++++++++++++++++++++++++------
 1 file changed, 25 insertions(+), 6 deletions(-)
object-file.c +25 -6
index f4d7f8c109..cfae54762e 100644
@@ -2000,6 +2000,28 @@ static int start_loose_object_common(struct strbuf *tmp_file,
 	return fd;
 }
 
+/**
+ * Common steps for the inner git_deflate() loop for writing loose
+ * objects. Returns what git_deflate() returns.
+ */
+static int write_loose_object_common(git_hash_ctx *c,
+				     git_zstream *stream, const int flush,
+				     unsigned char *in0, const int fd,
+				     unsigned char *compressed,
+				     const size_t compressed_len)
+{
+	int ret;
+
+	ret = git_deflate(stream, flush ? Z_FINISH : 0);
+	the_hash_algo->update_fn(c, in0, stream->next_in - in0);
+	if (write_buffer(fd, compressed, stream->next_out - compressed) < 0)
+		die(_("unable to write loose object file"));
+	stream->next_out = compressed;
+	stream->avail_out = compressed_len;
+
+	return ret;
+}
+
 /**
  * Common steps for loose object writers to end writing loose objects:
  *
@@ -2047,12 +2069,9 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	stream.avail_in = len;
 	do {
 		unsigned char *in0 = stream.next_in;
-		ret = git_deflate(&stream, Z_FINISH);
-		the_hash_algo->update_fn(&c, in0, stream.next_in - in0);
-		if (write_buffer(fd, compressed, stream.next_out - compressed) < 0)
-			die(_("unable to write loose object file"));
-		stream.next_out = compressed;
-		stream.avail_out = sizeof(compressed);
+
+		ret = write_loose_object_common(&c, &stream, 1, in0, fd,
+						compressed, sizeof(compressed));
 	} while (ret == Z_OK);
 
 	if (ret != Z_STREAM_END)
Signature · 1 line
2.36.1
author
#120 lore raw

PATCH v14 5/7 object-file.c: add "stream_loose_object()" to handle large object

In reply to Han Xin

From: Han Xin <[email protected]>

If we want unpack and write a loose object using "write_loose_object", we have to feed it with a buffer with the same size of the object, which will consume lots of memory and may cause OOM. This can be improved by feeding data to "stream_loose_object()" in a stream.

Add a new function "stream_loose_object()", which is a stream version of "write_loose_object()" but with a low memory footprint. We will use this function to unpack large blob object in later commit.

Another difference with "write_loose_object()" is that we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. In "write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object.

Still, we need to save the temporary file we're preparing somewhere. We'll do that in the top-level ".git/objects/" directory (or whatever "GIT_OBJECT_DIRECTORY" is set to). Once we've streamed it we'll know the OID, and will move it to its canonical path.

"freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid". After the temporary file is written, we wants to mark the object to recent and we may find that where indeed is already the object. We should remove the temporary and do not leave a new copy of the object.

---
 object-file.c  | 104 +++++++++++++++++++++++++++++++++++++++++++++++++
 object-store.h |   8 ++++
 2 files changed, 112 insertions(+)
object-file.c +104 -0
index cfae54762e..0b8383ad47 100644
@@ -2118,6 +2118,110 @@ static int freshen_packed_object(const struct object_id *oid)
 	return 1;
 }
 
+int stream_loose_object(struct input_stream *in_stream, size_t len,
+			struct object_id *oid)
+{
+	int fd, ret, err = 0, flush = 0;
+	unsigned char compressed[4096];
+	git_zstream stream;
+	git_hash_ctx c;
+	struct strbuf tmp_file = STRBUF_INIT;
+	struct strbuf filename = STRBUF_INIT;
+	int dirlen;
+	char hdr[MAX_HEADER_LEN];
+	int hdrlen;
+
+	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
+		prepare_loose_object_bulk_checkin();
+
+	/* Since oid is not determined, save tmp file to odb path. */
+	strbuf_addf(&filename, "%s/", get_object_directory());
+	hdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, len);
+
+	/*
+	 * Common steps for write_loose_object and stream_loose_object to
+	 * start writing loose objects:
+	 *
+	 *  - Create tmpfile for the loose object.
+	 *  - Setup zlib stream for compression.
+	 *  - Start to feed header to zlib stream.
+	 */
+	fd = start_loose_object_common(&tmp_file, filename.buf, 0,
+				       &stream, compressed, sizeof(compressed),
+				       &c, hdr, hdrlen);
+	if (fd < 0) {
+		err = -1;
+		goto cleanup;
+	}
+
+	/* Then the data itself.. */
+	do {
+		unsigned char *in0 = stream.next_in;
+
+		if (!stream.avail_in && !in_stream->is_finished) {
+			const void *in = in_stream->read(in_stream, &stream.avail_in);
+			stream.next_in = (void *)in;
+			in0 = (unsigned char *)in;
+			/* All data has been read. */
+			if (in_stream->is_finished)
+				flush = 1;
+		}
+		ret = write_loose_object_common(&c, &stream, flush, in0, fd,
+						compressed, sizeof(compressed));
+		/*
+		 * Unlike write_loose_object(), we do not have the entire
+		 * buffer. If we get Z_BUF_ERROR due to too few input bytes,
+		 * then we'll replenish them in the next input_stream->read()
+		 * call when we loop.
+		 */
+	} while (ret == Z_OK || ret == Z_BUF_ERROR);
+
+	if (stream.total_in != len + hdrlen)
+		die(_("write stream object %ld != %"PRIuMAX), stream.total_in,
+		    (uintmax_t)len + hdrlen);
+
+	/*
+	 * Common steps for write_loose_object and stream_loose_object to
+	 * end writing loose oject:
+	 *
+	 *  - End the compression of zlib stream.
+	 *  - Get the calculated oid.
+	 */
+	if (ret != Z_STREAM_END)
+		die(_("unable to stream deflate new object (%d)"), ret);
+	ret = end_loose_object_common(&c, &stream, oid);
+	if (ret != Z_OK)
+		die(_("deflateEnd on stream object failed (%d)"), ret);
+	close_loose_object(fd, tmp_file.buf);
+
+	if (freshen_packed_object(oid) || freshen_loose_object(oid)) {
+		unlink_or_warn(tmp_file.buf);
+		goto cleanup;
+	}
+
+	loose_object_path(the_repository, &filename, oid);
+
+	/* We finally know the object path, and create the missing dir. */
+	dirlen = directory_size(filename.buf);
+	if (dirlen) {
+		struct strbuf dir = STRBUF_INIT;
+		strbuf_add(&dir, filename.buf, dirlen);
+
+		if (mkdir_in_gitdir(dir.buf) && errno != EEXIST) {
+			err = error_errno(_("unable to create directory %s"), dir.buf);
+			strbuf_release(&dir);
+			goto cleanup;
+		}
+		strbuf_release(&dir);
+	}
+
+	err = finalize_object_file(tmp_file.buf, filename.buf);
+cleanup:
+	strbuf_release(&tmp_file);
+	strbuf_release(&filename);
+	return err;
+}
+
 int write_object_file_flags(const void *buf, unsigned long len,
 			    enum object_type type, struct object_id *oid,
 			    unsigned flags)
object-store.h +8 -0
index 539ea43904..5222ee5460 100644
@@ -46,6 +46,12 @@ struct object_directory {
 	char *path;
 };
 
+struct input_stream {
+	const void *(*read)(struct input_stream *, unsigned long *len);
+	void *data;
+	int is_finished;
+};
+
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
 	struct object_directory *, 1, fspathhash, fspatheq)
 
@@ -269,6 +275,8 @@ static inline int write_object_file(const void *buf, unsigned long len,
 int write_object_file_literally(const void *buf, unsigned long len,
 				const char *type, struct object_id *oid,
 				unsigned flags);
+int stream_loose_object(struct input_stream *in_stream, size_t len,
+			struct object_id *oid);
 
 /*
  * Add an object file to the in-memory object store, without writing it
Signature · 1 line
2.36.1
author
#121 lore raw

PATCH v14 6/7 core doc: modernize core.bigFileThreshold documentation

In reply to Han Xin

From: Ævar Arnfjörð Bjarmason <[email protected]>

The core.bigFileThreshold documentation has been largely unchanged since 5eef828bc03 (fast-import: Stream very large blobs directly to pack, 2010-02-01).

But since then this setting has been expanded to affect a lot more than that description indicated. Most notably in how "git diff" treats them, see 6bf3b813486 (diff --stat: mark any file larger than core.bigfilethreshold binary, 2014-08-16).

In addition to that, numerous commands and APIs make use of a streaming mode for files above this threshold.

So let's attempt to summarize 12 years of changes in behavior, which can be seen with:

    git log --oneline -Gbig_file_thre 5eef828bc03.. -- '*.c'

To do that turn this into a bullet-point list. The summary Han Xin produced in [1] helped a lot, but is a bit too detailed for documentation aimed at users. Let's instead summarize how user-observable behavior differs, and generally describe how we tend to stream these files in various commands.

1. https://lore.kernel.org/git/[email protected]/

---
 Documentation/config/core.txt | 33 ++++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 9 deletions(-)
Documentation/config/core.txt +24 -9
index 41e330f306..f2e75dd824 100644
@@ -444,17 +444,32 @@ You probably do not need to adjust this value.
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
 core.bigFileThreshold::
-	Files larger than this size are stored deflated, without
-	attempting delta compression.  Storing large files without
-	delta compression avoids excessive memory usage, at the
-	slight expense of increased disk usage. Additionally files
-	larger than this size are always treated as binary.
+	The size of files considered "big", which as discussed below
+	changes the behavior of numerous git commands, as well as how
+	such files are stored within the repository. The default is
+	512 MiB. Common unit suffixes of 'k', 'm', or 'g' are
+	supported.
 +
-Default is 512 MiB on all platforms.  This should be reasonable
-for most projects as source code and other text files can still
-be delta compressed, but larger binary media files won't be.
+Files above the configured limit will be:
 +
-Common unit suffixes of 'k', 'm', or 'g' are supported.
+* Stored deflated in packfiles, without attempting delta compression.
++
+The default limit is primarily set with this use-case in mind. With it
+most projects will have their source code and other text files delta
+compressed, but not larger binary media files.
++
+Storing large files without delta compression avoids excessive memory
+usage, at the slight expense of increased disk usage.
++
+* Will be treated as if though they were labeled "binary" (see
+  linkgit:gitattributes[5]). e.g. linkgit:git-log[1] and
+  linkgit:git-diff[1] will not diffs for files above this limit.
++
+* Will be generally be streamed when written, which avoids excessive
+memory usage, at the cost of some fixed overhead. Commands that make
+use of this include linkgit:git-archive[1],
+linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
+linkgit:git-fsck[1].
 
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
Signature · 1 line
2.36.1
#122 lore raw

Re: [PATCH v14 6/7] core doc: modernize core.bigFileThreshold documentation

Documentation/config/core.txt You probably do not need to adjust this value. · line 457

Quoted 13 lines · Han Xin

core.bigFileThreshold::

Documentation/config/core.txt +9 -0
+	The size of files considered "big", which as discussed below
+	changes the behavior of numerous git commands, as well as how
+	such files are stored within the repository. The default is
+	512 MiB. Common unit suffixes of 'k', 'm', or 'g' are
+	supported.
 +
+Files above the configured limit will be:
 +
+* Stored deflated in packfiles, without attempting delta compression.
++
+The default limit is primarily set with this use-case in mind. With it

"With it" -> "With it,"

Documentation/config/core.txt You probably do not need to adjust this value. · line 462

Quoted 5 lines
Documentation/config/core.txt +5 -0
+most projects will have their source code and other text files delta
+compressed, but not larger binary media files.
+
+Storing large files without delta compression avoids excessive memory
+usage, at the slight expense of increased disk usage.

Makes sense.

Quoted 1 line

+* Will be treated as if though they were labeled "binary" (see

"as if though" -> "as if"

Documentation/config/core.txt You probably do not need to adjust this value. · line 466

Quoted 2 lines
Documentation/config/core.txt +2 -0
+  linkgit:gitattributes[5]). e.g. linkgit:git-log[1] and
+  linkgit:git-diff[1] will not diffs for files above this limit.

"will not diffs" -ECANNOTPARSE. "will not compute diffs", probably?

Quoted 2 lines

++
+* Will be generally be streamed when written, which avoids excessive

"be generally be" -> "generally be"

Quoted 7 lines
Documentation/config/core.txt +4 -0
+memory usage, at the cost of some fixed overhead. Commands that make
+use of this include linkgit:git-archive[1],
+linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
+linkgit:git-fsck[1].
 
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
author
#123 lore raw

PATCH v14 7/7 unpack-objects: use stream_loose_object() to unpack large objects

In reply to Han Xin

From: Han Xin <[email protected]>

Make use of the stream_loose_object() function introduced in the preceding commit to unpack large objects. Before this we'd need to malloc() the size of the blob before unpacking it, which could cause OOM with very large blobs.

We could use the new streaming interface to unpack all blobs, but doing so would be much slower, as demonstrated e.g. with this benchmark using git-hyperfine[0]:

	rm -rf /tmp/scalar.git &&
	git clone --bare https://github.com/Microsoft/scalar.git /tmp/scalar.git &&
	mv /tmp/scalar.git/objects/pack/*.pack /tmp/scalar.git/my.pack &&
	git hyperfine \
		-r 2 --warmup 1 \
		-L rev origin/master,HEAD -L v "10,512,1k,1m" \
		-s 'make' \
		-p 'git init --bare dest.git' \
		-c 'rm -rf dest.git' \
		'./git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/scalar.git/my.pack'

Here we'll perform worse with lower core.bigFileThreshold settings with this change in terms of speed, but we're getting lower memory use in return:

Summary
'./git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' ran
1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.02 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.02 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.09 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.10 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.11 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD'

A better benchmark to demonstrate the benefits of that this one, which creates an artificial repo with a 1, 25, 50, 75 and 100MB blob:

	rm -rf /tmp/repo &&
	git init /tmp/repo &&
	(
		cd /tmp/repo &&
		for i in 1 25 50 75 100
		do
			dd if=/dev/urandom of=blob.$i count=$(($i*1024)) bs=1024
		done &&
		git add blob.* &&
		git commit -mblobs &&
		git gc &&
		PACK=$(echo .git/objects/pack/pack-*.pack) &&
		cp "$PACK" my.pack
	) &&
	git hyperfine \
		--show-output \
		-L rev origin/master,HEAD -L v "512,50m,100m" \
		-s 'make' \
		-p 'git init --bare dest.git' \
		-c 'rm -rf dest.git' \
		'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum'

Using this test we'll always use >100MB of memory on origin/master (around ~105MB), but max out at e.g. ~55MB if we set core.bigFileThreshold=50m.

The relevant "Maximum resident set size" lines were manually added below the relevant benchmark:

'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master' ran
Maximum resident set size (kbytes): 107080
1.02 ± 0.78 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 106968
1.09 ± 0.79 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 107032
1.42 ± 1.07 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 107072
1.83 ± 1.02 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 55704
2.16 ± 1.19 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 4564

This shows that if you have enough memory this new streaming method is slower the lower you set the streaming threshold, but the benefit is more bounded memory use.

An earlier version of this patch introduced a new "core.bigFileStreamingThreshold" instead of re-using the existing "core.bigFileThreshold" variable[1]. As noted in a detailed overview of its users in [2] using it has several different meanings.

Still, we consider it good enough to simply re-use it. While it's possible that someone might want to e.g. consider objects "small" for the purposes of diffing but "big" for the purposes of writing them such use-cases are probably too obscure to worry about. We can always split up "core.bigFileThreshold" in the future if there's a need for that.

0. https://github.com/avar/git-hyperfine/
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/

---
 Documentation/config/core.txt   |  4 +-
 builtin/unpack-objects.c        | 69 ++++++++++++++++++++++++++++++++-
 t/t5351-unpack-large-objects.sh | 43 ++++++++++++++++++--
 3 files changed, 109 insertions(+), 7 deletions(-)
Documentation/config/core.txt +2 -2
index f2e75dd824..a599dcb96b 100644
@@ -468,8 +468,8 @@ usage, at the slight expense of increased disk usage.
 * Will be generally be streamed when written, which avoids excessive
 memory usage, at the cost of some fixed overhead. Commands that make
 use of this include linkgit:git-archive[1],
-linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
-linkgit:git-fsck[1].
+linkgit:git-fast-import[1], linkgit:git-index-pack[1],
+linkgit:git-unpack-objects[1] and linkgit:git-fsck[1].
 
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
builtin/unpack-objects.c +68 -1

83 lines not on this page · raw message

t/t5351-unpack-large-objects.sh +39 -4

69 lines not on this page · raw message

Signature · 1 line
2.36.1
author
#124 lore raw

v15 cover unpack-objects: support streaming blobs to disk

In reply to Han Xin

This series makes "unpack-objects" capable of streaming large objects to disk.

As 6/6 shows streaming e.g. a 100MB blob now uses ~5MB of memory instead of ~105MB. This streaming method is slower if you've got memory to handle the blobs in-core, but if you don't it allows you to unpack objects at all, as you might otherwise OOM.

Changes since v14:

* Remove "object-file.c: do fsync() and close() before post-write die()" as it's not necessary anymore. It was first introduced in v10 and was no longer in the utility function end_loose_object_common() since v12. We can see the discussion[1].

* Minor grammar/comment etc. fixes throughout.

1. https://lore.kernel.org/git/[email protected]/

Han Xin (4):
unpack-objects: low memory footprint for get_data() in dry_run mode object-file.c: refactor write_loose_object() to several steps object-file.c: add "stream_loose_object()" to handle large object
unpack-objects: use stream_loose_object() to unpack large objects

Ævar Arnfjörð Bjarmason (2):
object-file.c: factor out deflate part of write_loose_object() core doc: modernize core.bigFileThreshold documentation

 Documentation/config/core.txt   |  33 +++--
 builtin/unpack-objects.c        | 106 +++++++++++++--
 object-file.c                   | 233 ++++++++++++++++++++++++++++----
 object-store.h                  |   8 ++
 t/t5351-unpack-large-objects.sh |  76 +++++++++++
 5 files changed, 405 insertions(+), 51 deletions(-)
 create mode 100755 t/t5351-unpack-large-objects.sh

Range-diff against v14:
1: bf600a2fa8 ! 1: 9a776f717d unpack-objects: low memory footprint for get_data() in dry_run mode
@@ Commit message
d9545c7f465 (fast-import: implement unpack limit, 2016-04-25).

- Signed-off-by: Han Xin <[email protected]>
+ Signed-off-by: Han Xin <[email protected]>

## builtin/unpack-objects.c ##

2:  a327f484f7 < -:  ---------- object-file.c: do fsync() and close() before post-write die()
3:  9bc8002282 ! 2:  a1e090d338 object-file.c: refactor write_loose_object() to several steps

@@ Commit message

- Signed-off-by: Han Xin <[email protected]>
+ Signed-off-by: Han Xin <[email protected]>

## object-file.c ##
@@ object-file.c: static int write_loose_object(const struct object_id *oid, char *
- ret = git_deflate_end_gently(&stream);
+ ret = end_loose_object_common(&c, &stream, &parano_oid);
if (ret != Z_OK)
-- die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
-- ret);
+ die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
+ ret);
- the_hash_algo->final_oid_fn(&parano_oid, &c);
-+ die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid), ret);
- close_loose_object(fd, tmp_file.buf);
-
if (!oideq(oid, &parano_oid))
+ die(_("confused by unstable object source data for %s"),
+ oid_to_hex(oid));

4:  7c73815f18 = 3:  0ddf912d47 object-file.c: factor out deflate part of write_loose_object()
5:  28a9588f9c ! 4:  f9e51d3c68 object-file.c: add "stream_loose_object()" to handle large object

@@ Commit message

- Signed-off-by: Han Xin <[email protected]>
+ Signed-off-by: Han Xin <[email protected]>

## object-file.c ##
6: dea5c4172b ! 5: 61ae1c1632 core doc: modernize core.bigFileThreshold documentation
@@ Documentation/config/core.txt: You probably do not need to adjust this value.
-Common unit suffixes of 'k', 'm', or 'g' are supported.
+* Stored deflated in packfiles, without attempting delta compression.
++
-+The default limit is primarily set with this use-case in mind. With it ++The default limit is primarily set with this use-case in mind. With it,
+most projects will have their source code and other text files delta +compressed, but not larger binary media files.
++
+Storing large files without delta compression avoids excessive memory +usage, at the slight expense of increased disk usage.
++
-+* Will be treated as if though they were labeled "binary" (see
++* Will be treated as if they were labeled "binary" (see
+ linkgit:gitattributes[5]). e.g. linkgit:git-log[1] and
-+ linkgit:git-diff[1] will not diffs for files above this limit.
++ linkgit:git-diff[1] will not compute diffs for files above this limit.
++
-+* Will be generally be streamed when written, which avoids excessive
++* Will generally be streamed when written, which avoids excessive
+memory usage, at the cost of some fixed overhead. Commands that make
+use of this include linkgit:git-archive[1],
+linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
7: d236230a4c ! 6: 5a4782d746 unpack-objects: use stream_loose_object() to unpack large objects
@@ Commit message

- Signed-off-by: Han Xin <[email protected]>
+ Signed-off-by: Han Xin <[email protected]>

## Documentation/config/core.txt ##
@@ Documentation/config/core.txt: usage, at the slight expense of increased disk usage.
- * Will be generally be streamed when written, which avoids excessive
+ * Will generally be streamed when written, which avoids excessive memory usage, at the cost of some fixed overhead. Commands that make use of this include linkgit:git-archive[1], -linkgit:git-fast-import[1], linkgit:git-index-pack[1] and

Signature · 1 line
2.36.1
author
#125 lore raw

PATCH v15 1/6 unpack-objects: low memory footprint for get_data() in dry_run mode

From: Han Xin <[email protected]>

As the name implies, "get_data(size)" will allocate and return a given amount of memory. Allocating memory for a large blob object may cause the system to run out of memory. Before preparing to replace calling of "get_data()" to unpack large blob objects in latter commits, refactor "get_data()" to reduce memory footprint for dry_run mode.

Because in dry_run mode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can allocate a smaller buffer and use it as zstream output. Make the function return NULL in the dry-run mode, as no callers use the returned buffer.

The "find [...]objects/?? -type f | wc -l" test idiom being used here is adapted from the same "find" use added to another test in d9545c7f465 (fast-import: implement unpack limit, 2016-04-25).

---
 builtin/unpack-objects.c        | 37 ++++++++++++++++++++---------
 t/t5351-unpack-large-objects.sh | 41 +++++++++++++++++++++++++++++++++
 2 files changed, 67 insertions(+), 11 deletions(-)
 create mode 100755 t/t5351-unpack-large-objects.sh
builtin/unpack-objects.c +26 -11
index 56d05e2725..32e8b47059 100644
@@ -97,15 +97,27 @@ static void use(int bytes)
 	display_throughput(progress, consumed_bytes);
 }
 
+/*
+ * Decompress zstream from the standard input into a newly
+ * allocated buffer of specified size and return the buffer.
+ * The caller is responsible to free the returned buffer.
+ *
+ * But for dry_run mode, "get_data()" is only used to check the
+ * integrity of data, and the returned buffer is not used at all.
+ * Therefore, in dry_run mode, "get_data()" will release the small
+ * allocated buffer which is reused to hold temporary zstream output
+ * and return NULL instead of returning garbage data.
+ */
 static void *get_data(unsigned long size)
 {
 	git_zstream stream;
-	void *buf = xmallocz(size);
+	unsigned long bufsize = dry_run && size > 8192 ? 8192 : size;
+	void *buf = xmallocz(bufsize);
 
 	memset(&stream, 0, sizeof(stream));
 
 	stream.next_out = buf;
-	stream.avail_out = size;
+	stream.avail_out = bufsize;
 	stream.next_in = fill(1);
 	stream.avail_in = len;
 	git_inflate_init(&stream);
@@ -125,8 +137,17 @@ static void *get_data(unsigned long size)
 		}
 		stream.next_in = fill(1);
 		stream.avail_in = len;
+		if (dry_run) {
+			/* reuse the buffer in dry_run mode */
+			stream.next_out = buf;
+			stream.avail_out = bufsize > size - stream.total_out ?
+						   size - stream.total_out :
+						   bufsize;
+		}
 	}
 	git_inflate_end(&stream);
+	if (dry_run)
+		FREE_AND_NULL(buf);
 	return buf;
 }
 
@@ -326,10 +347,8 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 {
 	void *buf = get_data(size);
 
-	if (!dry_run && buf)
+	if (buf)
 		write_object(nr, type, buf, size);
-	else
-		free(buf);
 }
 
 static int resolve_against_held(unsigned nr, const struct object_id *base,
@@ -359,10 +378,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 		oidread(&base_oid, fill(the_hash_algo->rawsz));
 		use(the_hash_algo->rawsz);
 		delta_data = get_data(delta_size);
-		if (dry_run || !delta_data) {
-			free(delta_data);
+		if (!delta_data)
 			return;
-		}
 		if (has_object_file(&base_oid))
 			; /* Ok we have this one */
 		else if (resolve_against_held(nr, &base_oid,
@@ -398,10 +415,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 			die("offset value out of bound for delta base object");
 
 		delta_data = get_data(delta_size);
-		if (dry_run || !delta_data) {
-			free(delta_data);
+		if (!delta_data)
 			return;
-		}
 		lo = 0;
 		hi = nr;
 		while (lo < hi) {
t/t5351-unpack-large-objects.sh new file +41 -0

41 lines not on this page · raw message

Signature · 1 line
2.36.1
author
#126 lore raw

PATCH v15 2/6 object-file.c: refactor write_loose_object() to several steps

In reply to Han Xin

From: Han Xin <[email protected]>

When writing a large blob using "write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in later commit to resolve this issue.

Before introducing that streaming function, do some refactoring on "write_loose_object()" to reuse code for both versions.

Rewrite "write_loose_object()" as follows:

1. Figure out a path for the (temp) object file. This step is only used in "write_loose_object()".

2. Move common steps for starting to write loose objects into a new function "start_loose_object_common()".

3. Compress data.

4. Move common steps for ending zlib stream into a new function

    "end_loose_object_common()".

5. Close fd and finalize the object file.

---
 object-file.c | 98 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 74 insertions(+), 24 deletions(-)
object-file.c +74 -24
index 79eb8339b6..b5bce03274 100644
@@ -1951,6 +1951,74 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	return fd;
 }
 
+/**
+ * Common steps for loose object writers to start writing loose
+ * objects:
+ *
+ * - Create tmpfile for the loose object.
+ * - Setup zlib stream for compression.
+ * - Start to feed header to zlib stream.
+ *
+ * Returns a "fd", which should later be provided to
+ * end_loose_object_common().
+ */
+static int start_loose_object_common(struct strbuf *tmp_file,
+				     const char *filename, unsigned flags,
+				     git_zstream *stream,
+				     unsigned char *buf, size_t buflen,
+				     git_hash_ctx *c,
+				     char *hdr, int hdrlen)
+{
+	int fd;
+
+	fd = create_tmpfile(tmp_file, filename);
+	if (fd < 0) {
+		if (flags & HASH_SILENT)
+			return -1;
+		else if (errno == EACCES)
+			return error(_("insufficient permission for adding "
+				       "an object to repository database %s"),
+				     get_object_directory());
+		else
+			return error_errno(
+				_("unable to create temporary file"));
+	}
+
+	/*  Setup zlib stream for compression */
+	git_deflate_init(stream, zlib_compression_level);
+	stream->next_out = buf;
+	stream->avail_out = buflen;
+	the_hash_algo->init_fn(c);
+
+	/*  Start to feed header to zlib stream */
+	stream->next_in = (unsigned char *)hdr;
+	stream->avail_in = hdrlen;
+	while (git_deflate(stream, 0) == Z_OK)
+		; /* nothing */
+	the_hash_algo->update_fn(c, hdr, hdrlen);
+
+	return fd;
+}
+
+/**
+ * Common steps for loose object writers to end writing loose objects:
+ *
+ * - End the compression of zlib stream.
+ * - Get the calculated oid to "oid".
+ */
+static int end_loose_object_common(git_hash_ctx *c, git_zstream *stream,
+				   struct object_id *oid)
+{
+	int ret;
+
+	ret = git_deflate_end_gently(stream);
+	if (ret != Z_OK)
+		return ret;
+	the_hash_algo->final_oid_fn(oid, c);
+
+	return Z_OK;
+}
+
 static int write_loose_object(const struct object_id *oid, char *hdr,
 			      int hdrlen, const void *buf, unsigned long len,
 			      time_t mtime, unsigned flags)
@@ -1968,28 +2036,11 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 
 	loose_object_path(the_repository, &filename, oid);
 
-	fd = create_tmpfile(&tmp_file, filename.buf);
-	if (fd < 0) {
-		if (flags & HASH_SILENT)
-			return -1;
-		else if (errno == EACCES)
-			return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
-		else
-			return error_errno(_("unable to create temporary file"));
-	}
-
-	/* Set it up */
-	git_deflate_init(&stream, zlib_compression_level);
-	stream.next_out = compressed;
-	stream.avail_out = sizeof(compressed);
-	the_hash_algo->init_fn(&c);
-
-	/* First header.. */
-	stream.next_in = (unsigned char *)hdr;
-	stream.avail_in = hdrlen;
-	while (git_deflate(&stream, 0) == Z_OK)
-		; /* nothing */
-	the_hash_algo->update_fn(&c, hdr, hdrlen);
+	fd = start_loose_object_common(&tmp_file, filename.buf, flags,
+				       &stream, compressed, sizeof(compressed),
+				       &c, hdr, hdrlen);
+	if (fd < 0)
+		return -1;
 
 	/* Then the data itself.. */
 	stream.next_in = (void *)buf;
@@ -2007,11 +2058,10 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	if (ret != Z_STREAM_END)
 		die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
 		    ret);
-	ret = git_deflate_end_gently(&stream);
+	ret = end_loose_object_common(&c, &stream, &parano_oid);
 	if (ret != Z_OK)
 		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
 		    ret);
-	the_hash_algo->final_oid_fn(&parano_oid, &c);
 	if (!oideq(oid, &parano_oid))
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));
Signature · 1 line
2.36.1
author
#127 lore raw

PATCH v15 3/6 object-file.c: factor out deflate part of write_loose_object()

In reply to Han Xin

From: Ævar Arnfjörð Bjarmason <[email protected]>

Split out the part of write_loose_object() that deals with calling git_deflate() into a utility function, a subsequent commit will introduce another function that'll make use of it.

---
 object-file.c | 31 +++++++++++++++++++++++++------
 1 file changed, 25 insertions(+), 6 deletions(-)
object-file.c +25 -6
index b5bce03274..18dbf2a4e4 100644
@@ -2000,6 +2000,28 @@ static int start_loose_object_common(struct strbuf *tmp_file,
 	return fd;
 }
 
+/**
+ * Common steps for the inner git_deflate() loop for writing loose
+ * objects. Returns what git_deflate() returns.
+ */
+static int write_loose_object_common(git_hash_ctx *c,
+				     git_zstream *stream, const int flush,
+				     unsigned char *in0, const int fd,
+				     unsigned char *compressed,
+				     const size_t compressed_len)
+{
+	int ret;
+
+	ret = git_deflate(stream, flush ? Z_FINISH : 0);
+	the_hash_algo->update_fn(c, in0, stream->next_in - in0);
+	if (write_buffer(fd, compressed, stream->next_out - compressed) < 0)
+		die(_("unable to write loose object file"));
+	stream->next_out = compressed;
+	stream->avail_out = compressed_len;
+
+	return ret;
+}
+
 /**
  * Common steps for loose object writers to end writing loose objects:
  *
@@ -2047,12 +2069,9 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	stream.avail_in = len;
 	do {
 		unsigned char *in0 = stream.next_in;
-		ret = git_deflate(&stream, Z_FINISH);
-		the_hash_algo->update_fn(&c, in0, stream.next_in - in0);
-		if (write_buffer(fd, compressed, stream.next_out - compressed) < 0)
-			die(_("unable to write loose object file"));
-		stream.next_out = compressed;
-		stream.avail_out = sizeof(compressed);
+
+		ret = write_loose_object_common(&c, &stream, 1, in0, fd,
+						compressed, sizeof(compressed));
 	} while (ret == Z_OK);
 
 	if (ret != Z_STREAM_END)
Signature · 1 line
2.36.1
author
#128 lore raw

PATCH v15 4/6 object-file.c: add "stream_loose_object()" to handle large object

In reply to Han Xin

From: Han Xin <[email protected]>

If we want unpack and write a loose object using "write_loose_object", we have to feed it with a buffer with the same size of the object, which will consume lots of memory and may cause OOM. This can be improved by feeding data to "stream_loose_object()" in a stream.

Add a new function "stream_loose_object()", which is a stream version of "write_loose_object()" but with a low memory footprint. We will use this function to unpack large blob object in later commit.

Another difference with "write_loose_object()" is that we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. In "write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object.

Still, we need to save the temporary file we're preparing somewhere. We'll do that in the top-level ".git/objects/" directory (or whatever "GIT_OBJECT_DIRECTORY" is set to). Once we've streamed it we'll know the OID, and will move it to its canonical path.

"freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid". After the temporary file is written, we wants to mark the object to recent and we may find that where indeed is already the object. We should remove the temporary and do not leave a new copy of the object.

---
 object-file.c  | 104 +++++++++++++++++++++++++++++++++++++++++++++++++
 object-store.h |   8 ++++
 2 files changed, 112 insertions(+)
object-file.c +104 -0
index 18dbf2a4e4..2ca2576ab1 100644
@@ -2119,6 +2119,110 @@ static int freshen_packed_object(const struct object_id *oid)
 	return 1;
 }
 
+int stream_loose_object(struct input_stream *in_stream, size_t len,
+			struct object_id *oid)
+{
+	int fd, ret, err = 0, flush = 0;
+	unsigned char compressed[4096];
+	git_zstream stream;
+	git_hash_ctx c;
+	struct strbuf tmp_file = STRBUF_INIT;
+	struct strbuf filename = STRBUF_INIT;
+	int dirlen;
+	char hdr[MAX_HEADER_LEN];
+	int hdrlen;
+
+	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
+		prepare_loose_object_bulk_checkin();
+
+	/* Since oid is not determined, save tmp file to odb path. */
+	strbuf_addf(&filename, "%s/", get_object_directory());
+	hdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, len);
+
+	/*
+	 * Common steps for write_loose_object and stream_loose_object to
+	 * start writing loose objects:
+	 *
+	 *  - Create tmpfile for the loose object.
+	 *  - Setup zlib stream for compression.
+	 *  - Start to feed header to zlib stream.
+	 */
+	fd = start_loose_object_common(&tmp_file, filename.buf, 0,
+				       &stream, compressed, sizeof(compressed),
+				       &c, hdr, hdrlen);
+	if (fd < 0) {
+		err = -1;
+		goto cleanup;
+	}
+
+	/* Then the data itself.. */
+	do {
+		unsigned char *in0 = stream.next_in;
+
+		if (!stream.avail_in && !in_stream->is_finished) {
+			const void *in = in_stream->read(in_stream, &stream.avail_in);
+			stream.next_in = (void *)in;
+			in0 = (unsigned char *)in;
+			/* All data has been read. */
+			if (in_stream->is_finished)
+				flush = 1;
+		}
+		ret = write_loose_object_common(&c, &stream, flush, in0, fd,
+						compressed, sizeof(compressed));
+		/*
+		 * Unlike write_loose_object(), we do not have the entire
+		 * buffer. If we get Z_BUF_ERROR due to too few input bytes,
+		 * then we'll replenish them in the next input_stream->read()
+		 * call when we loop.
+		 */
+	} while (ret == Z_OK || ret == Z_BUF_ERROR);
+
+	if (stream.total_in != len + hdrlen)
+		die(_("write stream object %ld != %"PRIuMAX), stream.total_in,
+		    (uintmax_t)len + hdrlen);
+
+	/*
+	 * Common steps for write_loose_object and stream_loose_object to
+	 * end writing loose oject:
+	 *
+	 *  - End the compression of zlib stream.
+	 *  - Get the calculated oid.
+	 */
+	if (ret != Z_STREAM_END)
+		die(_("unable to stream deflate new object (%d)"), ret);
+	ret = end_loose_object_common(&c, &stream, oid);
+	if (ret != Z_OK)
+		die(_("deflateEnd on stream object failed (%d)"), ret);
+	close_loose_object(fd, tmp_file.buf);
+
+	if (freshen_packed_object(oid) || freshen_loose_object(oid)) {
+		unlink_or_warn(tmp_file.buf);
+		goto cleanup;
+	}
+
+	loose_object_path(the_repository, &filename, oid);
+
+	/* We finally know the object path, and create the missing dir. */
+	dirlen = directory_size(filename.buf);
+	if (dirlen) {
+		struct strbuf dir = STRBUF_INIT;
+		strbuf_add(&dir, filename.buf, dirlen);
+
+		if (mkdir_in_gitdir(dir.buf) && errno != EEXIST) {
+			err = error_errno(_("unable to create directory %s"), dir.buf);
+			strbuf_release(&dir);
+			goto cleanup;
+		}
+		strbuf_release(&dir);
+	}
+
+	err = finalize_object_file(tmp_file.buf, filename.buf);
+cleanup:
+	strbuf_release(&tmp_file);
+	strbuf_release(&filename);
+	return err;
+}
+
 int write_object_file_flags(const void *buf, unsigned long len,
 			    enum object_type type, struct object_id *oid,
 			    unsigned flags)
object-store.h +8 -0
index 539ea43904..5222ee5460 100644
@@ -46,6 +46,12 @@ struct object_directory {
 	char *path;
 };
 
+struct input_stream {
+	const void *(*read)(struct input_stream *, unsigned long *len);
+	void *data;
+	int is_finished;
+};
+
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
 	struct object_directory *, 1, fspathhash, fspatheq)
 
@@ -269,6 +275,8 @@ static inline int write_object_file(const void *buf, unsigned long len,
 int write_object_file_literally(const void *buf, unsigned long len,
 				const char *type, struct object_id *oid,
 				unsigned flags);
+int stream_loose_object(struct input_stream *in_stream, size_t len,
+			struct object_id *oid);
 
 /*
  * Add an object file to the in-memory object store, without writing it
Signature · 1 line
2.36.1
author
#129 lore raw

PATCH v15 5/6 core doc: modernize core.bigFileThreshold documentation

In reply to Han Xin

From: Ævar Arnfjörð Bjarmason <[email protected]>

The core.bigFileThreshold documentation has been largely unchanged since 5eef828bc03 (fast-import: Stream very large blobs directly to pack, 2010-02-01).

But since then this setting has been expanded to affect a lot more than that description indicated. Most notably in how "git diff" treats them, see 6bf3b813486 (diff --stat: mark any file larger than core.bigfilethreshold binary, 2014-08-16).

In addition to that, numerous commands and APIs make use of a streaming mode for files above this threshold.

So let's attempt to summarize 12 years of changes in behavior, which can be seen with:

    git log --oneline -Gbig_file_thre 5eef828bc03.. -- '*.c'

To do that turn this into a bullet-point list. The summary Han Xin produced in [1] helped a lot, but is a bit too detailed for documentation aimed at users. Let's instead summarize how user-observable behavior differs, and generally describe how we tend to stream these files in various commands.

1. https://lore.kernel.org/git/[email protected]/

---
 Documentation/config/core.txt | 33 ++++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 9 deletions(-)
Documentation/config/core.txt +24 -9
index 41e330f306..87e4c04836 100644
@@ -444,17 +444,32 @@ You probably do not need to adjust this value.
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
 core.bigFileThreshold::
-	Files larger than this size are stored deflated, without
-	attempting delta compression.  Storing large files without
-	delta compression avoids excessive memory usage, at the
-	slight expense of increased disk usage. Additionally files
-	larger than this size are always treated as binary.
+	The size of files considered "big", which as discussed below
+	changes the behavior of numerous git commands, as well as how
+	such files are stored within the repository. The default is
+	512 MiB. Common unit suffixes of 'k', 'm', or 'g' are
+	supported.
 +
-Default is 512 MiB on all platforms.  This should be reasonable
-for most projects as source code and other text files can still
-be delta compressed, but larger binary media files won't be.
+Files above the configured limit will be:
 +
-Common unit suffixes of 'k', 'm', or 'g' are supported.
+* Stored deflated in packfiles, without attempting delta compression.
++
+The default limit is primarily set with this use-case in mind. With it,
+most projects will have their source code and other text files delta
+compressed, but not larger binary media files.
++
+Storing large files without delta compression avoids excessive memory
+usage, at the slight expense of increased disk usage.
++
+* Will be treated as if they were labeled "binary" (see
+  linkgit:gitattributes[5]). e.g. linkgit:git-log[1] and
+  linkgit:git-diff[1] will not compute diffs for files above this limit.
++
+* Will generally be streamed when written, which avoids excessive
+memory usage, at the cost of some fixed overhead. Commands that make
+use of this include linkgit:git-archive[1],
+linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
+linkgit:git-fsck[1].
 
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
Signature · 1 line
2.36.1
author
#130 lore raw

PATCH v15 6/6 unpack-objects: use stream_loose_object() to unpack large objects

In reply to Han Xin

From: Han Xin <[email protected]>

Make use of the stream_loose_object() function introduced in the preceding commit to unpack large objects. Before this we'd need to malloc() the size of the blob before unpacking it, which could cause OOM with very large blobs.

We could use the new streaming interface to unpack all blobs, but doing so would be much slower, as demonstrated e.g. with this benchmark using git-hyperfine[0]:

	rm -rf /tmp/scalar.git &&
	git clone --bare https://github.com/Microsoft/scalar.git /tmp/scalar.git &&
	mv /tmp/scalar.git/objects/pack/*.pack /tmp/scalar.git/my.pack &&
	git hyperfine \
		-r 2 --warmup 1 \
		-L rev origin/master,HEAD -L v "10,512,1k,1m" \
		-s 'make' \
		-p 'git init --bare dest.git' \
		-c 'rm -rf dest.git' \
		'./git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/scalar.git/my.pack'

Here we'll perform worse with lower core.bigFileThreshold settings with this change in terms of speed, but we're getting lower memory use in return:

Summary
'./git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' ran
1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.01 ± 0.02 times faster than './git -C dest.git -c core.bigFileThreshold=1m unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.02 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'origin/master' 1.09 ± 0.01 times faster than './git -C dest.git -c core.bigFileThreshold=1k unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.10 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD' 1.11 ± 0.00 times faster than './git -C dest.git -c core.bigFileThreshold=10 unpack-objects </tmp/scalar.git/my.pack' in 'HEAD'

A better benchmark to demonstrate the benefits of that this one, which creates an artificial repo with a 1, 25, 50, 75 and 100MB blob:

	rm -rf /tmp/repo &&
	git init /tmp/repo &&
	(
		cd /tmp/repo &&
		for i in 1 25 50 75 100
		do
			dd if=/dev/urandom of=blob.$i count=$(($i*1024)) bs=1024
		done &&
		git add blob.* &&
		git commit -mblobs &&
		git gc &&
		PACK=$(echo .git/objects/pack/pack-*.pack) &&
		cp "$PACK" my.pack
	) &&
	git hyperfine \
		--show-output \
		-L rev origin/master,HEAD -L v "512,50m,100m" \
		-s 'make' \
		-p 'git init --bare dest.git' \
		-c 'rm -rf dest.git' \
		'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold={v} unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum'

Using this test we'll always use >100MB of memory on origin/master (around ~105MB), but max out at e.g. ~55MB if we set core.bigFileThreshold=50m.

The relevant "Maximum resident set size" lines were manually added below the relevant benchmark:

'/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master' ran
Maximum resident set size (kbytes): 107080
1.02 ± 0.78 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 106968
1.09 ± 0.79 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'origin/master'
Maximum resident set size (kbytes): 107032
1.42 ± 1.07 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=100m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 107072
1.83 ± 1.02 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=50m unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 55704
2.16 ± 1.19 times faster than '/usr/bin/time -v ./git -C dest.git -c core.bigFileThreshold=512 unpack-objects </tmp/repo/my.pack 2>&1 | grep Maximum' in 'HEAD'
Maximum resident set size (kbytes): 4564

This shows that if you have enough memory this new streaming method is slower the lower you set the streaming threshold, but the benefit is more bounded memory use.

An earlier version of this patch introduced a new "core.bigFileStreamingThreshold" instead of re-using the existing "core.bigFileThreshold" variable[1]. As noted in a detailed overview of its users in [2] using it has several different meanings.

Still, we consider it good enough to simply re-use it. While it's possible that someone might want to e.g. consider objects "small" for the purposes of diffing but "big" for the purposes of writing them such use-cases are probably too obscure to worry about. We can always split up "core.bigFileThreshold" in the future if there's a need for that.

0. https://github.com/avar/git-hyperfine/
1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/

---
 Documentation/config/core.txt   |  4 +-
 builtin/unpack-objects.c        | 69 ++++++++++++++++++++++++++++++++-
 t/t5351-unpack-large-objects.sh | 43 ++++++++++++++++++--
 3 files changed, 109 insertions(+), 7 deletions(-)
Documentation/config/core.txt +2 -2
index 87e4c04836..3ea3124f7f 100644
@@ -468,8 +468,8 @@ usage, at the slight expense of increased disk usage.
 * Will generally be streamed when written, which avoids excessive
 memory usage, at the cost of some fixed overhead. Commands that make
 use of this include linkgit:git-archive[1],
-linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
-linkgit:git-fsck[1].
+linkgit:git-fast-import[1], linkgit:git-index-pack[1],
+linkgit:git-unpack-objects[1] and linkgit:git-fsck[1].
 
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
builtin/unpack-objects.c +68 -1

83 lines not on this page · raw message

t/t5351-unpack-large-objects.sh +39 -4

69 lines not on this page · raw message

Signature · 1 line
2.36.1
#131 lore raw

Re: [PATCH v15 6/6] unpack-objects: use stream_loose_object() to unpack large objects

t/t5351-unpack-large-objects.sh test_expect_success 'set memory limitation to 1MB' ' · line 61

Quoted 11 lines · Han Xin
t/t5351-unpack-large-objects.sh +11 -0
+BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch'
+
+test_expect_success 'unpack big object in stream (core.fsyncmethod=batch)' '
+	prepare_dest 1m &&
+	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \
+		git -C dest.git $BATCH_CONFIGURATION unpack-objects <pack-$PACK.pack &&
+	grep fsync/hardware-flush trace2.txt &&
+	test_dir_is_empty dest.git/objects/pack &&
+	git -C dest.git cat-file --batch-check="%(objectname)" <obj-list >current &&
+	cmp obj-list current
+'

This test without any prerequisite expects that "hardware-flush" will always appear in the trace, but is that reasonable? Don't we need either

(1) some sort of prerequisite to make sure this test piece runs only on platforms that will use hardware-flush, or

(2) loosen grep pattern to look for just "fsync/", or

(3) something else?

It will become even worse when we queue Ævar's "trace2 squelch" patch on top, as we will stop emitting trace entries for that did not trigger.

author
#132 lore raw

cover unpack-objects: low memory footprint for get_data() in dry_run mode

In reply to Ævar Arnfjörð Bjarmason

This patch teaches "git unpack-objects" to use a lower memory footprint for "get_data()" in dry-run mode since the returned data is not used.

This patch is separeted from "[PATCH v12 0/8] unpack-objects: support streaming blobs to disk"[1] because it has less impact and less controversy on existing ones.

1. https://lore.kernel.org/git/[email protected]/

Han Xin (1):
unpack-objects: low memory footprint for get_data() in dry_run mode

 builtin/unpack-objects.c        | 34 ++++++++++++++++++---------
 t/t5351-unpack-large-objects.sh | 41 +++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+), 11 deletions(-)
 create mode 100755 t/t5351-unpack-large-objects.sh
Signature · 1 line
2.36.1
author
#133 lore raw

PATCH 1/1 unpack-objects: low memory footprint for get_data() in dry_run mode

From: Han Xin <[email protected]>

As the name implies, "get_data(size)" will allocate and return a given amount of memory. Allocating memory for a large blob object may cause the system to run out of memory. Before preparing to replace calling of "get_data()" to unpack large blob objects in latter commits, refactor "get_data()" to reduce memory footprint for dry_run mode.

Because in dry_run mode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can allocate a smaller buffer and reuse it as zstream output. Therefore, in dry_run mode, "get_data()" will release the allocated buffer and return NULL instead of returning garbage data.

The "find [...]objects/?? -type f | wc -l" test idiom being used here is adapted from the same "find" use added to another test in d9545c7f465 (fast-import: implement unpack limit, 2016-04-25).

---
 builtin/unpack-objects.c        | 34 ++++++++++++++++++---------
 t/t5351-unpack-large-objects.sh | 41 +++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+), 11 deletions(-)
 create mode 100755 t/t5351-unpack-large-objects.sh
builtin/unpack-objects.c +23 -11
index dbeb0680a5..e3d3002597 100644
@@ -96,15 +96,26 @@ static void use(int bytes)
 	display_throughput(progress, consumed_bytes);
 }
 
+/*
+ * Decompress zstream from stdin and return specific size of data.
+ * The caller is responsible to free the returned buffer.
+ *
+ * But for dry_run mode, "get_data()" is only used to check the
+ * integrity of data, and the returned buffer is not used at all.
+ * Therefore, in dry_run mode, "get_data()" will release the small
+ * allocated buffer which is reused to hold temporary zstream output
+ * and return NULL instead of returning garbage data.
+ */
 static void *get_data(unsigned long size)
 {
 	git_zstream stream;
-	void *buf = xmallocz(size);
+	unsigned long bufsize = dry_run && size > 8192 ? 8192 : size;
+	void *buf = xmallocz(bufsize);
 
 	memset(&stream, 0, sizeof(stream));
 
 	stream.next_out = buf;
-	stream.avail_out = size;
+	stream.avail_out = bufsize;
 	stream.next_in = fill(1);
 	stream.avail_in = len;
 	git_inflate_init(&stream);
@@ -124,8 +135,15 @@ static void *get_data(unsigned long size)
 		}
 		stream.next_in = fill(1);
 		stream.avail_in = len;
+		if (dry_run) {
+			/* reuse the buffer in dry_run mode */
+			stream.next_out = buf;
+			stream.avail_out = bufsize;
+		}
 	}
 	git_inflate_end(&stream);
+	if (dry_run)
+		FREE_AND_NULL(buf);
 	return buf;
 }
 
@@ -325,10 +343,8 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 {
 	void *buf = get_data(size);
 
-	if (!dry_run && buf)
+	if (buf)
 		write_object(nr, type, buf, size);
-	else
-		free(buf);
 }
 
 static int resolve_against_held(unsigned nr, const struct object_id *base,
@@ -358,10 +374,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 		oidread(&base_oid, fill(the_hash_algo->rawsz));
 		use(the_hash_algo->rawsz);
 		delta_data = get_data(delta_size);
-		if (dry_run || !delta_data) {
-			free(delta_data);
+		if (!delta_data)
 			return;
-		}
 		if (has_object_file(&base_oid))
 			; /* Ok we have this one */
 		else if (resolve_against_held(nr, &base_oid,
@@ -397,10 +411,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 			die("offset value out of bound for delta base object");
 
 		delta_data = get_data(delta_size);
-		if (dry_run || !delta_data) {
-			free(delta_data);
+		if (!delta_data)
 			return;
-		}
 		lo = 0;
 		hi = nr;
 		while (lo < hi) {
t/t5351-unpack-large-objects.sh new file +41 -0

41 lines not on this page · raw message

Signature · 1 line
2.36.1
author
#134 lore raw

PATCH v9 2/5 object-file.c: refactor write_loose_object() to several steps

In reply to Han Xin

From: Han Xin <[email protected]>

When writing a large blob using "write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in latter commit to resolve this issue.

Before introducing a stream vesion function for writing loose object, do some refactoring on "write_loose_object()" to reuse code for both versions.

Rewrite "write_loose_object()" as follows:

1. Figure out a path for the (temp) object file. This step is only used in "write_loose_object()".

2. Move common steps for starting to write loose objects into a new function "start_loose_object_common()".

3. Compress data.

4. Move common steps for ending zlib stream into a new funciton

    "end_loose_object_common()".

5. Close fd and finalize the object file.

---
 object-file.c | 105 +++++++++++++++++++++++++++++++++++---------------
 1 file changed, 75 insertions(+), 30 deletions(-)
object-file.c +75 -30
index eb1426f98c..422b43212a 100644
@@ -1857,6 +1857,59 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	return fd;
 }
 
+static int start_loose_object_common(struct strbuf *tmp_file,
+				     const char *filename, unsigned flags,
+				     git_zstream *stream,
+				     unsigned char *buf, size_t buflen,
+				     git_hash_ctx *c,
+				     char *hdr, int hdrlen)
+{
+	int fd;
+
+	fd = create_tmpfile(tmp_file, filename);
+	if (fd < 0) {
+		if (flags & HASH_SILENT)
+			return -1;
+		else if (errno == EACCES)
+			return error(_("insufficient permission for adding "
+				       "an object to repository database %s"),
+				     get_object_directory());
+		else
+			return error_errno(
+				_("unable to create temporary file"));
+	}
+
+	/*  Setup zlib stream for compression */
+	git_deflate_init(stream, zlib_compression_level);
+	stream->next_out = buf;
+	stream->avail_out = buflen;
+	the_hash_algo->init_fn(c);
+
+	/*  Start to feed header to zlib stream */
+	stream->next_in = (unsigned char *)hdr;
+	stream->avail_in = hdrlen;
+	while (git_deflate(stream, 0) == Z_OK)
+		; /* nothing */
+	the_hash_algo->update_fn(c, hdr, hdrlen);
+
+	return fd;
+}
+
+static void end_loose_object_common(int ret, git_hash_ctx *c,
+				    git_zstream *stream,
+				    struct object_id *parano_oid,
+				    const struct object_id *expected_oid,
+				    const char *die_msg1_fmt,
+				    const char *die_msg2_fmt)
+{
+	if (ret != Z_STREAM_END)
+		die(_(die_msg1_fmt), ret, expected_oid);
+	ret = git_deflate_end_gently(stream);
+	if (ret != Z_OK)
+		die(_(die_msg2_fmt), ret, expected_oid);
+	the_hash_algo->final_oid_fn(parano_oid, c);
+}
+
 static int write_loose_object(const struct object_id *oid, char *hdr,
 			      int hdrlen, const void *buf, unsigned long len,
 			      time_t mtime, unsigned flags)
@@ -1871,28 +1924,18 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 
 	loose_object_path(the_repository, &filename, oid);
 
-	fd = create_tmpfile(&tmp_file, filename.buf);
-	if (fd < 0) {
-		if (flags & HASH_SILENT)
-			return -1;
-		else if (errno == EACCES)
-			return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
-		else
-			return error_errno(_("unable to create temporary file"));
-	}
-
-	/* Set it up */
-	git_deflate_init(&stream, zlib_compression_level);
-	stream.next_out = compressed;
-	stream.avail_out = sizeof(compressed);
-	the_hash_algo->init_fn(&c);
-
-	/* First header.. */
-	stream.next_in = (unsigned char *)hdr;
-	stream.avail_in = hdrlen;
-	while (git_deflate(&stream, 0) == Z_OK)
-		; /* nothing */
-	the_hash_algo->update_fn(&c, hdr, hdrlen);
+	/* Common steps for write_loose_object and stream_loose_object to
+	 * start writing loose oject:
+	 *
+	 *  - Create tmpfile for the loose object.
+	 *  - Setup zlib stream for compression.
+	 *  - Start to feed header to zlib stream.
+	 */
+	fd = start_loose_object_common(&tmp_file, filename.buf, flags,
+				       &stream, compressed, sizeof(compressed),
+				       &c, hdr, hdrlen);
+	if (fd < 0)
+		return -1;
 
 	/* Then the data itself.. */
 	stream.next_in = (void *)buf;
@@ -1907,14 +1950,16 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 		stream.avail_out = sizeof(compressed);
 	} while (ret == Z_OK);
 
-	if (ret != Z_STREAM_END)
-		die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
-		    ret);
-	ret = git_deflate_end_gently(&stream);
-	if (ret != Z_OK)
-		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
-		    ret);
-	the_hash_algo->final_oid_fn(&parano_oid, &c);
+	/* Common steps for write_loose_object and stream_loose_object to
+	 * end writing loose oject:
+	 *
+	 *  - End the compression of zlib stream.
+	 *  - Get the calculated oid to "parano_oid".
+	 */
+	end_loose_object_common(ret, &c, &stream, &parano_oid, oid,
+				N_("unable to deflate new object %s (%d)"),
+				N_("deflateEnd on object %s failed (%d)"));
+
 	if (!oideq(oid, &parano_oid))
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
author
#135 lore raw

PATCH v9 3/5 object-file.c: add "stream_loose_object()" to handle large object

In reply to Han Xin

From: Han Xin <[email protected]>

If we want unpack and write a loose object using "write_loose_object", we have to feed it with a buffer with the same size of the object, which will consume lots of memory and may cause OOM. This can be improved by feeding data to "stream_loose_object()" in a stream.

Add a new function "stream_loose_object()", which is a stream version of "write_loose_object()" but with a low memory footprint. We will use this function to unpack large blob object in latter commit.

Another difference with "write_loose_object()" is that we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. In "write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

"freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid".

---
 object-file.c  | 101 +++++++++++++++++++++++++++++++++++++++++++++++++
 object-store.h |   9 +++++
 2 files changed, 110 insertions(+)
object-file.c +101 -0
index 422b43212a..a738f47cb2 100644
@@ -1996,6 +1996,107 @@ static int freshen_packed_object(const struct object_id *oid)
 	return 1;
 }
 
+int stream_loose_object(struct input_stream *in_stream, size_t len,
+			struct object_id *oid)
+{
+	int fd, ret, err = 0, flush = 0;
+	unsigned char compressed[4096];
+	git_zstream stream;
+	git_hash_ctx c;
+	struct strbuf tmp_file = STRBUF_INIT;
+	struct strbuf filename = STRBUF_INIT;
+	int dirlen;
+	char hdr[MAX_HEADER_LEN];
+	int hdrlen;
+
+	/* Since oid is not determined, save tmp file to odb path. */
+	strbuf_addf(&filename, "%s/", get_object_directory());
+	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX, type_name(OBJ_BLOB), len) + 1;
+
+	/* Common steps for write_loose_object and stream_loose_object to
+	 * start writing loose oject:
+	 *
+	 *  - Create tmpfile for the loose object.
+	 *  - Setup zlib stream for compression.
+	 *  - Start to feed header to zlib stream.
+	 */
+	fd = start_loose_object_common(&tmp_file, filename.buf, 0,
+				       &stream, compressed, sizeof(compressed),
+				       &c, hdr, hdrlen);
+	if (fd < 0) {
+		err = -1;
+		goto cleanup;
+	}
+
+	/* Then the data itself.. */
+	do {
+		unsigned char *in0 = stream.next_in;
+		if (!stream.avail_in && !in_stream->is_finished) {
+			const void *in = in_stream->read(in_stream, &stream.avail_in);
+			stream.next_in = (void *)in;
+			in0 = (unsigned char *)in;
+			/* All data has been read. */
+			if (in_stream->is_finished)
+				flush = Z_FINISH;
+		}
+		ret = git_deflate(&stream, flush);
+		the_hash_algo->update_fn(&c, in0, stream.next_in - in0);
+		if (write_buffer(fd, compressed, stream.next_out - compressed) < 0)
+			die(_("unable to write loose object file"));
+		stream.next_out = compressed;
+		stream.avail_out = sizeof(compressed);
+		/*
+		 * Unlike write_loose_object(), we do not have the entire
+		 * buffer. If we get Z_BUF_ERROR due to too few input bytes,
+		 * then we'll replenish them in the next input_stream->read()
+		 * call when we loop.
+		 */
+	} while (ret == Z_OK || ret == Z_BUF_ERROR);
+
+	if (stream.total_in != len + hdrlen)
+		die(_("write stream object %ld != %"PRIuMAX), stream.total_in,
+		    (uintmax_t)len + hdrlen);
+
+	/* Common steps for write_loose_object and stream_loose_object to
+	 * end writing loose oject:
+	 *
+	 *  - End the compression of zlib stream.
+	 *  - Get the calculated oid.
+	 */
+	end_loose_object_common(ret, &c, &stream, oid, NULL,
+				N_("unable to stream deflate new object (%d)"),
+				N_("deflateEnd on stream object failed (%d)"));
+
+	close_loose_object(fd);
+
+	if (freshen_packed_object(oid) || freshen_loose_object(oid)) {
+		unlink_or_warn(tmp_file.buf);
+		goto cleanup;
+	}
+
+	loose_object_path(the_repository, &filename, oid);
+
+	/* We finally know the object path, and create the missing dir. */
+	dirlen = directory_size(filename.buf);
+	if (dirlen) {
+		struct strbuf dir = STRBUF_INIT;
+		strbuf_add(&dir, filename.buf, dirlen);
+
+		if (mkdir_in_gitdir(dir.buf) && errno != EEXIST) {
+			err = error_errno(_("unable to create directory %s"), dir.buf);
+			strbuf_release(&dir);
+			goto cleanup;
+		}
+		strbuf_release(&dir);
+	}
+
+	err = finalize_object_file(tmp_file.buf, filename.buf);
+cleanup:
+	strbuf_release(&tmp_file);
+	strbuf_release(&filename);
+	return err;
+}
+
 int write_object_file_flags(const void *buf, unsigned long len,
 			    const char *type, struct object_id *oid,
 			    unsigned flags)
object-store.h +9 -0
index 952efb6a4b..cc41c64d69 100644
@@ -34,6 +34,12 @@ struct object_directory {
 	char *path;
 };
 
+struct input_stream {
+	const void *(*read)(struct input_stream *, unsigned long *len);
+	void *data;
+	int is_finished;
+};
+
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
 	struct object_directory *, 1, fspathhash, fspatheq)
 
@@ -232,6 +238,9 @@ static inline int write_object_file(const void *buf, unsigned long len,
 	return write_object_file_flags(buf, len, type, oid, 0);
 }
 
+int stream_loose_object(struct input_stream *in_stream, size_t len,
+			struct object_id *oid);
+
 int hash_object_file_literally(const void *buf, unsigned long len,
 			       const char *type, struct object_id *oid,
 			       unsigned flags);
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
author
#136 lore raw

PATCH v9 4/5 unpack-objects: unpack_non_delta_entry() read data in a stream

In reply to Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()". However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty.

    $ hyperfine \
      --setup \
      'if ! test -d scalar.git; then git clone --bare
       https://github.com/microsoft/scalar.git;
       cp scalar.git/objects/pack/*.pack small.pack; fi' \
      --prepare 'rm -rf dest.git && git init --bare dest.git' \
      ...
    Summary
      './git -C dest.git -c core.bigFileThreshold=512m
      unpack-objects <small.pack' in 'origin/master'
        1.01 ± 0.04 times faster than './git -C dest.git
                -c core.bigFileThreshold=512m unpack-objects
                <small.pack' in 'HEAD~1'
        1.01 ± 0.04 times faster than './git -C dest.git
                -c core.bigFileThreshold=512m unpack-objects
                <small.pack' in 'HEAD~0'
        1.03 ± 0.10 times faster than './git -C dest.git
                -c core.bigFileThreshold=16k unpack-objects
                <small.pack' in 'origin/master'
        1.02 ± 0.07 times faster than './git -C dest.git
                -c core.bigFileThreshold=16k unpack-objects
                <small.pack' in 'HEAD~0'
        1.10 ± 0.04 times faster than './git -C dest.git
                -c core.bigFileThreshold=16k unpack-objects
                <small.pack' in 'HEAD~1'

Therefore, only unpack objects larger than the "core.bigFileThreshold" in zstream. Until now, the config variable has been used in the following cases, and our new case belongs to the packfile category.

* Archive:

+ archive.c: write_entry(): write large blob entries to archive in stream.

* Loose objects:

+ object-file.c: index_fd(): when hashing large files in worktree, read files in a stream, and create one packfile per large blob if want to save files to git object store.

+ object-file.c: read_loose_object(): when checking loose objects using "git-fsck", do not read full content of large loose objects.

* Packfile:

+ fast-import.c: parse_and_store_blob(): streaming large blob from foreign source to packfile.

+ index-pack.c: check_collison(): read and check large blob in stream.

+ index-pack.c: unpack_entry_data(): do not return the entire contents of the big blob from packfile, but uses a fixed buf to perform some integrity checks on the object.

+ pack-check.c: verify_packfile(): used by "git-fsck" and will call check_object_signature() to check large blob in pack with the streaming interface.

+ pack-objects.c: get_object_details(): set "no_try_delta" for large blobs when counting objects.

+ pack-objects.c: write_no_reuse_object(): streaming large blob to pack.

+ unpack-objects.c: unpack_non_delta_entry(): unpack large blob in stream from packfile.

* Others:

+ diff.c: diff_populate_filespec(): treat large blob file as binary.

+ streaming.c: istream_source(): as a helper of "open_istream()" to select proper streaming interface to read large blob from packfile.

---
 builtin/unpack-objects.c        | 71 ++++++++++++++++++++++++++++++++-
 t/t5328-unpack-large-objects.sh | 23 +++++++++--
 2 files changed, 90 insertions(+), 4 deletions(-)
builtin/unpack-objects.c +70 -1
index c6d6c17072..e9ec2b349d 100644
@@ -343,11 +343,80 @@ static void added_object(unsigned nr, enum object_type type,
 	}
 }
 
+struct input_zstream_data {
+	git_zstream *zstream;
+	unsigned char buf[8192];
+	int status;
+};
+
+static const void *feed_input_zstream(struct input_stream *in_stream,
+				      unsigned long *readlen)
+{
+	struct input_zstream_data *data = in_stream->data;
+	git_zstream *zstream = data->zstream;
+	void *in = fill(1);
+
+	if (in_stream->is_finished) {
+		*readlen = 0;
+		return NULL;
+	}
+
+	zstream->next_out = data->buf;
+	zstream->avail_out = sizeof(data->buf);
+	zstream->next_in = in;
+	zstream->avail_in = len;
+
+	data->status = git_inflate(zstream, 0);
+
+	in_stream->is_finished = data->status != Z_OK;
+	use(len - zstream->avail_in);
+	*readlen = sizeof(data->buf) - zstream->avail_out;
+
+	return data->buf;
+}
+
+static void write_stream_blob(unsigned nr, size_t size)
+{
+	git_zstream zstream = { 0 };
+	struct input_zstream_data data = { 0 };
+	struct input_stream in_stream = {
+		.read = feed_input_zstream,
+		.data = &data,
+	};
+
+	data.zstream = &zstream;
+	git_inflate_init(&zstream);
+
+	if (stream_loose_object(&in_stream, size, &obj_list[nr].oid))
+		die(_("failed to write object in stream"));
+
+	if (data.status != Z_STREAM_END)
+		die(_("inflate returned (%d)"), data.status);
+	git_inflate_end(&zstream);
+
+	if (strict) {
+		struct blob *blob =
+			lookup_blob(the_repository, &obj_list[nr].oid);
+		if (blob)
+			blob->object.flags |= FLAG_WRITTEN;
+		else
+			die(_("invalid blob object from stream"));
+	}
+	obj_list[nr].obj = NULL;
+}
+
 static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 				   unsigned nr)
 {
-	void *buf = get_data(size);
+	void *buf;
+
+	/* Write large blob in stream without allocating full buffer. */
+	if (!dry_run && type == OBJ_BLOB && size > big_file_threshold) {
+		write_stream_blob(nr, size);
+		return;
+	}
 
+	buf = get_data(size);
 	if (buf)
 		write_object(nr, type, buf, size);
 }
t/t5328-unpack-large-objects.sh +20 -3
index 45a3316e06..f4129979f9 100755
@@ -9,7 +9,11 @@ test_description='git unpack-objects with large objects'
 
 prepare_dest () {
 	test_when_finished "rm -rf dest.git" &&
-	git init --bare dest.git
+	git init --bare dest.git &&
+	if test -n "$1"
+	then
+		git -C dest.git config core.bigFileThreshold $1
+	fi
 }
 
 test_no_loose () {
@@ -33,16 +37,29 @@ test_expect_success 'set memory limitation to 1MB' '
 '
 
 test_expect_success 'unpack-objects failed under memory limitation' '
-	prepare_dest &&
+	prepare_dest 2m &&
 	test_must_fail git -C dest.git unpack-objects <test-$PACK.pack 2>err &&
 	grep "fatal: attempting to allocate" err
 '
 
 test_expect_success 'unpack-objects works with memory limitation in dry-run mode' '
-	prepare_dest &&
+	prepare_dest 2m &&
 	git -C dest.git unpack-objects -n <test-$PACK.pack &&
 	test_no_loose &&
 	test_dir_is_empty dest.git/objects/pack
 '
 
+test_expect_success 'unpack big object in stream' '
+	prepare_dest 1m &&
+	git -C dest.git unpack-objects <test-$PACK.pack &&
+	test_dir_is_empty dest.git/objects/pack
+'
+
+test_expect_success 'do not unpack existing large objects' '
+	prepare_dest 1m &&
+	git -C dest.git index-pack --stdin <test-$PACK.pack &&
+	git -C dest.git unpack-objects <test-$PACK.pack &&
+	test_no_loose
+'
+
 test_done
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
author
#137 lore raw

PATCH v9 5/5 object-file API: add a format_object_header() function

In reply to Han Xin

From: Ævar Arnfjörð Bjarmason <[email protected]>

Add a convenience function to wrap the xsnprintf() command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.

All except one caller of it had a valid "enum object_type" for us, it's only write_object_file_prepare() which might need to deal with "git hash-object --literally" and a potential garbage type. Let's have the primary API use an "enum object_type", and define an *_extended() function that can take an arbitrary "const char *" for the type.

See [1] for the discussion that prompted this patch, i.e. new code in object-file.c that wanted to copy/paste the xsnprintf() invocation.

1. https://lore.kernel.org/git/[email protected]/

---
 builtin/index-pack.c |  3 +--
 bulk-checkin.c       |  4 ++--
 cache.h              | 21 +++++++++++++++++++++
 http-push.c          |  2 +-
 object-file.c        | 16 ++++++++++++----
 5 files changed, 37 insertions(+), 9 deletions(-)
builtin/index-pack.c +1 -2
index c23d01de7d..8a6ce77940 100644
@@ -449,8 +449,7 @@ static void *unpack_entry_data(off_t offset, unsigned long size,
 	int hdrlen;
 
 	if (!is_delta_type(type)) {
-		hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX,
-				   type_name(type),(uintmax_t)size) + 1;
+		hdrlen = format_object_header(hdr, sizeof(hdr), type, size);
 		the_hash_algo->init_fn(&c);
 		the_hash_algo->update_fn(&c, hdr, hdrlen);
 	} else
bulk-checkin.c +2 -2
index 8785b2ac80..9e685f0f1a 100644
@@ -220,8 +220,8 @@ static int deflate_to_pack(struct bulk_checkin_state *state,
 	if (seekback == (off_t) -1)
 		return error("cannot find the current offset");
 
-	header_len = xsnprintf((char *)obuf, sizeof(obuf), "%s %" PRIuMAX,
-			       type_name(type), (uintmax_t)size) + 1;
+	header_len = format_object_header((char *)obuf, sizeof(obuf),
+					 type, size);
 	the_hash_algo->init_fn(&ctx);
 	the_hash_algo->update_fn(&ctx, obuf, header_len);
 
cache.h +21 -0
index cfba463aa9..64071a8d80 100644
@@ -1310,6 +1310,27 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
 						    unsigned long bufsiz,
 						    struct strbuf *hdrbuf);
 
+/**
+ * format_object_header() is a thin wrapper around s xsnprintf() that
+ * writes the initial "<type> <obj-len>" part of the loose object
+ * header. It returns the size that snprintf() returns + 1.
+ *
+ * The format_object_header_extended() function allows for writing a
+ * type_name that's not one of the "enum object_type" types. This is
+ * used for "git hash-object --literally". Pass in a OBJ_NONE as the
+ * type, and a non-NULL "type_str" to do that.
+ *
+ * format_object_header() is a convenience wrapper for
+ * format_object_header_extended().
+ */
+int format_object_header_extended(char *str, size_t size, enum object_type type,
+				 const char *type_str, size_t objsize);
+static inline int format_object_header(char *str, size_t size,
+				      enum object_type type, size_t objsize)
+{
+	return format_object_header_extended(str, size, type, NULL, objsize);
+}
+
 /**
  * parse_loose_header() parses the starting "<type> <len>\0" of an
  * object. If it doesn't follow that format -1 is returned. To check
http-push.c +1 -1
index 3309aaf004..f0c044dcf7 100644
@@ -363,7 +363,7 @@ static void start_put(struct transfer_request *request)
 	git_zstream stream;
 
 	unpacked = read_object_file(&request->obj->oid, &type, &len);
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
+	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
 
 	/* Set it up */
 	git_deflate_init(&stream, zlib_compression_level);
object-file.c +12 -4
index a738f47cb2..0dce5d2fec 100644
@@ -1006,6 +1006,14 @@ void *xmmap(void *start, size_t length,
 	return ret;
 }
 
+int format_object_header_extended(char *str, size_t size, enum object_type type,
+				 const char *typestr, size_t objsize)
+{
+	const char *s = type == OBJ_NONE ? typestr : type_name(type);
+
+	return xsnprintf(str, size, "%s %"PRIuMAX, s, (uintmax_t)objsize) + 1;
+}
+
 /*
  * With an in-core object data in "map", rehash it to make sure the
  * object name actually matches "oid" to detect object corruption.
@@ -1034,7 +1042,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,
 		return -1;
 
 	/* Generate the header */
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(obj_type), (uintmax_t)size) + 1;
+	hdrlen = format_object_header(hdr, sizeof(hdr), obj_type, size);
 
 	/* Sha1.. */
 	r->hash_algo->init_fn(&c);
@@ -1734,7 +1742,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,
 	git_hash_ctx c;
 
 	/* Generate the header */
-	*hdrlen = xsnprintf(hdr, *hdrlen, "%s %"PRIuMAX , type, (uintmax_t)len)+1;
+	*hdrlen = format_object_header_extended(hdr, *hdrlen, OBJ_NONE, type, len);
 
 	/* Sha1.. */
 	algo->init_fn(&c);
@@ -2011,7 +2019,7 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
 
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", get_object_directory());
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX, type_name(OBJ_BLOB), len) + 1;
+	hdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, len);
 
 	/* Common steps for write_loose_object and stream_loose_object to
 	 * start writing loose oject:
@@ -2152,7 +2160,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 	buf = read_object(the_repository, oid, &type, &len);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
+	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
 	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
 	free(buf);
 
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
author
#138 lore raw

PATCH v8 1/6 unpack-objects: low memory footprint for get_data() in dry_run mode

In reply to Han Xin

From: Han Xin <[email protected]>

As the name implies, "get_data(size)" will allocate and return a given size of memory. Allocating memory for a large blob object may cause the system to run out of memory. Before preparing to replace calling of "get_data()" to unpack large blob objects in latter commits, refactor "get_data()" to reduce memory footprint for dry_run mode.

Because in dry_run mode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can allocate a smaller buffer and reuse it as zstream output. Therefore, in dry_run mode, "get_data()" will release the allocated buffer and return NULL instead of returning garbage data.

---
 builtin/unpack-objects.c        | 39 ++++++++++++++++++-------
 t/t5329-unpack-large-objects.sh | 52 +++++++++++++++++++++++++++++++++
 2 files changed, 80 insertions(+), 11 deletions(-)
 create mode 100755 t/t5329-unpack-large-objects.sh
builtin/unpack-objects.c +28 -11
index 4a9466295b..c6d6c17072 100644
@@ -96,15 +96,31 @@ static void use(int bytes)
 	display_throughput(progress, consumed_bytes);
 }
 
+/*
+ * Decompress zstream from stdin and return specific size of data.
+ * The caller is responsible to free the returned buffer.
+ *
+ * But for dry_run mode, "get_data()" is only used to check the
+ * integrity of data, and the returned buffer is not used at all.
+ * Therefore, in dry_run mode, "get_data()" will release the small
+ * allocated buffer which is reused to hold temporary zstream output
+ * and return NULL instead of returning garbage data.
+ */
 static void *get_data(unsigned long size)
 {
 	git_zstream stream;
-	void *buf = xmallocz(size);
+	unsigned long bufsize;
+	void *buf;
 
 	memset(&stream, 0, sizeof(stream));
+	if (dry_run && size > 8192)
+		bufsize = 8192;
+	else
+		bufsize = size;
+	buf = xmallocz(bufsize);
 
 	stream.next_out = buf;
-	stream.avail_out = size;
+	stream.avail_out = bufsize;
 	stream.next_in = fill(1);
 	stream.avail_in = len;
 	git_inflate_init(&stream);
@@ -124,8 +140,15 @@ static void *get_data(unsigned long size)
 		}
 		stream.next_in = fill(1);
 		stream.avail_in = len;
+		if (dry_run) {
+			/* reuse the buffer in dry_run mode */
+			stream.next_out = buf;
+			stream.avail_out = bufsize;
+		}
 	}
 	git_inflate_end(&stream);
+	if (dry_run)
+		FREE_AND_NULL(buf);
 	return buf;
 }
 
@@ -325,10 +348,8 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 {
 	void *buf = get_data(size);
 
-	if (!dry_run && buf)
+	if (buf)
 		write_object(nr, type, buf, size);
-	else
-		free(buf);
 }
 
 static int resolve_against_held(unsigned nr, const struct object_id *base,
@@ -358,10 +379,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 		oidread(&base_oid, fill(the_hash_algo->rawsz));
 		use(the_hash_algo->rawsz);
 		delta_data = get_data(delta_size);
-		if (dry_run || !delta_data) {
-			free(delta_data);
+		if (!delta_data)
 			return;
-		}
 		if (has_object_file(&base_oid))
 			; /* Ok we have this one */
 		else if (resolve_against_held(nr, &base_oid,
@@ -397,10 +416,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 			die("offset value out of bound for delta base object");
 
 		delta_data = get_data(delta_size);
-		if (dry_run || !delta_data) {
-			free(delta_data);
+		if (!delta_data)
 			return;
-		}
 		lo = 0;
 		hi = nr;
 		while (lo < hi) {
t/t5329-unpack-large-objects.sh new file +52 -0

52 lines not on this page · raw message

Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
#139 lore raw

Re: [PATCH v8 1/6] unpack-objects: low memory footprint for get_data() in dry_run mode

Am 08.01.22 um 09:54 schrieb Han Xin:

Quoted 102 lines

From: Han Xin <[email protected]>

As the name implies, "get_data(size)" will allocate and return a given size of memory. Allocating memory for a large blob object may cause the system to run out of memory. Before preparing to replace calling of "get_data()" to unpack large blob objects in latter commits, refactor "get_data()" to reduce memory footprint for dry_run mode.

Because in dry_run mode, "get_data()" is only used to check the integrity of data, and the returned buffer is not used at all, we can allocate a smaller buffer and reuse it as zstream output. Therefore, in dry_run mode, "get_data()" will release the allocated buffer and return NULL instead of returning garbage data.

---
 builtin/unpack-objects.c        | 39 ++++++++++++++++++-------
 t/t5329-unpack-large-objects.sh | 52 +++++++++++++++++++++++++++++++++
 2 files changed, 80 insertions(+), 11 deletions(-)
 create mode 100755 t/t5329-unpack-large-objects.sh
builtin/unpack-objects.c +27 -8
index 4a9466295b..c6d6c17072 100644
@@ -96,15 +96,31 @@ static void use(int bytes)
 	display_throughput(progress, consumed_bytes);
 }
 
+/*
+ * Decompress zstream from stdin and return specific size of data.
+ * The caller is responsible to free the returned buffer.
+ *
+ * But for dry_run mode, "get_data()" is only used to check the
+ * integrity of data, and the returned buffer is not used at all.
+ * Therefore, in dry_run mode, "get_data()" will release the small
+ * allocated buffer which is reused to hold temporary zstream output
+ * and return NULL instead of returning garbage data.
+ */
 static void *get_data(unsigned long size)
 {
 	git_zstream stream;
-	void *buf = xmallocz(size);
+	unsigned long bufsize;
+	void *buf;
 
 	memset(&stream, 0, sizeof(stream));
+	if (dry_run && size > 8192)
+		bufsize = 8192;
+	else
+		bufsize = size;
+	buf = xmallocz(bufsize);
 
 	stream.next_out = buf;
-	stream.avail_out = size;
+	stream.avail_out = bufsize;
 	stream.next_in = fill(1);
 	stream.avail_in = len;
 	git_inflate_init(&stream);
@@ -124,8 +140,15 @@ static void *get_data(unsigned long size)
 		}
 		stream.next_in = fill(1);
 		stream.avail_in = len;
+		if (dry_run) {
+			/* reuse the buffer in dry_run mode */
+			stream.next_out = buf;
+			stream.avail_out = bufsize;
+		}
 	}
 	git_inflate_end(&stream);
+	if (dry_run)
+		FREE_AND_NULL(buf);
 	return buf;
 }
 
@@ -325,10 +348,8 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 {
 	void *buf = get_data(size);
 
-	if (!dry_run && buf)
+	if (buf)
 		write_object(nr, type, buf, size);
-	else
-		free(buf);
 }
 
 static int resolve_against_held(unsigned nr, const struct object_id *base,
@@ -358,10 +379,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 		oidread(&base_oid, fill(the_hash_algo->rawsz));
 		use(the_hash_algo->rawsz);
 		delta_data = get_data(delta_size);
-		if (dry_run || !delta_data) {
-			free(delta_data);
+		if (!delta_data)
 			return;
-		}
 		if (has_object_file(&base_oid))
 			; /* Ok we have this one */
 		else if (resolve_against_held(nr, &base_oid,
@@ -397,10 +416,8 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 			die("offset value out of bound for delta base object");

builtin/unpack-objects.c static void unpack_delta_entry(enum object_type type, unsigned long delta_size, · line 423

Quoted 10 lines
builtin/unpack-objects.c +1 -3
 
 		delta_data = get_data(delta_size);
-		if (dry_run || !delta_data) {
-			free(delta_data);
+		if (!delta_data)
 			return;
-		}
 		lo = 0;
 		hi = nr;
 		while (lo < hi) {

Nice!

Quoted 19 lines
t/t5329-unpack-large-objects.sh new file +13 -0
new file mode 100755
index 0000000000..39c7a62d94
@@ -0,0 +1,52 @@
+#!/bin/sh
+#
+# Copyright (c) 2021 Han Xin
+#
+
+test_description='git unpack-objects with large objects'
+
+. ./test-lib.sh
+
+prepare_dest () {
+	test_when_finished "rm -rf dest.git" &&
+	git init --bare dest.git
+}

t/t5329-unpack-large-objects.sh line 23

Quoted 10 lines
t/t5329-unpack-large-objects.sh new file +10 -0
+
+assert_no_loose () {
+	glob=dest.git/objects/?? &&
+	echo "$glob" >expect &&
+	eval "echo $glob" >actual &&
+	test_cmp expect actual
+}
+
+assert_no_pack () {
+	rmdir dest.git/objects/pack

I would expect a function whose name starts with "assert" to have no side effects. It doesn't matter here, because it's called only at the very end, but that might change. You can use test_dir_is_empty instead of rmdir.

Quoted 29 lines
t/t5329-unpack-large-objects.sh new file +29 -0
+}
+
+test_expect_success "create large objects (1.5 MB) and PACK" '
+	test-tool genrandom foo 1500000 >big-blob &&
+	test_commit --append foo big-blob &&
+	test-tool genrandom bar 1500000 >big-blob &&
+	test_commit --append bar big-blob &&
+	PACK=$(echo HEAD | git pack-objects --revs test)
+'
+
+test_expect_success 'set memory limitation to 1MB' '
+	GIT_ALLOC_LIMIT=1m &&
+	export GIT_ALLOC_LIMIT
+'
+
+test_expect_success 'unpack-objects failed under memory limitation' '
+	prepare_dest &&
+	test_must_fail git -C dest.git unpack-objects <test-$PACK.pack 2>err &&
+	grep "fatal: attempting to allocate" err
+'
+
+test_expect_success 'unpack-objects works with memory limitation in dry-run mode' '
+	prepare_dest &&
+	git -C dest.git unpack-objects -n <test-$PACK.pack &&
+	assert_no_loose &&
+	assert_no_pack
+'
+
+test_done
author
#140 lore raw

Re: [PATCH v8 1/6] unpack-objects: low memory footprint for get_data() in dry_run mode

Quoted 4 lines · René Scharfe

Am 08.01.22 um 09:54 schrieb Han Xin:

Quoted 2 lines

From: Han Xin <[email protected]>

Quoted 15 lines

t/t5329-unpack-large-objects.sh line 23

Quoted 9 lines
t/t5329-unpack-large-objects.sh new file +9 -0
+assert_no_loose () {
+     glob=dest.git/objects/?? &&
+     echo "$glob" >expect &&
+     eval "echo $glob" >actual &&
+     test_cmp expect actual
+}
+
+assert_no_pack () {
+     rmdir dest.git/objects/pack

I would expect a function whose name starts with "assert" to have no side effects. It doesn't matter here, because it's called only at the very end, but that might change. You can use test_dir_is_empty instead of rmdir.

*nod*
I think it would be better to rename "assert_no_loose()" to "test_no_loose()". I will remove "assert_no_pack()" and use "test_dir_is_empty()" instead.

Thanks
-Han Xin

author
#141 lore raw

PATCH v8 2/6 object-file.c: refactor write_loose_object() to several steps

In reply to Han Xin

From: Han Xin <[email protected]>

When writing a large blob using "write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in latter commit to resolve this issue.

Before introducing a stream vesion function for writing loose object, do some refactoring on "write_loose_object()" to reuse code for both versions.

Rewrite "write_loose_object()" as follows:

1. Figure out a path for the (temp) object file. This step is only used in "write_loose_object()".

2. Move common steps for starting to write loose objects into a new function "start_loose_object_common()".

3. Compress data.

4. Move common steps for ending zlib stream into a new funciton

    "end_loose_object_common()".

5. Close fd and finalize the object file.

---
 object-file.c | 149 +++++++++++++++++++++++++++++++++++---------------
 1 file changed, 105 insertions(+), 44 deletions(-)
object-file.c +105 -44
index eb1426f98c..5d163081b1 100644
@@ -1743,6 +1743,25 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,
 	algo->final_oid_fn(oid, &c);
 }
 
+/*
+ * Move the just written object with proper mtime into its final resting place.
+ */
+static int finalize_object_file_with_mtime(const char *tmpfile,
+					   const char *filename,
+					   time_t mtime,
+					   unsigned flags)
+{
+	struct utimbuf utb;
+
+	if (mtime) {
+		utb.actime = mtime;
+		utb.modtime = mtime;
+		if (utime(tmpfile, &utb) < 0 && !(flags & HASH_SILENT))
+			warning_errno(_("failed utime() on %s"), tmpfile);
+	}
+	return finalize_object_file(tmpfile, filename);
+}
+
 /*
  * Move the just written object into its final resting place.
  */
@@ -1828,7 +1847,8 @@ static inline int directory_size(const char *filename)
  * We want to avoid cross-directory filename renames, because those
  * can have problems on various filesystems (FAT, NFS, Coda).
  */
-static int create_tmpfile(struct strbuf *tmp, const char *filename)
+static int create_tmpfile(struct strbuf *tmp, const char *filename,
+			  unsigned flags)
 {
 	int fd, dirlen = directory_size(filename);
 
@@ -1836,7 +1856,9 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	strbuf_add(tmp, filename, dirlen);
 	strbuf_addstr(tmp, "tmp_obj_XXXXXX");
 	fd = git_mkstemp_mode(tmp->buf, 0444);
-	if (fd < 0 && dirlen && errno == ENOENT) {
+	do {
+		if (fd >= 0 || !dirlen || errno != ENOENT)
+			break;
 		/*
 		 * Make sure the directory exists; note that the contents
 		 * of the buffer are undefined after mkstemp returns an
@@ -1846,17 +1868,72 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 		strbuf_reset(tmp);
 		strbuf_add(tmp, filename, dirlen - 1);
 		if (mkdir(tmp->buf, 0777) && errno != EEXIST)
-			return -1;
+			break;
 		if (adjust_shared_perm(tmp->buf))
-			return -1;
+			break;
 
 		/* Try again */
 		strbuf_addstr(tmp, "/tmp_obj_XXXXXX");
 		fd = git_mkstemp_mode(tmp->buf, 0444);
+	} while (0);
+
+	if (fd < 0 && !(flags & HASH_SILENT)) {
+		if (errno == EACCES)
+			return error(_("insufficient permission for adding an "
+				       "object to repository database %s"),
+				     get_object_directory());
+		else
+			return error_errno(_("unable to create temporary file"));
 	}
+
 	return fd;
 }
 
+static int start_loose_object_common(struct strbuf *tmp_file,
+				     const char *filename, unsigned flags,
+				     git_zstream *stream,
+				     unsigned char *buf, size_t buflen,
+				     git_hash_ctx *c,
+				     enum object_type type, size_t len,
+				     char *hdr, int hdrlen)
+{
+	int fd;
+
+	fd = create_tmpfile(tmp_file, filename, flags);
+	if (fd < 0)
+		return -1;
+
+	/*  Setup zlib stream for compression */
+	git_deflate_init(stream, zlib_compression_level);
+	stream->next_out = buf;
+	stream->avail_out = buflen;
+	the_hash_algo->init_fn(c);
+
+	/*  Start to feed header to zlib stream */
+	stream->next_in = (unsigned char *)hdr;
+	stream->avail_in = hdrlen;
+	while (git_deflate(stream, 0) == Z_OK)
+		; /* nothing */
+	the_hash_algo->update_fn(c, hdr, hdrlen);
+
+	return fd;
+}
+
+static void end_loose_object_common(int ret, git_hash_ctx *c,
+				    git_zstream *stream,
+				    struct object_id *parano_oid,
+				    const struct object_id *expected_oid,
+				    const char *die_msg1_fmt,
+				    const char *die_msg2_fmt)
+{
+	if (ret != Z_STREAM_END)
+		die(_(die_msg1_fmt), ret, expected_oid);
+	ret = git_deflate_end_gently(stream);
+	if (ret != Z_OK)
+		die(_(die_msg2_fmt), ret, expected_oid);
+	the_hash_algo->final_oid_fn(parano_oid, c);
+}
+
 static int write_loose_object(const struct object_id *oid, char *hdr,
 			      int hdrlen, const void *buf, unsigned long len,
 			      time_t mtime, unsigned flags)
@@ -1871,28 +1948,18 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 
 	loose_object_path(the_repository, &filename, oid);
 
-	fd = create_tmpfile(&tmp_file, filename.buf);
-	if (fd < 0) {
-		if (flags & HASH_SILENT)
-			return -1;
-		else if (errno == EACCES)
-			return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
-		else
-			return error_errno(_("unable to create temporary file"));
-	}
-
-	/* Set it up */
-	git_deflate_init(&stream, zlib_compression_level);
-	stream.next_out = compressed;
-	stream.avail_out = sizeof(compressed);
-	the_hash_algo->init_fn(&c);
-
-	/* First header.. */
-	stream.next_in = (unsigned char *)hdr;
-	stream.avail_in = hdrlen;
-	while (git_deflate(&stream, 0) == Z_OK)
-		; /* nothing */
-	the_hash_algo->update_fn(&c, hdr, hdrlen);
+	/* Common steps for write_loose_object and stream_loose_object to
+	 * start writing loose oject:
+	 *
+	 *  - Create tmpfile for the loose object.
+	 *  - Setup zlib stream for compression.
+	 *  - Start to feed header to zlib stream.
+	 */
+	fd = start_loose_object_common(&tmp_file, filename.buf, flags,
+				       &stream, compressed, sizeof(compressed),
+				       &c, OBJ_NONE, 0, hdr, hdrlen);
+	if (fd < 0)
+		return -1;
 
 	/* Then the data itself.. */
 	stream.next_in = (void *)buf;
@@ -1907,30 +1974,24 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 		stream.avail_out = sizeof(compressed);
 	} while (ret == Z_OK);
 
-	if (ret != Z_STREAM_END)
-		die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
-		    ret);
-	ret = git_deflate_end_gently(&stream);
-	if (ret != Z_OK)
-		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
-		    ret);
-	the_hash_algo->final_oid_fn(&parano_oid, &c);
+	/* Common steps for write_loose_object and stream_loose_object to
+	 * end writing loose oject:
+	 *
+	 *  - End the compression of zlib stream.
+	 *  - Get the calculated oid to "parano_oid".
+	 */
+	end_loose_object_common(ret, &c, &stream, &parano_oid, oid,
+				N_("unable to deflate new object %s (%d)"),
+				N_("deflateEnd on object %s failed (%d)"));
+
 	if (!oideq(oid, &parano_oid))
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));
 
 	close_loose_object(fd);
 
-	if (mtime) {
-		struct utimbuf utb;
-		utb.actime = mtime;
-		utb.modtime = mtime;
-		if (utime(tmp_file.buf, &utb) < 0 &&
-		    !(flags & HASH_SILENT))
-			warning_errno(_("failed utime() on %s"), tmp_file.buf);
-	}
-
-	return finalize_object_file(tmp_file.buf, filename.buf);
+	return finalize_object_file_with_mtime(tmp_file.buf, filename.buf,
+					       mtime, flags);
 }
 
 static int freshen_loose_object(const struct object_id *oid)
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
#142 lore raw

Re: [PATCH v8 2/6] object-file.c: refactor write_loose_object() to several steps

Am 08.01.22 um 09:54 schrieb Han Xin:

Quoted 39 lines

From: Han Xin <[email protected]>

When writing a large blob using "write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in latter commit to resolve this issue.

Before introducing a stream vesion function for writing loose object, do some refactoring on "write_loose_object()" to reuse code for both versions.

Rewrite "write_loose_object()" as follows:

1. Figure out a path for the (temp) object file. This step is only used in "write_loose_object()".

2. Move common steps for starting to write loose objects into a new function "start_loose_object_common()".

3. Compress data.

4. Move common steps for ending zlib stream into a new funciton

    "end_loose_object_common()".

5. Close fd and finalize the object file.

---
 object-file.c | 149 +++++++++++++++++++++++++++++++++++---------------
 1 file changed, 105 insertions(+), 44 deletions(-)
object-file.c +0 -0
index eb1426f98c..5d163081b1 100644

object-file.c static void write_object_file_prepare(const struct git_hash_algo *algo, · line 1752

Quoted 10 lines
object-file.c +7 -0
@@ -1743,6 +1743,25 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,
 	algo->final_oid_fn(oid, &c);
 }
 
+/*
+ * Move the just written object with proper mtime into its final resting place.
+ */
+static int finalize_object_file_with_mtime(const char *tmpfile,
+					   const char *filename,
+					   time_t mtime,
+					   unsigned flags)

This function is called only once after your series. Should it be used by stream_loose_object()? Probably not -- the latter doesn't have a way to force a certain modification time and its caller doesn't need one. So creating finalize_object_file_with_mtime() seems unnecessary for this series.

Quoted 16 lines
object-file.c +12 -0
+{
+	struct utimbuf utb;
+
+	if (mtime) {
+		utb.actime = mtime;
+		utb.modtime = mtime;
+		if (utime(tmpfile, &utb) < 0 && !(flags & HASH_SILENT))
+			warning_errno(_("failed utime() on %s"), tmpfile);
+	}
+	return finalize_object_file(tmpfile, filename);
+}
+
 /*
  * Move the just written object into its final resting place.
  */

object-file.c static inline int directory_size(const char *filename) · line 1851

Quoted 6 lines
object-file.c +2 -1
@@ -1828,7 +1847,8 @@ static inline int directory_size(const char *filename)
  * We want to avoid cross-directory filename renames, because those
  * can have problems on various filesystems (FAT, NFS, Coda).
  */
-static int create_tmpfile(struct strbuf *tmp, const char *filename)
+static int create_tmpfile(struct strbuf *tmp, const char *filename,
+			  unsigned flags)

create_tmpfile() is not mentioned in the commit message, yet it's changed here. Hrm.

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1861

Quoted 11 lines
object-file.c +3 -1
 {
 	int fd, dirlen = directory_size(filename);
 
@@ -1836,7 +1856,9 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	strbuf_add(tmp, filename, dirlen);
 	strbuf_addstr(tmp, "tmp_obj_XXXXXX");
 	fd = git_mkstemp_mode(tmp->buf, 0444);
-	if (fd < 0 && dirlen && errno == ENOENT) {
+	do {
+		if (fd >= 0 || !dirlen || errno != ENOENT)
+			break;

Why turn this branch into a loop? Is this done to mkdir multiple components, e.g. with filename being "a/b/c/file" to create "a", "a/b", and "a/b/c"? It's only used for loose objects, so a fan-out directory (e.g. ".git/objects/ff") can certainly be missing, but can their parent be missing as well sometimes? If that's the point then such a fix would be worth its own patch. (Which probably would benefit from using safe_create_leading_directories()).

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1873

Quoted 12 lines
object-file.c +2 -2
 		/*
 		 * Make sure the directory exists; note that the contents
 		 * of the buffer are undefined after mkstemp returns an
@@ -1846,17 +1868,72 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 		strbuf_reset(tmp);
 		strbuf_add(tmp, filename, dirlen - 1);
 		if (mkdir(tmp->buf, 0777) && errno != EEXIST)
-			return -1;
+			break;
 		if (adjust_shared_perm(tmp->buf))
-			return -1;
+			break;

Or is it just to replace these returns with a jump to the new error reporting section?

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1877

Quoted 4 lines
object-file.c +0 -0
 		/* Try again */
 		strbuf_addstr(tmp, "/tmp_obj_XXXXXX");
 		fd = git_mkstemp_mode(tmp->buf, 0444);

In that case a break would be missing here.

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1887

Quoted 10 lines
object-file.c +9 -0
+	} while (0);
+
+	if (fd < 0 && !(flags & HASH_SILENT)) {
+		if (errno == EACCES)
+			return error(_("insufficient permission for adding an "
+				       "object to repository database %s"),
+				     get_object_directory());
+		else
+			return error_errno(_("unable to create temporary file"));
 	}

Why move this error reporting code into create_tmpfile()? This function has a single caller both before and after your series, so the code could just as well stay at its call-site, avoiding the need to add the flags parameter.

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1897

Quoted 10 lines
object-file.c +7 -0
+
 	return fd;
 }
 
+static int start_loose_object_common(struct strbuf *tmp_file,
+				     const char *filename, unsigned flags,
+				     git_zstream *stream,
+				     unsigned char *buf, size_t buflen,
+				     git_hash_ctx *c,
+				     enum object_type type, size_t len,

The parameters type and len are not used by this function and thus can be dropped.

Quoted 20 lines
object-file.c +20 -0
+				     char *hdr, int hdrlen)
+{
+	int fd;
+
+	fd = create_tmpfile(tmp_file, filename, flags);
+	if (fd < 0)
+		return -1;
+
+	/*  Setup zlib stream for compression */
+	git_deflate_init(stream, zlib_compression_level);
+	stream->next_out = buf;
+	stream->avail_out = buflen;
+	the_hash_algo->init_fn(c);
+
+	/*  Start to feed header to zlib stream */
+	stream->next_in = (unsigned char *)hdr;
+	stream->avail_in = hdrlen;
+	while (git_deflate(stream, 0) == Z_OK)
+		; /* nothing */
+	the_hash_algo->update_fn(c, hdr, hdrlen);

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1927

Quoted 10 lines
object-file.c +10 -0
+
+	return fd;
+}
+
+static void end_loose_object_common(int ret, git_hash_ctx *c,
+				    git_zstream *stream,
+				    struct object_id *parano_oid,
+				    const struct object_id *expected_oid,
+				    const char *die_msg1_fmt,
+				    const char *die_msg2_fmt)

Hmm, the signature needs as many lines as the function body.

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1933

Quoted 6 lines
object-file.c +6 -0
+{
+	if (ret != Z_STREAM_END)
+		die(_(die_msg1_fmt), ret, expected_oid);
+	ret = git_deflate_end_gently(stream);
+	if (ret != Z_OK)
+		die(_(die_msg2_fmt), ret, expected_oid);

These format strings cannot be checked by the compiler.

Considering those two together I think I'd either unify the error messages and move their strings here (losing the ability for users to see if streaming was used) or not extract the function and duplicate its few shared lines. Just a feeling, though.

Quoted 90 lines
object-file.c +27 -40
+	the_hash_algo->final_oid_fn(parano_oid, c);
+}
+
 static int write_loose_object(const struct object_id *oid, char *hdr,
 			      int hdrlen, const void *buf, unsigned long len,
 			      time_t mtime, unsigned flags)
@@ -1871,28 +1948,18 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 
 	loose_object_path(the_repository, &filename, oid);
 
-	fd = create_tmpfile(&tmp_file, filename.buf);
-	if (fd < 0) {
-		if (flags & HASH_SILENT)
-			return -1;
-		else if (errno == EACCES)
-			return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
-		else
-			return error_errno(_("unable to create temporary file"));
-	}
-
-	/* Set it up */
-	git_deflate_init(&stream, zlib_compression_level);
-	stream.next_out = compressed;
-	stream.avail_out = sizeof(compressed);
-	the_hash_algo->init_fn(&c);
-
-	/* First header.. */
-	stream.next_in = (unsigned char *)hdr;
-	stream.avail_in = hdrlen;
-	while (git_deflate(&stream, 0) == Z_OK)
-		; /* nothing */
-	the_hash_algo->update_fn(&c, hdr, hdrlen);
+	/* Common steps for write_loose_object and stream_loose_object to
+	 * start writing loose oject:
+	 *
+	 *  - Create tmpfile for the loose object.
+	 *  - Setup zlib stream for compression.
+	 *  - Start to feed header to zlib stream.
+	 */
+	fd = start_loose_object_common(&tmp_file, filename.buf, flags,
+				       &stream, compressed, sizeof(compressed),
+				       &c, OBJ_NONE, 0, hdr, hdrlen);
+	if (fd < 0)
+		return -1;
 
 	/* Then the data itself.. */
 	stream.next_in = (void *)buf;
@@ -1907,30 +1974,24 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 		stream.avail_out = sizeof(compressed);
 	} while (ret == Z_OK);
 
-	if (ret != Z_STREAM_END)
-		die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
-		    ret);
-	ret = git_deflate_end_gently(&stream);
-	if (ret != Z_OK)
-		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
-		    ret);
-	the_hash_algo->final_oid_fn(&parano_oid, &c);
+	/* Common steps for write_loose_object and stream_loose_object to
+	 * end writing loose oject:
+	 *
+	 *  - End the compression of zlib stream.
+	 *  - Get the calculated oid to "parano_oid".
+	 */
+	end_loose_object_common(ret, &c, &stream, &parano_oid, oid,
+				N_("unable to deflate new object %s (%d)"),
+				N_("deflateEnd on object %s failed (%d)"));
+
 	if (!oideq(oid, &parano_oid))
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));
 
 	close_loose_object(fd);
 
-	if (mtime) {
-		struct utimbuf utb;
-		utb.actime = mtime;
-		utb.modtime = mtime;
-		if (utime(tmp_file.buf, &utb) < 0 &&
-		    !(flags & HASH_SILENT))
-			warning_errno(_("failed utime() on %s"), tmp_file.buf);
-	}
-
-	return finalize_object_file(tmp_file.buf, filename.buf);
+	return finalize_object_file_with_mtime(tmp_file.buf, filename.buf,
+					       mtime, flags);
 }
 
 static int freshen_loose_object(const struct object_id *oid)
author
#143 lore raw

Re: [PATCH v8 2/6] object-file.c: refactor write_loose_object() to several steps

Quoted 44 lines · René Scharfe

Am 08.01.22 um 09:54 schrieb Han Xin:

Quoted 42 lines

From: Han Xin <[email protected]>

When writing a large blob using "write_loose_object()", we have to pass a buffer with the whole content of the blob, and this behavior will consume lots of memory and may cause OOM. We will introduce a stream version function ("stream_loose_object()") in latter commit to resolve this issue.

Before introducing a stream vesion function for writing loose object, do some refactoring on "write_loose_object()" to reuse code for both versions.

Rewrite "write_loose_object()" as follows:

1. Figure out a path for the (temp) object file. This step is only used in "write_loose_object()".

2. Move common steps for starting to write loose objects into a new function "start_loose_object_common()".

3. Compress data.

4. Move common steps for ending zlib stream into a new funciton

    "end_loose_object_common()".

5. Close fd and finalize the object file.

---
 object-file.c | 149 +++++++++++++++++++++++++++++++++++---------------
 1 file changed, 105 insertions(+), 44 deletions(-)
object-file.c +0 -0
index eb1426f98c..5d163081b1 100644
@@ -1743,6 +1743,25 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,
      algo->final_oid_fn(oid, &c);
 }
 
Quoted 14 lines

object-file.c static void write_object_file_prepare(const struct git_hash_algo *algo, · line 1752

Quoted 7 lines
object-file.c +7 -0
+/*
+ * Move the just written object with proper mtime into its final resting place.
+ */
+static int finalize_object_file_with_mtime(const char *tmpfile,
+                                        const char *filename,
+                                        time_t mtime,
+                                        unsigned flags)

This function is called only once after your series. Should it be used by stream_loose_object()? Probably not -- the latter doesn't have a way to force a certain modification time and its caller doesn't need one. So creating finalize_object_file_with_mtime() seems unnecessary for this series.

After accepting the suggestion by Ævar Arnfjörð Bjarmason[1] to remove finalize_object_file_with_mtime() from stream_loose_object() , it seems to be an overkill for write_loose_object() now. I'll put it back into write_loose_object() .

1. https://lore.kernel.org/git/[email protected]/

Thanks
-Han Xin

Quoted 80 lines
Quoted 16 lines
object-file.c +12 -0
+{
+     struct utimbuf utb;
+
+     if (mtime) {
+             utb.actime = mtime;
+             utb.modtime = mtime;
+             if (utime(tmpfile, &utb) < 0 && !(flags & HASH_SILENT))
+                     warning_errno(_("failed utime() on %s"), tmpfile);
+     }
+     return finalize_object_file(tmpfile, filename);
+}
+
 /*
  * Move the just written object into its final resting place.
  */

object-file.c static inline int directory_size(const char *filename) · line 1851

Quoted 6 lines
object-file.c +2 -1
@@ -1828,7 +1847,8 @@ static inline int directory_size(const char *filename)
  * We want to avoid cross-directory filename renames, because those
  * can have problems on various filesystems (FAT, NFS, Coda).
  */
-static int create_tmpfile(struct strbuf *tmp, const char *filename)
+static int create_tmpfile(struct strbuf *tmp, const char *filename,
+                       unsigned flags)

create_tmpfile() is not mentioned in the commit message, yet it's changed here. Hrm.

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1861

Quoted 11 lines
object-file.c +3 -1
 {
      int fd, dirlen = directory_size(filename);
 
@@ -1836,7 +1856,9 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
      strbuf_add(tmp, filename, dirlen);
      strbuf_addstr(tmp, "tmp_obj_XXXXXX");
      fd = git_mkstemp_mode(tmp->buf, 0444);
-     if (fd < 0 && dirlen && errno == ENOENT) {
+     do {
+             if (fd >= 0 || !dirlen || errno != ENOENT)
+                     break;

Why turn this branch into a loop? Is this done to mkdir multiple components, e.g. with filename being "a/b/c/file" to create "a", "a/b", and "a/b/c"? It's only used for loose objects, so a fan-out directory (e.g. ".git/objects/ff") can certainly be missing, but can their parent be missing as well sometimes? If that's the point then such a fix would be worth its own patch. (Which probably would benefit from using safe_create_leading_directories()).

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1873

Quoted 12 lines
object-file.c +2 -2
              /*
               * Make sure the directory exists; note that the contents
               * of the buffer are undefined after mkstemp returns an
@@ -1846,17 +1868,72 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
              strbuf_reset(tmp);
              strbuf_add(tmp, filename, dirlen - 1);
              if (mkdir(tmp->buf, 0777) && errno != EEXIST)
-                     return -1;
+                     break;
              if (adjust_shared_perm(tmp->buf))
-                     return -1;
+                     break;

Or is it just to replace these returns with a jump to the new error reporting section?

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1877

Quoted 4 lines
object-file.c +0 -0
              /* Try again */
              strbuf_addstr(tmp, "/tmp_obj_XXXXXX");
              fd = git_mkstemp_mode(tmp->buf, 0444);

In that case a break would be missing here.

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1887

Quoted 10 lines
object-file.c +9 -0
+     } while (0);
+
+     if (fd < 0 && !(flags & HASH_SILENT)) {
+             if (errno == EACCES)
+                     return error(_("insufficient permission for adding an "
+                                    "object to repository database %s"),
+                                  get_object_directory());
+             else
+                     return error_errno(_("unable to create temporary file"));
      }
Quoted 5 lines

Why move this error reporting code into create_tmpfile()? This function has a single caller both before and after your series, so the code could just as well stay at its call-site, avoiding the need to add the flags parameter.

Here is a legacy from v7, now there is no step called "Figuring out a path for the (temp) object file.", and it's only used in start_loose_object_common(). I will bring it back to what it was.

Thanks
-Han Xin

Quoted 4 lines
Quoted 4 lines
object-file.c +1 -0
+
      return fd;
 }
 
Quoted 10 lines

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1897

Quoted 6 lines
object-file.c +6 -0
+static int start_loose_object_common(struct strbuf *tmp_file,
+                                  const char *filename, unsigned flags,
+                                  git_zstream *stream,
+                                  unsigned char *buf, size_t buflen,
+                                  git_hash_ctx *c,
+                                  enum object_type type, size_t len,

The parameters type and len are not used by this function and thus can be dropped.

*nod*

Quoted 137 lines
Quoted 20 lines
object-file.c +20 -0
+                                  char *hdr, int hdrlen)
+{
+     int fd;
+
+     fd = create_tmpfile(tmp_file, filename, flags);
+     if (fd < 0)
+             return -1;
+
+     /*  Setup zlib stream for compression */
+     git_deflate_init(stream, zlib_compression_level);
+     stream->next_out = buf;
+     stream->avail_out = buflen;
+     the_hash_algo->init_fn(c);
+
+     /*  Start to feed header to zlib stream */
+     stream->next_in = (unsigned char *)hdr;
+     stream->avail_in = hdrlen;
+     while (git_deflate(stream, 0) == Z_OK)
+             ; /* nothing */
+     the_hash_algo->update_fn(c, hdr, hdrlen);

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1927

Quoted 10 lines
object-file.c +10 -0
+
+     return fd;
+}
+
+static void end_loose_object_common(int ret, git_hash_ctx *c,
+                                 git_zstream *stream,
+                                 struct object_id *parano_oid,
+                                 const struct object_id *expected_oid,
+                                 const char *die_msg1_fmt,
+                                 const char *die_msg2_fmt)

Hmm, the signature needs as many lines as the function body.

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1933

Quoted 6 lines
object-file.c +6 -0
+{
+     if (ret != Z_STREAM_END)
+             die(_(die_msg1_fmt), ret, expected_oid);
+     ret = git_deflate_end_gently(stream);
+     if (ret != Z_OK)
+             die(_(die_msg2_fmt), ret, expected_oid);

These format strings cannot be checked by the compiler.

Considering those two together I think I'd either unify the error messages and move their strings here (losing the ability for users to see if streaming was used) or not extract the function and duplicate its few shared lines. Just a feeling, though.

Quoted 90 lines
object-file.c +27 -40
+     the_hash_algo->final_oid_fn(parano_oid, c);
+}
+
 static int write_loose_object(const struct object_id *oid, char *hdr,
                            int hdrlen, const void *buf, unsigned long len,
                            time_t mtime, unsigned flags)
@@ -1871,28 +1948,18 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 
      loose_object_path(the_repository, &filename, oid);
 
-     fd = create_tmpfile(&tmp_file, filename.buf);
-     if (fd < 0) {
-             if (flags & HASH_SILENT)
-                     return -1;
-             else if (errno == EACCES)
-                     return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
-             else
-                     return error_errno(_("unable to create temporary file"));
-     }
-
-     /* Set it up */
-     git_deflate_init(&stream, zlib_compression_level);
-     stream.next_out = compressed;
-     stream.avail_out = sizeof(compressed);
-     the_hash_algo->init_fn(&c);
-
-     /* First header.. */
-     stream.next_in = (unsigned char *)hdr;
-     stream.avail_in = hdrlen;
-     while (git_deflate(&stream, 0) == Z_OK)
-             ; /* nothing */
-     the_hash_algo->update_fn(&c, hdr, hdrlen);
+     /* Common steps for write_loose_object and stream_loose_object to
+      * start writing loose oject:
+      *
+      *  - Create tmpfile for the loose object.
+      *  - Setup zlib stream for compression.
+      *  - Start to feed header to zlib stream.
+      */
+     fd = start_loose_object_common(&tmp_file, filename.buf, flags,
+                                    &stream, compressed, sizeof(compressed),
+                                    &c, OBJ_NONE, 0, hdr, hdrlen);
+     if (fd < 0)
+             return -1;
 
      /* Then the data itself.. */
      stream.next_in = (void *)buf;
@@ -1907,30 +1974,24 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
              stream.avail_out = sizeof(compressed);
      } while (ret == Z_OK);
 
-     if (ret != Z_STREAM_END)
-             die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
-                 ret);
-     ret = git_deflate_end_gently(&stream);
-     if (ret != Z_OK)
-             die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
-                 ret);
-     the_hash_algo->final_oid_fn(&parano_oid, &c);
+     /* Common steps for write_loose_object and stream_loose_object to
+      * end writing loose oject:
+      *
+      *  - End the compression of zlib stream.
+      *  - Get the calculated oid to "parano_oid".
+      */
+     end_loose_object_common(ret, &c, &stream, &parano_oid, oid,
+                             N_("unable to deflate new object %s (%d)"),
+                             N_("deflateEnd on object %s failed (%d)"));
+
      if (!oideq(oid, &parano_oid))
              die(_("confused by unstable object source data for %s"),
                  oid_to_hex(oid));
 
      close_loose_object(fd);
 
-     if (mtime) {
-             struct utimbuf utb;
-             utb.actime = mtime;
-             utb.modtime = mtime;
-             if (utime(tmp_file.buf, &utb) < 0 &&
-                 !(flags & HASH_SILENT))
-                     warning_errno(_("failed utime() on %s"), tmp_file.buf);
-     }
-
-     return finalize_object_file(tmp_file.buf, filename.buf);
+     return finalize_object_file_with_mtime(tmp_file.buf, filename.buf,
+                                            mtime, flags);
 }
 
 static int freshen_loose_object(const struct object_id *oid)
author
#144 lore raw

PATCH v8 3/6 object-file.c: remove the slash for directory_size()

In reply to Han Xin

From: Han Xin <[email protected]>

Since "mkdir foo/" works as well as "mkdir foo", let's remove the end slash as many users of it want.

---
 object-file.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
object-file.c +4 -4
index 5d163081b1..4f0127e823 100644
@@ -1831,13 +1831,13 @@ static void close_loose_object(int fd)
 		die_errno(_("error when closing loose object file"));
 }
 
-/* Size of directory component, including the ending '/' */
+/* Size of directory component, excluding the ending '/' */
 static inline int directory_size(const char *filename)
 {
 	const char *s = strrchr(filename, '/');
 	if (!s)
 		return 0;
-	return s - filename + 1;
+	return s - filename;
 }
 
 /*
@@ -1854,7 +1854,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,
 
 	strbuf_reset(tmp);
 	strbuf_add(tmp, filename, dirlen);
-	strbuf_addstr(tmp, "tmp_obj_XXXXXX");
+	strbuf_addstr(tmp, "/tmp_obj_XXXXXX");
 	fd = git_mkstemp_mode(tmp->buf, 0444);
 	do {
 		if (fd >= 0 || !dirlen || errno != ENOENT)
@@ -1866,7 +1866,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,
 		 * scratch.
 		 */
 		strbuf_reset(tmp);
-		strbuf_add(tmp, filename, dirlen - 1);
+		strbuf_add(tmp, filename, dirlen);
 		if (mkdir(tmp->buf, 0777) && errno != EEXIST)
 			break;
 		if (adjust_shared_perm(tmp->buf))
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
#145 lore raw

Re: [PATCH v8 3/6] object-file.c: remove the slash for directory_size()

Am 08.01.22 um 09:54 schrieb Han Xin:

Quoted 18 lines

From: Han Xin <[email protected]>

Since "mkdir foo/" works as well as "mkdir foo", let's remove the end slash as many users of it want.

---
 object-file.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
object-file.c +0 -0
index 5d163081b1..4f0127e823 100644
@@ -1831,13 +1831,13 @@ static void close_loose_object(int fd)
 		die_errno(_("error when closing loose object file"));
 }

object-file.c static void close_loose_object(int fd) · line 1840

Quoted 10 lines
object-file.c +2 -2
 
-/* Size of directory component, including the ending '/' */
+/* Size of directory component, excluding the ending '/' */
 static inline int directory_size(const char *filename)
 {
 	const char *s = strrchr(filename, '/');
 	if (!s)
 		return 0;
-	return s - filename + 1;
+	return s - filename;

This will return zero both for "filename" and "/filename". Hmm. Since it's only used for loose object files we can assume that at least one slash is present, so this removal of functionality is not actually a problem. But I don't understand its benefit.

Quoted 13 lines
object-file.c +1 -1
 }
 
 /*
@@ -1854,7 +1854,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,
 
 	strbuf_reset(tmp);
 	strbuf_add(tmp, filename, dirlen);
-	strbuf_addstr(tmp, "tmp_obj_XXXXXX");
+	strbuf_addstr(tmp, "/tmp_obj_XXXXXX");
 	fd = git_mkstemp_mode(tmp->buf, 0444);
 	do {
 		if (fd >= 0 || !dirlen || errno != ENOENT)

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename, · line 1870

Quoted 6 lines
object-file.c +1 -1
@@ -1866,7 +1866,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,
 		 * scratch.
 		 */
 		strbuf_reset(tmp);
-		strbuf_add(tmp, filename, dirlen - 1);
+		strbuf_add(tmp, filename, dirlen);
 		if (mkdir(tmp->buf, 0777) && errno != EEXIST)

This code makes sure that mkdir(2) is called without the trailing slash, both with or without this patch. From the commit message above I somehow expected a change in this regard -- but again I wouldn't understand its benefit.

Is this change really needed? Is streaming unpack not possible with the original directory_size() function?

Quoted 2 lines
object-file.c +0 -0
 			break;
 		if (adjust_shared_perm(tmp->buf))
author
#146 lore raw

Re: [PATCH v8 3/6] object-file.c: remove the slash for directory_size()

Quoted 61 lines · René Scharfe

Am 08.01.22 um 09:54 schrieb Han Xin:

Quoted 18 lines

From: Han Xin <[email protected]>

Since "mkdir foo/" works as well as "mkdir foo", let's remove the end slash as many users of it want.

---
 object-file.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
object-file.c +0 -0
index 5d163081b1..4f0127e823 100644
@@ -1831,13 +1831,13 @@ static void close_loose_object(int fd)
              die_errno(_("error when closing loose object file"));
 }

object-file.c static void close_loose_object(int fd) · line 1840

Quoted 10 lines
object-file.c +2 -2
 
-/* Size of directory component, including the ending '/' */
+/* Size of directory component, excluding the ending '/' */
 static inline int directory_size(const char *filename)
 {
      const char *s = strrchr(filename, '/');
      if (!s)
              return 0;
-     return s - filename + 1;
+     return s - filename;

This will return zero both for "filename" and "/filename". Hmm. Since it's only used for loose object files we can assume that at least one slash is present, so this removal of functionality is not actually a problem. But I don't understand its benefit.

Quoted 13 lines
object-file.c +1 -1
 }
 
 /*
@@ -1854,7 +1854,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,
 
      strbuf_reset(tmp);
      strbuf_add(tmp, filename, dirlen);
-     strbuf_addstr(tmp, "tmp_obj_XXXXXX");
+     strbuf_addstr(tmp, "/tmp_obj_XXXXXX");
      fd = git_mkstemp_mode(tmp->buf, 0444);
      do {
              if (fd >= 0 || !dirlen || errno != ENOENT)

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename, · line 1870

Quoted 6 lines
object-file.c +1 -1
@@ -1866,7 +1866,7 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename,
               * scratch.
               */
              strbuf_reset(tmp);
-             strbuf_add(tmp, filename, dirlen - 1);
+             strbuf_add(tmp, filename, dirlen);
              if (mkdir(tmp->buf, 0777) && errno != EEXIST)

This code makes sure that mkdir(2) is called without the trailing slash, both with or without this patch. From the commit message above I somehow expected a change in this regard -- but again I wouldn't understand its benefit.

Quoted 3 lines

Is this change really needed? Is streaming unpack not possible with the original directory_size() function?

*nod*
Streaming unpacking still works with the original directory_size().

This patch is more of a code cleanup that reduces the extra handling of directory size first increasing and then decreasing. I'll seriously consider if I should remove this patch, or move it to the tail.

Thanks
-Han Xin

Quoted 2 lines
Quoted 2 lines
object-file.c +0 -0
                      break;
              if (adjust_shared_perm(tmp->buf))
author
#147 lore raw

PATCH v8 4/6 object-file.c: add "stream_loose_object()" to handle large object

In reply to Han Xin

From: Han Xin <[email protected]>

If we want unpack and write a loose object using "write_loose_object", we have to feed it with a buffer with the same size of the object, which will consume lots of memory and may cause OOM. This can be improved by feeding data to "stream_loose_object()" in a stream.

Add a new function "stream_loose_object()", which is a stream version of "write_loose_object()" but with a low memory footprint. We will use this function to unpack large blob object in latter commit.

Another difference with "write_loose_object()" is that we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. In "write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

"freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid".

---
 object-file.c  | 101 +++++++++++++++++++++++++++++++++++++++++++++++++
 object-store.h |   9 +++++
 2 files changed, 110 insertions(+)
object-file.c +101 -0
index 4f0127e823..a462a21629 100644
@@ -2012,6 +2012,107 @@ static int freshen_packed_object(const struct object_id *oid)
 	return 1;
 }
 
+int stream_loose_object(struct input_stream *in_stream, size_t len,
+			struct object_id *oid)
+{
+	int fd, ret, err = 0, flush = 0;
+	unsigned char compressed[4096];
+	git_zstream stream;
+	git_hash_ctx c;
+	struct strbuf tmp_file = STRBUF_INIT;
+	struct strbuf filename = STRBUF_INIT;
+	int dirlen;
+	char hdr[MAX_HEADER_LEN];
+	int hdrlen;
+
+	/* Since oid is not determined, save tmp file to odb path. */
+	strbuf_addf(&filename, "%s/", get_object_directory());
+	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX, type_name(OBJ_BLOB), len) + 1;
+
+	/* Common steps for write_loose_object and stream_loose_object to
+	 * start writing loose oject:
+	 *
+	 *  - Create tmpfile for the loose object.
+	 *  - Setup zlib stream for compression.
+	 *  - Start to feed header to zlib stream.
+	 */
+	fd = start_loose_object_common(&tmp_file, filename.buf, 0,
+				       &stream, compressed, sizeof(compressed),
+				       &c, OBJ_BLOB, len, hdr, hdrlen);
+	if (fd < 0) {
+		err = -1;
+		goto cleanup;
+	}
+
+	/* Then the data itself.. */
+	do {
+		unsigned char *in0 = stream.next_in;
+		if (!stream.avail_in && !in_stream->is_finished) {
+			const void *in = in_stream->read(in_stream, &stream.avail_in);
+			stream.next_in = (void *)in;
+			in0 = (unsigned char *)in;
+			/* All data has been read. */
+			if (in_stream->is_finished)
+				flush = Z_FINISH;
+		}
+		ret = git_deflate(&stream, flush);
+		the_hash_algo->update_fn(&c, in0, stream.next_in - in0);
+		if (write_buffer(fd, compressed, stream.next_out - compressed) < 0)
+			die(_("unable to write loose object file"));
+		stream.next_out = compressed;
+		stream.avail_out = sizeof(compressed);
+		/*
+		 * Unlike write_loose_object(), we do not have the entire
+		 * buffer. If we get Z_BUF_ERROR due to too few input bytes,
+		 * then we'll replenish them in the next input_stream->read()
+		 * call when we loop.
+		 */
+	} while (ret == Z_OK || ret == Z_BUF_ERROR);
+
+	if (stream.total_in != len + hdrlen)
+		die(_("write stream object %ld != %"PRIuMAX), stream.total_in,
+		    (uintmax_t)len + hdrlen);
+
+	/* Common steps for write_loose_object and stream_loose_object to
+	 * end writing loose oject:
+	 *
+	 *  - End the compression of zlib stream.
+	 *  - Get the calculated oid.
+	 */
+	end_loose_object_common(ret, &c, &stream, oid, NULL,
+				N_("unable to stream deflate new object (%d)"),
+				N_("deflateEnd on stream object failed (%d)"));
+
+	close_loose_object(fd);
+
+	if (freshen_packed_object(oid) || freshen_loose_object(oid)) {
+		unlink_or_warn(tmp_file.buf);
+		goto cleanup;
+	}
+
+	loose_object_path(the_repository, &filename, oid);
+
+	/* We finally know the object path, and create the missing dir. */
+	dirlen = directory_size(filename.buf);
+	if (dirlen) {
+		struct strbuf dir = STRBUF_INIT;
+		strbuf_add(&dir, filename.buf, dirlen);
+
+		if (mkdir_in_gitdir(dir.buf) && errno != EEXIST) {
+			err = error_errno(_("unable to create directory %s"), dir.buf);
+			strbuf_release(&dir);
+			goto cleanup;
+		}
+		strbuf_release(&dir);
+	}
+
+	err = finalize_object_file(tmp_file.buf, filename.buf);
+cleanup:
+	strbuf_release(&tmp_file);
+	strbuf_release(&filename);
+	return err;
+}
+
 int write_object_file_flags(const void *buf, unsigned long len,
 			    const char *type, struct object_id *oid,
 			    unsigned flags)
object-store.h +9 -0
index 952efb6a4b..cc41c64d69 100644
@@ -34,6 +34,12 @@ struct object_directory {
 	char *path;
 };
 
+struct input_stream {
+	const void *(*read)(struct input_stream *, unsigned long *len);
+	void *data;
+	int is_finished;
+};
+
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
 	struct object_directory *, 1, fspathhash, fspatheq)
 
@@ -232,6 +238,9 @@ static inline int write_object_file(const void *buf, unsigned long len,
 	return write_object_file_flags(buf, len, type, oid, 0);
 }
 
+int stream_loose_object(struct input_stream *in_stream, size_t len,
+			struct object_id *oid);
+
 int hash_object_file_literally(const void *buf, unsigned long len,
 			       const char *type, struct object_id *oid,
 			       unsigned flags);
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
author
#148 lore raw

PATCH v8 5/6 unpack-objects: unpack_non_delta_entry() read data in a stream

In reply to Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()". However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty.

    $ hyperfine \
      --setup \
      'if ! test -d scalar.git; then git clone --bare
       https://github.com/microsoft/scalar.git;
       cp scalar.git/objects/pack/*.pack small.pack; fi' \
      --prepare 'rm -rf dest.git && git init --bare dest.git' \
      ...
    Summary
      './git -C dest.git -c core.bigFileThreshold=512m
      unpack-objects <small.pack' in 'origin/master'
        1.01 ± 0.04 times faster than './git -C dest.git
                -c core.bigFileThreshold=512m unpack-objects
                <small.pack' in 'HEAD~1'
        1.01 ± 0.04 times faster than './git -C dest.git
                -c core.bigFileThreshold=512m unpack-objects
                <small.pack' in 'HEAD~0'
        1.03 ± 0.10 times faster than './git -C dest.git
                -c core.bigFileThreshold=16k unpack-objects
                <small.pack' in 'origin/master'
        1.02 ± 0.07 times faster than './git -C dest.git
                -c core.bigFileThreshold=16k unpack-objects
                <small.pack' in 'HEAD~0'
        1.10 ± 0.04 times faster than './git -C dest.git
                -c core.bigFileThreshold=16k unpack-objects
                <small.pack' in 'HEAD~1'

Therefore, only unpack objects larger than the "core.bigFileThreshold" in zstream. Until now, the config variable has been used in the following cases, and our new case belongs to the packfile category.

* Archive:

+ archive.c: write_entry(): write large blob entries to archive in stream.

* Loose objects:

+ object-file.c: index_fd(): when hashing large files in worktree, read files in a stream, and create one packfile per large blob if want to save files to git object store.

+ object-file.c: read_loose_object(): when checking loose objects using "git-fsck", do not read full content of large loose objects.

* Packfile:

+ fast-import.c: parse_and_store_blob(): streaming large blob from foreign source to packfile.

+ index-pack.c: check_collison(): read and check large blob in stream.

+ index-pack.c: unpack_entry_data(): do not return the entire contents of the big blob from packfile, but uses a fixed buf to perform some integrity checks on the object.

+ pack-check.c: verify_packfile(): used by "git-fsck" and will call check_object_signature() to check large blob in pack with the streaming interface.

+ pack-objects.c: get_object_details(): set "no_try_delta" for large blobs when counting objects.

+ pack-objects.c: write_no_reuse_object(): streaming large blob to pack.

+ unpack-objects.c: unpack_non_delta_entry(): unpack large blob in stream from packfile.

* Others:

+ diff.c: diff_populate_filespec(): treat large blob file as binary.

+ streaming.c: istream_source(): as a helper of "open_istream()" to select proper streaming interface to read large blob from packfile.

---
 builtin/unpack-objects.c        | 71 ++++++++++++++++++++++++++++++++-
 t/t5329-unpack-large-objects.sh | 23 +++++++++--
 2 files changed, 90 insertions(+), 4 deletions(-)
builtin/unpack-objects.c +70 -1
index c6d6c17072..e9ec2b349d 100644
@@ -343,11 +343,80 @@ static void added_object(unsigned nr, enum object_type type,
 	}
 }
 
+struct input_zstream_data {
+	git_zstream *zstream;
+	unsigned char buf[8192];
+	int status;
+};
+
+static const void *feed_input_zstream(struct input_stream *in_stream,
+				      unsigned long *readlen)
+{
+	struct input_zstream_data *data = in_stream->data;
+	git_zstream *zstream = data->zstream;
+	void *in = fill(1);
+
+	if (in_stream->is_finished) {
+		*readlen = 0;
+		return NULL;
+	}
+
+	zstream->next_out = data->buf;
+	zstream->avail_out = sizeof(data->buf);
+	zstream->next_in = in;
+	zstream->avail_in = len;
+
+	data->status = git_inflate(zstream, 0);
+
+	in_stream->is_finished = data->status != Z_OK;
+	use(len - zstream->avail_in);
+	*readlen = sizeof(data->buf) - zstream->avail_out;
+
+	return data->buf;
+}
+
+static void write_stream_blob(unsigned nr, size_t size)
+{
+	git_zstream zstream = { 0 };
+	struct input_zstream_data data = { 0 };
+	struct input_stream in_stream = {
+		.read = feed_input_zstream,
+		.data = &data,
+	};
+
+	data.zstream = &zstream;
+	git_inflate_init(&zstream);
+
+	if (stream_loose_object(&in_stream, size, &obj_list[nr].oid))
+		die(_("failed to write object in stream"));
+
+	if (data.status != Z_STREAM_END)
+		die(_("inflate returned (%d)"), data.status);
+	git_inflate_end(&zstream);
+
+	if (strict) {
+		struct blob *blob =
+			lookup_blob(the_repository, &obj_list[nr].oid);
+		if (blob)
+			blob->object.flags |= FLAG_WRITTEN;
+		else
+			die(_("invalid blob object from stream"));
+	}
+	obj_list[nr].obj = NULL;
+}
+
 static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 				   unsigned nr)
 {
-	void *buf = get_data(size);
+	void *buf;
+
+	/* Write large blob in stream without allocating full buffer. */
+	if (!dry_run && type == OBJ_BLOB && size > big_file_threshold) {
+		write_stream_blob(nr, size);
+		return;
+	}
 
+	buf = get_data(size);
 	if (buf)
 		write_object(nr, type, buf, size);
 }
t/t5329-unpack-large-objects.sh +20 -3
index 39c7a62d94..6f3bfb3df7 100755
@@ -9,7 +9,11 @@ test_description='git unpack-objects with large objects'
 
 prepare_dest () {
 	test_when_finished "rm -rf dest.git" &&
-	git init --bare dest.git
+	git init --bare dest.git &&
+	if test -n "$1"
+	then
+		git -C dest.git config core.bigFileThreshold $1
+	fi
 }
 
 assert_no_loose () {
@@ -37,16 +41,29 @@ test_expect_success 'set memory limitation to 1MB' '
 '
 
 test_expect_success 'unpack-objects failed under memory limitation' '
-	prepare_dest &&
+	prepare_dest 2m &&
 	test_must_fail git -C dest.git unpack-objects <test-$PACK.pack 2>err &&
 	grep "fatal: attempting to allocate" err
 '
 
 test_expect_success 'unpack-objects works with memory limitation in dry-run mode' '
-	prepare_dest &&
+	prepare_dest 2m &&
 	git -C dest.git unpack-objects -n <test-$PACK.pack &&
 	assert_no_loose &&
 	assert_no_pack
 '
 
+test_expect_success 'unpack big object in stream' '
+	prepare_dest 1m &&
+	git -C dest.git unpack-objects <test-$PACK.pack &&
+	assert_no_pack
+'
+
+test_expect_success 'do not unpack existing large objects' '
+	prepare_dest 1m &&
+	git -C dest.git index-pack --stdin <test-$PACK.pack &&
+	git -C dest.git unpack-objects <test-$PACK.pack &&
+	assert_no_loose
+'
+
 test_done
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
author
#149 lore raw

PATCH v8 6/6 object-file API: add a format_object_header() function

In reply to Han Xin

From: Ævar Arnfjörð Bjarmason <[email protected]>

Add a convenience function to wrap the xsnprintf() command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.

All except one caller of it had a valid "enum object_type" for us, it's only write_object_file_prepare() which might need to deal with "git hash-object --literally" and a potential garbage type. Let's have the primary API use an "enum object_type", and define an *_extended() function that can take an arbitrary "const char *" for the type.

See [1] for the discussion that prompted this patch, i.e. new code in object-file.c that wanted to copy/paste the xsnprintf() invocation.

1. https://lore.kernel.org/git/[email protected]/

---
 builtin/index-pack.c |  3 +--
 bulk-checkin.c       |  4 ++--
 cache.h              | 21 +++++++++++++++++++++
 http-push.c          |  2 +-
 object-file.c        | 16 ++++++++++++----
 5 files changed, 37 insertions(+), 9 deletions(-)
builtin/index-pack.c +1 -2
index c23d01de7d..8a6ce77940 100644
@@ -449,8 +449,7 @@ static void *unpack_entry_data(off_t offset, unsigned long size,
 	int hdrlen;
 
 	if (!is_delta_type(type)) {
-		hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX,
-				   type_name(type),(uintmax_t)size) + 1;
+		hdrlen = format_object_header(hdr, sizeof(hdr), type, size);
 		the_hash_algo->init_fn(&c);
 		the_hash_algo->update_fn(&c, hdr, hdrlen);
 	} else
bulk-checkin.c +2 -2
index 8785b2ac80..9e685f0f1a 100644
@@ -220,8 +220,8 @@ static int deflate_to_pack(struct bulk_checkin_state *state,
 	if (seekback == (off_t) -1)
 		return error("cannot find the current offset");
 
-	header_len = xsnprintf((char *)obuf, sizeof(obuf), "%s %" PRIuMAX,
-			       type_name(type), (uintmax_t)size) + 1;
+	header_len = format_object_header((char *)obuf, sizeof(obuf),
+					 type, size);
 	the_hash_algo->init_fn(&ctx);
 	the_hash_algo->update_fn(&ctx, obuf, header_len);
 
cache.h +21 -0
index cfba463aa9..64071a8d80 100644
@@ -1310,6 +1310,27 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
 						    unsigned long bufsiz,
 						    struct strbuf *hdrbuf);
 
+/**
+ * format_object_header() is a thin wrapper around s xsnprintf() that
+ * writes the initial "<type> <obj-len>" part of the loose object
+ * header. It returns the size that snprintf() returns + 1.
+ *
+ * The format_object_header_extended() function allows for writing a
+ * type_name that's not one of the "enum object_type" types. This is
+ * used for "git hash-object --literally". Pass in a OBJ_NONE as the
+ * type, and a non-NULL "type_str" to do that.
+ *
+ * format_object_header() is a convenience wrapper for
+ * format_object_header_extended().
+ */
+int format_object_header_extended(char *str, size_t size, enum object_type type,
+				 const char *type_str, size_t objsize);
+static inline int format_object_header(char *str, size_t size,
+				      enum object_type type, size_t objsize)
+{
+	return format_object_header_extended(str, size, type, NULL, objsize);
+}
+
 /**
  * parse_loose_header() parses the starting "<type> <len>\0" of an
  * object. If it doesn't follow that format -1 is returned. To check
http-push.c +1 -1
index 3309aaf004..f0c044dcf7 100644
@@ -363,7 +363,7 @@ static void start_put(struct transfer_request *request)
 	git_zstream stream;
 
 	unpacked = read_object_file(&request->obj->oid, &type, &len);
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
+	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
 
 	/* Set it up */
 	git_deflate_init(&stream, zlib_compression_level);
object-file.c +12 -4
index a462a21629..d384ef2952 100644
@@ -1006,6 +1006,14 @@ void *xmmap(void *start, size_t length,
 	return ret;
 }
 
+int format_object_header_extended(char *str, size_t size, enum object_type type,
+				 const char *typestr, size_t objsize)
+{
+	const char *s = type == OBJ_NONE ? typestr : type_name(type);
+
+	return xsnprintf(str, size, "%s %"PRIuMAX, s, (uintmax_t)objsize) + 1;
+}
+
 /*
  * With an in-core object data in "map", rehash it to make sure the
  * object name actually matches "oid" to detect object corruption.
@@ -1034,7 +1042,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,
 		return -1;
 
 	/* Generate the header */
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(obj_type), (uintmax_t)size) + 1;
+	hdrlen = format_object_header(hdr, sizeof(hdr), obj_type, size);
 
 	/* Sha1.. */
 	r->hash_algo->init_fn(&c);
@@ -1734,7 +1742,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,
 	git_hash_ctx c;
 
 	/* Generate the header */
-	*hdrlen = xsnprintf(hdr, *hdrlen, "%s %"PRIuMAX , type, (uintmax_t)len)+1;
+	*hdrlen = format_object_header_extended(hdr, *hdrlen, OBJ_NONE, type, len);
 
 	/* Sha1.. */
 	algo->init_fn(&c);
@@ -2027,7 +2035,7 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
 
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", get_object_directory());
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX, type_name(OBJ_BLOB), len) + 1;
+	hdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, len);
 
 	/* Common steps for write_loose_object and stream_loose_object to
 	 * start writing loose oject:
@@ -2168,7 +2176,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 	buf = read_object(the_repository, oid, &type, &len);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
+	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
 	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
 	free(buf);
 
Signature · 1 line
2.34.1.52.gc288e771b4.agit.6.5.6
author
#150 lore raw

PATCH v6 1/6 object-file.c: release strbuf in write_loose_object()

In reply to Han Xin

From: Han Xin <[email protected]>

Fix a strbuf leak in "write_loose_object()" sugguested by Ævar Arnfjörð Bjarmason.

---
 object-file.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
object-file.c +11 -4
index eb1426f98c..32acf1dad6 100644
@@ -1874,11 +1874,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
 		if (flags & HASH_SILENT)
-			return -1;
+			ret = -1;
 		else if (errno == EACCES)
-			return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
+			ret = error(_("insufficient permission for adding an "
+				      "object to repository database %s"),
+				    get_object_directory());
 		else
-			return error_errno(_("unable to create temporary file"));
+			ret = error_errno(_("unable to create temporary file"));
+		goto cleanup;
 	}
 
 	/* Set it up */
@@ -1930,7 +1933,11 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 			warning_errno(_("failed utime() on %s"), tmp_file.buf);
 	}
 
-	return finalize_object_file(tmp_file.buf, filename.buf);
+	ret = finalize_object_file(tmp_file.buf, filename.buf);
+cleanup:
+	strbuf_release(&filename);
+	strbuf_release(&tmp_file);
+	return ret;
 }
 
 static int freshen_loose_object(const struct object_id *oid)
Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
#151 lore raw

Re: [PATCH v6 1/6] object-file.c: release strbuf in write_loose_object()

Am 17.12.21 um 12:26 schrieb Han Xin:

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr,

Quoted 16 lines

From: Han Xin <[email protected]>

Fix a strbuf leak in "write_loose_object()" sugguested by Ævar Arnfjörð Bjarmason.

---
 object-file.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
object-file.c +0 -0
index eb1426f98c..32acf1dad6 100644
@@ -1874,11 +1874,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,

Relevant context lines:

	static struct strbuf tmp_file = STRBUF_INIT;
	static struct strbuf filename = STRBUF_INIT;

	loose_object_path(the_repository, &filename, oid);
Quoted 18 lines
object-file.c +6 -3
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
 		if (flags & HASH_SILENT)
-			return -1;
+			ret = -1;
 		else if (errno == EACCES)
-			return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
+			ret = error(_("insufficient permission for adding an "
+				      "object to repository database %s"),
+				    get_object_directory());
 		else
-			return error_errno(_("unable to create temporary file"));
+			ret = error_errno(_("unable to create temporary file"));
+		goto cleanup;
 	}
 
 	/* Set it up */

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1939

Quoted 8 lines
object-file.c +4 -1
@@ -1930,7 +1933,11 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 			warning_errno(_("failed utime() on %s"), tmp_file.buf);
 	}
 
-	return finalize_object_file(tmp_file.buf, filename.buf);
+	ret = finalize_object_file(tmp_file.buf, filename.buf);
+cleanup:
+	strbuf_release(&filename);
+	strbuf_release(&tmp_file);

There was no leak before. Both strbufs are static and both functions they are passed to (loose_object_path() and create_tmpfile()) reset them first. So while the allocated memory was not released before, it was reused.

Not sure if making write_loose_object() allocate and release these buffers on every call has much of a performance impact. The only reason I can think of for wanting such a change is to get rid of the static buffers, to allow the function to be used by concurrent threads.

So I think either keeping the code as-is or also making the strbufs non-static would be better (but then discussing a possible performance impact in the commit message would be nice).

Quoted 4 lines
object-file.c +1 -0
+	return ret;
 }
 
 static int freshen_loose_object(const struct object_id *oid)
#152 lore raw

Re: [PATCH v6 1/6] object-file.c: release strbuf in write_loose_object()

Quoted 11 lines · René Scharfe

There was no leak before. Both strbufs are static and both functions they are passed to (loose_object_path() and create_tmpfile()) reset them first. So while the allocated memory was not released before, it was reused.

Not sure if making write_loose_object() allocate and release these buffers on every call has much of a performance impact. The only reason I can think of for wanting such a change is to get rid of the static buffers, to allow the function to be used by concurrent threads.

Quoted 3 lines

So I think either keeping the code as-is or also making the strbufs non-static would be better (but then discussing a possible performance impact in the commit message would be nice).

Makes sense.

author
#153 lore raw

PATCH v6 2/6 object-file.c: refactor object header generation into a function

In reply to Han Xin

From: Han Xin <[email protected]>

There are 3 places where "xsnprintf" is used to generate the object header, and I originally planned to add a fourth in the latter patch.

According to Ævar Arnfjörð Bjarmason’s suggestion, although it's just one line, it's also code that's very central to git, so reafactor them into a function which will help later readability.

---
 object-file.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
object-file.c +11 -3
index 32acf1dad6..95fcd5435d 100644
@@ -1006,6 +1006,14 @@ void *xmmap(void *start, size_t length,
 	return ret;
 }
 
+static inline int generate_object_header(char *buf, int bufsz,
+					 const char *type_name,
+					 unsigned long size)
+{
+	return xsnprintf(buf, bufsz, "%s %"PRIuMAX, type_name,
+			 (uintmax_t)size) + 1;
+}
+
 /*
  * With an in-core object data in "map", rehash it to make sure the
  * object name actually matches "oid" to detect object corruption.
@@ -1034,7 +1042,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,
 		return -1;
 
 	/* Generate the header */
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(obj_type), (uintmax_t)size) + 1;
+	hdrlen = generate_object_header(hdr, sizeof(hdr), type_name(obj_type), size);
 
 	/* Sha1.. */
 	r->hash_algo->init_fn(&c);
@@ -1734,7 +1742,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,
 	git_hash_ctx c;
 
 	/* Generate the header */
-	*hdrlen = xsnprintf(hdr, *hdrlen, "%s %"PRIuMAX , type, (uintmax_t)len)+1;
+	*hdrlen = generate_object_header(hdr, *hdrlen, type, len);
 
 	/* Sha1.. */
 	algo->init_fn(&c);
@@ -2013,7 +2021,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 	buf = read_object(the_repository, oid, &type, &len);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
+	hdrlen = generate_object_header(hdr, sizeof(hdr), type_name(type), len);
 	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
 	free(buf);
 
Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
#154 lore raw

RFC PATCH 1/1 object-file API: add a format_loose_header() function

Add a convenience function to wrap the xsnprintf() command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.

All except one caller of it had a valid "enum object_type" for us, it's only write_object_file_prepare() which might need to deal with "git hash-object --literally" and a potential garbage type. Let's have the primary API use an "enum object_type", and define an *_extended() function that can take an arbitrary "const char *" for the type.

See [1] for the discussion that prompted this patch, i.e. new code in object-file.c that wanted to copy/paste the xsnprintf() invocation.

1. https://lore.kernel.org/git/[email protected]/

---

Quoted 11 lines · Han Xin

From: Han Xin <[email protected]>

There are 3 places where "xsnprintf" is used to generate the object header, and I originally planned to add a fourth in the latter patch.

According to Ævar Arnfjörð Bjarmason’s suggestion, although it's just one line, it's also code that's very central to git, so reafactor them into a function which will help later readability.

I came up with this after my comment on the earlier round suggesting to factor out that header formatting. I don't know if this more thorough approach is worth it or if you'd like to replace your change with this one, but just posting it here as an RFC.

 builtin/index-pack.c |  3 +--
 bulk-checkin.c       |  4 ++--
 cache.h              | 21 +++++++++++++++++++++
 http-push.c          |  2 +-
 object-file.c        | 14 +++++++++++---
 5 files changed, 36 insertions(+), 8 deletions(-)
builtin/index-pack.c +1 -2
index c23d01de7dc..900c6539f68 100644
@@ -449,8 +449,7 @@ static void *unpack_entry_data(off_t offset, unsigned long size,
 	int hdrlen;
 
 	if (!is_delta_type(type)) {
-		hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX,
-				   type_name(type),(uintmax_t)size) + 1;
+		hdrlen = format_loose_header(hdr, sizeof(hdr), type, (uintmax_t)size);
 		the_hash_algo->init_fn(&c);
 		the_hash_algo->update_fn(&c, hdr, hdrlen);
 	} else
bulk-checkin.c +2 -2
index 8785b2ac806..446dea7c516 100644
@@ -220,8 +220,8 @@ static int deflate_to_pack(struct bulk_checkin_state *state,
 	if (seekback == (off_t) -1)
 		return error("cannot find the current offset");
 
-	header_len = xsnprintf((char *)obuf, sizeof(obuf), "%s %" PRIuMAX,
-			       type_name(type), (uintmax_t)size) + 1;
+	header_len = format_loose_header((char *)obuf, sizeof(obuf),
+					 type, (uintmax_t)size);
 	the_hash_algo->init_fn(&ctx);
 	the_hash_algo->update_fn(&ctx, obuf, header_len);
 
cache.h +21 -0
index d5cafba17d4..ccece21a4a2 100644
@@ -1309,6 +1309,27 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
 						    unsigned long bufsiz,
 						    struct strbuf *hdrbuf);
 
+/**
+ * format_loose_header() is a thin wrapper around s xsnprintf() that
+ * writes the initial "<type> <obj-len>" part of the loose object
+ * header. It returns the size that snprintf() returns + 1.
+ *
+ * The format_loose_header_extended() function allows for writing a
+ * type_name that's not one of the "enum object_type" types. This is
+ * used for "git hash-object --literally". Pass in a OBJ_NONE as the
+ * type, and a non-NULL "type_str" to do that.
+ *
+ * format_loose_header() is a convenience wrapper for
+ * format_loose_header_extended().
+ */
+int format_loose_header_extended(char *str, size_t size, enum object_type type,
+				 const char *type_str, size_t objsize);
+static inline int format_loose_header(char *str, size_t size,
+				      enum object_type type, size_t objsize)
+{
+	return format_loose_header_extended(str, size, type, NULL, objsize);
+}
+
 /**
  * parse_loose_header() parses the starting "<type> <len>\0" of an
  * object. If it doesn't follow that format -1 is returned. To check
http-push.c +1 -1
index 3309aaf004a..d1a8619e0af 100644
@@ -363,7 +363,7 @@ static void start_put(struct transfer_request *request)
 	git_zstream stream;
 
 	unpacked = read_object_file(&request->obj->oid, &type, &len);
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
+	hdrlen = format_loose_header(hdr, sizeof(hdr), type, (uintmax_t)len);
 
 	/* Set it up */
 	git_deflate_init(&stream, zlib_compression_level);
object-file.c +11 -3
index eac67f6f5f9..d94609ee48d 100644
@@ -1009,6 +1009,14 @@ void *xmmap(void *start, size_t length,
 	return ret;
 }
 
+int format_loose_header_extended(char *str, size_t size, enum object_type type,
+				 const char *typestr, size_t objsize)
+{
+	const char *s = type == OBJ_NONE ? typestr : type_name(type);
+
+	return xsnprintf(str, size, "%s %"PRIuMAX, s, (uintmax_t)objsize) + 1;
+}
+
 /*
  * With an in-core object data in "map", rehash it to make sure the
  * object name actually matches "oid" to detect object corruption.
@@ -1037,7 +1045,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,
 		return -1;
 
 	/* Generate the header */
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(obj_type), (uintmax_t)size) + 1;
+	hdrlen = format_loose_header(hdr, sizeof(hdr), obj_type, size);
 
 	/* Sha1.. */
 	r->hash_algo->init_fn(&c);
@@ -1737,7 +1745,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,
 	git_hash_ctx c;
 
 	/* Generate the header */
-	*hdrlen = xsnprintf(hdr, *hdrlen, "%s %"PRIuMAX , type, (uintmax_t)len)+1;
+	*hdrlen = format_loose_header_extended(hdr, *hdrlen, OBJ_NONE, type, len);
 
 	/* Sha1.. */
 	algo->init_fn(&c);
@@ -2009,7 +2017,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 	buf = read_object(the_repository, oid, &type, &len);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
+	hdrlen = format_loose_header(hdr, sizeof(hdr), type, len);
 	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
 	free(buf);
 
Signature · 1 line
2.34.1.1119.g606023410ba
#155 lore raw

Re: [RFC PATCH] object-file API: add a format_loose_header() function

Hi Ævar,
(catching up after a week away, and noticed your patch today..)

Quoted 10 lines · Ævar Arnfjörð Bjarmason

Add a convenience function to wrap the xsnprintf() command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.

All except one caller of it had a valid "enum object_type" for us, it's only write_object_file_prepare() which might need to deal with "git hash-object --literally" and a potential garbage type. Let's have the primary API use an "enum object_type", and define an *_extended() function that can take an arbitrary "const char *" for the type.

I recently completed a PR in the Git for Windows build that is focused on "git hash-object --literally" as a starter for LLP64 large file (>4GB) compatibility.
(https://github.com/git-for-windows/git/pull/3533), which Dscho has merged (cc'd).

I'm not sure that the extended version will work as expected across the test suite
as multiple fake object types are tried, though I only skimmed the patch.

I'd support the general thrust, but just wanted to synchronise any changes.

Philip

Quoted 154 lines

See [1] for the discussion that prompted this patch, i.e. new code in object-file.c that wanted to copy/paste the xsnprintf() invocation.

1. https://lore.kernel.org/git/[email protected]/

---

Quoted 11 lines · Han Xin

From: Han Xin <[email protected]>

There are 3 places where "xsnprintf" is used to generate the object header, and I originally planned to add a fourth in the latter patch.

According to Ævar Arnfjörð Bjarmason’s suggestion, although it's just one line, it's also code that's very central to git, so reafactor them into a function which will help later readability.

I came up with this after my comment on the earlier round suggesting to factor out that header formatting. I don't know if this more thorough approach is worth it or if you'd like to replace your change with this one, but just posting it here as an RFC.

 builtin/index-pack.c |  3 +--
 bulk-checkin.c       |  4 ++--
 cache.h              | 21 +++++++++++++++++++++
 http-push.c          |  2 +-
 object-file.c        | 14 +++++++++++---
 5 files changed, 36 insertions(+), 8 deletions(-)
builtin/index-pack.c +1 -2
index c23d01de7dc..900c6539f68 100644
@@ -449,8 +449,7 @@ static void *unpack_entry_data(off_t offset, unsigned long size,
 	int hdrlen;
 
 	if (!is_delta_type(type)) {
-		hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX,
-				   type_name(type),(uintmax_t)size) + 1;
+		hdrlen = format_loose_header(hdr, sizeof(hdr), type, (uintmax_t)size);
 		the_hash_algo->init_fn(&c);
 		the_hash_algo->update_fn(&c, hdr, hdrlen);
 	} else
bulk-checkin.c +2 -2
index 8785b2ac806..446dea7c516 100644
@@ -220,8 +220,8 @@ static int deflate_to_pack(struct bulk_checkin_state *state,
 	if (seekback == (off_t) -1)
 		return error("cannot find the current offset");
 
-	header_len = xsnprintf((char *)obuf, sizeof(obuf), "%s %" PRIuMAX,
-			       type_name(type), (uintmax_t)size) + 1;
+	header_len = format_loose_header((char *)obuf, sizeof(obuf),
+					 type, (uintmax_t)size);
 	the_hash_algo->init_fn(&ctx);
 	the_hash_algo->update_fn(&ctx, obuf, header_len);
 
cache.h +21 -0
index d5cafba17d4..ccece21a4a2 100644
@@ -1309,6 +1309,27 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
 						    unsigned long bufsiz,
 						    struct strbuf *hdrbuf);
 
+/**
+ * format_loose_header() is a thin wrapper around s xsnprintf() that
+ * writes the initial "<type> <obj-len>" part of the loose object
+ * header. It returns the size that snprintf() returns + 1.
+ *
+ * The format_loose_header_extended() function allows for writing a
+ * type_name that's not one of the "enum object_type" types. This is
+ * used for "git hash-object --literally". Pass in a OBJ_NONE as the
+ * type, and a non-NULL "type_str" to do that.
+ *
+ * format_loose_header() is a convenience wrapper for
+ * format_loose_header_extended().
+ */
+int format_loose_header_extended(char *str, size_t size, enum object_type type,
+				 const char *type_str, size_t objsize);
+static inline int format_loose_header(char *str, size_t size,
+				      enum object_type type, size_t objsize)
+{
+	return format_loose_header_extended(str, size, type, NULL, objsize);
+}
+
 /**
  * parse_loose_header() parses the starting "<type> <len>\0" of an
  * object. If it doesn't follow that format -1 is returned. To check
http-push.c +1 -1
index 3309aaf004a..d1a8619e0af 100644
@@ -363,7 +363,7 @@ static void start_put(struct transfer_request *request)
 	git_zstream stream;
 
 	unpacked = read_object_file(&request->obj->oid, &type, &len);
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
+	hdrlen = format_loose_header(hdr, sizeof(hdr), type, (uintmax_t)len);
 
 	/* Set it up */
 	git_deflate_init(&stream, zlib_compression_level);
object-file.c +11 -3
index eac67f6f5f9..d94609ee48d 100644
@@ -1009,6 +1009,14 @@ void *xmmap(void *start, size_t length,
 	return ret;
 }
 
+int format_loose_header_extended(char *str, size_t size, enum object_type type,
+				 const char *typestr, size_t objsize)
+{
+	const char *s = type == OBJ_NONE ? typestr : type_name(type);
+
+	return xsnprintf(str, size, "%s %"PRIuMAX, s, (uintmax_t)objsize) + 1;
+}
+
 /*
  * With an in-core object data in "map", rehash it to make sure the
  * object name actually matches "oid" to detect object corruption.
@@ -1037,7 +1045,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,
 		return -1;
 
 	/* Generate the header */
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(obj_type), (uintmax_t)size) + 1;
+	hdrlen = format_loose_header(hdr, sizeof(hdr), obj_type, size);
 
 	/* Sha1.. */
 	r->hash_algo->init_fn(&c);
@@ -1737,7 +1745,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,
 	git_hash_ctx c;
 
 	/* Generate the header */
-	*hdrlen = xsnprintf(hdr, *hdrlen, "%s %"PRIuMAX , type, (uintmax_t)len)+1;
+	*hdrlen = format_loose_header_extended(hdr, *hdrlen, OBJ_NONE, type, len);
 
 	/* Sha1.. */
 	algo->init_fn(&c);
@@ -2009,7 +2017,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 	buf = read_object(the_repository, oid, &type, &len);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
+	hdrlen = format_loose_header(hdr, sizeof(hdr), type, len);
 	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
 	free(buf);
 
#156 lore raw

Re: [RFC PATCH] object-file API: add a format_loose_header() function

In reply to Ævar Arnfjörð Bjarmason
Quoted 7 lines · Ævar Arnfjörð Bjarmason

Add a convenience function to wrap the xsnprintf() command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.
...
+/**
+ * format_loose_header() is a thin wrapper around s xsnprintf() that

The name should have "object" somewhere in it. Not all readers can be expected to know that you meant "loose" to be an acceptable short hand for "loose object".

That nit aside, I think it is a good idea to give people a common helper function to call. I am undecided if it is a good idea to make it take enum or "const char *"; most everybody should be able to say

format_object_header(type_name(OBJ_COMMIT), ...)

just fine, so two variants might be overkill, just to allow

format_object_header(OBJ_COMMIT, ...)

and to forbid

	format_object_header("connit", ...)

I dunno.

#157 lore raw

Re: [RFC PATCH] object-file API: add a format_loose_header() function

Quoted 10 lines · Junio C Hamano
Quoted 7 lines · Ævar Arnfjörð Bjarmason

Add a convenience function to wrap the xsnprintf() command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there.
...
+/**
+ * format_loose_header() is a thin wrapper around s xsnprintf() that

Quoted 3 lines

The name should have "object" somewhere in it. Not all readers can be expected to know that you meant "loose" to be an acceptable short hand for "loose object".

*nod*

Quoted 9 lines

That nit aside, I think it is a good idea to give people a common helper function to call. I am undecided if it is a good idea to make it take enum or "const char *"; most everybody should be able to say

format_object_header(type_name(OBJ_COMMIT), ...)

just fine, so two variants might be overkill, just to allow

Quoted 7 lines

format_object_header(OBJ_COMMIT, ...)

and to forbid

	format_object_header("connit", ...)

I dunno.

Ultimately only a single API caller in hash-object.c really cares about something else than the enum.

I've got some patches locally to convert e.g. write_object_file() to use the enum, and it removes the need for some callers to convert enum to char *, only to have other things convert it back.

So I think for any new APIs it makes sense to work towards sidelining the hash-object.c --literally caller.

#158 lore raw

Re: [RFC PATCH] object-file API: add a format_loose_header() function

Quoted 6 lines · Ævar Arnfjörð Bjarmason

I've got some patches locally to convert e.g. write_object_file() to use the enum, and it removes the need for some callers to convert enum to char *, only to have other things convert it back.

So I think for any new APIs it makes sense to work towards sidelining the hash-object.c --literally caller.

Your logic is backwards to argue "because I did something this way, it makes sense to do it this way"?

#159 lore raw

Re: [RFC PATCH] object-file API: add a format_loose_header() function

Quoted 11 lines · Junio C Hamano
Quoted 6 lines · Ævar Arnfjörð Bjarmason

I've got some patches locally to convert e.g. write_object_file() to use the enum, and it removes the need for some callers to convert enum to char *, only to have other things convert it back.

So I think for any new APIs it makes sense to work towards sidelining the hash-object.c --literally caller.

Your logic is backwards to argue "because I did something this way, it makes sense to do it this way"?

No, it's that if you look at the write_object_file() and hash_object_file() callers in-tree now many, including in object-file.c itself are taking an "enum object_type" only to convert it to a string, and then we'll in turn sometimes convert that to the "enum object_type" again at some lower level.

That API inconsistency dates back to at least Linus's a733cb606fe (Change pack file format. Hopefully for the last time., 2005-06-28).

I'm just pointing out that I have local patches that prove that a lot of back & forth is done for no good reason, and that this is one of the codepaths that's tangentally involved. So it makes sense in this case to make any new API take "enum object_type" as the primary interface.

author
#160 lore raw

Re: [RFC PATCH] object-file API: add a format_loose_header() function

In reply to Ævar Arnfjörð Bjarmason
Quoted 5 lines · Ævar Arnfjörð Bjarmason

I came up with this after my comment on the earlier round suggesting to factor out that header formatting. I don't know if this more thorough approach is worth it or if you'd like to replace your change with this one, but just posting it here as an RFC.

I will take this patch and rename the function name from "format_loose_header()" to "format_object_header()".

Thanks
-Han Xin

author
#161 lore raw

PATCH v6 3/6 object-file.c: refactor write_loose_object() to reuse in stream version

In reply to Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

This can be improved by feeding data to "stream_loose_object()" in stream instead of read into the whole buf.

As this new method "stream_loose_object()" has many similarities with "write_loose_object()", we split up "write_loose_object()" into some steps:
1. Figuring out a path for the (temp) object file.
2. Creating the tempfile.
3. Setting up zlib and write header.
4. Write object data and handle errors.
5. Optionally, do someting after write, maybe force a loose object if
"mtime".

---
 object-file.c | 98 +++++++++++++++++++++++++++++++++------------------
 1 file changed, 63 insertions(+), 35 deletions(-)
object-file.c +63 -35
index 95fcd5435d..dd29e5372e 100644
@@ -1751,6 +1751,25 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,
 	algo->final_oid_fn(oid, &c);
 }
 
+/*
+ * Move the just written object with proper mtime into its final resting place.
+ */
+static int finalize_object_file_with_mtime(const char *tmpfile,
+					   const char *filename,
+					   time_t mtime,
+					   unsigned flags)
+{
+	struct utimbuf utb;
+
+	if (mtime) {
+		utb.actime = mtime;
+		utb.modtime = mtime;
+		if (utime(tmpfile, &utb) < 0 && !(flags & HASH_SILENT))
+			warning_errno(_("failed utime() on %s"), tmpfile);
+	}
+	return finalize_object_file(tmpfile, filename);
+}
+
 /*
  * Move the just written object into its final resting place.
  */
@@ -1836,7 +1855,8 @@ static inline int directory_size(const char *filename)
  * We want to avoid cross-directory filename renames, because those
  * can have problems on various filesystems (FAT, NFS, Coda).
  */
-static int create_tmpfile(struct strbuf *tmp, const char *filename)
+static int create_tmpfile(struct strbuf *tmp, const char *filename,
+			  unsigned flags)
 {
 	int fd, dirlen = directory_size(filename);
 
@@ -1844,7 +1864,9 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	strbuf_add(tmp, filename, dirlen);
 	strbuf_addstr(tmp, "tmp_obj_XXXXXX");
 	fd = git_mkstemp_mode(tmp->buf, 0444);
-	if (fd < 0 && dirlen && errno == ENOENT) {
+	do {
+		if (fd >= 0 || !dirlen || errno != ENOENT)
+			break;
 		/*
 		 * Make sure the directory exists; note that the contents
 		 * of the buffer are undefined after mkstemp returns an
@@ -1854,17 +1876,48 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 		strbuf_reset(tmp);
 		strbuf_add(tmp, filename, dirlen - 1);
 		if (mkdir(tmp->buf, 0777) && errno != EEXIST)
-			return -1;
+			break;
 		if (adjust_shared_perm(tmp->buf))
-			return -1;
+			break;
 
 		/* Try again */
 		strbuf_addstr(tmp, "/tmp_obj_XXXXXX");
 		fd = git_mkstemp_mode(tmp->buf, 0444);
+	} while (0);
+
+	if (fd < 0 && !(flags & HASH_SILENT)) {
+		if (errno == EACCES)
+			return error(_("insufficient permission for adding an "
+				       "object to repository database %s"),
+				     get_object_directory());
+		else
+			return error_errno(_("unable to create temporary file"));
 	}
+
 	return fd;
 }
 
+static void setup_stream_and_header(git_zstream *stream,
+				    unsigned char *compressed,
+				    unsigned long compressed_size,
+				    git_hash_ctx *c,
+				    char *hdr,
+				    int hdrlen)
+{
+	/* Set it up */
+	git_deflate_init(stream, zlib_compression_level);
+	stream->next_out = compressed;
+	stream->avail_out = compressed_size;
+	the_hash_algo->init_fn(c);
+
+	/* First header.. */
+	stream->next_in = (unsigned char *)hdr;
+	stream->avail_in = hdrlen;
+	while (git_deflate(stream, 0) == Z_OK)
+		; /* nothing */
+	the_hash_algo->update_fn(c, hdr, hdrlen);
+}
+
 static int write_loose_object(const struct object_id *oid, char *hdr,
 			      int hdrlen, const void *buf, unsigned long len,
 			      time_t mtime, unsigned flags)
@@ -1879,31 +1932,15 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 
 	loose_object_path(the_repository, &filename, oid);
 
-	fd = create_tmpfile(&tmp_file, filename.buf);
+	fd = create_tmpfile(&tmp_file, filename.buf, flags);
 	if (fd < 0) {
-		if (flags & HASH_SILENT)
-			ret = -1;
-		else if (errno == EACCES)
-			ret = error(_("insufficient permission for adding an "
-				      "object to repository database %s"),
-				    get_object_directory());
-		else
-			ret = error_errno(_("unable to create temporary file"));
+		ret = -1;
 		goto cleanup;
 	}
 
-	/* Set it up */
-	git_deflate_init(&stream, zlib_compression_level);
-	stream.next_out = compressed;
-	stream.avail_out = sizeof(compressed);
-	the_hash_algo->init_fn(&c);
-
-	/* First header.. */
-	stream.next_in = (unsigned char *)hdr;
-	stream.avail_in = hdrlen;
-	while (git_deflate(&stream, 0) == Z_OK)
-		; /* nothing */
-	the_hash_algo->update_fn(&c, hdr, hdrlen);
+	/* Set it up and write header */
+	setup_stream_and_header(&stream, compressed, sizeof(compressed),
+				&c, hdr, hdrlen);
 
 	/* Then the data itself.. */
 	stream.next_in = (void *)buf;
@@ -1932,16 +1969,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 
 	close_loose_object(fd);
 
-	if (mtime) {
-		struct utimbuf utb;
-		utb.actime = mtime;
-		utb.modtime = mtime;
-		if (utime(tmp_file.buf, &utb) < 0 &&
-		    !(flags & HASH_SILENT))
-			warning_errno(_("failed utime() on %s"), tmp_file.buf);
-	}
-
-	ret = finalize_object_file(tmp_file.buf, filename.buf);
+	ret = finalize_object_file_with_mtime(tmp_file.buf, filename.buf, mtime, flags);
 cleanup:
 	strbuf_release(&filename);
 	strbuf_release(&tmp_file);
Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
author
#162 lore raw

PATCH v6 4/6 object-file.c: make "write_object_file_flags()" to support read in stream

In reply to Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

This can be improved by feeding data to "stream_loose_object()" in a stream. The input stream is implemented as an interface.

When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in a new function called "stream_loose_object()".

In "write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

We will reuse "write_object_file_flags()" in "unpack_non_delta_entry()" to read the entire data contents in stream, so a new flag "HASH_STREAM" is added. When read in stream, we needn't prepare the "oid" before "write_loose_object()", only generate the header. "freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid".

---
 cache.h        |  1 +
 object-file.c  | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++
 object-store.h |  5 +++
 3 files changed, 98 insertions(+)
cache.h +1 -0
index cfba463aa9..6d68fd10a3 100644
@@ -898,6 +898,7 @@ int ie_modified(struct index_state *, const struct cache_entry *, struct stat *,
 #define HASH_FORMAT_CHECK 2
 #define HASH_RENORMALIZE  4
 #define HASH_SILENT 8
+#define HASH_STREAM 16
 int index_fd(struct index_state *istate, struct object_id *oid, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);
 int index_path(struct index_state *istate, struct object_id *oid, const char *path, struct stat *st, unsigned flags);
 
object-file.c +92 -0

104 lines not on this page · raw message

object-store.h +5 -0
index 952efb6a4b..4040e2c40a 100644
@@ -34,6 +34,11 @@ struct object_directory {
 	char *path;
 };
 
+struct input_stream {
+	const void *(*read)(const struct input_stream *, unsigned long *len);
+	void *data;
+};
+
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
 	struct object_directory *, 1, fspathhash, fspatheq)
 
Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
#163 lore raw

Re: [PATCH v6 4/6] object-file.c: make "write_object_file_flags()" to support read in stream

Am 17.12.21 um 12:26 schrieb Han Xin:

Quoted 57 lines

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

This can be improved by feeding data to "stream_loose_object()" in a stream. The input stream is implemented as an interface.

When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in a new function called "stream_loose_object()".

In "write_loose_object()", we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

We will reuse "write_object_file_flags()" in "unpack_non_delta_entry()" to read the entire data contents in stream, so a new flag "HASH_STREAM" is added. When read in stream, we needn't prepare the "oid" before "write_loose_object()", only generate the header. "freshen_packed_object()" or "freshen_loose_object()" will be called inside "stream_loose_object()" after obtaining the "oid".

---
 cache.h        |  1 +
 object-file.c  | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++
 object-store.h |  5 +++
 3 files changed, 98 insertions(+)
cache.h +1 -0
index cfba463aa9..6d68fd10a3 100644
@@ -898,6 +898,7 @@ int ie_modified(struct index_state *, const struct cache_entry *, struct stat *,
 #define HASH_FORMAT_CHECK 2
 #define HASH_RENORMALIZE  4
 #define HASH_SILENT 8
+#define HASH_STREAM 16
 int index_fd(struct index_state *istate, struct object_id *oid, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);
 int index_path(struct index_state *istate, struct object_id *oid, const char *path, struct stat *st, unsigned flags);
 
object-file.c +1 -0
index dd29e5372e..2ef1d4fb00 100644
@@ -1994,6 +1994,88 @@ static int freshen_packed_object(const struct object_id *oid)
 	return 1;
 }
 
+static int stream_loose_object(struct object_id *oid, char *hdr, int hdrlen,

object-file.c static int freshen_packed_object(const struct object_id *oid) · line 2007

Quoted 10 lines
object-file.c +10 -0
+			       const struct input_stream *in_stream,
+			       unsigned long len, time_t mtime, unsigned flags)
+{
+	int fd, ret, err = 0, flush = 0;
+	unsigned char compressed[4096];
+	git_zstream stream;
+	git_hash_ctx c;
+	struct object_id parano_oid;
+	static struct strbuf tmp_file = STRBUF_INIT;
+	static struct strbuf filename = STRBUF_INIT;

Note these static strbufs.

Quoted 58 lines
object-file.c +58 -0
+	int dirlen;
+
+	/* When oid is not determined, save tmp file to odb path. */
+	strbuf_addf(&filename, "%s/", get_object_directory());
+
+	fd = create_tmpfile(&tmp_file, filename.buf, flags);
+	if (fd < 0) {
+		err = -1;
+		goto cleanup;
+	}
+
+	/* Set it up and write header */
+	setup_stream_and_header(&stream, compressed, sizeof(compressed),
+				&c, hdr, hdrlen);
+
+	/* Then the data itself.. */
+	do {
+		unsigned char *in0 = stream.next_in;
+		if (!stream.avail_in) {
+			const void *in = in_stream->read(in_stream, &stream.avail_in);
+			stream.next_in = (void *)in;
+			in0 = (unsigned char *)in;
+			/* All data has been read. */
+			if (len + hdrlen == stream.total_in + stream.avail_in)
+				flush = Z_FINISH;
+		}
+		ret = git_deflate(&stream, flush);
+		the_hash_algo->update_fn(&c, in0, stream.next_in - in0);
+		if (write_buffer(fd, compressed, stream.next_out - compressed) < 0)
+			die(_("unable to write loose object file"));
+		stream.next_out = compressed;
+		stream.avail_out = sizeof(compressed);
+	} while (ret == Z_OK || ret == Z_BUF_ERROR);
+
+	if (ret != Z_STREAM_END)
+		die(_("unable to deflate new object streamingly (%d)"), ret);
+	ret = git_deflate_end_gently(&stream);
+	if (ret != Z_OK)
+		die(_("deflateEnd on object streamingly failed (%d)"), ret);
+	the_hash_algo->final_oid_fn(&parano_oid, &c);
+
+	close_loose_object(fd);
+
+	oidcpy(oid, &parano_oid);
+
+	if (freshen_packed_object(oid) || freshen_loose_object(oid)) {
+		unlink_or_warn(tmp_file.buf);
+		goto cleanup;
+	}
+
+	loose_object_path(the_repository, &filename, oid);
+
+	/* We finally know the object path, and create the missing dir. */
+	dirlen = directory_size(filename.buf);
+	if (dirlen) {
+		struct strbuf dir = STRBUF_INIT;
+		strbuf_add(&dir, filename.buf, dirlen - 1);
+

object-file.c static int freshen_packed_object(const struct object_id *oid) · line 2075

Quoted 10 lines
object-file.c +10 -0
+		if (mkdir_in_gitdir(dir.buf) < 0) {
+			err = -1;
+			goto cleanup;
+		}
+	}
+
+	err = finalize_object_file_with_mtime(tmp_file.buf, filename.buf, mtime, flags);
+cleanup:
+	strbuf_release(&tmp_file);
+	strbuf_release(&filename);

The static strbufs are released here. That combination is strange -- why keep the variable values between calls by making them static, but throw away the allocated buffers instead of reusing them?

Given that this function is only used for huge objects I think making the strbufs non-static and releasing them is the best choice here.

Quoted 9 lines
object-file.c +3 -0
+	return err;
+}
+
 int write_object_file_flags(const void *buf, unsigned long len,
 			    const char *type, struct object_id *oid,
 			    unsigned flags)
@@ -2001,6 +2083,16 @@ int write_object_file_flags(const void *buf, unsigned long len,
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen = sizeof(hdr);

object-file.c int write_object_file_flags(const void *buf, unsigned long len, · line 2094

Quoted 10 lines
object-file.c +9 -0
 
+	/* When streaming a large blob object (marked as HASH_STREAM),
+	 * we have no chance to run "write_object_file_prepare()" to
+	 * calculate the "oid" in advance.  Call "stream_loose_object()"
+	 * to write loose object in stream.
+	 */
+	if (flags & HASH_STREAM) {
+		hdrlen = generate_object_header(hdr, hdrlen, type, len);
+		return stream_loose_object(oid, hdr, hdrlen, buf, len, 0, flags);
+	}

So stream_loose_object() is called by passing the flag HASH_STREAM to write_object_file_flags() and passing a struct input_stream via its buf pointer. That's ... unconventional. Certainly scary. Why not export stream_loose_object() and call it directly? Demo patch below.

Quoted 10 lines
object-file.c +1 -0
+
 	/* Normally if we have it in the pack then we do not bother writing
 	 * it out into .git/objects/??/?{38} file.
 	 */
object-store.h +0 -0
index 952efb6a4b..4040e2c40a 100644
@@ -34,6 +34,11 @@ struct object_directory {
 	char *path;

object-store.h struct object_directory { · line 44

Quoted 10 lines
object-store.h +5 -0
 };
 
+struct input_stream {
+	const void *(*read)(const struct input_stream *, unsigned long *len);
+	void *data;
+};
+
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
 	struct object_directory *, 1, fspathhash, fspatheq)
 
builtin/unpack-objects.c +2 -4
index 42e1033d85..07d186bd20 100644
@@ -375,10 +375,8 @@ static void write_stream_blob(unsigned nr, unsigned long size)
 	data.zstream = &zstream;
 	git_inflate_init(&zstream);
 
-	if (write_object_file_flags(&in_stream, size,
-				    type_name(OBJ_BLOB),
-				    &obj_list[nr].oid,
-				    HASH_STREAM))
+	if (stream_loose_object(&in_stream, size, type_name(OBJ_BLOB), 0, 0,
+				&obj_list[nr].oid))
 		die(_("failed to write object in stream"));
 
 	if (zstream.total_out != size || data.status != Z_STREAM_END)
object-file.c +7 -13
index 2ef1d4fb00..0a6b65ab26 100644
@@ -1994,9 +1994,9 @@ static int freshen_packed_object(const struct object_id *oid)
 	return 1;
 }
 
-static int stream_loose_object(struct object_id *oid, char *hdr, int hdrlen,
-			       const struct input_stream *in_stream,
-			       unsigned long len, time_t mtime, unsigned flags)
+int stream_loose_object(struct input_stream *in_stream, unsigned long len,
+			const char *type, time_t mtime, unsigned flags,
+			struct object_id *oid)
 {
 	int fd, ret, err = 0, flush = 0;
 	unsigned char compressed[4096];
@@ -2006,6 +2006,10 @@ static int stream_loose_object(struct object_id *oid, char *hdr, int hdrlen,
 	static struct strbuf tmp_file = STRBUF_INIT;
 	static struct strbuf filename = STRBUF_INIT;
 	int dirlen;
+	char hdr[MAX_HEADER_LEN];
+	int hdrlen = sizeof(hdr);
+
+	hdrlen = generate_object_header(hdr, hdrlen, type, len);
 
 	/* When oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", get_object_directory());
@@ -2083,16 +2087,6 @@ int write_object_file_flags(const void *buf, unsigned long len,
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen = sizeof(hdr);
 
-	/* When streaming a large blob object (marked as HASH_STREAM),
-	 * we have no chance to run "write_object_file_prepare()" to
-	 * calculate the "oid" in advance.  Call "stream_loose_object()"
-	 * to write loose object in stream.
-	 */
-	if (flags & HASH_STREAM) {
-		hdrlen = generate_object_header(hdr, hdrlen, type, len);
-		return stream_loose_object(oid, hdr, hdrlen, buf, len, 0, flags);
-	}
-
 	/* Normally if we have it in the pack then we do not bother writing
 	 * it out into .git/objects/??/?{38} file.
 	 */
object-store.h +4 -0
index 4040e2c40a..786b6435b1 100644
@@ -237,6 +237,10 @@ static inline int write_object_file(const void *buf, unsigned long len,
 	return write_object_file_flags(buf, len, type, oid, 0);
 }
 
+int stream_loose_object(struct input_stream *in_stream, unsigned long len,
+			const char *type, time_t mtime, unsigned flags,
+			struct object_id *oid);
+
 int hash_object_file_literally(const void *buf, unsigned long len,
 			       const char *type, struct object_id *oid,
 			       unsigned flags);
author
#164 lore raw

PATCH v6 5/6 unpack-objects.c: add dry_run mode for get_data()

In reply to Han Xin

From: Han Xin <[email protected]>

In dry_run mode, "get_data()" is used to verify the inflation of data, and the returned buffer will not be used at all and will be freed immediately. Even in dry_run mode, it is dangerous to allocate a full-size buffer for a large blob object. Therefore, only allocate a low memory footprint when calling "get_data()" in dry_run mode.

---
 builtin/unpack-objects.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
builtin/unpack-objects.c +17 -6
index 4a9466295b..c4a17bdb44 100644
@@ -96,15 +96,21 @@ static void use(int bytes)
 	display_throughput(progress, consumed_bytes);
 }
 
-static void *get_data(unsigned long size)
+static void *get_data(unsigned long size, int dry_run)
 {
 	git_zstream stream;
-	void *buf = xmallocz(size);
+	unsigned long bufsize;
+	void *buf;
 
 	memset(&stream, 0, sizeof(stream));
+	if (dry_run && size > 8192)
+		bufsize = 8192;
+	else
+		bufsize = size;
+	buf = xmallocz(bufsize);
 
 	stream.next_out = buf;
-	stream.avail_out = size;
+	stream.avail_out = bufsize;
 	stream.next_in = fill(1);
 	stream.avail_in = len;
 	git_inflate_init(&stream);
@@ -124,6 +130,11 @@ static void *get_data(unsigned long size)
 		}
 		stream.next_in = fill(1);
 		stream.avail_in = len;
+		if (dry_run) {
+			/* reuse the buffer in dry_run mode */
+			stream.next_out = buf;
+			stream.avail_out = bufsize;
+		}
 	}
 	git_inflate_end(&stream);
 	return buf;
@@ -323,7 +334,7 @@ static void added_object(unsigned nr, enum object_type type,
 static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 				   unsigned nr)
 {
-	void *buf = get_data(size);
+	void *buf = get_data(size, dry_run);
 
 	if (!dry_run && buf)
 		write_object(nr, type, buf, size);
@@ -357,7 +368,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 	if (type == OBJ_REF_DELTA) {
 		oidread(&base_oid, fill(the_hash_algo->rawsz));
 		use(the_hash_algo->rawsz);
-		delta_data = get_data(delta_size);
+		delta_data = get_data(delta_size, dry_run);
 		if (dry_run || !delta_data) {
 			free(delta_data);
 			return;
@@ -396,7 +407,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 		if (base_offset <= 0 || base_offset >= obj_list[nr].offset)
 			die("offset value out of bound for delta base object");
 
-		delta_data = get_data(delta_size);
+		delta_data = get_data(delta_size, dry_run);
 		if (dry_run || !delta_data) {
 			free(delta_data);
 			return;
Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
#165 lore raw

Re: [PATCH v6 5/6] unpack-objects.c: add dry_run mode for get_data()

Am 17.12.21 um 12:26 schrieb Han Xin:

Quoted 7 lines

From: Han Xin <[email protected]>

In dry_run mode, "get_data()" is used to verify the inflation of data, and the returned buffer will not be used at all and will be freed immediately. Even in dry_run mode, it is dangerous to allocate a full-size buffer for a large blob object. Therefore, only allocate a low memory footprint when calling "get_data()" in dry_run mode.

Clever. Looks good to me.

For some reason I was expecting this patch to have some connection to one of the earlier ones (perhaps because get_data() was mentioned), but it is technically independent.

Quoted 75 lines
---
 builtin/unpack-objects.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
builtin/unpack-objects.c +17 -6
index 4a9466295b..c4a17bdb44 100644
@@ -96,15 +96,21 @@ static void use(int bytes)
 	display_throughput(progress, consumed_bytes);
 }
 
-static void *get_data(unsigned long size)
+static void *get_data(unsigned long size, int dry_run)
 {
 	git_zstream stream;
-	void *buf = xmallocz(size);
+	unsigned long bufsize;
+	void *buf;
 
 	memset(&stream, 0, sizeof(stream));
+	if (dry_run && size > 8192)
+		bufsize = 8192;
+	else
+		bufsize = size;
+	buf = xmallocz(bufsize);
 
 	stream.next_out = buf;
-	stream.avail_out = size;
+	stream.avail_out = bufsize;
 	stream.next_in = fill(1);
 	stream.avail_in = len;
 	git_inflate_init(&stream);
@@ -124,6 +130,11 @@ static void *get_data(unsigned long size)
 		}
 		stream.next_in = fill(1);
 		stream.avail_in = len;
+		if (dry_run) {
+			/* reuse the buffer in dry_run mode */
+			stream.next_out = buf;
+			stream.avail_out = bufsize;
+		}
 	}
 	git_inflate_end(&stream);
 	return buf;
@@ -323,7 +334,7 @@ static void added_object(unsigned nr, enum object_type type,
 static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 				   unsigned nr)
 {
-	void *buf = get_data(size);
+	void *buf = get_data(size, dry_run);
 
 	if (!dry_run && buf)
 		write_object(nr, type, buf, size);
@@ -357,7 +368,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 	if (type == OBJ_REF_DELTA) {
 		oidread(&base_oid, fill(the_hash_algo->rawsz));
 		use(the_hash_algo->rawsz);
-		delta_data = get_data(delta_size);
+		delta_data = get_data(delta_size, dry_run);
 		if (dry_run || !delta_data) {
 			free(delta_data);
 			return;
@@ -396,7 +407,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 		if (base_offset <= 0 || base_offset >= obj_list[nr].offset)
 			die("offset value out of bound for delta base object");
 
-		delta_data = get_data(delta_size);
+		delta_data = get_data(delta_size, dry_run);
 		if (dry_run || !delta_data) {
 			free(delta_data);
 			return;
author
#166 lore raw

PATCH v6 6/6 unpack-objects: unpack_non_delta_entry() read data in a stream

In reply to Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".

However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "core.BigFileStreamingThreshold" in zstream. See the following benchmarks:

hyperfine \
--setup \
'if ! test -d scalar.git; then git clone --bare https://github.com/microsoft/scalar.git; cp scalar.git/objects/pack/*.pack small.pack; fi' \
--prepare 'rm -rf dest.git && git init --bare dest.git'

Summary
'./git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'origin/master'
1.01 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~1' 1.01 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~0' 1.03 ± 0.10 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'origin/master' 1.02 ± 0.07 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~0' 1.10 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~1'

---
 Documentation/config/core.txt       | 11 ++++
 builtin/unpack-objects.c            | 73 +++++++++++++++++++++++-
 cache.h                             |  1 +
 config.c                            |  5 ++
 environment.c                       |  1 +
 t/t5590-unpack-non-delta-objects.sh | 87 +++++++++++++++++++++++++++++
 6 files changed, 177 insertions(+), 1 deletion(-)
 create mode 100755 t/t5590-unpack-non-delta-objects.sh
Documentation/config/core.txt +11 -0
index c04f62a54a..601b7a2418 100644
@@ -424,6 +424,17 @@ be delta compressed, but larger binary media files won't be.
 +
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
+core.bigFileStreamingThreshold::
+	Files larger than this will be streamed out to a temporary
+	object file while being hashed, which will when be renamed
+	in-place to a loose object, particularly if the
+	`core.bigFileThreshold' setting dictates that they're always
+	written out as loose objects.
++
+Default is 128 MiB on all platforms.
++
+Common unit suffixes of 'k', 'm', or 'g' are supported.
+
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
 	describe paths that are not meant to be tracked, in addition
builtin/unpack-objects.c +72 -1

83 lines not on this page · raw message

cache.h +1 -0
index 6d68fd10a3..976f9cf656 100644
@@ -975,6 +975,7 @@ extern size_t packed_git_window_size;
 extern size_t packed_git_limit;
 extern size_t delta_base_cache_limit;
 extern unsigned long big_file_threshold;
+extern unsigned long big_file_streaming_threshold;
 extern unsigned long pack_size_limit_cfg;
 
 /*
config.c +5 -0
index c5873f3a70..7b122a142a 100644
@@ -1408,6 +1408,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (!strcmp(var, "core.bigfilestreamingthreshold")) {
+		big_file_streaming_threshold = git_config_ulong(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "core.packedgitlimit")) {
 		packed_git_limit = git_config_ulong(var, value);
 		return 0;
environment.c +1 -0
index 0d06a31024..04bba593de 100644
@@ -47,6 +47,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
 size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
 size_t delta_base_cache_limit = 96 * 1024 * 1024;
 unsigned long big_file_threshold = 512 * 1024 * 1024;
+unsigned long big_file_streaming_threshold = 128 * 1024 * 1024;
 int pager_use_color = 1;
 const char *editor_program;
 const char *askpass_program;
t/t5590-unpack-non-delta-objects.sh new file +87 -0

87 lines not on this page · raw message

Signature · 1 line
2.34.1.52.gfcc2252aea.agit.6.5.6
author
#167 lore raw

PATCH v5 1/6 object-file: refactor write_loose_object() to support read from stream

In reply to Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

This can be improved by feeding data to "write_loose_object()" in a stream. The input stream is implemented as an interface.

In the first step, we add a new flag called "HASH_STREAM" and make a simple implementation, feeding the entire buffer in the stream to "write_loose_object()" as a refactor.

---
 cache.h        | 1 +
 object-file.c  | 7 ++++++-
 object-store.h | 5 +++++
 3 files changed, 12 insertions(+), 1 deletion(-)
cache.h +1 -0
index eba12487b9..51bd435dea 100644
@@ -888,6 +888,7 @@ int ie_modified(struct index_state *, const struct cache_entry *, struct stat *,
 #define HASH_FORMAT_CHECK 2
 #define HASH_RENORMALIZE  4
 #define HASH_SILENT 8
+#define HASH_STREAM 16
 int index_fd(struct index_state *istate, struct object_id *oid, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);
 int index_path(struct index_state *istate, struct object_id *oid, const char *path, struct stat *st, unsigned flags);
 
object-file.c +6 -1
index eb972cdccd..06375a90d6 100644
@@ -1898,7 +1898,12 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	the_hash_algo->update_fn(&c, hdr, hdrlen);
 
 	/* Then the data itself.. */
-	stream.next_in = (void *)buf;
+	if (flags & HASH_STREAM) {
+		struct input_stream *in_stream = (struct input_stream *)buf;
+		stream.next_in = (void *)in_stream->read(in_stream, &len);
+	} else {
+		stream.next_in = (void *)buf;
+	}
 	stream.avail_in = len;
 	do {
 		unsigned char *in0 = stream.next_in;
object-store.h +5 -0
index 952efb6a4b..ccc1fc9c1a 100644
@@ -34,6 +34,11 @@ struct object_directory {
 	char *path;
 };
 
+struct input_stream {
+	const void *(*read)(struct input_stream *, unsigned long *len);
+	void *data;
+};
+
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
 	struct object_directory *, 1, fspathhash, fspatheq)
 
Signature · 1 line
2.34.0
author
#168 lore raw

PATCH v5 2/6 object-file.c: handle undetermined oid in write_loose_object()

In reply to Han Xin

From: Han Xin <[email protected]>

When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".

In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

The promise that "oid" is constant in "write_loose_object()" has been removed because it will be filled after reading all stream data.

---
 object-file.c | 48 +++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 39 insertions(+), 9 deletions(-)
object-file.c +39 -9
index 06375a90d6..41099b137f 100644
@@ -1860,11 +1860,11 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	return fd;
 }
 
-static int write_loose_object(const struct object_id *oid, char *hdr,
+static int write_loose_object(struct object_id *oid, char *hdr,
 			      int hdrlen, const void *buf, unsigned long len,
 			      time_t mtime, unsigned flags)
 {
-	int fd, ret;
+	int fd, ret, err = 0;
 	unsigned char compressed[4096];
 	git_zstream stream;
 	git_hash_ctx c;
@@ -1872,16 +1872,21 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	static struct strbuf tmp_file = STRBUF_INIT;
 	static struct strbuf filename = STRBUF_INIT;
 
-	loose_object_path(the_repository, &filename, oid);
+	if (flags & HASH_STREAM)
+		/* When oid is not determined, save tmp file to odb path. */
+		strbuf_addf(&filename, "%s/", get_object_directory());
+	else
+		loose_object_path(the_repository, &filename, oid);
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
 		if (flags & HASH_SILENT)
-			return -1;
+			err = -1;
 		else if (errno == EACCES)
-			return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
+			err = error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
 		else
-			return error_errno(_("unable to create temporary file"));
+			err = error_errno(_("unable to create temporary file"));
+		goto cleanup;
 	}
 
 	/* Set it up */
@@ -1923,12 +1928,34 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
 		    ret);
 	the_hash_algo->final_oid_fn(&parano_oid, &c);
-	if (!oideq(oid, &parano_oid))
+	if (!(flags & HASH_STREAM) && !oideq(oid, &parano_oid))
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));
 
 	close_loose_object(fd);
 
+	if (flags & HASH_STREAM) {
+		int dirlen;
+
+		oidcpy((struct object_id *)oid, &parano_oid);
+		loose_object_path(the_repository, &filename, oid);
+
+		/* We finally know the object path, and create the missing dir. */
+		dirlen = directory_size(filename.buf);
+		if (dirlen) {
+			struct strbuf dir = STRBUF_INIT;
+			strbuf_add(&dir, filename.buf, dirlen - 1);
+			if (mkdir(dir.buf, 0777) && errno != EEXIST)
+				err = -1;
+			else if (adjust_shared_perm(dir.buf))
+				err = -1;
+			else
+				strbuf_release(&dir);
+			if (err < 0)
+				goto cleanup;
+		}
+	}
+
 	if (mtime) {
 		struct utimbuf utb;
 		utb.actime = mtime;
@@ -1938,7 +1965,10 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 			warning_errno(_("failed utime() on %s"), tmp_file.buf);
 	}
 
-	return finalize_object_file(tmp_file.buf, filename.buf);
+	err = finalize_object_file(tmp_file.buf, filename.buf);
+cleanup:
+	strbuf_release(&filename);
+	return err;
 }
 
 static int freshen_loose_object(const struct object_id *oid)
@@ -2015,7 +2045,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
 	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
-	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
+	ret = write_loose_object((struct object_id*) oid, hdr, hdrlen, buf, len, mtime, 0);
 	free(buf);
 
 	return ret;
Signature · 1 line
2.34.0
#169 lore raw

Re: [PATCH v5 2/6] object-file.c: handle undetermined oid in write_loose_object()

Quoted 68 lines · Han Xin

From: Han Xin <[email protected]>

When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".

In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

The promise that "oid" is constant in "write_loose_object()" has been removed because it will be filled after reading all stream data.

---
 object-file.c | 48 +++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 39 insertions(+), 9 deletions(-)
object-file.c +11 -6
index 06375a90d6..41099b137f 100644
@@ -1860,11 +1860,11 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	return fd;
 }
 
-static int write_loose_object(const struct object_id *oid, char *hdr,
+static int write_loose_object(struct object_id *oid, char *hdr,
 			      int hdrlen, const void *buf, unsigned long len,
 			      time_t mtime, unsigned flags)
 {
-	int fd, ret;
+	int fd, ret, err = 0;
 	unsigned char compressed[4096];
 	git_zstream stream;
 	git_hash_ctx c;
@@ -1872,16 +1872,21 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	static struct strbuf tmp_file = STRBUF_INIT;
 	static struct strbuf filename = STRBUF_INIT;
 
-	loose_object_path(the_repository, &filename, oid);
+	if (flags & HASH_STREAM)
+		/* When oid is not determined, save tmp file to odb path. */
+		strbuf_addf(&filename, "%s/", get_object_directory());
+	else
+		loose_object_path(the_repository, &filename, oid);
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
 		if (flags & HASH_SILENT)
-			return -1;
+			err = -1;
 		else if (errno == EACCES)
-			return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
+			err = error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
 		else
-			return error_errno(_("unable to create temporary file"));
+			err = error_errno(_("unable to create temporary file"));
+		goto cleanup;
 	}
 
 	/* Set it up */

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1933

Quoted 7 lines
object-file.c +1 -1
@@ -1923,12 +1928,34 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
 		    ret);
 	the_hash_algo->final_oid_fn(&parano_oid, &c);
-	if (!oideq(oid, &parano_oid))
+	if (!(flags & HASH_STREAM) && !oideq(oid, &parano_oid))
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));

Here we don't have a meaningful "const" OID anymore, but still if we die we use the "oid".

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1940

Quoted 6 lines
object-file.c +4 -0
 	close_loose_object(fd);
 
+	if (flags & HASH_STREAM) {
+		int dirlen;
+
+		oidcpy((struct object_id *)oid, &parano_oid);

This cast isn't needed anymore now that you stripped the "const" off, but more on that later...

Quoted 5 lines
object-file.c +5 -0
+		loose_object_path(the_repository, &filename, oid);
+
+		/* We finally know the object path, and create the missing dir. */
+		dirlen = directory_size(filename.buf);
+		if (dirlen) {

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1955

Quoted 10 lines
object-file.c +10 -0
+			struct strbuf dir = STRBUF_INIT;
+			strbuf_add(&dir, filename.buf, dirlen - 1);
+			if (mkdir(dir.buf, 0777) && errno != EEXIST)
+				err = -1;
+			else if (adjust_shared_perm(dir.buf))
+				err = -1;
+			else
+				strbuf_release(&dir);
+			if (err < 0)
+				goto cleanup;

Can't we use one of the existing utility functions for this? Testing locally I could replace this with:

	diff --git a/object-file.c b/object-file.c
	index 7c93db11b2d..05e1fae893d 100644
	--- a/object-file.c
	+++ b/object-file.c
	@@ -1952,14 +1952,11 @@ static int write_loose_object(struct object_id *oid, char *hdr,
	 		if (dirlen) {
	 			struct strbuf dir = STRBUF_INIT;
	 			strbuf_add(&dir, filename.buf, dirlen - 1);
	-			if (mkdir(dir.buf, 0777) && errno != EEXIST)
	+			
	+			if (mkdir_in_gitdir(dir.buf) < 0) {
	 				err = -1;
	-			else if (adjust_shared_perm(dir.buf))
	-				err = -1;
	-			else
	-				strbuf_release(&dir);
	-			if (err < 0)
	 				goto cleanup;
	+			}
	 		}
	 	}

And your tests still pass. Maybe they have a blind spot, or maybe we can just use the existing function.

Quoted 7 lines
object-file.c +3 -0
+		}
+	}
+
 	if (mtime) {
 		struct utimbuf utb;
 		utb.actime = mtime;

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1972

Quoted 9 lines
object-file.c +4 -1
@@ -1938,7 +1965,10 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 			warning_errno(_("failed utime() on %s"), tmp_file.buf);
 	}
 
-	return finalize_object_file(tmp_file.buf, filename.buf);
+	err = finalize_object_file(tmp_file.buf, filename.buf);
+cleanup:
+	strbuf_release(&filename);
+	return err;
 }

Reading this series is an odd mixture of of things that would really be much easier to understand if they were combined, e.g. 1/6 adding APIs that aren't used by anything, but then adding one codepath (also unused), that we then use later. Could just add it at the same time as the use and the patch would be easier to read....

...and then this, which *is* something that could be split up into an earlier cleanup step, i.e. the strbuf leak here exists before this series, fixing it is good, but splitting that up into its own patch would make this diff smaller & the actual behavior changes easier to reason about.

object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2051

Quoted 10 lines
object-file.c +1 -1
 static int freshen_loose_object(const struct object_id *oid)
@@ -2015,7 +2045,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
 	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
-	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
+	ret = write_loose_object((struct object_id*) oid, hdr, hdrlen, buf, len, mtime, 0);
 	free(buf);
 
 	return ret;

...on the "more on that later", here we're casting the "oid" from const for a function that's never going to be involved in the streaming codepath.

I know I suggested the HASH_STREAM flag, but what I was really going for was "let's share more of the code?", looking at this v5 (which is already much better than v4) I think a better approach is to split up write_loose_object().

I.e. it already calls close_loose_object() and finalize_object_file() to do some of its work, but around that we have:

1. Figuring out a path for the (temp) object file
2. Creating the tempfile
3. Setting up zlib
4. Once zlib is set up inspect its state, die with a message about oid_to_hex(oid) if we failed
5. Optionally, do HASH_STREAM stuff
Maybe force a loose object if "mtime".

I think if that's split up so that each of those is its own little function what's now write_loose_object() can call those in sequence, and a new stream_loose_object() can just do #1 differentl, followed by the same #2 and #4, but do #4 differently etc.

You'll still be able to re-use the write_object_file_prepare() etc. logic.

As an example your 5/6 copy/pastes the xsnprintf() formatting of the object header. It's just one line, but it's also code that's very central to git, so I think instead of just copy/pasting it a prep step of factoring it out would make sense, and that would be a prep cleanup that would help later readability. E.g.:

	diff --git a/object-file.c b/object-file.c
	index eac67f6f5f9..a7dcbd929e9 100644
	--- a/object-file.c
	+++ b/object-file.c
	@@ -1009,6 +1009,13 @@ void *xmmap(void *start, size_t length,
	 	return ret;
	 }
	 
	+static int generate_object_header(char *buf, int bufsz, const char *type_name,
	+				  unsigned long size)
	+{
	+	return xsnprintf(buf, bufsz, "%s %"PRIuMAX , type_name,
	+			 (uintmax_t)size) + 1;
	+}
	+
	 /*
	  * With an in-core object data in "map", rehash it to make sure the
	  * object name actually matches "oid" to detect object corruption.
	@@ -1037,7 +1044,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,
	 		return -1;
	 
	 	/* Generate the header */
	-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(obj_type), (uintmax_t)size) + 1;
	+	hdrlen = generate_object_header(hdr, sizeof(hdr), type_name(obj_type), size);
	 
	 	/* Sha1.. */
	 	r->hash_algo->init_fn(&c);
	@@ -1737,7 +1744,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo,
	 	git_hash_ctx c;
	 
	 	/* Generate the header */
	-	*hdrlen = xsnprintf(hdr, *hdrlen, "%s %"PRIuMAX , type, (uintmax_t)len)+1;
	+	*hdrlen = generate_object_header(hdr, *hdrlen, type, len);
	 
	 	/* Sha1.. */
	 	algo->init_fn(&c);
	@@ -2009,7 +2016,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
	 	buf = read_object(the_repository, oid, &type, &len);
	 	if (!buf)
	 		return error(_("cannot read object for %s"), oid_to_hex(oid));
	-	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
	+	hdrlen = generate_object_header(hdr, sizeof(hdr), type_name(type), len);
	 	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
	 	free(buf);

Then in your change on top you just call that generate_object_header(), or better yet your amended write_object_file_flags() can just call a similarly amended write_object_file_prepare() directly.

author
#170 lore raw

PATCH v5 3/6 object-file.c: read stream in a loop in write_loose_object()

In reply to Han Xin

From: Han Xin <[email protected]>

In order to prepare the stream version of "write_loose_object()", read the input stream in a loop in "write_loose_object()", so that we can feed the contents of large blob object to "write_loose_object()" using a small fixed buffer.

---
 object-file.c | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)
object-file.c +15 -8
index 41099b137f..455ab3c06e 100644
@@ -1864,7 +1864,7 @@ static int write_loose_object(struct object_id *oid, char *hdr,
 			      int hdrlen, const void *buf, unsigned long len,
 			      time_t mtime, unsigned flags)
 {
-	int fd, ret, err = 0;
+	int fd, ret, err = 0, flush = 0;
 	unsigned char compressed[4096];
 	git_zstream stream;
 	git_hash_ctx c;
@@ -1903,22 +1903,29 @@ static int write_loose_object(struct object_id *oid, char *hdr,
 	the_hash_algo->update_fn(&c, hdr, hdrlen);
 
 	/* Then the data itself.. */
-	if (flags & HASH_STREAM) {
-		struct input_stream *in_stream = (struct input_stream *)buf;
-		stream.next_in = (void *)in_stream->read(in_stream, &len);
-	} else {
+	if (!(flags & HASH_STREAM)) {
 		stream.next_in = (void *)buf;
+		stream.avail_in = len;
+		flush = Z_FINISH;
 	}
-	stream.avail_in = len;
 	do {
 		unsigned char *in0 = stream.next_in;
-		ret = git_deflate(&stream, Z_FINISH);
+		if (flags & HASH_STREAM && !stream.avail_in) {
+			struct input_stream *in_stream = (struct input_stream *)buf;
+			const void *in = in_stream->read(in_stream, &stream.avail_in);
+			stream.next_in = (void *)in;
+			in0 = (unsigned char *)in;
+			/* All data has been read. */
+			if (len + hdrlen == stream.total_in + stream.avail_in)
+				flush = Z_FINISH;
+		}
+		ret = git_deflate(&stream, flush);
 		the_hash_algo->update_fn(&c, in0, stream.next_in - in0);
 		if (write_buffer(fd, compressed, stream.next_out - compressed) < 0)
 			die(_("unable to write loose object file"));
 		stream.next_out = compressed;
 		stream.avail_out = sizeof(compressed);
-	} while (ret == Z_OK);
+	} while (ret == Z_OK || ret == Z_BUF_ERROR);
 
 	if (ret != Z_STREAM_END)
 		die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
Signature · 1 line
2.34.0
author
#171 lore raw

PATCH v5 4/6 unpack-objects.c: add dry_run mode for get_data()

In reply to Han Xin

From: Han Xin <[email protected]>

In dry_run mode, "get_data()" is used to verify the inflation of data, and the returned buffer will not be used at all and will be freed immediately. Even in dry_run mode, it is dangerous to allocate a full-size buffer for a large blob object. Therefore, only allocate a low memory footprint when calling "get_data()" in dry_run mode.

---
 builtin/unpack-objects.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
builtin/unpack-objects.c +12 -6
index 4a9466295b..d878e2f8b4 100644
@@ -96,15 +96,16 @@ static void use(int bytes)
 	display_throughput(progress, consumed_bytes);
 }
 
-static void *get_data(unsigned long size)
+static void *get_data(unsigned long size, int dry_run)
 {
 	git_zstream stream;
-	void *buf = xmallocz(size);
+	unsigned long bufsize = dry_run ? 8192 : size;
+	void *buf = xmallocz(bufsize);
 
 	memset(&stream, 0, sizeof(stream));
 
 	stream.next_out = buf;
-	stream.avail_out = size;
+	stream.avail_out = bufsize;
 	stream.next_in = fill(1);
 	stream.avail_in = len;
 	git_inflate_init(&stream);
@@ -124,6 +125,11 @@ static void *get_data(unsigned long size)
 		}
 		stream.next_in = fill(1);
 		stream.avail_in = len;
+		if (dry_run) {
+			/* reuse the buffer in dry_run mode */
+			stream.next_out = buf;
+			stream.avail_out = bufsize;
+		}
 	}
 	git_inflate_end(&stream);
 	return buf;
@@ -323,7 +329,7 @@ static void added_object(unsigned nr, enum object_type type,
 static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 				   unsigned nr)
 {
-	void *buf = get_data(size);
+	void *buf = get_data(size, dry_run);
 
 	if (!dry_run && buf)
 		write_object(nr, type, buf, size);
@@ -357,7 +363,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 	if (type == OBJ_REF_DELTA) {
 		oidread(&base_oid, fill(the_hash_algo->rawsz));
 		use(the_hash_algo->rawsz);
-		delta_data = get_data(delta_size);
+		delta_data = get_data(delta_size, dry_run);
 		if (dry_run || !delta_data) {
 			free(delta_data);
 			return;
@@ -396,7 +402,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 		if (base_offset <= 0 || base_offset >= obj_list[nr].offset)
 			die("offset value out of bound for delta base object");
 
-		delta_data = get_data(delta_size);
+		delta_data = get_data(delta_size, dry_run);
 		if (dry_run || !delta_data) {
 			free(delta_data);
 			return;
Signature · 1 line
2.34.0
author
#172 lore raw

PATCH v5 5/6 object-file.c: make "write_object_file_flags()" to support "HASH_STREAM"

In reply to Han Xin

From: Han Xin <[email protected]>

We will use "write_object_file_flags()" in "unpack_non_delta_entry()" to read the entire data contents in stream. When read in stream, we needn't prepare "oid" before "write_loose_object()", only generate the header.

---
 object-file.c | 5 +++++
 1 file changed, 5 insertions(+)
object-file.c +5 -0
index 455ab3c06e..906590dae5 100644
@@ -2002,6 +2002,11 @@ int write_object_file_flags(const void *buf, unsigned long len,
 {
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen = sizeof(hdr);
+	if (flags & HASH_STREAM) {
+		/* Generate the header */
+		hdrlen = xsnprintf(hdr, hdrlen, "%s %"PRIuMAX , type, (uintmax_t)len)+1;
+		return write_loose_object(oid, hdr, hdrlen, buf, len, 0, flags);
+	}
 
 	/* Normally if we have it in the pack then we do not bother writing
 	 * it out into .git/objects/??/?{38} file.
Signature · 1 line
2.34.0
author
#173 lore raw

PATCH v5 6/6 unpack-objects: unpack_non_delta_entry() read data in a stream

In reply to Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".

However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "core.BigFileStreamingThreshold" in zstream. See the following benchmarks:

hyperfine \
--setup \
'if ! test -d scalar.git; then git clone --bare https://github.com/microsoft/scalar.git; cp scalar.git/objects/pack/*.pack small.pack; fi' \
--prepare 'rm -rf dest.git && git init --bare dest.git'

Summary
'./git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'origin/master'
1.01 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~1' 1.01 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~0' 1.03 ± 0.10 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'origin/master' 1.02 ± 0.07 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~0' 1.10 ± 0.04 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~1'

---
 Documentation/config/core.txt       | 11 +++++
 builtin/unpack-objects.c            | 70 ++++++++++++++++++++++++++++-
 cache.h                             |  1 +
 config.c                            |  5 +++
 environment.c                       |  1 +
 t/t5590-unpack-non-delta-objects.sh | 70 +++++++++++++++++++++++++++++
 6 files changed, 157 insertions(+), 1 deletion(-)
 create mode 100755 t/t5590-unpack-non-delta-objects.sh
Documentation/config/core.txt +11 -0
index c04f62a54a..601b7a2418 100644
@@ -424,6 +424,17 @@ be delta compressed, but larger binary media files won't be.
 +
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
+core.bigFileStreamingThreshold::
+	Files larger than this will be streamed out to a temporary
+	object file while being hashed, which will when be renamed
+	in-place to a loose object, particularly if the
+	`core.bigFileThreshold' setting dictates that they're always
+	written out as loose objects.
++
+Default is 128 MiB on all platforms.
++
+Common unit suffixes of 'k', 'm', or 'g' are supported.
+
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
 	describe paths that are not meant to be tracked, in addition
builtin/unpack-objects.c +69 -1

80 lines not on this page · raw message

cache.h +1 -0
index 51bd435dea..78548cd67a 100644
@@ -965,6 +965,7 @@ extern size_t packed_git_window_size;
 extern size_t packed_git_limit;
 extern size_t delta_base_cache_limit;
 extern unsigned long big_file_threshold;
+extern unsigned long big_file_streaming_threshold;
 extern unsigned long pack_size_limit_cfg;
 
 /*
config.c +5 -0
index c5873f3a70..7b122a142a 100644
@@ -1408,6 +1408,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (!strcmp(var, "core.bigfilestreamingthreshold")) {
+		big_file_streaming_threshold = git_config_ulong(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "core.packedgitlimit")) {
 		packed_git_limit = git_config_ulong(var, value);
 		return 0;
environment.c +1 -0
index 9da7f3c1a1..4fcc3de741 100644
@@ -46,6 +46,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
 size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
 size_t delta_base_cache_limit = 96 * 1024 * 1024;
 unsigned long big_file_threshold = 512 * 1024 * 1024;
+unsigned long big_file_streaming_threshold = 128 * 1024 * 1024;
 int pager_use_color = 1;
 const char *editor_program;
 const char *askpass_program;
t/t5590-unpack-non-delta-objects.sh new file +70 -0

70 lines not on this page · raw message

Signature · 1 line
2.34.0
#174 lore raw

Re: [PATCH v5 6/6] unpack-objects: unpack_non_delta_entry() read data in a stream

builtin/unpack-objects.c static void added_object(unsigned nr, enum object_type type, · line 373

Quoted 3 lines · Han Xin

From: Han Xin <[email protected]>
[...]

builtin/unpack-objects.c +1 -0
+	if ((ret = write_object_file_flags(&in_stream, size, type_name(OBJ_BLOB) ,&obj_list[nr].oid, HASH_STREAM)))

There's some odd code formatting here, i.e.. ") ,&" not "), &". Could also use line-wrapping at 79 characters.

author
#175 lore raw

PATCH v4 1/5 object-file: refactor write_loose_object() to read buffer from stream

In reply to Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

This can be improved by feeding data to "write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.

---
 object-file.c  | 53 ++++++++++++++++++++++++++++++++++++++++++++++----
 object-store.h |  6 ++++++
 2 files changed, 55 insertions(+), 4 deletions(-)
object-file.c +49 -4
index eb972cdccd..82656f7428 100644
@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	return fd;
 }
 
+struct simple_input_stream_data {
+	const void *buf;
+	unsigned long len;
+};
+
+static const void *feed_simple_input_stream(struct input_stream *in_stream, unsigned long *len)
+{
+	struct simple_input_stream_data *data = in_stream->data;
+
+	if (data->len == 0) {
+		*len = 0;
+		return NULL;
+	}
+	*len = data->len;
+	data->len = 0;
+	return data->buf;
+}
+
 static int write_loose_object(const struct object_id *oid, char *hdr,
-			      int hdrlen, const void *buf, unsigned long len,
+			      int hdrlen, struct input_stream *in_stream,
 			      time_t mtime, unsigned flags)
 {
 	int fd, ret;
@@ -1871,6 +1889,8 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	struct object_id parano_oid;
 	static struct strbuf tmp_file = STRBUF_INIT;
 	static struct strbuf filename = STRBUF_INIT;
+	const void *buf;
+	unsigned long len;
 
 	loose_object_path(the_repository, &filename, oid);
 
@@ -1898,6 +1918,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	the_hash_algo->update_fn(&c, hdr, hdrlen);
 
 	/* Then the data itself.. */
+	buf = in_stream->read(in_stream, &len);
 	stream.next_in = (void *)buf;
 	stream.avail_in = len;
 	do {
@@ -1960,6 +1981,14 @@ int write_object_file_flags(const void *buf, unsigned long len,
 {
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen = sizeof(hdr);
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = (void *)&(struct simple_input_stream_data) {
+			.buf = buf,
+			.len = len,
+		},
+		.size = len,
+	};
 
 	/* Normally if we have it in the pack then we do not bother writing
 	 * it out into .git/objects/??/?{38} file.
@@ -1968,7 +1997,7 @@ int write_object_file_flags(const void *buf, unsigned long len,
 				  &hdrlen);
 	if (freshen_packed_object(oid) || freshen_loose_object(oid))
 		return 0;
-	return write_loose_object(oid, hdr, hdrlen, buf, len, 0, flags);
+	return write_loose_object(oid, hdr, hdrlen, &in_stream, 0, flags);
 }
 
 int hash_object_file_literally(const void *buf, unsigned long len,
@@ -1977,6 +2006,14 @@ int hash_object_file_literally(const void *buf, unsigned long len,
 {
 	char *header;
 	int hdrlen, status = 0;
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = (void *)&(struct simple_input_stream_data) {
+			.buf = buf,
+			.len = len,
+		},
+		.size = len,
+	};
 
 	/* type string, SP, %lu of the length plus NUL must fit this */
 	hdrlen = strlen(type) + MAX_HEADER_LEN;
@@ -1988,7 +2025,7 @@ int hash_object_file_literally(const void *buf, unsigned long len,
 		goto cleanup;
 	if (freshen_packed_object(oid) || freshen_loose_object(oid))
 		goto cleanup;
-	status = write_loose_object(oid, header, hdrlen, buf, len, 0, 0);
+	status = write_loose_object(oid, header, hdrlen, &in_stream, 0, 0);
 
 cleanup:
 	free(header);
@@ -2003,14 +2040,22 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen;
 	int ret;
+	struct simple_input_stream_data data;
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = &data,
+	};
 
 	if (has_loose_object(oid))
 		return 0;
 	buf = read_object(the_repository, oid, &type, &len);
+	in_stream.size = len;
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
+	data.buf = buf;
+	data.len = len;
 	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
-	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
+	ret = write_loose_object(oid, hdr, hdrlen, &in_stream, mtime, 0);
 	free(buf);
 
 	return ret;
object-store.h +6 -0
index 952efb6a4b..a84d891d60 100644
@@ -34,6 +34,12 @@ struct object_directory {
 	char *path;
 };
 
+struct input_stream {
+	const void *(*read)(struct input_stream *, unsigned long *len);
+	void *data;
+	size_t size;
+};
+
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
 	struct object_directory *, 1, fspathhash, fspatheq)
 
Signature · 1 line
2.34.0
#176 lore raw

Re: [PATCH v4 1/5] object-file: refactor write_loose_object() to read buffer from stream

Quoted 23 lines · Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

This can be improved by feeding data to "write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.

---
 object-file.c  | 53 ++++++++++++++++++++++++++++++++++++++++++++++----
 object-store.h |  6 ++++++
 2 files changed, 55 insertions(+), 4 deletions(-)
object-file.c +0 -0
index eb972cdccd..82656f7428 100644

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1866

Quoted 7 lines
object-file.c +4 -0
@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	return fd;
 }
 
+struct simple_input_stream_data {
+	const void *buf;
+	unsigned long len;
+};

I see why you picked "const void *buf" here, over say const char *, it's what "struct input_stream" uses.

But why not use size_t for the length, as input_stream does?

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1872

Quoted 5 lines
object-file.c +5 -0
+static const void *feed_simple_input_stream(struct input_stream *in_stream, unsigned long *len)
+{
+	struct simple_input_stream_data *data = in_stream->data;
+
+	if (data->len == 0) {

nit: if (!data->len)...

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1878

Quoted 6 lines
object-file.c +6 -0
+		*len = 0;
+		return NULL;
+	}
+	*len = data->len;
+	data->len = 0;
+	return data->buf;

But isn't the body of this functin the same as:

        *len = data->len;
        if (!len)
                return NULL;
        data->len = 0;
        return data->buf;

I.e. you don't need the condition for setting "*len" if it's 0, then data->len is also 0. You just want to return NULL afterwards, and not set (harmless, but no need) data->len to 0)< or return data->buf.

object-file.c int write_object_file_flags(const void *buf, unsigned long len, · line 1991

Quoted 8 lines
object-file.c +8 -0
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = (void *)&(struct simple_input_stream_data) {
+			.buf = buf,
+			.len = len,
+		},
+		.size = len,
+	};

Maybe it's that I'm unused to it, but I find this a bit more readable:

	@@ -2013,12 +2011,13 @@ int write_object_file_flags(const void *buf, unsigned long len,
	 {
	 	char hdr[MAX_HEADER_LEN];
	 	int hdrlen = sizeof(hdr);
	+	struct simple_input_stream_data tmp = {
	+		.buf = buf,
	+		.len = len,
	+	};
	 	struct input_stream in_stream = {
	 		.read = feed_simple_input_stream,
	-		.data = (void *)&(struct simple_input_stream_data) {
	-			.buf = buf,
	-			.len = len,
	-		},
	+		.data = (void *)&tmp,
	 		.size = len,
	 	};

Yes there's a temporary variable, but no denser inline casting. Also easier to strep through in a debugger (which will have the type information on "tmp".

object-file.c int hash_object_file_literally(const void *buf, unsigned long len, · line 2016

Quoted 13 lines
object-file.c +8 -0
 int hash_object_file_literally(const void *buf, unsigned long len,
@@ -1977,6 +2006,14 @@ int hash_object_file_literally(const void *buf, unsigned long len,
 {
 	char *header;
 	int hdrlen, status = 0;
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = (void *)&(struct simple_input_stream_data) {
+			.buf = buf,
+			.len = len,
+		},
+		.size = len,
+	};

ditto..

Quoted 15 lines
object-file.c +1 -1
 	/* type string, SP, %lu of the length plus NUL must fit this */
 	hdrlen = strlen(type) + MAX_HEADER_LEN;
@@ -1988,7 +2025,7 @@ int hash_object_file_literally(const void *buf, unsigned long len,
 		goto cleanup;
 	if (freshen_packed_object(oid) || freshen_loose_object(oid))
 		goto cleanup;
-	status = write_loose_object(oid, header, hdrlen, buf, len, 0, 0);
+	status = write_loose_object(oid, header, hdrlen, &in_stream, 0, 0);
 
 cleanup:
 	free(header);
@@ -2003,14 +2040,22 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen;
 	int ret;

object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2052

Quoted 10 lines
object-file.c +6 -0
+	struct simple_input_stream_data data;
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = &data,
+	};
 
 	if (has_loose_object(oid))
 		return 0;
 	buf = read_object(the_repository, oid, &type, &len);
+	in_stream.size = len;

Why are we setting this here?...

object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2054

Quoted 2 lines
object-file.c +0 -0
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));

...Insted of after this point, as we may error and never use it?

object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2056

Quoted 2 lines
object-file.c +2 -0
+	data.buf = buf;
+	data.len = len;

Probably won't matter, just a nit...

object-file.c int force_object_loose(const struct object_id *oid, time_t mtime)

Quoted 6 lines
object-file.c +6 -0
+struct input_stream {
+	const void *(*read)(struct input_stream *, unsigned long *len);
+	void *data;
+	size_t size;
+};
+

Ah, and here's the size_t... :)

author
#177 lore raw

Re: [PATCH v4 1/5] object-file: refactor write_loose_object() to read buffer from stream

Quoted 38 lines · Ævar Arnfjörð Bjarmason
Quoted 23 lines · Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

This can be improved by feeding data to "write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.

---
 object-file.c  | 53 ++++++++++++++++++++++++++++++++++++++++++++++----
 object-store.h |  6 ++++++
 2 files changed, 55 insertions(+), 4 deletions(-)
object-file.c +0 -0
index eb972cdccd..82656f7428 100644

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1866

Quoted 7 lines
object-file.c +4 -0
@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
      return fd;
 }
 
+struct simple_input_stream_data {
+     const void *buf;
+     unsigned long len;
+};

I see why you picked "const void *buf" here, over say const char *, it's what "struct input_stream" uses.

Quoted 2 lines

But why not use size_t for the length, as input_stream does?

Yes, "size_t" will be better here.

Quoted 8 lines

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1872

Quoted 5 lines
object-file.c +5 -0
+static const void *feed_simple_input_stream(struct input_stream *in_stream, unsigned long *len)
+{
+     struct simple_input_stream_data *data = in_stream->data;
+
+     if (data->len == 0) {

nit: if (!data->len)...

Will apply.

Quoted 15 lines

object-file.c static int create_tmpfile(struct strbuf *tmp, const char *filename) · line 1878

Quoted 6 lines
object-file.c +6 -0
+             *len = 0;
+             return NULL;
+     }
+     *len = data->len;
+     data->len = 0;
+     return data->buf;

But isn't the body of this functin the same as:

        *len = data->len;
        if (!len)
                return NULL;
        data->len = 0;
        return data->buf;
Quoted 3 lines

I.e. you don't need the condition for setting "*len" if it's 0, then data->len is also 0. You just want to return NULL afterwards, and not set (harmless, but no need) data->len to 0)< or return data->buf.

Will apply.

Quoted 29 lines

object-file.c int write_object_file_flags(const void *buf, unsigned long len, · line 1991

Quoted 8 lines
object-file.c +8 -0
+     struct input_stream in_stream = {
+             .read = feed_simple_input_stream,
+             .data = (void *)&(struct simple_input_stream_data) {
+                     .buf = buf,
+                     .len = len,
+             },
+             .size = len,
+     };

Maybe it's that I'm unused to it, but I find this a bit more readable:

@@ -2013,12 +2011,13 @@ int write_object_file_flags(const void *buf, unsigned long len,
{
char hdr[MAX_HEADER_LEN];
int hdrlen = sizeof(hdr);
+ struct simple_input_stream_data tmp = {
+ .buf = buf,
+ .len = len,
+ };
struct input_stream in_stream = {
.read = feed_simple_input_stream,
- .data = (void *)&(struct simple_input_stream_data) {
- .buf = buf,
- .len = len,
- },
+ .data = (void *)&tmp,
.size = len,
};

Quoted 4 lines

Yes there's a temporary variable, but no denser inline casting. Also easier to strep through in a debugger (which will have the type information on "tmp".

Will apply.

Quoted 37 lines

object-file.c int hash_object_file_literally(const void *buf, unsigned long len, · line 2016

Quoted 13 lines
object-file.c +8 -0
 int hash_object_file_literally(const void *buf, unsigned long len,
@@ -1977,6 +2006,14 @@ int hash_object_file_literally(const void *buf, unsigned long len,
 {
      char *header;
      int hdrlen, status = 0;
+     struct input_stream in_stream = {
+             .read = feed_simple_input_stream,
+             .data = (void *)&(struct simple_input_stream_data) {
+                     .buf = buf,
+                     .len = len,
+             },
+             .size = len,
+     };

ditto..

Quoted 21 lines
object-file.c +6 -1
      /* type string, SP, %lu of the length plus NUL must fit this */
      hdrlen = strlen(type) + MAX_HEADER_LEN;
@@ -1988,7 +2025,7 @@ int hash_object_file_literally(const void *buf, unsigned long len,
              goto cleanup;
      if (freshen_packed_object(oid) || freshen_loose_object(oid))
              goto cleanup;
-     status = write_loose_object(oid, header, hdrlen, buf, len, 0, 0);
+     status = write_loose_object(oid, header, hdrlen, &in_stream, 0, 0);
 
 cleanup:
      free(header);
@@ -2003,14 +2040,22 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
      char hdr[MAX_HEADER_LEN];
      int hdrlen;
      int ret;
+     struct simple_input_stream_data data;
+     struct input_stream in_stream = {
+             .read = feed_simple_input_stream,
+             .data = &data,
+     };
 
Quoted 7 lines

object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2052

Quoted 4 lines
object-file.c +1 -0
      if (has_loose_object(oid))
              return 0;
      buf = read_object(the_repository, oid, &type, &len);
+     in_stream.size = len;

Why are we setting this here?...

Yes, putting "in_stream.size=len;" here was a stupid decision.

Quoted 18 lines

object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2054

Quoted 2 lines
object-file.c +0 -0
      if (!buf)
              return error(_("cannot read object for %s"), oid_to_hex(oid));

...Insted of after this point, as we may error and never use it?

object-file.c int force_object_loose(const struct object_id *oid, time_t mtime) · line 2056

Quoted 2 lines
object-file.c +2 -0
+     data.buf = buf;
+     data.len = len;

Probably won't matter, just a nit...

object-file.c int force_object_loose(const struct object_id *oid, time_t mtime)

Quoted 6 lines
object-file.c +6 -0
+struct input_stream {
+     const void *(*read)(struct input_stream *, unsigned long *len);
+     void *data;
+     size_t size;
+};
+

Ah, and here's the size_t... :)

author
#178 lore raw

PATCH v4 2/5 object-file.c: handle undetermined oid in write_loose_object()

In reply to Han Xin

From: Han Xin <[email protected]>

When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".

In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

---
 object-file.c | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
object-file.c +28 -2
index 82656f7428..1c41587bfb 100644
@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	const void *buf;
 	unsigned long len;
 
-	loose_object_path(the_repository, &filename, oid);
+	if (is_null_oid(oid)) {
+		/* When oid is not determined, save tmp file to odb path. */
+		strbuf_reset(&filename);
+		strbuf_addstr(&filename, the_repository->objects->odb->path);
+		strbuf_addch(&filename, '/');
+	} else {
+		loose_object_path(the_repository, &filename, oid);
+	}
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
@@ -1939,12 +1946,31 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
 		    ret);
 	the_hash_algo->final_oid_fn(&parano_oid, &c);
-	if (!oideq(oid, &parano_oid))
+	if (!is_null_oid(oid) && !oideq(oid, &parano_oid))
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));
 
 	close_loose_object(fd);
 
+	if (is_null_oid(oid)) {
+		int dirlen;
+
+		oidcpy((struct object_id *)oid, &parano_oid);
+		loose_object_path(the_repository, &filename, oid);
+
+		/* We finally know the object path, and create the missing dir. */
+		dirlen = directory_size(filename.buf);
+		if (dirlen) {
+			struct strbuf dir = STRBUF_INIT;
+			strbuf_add(&dir, filename.buf, dirlen - 1);
+			if (mkdir(dir.buf, 0777) && errno != EEXIST)
+				return -1;
+			if (adjust_shared_perm(dir.buf))
+				return -1;
+			strbuf_release(&dir);
+		}
+	}
+
 	if (mtime) {
 		struct utimbuf utb;
 		utb.actime = mtime;
Signature · 1 line
2.34.0
#179 lore raw

Re: [PATCH v4 2/5] object-file.c: handle undetermined oid in write_loose_object()

Quoted 45 lines · Han Xin

From: Han Xin <[email protected]>

When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".

In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

---
 object-file.c | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
object-file.c +9 -2
index 82656f7428..1c41587bfb 100644
@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	const void *buf;
 	unsigned long len;
 
-	loose_object_path(the_repository, &filename, oid);
+	if (is_null_oid(oid)) {
+		/* When oid is not determined, save tmp file to odb path. */
+		strbuf_reset(&filename);
+		strbuf_addstr(&filename, the_repository->objects->odb->path);
+		strbuf_addch(&filename, '/');
+	} else {
+		loose_object_path(the_repository, &filename, oid);
+	}
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
@@ -1939,12 +1946,31 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
 		    ret);
 	the_hash_algo->final_oid_fn(&parano_oid, &c);
-	if (!oideq(oid, &parano_oid))
+	if (!is_null_oid(oid) && !oideq(oid, &parano_oid))

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1959

Quoted 10 lines
object-file.c +5 -0
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));
 
 	close_loose_object(fd);
 
+	if (is_null_oid(oid)) {
+		int dirlen;
+
+		oidcpy((struct object_id *)oid, &parano_oid);
+		loose_object_path(the_repository, &filename, oid);

Why are we breaking the promise that "oid" is constant here? I tested locally with the below on top, and it seems to work (at least no tests broke). Isn't it preferrable to the cast & the caller having its "oid" changed?

object-file.c +3 -2
index 71d510614b9..d014e6942ea 100644
@@ -1958,10 +1958,11 @@ int write_loose_object(const struct object_id *oid, char *hdr,
 	close_loose_object(fd);
 
 	if (is_null_oid(oid)) {
+		struct object_id oid2;
 		int dirlen;
 
-		oidcpy((struct object_id *)oid, &parano_oid);
-		loose_object_path(the_repository, &filename, oid);
+		oidcpy(&oid2, &parano_oid);
+		loose_object_path(the_repository, &filename, &oid2);
 
 		/* We finally know the object path, and create the missing dir. */
 		dirlen = directory_size(filename.buf);
author
#180 lore raw

Re: [PATCH v4 2/5] object-file.c: handle undetermined oid in write_loose_object()

Quoted 73 lines · Ævar Arnfjörð Bjarmason
Quoted 45 lines · Han Xin

From: Han Xin <[email protected]>

When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".

In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

---
 object-file.c | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
object-file.c +9 -2
index 82656f7428..1c41587bfb 100644
@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
      const void *buf;
      unsigned long len;
 
-     loose_object_path(the_repository, &filename, oid);
+     if (is_null_oid(oid)) {
+             /* When oid is not determined, save tmp file to odb path. */
+             strbuf_reset(&filename);
+             strbuf_addstr(&filename, the_repository->objects->odb->path);
+             strbuf_addch(&filename, '/');
+     } else {
+             loose_object_path(the_repository, &filename, oid);
+     }
 
      fd = create_tmpfile(&tmp_file, filename.buf);
      if (fd < 0) {
@@ -1939,12 +1946,31 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
              die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
                  ret);
      the_hash_algo->final_oid_fn(&parano_oid, &c);
-     if (!oideq(oid, &parano_oid))
+     if (!is_null_oid(oid) && !oideq(oid, &parano_oid))

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1959

Quoted 10 lines
object-file.c +5 -0
              die(_("confused by unstable object source data for %s"),
                  oid_to_hex(oid));
 
      close_loose_object(fd);
 
+     if (is_null_oid(oid)) {
+             int dirlen;
+
+             oidcpy((struct object_id *)oid, &parano_oid);
+             loose_object_path(the_repository, &filename, oid);

Why are we breaking the promise that "oid" is constant here? I tested locally with the below on top, and it seems to work (at least no tests broke). Isn't it preferrable to the cast & the caller having its "oid" changed?

object-file.c +0 -0
index 71d510614b9..d014e6942ea 100644
@@ -1958,10 +1958,11 @@ int write_loose_object(const struct object_id *oid, char *hdr,
        close_loose_object(fd);
 
        if (is_null_oid(oid)) {

object-file.c int write_loose_object(const struct object_id *oid, char *hdr, · line 1968

Quoted 10 lines
object-file.c +3 -2
+               struct object_id oid2;
                int dirlen;
 
-               oidcpy((struct object_id *)oid, &parano_oid);
-               loose_object_path(the_repository, &filename, oid);
+               oidcpy(&oid2, &parano_oid);
+               loose_object_path(the_repository, &filename, &oid2);
 
                /* We finally know the object path, and create the missing dir. */
                dirlen = directory_size(filename.buf);

Maybe I should change the promise that "oid" is constant in "write_loose_object()".

The original write_object_file_flags() defines a variable "oid", and completes the calculation of the "oid" in "write_object_file_prepare()" which will be passed to "write_loose_object()".

If a null oid is maintained after calling "write_loose_object()", "--strict" will become meaningless, although it does not break existing test cases.

#181 lore raw

Re: [PATCH v4 2/5] object-file.c: handle undetermined oid in write_loose_object()

In reply to Han Xin
Quoted 24 lines · Han Xin

From: Han Xin <[email protected]>

When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".

In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

---
 object-file.c | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
object-file.c +0 -0
index 82656f7428..1c41587bfb 100644

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1897

Quoted 7 lines
object-file.c +3 -1
@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	const void *buf;
 	unsigned long len;
 
-	loose_object_path(the_repository, &filename, oid);
+	if (is_null_oid(oid)) {
+		/* When oid is not determined, save tmp file to odb path. */
+		strbuf_reset(&filename);

Why re-use this & leak memory? An existing strbuf use in this function doesn't leak in the same way. Just release it as in the below patch on top (the ret v.s. err variable naming is a bit confused, maybe could do with a prep cleanup step.).

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1899

Quoted 2 lines
object-file.c +2 -0
+		strbuf_addstr(&filename, the_repository->objects->odb->path);
+		strbuf_addch(&filename, '/');

And once we do that this could just become:

	strbuf_addf($filename, "%s/", ...)

Is there's existing uses of this pattern, so mayb e not worth it, but it allows you to remove the braces on the if/else.

object-file.c +16 -10
index 8bd89e7b7ba..2b52f3fc1cc 100644
@@ -1880,7 +1880,7 @@ int write_loose_object(const struct object_id *oid, char *hdr,
 		       int hdrlen, struct input_stream *in_stream,
 		       time_t mtime, unsigned flags)
 {
-	int fd, ret;
+	int fd, ret, err = 0;
 	unsigned char compressed[4096];
 	git_zstream stream;
 	git_hash_ctx c;
@@ -1892,7 +1892,6 @@ int write_loose_object(const struct object_id *oid, char *hdr,
 
 	if (is_null_oid(oid)) {
 		/* When oid is not determined, save tmp file to odb path. */
-		strbuf_reset(&filename);
 		strbuf_addstr(&filename, the_repository->objects->odb->path);
 		strbuf_addch(&filename, '/');
 	} else {
@@ -1902,11 +1901,12 @@ int write_loose_object(const struct object_id *oid, char *hdr,
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
 		if (flags & HASH_SILENT)
-			return -1;
+			err = -1;
 		else if (errno == EACCES)
-			return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
+			err = error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
 		else
-			return error_errno(_("unable to create temporary file"));
+			err = error_errno(_("unable to create temporary file"));
+		goto cleanup;
 	}
 
 	/* Set it up */
@@ -1968,10 +1968,13 @@ int write_loose_object(const struct object_id *oid, char *hdr,
 			struct strbuf dir = STRBUF_INIT;
 			strbuf_add(&dir, filename.buf, dirlen - 1);
 			if (mkdir(dir.buf, 0777) && errno != EEXIST)
-				return -1;
-			if (adjust_shared_perm(dir.buf))
-				return -1;
-			strbuf_release(&dir);
+				err = -1;
+			else if (adjust_shared_perm(dir.buf))
+				err = -1;
+			else
+				strbuf_release(&dir);
+			if (err < 0)
+				goto cleanup;
 		}
 	}
 
@@ -1984,7 +1987,10 @@ int write_loose_object(const struct object_id *oid, char *hdr,
 			warning_errno(_("failed utime() on %s"), tmp_file.buf);
 	}
 
-	return finalize_object_file(tmp_file.buf, filename.buf);
+	err = finalize_object_file(tmp_file.buf, filename.buf);
+cleanup:
+	strbuf_release(&filename);
+	return err;
 }
 
 static int freshen_loose_object(const struct object_id *oid)
author
#182 lore raw

Re: [PATCH v4 2/5] object-file.c: handle undetermined oid in write_loose_object()

Quoted 108 lines · Ævar Arnfjörð Bjarmason
Quoted 24 lines · Han Xin

From: Han Xin <[email protected]>

When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".

In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

---
 object-file.c | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
object-file.c +0 -0
index 82656f7428..1c41587bfb 100644

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1897

Quoted 7 lines
object-file.c +3 -1
@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
      const void *buf;
      unsigned long len;
 
-     loose_object_path(the_repository, &filename, oid);
+     if (is_null_oid(oid)) {
+             /* When oid is not determined, save tmp file to odb path. */
+             strbuf_reset(&filename);

Why re-use this & leak memory? An existing strbuf use in this function doesn't leak in the same way. Just release it as in the below patch on top (the ret v.s. err variable naming is a bit confused, maybe could do with a prep cleanup step.).

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1899

Quoted 2 lines
object-file.c +2 -0
+             strbuf_addstr(&filename, the_repository->objects->odb->path);
+             strbuf_addch(&filename, '/');

And once we do that this could just become:

        strbuf_addf($filename, "%s/", ...)

Is there's existing uses of this pattern, so mayb e not worth it, but it allows you to remove the braces on the if/else.

object-file.c +12 -9
index 8bd89e7b7ba..2b52f3fc1cc 100644
@@ -1880,7 +1880,7 @@ int write_loose_object(const struct object_id *oid, char *hdr,
                       int hdrlen, struct input_stream *in_stream,
                       time_t mtime, unsigned flags)
 {
-       int fd, ret;
+       int fd, ret, err = 0;
        unsigned char compressed[4096];
        git_zstream stream;
        git_hash_ctx c;
@@ -1892,7 +1892,6 @@ int write_loose_object(const struct object_id *oid, char *hdr,
 
        if (is_null_oid(oid)) {
                /* When oid is not determined, save tmp file to odb path. */
-               strbuf_reset(&filename);
                strbuf_addstr(&filename, the_repository->objects->odb->path);
                strbuf_addch(&filename, '/');
        } else {
@@ -1902,11 +1901,12 @@ int write_loose_object(const struct object_id *oid, char *hdr,
        fd = create_tmpfile(&tmp_file, filename.buf);
        if (fd < 0) {
                if (flags & HASH_SILENT)
-                       return -1;
+                       err = -1;
                else if (errno == EACCES)
-                       return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
+                       err = error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
                else
-                       return error_errno(_("unable to create temporary file"));
+                       err = error_errno(_("unable to create temporary file"));
+               goto cleanup;
        }
 
        /* Set it up */
@@ -1968,10 +1968,13 @@ int write_loose_object(const struct object_id *oid, char *hdr,
                        struct strbuf dir = STRBUF_INIT;
                        strbuf_add(&dir, filename.buf, dirlen - 1);
                        if (mkdir(dir.buf, 0777) && errno != EEXIST)
-                               return -1;
-                       if (adjust_shared_perm(dir.buf))
-                               return -1;
-                       strbuf_release(&dir);
+                               err = -1;
+                       else if (adjust_shared_perm(dir.buf))
+                               err = -1;
+                       else
+                               strbuf_release(&dir);
+                       if (err < 0)
+                               goto cleanup;
                }
        }
 
@@ -1984,7 +1987,10 @@ int write_loose_object(const struct object_id *oid, char *hdr,
                        warning_errno(_("failed utime() on %s"), tmp_file.buf);

object-file.c int write_loose_object(const struct object_id *oid, char *hdr, · line 1996

Quoted 10 lines
object-file.c +4 -1
        }
 
-       return finalize_object_file(tmp_file.buf, filename.buf);
+       err = finalize_object_file(tmp_file.buf, filename.buf);
+cleanup:
+       strbuf_release(&filename);
+       return err;
 }
 
 static int freshen_loose_object(const struct object_id *oid)

Yes, this will be much better. Will apply.

author
#183 lore raw

PATCH v4 3/5 object-file.c: read stream in a loop in write_loose_object()

In reply to Han Xin

From: Han Xin <[email protected]>

In order to prepare the stream version of "write_loose_object()", read the input stream in a loop in "write_loose_object()", so that we can feed the contents of large blob object to "write_loose_object()" using a small fixed buffer.

---
 object-file.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
object-file.c +11 -6
index 1c41587bfb..fa54e39c2c 100644
@@ -1890,7 +1890,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	static struct strbuf tmp_file = STRBUF_INIT;
 	static struct strbuf filename = STRBUF_INIT;
 	const void *buf;
-	unsigned long len;
+	int flush = 0;
 
 	if (is_null_oid(oid)) {
 		/* When oid is not determined, save tmp file to odb path. */
@@ -1925,18 +1925,23 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	the_hash_algo->update_fn(&c, hdr, hdrlen);
 
 	/* Then the data itself.. */
-	buf = in_stream->read(in_stream, &len);
-	stream.next_in = (void *)buf;
-	stream.avail_in = len;
 	do {
 		unsigned char *in0 = stream.next_in;
-		ret = git_deflate(&stream, Z_FINISH);
+		if (!stream.avail_in) {
+			buf = in_stream->read(in_stream, &stream.avail_in);
+			stream.next_in = (void *)buf;
+			in0 = (unsigned char *)buf;
+			/* All data has been read. */
+			if (in_stream->size + hdrlen == stream.total_in + stream.avail_in)
+				flush = Z_FINISH;
+		}
+		ret = git_deflate(&stream, flush);
 		the_hash_algo->update_fn(&c, in0, stream.next_in - in0);
 		if (write_buffer(fd, compressed, stream.next_out - compressed) < 0)
 			die(_("unable to write loose object file"));
 		stream.next_out = compressed;
 		stream.avail_out = sizeof(compressed);
-	} while (ret == Z_OK);
+	} while (ret == Z_OK || ret == Z_BUF_ERROR);
 
 	if (ret != Z_STREAM_END)
 		die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
Signature · 1 line
2.34.0
author
#184 lore raw

PATCH v4 4/5 unpack-objects.c: add dry_run mode for get_data()

In reply to Han Xin

From: Han Xin <[email protected]>

In dry_run mode, "get_data()" is used to verify the inflation of data, and the returned buffer will not be used at all and will be freed immediately. Even in dry_run mode, it is dangerous to allocate a full-size buffer for a large blob object. Therefore, only allocate a low memory footprint when calling "get_data()" in dry_run mode.

---
 builtin/unpack-objects.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
builtin/unpack-objects.c +12 -6
index 4a9466295b..8d68acd662 100644
@@ -96,15 +96,16 @@ static void use(int bytes)
 	display_throughput(progress, consumed_bytes);
 }
 
-static void *get_data(unsigned long size)
+static void *get_data(unsigned long size, int dry_run)
 {
 	git_zstream stream;
-	void *buf = xmallocz(size);
+	unsigned long bufsize = dry_run ? 4096 : size;
+	void *buf = xmallocz(bufsize);
 
 	memset(&stream, 0, sizeof(stream));
 
 	stream.next_out = buf;
-	stream.avail_out = size;
+	stream.avail_out = bufsize;
 	stream.next_in = fill(1);
 	stream.avail_in = len;
 	git_inflate_init(&stream);
@@ -124,6 +125,11 @@ static void *get_data(unsigned long size)
 		}
 		stream.next_in = fill(1);
 		stream.avail_in = len;
+		if (dry_run) {
+			/* reuse the buffer in dry_run mode */
+			stream.next_out = buf;
+			stream.avail_out = bufsize;
+		}
 	}
 	git_inflate_end(&stream);
 	return buf;
@@ -323,7 +329,7 @@ static void added_object(unsigned nr, enum object_type type,
 static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 				   unsigned nr)
 {
-	void *buf = get_data(size);
+	void *buf = get_data(size, dry_run);
 
 	if (!dry_run && buf)
 		write_object(nr, type, buf, size);
@@ -357,7 +363,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 	if (type == OBJ_REF_DELTA) {
 		oidread(&base_oid, fill(the_hash_algo->rawsz));
 		use(the_hash_algo->rawsz);
-		delta_data = get_data(delta_size);
+		delta_data = get_data(delta_size, dry_run);
 		if (dry_run || !delta_data) {
 			free(delta_data);
 			return;
@@ -396,7 +402,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 		if (base_offset <= 0 || base_offset >= obj_list[nr].offset)
 			die("offset value out of bound for delta base object");
 
-		delta_data = get_data(delta_size);
+		delta_data = get_data(delta_size, dry_run);
 		if (dry_run || !delta_data) {
 			free(delta_data);
 			return;
Signature · 1 line
2.34.0
#185 lore raw

Re: [PATCH v4 4/5] unpack-objects.c: add dry_run mode for get_data()

builtin/unpack-objects.c static void use(int bytes) · line 103

Quoted 2 lines · Han Xin
builtin/unpack-objects.c +2 -0
+	unsigned long bufsize = dry_run ? 4096 : size;
+	void *buf = xmallocz(bufsize);

It's probably nothing, but in your CL you note that you changed another hardcoding from 4k to 8k, should this one still be 4k?

It's probably fine, just wondering...

author
#186 lore raw

Re: [PATCH v4 4/5] unpack-objects.c: add dry_run mode for get_data()

Quoted 11 lines · Ævar Arnfjörð Bjarmason

builtin/unpack-objects.c static void use(int bytes) · line 103

Quoted 2 lines · Han Xin
builtin/unpack-objects.c +2 -0
+     unsigned long bufsize = dry_run ? 4096 : size;
+     void *buf = xmallocz(bufsize);

It's probably nothing, but in your CL you note that you changed another hardcoding from 4k to 8k, should this one still be 4k?

It's probably fine, just wondering...

Yes, I think this is an omission from my work.

author
#187 lore raw

PATCH v4 5/5 unpack-objects: unpack_non_delta_entry() read data in a stream

In reply to Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".

However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "big_file_threshold" in zstream. See the following benchmarks:

    hyperfine \
      --setup \
      'if ! test -d scalar.git; then git clone --bare https://github.com/microsoft/scalar.git; cp scalar.git/objects/pack/*.pack small.pack; fi' \
      --prepare 'rm -rf dest.git && git init --bare dest.git' \
      -n 'old' 'git -C dest.git unpack-objects <small.pack' \
      -n 'new' 'new/git -C dest.git unpack-objects <small.pack' \
      -n 'new (small threshold)' \
      'new/git -c core.bigfilethreshold=16k -C dest.git unpack-objects <small.pack'
    Benchmark 1: old
      Time (mean ± σ):      6.075 s ±  0.069 s    [User: 5.047 s, System: 0.991 s]
      Range (min … max):    6.018 s …  6.189 s    10 runs

    Benchmark 2: new
      Time (mean ± σ):      6.090 s ±  0.033 s    [User: 5.075 s, System: 0.976 s]
      Range (min … max):    6.030 s …  6.142 s    10 runs

    Benchmark 3: new (small threshold)
      Time (mean ± σ):      6.755 s ±  0.029 s    [User: 5.150 s, System: 1.560 s]
      Range (min … max):    6.711 s …  6.809 s    10 runs

    Summary
      'old' ran
        1.00 ± 0.01 times faster than 'new'
        1.11 ± 0.01 times faster than 'new (small threshold)'
---
 builtin/unpack-objects.c            | 77 ++++++++++++++++++++++++++++-
 object-file.c                       |  6 +--
 object-store.h                      |  4 ++
 t/t5590-unpack-non-delta-objects.sh | 76 ++++++++++++++++++++++++++++
 4 files changed, 159 insertions(+), 4 deletions(-)
 create mode 100755 t/t5590-unpack-non-delta-objects.sh
builtin/unpack-objects.c +76 -1
index 8d68acd662..bedc494e2d 100644
@@ -326,11 +326,86 @@ static void added_object(unsigned nr, enum object_type type,
 	}
 }
 
+struct input_zstream_data {
+	git_zstream *zstream;
+	unsigned char buf[8192];
+	int status;
+};
+
+static const void *feed_input_zstream(struct input_stream *in_stream, unsigned long *readlen)
+{
+	struct input_zstream_data *data = in_stream->data;
+	git_zstream *zstream = data->zstream;
+	void *in = fill(1);
+
+	if (!len || data->status == Z_STREAM_END) {
+		*readlen = 0;
+		return NULL;
+	}
+
+	zstream->next_out = data->buf;
+	zstream->avail_out = sizeof(data->buf);
+	zstream->next_in = in;
+	zstream->avail_in = len;
+
+	data->status = git_inflate(zstream, 0);
+	use(len - zstream->avail_in);
+	*readlen = sizeof(data->buf) - zstream->avail_out;
+
+	return data->buf;
+}
+
+static void write_stream_blob(unsigned nr, unsigned long size)
+{
+	char hdr[32];
+	int hdrlen;
+	git_zstream zstream;
+	struct input_zstream_data data;
+	struct input_stream in_stream = {
+		.read = feed_input_zstream,
+		.data = &data,
+		.size = size,
+	};
+	struct object_id *oid = &obj_list[nr].oid;
+	int ret;
+
+	memset(&zstream, 0, sizeof(zstream));
+	memset(&data, 0, sizeof(data));
+	data.zstream = &zstream;
+	git_inflate_init(&zstream);
+
+	/* Generate the header */
+	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX, type_name(OBJ_BLOB), (uintmax_t)size) + 1;
+
+	if ((ret = write_loose_object(oid, hdr, hdrlen, &in_stream, 0, 0)))
+		die(_("failed to write object in stream %d"), ret);
+
+	if (zstream.total_out != size || data.status != Z_STREAM_END)
+		die(_("inflate returned %d"), data.status);
+	git_inflate_end(&zstream);
+
+	if (strict && !dry_run) {
+		struct blob *blob = lookup_blob(the_repository, oid);
+		if (blob)
+			blob->object.flags |= FLAG_WRITTEN;
+		else
+			die("invalid blob object from stream");
+	}
+	obj_list[nr].obj = NULL;
+}
+
 static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 				   unsigned nr)
 {
-	void *buf = get_data(size, dry_run);
+	void *buf;
+
+	/* Write large blob in stream without allocating full buffer. */
+	if (!dry_run && type == OBJ_BLOB && size > big_file_threshold) {
+		write_stream_blob(nr, size);
+		return;
+	}
 
+	buf = get_data(size, dry_run);
 	if (!dry_run && buf)
 		write_object(nr, type, buf, size);
 	else
object-file.c +3 -3
index fa54e39c2c..71d510614b 100644
@@ -1878,9 +1878,9 @@ static const void *feed_simple_input_stream(struct input_stream *in_stream, unsi
 	return data->buf;
 }
 
-static int write_loose_object(const struct object_id *oid, char *hdr,
-			      int hdrlen, struct input_stream *in_stream,
-			      time_t mtime, unsigned flags)
+int write_loose_object(const struct object_id *oid, char *hdr,
+		       int hdrlen, struct input_stream *in_stream,
+		       time_t mtime, unsigned flags)
 {
 	int fd, ret;
 	unsigned char compressed[4096];
object-store.h +4 -0
index a84d891d60..ac5b11ec16 100644
@@ -229,6 +229,10 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf,
 		     unsigned long len, const char *type,
 		     struct object_id *oid);
 
+int write_loose_object(const struct object_id *oid, char *hdr,
+		       int hdrlen, struct input_stream *in_stream,
+		       time_t mtime, unsigned flags);
+
 int write_object_file_flags(const void *buf, unsigned long len,
 			    const char *type, struct object_id *oid,
 			    unsigned flags);
t/t5590-unpack-non-delta-objects.sh new file +76 -0

76 lines not on this page · raw message

Signature · 1 line
2.34.0
#188 lore raw

Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream

Quoted 30 lines · Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".

However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "big_file_threshold" in zstream. See the following benchmarks:

    hyperfine \
      --setup \
      'if ! test -d scalar.git; then git clone --bare https://github.com/microsoft/scalar.git; cp scalar.git/objects/pack/*.pack small.pack; fi' \
      --prepare 'rm -rf dest.git && git init --bare dest.git' \
      -n 'old' 'git -C dest.git unpack-objects <small.pack' \
      -n 'new' 'new/git -C dest.git unpack-objects <small.pack' \
      -n 'new (small threshold)' \
      'new/git -c core.bigfilethreshold=16k -C dest.git unpack-objects <small.pack'
    Benchmark 1: old
      Time (mean ± σ):      6.075 s ±  0.069 s    [User: 5.047 s, System: 0.991 s]
      Range (min … max):    6.018 s …  6.189 s    10 runs

    Benchmark 2: new
      Time (mean ± σ):      6.090 s ±  0.033 s    [User: 5.075 s, System: 0.976 s]
      Range (min … max):    6.030 s …  6.142 s    10 runs
Quoted 8 lines
    Benchmark 3: new (small threshold)
      Time (mean ± σ):      6.755 s ±  0.029 s    [User: 5.150 s, System: 1.560 s]
      Range (min … max):    6.711 s …  6.809 s    10 runs

    Summary
      'old' ran
        1.00 ± 0.01 times faster than 'new'
        1.11 ± 0.01 times faster than 'new (small threshold)'

So before we wrote used core.bigfilethreshold for two things (or more?):
Whether we show a diff for it (we mark it "binary") and whether it's split into a loose object.

Now it's three things, we've added a "this is a threshold when we'll stream the object" to that.

Might it make sense to squash something like this in, so we can have our cake & eat it too?

With this I get, where HEAD~0 is this change:

Summary
'./git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~0' ran
1.00 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~1' 1.00 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'origin/master' 1.01 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~0' 1.06 ± 0.14 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'origin/master' 1.20 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~1'

I.e. it's 5% slower, not 20% (haven't looked into why), but we'll not stream out 16k..128MB objects (maybe the repo has even bigger ones?)

Documentation/config/core.txt +11 -0
index c04f62a54a1..601b7a2418f 100644
@@ -424,6 +424,17 @@ be delta compressed, but larger binary media files won't be.
 +
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
+core.bigFileStreamingThreshold::
+	Files larger than this will be streamed out to a temporary
+	object file while being hashed, which will when be renamed
+	in-place to a loose object, particularly if the
+	`core.bigFileThreshold' setting dictates that they're always
+	written out as loose objects.
++
+Default is 128 MiB on all platforms.
++
+Common unit suffixes of 'k', 'm', or 'g' are supported.
+
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
 	describe paths that are not meant to be tracked, in addition
builtin/unpack-objects.c +1 -1
index bedc494e2db..94ce275c807 100644
@@ -400,7 +400,7 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 	void *buf;
 
 	/* Write large blob in stream without allocating full buffer. */
-	if (!dry_run && type == OBJ_BLOB && size > big_file_threshold) {
+	if (!dry_run && type == OBJ_BLOB && size > big_file_streaming_threshold) {
 		write_stream_blob(nr, size);
 		return;
 	}
cache.h +1 -0
index eba12487b99..4037c7fd849 100644
@@ -964,6 +964,7 @@ extern size_t packed_git_window_size;
 extern size_t packed_git_limit;
 extern size_t delta_base_cache_limit;
 extern unsigned long big_file_threshold;
+extern unsigned long big_file_streaming_threshold;
 extern unsigned long pack_size_limit_cfg;
 
 /*
config.c +5 -0
index c5873f3a706..7b122a142a8 100644
@@ -1408,6 +1408,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (!strcmp(var, "core.bigfilestreamingthreshold")) {
+		big_file_streaming_threshold = git_config_ulong(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "core.packedgitlimit")) {
 		packed_git_limit = git_config_ulong(var, value);
 		return 0;
environment.c +1 -0
index 9da7f3c1a19..4fcc3de7417 100644
@@ -46,6 +46,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
 size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
 size_t delta_base_cache_limit = 96 * 1024 * 1024;
 unsigned long big_file_threshold = 512 * 1024 * 1024;
+unsigned long big_file_streaming_threshold = 128 * 1024 * 1024;
 int pager_use_color = 1;
 const char *editor_program;
 const char *askpass_program;
author
#189 lore raw

Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream

Quoted 134 lines · Ævar Arnfjörð Bjarmason
Quoted 30 lines · Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".

However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "big_file_threshold" in zstream. See the following benchmarks:

    hyperfine \
      --setup \
      'if ! test -d scalar.git; then git clone --bare https://github.com/microsoft/scalar.git; cp scalar.git/objects/pack/*.pack small.pack; fi' \
      --prepare 'rm -rf dest.git && git init --bare dest.git' \
      -n 'old' 'git -C dest.git unpack-objects <small.pack' \
      -n 'new' 'new/git -C dest.git unpack-objects <small.pack' \
      -n 'new (small threshold)' \
      'new/git -c core.bigfilethreshold=16k -C dest.git unpack-objects <small.pack'
    Benchmark 1: old
      Time (mean ± σ):      6.075 s ±  0.069 s    [User: 5.047 s, System: 0.991 s]
      Range (min … max):    6.018 s …  6.189 s    10 runs

    Benchmark 2: new
      Time (mean ± σ):      6.090 s ±  0.033 s    [User: 5.075 s, System: 0.976 s]
      Range (min … max):    6.030 s …  6.142 s    10 runs
Quoted 8 lines
    Benchmark 3: new (small threshold)
      Time (mean ± σ):      6.755 s ±  0.029 s    [User: 5.150 s, System: 1.560 s]
      Range (min … max):    6.711 s …  6.809 s    10 runs

    Summary
      'old' ran
        1.00 ± 0.01 times faster than 'new'
        1.11 ± 0.01 times faster than 'new (small threshold)'

So before we wrote used core.bigfilethreshold for two things (or more?):
Whether we show a diff for it (we mark it "binary") and whether it's split into a loose object.

Now it's three things, we've added a "this is a threshold when we'll stream the object" to that.

Might it make sense to squash something like this in, so we can have our cake & eat it too?

With this I get, where HEAD~0 is this change:

Summary
'./git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~0' ran
1.00 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'HEAD~1' 1.00 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=512m unpack-objects <small.pack' in 'origin/master' 1.01 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~0' 1.06 ± 0.14 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'origin/master' 1.20 ± 0.01 times faster than './git -C dest.git -c core.bigfilethreshold=16k unpack-objects <small.pack' in 'HEAD~1'

I.e. it's 5% slower, not 20% (haven't looked into why), but we'll not stream out 16k..128MB objects (maybe the repo has even bigger ones?)

Documentation/config/core.txt +11 -0
index c04f62a54a1..601b7a2418f 100644
@@ -424,6 +424,17 @@ be delta compressed, but larger binary media files won't be.
 +
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
+core.bigFileStreamingThreshold::
+       Files larger than this will be streamed out to a temporary
+       object file while being hashed, which will when be renamed
+       in-place to a loose object, particularly if the
+       `core.bigFileThreshold' setting dictates that they're always
+       written out as loose objects.
++
+Default is 128 MiB on all platforms.
++
+Common unit suffixes of 'k', 'm', or 'g' are supported.
+
 core.excludesFile::
        Specifies the pathname to the file that contains patterns to
        describe paths that are not meant to be tracked, in addition
builtin/unpack-objects.c +1 -1
index bedc494e2db..94ce275c807 100644
@@ -400,7 +400,7 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,
        void *buf;
 
        /* Write large blob in stream without allocating full buffer. */
-       if (!dry_run && type == OBJ_BLOB && size > big_file_threshold) {
+       if (!dry_run && type == OBJ_BLOB && size > big_file_streaming_threshold) {
                write_stream_blob(nr, size);
                return;
        }
cache.h +1 -0
index eba12487b99..4037c7fd849 100644
@@ -964,6 +964,7 @@ extern size_t packed_git_window_size;
 extern size_t packed_git_limit;
 extern size_t delta_base_cache_limit;
 extern unsigned long big_file_threshold;
+extern unsigned long big_file_streaming_threshold;
 extern unsigned long pack_size_limit_cfg;
 
 /*
config.c +5 -0
index c5873f3a706..7b122a142a8 100644
@@ -1408,6 +1408,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
                return 0;
        }
 
+       if (!strcmp(var, "core.bigfilestreamingthreshold")) {
+               big_file_streaming_threshold = git_config_ulong(var, value);
+               return 0;
+       }
+
        if (!strcmp(var, "core.packedgitlimit")) {
                packed_git_limit = git_config_ulong(var, value);
                return 0;
environment.c +0 -0
index 9da7f3c1a19..4fcc3de7417 100644

environment.c size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; · line 52

Quoted 7 lines
environment.c +1 -0
@@ -46,6 +46,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
 size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
 size_t delta_base_cache_limit = 96 * 1024 * 1024;
 unsigned long big_file_threshold = 512 * 1024 * 1024;
+unsigned long big_file_streaming_threshold = 128 * 1024 * 1024;
 int pager_use_color = 1;
 const char *editor_program;
 const char *askpass_program;

I'm not sure if we need an additional "core.bigFileStreamingThreshold" here, because "core.bigFileThreshold" has been widely used in "index-pack", "read_object" and so on.

In the test case which uses "core.bigFileStreamingThreshold" instead of "core.bigFileThreshold", I found the test case execution failed because of "fsck", who tried to allocate 15MB of memory.
In the process of "fsck_loose()", "read_loose_object()" will be called, which contains the following content:

  if (*oi->typep == OBJ_BLOB && *size> big_file_threshold) {
    if (check_stream_oid(&stream, hdr, *size, path, expected_oid) <0)
    goto out;
  } else {
    /* this will allocate 15MB of memory */
    *contents = unpack_loose_rest(&stream, hdr, *size, expected_oid);
    ...
  }

The same case can be found in "unpack_entry_data()":

static char fixed_buf[8192];
...
if (type == OBJ_BLOB && size > big_file_threshold)

    buf = fixed_buf;
  else
    buf = xmallocz(size);

...

Although I know that setting a "core.bigfilethreshold" smaller than the default value on the server side does not help me prevent users from creating large delta objects on the client side, it can still effectively help me reduce the Memory allocation in "receive-pack".

If this is not the correct way to use "core.bigfilethreshold", maybe you can share some better solutions to me, if you want.

Thanks.
-Han Xin

#190 lore raw

Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream

In reply to Han Xin
Quoted 16 lines · Han Xin
t/t5590-unpack-non-delta-objects.sh new file +10 -0
new file mode 100755
index 0000000000..01d950d119
@@ -0,0 +1,76 @@
+#!/bin/sh
+#
+# Copyright (c) 2021 Han Xin
+#
+
+test_description='Test unpack-objects when receive pack'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+

t/t5590-unpack-non-delta-objects.sh line 20

Quoted 10 lines
t/t5590-unpack-non-delta-objects.sh new file +10 -0
+. ./test-lib.sh
+
+test_expect_success "create commit with big blobs (1.5 MB)" '
+	test-tool genrandom foo 1500000 >big-blob &&
+	test_commit --append foo big-blob &&
+	test-tool genrandom bar 1500000 >big-blob &&
+	test_commit --append bar big-blob &&
+	(
+		cd .git &&
+		find objects/?? -type f | sort

...are thse...

t/t5590-unpack-non-delta-objects.sh line 22

Quoted 2 lines
t/t5590-unpack-non-delta-objects.sh new file +2 -0
+	) >expect &&
+	PACK=$(echo main | git pack-objects --progress --revs test)

Is --progress needed?

t/t5590-unpack-non-delta-objects.sh line 33

Quoted 11 lines
t/t5590-unpack-non-delta-objects.sh new file +11 -0
+'
+
+test_expect_success 'setup GIT_ALLOC_LIMIT to 1MB' '
+	GIT_ALLOC_LIMIT=1m &&
+	export GIT_ALLOC_LIMIT
+'
+
+test_expect_success 'prepare dest repository' '
+	git init --bare dest.git &&
+	git -C dest.git config core.bigFileThreshold 2m &&
+	git -C dest.git config receive.unpacklimit 100

I think it would be better to just (could roll this into a function):

	test_when_finished "rm -rf dest.git" &&
	git init dest.git &&
	git -C dest.git config ...

Then you can use it with e.g. --run=3-4 and not have it error out because of skipped setup.

A lot of our tests fail like that, but in this case fixing it seems trivial.

t/t5590-unpack-non-delta-objects.sh line 38

Quoted 5 lines
t/t5590-unpack-non-delta-objects.sh new file +5 -0
+'
+
+test_expect_success 'fail to unpack-objects: cannot allocate' '
+	test_must_fail git -C dest.git unpack-objects <test-$PACK.pack 2>err &&
+	test_i18ngrep "fatal: attempting to allocate" err &&

nit: just "grep", not "test_i18ngrep"

t/t5590-unpack-non-delta-objects.sh line 41

Quoted 3 lines
t/t5590-unpack-non-delta-objects.sh new file +3 -0
+	(
+		cd dest.git &&
+		find objects/?? -type f | sort

..."find" needed over just globbing?:

    obj=$(echo objects/*/*)

?

author
#191 lore raw

Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream

Quoted 33 lines · Ævar Arnfjörð Bjarmason
Quoted 16 lines · Han Xin
t/t5590-unpack-non-delta-objects.sh new file +10 -0
new file mode 100755
index 0000000000..01d950d119
@@ -0,0 +1,76 @@
+#!/bin/sh
+#
+# Copyright (c) 2021 Han Xin
+#
+
+test_description='Test unpack-objects when receive pack'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+

t/t5590-unpack-non-delta-objects.sh line 20

Quoted 10 lines
t/t5590-unpack-non-delta-objects.sh new file +10 -0
+. ./test-lib.sh
+
+test_expect_success "create commit with big blobs (1.5 MB)" '
+     test-tool genrandom foo 1500000 >big-blob &&
+     test_commit --append foo big-blob &&
+     test-tool genrandom bar 1500000 >big-blob &&
+     test_commit --append bar big-blob &&
+     (
+             cd .git &&
+             find objects/?? -type f | sort

...are thse...

Quoted 5 lines

t/t5590-unpack-non-delta-objects.sh line 22

Quoted 2 lines
t/t5590-unpack-non-delta-objects.sh new file +2 -0
+     ) >expect &&
+     PACK=$(echo main | git pack-objects --progress --revs test)

Is --progress needed?

"--progress" is not necessary.

Quoted 18 lines

t/t5590-unpack-non-delta-objects.sh line 33

Quoted 11 lines
t/t5590-unpack-non-delta-objects.sh new file +11 -0
+'
+
+test_expect_success 'setup GIT_ALLOC_LIMIT to 1MB' '
+     GIT_ALLOC_LIMIT=1m &&
+     export GIT_ALLOC_LIMIT
+'
+
+test_expect_success 'prepare dest repository' '
+     git init --bare dest.git &&
+     git -C dest.git config core.bigFileThreshold 2m &&
+     git -C dest.git config receive.unpacklimit 100

I think it would be better to just (could roll this into a function):

        test_when_finished "rm -rf dest.git" &&
        git init dest.git &&
        git -C dest.git config ...
Quoted 7 lines

Then you can use it with e.g. --run=3-4 and not have it error out because of skipped setup.

A lot of our tests fail like that, but in this case fixing it seems trivial.

OK, I will take it.

Quoted 13 lines

t/t5590-unpack-non-delta-objects.sh line 38

Quoted 6 lines
t/t5590-unpack-non-delta-objects.sh new file +5 -0
+'
+
+test_expect_success 'fail to unpack-objects: cannot allocate' '
+     test_must_fail git -C dest.git unpack-objects <test-$PACK.pack 2>err &&
+     test_i18ngrep "fatal: attempting to allocate" err &&

nit: just "grep", not "test_i18ngrep"

t/t5590-unpack-non-delta-objects.sh line 41

Quoted 3 lines
t/t5590-unpack-non-delta-objects.sh new file +3 -0
+     (
+             cd dest.git &&
+             find objects/?? -type f | sort
Quoted 5 lines

..."find" needed over just globbing?:

    obj=$(echo objects/*/*)

?

I tried to use "echo" instead of "find". It works well on my personal computer, but fails due to the "info/commit-graph" generated when CI on Github.
So it seems that ".git/objects/??" will be more rigorous?

#192 lore raw

Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream

In reply to Han Xin
Quoted 30 lines · Han Xin

From: Han Xin <[email protected]>
[..]

builtin/unpack-objects.c +28 -0
+static void write_stream_blob(unsigned nr, unsigned long size)
+{
+	char hdr[32];
+	int hdrlen;
+	git_zstream zstream;
+	struct input_zstream_data data;
+	struct input_stream in_stream = {
+		.read = feed_input_zstream,
+		.data = &data,
+		.size = size,
+	};
+	struct object_id *oid = &obj_list[nr].oid;
+	int ret;
+
+	memset(&zstream, 0, sizeof(zstream));
+	memset(&data, 0, sizeof(data));
+	data.zstream = &zstream;
+	git_inflate_init(&zstream);
+
+	/* Generate the header */
+	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX, type_name(OBJ_BLOB), (uintmax_t)size) + 1;
+
+	if ((ret = write_loose_object(oid, hdr, hdrlen, &in_stream, 0, 0)))
+		die(_("failed to write object in stream %d"), ret);
+
+	if (zstream.total_out != size || data.status != Z_STREAM_END)
+		die(_("inflate returned %d"), data.status);
+	git_inflate_end(&zstream);

builtin/unpack-objects.c static void added_object(unsigned nr, enum object_type type, · line 395

Quoted 10 lines
builtin/unpack-objects.c +10 -0
+
+	if (strict && !dry_run) {
+		struct blob *blob = lookup_blob(the_repository, oid);
+		if (blob)
+			blob->object.flags |= FLAG_WRITTEN;
+		else
+			die("invalid blob object from stream");
+	}
+	obj_list[nr].obj = NULL;
+}

Just a side-note, I think (but am not 100% sure) that these existing occurances aren't needed due to our use of CALLOC_ARRAY():

    diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
    index 4a9466295ba..00b349412c5 100644
    --- a/builtin/unpack-objects.c
    +++ b/builtin/unpack-objects.c
    @@ -248,7 +248,6 @@ static void write_object(unsigned nr, enum object_type type,
                            die("failed to write object");
                    added_object(nr, type, buf, size);
                    free(buf);
    -               obj_list[nr].obj = NULL;
            } else if (type == OBJ_BLOB) {
                    struct blob *blob;
                    if (write_object_file(buf, size, type_name(type),
    @@ -262,7 +261,6 @@ static void write_object(unsigned nr, enum object_type type,
                            blob->object.flags |= FLAG_WRITTEN;
                    else
                            die("invalid blob object");
    -               obj_list[nr].obj = NULL;
            } else {
                    struct object *obj;
                    int eaten;

The reason I'm noting it is that the same seems to be true of your new addition here. I.e. are these assignments to NULL needed?

Anyway, the reason I started poking at this it tha this write_stream_blob() seems to duplicate much of write_object(). AFAICT only the writing part is really different, the part where we lookup_blob() after, set FLAG_WRITTEN etc. is all the same.

Why can't we call write_object() here?

The obvious answer seems to be that the call to write_object_file() isn't prepared to do the sort of streaming that you want, so instead you're bypassing it and calling write_loose_object() directly.

I haven't tried this myself, but isn't a better and cleaner approach here to not add another meaning to what is_null_oid() means, but to just add a HASH_STREAM flag that'll get passed down as "unsigned flags" to write_loose_object()? See FLAG_BITS in object.h.

Then the "obj_list[nr].obj" here could also become "obj_list[nr].obj.flags |= (1u<<12)" or whatever (but that wouldn't strictly be needed I think.

But by adding the "HASH_STREAM" flag you could I think stop duplicating the "Generate the header" etc. here and call write_object_file_flags().

I don't so much care about how it's done within unpack-objects.c, but not having another meaning to is_null_oid() in play would be really nice, and it this case it seems entirely avoidable.

author
#193 lore raw

Re: [PATCH v4 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream

Quoted 92 lines · Ævar Arnfjörð Bjarmason
Quoted 30 lines · Han Xin

From: Han Xin <[email protected]>
[..]

builtin/unpack-objects.c +28 -0
+static void write_stream_blob(unsigned nr, unsigned long size)
+{
+     char hdr[32];
+     int hdrlen;
+     git_zstream zstream;
+     struct input_zstream_data data;
+     struct input_stream in_stream = {
+             .read = feed_input_zstream,
+             .data = &data,
+             .size = size,
+     };
+     struct object_id *oid = &obj_list[nr].oid;
+     int ret;
+
+     memset(&zstream, 0, sizeof(zstream));
+     memset(&data, 0, sizeof(data));
+     data.zstream = &zstream;
+     git_inflate_init(&zstream);
+
+     /* Generate the header */
+     hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX, type_name(OBJ_BLOB), (uintmax_t)size) + 1;
+
+     if ((ret = write_loose_object(oid, hdr, hdrlen, &in_stream, 0, 0)))
+             die(_("failed to write object in stream %d"), ret);
+
+     if (zstream.total_out != size || data.status != Z_STREAM_END)
+             die(_("inflate returned %d"), data.status);
+     git_inflate_end(&zstream);

builtin/unpack-objects.c static void added_object(unsigned nr, enum object_type type, · line 395

Quoted 10 lines
builtin/unpack-objects.c +10 -0
+
+     if (strict && !dry_run) {
+             struct blob *blob = lookup_blob(the_repository, oid);
+             if (blob)
+                     blob->object.flags |= FLAG_WRITTEN;
+             else
+                     die("invalid blob object from stream");
+     }
+     obj_list[nr].obj = NULL;
+}

Just a side-note, I think (but am not 100% sure) that these existing occurances aren't needed due to our use of CALLOC_ARRAY():

    diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
    index 4a9466295ba..00b349412c5 100644
    --- a/builtin/unpack-objects.c
    +++ b/builtin/unpack-objects.c
    @@ -248,7 +248,6 @@ static void write_object(unsigned nr, enum object_type type,
                            die("failed to write object");
                    added_object(nr, type, buf, size);
                    free(buf);
    -               obj_list[nr].obj = NULL;
            } else if (type == OBJ_BLOB) {
                    struct blob *blob;
                    if (write_object_file(buf, size, type_name(type),
    @@ -262,7 +261,6 @@ static void write_object(unsigned nr, enum object_type type,
                            blob->object.flags |= FLAG_WRITTEN;
                    else
                            die("invalid blob object");
    -               obj_list[nr].obj = NULL;
            } else {
                    struct object *obj;
                    int eaten;

The reason I'm noting it is that the same seems to be true of your new addition here. I.e. are these assignments to NULL needed?

Anyway, the reason I started poking at this it tha this write_stream_blob() seems to duplicate much of write_object(). AFAICT only the writing part is really different, the part where we lookup_blob() after, set FLAG_WRITTEN etc. is all the same.

Why can't we call write_object() here?

The obvious answer seems to be that the call to write_object_file() isn't prepared to do the sort of streaming that you want, so instead you're bypassing it and calling write_loose_object() directly.

I haven't tried this myself, but isn't a better and cleaner approach here to not add another meaning to what is_null_oid() means, but to just add a HASH_STREAM flag that'll get passed down as "unsigned flags" to write_loose_object()? See FLAG_BITS in object.h.

Then the "obj_list[nr].obj" here could also become "obj_list[nr].obj.flags |= (1u<<12)" or whatever (but that wouldn't strictly be needed I think.

Quoted 6 lines

But by adding the "HASH_STREAM" flag you could I think stop duplicating the "Generate the header" etc. here and call write_object_file_flags().

I don't so much care about how it's done within unpack-objects.c, but not having another meaning to is_null_oid() in play would be really nice, and it this case it seems entirely avoidable.

I did refactor it according to your suggestions in my next patch version. Using a HASH_STREAM tag is indeed a better way to deal with it, and it can also reduce my refactor to the original contents.

Thanks.
-Han Xin

author
#194 lore raw

PATCH v3 1/5 object-file: refactor write_loose_object() to read buffer from stream

In reply to Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

This can be improved by feeding data to "write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.

---
 object-file.c  | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
 object-store.h |  5 +++++
 2 files changed, 51 insertions(+), 4 deletions(-)
object-file.c +46 -4
index c3d866a287..227f53a0de 100644
@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	return fd;
 }
 
+struct simple_input_stream_data {
+	const void *buf;
+	unsigned long len;
+};
+
+static const void *feed_simple_input_stream(struct input_stream *in_stream, unsigned long *len)
+{
+	struct simple_input_stream_data *data = in_stream->data;
+
+	if (data->len == 0) {
+		*len = 0;
+		return NULL;
+	}
+	*len = data->len;
+	data->len = 0;
+	return data->buf;
+}
+
 static int write_loose_object(const struct object_id *oid, char *hdr,
-			      int hdrlen, const void *buf, unsigned long len,
+			      int hdrlen, struct input_stream *in_stream,
 			      time_t mtime, unsigned flags)
 {
 	int fd, ret;
@@ -1871,6 +1889,8 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	struct object_id parano_oid;
 	static struct strbuf tmp_file = STRBUF_INIT;
 	static struct strbuf filename = STRBUF_INIT;
+	const void *buf;
+	unsigned long len;
 
 	loose_object_path(the_repository, &filename, oid);
 
@@ -1898,6 +1918,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	the_hash_algo->update_fn(&c, hdr, hdrlen);
 
 	/* Then the data itself.. */
+	buf = in_stream->read(in_stream, &len);
 	stream.next_in = (void *)buf;
 	stream.avail_in = len;
 	do {
@@ -1960,6 +1981,13 @@ int write_object_file_flags(const void *buf, unsigned long len,
 {
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen = sizeof(hdr);
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = (void *)&(struct simple_input_stream_data) {
+			.buf = buf,
+			.len = len,
+		},
+	};
 
 	/* Normally if we have it in the pack then we do not bother writing
 	 * it out into .git/objects/??/?{38} file.
@@ -1968,7 +1996,7 @@ int write_object_file_flags(const void *buf, unsigned long len,
 				  &hdrlen);
 	if (freshen_packed_object(oid) || freshen_loose_object(oid))
 		return 0;
-	return write_loose_object(oid, hdr, hdrlen, buf, len, 0, flags);
+	return write_loose_object(oid, hdr, hdrlen, &in_stream, 0, flags);
 }
 
 int hash_object_file_literally(const void *buf, unsigned long len,
@@ -1977,6 +2005,13 @@ int hash_object_file_literally(const void *buf, unsigned long len,
 {
 	char *header;
 	int hdrlen, status = 0;
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = (void *)&(struct simple_input_stream_data) {
+			.buf = buf,
+			.len = len,
+		},
+	};
 
 	/* type string, SP, %lu of the length plus NUL must fit this */
 	hdrlen = strlen(type) + MAX_HEADER_LEN;
@@ -1988,7 +2023,7 @@ int hash_object_file_literally(const void *buf, unsigned long len,
 		goto cleanup;
 	if (freshen_packed_object(oid) || freshen_loose_object(oid))
 		goto cleanup;
-	status = write_loose_object(oid, header, hdrlen, buf, len, 0, 0);
+	status = write_loose_object(oid, header, hdrlen, &in_stream, 0, 0);
 
 cleanup:
 	free(header);
@@ -2003,14 +2038,21 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen;
 	int ret;
+	struct simple_input_stream_data data;
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = &data,
+	};
 
 	if (has_loose_object(oid))
 		return 0;
 	buf = read_object(the_repository, oid, &type, &len);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
+	data.buf = buf;
+	data.len = len;
 	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
-	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
+	ret = write_loose_object(oid, hdr, hdrlen, &in_stream, mtime, 0);
 	free(buf);
 
 	return ret;
object-store.h +5 -0
index 952efb6a4b..ccc1fc9c1a 100644
@@ -34,6 +34,11 @@ struct object_directory {
 	char *path;
 };
 
+struct input_stream {
+	const void *(*read)(struct input_stream *, unsigned long *len);
+	void *data;
+};
+
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
 	struct object_directory *, 1, fspathhash, fspatheq)
 
Signature · 1 line
2.34.0.6.g676eedc724
#195 lore raw

Re: [PATCH v3 1/5] object-file: refactor write_loose_object() to read buffer from stream

Quoted 10 lines · Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

This can be improved by feeding data to "write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.

Possibly a stupid question (not a review).

How does this compare with "struct git_istream" implemented for a few existing codepaths? It seems that the existing users are pack-objects, index-pack and archive and all of them use the interface to obtain data given an object name without having to grab everything in core at once.

If we are adding a new streaming interface to go in the opposite direction, i.e. from the working tree data to object store, I would understand it as a complementary interface (but then I suspect there is a half of it already in bulk-checkin API), but I am not sure how this new thing fits in the larger picture.

Quoted 140 lines
---
 object-file.c  | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
 object-store.h |  5 +++++
 2 files changed, 51 insertions(+), 4 deletions(-)
object-file.c +46 -4
index c3d866a287..227f53a0de 100644
@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
 	return fd;
 }
 
+struct simple_input_stream_data {
+	const void *buf;
+	unsigned long len;
+};
+
+static const void *feed_simple_input_stream(struct input_stream *in_stream, unsigned long *len)
+{
+	struct simple_input_stream_data *data = in_stream->data;
+
+	if (data->len == 0) {
+		*len = 0;
+		return NULL;
+	}
+	*len = data->len;
+	data->len = 0;
+	return data->buf;
+}
+
 static int write_loose_object(const struct object_id *oid, char *hdr,
-			      int hdrlen, const void *buf, unsigned long len,
+			      int hdrlen, struct input_stream *in_stream,
 			      time_t mtime, unsigned flags)
 {
 	int fd, ret;
@@ -1871,6 +1889,8 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	struct object_id parano_oid;
 	static struct strbuf tmp_file = STRBUF_INIT;
 	static struct strbuf filename = STRBUF_INIT;
+	const void *buf;
+	unsigned long len;
 
 	loose_object_path(the_repository, &filename, oid);
 
@@ -1898,6 +1918,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	the_hash_algo->update_fn(&c, hdr, hdrlen);
 
 	/* Then the data itself.. */
+	buf = in_stream->read(in_stream, &len);
 	stream.next_in = (void *)buf;
 	stream.avail_in = len;
 	do {
@@ -1960,6 +1981,13 @@ int write_object_file_flags(const void *buf, unsigned long len,
 {
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen = sizeof(hdr);
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = (void *)&(struct simple_input_stream_data) {
+			.buf = buf,
+			.len = len,
+		},
+	};
 
 	/* Normally if we have it in the pack then we do not bother writing
 	 * it out into .git/objects/??/?{38} file.
@@ -1968,7 +1996,7 @@ int write_object_file_flags(const void *buf, unsigned long len,
 				  &hdrlen);
 	if (freshen_packed_object(oid) || freshen_loose_object(oid))
 		return 0;
-	return write_loose_object(oid, hdr, hdrlen, buf, len, 0, flags);
+	return write_loose_object(oid, hdr, hdrlen, &in_stream, 0, flags);
 }
 
 int hash_object_file_literally(const void *buf, unsigned long len,
@@ -1977,6 +2005,13 @@ int hash_object_file_literally(const void *buf, unsigned long len,
 {
 	char *header;
 	int hdrlen, status = 0;
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = (void *)&(struct simple_input_stream_data) {
+			.buf = buf,
+			.len = len,
+		},
+	};
 
 	/* type string, SP, %lu of the length plus NUL must fit this */
 	hdrlen = strlen(type) + MAX_HEADER_LEN;
@@ -1988,7 +2023,7 @@ int hash_object_file_literally(const void *buf, unsigned long len,
 		goto cleanup;
 	if (freshen_packed_object(oid) || freshen_loose_object(oid))
 		goto cleanup;
-	status = write_loose_object(oid, header, hdrlen, buf, len, 0, 0);
+	status = write_loose_object(oid, header, hdrlen, &in_stream, 0, 0);
 
 cleanup:
 	free(header);
@@ -2003,14 +2038,21 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen;
 	int ret;
+	struct simple_input_stream_data data;
+	struct input_stream in_stream = {
+		.read = feed_simple_input_stream,
+		.data = &data,
+	};
 
 	if (has_loose_object(oid))
 		return 0;
 	buf = read_object(the_repository, oid, &type, &len);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
+	data.buf = buf;
+	data.len = len;
 	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
-	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
+	ret = write_loose_object(oid, hdr, hdrlen, &in_stream, mtime, 0);
 	free(buf);
 
 	return ret;
object-store.h +5 -0
index 952efb6a4b..ccc1fc9c1a 100644
@@ -34,6 +34,11 @@ struct object_directory {
 	char *path;
 };
 
+struct input_stream {
+	const void *(*read)(struct input_stream *, unsigned long *len);
+	void *data;
+};
+
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
 	struct object_directory *, 1, fspathhash, fspatheq)
author
#196 lore raw

Re: [PATCH v3 1/5] object-file: refactor write_loose_object() to read buffer from stream

Quoted 22 lines · Junio C Hamano
Quoted 10 lines · Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

This can be improved by feeding data to "write_loose_object()" in a stream. The input stream is implemented as an interface. In the first step, we make a simple implementation, feeding the entire buffer in the "stream" to "write_loose_object()" as a refactor.

Possibly a stupid question (not a review).

How does this compare with "struct git_istream" implemented for a few existing codepaths? It seems that the existing users are pack-objects, index-pack and archive and all of them use the interface to obtain data given an object name without having to grab everything in core at once.

Quoted 6 lines

If we are adding a new streaming interface to go in the opposite direction, i.e. from the working tree data to object store, I would understand it as a complementary interface (but then I suspect there is a half of it already in bulk-checkin API), but I am not sure how this new thing fits in the larger picture.

Thank you for your reply.

Before starting to make this patch, I did consider whether I should reuse "struct git_istream" to solve the problem, but I found that in the process of git unpack-objects, the data comes from stdin, and we cannot get an oid in advance until the whole object data is read. Also, we can't do "lseek()“ on stdin to change the data reading position.

I compared the implementation of "bulk-checkin", and they do have some similarities.
I think the difference in the reverse implementation is that we do not always clearly know where the boundary of the target data is. For example, in the process of "unpack-objects", the "buffer" has been partially read after calling "fill()". And the "buffer" remaining after reading cannot be discarded because it is the beginning of the next object.
Perhaps "struct input_stream" can make some improvements to "index_bulk_checkin()", so that it can read from an inner buffer in addition to reading from "fd" if necessary.

Quoted 142 lines
Quoted 142 lines
---
 object-file.c  | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
 object-store.h |  5 +++++
 2 files changed, 51 insertions(+), 4 deletions(-)
object-file.c +46 -4
index c3d866a287..227f53a0de 100644
@@ -1860,8 +1860,26 @@ static int create_tmpfile(struct strbuf *tmp, const char *filename)
      return fd;
 }
 
+struct simple_input_stream_data {
+     const void *buf;
+     unsigned long len;
+};
+
+static const void *feed_simple_input_stream(struct input_stream *in_stream, unsigned long *len)
+{
+     struct simple_input_stream_data *data = in_stream->data;
+
+     if (data->len == 0) {
+             *len = 0;
+             return NULL;
+     }
+     *len = data->len;
+     data->len = 0;
+     return data->buf;
+}
+
 static int write_loose_object(const struct object_id *oid, char *hdr,
-                           int hdrlen, const void *buf, unsigned long len,
+                           int hdrlen, struct input_stream *in_stream,
                            time_t mtime, unsigned flags)
 {
      int fd, ret;
@@ -1871,6 +1889,8 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
      struct object_id parano_oid;
      static struct strbuf tmp_file = STRBUF_INIT;
      static struct strbuf filename = STRBUF_INIT;
+     const void *buf;
+     unsigned long len;
 
      loose_object_path(the_repository, &filename, oid);
 
@@ -1898,6 +1918,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
      the_hash_algo->update_fn(&c, hdr, hdrlen);
 
      /* Then the data itself.. */
+     buf = in_stream->read(in_stream, &len);
      stream.next_in = (void *)buf;
      stream.avail_in = len;
      do {
@@ -1960,6 +1981,13 @@ int write_object_file_flags(const void *buf, unsigned long len,
 {
      char hdr[MAX_HEADER_LEN];
      int hdrlen = sizeof(hdr);
+     struct input_stream in_stream = {
+             .read = feed_simple_input_stream,
+             .data = (void *)&(struct simple_input_stream_data) {
+                     .buf = buf,
+                     .len = len,
+             },
+     };
 
      /* Normally if we have it in the pack then we do not bother writing
       * it out into .git/objects/??/?{38} file.
@@ -1968,7 +1996,7 @@ int write_object_file_flags(const void *buf, unsigned long len,
                                &hdrlen);
      if (freshen_packed_object(oid) || freshen_loose_object(oid))
              return 0;
-     return write_loose_object(oid, hdr, hdrlen, buf, len, 0, flags);
+     return write_loose_object(oid, hdr, hdrlen, &in_stream, 0, flags);
 }
 
 int hash_object_file_literally(const void *buf, unsigned long len,
@@ -1977,6 +2005,13 @@ int hash_object_file_literally(const void *buf, unsigned long len,
 {
      char *header;
      int hdrlen, status = 0;
+     struct input_stream in_stream = {
+             .read = feed_simple_input_stream,
+             .data = (void *)&(struct simple_input_stream_data) {
+                     .buf = buf,
+                     .len = len,
+             },
+     };
 
      /* type string, SP, %lu of the length plus NUL must fit this */
      hdrlen = strlen(type) + MAX_HEADER_LEN;
@@ -1988,7 +2023,7 @@ int hash_object_file_literally(const void *buf, unsigned long len,
              goto cleanup;
      if (freshen_packed_object(oid) || freshen_loose_object(oid))
              goto cleanup;
-     status = write_loose_object(oid, header, hdrlen, buf, len, 0, 0);
+     status = write_loose_object(oid, header, hdrlen, &in_stream, 0, 0);
 
 cleanup:
      free(header);
@@ -2003,14 +2038,21 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
      char hdr[MAX_HEADER_LEN];
      int hdrlen;
      int ret;
+     struct simple_input_stream_data data;
+     struct input_stream in_stream = {
+             .read = feed_simple_input_stream,
+             .data = &data,
+     };
 
      if (has_loose_object(oid))
              return 0;
      buf = read_object(the_repository, oid, &type, &len);
      if (!buf)
              return error(_("cannot read object for %s"), oid_to_hex(oid));
+     data.buf = buf;
+     data.len = len;
      hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX , type_name(type), (uintmax_t)len) + 1;
-     ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
+     ret = write_loose_object(oid, hdr, hdrlen, &in_stream, mtime, 0);
      free(buf);
 
      return ret;
object-store.h +5 -0
index 952efb6a4b..ccc1fc9c1a 100644
@@ -34,6 +34,11 @@ struct object_directory {
      char *path;
 };
 
+struct input_stream {
+     const void *(*read)(struct input_stream *, unsigned long *len);
+     void *data;
+};
+
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
      struct object_directory *, 1, fspathhash, fspatheq)
author
#197 lore raw

PATCH v3 2/5 object-file.c: handle undetermined oid in write_loose_object()

In reply to Han Xin

From: Han Xin <[email protected]>

When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".

In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

---
 object-file.c | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
object-file.c +28 -2
index 227f53a0de..78fd2a5d39 100644
@@ -1892,7 +1892,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	const void *buf;
 	unsigned long len;
 
-	loose_object_path(the_repository, &filename, oid);
+	if (is_null_oid(oid)) {
+		/* When oid is not determined, save tmp file to odb path. */
+		strbuf_reset(&filename);
+		strbuf_addstr(&filename, the_repository->objects->odb->path);
+		strbuf_addch(&filename, '/');
+	} else {
+		loose_object_path(the_repository, &filename, oid);
+	}
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
@@ -1939,12 +1946,31 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
 		    ret);
 	the_hash_algo->final_oid_fn(&parano_oid, &c);
-	if (!oideq(oid, &parano_oid))
+	if (!is_null_oid(oid) && !oideq(oid, &parano_oid))
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));
 
 	close_loose_object(fd);
 
+	if (is_null_oid(oid)) {
+		int dirlen;
+
+		oidcpy((struct object_id *)oid, &parano_oid);
+		loose_object_path(the_repository, &filename, oid);
+
+		/* We finally know the object path, and create the missing dir. */
+		dirlen = directory_size(filename.buf);
+		if (dirlen) {
+			struct strbuf dir = STRBUF_INIT;
+			strbuf_add(&dir, filename.buf, dirlen - 1);
+			if (mkdir(dir.buf, 0777) && errno != EEXIST)
+				return -1;
+			if (adjust_shared_perm(dir.buf))
+				return -1;
+			strbuf_release(&dir);
+		}
+	}
+
 	if (mtime) {
 		struct utimbuf utb;
 		utb.actime = mtime;
Signature · 1 line
2.34.0.6.g676eedc724
#198 lore raw

Re: [PATCH v3 2/5] object-file.c: handle undetermined oid in write_loose_object()

Quoted 7 lines · Han Xin

From: Han Xin <[email protected]>

When streaming a large blob object to "write_loose_object()", we have no chance to run "write_object_file_prepare()" to calculate the oid in advance. So we need to handle undetermined oid in function "write_loose_object()".

Quoted 5 lines

In the original implementation, we know the oid and we can write the temporary file in the same directory as the final object, but for an object with an undetermined oid, we don't know the exact directory for the object, so we have to save the temporary file in ".git/objects/" directory instead.

My first reaction is to not write into .git/objects/ directly, but instead make a .git/objects/tmp/ directory and write within that directory. The idea is to prevent leaving stale files in the .git/objects/ directory if the process terminates strangely (say, a power outage or segfault).

If this was an interesting idea to pursue, it does leave a question:
should we clean up the tmp/ directory when it is empty? That would require adding a check in finalize_object_file() that is probably best left unchecked (the lstat() would add a cost per loose object write that is probably too costly). I would rather leave an empty tmp/ directory than add that cost per loose object write.

I suppose another way to do it would be to register the check as an event at the end of the process, so we only check once, and that only happens if we created a loose object with this streaming method.

With all of these complications in mind, I think cleaning up the stale tmp/ directory could (at the very least) be delayed to another commit or patch series. Hopefully adding the directory is not too much complication to add here.

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1899

Quoted 6 lines
object-file.c +5 -1
-	loose_object_path(the_repository, &filename, oid);
+	if (is_null_oid(oid)) {
+		/* When oid is not determined, save tmp file to odb path. */
+		strbuf_reset(&filename);
+		strbuf_addstr(&filename, the_repository->objects->odb->path);
+		strbuf_addch(&filename, '/');

Here, you could instead of the strbuf_addch() do

	strbuf_add(&filename, "/tmp/", 5);
	if (safe_create_leading_directories(filename.buf)) {
		error(_("failed to create '%s'"));
		strbuf_release(&filename);
		return -1;
	}		
Quoted 26 lines
object-file.c +13 -1
+	} else {
+		loose_object_path(the_repository, &filename, oid);
+	}
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
@@ -1939,12 +1946,31 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
 		    ret);
 	the_hash_algo->final_oid_fn(&parano_oid, &c);
-	if (!oideq(oid, &parano_oid))
+	if (!is_null_oid(oid) && !oideq(oid, &parano_oid))
 		die(_("confused by unstable object source data for %s"),
 		    oid_to_hex(oid));
 
 	close_loose_object(fd);
 
+	if (is_null_oid(oid)) {
+		int dirlen;
+
+		oidcpy((struct object_id *)oid, &parano_oid);
+		loose_object_path(the_repository, &filename, oid);
+
+		/* We finally know the object path, and create the missing dir. */
+		dirlen = directory_size(filename.buf);
+		if (dirlen) {

object-file.c static int write_loose_object(const struct object_id *oid, char *hdr, · line 1973

Quoted 10 lines
object-file.c +10 -0
+			struct strbuf dir = STRBUF_INIT;
+			strbuf_add(&dir, filename.buf, dirlen - 1);
+			if (mkdir(dir.buf, 0777) && errno != EEXIST)
+				return -1;
+			if (adjust_shared_perm(dir.buf))
+				return -1;
+			strbuf_release(&dir);
+		}
+	}
+

Upon first reading I was asking "where is the file rename?" but it is part of finalize_object_file() which is called further down.

Thanks,
-Stolee

#199 lore raw

Re: [PATCH v3 2/5] object-file.c: handle undetermined oid in write_loose_object()

Quoted 5 lines · Derrick Stolee

My first reaction is to not write into .git/objects/ directly, but instead make a .git/objects/tmp/ directory and write within that directory. The idea is to prevent leaving stale files in the .git/objects/ directory if the process terminates strangely (say, a power outage or segfault).

Even if we know the name of the object we are writing beforehand, I do not think it is a good idea to open-write-close the final object file. The approach we already use everywhere is to write into a tmpfile/lockfile and rename it to the final name

object-file.c::write_loose_object() uses create_tmpfile() to prepare a temporary file whose name begins with "tmp_obj_", so that "gc" can recognize stale ones and remove them.

Quoted 6 lines

If this was an interesting idea to pursue, it does leave a question:
should we clean up the tmp/ directory when it is empty? That would require adding a check in finalize_object_file() that is probably best left unchecked (the lstat() would add a cost per loose object write that is probably too costly). I would rather leave an empty tmp/ directory than add that cost per loose object write.

I am not sure why we want a new tmp/ directory.

#200 lore raw

Re: [PATCH v3 2/5] object-file.c: handle undetermined oid in write_loose_object()

Quoted 8 lines · Junio C Hamano
Quoted 5 lines · Derrick Stolee

My first reaction is to not write into .git/objects/ directly, but instead make a .git/objects/tmp/ directory and write within that directory. The idea is to prevent leaving stale files in the .git/objects/ directory if the process terminates strangely (say, a power outage or segfault).

Quoted 8 lines

Even if we know the name of the object we are writing beforehand, I do not think it is a good idea to open-write-close the final object file. The approach we already use everywhere is to write into a tmpfile/lockfile and rename it to the final name

object-file.c::write_loose_object() uses create_tmpfile() to prepare a temporary file whose name begins with "tmp_obj_", so that "gc" can recognize stale ones and remove them.

The only difference is that the tmp_obj_* file would go into the loose object directory corresponding to the first two hex characters of the OID, but that no longer happens now.

Quoted 8 lines
Quoted 6 lines

If this was an interesting idea to pursue, it does leave a question:
should we clean up the tmp/ directory when it is empty? That would require adding a check in finalize_object_file() that is probably best left unchecked (the lstat() would add a cost per loose object write that is probably too costly). I would rather leave an empty tmp/ directory than add that cost per loose object write.

I am not sure why we want a new tmp/ directory.

I'm just thinking of a case where this fails repeatedly I would rather have those failed tmp_obj_* files isolated in their own directory. It's an extremely minor point, so I'm fine to drop the recommendation.

Thanks,
-Stolee

author
#201 lore raw

Re: [PATCH v3 2/5] object-file.c: handle undetermined oid in write_loose_object()

Quoted 15 lines · Derrick Stolee
Quoted 13 lines · Junio C Hamano
Quoted 5 lines · Derrick Stolee

My first reaction is to not write into .git/objects/ directly, but instead make a .git/objects/tmp/ directory and write within that directory. The idea is to prevent leaving stale files in the .git/objects/ directory if the process terminates strangely (say, a power outage or segfault).

Even if we know the name of the object we are writing beforehand, I do not think it is a good idea to open-write-close the final object file. The approach we already use everywhere is to write into a tmpfile/lockfile and rename it to the final name

Quoted 8 lines
Quoted 3 lines

object-file.c::write_loose_object() uses create_tmpfile() to prepare a temporary file whose name begins with "tmp_obj_", so that "gc" can recognize stale ones and remove them.

The only difference is that the tmp_obj_* file would go into the loose object directory corresponding to the first two hex characters of the OID, but that no longer happens now.

At the beginning of this patch, I did save the temporary object in a two hex characters directory of "null_oid", but this is also a very strange behavior. "Gc" will indeed clean up these tmp_obj_* files, no matter if they are in .git/objects/ or .git/objects/xx.

Thanks,
-Han Xin

author
#202 lore raw

PATCH v3 3/5 object-file.c: read stream in a loop in write_loose_object()

In reply to Han Xin

From: Han Xin <[email protected]>

In order to prepare the stream version of "write_loose_object()", read the input stream in a loop in "write_loose_object()", so that we can feed the contents of large blob object to "write_loose_object()" using a small fixed buffer.

---
 object-file.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
object-file.c +11 -5
index 78fd2a5d39..93bcfaca50 100644
@@ -1890,7 +1890,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	static struct strbuf tmp_file = STRBUF_INIT;
 	static struct strbuf filename = STRBUF_INIT;
 	const void *buf;
-	unsigned long len;
+	int flush = 0;
 
 	if (is_null_oid(oid)) {
 		/* When oid is not determined, save tmp file to odb path. */
@@ -1925,12 +1925,18 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	the_hash_algo->update_fn(&c, hdr, hdrlen);
 
 	/* Then the data itself.. */
-	buf = in_stream->read(in_stream, &len);
-	stream.next_in = (void *)buf;
-	stream.avail_in = len;
 	do {
 		unsigned char *in0 = stream.next_in;
-		ret = git_deflate(&stream, Z_FINISH);
+		if (!stream.avail_in) {
+			buf = in_stream->read(in_stream, &stream.avail_in);
+			if (buf) {
+				stream.next_in = (void *)buf;
+				in0 = (unsigned char *)buf;
+			} else {
+				flush = Z_FINISH;
+			}
+		}
+		ret = git_deflate(&stream, flush);
 		the_hash_algo->update_fn(&c, in0, stream.next_in - in0);
 		if (write_buffer(fd, compressed, stream.next_out - compressed) < 0)
 			die(_("unable to write loose object file"));
Signature · 1 line
2.34.0.6.g676eedc724
author
#203 lore raw

PATCH v3 4/5 unpack-objects.c: add dry_run mode for get_data()

In reply to Han Xin

From: Han Xin <[email protected]>

In dry_run mode, "get_data()" is used to verify the inflation of data, and the returned buffer will not be used at all and will be freed immediately. Even in dry_run mode, it is dangerous to allocate a full-size buffer for a large blob object. Therefore, only allocate a low memory footprint when calling "get_data()" in dry_run mode.

---
 builtin/unpack-objects.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
builtin/unpack-objects.c +12 -6
index 4a9466295b..8d68acd662 100644
@@ -96,15 +96,16 @@ static void use(int bytes)
 	display_throughput(progress, consumed_bytes);
 }
 
-static void *get_data(unsigned long size)
+static void *get_data(unsigned long size, int dry_run)
 {
 	git_zstream stream;
-	void *buf = xmallocz(size);
+	unsigned long bufsize = dry_run ? 4096 : size;
+	void *buf = xmallocz(bufsize);
 
 	memset(&stream, 0, sizeof(stream));
 
 	stream.next_out = buf;
-	stream.avail_out = size;
+	stream.avail_out = bufsize;
 	stream.next_in = fill(1);
 	stream.avail_in = len;
 	git_inflate_init(&stream);
@@ -124,6 +125,11 @@ static void *get_data(unsigned long size)
 		}
 		stream.next_in = fill(1);
 		stream.avail_in = len;
+		if (dry_run) {
+			/* reuse the buffer in dry_run mode */
+			stream.next_out = buf;
+			stream.avail_out = bufsize;
+		}
 	}
 	git_inflate_end(&stream);
 	return buf;
@@ -323,7 +329,7 @@ static void added_object(unsigned nr, enum object_type type,
 static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 				   unsigned nr)
 {
-	void *buf = get_data(size);
+	void *buf = get_data(size, dry_run);
 
 	if (!dry_run && buf)
 		write_object(nr, type, buf, size);
@@ -357,7 +363,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 	if (type == OBJ_REF_DELTA) {
 		oidread(&base_oid, fill(the_hash_algo->rawsz));
 		use(the_hash_algo->rawsz);
-		delta_data = get_data(delta_size);
+		delta_data = get_data(delta_size, dry_run);
 		if (dry_run || !delta_data) {
 			free(delta_data);
 			return;
@@ -396,7 +402,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,
 		if (base_offset <= 0 || base_offset >= obj_list[nr].offset)
 			die("offset value out of bound for delta base object");
 
-		delta_data = get_data(delta_size);
+		delta_data = get_data(delta_size, dry_run);
 		if (dry_run || !delta_data) {
 			free(delta_data);
 			return;
Signature · 1 line
2.34.0.6.g676eedc724
author
#204 lore raw

PATCH v3 5/5 unpack-objects: unpack_non_delta_entry() read data in a stream

In reply to Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".

However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "big_file_threshold" in zstream. See the following benchmarks:

    $ hyperfine \
    --prepare 'rm -rf dest.git && git init --bare dest.git' \
    'git -C dest.git unpack-objects <binary_320M.pack'
    Benchmark 1: git -C dest.git unpack-objects <binary_320M.pack
      Time (mean ± σ):     10.029 s ±  0.270 s    [User: 8.265 s, System: 1.522 s]
      Range (min … max):    9.786 s … 10.603 s    10 runs
    $ hyperfine \
    --prepare 'rm -rf dest.git && git init --bare dest.git' \
    'git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack'
    Benchmark 1: git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack
      Time (mean ± σ):     10.859 s ±  0.774 s    [User: 8.813 s, System: 1.898 s]
      Range (min … max):    9.884 s … 12.192 s    10 runs
    $ hyperfine \
    --prepare 'rm -rf dest.git && git init --bare dest.git' \
    'git -C dest.git unpack-objects <binary_96M.pack'
    Benchmark 1: git -C dest.git unpack-objects <binary_96M.pack
      Time (mean ± σ):      2.678 s ±  0.037 s    [User: 2.205 s, System: 0.450 s]
      Range (min … max):    2.639 s …  2.743 s    10 runs
    $ hyperfine \
    --prepare 'rm -rf dest.git && git init --bare dest.git' \
    'git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_96M.pack'
    Benchmark 1: git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_96M.pack
      Time (mean ± σ):      2.819 s ±  0.124 s    [User: 2.216 s, System: 0.564 s]
      Range (min … max):    2.679 s …  3.125 s    10 runs
---
 builtin/unpack-objects.c            | 76 ++++++++++++++++++++++++++++-
 object-file.c                       |  6 +--
 object-store.h                      |  4 ++
 t/t5590-unpack-non-delta-objects.sh | 76 +++++++++++++++++++++++++++++
 4 files changed, 158 insertions(+), 4 deletions(-)
 create mode 100755 t/t5590-unpack-non-delta-objects.sh
builtin/unpack-objects.c +75 -1
index 8d68acd662..bfc254a236 100644
@@ -326,11 +326,85 @@ static void added_object(unsigned nr, enum object_type type,
 	}
 }
 
+struct input_zstream_data {
+	git_zstream *zstream;
+	unsigned char buf[4096];
+	int status;
+};
+
+static const void *feed_input_zstream(struct input_stream *in_stream, unsigned long *readlen)
+{
+	struct input_zstream_data *data = in_stream->data;
+	git_zstream *zstream = data->zstream;
+	void *in = fill(1);
+
+	if (!len || data->status == Z_STREAM_END) {
+		*readlen = 0;
+		return NULL;
+	}
+
+	zstream->next_out = data->buf;
+	zstream->avail_out = sizeof(data->buf);
+	zstream->next_in = in;
+	zstream->avail_in = len;
+
+	data->status = git_inflate(zstream, 0);
+	use(len - zstream->avail_in);
+	*readlen = sizeof(data->buf) - zstream->avail_out;
+
+	return data->buf;
+}
+
+static void write_stream_blob(unsigned nr, unsigned long size)
+{
+	char hdr[32];
+	int hdrlen;
+	git_zstream zstream;
+	struct input_zstream_data data;
+	struct input_stream in_stream = {
+		.read = feed_input_zstream,
+		.data = &data,
+	};
+	struct object_id *oid = &obj_list[nr].oid;
+	int ret;
+
+	memset(&zstream, 0, sizeof(zstream));
+	memset(&data, 0, sizeof(data));
+	data.zstream = &zstream;
+	git_inflate_init(&zstream);
+
+	/* Generate the header */
+	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX, type_name(OBJ_BLOB), (uintmax_t)size) + 1;
+
+	if ((ret = write_loose_object(oid, hdr, hdrlen, &in_stream, 0, 0)))
+		die(_("failed to write object in stream %d"), ret);
+
+	if (zstream.total_out != size || data.status != Z_STREAM_END)
+		die(_("inflate returned %d"), data.status);
+	git_inflate_end(&zstream);
+
+	if (strict && !dry_run) {
+		struct blob *blob = lookup_blob(the_repository, oid);
+		if (blob)
+			blob->object.flags |= FLAG_WRITTEN;
+		else
+			die("invalid blob object from stream");
+	}
+	obj_list[nr].obj = NULL;
+}
+
 static void unpack_non_delta_entry(enum object_type type, unsigned long size,
 				   unsigned nr)
 {
-	void *buf = get_data(size, dry_run);
+	void *buf;
+
+	/* Write large blob in stream without allocating full buffer. */
+	if (!dry_run && type == OBJ_BLOB && size > big_file_threshold) {
+		write_stream_blob(nr, size);
+		return;
+	}
 
+	buf = get_data(size, dry_run);
 	if (!dry_run && buf)
 		write_object(nr, type, buf, size);
 	else
object-file.c +3 -3
index 93bcfaca50..bd7631f7ef 100644
@@ -1878,9 +1878,9 @@ static const void *feed_simple_input_stream(struct input_stream *in_stream, unsi
 	return data->buf;
 }
 
-static int write_loose_object(const struct object_id *oid, char *hdr,
-			      int hdrlen, struct input_stream *in_stream,
-			      time_t mtime, unsigned flags)
+int write_loose_object(const struct object_id *oid, char *hdr,
+		       int hdrlen, struct input_stream *in_stream,
+		       time_t mtime, unsigned flags)
 {
 	int fd, ret;
 	unsigned char compressed[4096];
object-store.h +4 -0
index ccc1fc9c1a..cbd95c47e2 100644
@@ -228,6 +228,10 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf,
 		     unsigned long len, const char *type,
 		     struct object_id *oid);
 
+int write_loose_object(const struct object_id *oid, char *hdr,
+		       int hdrlen, struct input_stream *in_stream,
+		       time_t mtime, unsigned flags);
+
 int write_object_file_flags(const void *buf, unsigned long len,
 			    const char *type, struct object_id *oid,
 			    unsigned flags);
t/t5590-unpack-non-delta-objects.sh new file +76 -0

76 lines not on this page · raw message

Signature · 1 line
2.34.0.6.g676eedc724
#205 lore raw

Re: [PATCH v3 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream

Quoted 21 lines · Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".

However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "big_file_threshold" in zstream. See the following benchmarks:

    $ hyperfine \
    --prepare 'rm -rf dest.git && git init --bare dest.git' \
    'git -C dest.git unpack-objects <binary_320M.pack'
    Benchmark 1: git -C dest.git unpack-objects <binary_320M.pack
      Time (mean ± σ):     10.029 s ±  0.270 s    [User: 8.265 s, System: 1.522 s]
      Range (min … max):    9.786 s … 10.603 s    10 runs
Quoted 6 lines
    $ hyperfine \
    --prepare 'rm -rf dest.git && git init --bare dest.git' \
    'git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack'
    Benchmark 1: git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack
      Time (mean ± σ):     10.859 s ±  0.774 s    [User: 8.813 s, System: 1.898 s]
      Range (min … max):    9.884 s … 12.192 s    10 runs

It seems that you want us to compare this pair of results, and hyperfine can assist with that by including multiple benchmarks (with labels, using '-n') as follows:

$ hyperfine \
        --prepare 'rm -rf dest.git && git init --bare dest.git' \
        -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \
        -n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \
        -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'

Benchmark 1: old
Time (mean ± σ): 20.835 s ± 0.058 s [User: 14.510 s, System: 6.284 s]
Range (min … max): 20.741 s … 20.909 s 10 runs

Benchmark 2: new
Time (mean ± σ): 26.515 s ± 0.072 s [User: 19.783 s, System: 6.696 s]
Range (min … max): 26.419 s … 26.611 s 10 runs

Benchmark 3: new (small threshold)
Time (mean ± σ): 26.523 s ± 0.101 s [User: 19.805 s, System: 6.680 s]
Range (min … max): 26.416 s … 26.739 s 10 runs

Summary
'old' ran
1.27 ± 0.00 times faster than 'new'
1.27 ± 0.01 times faster than 'new (small threshold)'

(Here, 'old' is testing a compiled version of the latest 'master' branch, while 'new' has your patches applied on top.)

Notice from this example I had a pack with many small objects (mostly commits and trees) and I see that this change introduces significant overhead to this case.

It would be nice to understand this overhead and fix it before taking this change any further.

Thanks,
-Stolee

author
#206 lore raw

Re: [PATCH v3 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream

Quoted 64 lines · Derrick Stolee
Quoted 21 lines · Han Xin

From: Han Xin <[email protected]>

We used to call "get_data()" in "unpack_non_delta_entry()" to read the entire contents of a blob object, no matter how big it is. This implementation may consume all the memory and cause OOM.

By implementing a zstream version of input_stream interface, we can use a small fixed buffer for "unpack_non_delta_entry()".

However, unpack non-delta objects from a stream instead of from an entrie buffer will have 10% performance penalty. Therefore, only unpack object larger than the "big_file_threshold" in zstream. See the following benchmarks:

    $ hyperfine \
    --prepare 'rm -rf dest.git && git init --bare dest.git' \
    'git -C dest.git unpack-objects <binary_320M.pack'
    Benchmark 1: git -C dest.git unpack-objects <binary_320M.pack
      Time (mean ± σ):     10.029 s ±  0.270 s    [User: 8.265 s, System: 1.522 s]
      Range (min … max):    9.786 s … 10.603 s    10 runs
Quoted 6 lines
    $ hyperfine \
    --prepare 'rm -rf dest.git && git init --bare dest.git' \
    'git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack'
    Benchmark 1: git -c core.bigFileThreshold=2m -C dest.git unpack-objects <binary_320M.pack
      Time (mean ± σ):     10.859 s ±  0.774 s    [User: 8.813 s, System: 1.898 s]
      Range (min … max):    9.884 s … 12.192 s    10 runs

It seems that you want us to compare this pair of results, and hyperfine can assist with that by including multiple benchmarks (with labels, using '-n') as follows:

$ hyperfine \
        --prepare 'rm -rf dest.git && git init --bare dest.git' \
        -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \
        -n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \
        -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'

Benchmark 1: old
Time (mean ± σ): 20.835 s ± 0.058 s [User: 14.510 s, System: 6.284 s]
Range (min … max): 20.741 s … 20.909 s 10 runs

Benchmark 2: new
Time (mean ± σ): 26.515 s ± 0.072 s [User: 19.783 s, System: 6.696 s]
Range (min … max): 26.419 s … 26.611 s 10 runs

Benchmark 3: new (small threshold)
Time (mean ± σ): 26.523 s ± 0.101 s [User: 19.805 s, System: 6.680 s]
Range (min … max): 26.416 s … 26.739 s 10 runs

Summary
'old' ran
1.27 ± 0.00 times faster than 'new'
1.27 ± 0.01 times faster than 'new (small threshold)'

(Here, 'old' is testing a compiled version of the latest 'master' branch, while 'new' has your patches applied on top.)

Notice from this example I had a pack with many small objects (mostly commits and trees) and I see that this change introduces significant overhead to this case.

Quoted 5 lines

It would be nice to understand this overhead and fix it before taking this change any further.

Thanks,
-Stolee

Can you show me the specific information of the repository you tested, so that I can analyze it further.

I test this repository, but did not meet the problem:

Unpacking objects: 100% (18345/18345), 43.15 MiB

hyperfine \
        --prepare 'rm -rf dest.git && git init --bare dest.git' \
        -n 'old' 'git -C dest.git unpack-objects <big.pack' \
        -n 'new' 'new/git -C dest.git unpack-objects <big.pack' \
        -n 'new (small threshold)' 'new/git -c

core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack' Benchmark 1: old
Time (mean ± σ): 17.403 s ± 0.880 s [User: 4.996 s, System: 11.803 s]
Range (min … max): 15.911 s … 19.368 s 10 runs

Benchmark 2: new
Time (mean ± σ): 17.788 s ± 0.199 s [User: 5.054 s, System: 12.257 s]
Range (min … max): 17.420 s … 18.195 s 10 runs

Benchmark 3: new (small threshold)
Time (mean ± σ): 18.433 s ± 0.711 s [User: 4.982 s, System: 12.338 s]
Range (min … max): 17.518 s … 19.775 s 10 runs

Summary
'old' ran
1.02 ± 0.05 times faster than 'new'
1.06 ± 0.07 times faster than 'new (small threshold)'

Thanks,
- Han Xin

#207 lore raw

Re: [PATCH v3 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream

Quoted 31 lines · Han Xin
Quoted 30 lines · Derrick Stolee
$ hyperfine \
        --prepare 'rm -rf dest.git && git init --bare dest.git' \
        -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \
        -n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \
        -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'

Benchmark 1: old
Time (mean ± σ): 20.835 s ± 0.058 s [User: 14.510 s, System: 6.284 s]
Range (min … max): 20.741 s … 20.909 s 10 runs

Benchmark 2: new
Time (mean ± σ): 26.515 s ± 0.072 s [User: 19.783 s, System: 6.696 s]
Range (min … max): 26.419 s … 26.611 s 10 runs

Benchmark 3: new (small threshold)
Time (mean ± σ): 26.523 s ± 0.101 s [User: 19.805 s, System: 6.680 s]
Range (min … max): 26.416 s … 26.739 s 10 runs

Summary
'old' ran
1.27 ± 0.00 times faster than 'new'
1.27 ± 0.01 times faster than 'new (small threshold)'

(Here, 'old' is testing a compiled version of the latest 'master' branch, while 'new' has your patches applied on top.)

Notice from this example I had a pack with many small objects (mostly commits and trees) and I see that this change introduces significant overhead to this case.

Quoted 8 lines
Quoted 5 lines

It would be nice to understand this overhead and fix it before taking this change any further.

Thanks,
-Stolee

Can you show me the specific information of the repository you tested, so that I can analyze it further.

I used a pack-file from an internal repo. It happened to be using partial clone, so here is a repro with the git/git repository after cloning this way:

$ git clone --no-checkout --filter=blob:none https://github.com/git/git

(copy the large .pack from git/.git/objects/pack/ to big.pack)

$ hyperfine \
	--prepare 'rm -rf dest.git && git init --bare dest.git' \
	-n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \
	-n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \
	-n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'

Benchmark 1: old
Time (mean ± σ): 82.748 s ± 0.445 s [User: 50.512 s, System: 32.049 s]
Range (min … max): 82.042 s … 83.587 s 10 runs

Benchmark 2: new
Time (mean ± σ): 101.644 s ± 0.524 s [User: 67.470 s, System: 34.047 s]
Range (min … max): 100.866 s … 102.633 s 10 runs

Benchmark 3: new (small threshold)
Time (mean ± σ): 101.093 s ± 0.269 s [User: 67.404 s, System: 33.559 s]
Range (min … max): 100.639 s … 101.375 s 10 runs

Summary
'old' ran
1.22 ± 0.01 times faster than 'new (small threshold)' 1.23 ± 0.01 times faster than 'new'

I'm also able to repro this with a smaller repo (microsoft/scalar) so the tests complete much faster:

$ hyperfine \
        --prepare 'rm -rf dest.git && git init --bare dest.git' \
        -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <small.pack' \
        -n 'new' '~/_git/git/git -C dest.git unpack-objects <small.pack' \
        -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <small.pack'

Benchmark 1: old
Time (mean ± σ): 3.295 s ± 0.023 s [User: 1.063 s, System: 2.228 s]
Range (min … max): 3.269 s … 3.351 s 10 runs

Benchmark 2: new
Time (mean ± σ): 3.592 s ± 0.105 s [User: 1.261 s, System: 2.328 s]
Range (min … max): 3.378 s … 3.679 s 10 runs

Benchmark 3: new (small threshold)
Time (mean ± σ): 3.584 s ± 0.144 s [User: 1.241 s, System: 2.339 s]
Range (min … max): 3.359 s … 3.747 s 10 runs

Summary
'old' ran
1.09 ± 0.04 times faster than 'new (small threshold)' 1.09 ± 0.03 times faster than 'new'

It's not the same relative overhead, but still significant.

These pack-files contain (mostly) small objects, no large blobs. I know that's not the target of your efforts, but it would be good to avoid a regression here.

Thanks,
-Stolee

#208 lore raw

"git hyperfine" (was: [PATCH v3 5/5] unpack-objects[...])

I hadn't sent a shameless plug for my "git hyperfine" script to the list, perhaps this is a good time. It's just a thin shellscript wrapper around "hyperfine" that I wrote the other day, which...

Quoted 23 lines · Derrick Stolee

[...]
I used a pack-file from an internal repo. It happened to be using partial clone, so here is a repro with the git/git repository after cloning this way:

$ git clone --no-checkout --filter=blob:none https://github.com/git/git

(copy the large .pack from git/.git/objects/pack/ to big.pack)

$ hyperfine \
	--prepare 'rm -rf dest.git && git init --bare dest.git' \
	-n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \
	-n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \
	-n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'

Benchmark 1: old
Time (mean ± σ): 82.748 s ± 0.445 s [User: 50.512 s, System: 32.049 s]
Range (min … max): 82.042 s … 83.587 s 10 runs

Benchmark 2: new
Time (mean ± σ): 101.644 s ± 0.524 s [User: 67.470 s, System: 34.047 s]
Range (min … max): 100.866 s … 102.633 s 10 runs

Quoted 8 lines

Benchmark 3: new (small threshold)
Time (mean ± σ): 101.093 s ± 0.269 s [User: 67.404 s, System: 33.559 s]
Range (min … max): 100.639 s … 101.375 s 10 runs

Summary
'old' ran
1.22 ± 0.01 times faster than 'new (small threshold)' 1.23 ± 0.01 times faster than 'new'

...adds enough sugar around "hyperfine" itself to do this as e.g. (the "-s" is a feature I submitted to hyperfine itself, it's not in a release yet[1], but in this case you could also use "-p"):

    git hyperfine -L rev v2.20.0,origin/master \
        -s 'if ! test -d redis.git; then git clone --bare --filter=blob:none https://github.com/redis/redis; fi && make' \
        -p 'rm -rf dest.git; git init --bare dest.git' \
        './git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)'

The sugar being that for each named "rev" parameter it'll set up "git worktree" for you, so under the hood each of those is chdir-ing to the respective revision of:

    $ git worktree list
    [...]
    /run/user/1001/git-hyperfine/origin/master  abe6bb39053 (detached HEAD)
    /run/user/1001/git-hyperfine/v2.33.0        225bc32a989 (detached HEAD)

That they're named revisions and not git-rev-parse'd is intentional, since you'll benefit from faster incremental "make" (even if using "ccache"). I'm typically benchmarking HEAD~1,HEAD~0.

The output will then use those "rev" parameters, and be e.g.:

    Benchmark 1: ./git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'v2.20.0
      Time (mean ± σ):      6.678 s ±  0.046 s    [User: 4.525 s, System: 2.117 s]
      Range (min … max):    6.619 s …  6.765 s    10 runs
     
    Benchmark 2: ./git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'origin/master
      Time (mean ± σ):      6.756 s ±  0.074 s    [User: 4.586 s, System: 2.134 s]
      Range (min … max):    6.691 s …  6.941 s    10 runs
     
    Summary
      './git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'v2.20.0' ran
        1.01 ± 0.01 times faster than './git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'origin/master'

I think if you're routinely benchmarking N different git versions you'll find it handy, it also has configurable hook support (using git config), so e.g. it's easy to copy your config.mak in-place in the worktrees. E.g. my config is:

    $ git -P config --get-regexp '^hyperfine'
    hyperfine.run-dir $XDG_RUNTIME_DIR/git-hyperfine
    hyperfine.xargs-options -r
    hyperfine.hook.setup ~/g/git.meta/config.mak.sh

It's hosted at https://github.com/avar/git-hyperfine/ and
https://gitlab.com/avar/git-hyperfine/; It's implemented in (portable) POSIX shell script.

There's surely some bugs in it, one known one is that unlike hyperfine it doesn't accept there being spaces in the parameters to -L, because I'm screwing up some quoting-within-quoting in the (shellscript) implementation (suggestions for that particular one most welcome).

I hacked it up after this suggestion from Jeff King[2] of moving t/perf over to it.

I haven't done any of that legwork, but I think a wrapper like "git-hyperfine" that prepares worktrees for the N revisions we're benchmarking is a good direction to go in.

We don't use git-worktrees in t/perf, but probably could for most/all tests. In any case it would be easy to have the script setup the revs to be benchmarked in some hookable custom manner to have it do exactly what t/perf/run is doing now.

1. https://github.com/sharkdp/hyperfine/commit/017d55a
2. https://lore.kernel.org/git/[email protected]/

author
#209 lore raw

Re: [PATCH v3 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream

In reply to Derrick Stolee
Quoted 58 lines · Derrick Stolee

I used a pack-file from an internal repo. It happened to be using partial clone, so here is a repro with the git/git repository after cloning this way:

$ git clone --no-checkout --filter=blob:none https://github.com/git/git

(copy the large .pack from git/.git/objects/pack/ to big.pack)

$ hyperfine \
        --prepare 'rm -rf dest.git && git init --bare dest.git' \
        -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \
        -n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \
        -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack'

Benchmark 1: old
Time (mean ± σ): 82.748 s ± 0.445 s [User: 50.512 s, System: 32.049 s]
Range (min … max): 82.042 s … 83.587 s 10 runs

Benchmark 2: new
Time (mean ± σ): 101.644 s ± 0.524 s [User: 67.470 s, System: 34.047 s]
Range (min … max): 100.866 s … 102.633 s 10 runs

Benchmark 3: new (small threshold)
Time (mean ± σ): 101.093 s ± 0.269 s [User: 67.404 s, System: 33.559 s]
Range (min … max): 100.639 s … 101.375 s 10 runs

Summary
'old' ran
1.22 ± 0.01 times faster than 'new (small threshold)' 1.23 ± 0.01 times faster than 'new'

I'm also able to repro this with a smaller repo (microsoft/scalar) so the tests complete much faster:

$ hyperfine \
        --prepare 'rm -rf dest.git && git init --bare dest.git' \
        -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <small.pack' \
        -n 'new' '~/_git/git/git -C dest.git unpack-objects <small.pack' \
        -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <small.pack'

Benchmark 1: old
Time (mean ± σ): 3.295 s ± 0.023 s [User: 1.063 s, System: 2.228 s]
Range (min … max): 3.269 s … 3.351 s 10 runs

Benchmark 2: new
Time (mean ± σ): 3.592 s ± 0.105 s [User: 1.261 s, System: 2.328 s]
Range (min … max): 3.378 s … 3.679 s 10 runs

Benchmark 3: new (small threshold)
Time (mean ± σ): 3.584 s ± 0.144 s [User: 1.241 s, System: 2.339 s]
Range (min … max): 3.359 s … 3.747 s 10 runs

Summary
'old' ran
1.09 ± 0.04 times faster than 'new (small threshold)' 1.09 ± 0.03 times faster than 'new'

Quoted 8 lines

It's not the same relative overhead, but still significant.

These pack-files contain (mostly) small objects, no large blobs. I know that's not the target of your efforts, but it would be good to avoid a regression here.

Thanks,
-Stolee

With your help, I did catch this performance problem, which was introduced in this patch:
https://lore.kernel.org/git/[email protected]/

This patch changes the original data reading ino to stream reading, but its problem is that even for the original reading of the whole object data, it still generates an additional git_deflate() and subsequent transfer.

I will fix it in a follow-up patch.

Thanks,
-Han Xin

#210 lore raw

Re: [PATCH v3 5/5] unpack-objects: unpack_non_delta_entry() read data in a stream

Quoted 8 lines · Han Xin
Quoted 6 lines · Derrick Stolee

These pack-files contain (mostly) small objects, no large blobs. I know that's not the target of your efforts, but it would be good to avoid a regression here.

Thanks,
-Stolee

Quoted 7 lines

With your help, I did catch this performance problem, which was introduced in this patch:
https://lore.kernel.org/git/[email protected]/

This patch changes the original data reading ino to stream reading, but its problem is that even for the original reading of the whole object data, it still generates an additional git_deflate() and subsequent transfer.

I'm glad you found it!

Quoted 1 line

I will fix it in a follow-up patch.

Looking forward to it.

Thanks,
-Stolee