pack-objects: trace pack bytes written
PATCH 1/1 pack-objects: trace pack bytes written
From: Friel <[email protected]>
We want to measure how compression settings affect push performance on the client. Different settings can produce different-sized packs from the same objects. Trace2 records the object count, but we also need the pack size to compare those settings.
Add a write_pack_file/wrote_bytes Trace2 datum alongside write_pack_file/wrote. Count packs written to stdout or disk, including each pack's header and trailing checksum. When pack.packSizeLimit splits the output, report the sum of the pack sizes.
--- builtin/pack-objects.c | 7 +++++++ t/t5300-pack-object.sh | 24 ++++++++++++++++++++++++ 2 files changed, 31 insertions(+)
builtin/pack-objects.c +7 -0
@@ -1337,6 +1337,7 @@ static void write_pack_file(void)uint32_t nr_remaining = nr_result;time_t last_mtime =0 ;struct object_entry **write_order; +off_t bytes_written =0 ;if (progress > pack_to_stdout) progress_state = start_progress(the_repository,
@@ -1347,6 +1348,7 @@ static void write_pack_file(void) do { unsigned char hash[GIT_MAX_RAWSZ]; char *pack_tmp_name = NULL; + off_t pack_bytes; if (pack_to_stdout) { /*
@@ -1389,6 +1391,8 @@ static void write_pack_file(void)display_progress (progress_state, written); } + pack_bytes =hashfile_total (f) + + the_repository->hash_algo->rawsz;if (pack_to_stdout) {/* * We never fsync when writing to stdout since we may
@@ -1419,6 +1423,7 @@ static void write_pack_file(void) write_bitmap_index =0 ; } } + bytes_written += pack_bytes;if (!pack_to_stdout) {struct stat st;
@@ -1510,6 +1515,8 @@ static void write_pack_file(void) written, nr_result);trace2_data_intmax ("pack-objects" , the_repository,"write_pack_file/wrote" , nr_result); +trace2_data_intmax ("pack-objects" , the_repository, +"write_pack_file/wrote_bytes" , bytes_written); }static int no_try_delta (const char *path)
t/t5300-pack-object.sh +24 -0
@@ -33,6 +33,30 @@ test_expect_success 'setup' ' } >expect ' +test_expect_success 'pack-object traces bytes written to stdout' ' + test_when_finished "rm -f pack.trace pack.pack" && + GIT_TRACE2_EVENT="$PWD/pack.trace" \ + git pack-objects --quiet --revs --stdout >pack.pack <<-EOF && + $commit + EOF + bytes=$(test_file_size pack.pack) && + test_grep "\"key\":\"write_pack_file/wrote_bytes\",\"value\":\"$bytes\"" pack.trace +' + +test_expect_success 'pack-object traces bytes written to split pack files' ' + test_when_finished "rm -f split.trace traced-pack-*" && + GIT_TRACE2_EVENT="$PWD/split.trace" \ + git -c pack.packSizeLimit=3m pack-objects --quiet traced-pack <obj-list && + test 2 = $(ls traced-pack-*.pack | wc -l) && + bytes=0 && + for pack in traced-pack-*.pack + do + pack_size=$(test_file_size "$pack") && + bytes=$((bytes + pack_size)) || return 1 + done && + test_grep "\"key\":\"write_pack_file/wrote_bytes\",\"value\":\"$bytes\"" split.trace +' + test_expect_success 'setup pack-object <stdin' ' git init pack-object-stdin && test_commit -C pack-object-stdin one &&
base-commit: 18e66859d87fb4b76599f73460b54f0848c76b16
Quoted 48 lines · [email protected] writes:
From: Friel <[email protected]>
We want to measure how compression settings affect push performance on the client. Different settings can produce different-sized packs from the same objects. Trace2 records the object count, but we also need the pack size to compare those settings.
Add a write_pack_file/wrote_bytes Trace2 datum alongside write_pack_file/wrote. Count packs written to stdout or disk, including each pack's header and trailing checksum. When pack.packSizeLimit splits the output, report the sum of the pack sizes.
Signed-off-by Friel <[email protected]>--- builtin/pack-objects.c | 7 +++++++ t/t5300-pack-object.sh | 24 ++++++++++++++++++++++++ 2 files changed, 31 insertions(+)builtin/pack-objects.c +4
-0@@ -1337,6 +1337,7 @@ static void write_pack_file(void)uint32_t nr_remaining = nr_result;time_t last_mtime =0 ;struct object_entry **write_order; +off_t bytes_written =0 ;if (progress > pack_to_stdout) progress_state = start_progress(the_repository,@@ -1347,6 +1348,7 @@ static void write_pack_file(void) do { unsigned char hash[GIT_MAX_RAWSZ]; char *pack_tmp_name = NULL; + off_t pack_bytes; if (pack_to_stdout) { /*@@ -1389,6 +1391,8 @@ static void write_pack_file(void)display_progress (progress_state, written); } + pack_bytes =hashfile_total (f) + + the_repository->hash_algo->rawsz;if (pack_to_stdout) {/* * We never fsync when writing to stdout since we may
builtin/pack-objects.c static void write_pack_file(void) · line 1426
Quoted 4 lines
builtin/pack-objects.c +1
-0@@ -1419,6 +1423,7 @@ static void write_pack_file(void) write_bitmap_index =0 ; } } + bytes_written += pack_bytes;
I may very well be misreading the code, but it is unclear to me what role pack_bytes is playing, why we want to compute it before the finialization if/else cascade above, and increment bytes_written after that finalization if/else cascade above.
IOW, wouldn't it be equivalent to get rid of hunks 1347 and 1419, and in hunk 1389 to this instead?
bytes_written += hashfile_total(f) + the_hash_algo->rawsz;
The numbers for non stdout case are not that interesting (we can see how bit the on-disk files are very easily), but counting in the common code path (i.e., hunk 1389) sounds like the cleanest approach. I just found that the code with two variables confusing.
Thanks.
PATCH v2 1/1 pack-objects: trace pack bytes written
From: Friel <[email protected]>
We want to measure how compression settings affect push performance on the client. Different settings can produce different-sized packs from the same objects. Trace2 records the object count, but we also need the pack size to compare those settings.
Add a write_pack_file/wrote_bytes Trace2 datum alongside write_pack_file/wrote. Count packs written to stdout or disk, including each pack's header and trailing checksum. When pack.packSizeLimit splits the output, report the sum of the pack sizes.
---
Junio, you're right. Updating bytes_written before finalization is equivalent. I've dropped pack_bytes; everything else is unchanged. Thanks.
builtin/pack-objects.c | 5 +++++ t/t5300-pack-object.sh | 24 ++++++++++++++++++++++++ 2 files changed, 29 insertions(+)
builtin/pack-objects.c +5 -0
@@ -1337,6 +1337,7 @@ static void write_pack_file(void)uint32_t nr_remaining = nr_result;time_t last_mtime =0 ;struct object_entry **write_order; +off_t bytes_written =0 ;if (progress > pack_to_stdout) progress_state = start_progress(the_repository,
@@ -1389,6 +1390,8 @@ static void write_pack_file(void)display_progress (progress_state, written); } + bytes_written +=hashfile_total (f) + + the_repository->hash_algo->rawsz;if (pack_to_stdout) {/* * We never fsync when writing to stdout since we may
@@ -1510,6 +1513,8 @@ static void write_pack_file(void) written, nr_result);trace2_data_intmax ("pack-objects" , the_repository,"write_pack_file/wrote" , nr_result); +trace2_data_intmax ("pack-objects" , the_repository, +"write_pack_file/wrote_bytes" , bytes_written); }static int no_try_delta (const char *path)
t/t5300-pack-object.sh +24 -0
@@ -33,6 +33,30 @@ test_expect_success 'setup' ' } >expect ' +test_expect_success 'pack-object traces bytes written to stdout' ' + test_when_finished "rm -f pack.trace pack.pack" && + GIT_TRACE2_EVENT="$PWD/pack.trace" \ + git pack-objects --quiet --revs --stdout >pack.pack <<-EOF && + $commit + EOF + bytes=$(test_file_size pack.pack) && + test_grep "\"key\":\"write_pack_file/wrote_bytes\",\"value\":\"$bytes\"" pack.trace +' + +test_expect_success 'pack-object traces bytes written to split pack files' ' + test_when_finished "rm -f split.trace traced-pack-*" && + GIT_TRACE2_EVENT="$PWD/split.trace" \ + git -c pack.packSizeLimit=3m pack-objects --quiet traced-pack <obj-list && + test 2 = $(ls traced-pack-*.pack | wc -l) && + bytes=0 && + for pack in traced-pack-*.pack + do + pack_size=$(test_file_size "$pack") && + bytes=$((bytes + pack_size)) || return 1 + done && + test_grep "\"key\":\"write_pack_file/wrote_bytes\",\"value\":\"$bytes\"" split.trace +' + test_expect_success 'setup pack-object <stdin' ' git init pack-object-stdin && test_commit -C pack-object-stdin one &&
Interdiff against v1:
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index bbf1adb437..252530172c 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1348,7 +1348,6 @@ static void write_pack_file(void)
do {
unsigned char hash[GIT_MAX_RAWSZ]; char *pack_tmp_name = NULL;
- off_t pack_bytes;
if (pack_to_stdout) {
/*
@@ -1391,7 +1390,7 @@ static void write_pack_file(void)
display_progress(progress_state, written);
} - pack_bytes = hashfile_total(f) +
+ bytes_written += hashfile_total(f) +
the_repository->hash_algo->rawsz;
if (pack_to_stdout) {
/*
@@ -1423,7 +1422,6 @@ static void write_pack_file(void)
write_bitmap_index = 0;
}
}
- bytes_written += pack_bytes; if (!pack_to_stdout) {
struct stat st;base-commit: 18e66859d87fb4b76599f73460b54f0848c76b16
builtin/pack-objects.c static void write_pack_file(void) · line 1397
Quoted 13 lines · On Wed, Aug 19, 2026 at 04:28:10PM -0700, [email protected] wrote:
builtin/pack-objects.c +2
-0@@ -1389,6 +1390,8 @@ static void write_pack_file(void)display_progress (progress_state, written); } + bytes_written +=hashfile_total (f) + + the_repository->hash_algo->rawsz;if (pack_to_stdout) {/* * We never fsync when writing to stdout since we may
I guess the addition here accounts for the trailing hash written by the hashfile. If so, shouldn't we also use the algortihm that the hashfile uses in the first place via f->algop->rawsz?
builtin/pack-objects.c static void write_pack_file(void) · line 1520
Quoted 9 lines
builtin/pack-objects.c +2
-0@@ -1510,6 +1513,8 @@ static void write_pack_file(void) written, nr_result);trace2_data_intmax ("pack-objects" , the_repository,"write_pack_file/wrote" , nr_result); +trace2_data_intmax ("pack-objects" , the_repository, +"write_pack_file/wrote_bytes" , bytes_written); }static int no_try_delta (const char *path)
The "write_pack_file/wrote" event is quite awkwardly named, if you ask me, as it's not immediately obvious what exactly it's counting, and the second metric may make this even more confusing. In retrospect it would've been preferable to call this "wrote_objects" to clarify.
I don't really think we guarantee any kind of stability around those traces, so we could in theory change it here, too. But I don't feel like my argument is strong enough to really warrant such a change, so maybe we should just leave it as-is.
Thanks!
Patrick
Quoted 9 lines · Patrick Steinhardt
Quoted 8 lines · On Wed, Aug 19, 2026 at 04:28:10PM -0700, [email protected] wrote:
builtin/pack-objects.c +0
-0@@ -1389,6 +1390,8 @@ static void write_pack_file(void)display_progress (progress_state, written); }
Quoted 9 lines
builtin/pack-objects.c static void write_pack_file(void) · line 1397
Quoted 5 lines
builtin/pack-objects.c +2
-0+ bytes_written +=hashfile_total (f) + + the_repository->hash_algo->rawsz;if (pack_to_stdout) {/* * We never fsync when writing to stdout since we may I guess the addition here accounts for the trailing hash written by the hashfile. If so, shouldn't we also use the algortihm that the hashfile uses in the first place via
f->algop->rawsz?
Perhaps, though that is used to write the hash (via CSUM_HASH_IN_STREAM) only in two of the conditional blocks. In the third we finalize the hashfile and then use fixup_pack_header_footer(), passing the_hash_algo directly (not even the_repository->hash_algo, though of course they mean the same thing).
It all works out, of course, because we created the hashfile struct earlier using the_repository->hash_algo. So I think this is mostly academic in the first place, but your suggestion harmonizes two of the conditional blocks while creating disagreement with the third.
I think something like this would "fix" it by consistently using the hashfile's algo in all three blocks:
builtin/pack-objects.c +2 -2
@@ -1413,9 +1413,9 @@ static void write_pack_file(void)* If we wrote the wrong number of entries in the * header, rewrite it like in fast-import. */ - +const struct git_hash_algo *algo = f->algop;int fd =finalize_hashfile (f, hash,FSYNC_COMPONENT_PACK ,0 ); -fixup_pack_header_footer (the_hash_algo, fd, hash, +fixup_pack_header_footer (algo, fd, hash, pack_tmp_name, nr_written, hash, offset);close (fd);
But there's a subtle yet interesting difference here! f->algop won't necessarily be the same pointer as the_hash_algo. If we compiled with an unsafe variant, that will be used for hashfiles. If we're just looking at rawsz that's OK; the two variants should be identical (other than performance and collision detection), so taking rawsz from either is fine.
But fixup_pack_header_footer() actually recomputes the hash (as it must if we tweak the header). Right now it does it using the "normal" variant, but we should be able to use the unsafe one (which my diff snippet above would start to do).
Of course this whole thing is absurdly pessimal in the first place. If we are just going to throw out the hashfile's checksum, then why bother computing it in the first place? Because we don't trust a disk write at all, and actually verify the original hash computation as we read the bytes back in! So we'll actually sha1 the written packfile three times. Yikes. I wonder if it's really worth being so paranoid. But that is how it has always been.
Anyway, that is a bit of a tangent from the patch in question. I think either spelling is OK for the purposes of this patch. If somebody wants to pursue harmonizing the paths (and maybe even doing some timings to see if switching to the unsafe variant is noticeable here, and what the total cost of this triple-write approach is), that can happen separately.
-Peff
Quoted 59 lines · Jeff King
Quoted 9 lines · Patrick Steinhardt
Quoted 8 lines · On Wed, Aug 19, 2026 at 04:28:10PM -0700, [email protected] wrote:
builtin/pack-objects.c +0
-0@@ -1389,6 +1390,8 @@ static void write_pack_file(void)display_progress (progress_state, written); }Quoted 9 lines
builtin/pack-objects.c static void write_pack_file(void) · line 1397
Quoted 5 lines
builtin/pack-objects.c +2
-0+ bytes_written +=hashfile_total (f) + + the_repository->hash_algo->rawsz;if (pack_to_stdout) {/* * We never fsync when writing to stdout since we may I guess the addition here accounts for the trailing hash written by the hashfile. If so, shouldn't we also use the algortihm that the hashfile uses in the first place via
f->algop->rawsz?Perhaps, though that is used to write the hash (via
CSUM_HASH_IN_STREAM) only in two of the conditional blocks. In the third we finalize the hashfile and then usefixup_pack_header_footer(), passingthe_hash_algodirectly (not eventhe_repository->hash_algo, though of course they mean the same thing).It all works out, of course, because we created the hashfile struct earlier using
the_repository->hash_algo. So I think this is mostly academic in the first place, but your suggestion harmonizes two of the conditional blocks while creating disagreement with the third.I think something like this would "fix" it by consistently using the hashfile's algo in all three blocks:
builtin/pack-objects.c +2
-2@@ -1413,9 +1413,9 @@ static void write_pack_file(void)* If we wrote the wrong number of entries in the * header, rewrite it like in fast-import. */ - +const struct git_hash_algo *algo = f->algop;int fd =finalize_hashfile (f, hash,FSYNC_COMPONENT_PACK ,0 ); -fixup_pack_header_footer (the_hash_algo, fd, hash, +fixup_pack_header_footer (algo, fd, hash, pack_tmp_name, nr_written, hash, offset);close (fd);But there's a subtle yet interesting difference here!
f->algopwon't necessarily be the same pointer asthe_hash_algo. If we compiled with an unsafe variant, that will be used for hashfiles. If we're just looking at rawsz that's OK; the two variants should be identical (other than performance and collision detection), so taking rawsz from either is fine.
Quoted 4 lines
But
fixup_pack_header_footer()actually recomputes the hash (as it must if we tweak the header). Right now it does it using the "normal" variant, but we should be able to use the unsafe one (which my diff snippet above would start to do).
Yeah, I agree that switching over to the unsafe algortihm is sensible. Being able to speed up hashing of packfiles was one of the prime motivations of introducing the unsafe variants in the first place, so the fact that we still use the safe variant here feels like a plain oversight to me.
Quoted 7 lines
Of course this whole thing is absurdly pessimal in the first place. If we are just going to throw out the hashfile's checksum, then why bother computing it in the first place? Because we don't trust a disk write at all, and actually verify the original hash computation as we read the bytes back in! So we'll actually sha1 the written packfile three times. Yikes. I wonder if it's really worth being so paranoid. But that is how it has always been.
That's... awful. Honestly, if we cannot trust what we're writing to disk we're going to be kind of screwed anyway. We don't re-verify loose objects, refs or whatever other data structures we write to disk either. So doing this thrice here feels wrong.
Quoted 6 lines
Anyway, that is a bit of a tangent from the patch in question. I think either spelling is OK for the purposes of this patch. If somebody wants to pursue harmonizing the paths (and maybe even doing some timings to see if switching to the unsafe variant is noticeable here, and what the total cost of this triple-write approach is), that can happen separately.
I agree that this is definitely out of scope of this patch series.
Patrick
Quoted 12 lines · Patrick Steinhardt
Quoted 11 lines
But there's a subtle yet interesting difference here!
f->algopwon't necessarily be the same pointer asthe_hash_algo. If we compiled with an unsafe variant, that will be used for hashfiles. If we're just looking at rawsz that's OK; the two variants should be identical (other than performance and collision detection), so taking rawsz from either is fine.But
fixup_pack_header_footer()actually recomputes the hash (as it must if we tweak the header). Right now it does it using the "normal" variant, but we should be able to use the unsafe one (which my diff snippet above would start to do).
Quoted 5 lines
Yeah, I agree that switching over to the unsafe algortihm is sensible. Being able to speed up hashing of packfiles was one of the prime motivations of introducing the unsafe variants in the first place, so the fact that we still use the safe variant here feels like a plain oversight to me.
Yes, though I think the oversight can be forgiven here. The unsafe variants are purely for performance, so we started by converting a few hot code paths, knowing that it was OK to leave other spots using the collision-detecting implementation. The main one we cared about is "pack-objects --stdout" to serve fetches.
But this particular case is almost never exercised! It triggers only when --max-pack-size causes us to split the result into multiple packs (we can't write the header up front in that case, because we don't know how many objects we'll fit into the output). So I doubt anybody would have noticed or cared about the performance difference.
But it also means that cleaning up the triple-hash is tricky. The three hashes in this code path are:
a. we hash as we write, via struct hashfile
b. we hash as we read back the data to verify it
c. we re-hash the data on top of the fixed-up header
We can obviously drop (b) if we choose. We can't drop (c); it's the final value that goes into the on-disk packfile. So we'd like to drop (a), which is pointless (except for cross-checking step b).
But we don't know if we're in this code path until we've finished writing the file! If the output is smaller than --max-pack-size, then we just write the hash from (a) directly, and neither (b) nor (c) happens at all. This is the "nr_written == nr_remaining" conditional, the second in the chain.
We could pessimistically assume that we'll need to do (c), and skip the hash for (a). But that is worse for the usual case that we don't split the packfiles. Instead of hashing as the data is written, we have to re-read it (passing all those bytes through memory again).
So realistically the best we can do is drop (b).
Of course what I'd _really_ like to do is rip out --max-pack-size entirely. I don't think it's generally helpful, and it introduces all kinds of weird corner cases and complications like this. But obviously that's a much bigger change, and naturally if I seriously proposed it somebody would come out of the woodwork so with obscure case where it's useful.
Quoted 8 lines
Quoted 7 lines
Of course this whole thing is absurdly pessimal in the first place. If we are just going to throw out the hashfile's checksum, then why bother computing it in the first place? Because we don't trust a disk write at all, and actually verify the original hash computation as we read the bytes back in! So we'll actually sha1 the written packfile three times. Yikes. I wonder if it's really worth being so paranoid. But that is how it has always been.
Quoted 4 lines
That's... awful. Honestly, if we cannot trust what we're writing to disk we're going to be kind of screwed anyway. We don't re-verify loose objects, refs or whatever other data structures we write to disk either. So doing this thrice here feels wrong.
There was an attitude in the early days of Git that we should be checking hashes and checksums all the time. I.e., that the validity of the data was the most precious thing, and we should notice an on-disk corruption as quickly and reliably as possible, similar to filesystems that checksum the data.
But over time we've relaxed that quite a bit because of the quite noticeable costs. For example, we used to re-hash every object we loaded, but these days we have PARSE_OBJECT_SKIP_HASH_CHECK, and features like the commit graph.
I think this is a case where we could similarly relax. Especially because this is just the pack checksum. The actual object contents are still protected by their respective hashes.
-Peff
Quoted 6 lines · Jeff King
Of course what I'd _really_ like to do is rip out --max-pack-size entirely. I don't think it's generally helpful, and it introduces all kinds of weird corner cases and complications like this. But obviously that's a much bigger change, and naturally if I seriously proposed it somebody would come out of the woodwork so with obscure case where it's useful.
;-) Perhaps Git 3.0 boundary?
Quoted 3 lines
I think this is a case where we could similarly relax. Especially because this is just the pack checksum. The actual object contents are still protected by their respective hashes.
Yes. Dropping the "(b) validate as we re-read" step is a reasonable thing to do with the least disruption from that viewpoint.
Thanks.
Quoted 18 lines · Jeff King
builtin/pack-objects.c +2
-2@@ -1413,9 +1413,9 @@ static void write_pack_file(void)* If we wrote the wrong number of entries in the * header, rewrite it like in fast-import. */ - +const struct git_hash_algo *algo = f->algop;int fd =finalize_hashfile (f, hash,FSYNC_COMPONENT_PACK ,0 ); -fixup_pack_header_footer (the_hash_algo, fd, hash, +fixup_pack_header_footer (algo, fd, hash, pack_tmp_name, nr_written, hash, offset);close (fd);
Quoted 4 lines
But
fixup_pack_header_footer()actually recomputes the hash (as it must if we tweak the header). Right now it does it using the "normal" variant, but we should be able to use the unsafe one (which my diff snippet above would start to do).
I am amused. This is an interesting find.
Thanks.
Quoted 12 lines · On Wed, Aug 19, 2026 at 04:28:10PM -0700, [email protected] wrote:
From: Friel <[email protected]>
We want to measure how compression settings affect push performance on the client. Different settings can produce different-sized packs from the same objects. Trace2 records the object count, but we also need the pack size to compare those settings.
Add a write_pack_file/wrote_bytes Trace2 datum alongside write_pack_file/wrote. Count packs written to stdout or disk, including each pack's header and trailing checksum. When pack.packSizeLimit splits the output, report the sum of the pack sizes.
Quoted 5 lines
Signed-off-by Friel <[email protected]>---
Junio, you're right. Updatingbytes_writtenbefore finalization is equivalent. I've droppedpack_bytes; everything else is unchanged. Thanks.
The downthread discussion went pretty far off-topic, so for those who do not want to read it, the summary is: this patch looks good to me. ;)
-Peff
Quoted 14 lines · Jeff King
Quoted 12 lines · On Wed, Aug 19, 2026 at 04:28:10PM -0700, [email protected] wrote:
From: Friel <[email protected]>
We want to measure how compression settings affect push performance on the client. Different settings can produce different-sized packs from the same objects. Trace2 records the object count, but we also need the pack size to compare those settings.
Add a write_pack_file/wrote_bytes Trace2 datum alongside write_pack_file/wrote. Count packs written to stdout or disk, including each pack's header and trailing checksum. When pack.packSizeLimit splits the output, report the sum of the pack sizes.
Quoted 8 lines
Quoted 5 lines
Signed-off-by Friel <[email protected]>---
Junio, you're right. Updatingbytes_writtenbefore finalization is equivalent. I've droppedpack_bytes; everything else is unchanged. Thanks.The downthread discussion went pretty far off-topic, so for those who do not want to read it, the summary is: this patch looks good to me. ;)
It looks good to me, too. Thanks, all.