index-pack: optionally allow duplicate objects
RFC PATCH 1/1 index-pack: optionally allow duplicate objects
From: Friel <[email protected]>
index-pack accepts repeated object IDs by default. However, --check-self-contained-and-connected also enables strict index validation, which rejects them. A clone therefore rejects a pack containing duplicate objects even when all of its objects are connected.
By default, shallow and filtered clones do not request this pack-local check and already accept duplicate objects. Ordinary full clones reject the same pack because they enable strict index validation.
Git's upload-pack normally uses pack-objects to select each reachable object once before writing a response. A server can instead construct that response by streaming entries from existing packs. When those packs overlap, the same object can appear more than once.
Avoiding duplicates requires the producers to coordinate their object selection or track object IDs across all input packs. A duplicate can also be used as a delta base, so removing it can require buffering and rewriting the response. Doing that work at request time gives up the memory and latency benefits of streaming existing packs.
Add pack.allowDuplicateObjects so a client can accept these packs during its initial fetch:
git clone -c pack.allowDuplicateObjects <repository>
Add --[no-]allow-duplicate-objects to override the configuration for index-pack. Preserve the default rejection introduced by 68be2fea50 (receive-pack, fetch-pack: reject bogus pack that records objects twice, 2011-11-16). An explicit --strict or --verify remains strict regardless of the configuration and cannot be combined with --allow-duplicate-objects. Object validation, connectivity checks, and delta resolution remain unchanged.
---
Applies on top of tb/pack-with-duplicates.
Documentation/config/pack.adoc | 11 ++ Documentation/git-index-pack.adoc | 12 ++- builtin/index-pack.c | 41 ++++++- t/t5308-pack-detect-duplicates.sh | 171 ++++++++++++++++++++++++++++++ t/t5309-pack-delta-cycles.sh | 10 ++ 5 files changed, 241 insertions(+), 4 deletions(-)
Documentation/config/pack.adoc +11 -0
@@ -39,6 +39,17 @@ is set to "multi", reuse parts of just the bitmapped packfile. This can reduce memory and CPU usage to serve fetches, but might result in sending a slightly larger pack. Defaults to true. +pack.allowDuplicateObjects:: + Allow linkgit:git-index-pack[1] to accept a pack containing + multiple copies of the same object while checking that the pack + is self-contained and connected. For example, + `git clone -c pack.allowDuplicateObjects <repository>` can accept + a pack generated from overlapping existing packs. Object and + connectivity checks are preserved. Explicit `--strict` and + `--verify` continue to reject duplicate objects. + `--no-allow-duplicate-objects` overrides this setting. + Defaults to `false`. + pack.island:: An extended regular expression configuring a set of delta islands. See "DELTA ISLANDS" in linkgit:git-pack-objects[1]
Documentation/git-index-pack.adoc +11 -1
@@ -11,7 +11,9 @@ SYNOPSIS [verse] 'git index-pack' [-v] [-o <index-file>] [--[no-]rev-index] <pack-file> 'git index-pack' --stdin [--fix-thin] [--keep] [-v] [-o <index-file>] - [--[no-]rev-index] [<pack-file>] + [--[no-]rev-index] + [--[no-]allow-duplicate-objects] + [<pack-file>] DESCRIPTION
@@ -97,6 +99,14 @@ default and "Indexing objects" when `--stdin` is specified. --check-self-contained-and-connected:: Die if the pack contains broken links. For internal use only. +--allow-duplicate-objects:: +--no-allow-duplicate-objects:: + Allow or reject multiple copies of the same object while checking + that the pack is self-contained and connected. The default is + controlled by `pack.allowDuplicateObjects`. The command-line + option overrides the configuration. `--allow-duplicate-objects` + cannot be combined with `--strict` or `--verify`. + --fsck-objects[=<msg-id>=<severity>...]:: Die if the pack contains broken objects, but unlike `--strict`, don't choke on broken links. If the pack contains a tree pointing to a
builtin/index-pack.c +38 -3
84 lines not on this page · raw message
t/t5308-pack-detect-duplicates.sh +171 -0
175 lines not on this page · raw message
t/t5309-pack-delta-cycles.sh +10 -0
@@ -217,6 +217,16 @@ test_expect_success 'failover after a tail into a three-object delta cycle' ' check_blob "$T" tail ' +test_expect_success 'configured connectivity check recovers duplicate delta bases' ' + clear_packs && + git -c pack.allowDuplicateObjects index-pack --fix-thin \ + --check-self-contained-and-connected \ + --stdin <recoverable-1.pack && + printf "\7\0" >expect.duplicate-base && + check_blob "$A" expect.duplicate-base && + git cat-file -e "$B" +' + test_expect_success 'index-pack works with thin pack A->B->C with B on disk' ' git init server && (
base-commit: fd2739b159d075cd4c6fa69b2cd876ba1caa5c88
Quoted 10 lines · On Mon, Jul 27, 2026 at 09:25:32PM -0700, [email protected] wrote:
Git's upload-pack normally uses pack-objects to select each reachable object once before writing a response. A server can instead construct that response by streaming entries from existing packs. When those packs overlap, the same object can appear more than once.
Avoiding duplicates requires the producers to coordinate their object selection or track object IDs across all input packs. A duplicate can also be used as a delta base, so removing it can require buffering and rewriting the response. Doing that work at request time gives up the memory and latency benefits of streaming existing packs.
Right. An out-of-tree implementation of upload-pack may choose to stitch multiple individual packs together by concatenating them, trading some pack generation time for a pack which may contain duplicate objects.
While this series is primarily motivated by that use-case, I suspect that there are optimizations we could make within Git's implementation of upload-pack that would take advantage of environments where clients are prepared to accept packs that contain duplicate objects.
That's not a goal of this patch, of course, but something to keep in mind as others review this.
Quoted 18 lines
Applies on top of tb/pack-with-duplicates.
Documentation/config/pack.adoc | 11 ++ Documentation/git-index-pack.adoc | 12 ++- builtin/index-pack.c | 41 ++++++- t/t5308-pack-detect-duplicates.sh | 171 ++++++++++++++++++++++++++++++ t/t5309-pack-delta-cycles.sh | 10 ++ 5 files changed, 241 insertions(+), 4 deletions(-)Documentation/config/pack.adoc +1
-0@@ -39,6 +39,17 @@ is set to "multi", reuse parts of just the bitmapped packfile. This can reduce memory and CPU usage to serve fetches, but might result in sending a slightly larger pack. Defaults to true. +pack.allowDuplicateObjects::
Documentation/config/pack.adoc is set to "multi", reuse parts of just the bitmapped packfile. This · line 52
Quoted 10 lines
Documentation/config/pack.adoc +10
-0+ Allow linkgit:git-index-pack[1] to accept a pack containing + multiple copies of the same object while checking that the pack + is self-contained and connected. For example, + `git clone -c pack.allowDuplicateObjects <repository>` can accept + a pack generated from overlapping existing packs. Object and + connectivity checks are preserved. Explicit `--strict` and + `--verify` continue to reject duplicate objects. + `--no-allow-duplicate-objects` overrides this setting. + Defaults to `false`. +
A couple of brief thoughts here:
- Is "while checking that the pack is self-contained and connected" true in all cases? Certainly if we give the option '--check-self-contained-any-connected' to 'index-pack'. But if a user invokes "git -c pack.allowDuplicateObjects index-pack ...", we will not bother to perform the same checks.
- The "For example [...]" may be unnecessary here. I don't have a strong feeling here either way, but it feels somewhat specific to 'git-clone(1)' so perhaps belongs there instead?
- "Explicit --strict and --verify [...]" and the following sentence. I think that this means to suggest that --strict and --verify both continue to behave as-is, but setting this configuration option allows them to conditionally accept otherwise-good packs that happen to contain duplicate objects.
I wonder if these couple of sentences may be combined like: "When true, linkgit:git-index-pack[1] will accept otherwise-valid packs containing duplicate objects under --strict or --verify." But reading further, I don't think that that's actually what this option does. More below.
Quoted 8 lines
Documentation/config/pack.adoc +0
-0pack.island:: An extended regular expression configuring a set of delta islands. See "DELTA ISLANDS" in linkgit:git-pack-objects[1]Documentation/git-index-pack.adoc +0
-0
Documentation/git-index-pack.adoc SYNOPSIS · line 16
Quoted 7 lines
Documentation/git-index-pack.adoc +3
-1@@ -11,7 +11,9 @@ SYNOPSIS [verse] 'git index-pack' [-v] [-o <index-file>] [--[no-]rev-index] <pack-file> 'git index-pack' --stdin [--fix-thin] [--keep] [-v] [-o <index-file>] - [--[no-]rev-index] [<pack-file>] + [--[no-]rev-index] + [--[no-]allow-duplicate-objects] + [<pack-file>]
Not the fault of this patch, but the synopsis and usage string (index_pack_usage) do not agree, hence the 'index-pack' entry in t/t0450/adoc-help-mismatches. So putting this on a new line is OK, but I think it's fine to keep this and "[<pack-file>]" on the same line as "[--[no-]rev-index]" in the pre-image of this patch.
Documentation/git-index-pack.adoc default and "Indexing objects" when `--stdin` is specified. · line 108
Quoted 12 lines
Documentation/git-index-pack.adoc +7
-0DESCRIPTION@@ -97,6 +99,14 @@ default and "Indexing objects" when `--stdin` is specified. --check-self-contained-and-connected:: Die if the pack contains broken links. For internal use only. +--allow-duplicate-objects:: +--no-allow-duplicate-objects:: + Allow or reject multiple copies of the same object while checking + that the pack is self-contained and connected. The default is + controlled by `pack.allowDuplicateObjects`. The command-line + option overrides the configuration. `--allow-duplicate-objects` + cannot be combined with `--strict` or `--verify`.
Hmm. This suggests something other than what I gathered when reading the corresponding git-config(1) entry.
Are there cases where we would want want to allow duplicate object,s but retain the other "--strict" behavior of dying when the pack contains broken objects, or links off to objects that we don't have? I would imagine that 'git clone' would want to do just this. I imagine that such a use-case would expect that even if we are cloning from a source that is known to produce packs with duplicate objects we would still want to verify that none of the objects it references are missing, etc.
I think that suggests something more along the lines of having this option opt you out of this specific portion of "--strict"'s behavior, as in "git index-pack --strict --allow-duplicate-objects". I may be missing something here.
Documentation/git-index-pack.adoc static int nr_threads; · line 142
Quoted 9 lines
Documentation/git-index-pack.adoc +5
-0@@ -135,6 +135,11 @@ static int nr_threads; static int from_stdin; static int strict; +static enum { + DUPLICATE_OBJECTS_REJECT = 0, + DUPLICATE_OBJECTS_ALLOW_CONFIG, + DUPLICATE_OBJECTS_ALLOW_OPTION, +} allow_duplicate_objects;
I was initially a little surprised to see a new enum value here for what I imagined would be a true/false value. But looking at the diff below, I think that this is to silently ignore a "true" value for the config option 'pack.allowDuplicateObjects' in the presence of "--strict".
So I think that this tri-state is fine in that sense. But I imagine that much of this goes away if we take this option to instead carve out one specific behavior of --strict instead of being incompatible with it entirely.
Documentation/git-index-pack.adoc static int nr_threads;
Quoted 8 lines
Documentation/git-index-pack.adoc +8
-0+ if (write_idx_strict && + allow_duplicate_objects == DUPLICATE_OBJECTS_ALLOW_OPTION) + die(_("options '%s' and '%s' cannot be used together"), + "--allow-duplicate-objects", "--strict"); + if (verify && + allow_duplicate_objects == DUPLICATE_OBJECTS_ALLOW_OPTION) + die(_("options '%s' and '%s' cannot be used together"), + "--allow-duplicate-objects", "--verify");
If you end up keeping the existing meaning and need to declare this incompatible with write_idx_strict and verify, there is a helper for this case:
die_for_incompatible_opt2(allow_duplicate_objects == DUPLICATE_OBJECTS_ALLOW_OPTION,
"--allow-duplicate-objects",
write_idx_strict, "--strict");
die_for_incompatible_opt2(allow_duplicate_objects == DUPLICATE_OBJECTS_ALLOW_OPTION,
"--allow-duplicate-objects",
verify, "--verify");Alternatively, since writing "allow_duplicate_objects == DUPLICATE_OBJECTS_ALLOW_OPTION" is kind of a mouthful, you could instead write this abomination:
if (allow_duplicate_objects == DUPLICATE_OBJECTS_ALLOW_OPTION) {
die_for_incompatible_opt2(1, "--allow-duplicate-objects",
write_idx_strict, "--strict");
die_for_incompatible_opt2(1, "--allow-duplicate-objects",
verify, "--verify");
};-)
Documentation/git-index-pack.adoc int cmd_index_pack(int argc, · line 2096
Quoted 9 lines
Documentation/git-index-pack.adoc +1
-1@@ -2055,7 +2090,7 @@ int cmd_index_pack(int argc, read_idx_option(&opts, index_name); opts.flags |= WRITE_IDX_VERIFY | WRITE_IDX_STRICT; } - if (strict) + if (write_idx_strict) opts.flags |= WRITE_IDX_STRICT; if (HAVE_THREADS && !nr_threads) {
OK. Since we aren't treating this as a carve-out, we don't have any further changes in pack-write.c. Makes sense, though I am curious about your thoughts on whether the alternate interface makes more or less sense.
t/t5308-pack-detect-duplicates.sh
Quoted 4 lines
t/t5308-pack-detect-duplicates.sh +0
-0
I haven't read the tests carefully (under the assumption that they may change substantively if the meaning of "--allow-duplicate-objects" is altered). But from skimming, I wonder if there is some room to shrink the number of tests.
When working with an agent, I typically ask it to implement the minimal number of tests, along with a prompt that it must demonstrate that those tests still exercise all interesting behavior. Often I will repeat this a number of times until I am similarly convinced.
Thanks,
Taylor
Quoted 3 lines · [email protected] writes:
Signed-off-by Friel <[email protected]>---
Applies on top of tb/pack-with-duplicates.
I am really reluctant to take us in this direction. The last time I had a deep discussion on this was with Shawn Pearce (so those who knew him can tell how long ago that was), and the essence of his suggestion was that allowing malformed or invalid packfiles is a slippery slope. They complicate everything, from delta cycle detection to ensuring that repository data stays healthy.
Changes that help us detect such a broken pack as early as possible and prevent it from entering your repository are very much welcome. Changes that accept such a broken pack as if nothing were wrong, not so much.
Thanks.
Quoted 6 lines · Junio C Hamano
Quoted 3 lines · [email protected] writes:
Signed-off-by Friel <[email protected]>---
Applies on top of tb/pack-with-duplicates.
Quoted 6 lines
I am really reluctant to take us in this direction. The last time I had a deep discussion on this was with Shawn Pearce (so those who knew him can tell how long ago that was), and the essence of his suggestion was that allowing malformed or invalid packfiles is a slippery slope. They complicate everything, from delta cycle detection to ensuring that repository data stays healthy.
I share the concern, but I am not sure to what extent historical discussions should decide our stance on whether or not we want to provide support for packs containing duplicate objects.
A lot of this discussion pre-dates my involvement with the project, but the closest thread I could find on the topic with Shawn was from back in [1]. AFAICT the main concern at the time was that binary searching the pack would not handle duplicates well.
Your comment here around delta cycle detection is something that came up in the series I wrote (upon which this patch is based) trying to harden Git's handling of packs containing duplicate objects. The main thing I found when writing that series that Git did *not* already handle well was navigating cycles during delta resolution, in particular with REF_DELTAs. One of the patches in that series teaches Git how to handle this case in a way that (a) doesn't adversely impact the performance of delta resolution in packs that do *not* contain duplicates, and (b) is not especially complex.
My sense is that many of the things the project was concerned with at the time have since been hardened, and that Git is more well-equipped to deal with duplicate object-containing packs than we may give it credit for.
A couple of thoughts on why I think this direction is useful:
- I think that Peff makes a good point in [2], which effectively boils down to, "OK, maybe these packs are buggy, but if they contain the sole copy of an object we care about, we must be able to read them." As I understand it, that is effectively why we disambiguate between "index-pack" and "index-pack --strict".
I think a version of this patch that carves out duplicates as OK even in "--strict" mode results in better behavior for packs that contain duplicate objects (whereas before we had to drop all of the additional checks that "--strict" requires in order to index such a pack).
- There are genuinely useful scenarios by which a sender may wish to consolidate two or more packs together by appending (as opposed to repacking) them, in a way that is extremely cheap to do.
Friel may have a better example here, but a useful mental model for me has been: if an upload-pack implementation knows that it can serve a request by sending the objects from some known subset of all packs, it may make sense to simply combine those packs by appending them rather than generating a new pack with the union of their objects.
That trade-off is useful IMHO for serving fetches and clones for very large and fast-moving repositories where repacking may not always be able to keep up.
I am not proposing that we make packs containing duplicate objects the norm. But mine and Friel's approach here is to first demonstrate (via 'tb/pack-with-duplicates') that Git has good handling for packs containing duplicate objects, and subsequently (via this patch) to make it possible to index such packs.
If we can find useful ways to combine the ideas above with Git's in-tree implementation of upload-pack, one could imagine that Git itself may eventually send packs containing duplicate copies of some object(s) behind a capability. In other words, for clients that know how to process such a pack, the server may wish to ask the client to do just that in the name of saving some CPU cycles necessary to generate a pack that doesn't have any duplicate objects.
Quoted 4 lines
Changes that help us detect such a broken pack as early as possible and prevent it from entering your repository are very much welcome. Changes that accept such a broken pack as if nothing were wrong, not so much.
I agree that the RFC as written makes the exception look broader than it should.
A reroll should make --allow-duplicate-objects carve out only the duplicate-OID part of --strict, rather than make the two options incompatible. index-pack should still recompute every object ID, resolve every delta, perform the usual object and link checks, and check connectivity when requested. Only WRITE_IDX_STRICT's requirement that each OID have one physical entry would be relaxed. The default should, IMHO, remain that we reject such packs.
But I would note that having packs containing duplicate objects is not a new repository state for Git. Non-strict index-pack accepts duplicate entries today, and shallow and filtered clones can store the same pack. My series in 'tb/pack-with-duplicates' attempts to fix the known-broken assumptions in reverse indexes, delta resolution, MIDX verification, and bitmap reuse because those packs can already exist.
If duplicate entries are to be forbidden entirely in order for a pack to be considered valid, then I think we should reject them in every index-pack mode, including the shallow and filtered clone paths, and diagnose existing packs in fsck. Otherwise, a default-off opt-in for a known producer is a cleaner way IMHO to express that duplicate objects are OK, rather than relying on whether or not we performed a shallow/partial clone (and thus did not invoke index-pack with "--strict").
I think that narrower version is worth pursuing.
Thanks,
Taylor
[1]: https://lore.kernel.org/git/CAJo=hJs3mM7=LcOop-WD=bipA=Wx-7MDh6ObQwFUE38tjurvcw@mail.gmail.com/
[2]: https://lore.kernel.org/git/[email protected]/
Quoted 7 lines · Taylor Blau
If we can find useful ways to combine the ideas above with Git's in-tree implementation of upload-pack, one could imagine that Git itself may eventually send packs containing duplicate copies of some object(s) behind a capability. In other words, for clients that know how to process such a pack, the server may wish to ask the client to do just that in the name of saving some CPU cycles necessary to generate a pack that doesn't have any duplicate objects.
I can live with such an extension as long as we teach the receiving end to deduplicate the extra copy. Leaving packs with duplicate objects on disk is a completely different story, as it will become a source of spreading such broken packs elsewhere, though.
Quoted 3 lines
But I would note that having packs containing duplicate objects is not a new repository state for Git. Non-strict
index-packaccepts duplicate entries today, and shallow and filtered clones can store the same pack.
The same as what???
Quoted 8 lines
My series in 'tb/pack-with-duplicates' attempts to fix the known-broken assumptions in reverse indexes, delta resolution, MIDX verification, and bitmap reuse because those packs can already exist.
If duplicate entries are to be forbidden entirely in order for a pack to be considered valid, then I think we should reject them in every
index-packmode, including the shallow and filtered clone paths, and diagnose existing packs infsck.
Yup, I think that would be a sensible longer-term direction. We may need a bit more tool support to "fix" by reindexing at the receiving end, though. As you say, "cancatenate two packs, damn the duplicates" may be a cheap way for server side to give union of objects contained in these two packs, but doing so without even measuring how much they are duplicating cannot go on forever unchecked. Somebody needs to remove these duplicates, and the time the downloader indexes the incoming pack would be the best place to do so. It needs to read each and every object in the pack stream to make the .idx file out of the stream anyway.
From: Friel <[email protected]>
Quoted 2 lines · Junio C Hamano
I can live with such an extension as long as we teach the receiving end to deduplicate the extra copy.
That makes sense. We don't want packs containing duplicate objects to become a persistent source of duplicate objects in other repositories.
For our server, duplicate objects would be an exceptional consequence of an optimization, not normal operation. We have not seen duplicates in practice yet. But preventing them imposes a cost on every upload-pack request even when duplicates are rare.
I'll talk with Taylor about whether the client should repack when it detects duplicates, or whether Git already has a way to mark such a pack as dirty for reuse or retransmission until it has been cleaned up. The intent would be to pay that cost only when duplicates actually occur.
In all humility, thank you for reviewing and considering the RFC PATCH. I'm still getting acquainted with mailing list and the history of Git, and I'm happy to have Taylor & Ted's help.
Cheers,
Friel
Quoted 7 lines · On Wed, Jul 29, 2026 at 02:17:16PM -0700, [email protected] wrote:
From: Friel <[email protected]>
Quoted 2 lines · Junio C Hamano
I can live with such an extension as long as we teach the receiving end to deduplicate the extra copy.
Quoted 7 lines
That makes sense. We don't want packs containing duplicate objects to become a persistent source of duplicate objects in other repositories.
For our server, duplicate objects would be an exceptional consequence of an optimization, not normal operation. We have not seen duplicates in practice yet. But preventing them imposes a cost on every upload-pack request even when duplicates are rare.
To Junio's point about "these cannot go forever unchecked", I agree, and I think this is an important internal detail which I may not have made clear. We don't expect to send packs containing duplicate objects as a general case, but this patch and my series are a defensive measure to make sure clients don't immediately choke on them.
Quoted 5 lines
I'll talk with Taylor about whether the client should repack when it detects duplicates, or whether Git already has a way to mark such a pack as dirty for reuse or retransmission until it has been cleaned up. The intent would be to pay that cost only when duplicates actually occur.
Let me think a little bit more about this. It would be a shame to have to repack the entirety of the pack when there are only a few (or zero) duplicate object entries. But if we can efficiently pluck them out of the stream when fetching/cloning, that may be worthwhile.
FWIW, (and I'm biased, but) I still think the series this patch is based on is worth picking up. I think it makes sense to have better support when we *do* happen to see duplicate objects (e.g., to recover the only good copy of some object you have, similar to Peff's argument quoted earlier in the thread).
But "better support" can coexist with "...you still shouldn't do this".
Quoted 3 lines
In all humility, thank you for reviewing and considering the RFC PATCH. I'm still getting acquainted with mailing list and the history of Git, and I'm happy to have Taylor & Ted's help.
Thanks for saying that. Welcome to the list :-).
Thanks,
Taylor
Quoted 10 lines · Junio C Hamano
Quoted 7 lines · Taylor Blau
If we can find useful ways to combine the ideas above with Git's in-tree implementation of upload-pack, one could imagine that Git itself may eventually send packs containing duplicate copies of some object(s) behind a capability. In other words, for clients that know how to process such a pack, the server may wish to ask the client to do just that in the name of saving some CPU cycles necessary to generate a pack that doesn't have any duplicate objects.
Quoted 4 lines
I can live with such an extension as long as we teach the receiving end to deduplicate the extra copy. Leaving packs with duplicate objects on disk is a completely different story, as it will become a source of spreading such broken packs elsewhere, though.
I am trying to nudge us in the direction of reconsidering whether a pack containing duplicate objects *is* broken. After reading some of the historical discussions on the list, the only "broken" portion here is client-side support, which is what my series is trying to address.
Quoted 5 lines
Quoted 3 lines
But I would note that having packs containing duplicate objects is not a new repository state for Git. Non-strict
index-packaccepts duplicate entries today, and shallow and filtered clones can store the same pack.The same as what???
The same pack meaning the one which contains duplicate objects. As I understand, Friel configured clients to have a shallow depth equal to the 32-bit unsigned maximum value (which is gross), but does cause us to run "index-pack" without "--strict".
Thanks,
Taylor