Skip to content

Speed up mirror-fetches with many refs

Patrick Steinhardt git 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest7 patches 2 repliesIn review
  2. v27 patches 8 repliesSuperseded
  3. v16 patches 15 repliesSuperseded

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/7fetch: speed up lookup of want refs via commit-graphbuiltin/fetch.c+18 -60 replies
2/7fetch: avoid unpacking headers in object existence checkbuiltin/fetch.c+1 -30 replies
3/7connected: refactor iterator to return next object ID directlybuiltin/clone.c builtin/fetch.c builtin/receive-pack.c +3 more+25 -310 replies
4/7fetch-pack: optimize loading of refs via commit graphfetch-pack.c+5 -00 replies
5/7fetch: refactor fetch refs to be more extendablebuiltin/fetch.c+17 -70 replies
6/7fetch: merge fetching and consuming refsbuiltin/fetch.c+9 -210 replies
7/7fetch: avoid second connectivity check if we already have all objectsbuiltin/fetch.c+3 -40 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [PATCH v3 0/7] Speed up mirror-fetches with many refs

Hi,

this is the third version of my patch series to speed up mirror-fetches with many refs. This patch series applies on top of master with ps/connectivity-optim merged into it.

There's only some smallish changes based on Stolee's feedback (thanks for that!):

- A small typo in 1/7.

- A confict fix in 4/7 required now because it's based on master instead of directly on my merged topic.

- I've adjusted patch 5/7 such that I don't have to re-touch the logic in 6/7.

Patrick

Patrick Steinhardt (7):
fetch: speed up lookup of want refs via commit-graph
fetch: avoid unpacking headers in object existence check
connected: refactor iterator to return next object ID directly
fetch-pack: optimize loading of refs via commit graph
fetch: refactor fetch refs to be more extendable
fetch: merge fetching and consuming refs
fetch: avoid second connectivity check if we already have all objects

 builtin/clone.c        |  8 ++---
 builtin/fetch.c        | 74 +++++++++++++++++++++++-------------------
 builtin/receive-pack.c | 17 ++++------
 connected.c            | 15 +++++----
 connected.h            |  2 +-
 fetch-pack.c           | 12 ++++---
 6 files changed, 67 insertions(+), 61 deletions(-)

Range-diff against v2:
1: 4a819a6830 ! 1: 8214f04971 fetch: speed up lookup of want refs via commit-graph
@@ Commit message
that we repeatedly need to unpack object headers for each of the referenced objects.

- Speed this up by opportunistcally trying to resolve object IDs via the
+ Speed this up by opportunistically trying to resolve object IDs via the
commit graph. We only do so for any refs which are not in "refs/tags":
more likely than not, these are going to be a commit anyway, and this lets us avoid having to unpack object headers completely in case the

2:  81ebadabe8 = 2:  991a27cb82 fetch: avoid unpacking headers in object existence check
3:  98e981ced9 = 3:  ba834803ab connected: refactor iterator to return next object ID directly
4:  6311203f08 ! 4:  99d3316d48 fetch-pack: optimize loading of refs via commit graph

@@ fetch-pack.c: static struct commit *deref_without_lazy_fetch(const struct object

while (1) {
if (oid_object_info_extended(the_repository, oid, &info,
-@@ fetch-pack.c: static struct commit *deref_without_lazy_fetch(const struct object_id *oid,
- }
-
- if (type == OBJ_COMMIT) {
-- struct commit *commit = lookup_commit(the_repository, oid);
-+ commit = lookup_commit(the_repository, oid);
- if (!commit || repo_parse_commit(the_repository, commit))
- return NULL;
- return commit;
5: 56a9158ac3 ! 5: d64888e072 fetch: refactor fetch refs to be more extendable

    @@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
      		trace2_region_enter("fetch", "fetch_refs", the_repository);
      		ret = transport_fetch_refs(transport, ref_map);
      		trace2_region_leave("fetch", "fetch_refs", the_repository);
    -+		if (ret) {
    -+			transport_unlock_pack(transport);
    -+			return ret;
    -+		}
    ++		if (ret)
    ++			goto out;
      	}
     -	if (!ret)
     -		/*
    @@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
     -		 * time to update refs to reference the new objects.
     -		 */
     -		return 0;
    --	transport_unlock_pack(transport);
    --	return ret;
     +
     +	/*
     +	 * Keep the new pack's ".keep" file around to allow the caller
     +	 * time to update refs to reference the new objects.
     +	 */
    -+	return 0;
    ++	return ret;
    ++
    ++out:
    + 	transport_unlock_pack(transport);
    + 	return ret;
      }
    - 
    - /* Update local refs based on the ref values fetched from a remote */
6:  31d9f72edf ! 6:  56ecbfc9c3 fetch: merge fetching and consuming refs
    @@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
      
      	/*
     @@ builtin/fetch.c: static int fetch_refs(struct transport *transport, struct ref *ref_map)
    - 		trace2_region_enter("fetch", "fetch_refs", the_repository);
    - 		ret = transport_fetch_refs(transport, ref_map);
    - 		trace2_region_leave("fetch", "fetch_refs", the_repository);
    --		if (ret) {
    --			transport_unlock_pack(transport);
    --			return ret;
    --		}
    -+		if (ret)
    -+			goto out;
    + 			goto out;
      	}
      
     -	/*
     -	 * Keep the new pack's ".keep" file around to allow the caller
     -	 * time to update refs to reference the new objects.
     -	 */
    --	return 0;
    +-	return ret;
    +-
    +-out:
    +-	transport_unlock_pack(transport);
    +-	return ret;
     -}
     -
     -/* Update local refs based on the ref values fetched from a remote */

7: 84e39c847f = 7: c342fc0c69 fetch: avoid second connectivity check if we already have all objects

Signature · 1 line
2.33.0