Speed up mirror-fetches with many refs
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v3 Latest7 patches 2 repliesIn review
- v27 patches 8 repliesSuperseded
- v16 patches 15 repliesSuperseded
Patches in v3 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/7 | fetch: speed up lookup of want refs via commit-graphbuiltin/ | +18 | 0 replies | |
| 2/7 | fetch: avoid unpacking headers in object existence checkbuiltin/ | +1 | 0 replies | |
| 3/7 | connected: refactor iterator to return next object ID directlybuiltin/ builtin/ builtin/ +3 more | +25 | 0 replies | |
| 4/7 | fetch-pack: optimize loading of refs via commit graphfetch-pack.c | +5 | 0 replies | |
| 5/7 | fetch: refactor fetch refs to be more extendablebuiltin/ | +17 | 0 replies | |
| 6/7 | fetch: merge fetching and consuming refsbuiltin/ | +9 | 0 replies | |
| 7/7 | fetch: avoid second connectivity check if we already have all objectsbuiltin/ | +3 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.
Cover letter [PATCH v3 0/7] Speed up mirror-fetches with many refs
Hi,
this is the third version of my patch series to speed up mirror-fetches with many refs. This patch series applies on top of master with ps/connectivity-optim merged into it.
There's only some smallish changes based on Stolee's feedback (thanks for that!):
- A small typo in 1/7.
- A confict fix in 4/7 required now because it's based on master instead of directly on my merged topic.
- I've adjusted patch 5/7 such that I don't have to re-touch the logic in 6/7.
Patrick
Patrick Steinhardt (7):
fetch: speed up lookup of want refs via commit-graph
fetch: avoid unpacking headers in object existence check
connected: refactor iterator to return next object ID directly
fetch-pack: optimize loading of refs via commit graph
fetch: refactor fetch refs to be more extendable
fetch: merge fetching and consuming refs
fetch: avoid second connectivity check if we already have all objects
builtin/clone.c | 8 ++--- builtin/fetch.c | 74 +++++++++++++++++++++++------------------- builtin/receive-pack.c | 17 ++++------ connected.c | 15 +++++---- connected.h | 2 +- fetch-pack.c | 12 ++++--- 6 files changed, 67 insertions(+), 61 deletions(-)
Range-diff against v2:
1: 4a819a6830 ! 1: 8214f04971 fetch: speed up lookup of want refs via commit-graph
@@ Commit message
that we repeatedly need to unpack object headers for each of the referenced objects.
- Speed this up by opportunistcally trying to resolve object IDs via the
+ Speed this up by opportunistically trying to resolve object IDs via the
commit graph. We only do so for any refs which are not in "refs/tags":
more likely than not, these are going to be a commit anyway, and this lets us avoid having to unpack object headers completely in case the
2: 81ebadabe8 = 2: 991a27cb82 fetch: avoid unpacking headers in object existence check 3: 98e981ced9 = 3: ba834803ab connected: refactor iterator to return next object ID directly 4: 6311203f08 ! 4: 99d3316d48 fetch-pack: optimize loading of refs via commit graph
@@ fetch-pack.c: static struct commit *deref_without_lazy_fetch(const struct object
while (1) {
if (oid_object_info_extended(the_repository, oid, &info,
-@@ fetch-pack.c: static struct commit *deref_without_lazy_fetch(const struct object_id *oid,
- }
-
- if (type == OBJ_COMMIT) {
-- struct commit *commit = lookup_commit(the_repository, oid);
-+ commit = lookup_commit(the_repository, oid);
- if (!commit || repo_parse_commit(the_repository, commit))
- return NULL;
- return commit;
5: 56a9158ac3 ! 5: d64888e072 fetch: refactor fetch refs to be more extendable
@@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
trace2_region_enter("fetch", "fetch_refs", the_repository);
ret = transport_fetch_refs(transport, ref_map);
trace2_region_leave("fetch", "fetch_refs", the_repository);
-+ if (ret) {
-+ transport_unlock_pack(transport);
-+ return ret;
-+ }
++ if (ret)
++ goto out;
}
- if (!ret)
- /*
@@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
- * time to update refs to reference the new objects.
- */
- return 0;
-- transport_unlock_pack(transport);
-- return ret;
+
+ /*
+ * Keep the new pack's ".keep" file around to allow the caller
+ * time to update refs to reference the new objects.
+ */
-+ return 0;
++ return ret;
++
++out:
+ transport_unlock_pack(transport);
+ return ret;
}
-
- /* Update local refs based on the ref values fetched from a remote */
6: 31d9f72edf ! 6: 56ecbfc9c3 fetch: merge fetching and consuming refs
@@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
/*
@@ builtin/fetch.c: static int fetch_refs(struct transport *transport, struct ref *ref_map)
- trace2_region_enter("fetch", "fetch_refs", the_repository);
- ret = transport_fetch_refs(transport, ref_map);
- trace2_region_leave("fetch", "fetch_refs", the_repository);
-- if (ret) {
-- transport_unlock_pack(transport);
-- return ret;
-- }
-+ if (ret)
-+ goto out;
+ goto out;
}
- /*
- * Keep the new pack's ".keep" file around to allow the caller
- * time to update refs to reference the new objects.
- */
-- return 0;
+- return ret;
+-
+-out:
+- transport_unlock_pack(transport);
+- return ret;
-}
-
-/* Update local refs based on the ref values fetched from a remote */7: 84e39c847f = 7: c342fc0c69 fetch: avoid second connectivity check if we already have all objects
Signature · 1 line
2.33.0