Skip to content

a rabbit hole of update-server-info (and midx!) fixes

Jeff King git 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest13 patches 9 repliesIn review
  2. v112 patches 12 repliesSuperseded

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/13packfile.h: drop extern from function declarationspackfile.h+40 -406 replies
2/13pack-revindex: open index if necessarypack-bitmap.c pack-revindex.c pack-revindex.h +1 more+17 -70 replies
3/13t5319: fix bogus cat-file argumentt/t5319-multi-pack-index.sh+7 -60 replies
4/13t5319: drop useless --buffer from cat-filet/t5319-multi-pack-index.sh+2 -20 replies
5/13midx: check both pack and index names for containmentmidx.c midx.h+35 -31 reply
6/13packfile: fix pack basename computationpackfile.c packfile.h t/t5319-multi-pack-index.sh +1 more+32 -10 replies
7/13http: simplify parsing of remote objects/info/packshttp.c+14 -210 replies
8/13server-info: fix blind pointer arithmeticserver-info.c+12 -100 replies
9/13server-info: simplify cleanup in parse_pack_def()server-info.c+4 -31 reply
10/13server-info: use strbuf to read old info/packs fileserver-info.c+8 -100 replies
11/13server-info: drop nr_alloc struct memberserver-info.c+1 -10 replies
12/13server-info: drop objdirlen pointer arithmeticserver-info.c+2 -80 replies
13/13update_info_refs(): drop unused force parameterserver-info.c+2 -20 replies

Reviews trailers per patch

None of the 13 patches has a Reviewed-by or Acked-by.

PatchRené Scharfe
1/13
2/13
3/13
4/13
5/13Helped-by
6/13
7/13
8/13
9/13
10/13
11/13
12/13
13/13

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH v2 0/13] a rabbit hole of update-server-info (and midx!) fixes
Quoted 7 lines · Jeff King

This patch series started with patch 12: I just wanted to drop the unused "force" parameter from update_info_refs().
[...]
And here we are. I present them here in reverse rabbit-hole order (which is also roughly important fixes first, and minor cleanups last). The individual chunks are mostly independent, but the server-info cleanups rely on the shared pack_basename() helper added as part of the midx fix.

Here's a v2 which addresses the feedback on the last round (thank you both). In particular:

- the rabbit hole goes one deeper, as fixing the midx bug reveals that the revindex code just assumes that the pack .idx has been loaded. Patch 2 corrects this, which addresses most of the test failures with GIT_TEST_MULTI_PACK_INDEX.

- There was one other failure, which is that t5570 corrupted a .idx and expected us to notice (which no longer happens when the midx is able to provide the answer). I've tweaked the test to address this.

- I took René's suggestion of making midx_contains_pack() friendlier by handling either a .pack or a .idx argument. I had to tweak the suggested code just a little, and I added comments, since it took me a little staring to make sure everything was correct.

I split that out from the basename fix, so that's now patches 5 and
6. It also lets us drop the patch to factor out pack_name_to_idx().

Other than that, it's substantially the same as v1. I did move the public declaration of pack_basename() to the patch where it is added, rather than doing it later, which shows up in the range diff below.

[01/13]: packfile.h: drop extern from function declarations [02/13]: pack-revindex: open index if necessary [03/13]: t5319: fix bogus cat-file argument
[04/13]: t5319: drop useless --buffer from cat-file [05/13]: midx: check both pack and index names for containment [06/13]: packfile: fix pack basename computation [07/13]: http: simplify parsing of remote objects/info/packs [08/13]: server-info: fix blind pointer arithmetic [09/13]: server-info: simplify cleanup in parse_pack_def() [10/13]: server-info: use strbuf to read old info/packs file [11/13]: server-info: drop nr_alloc struct member [12/13]: server-info: drop objdirlen pointer arithmetic [13/13]: update_info_refs(): drop unused force parameter

 http.c                      | 35 ++++++---------
 midx.c                      | 36 +++++++++++++++-
 midx.h                      |  2 +-
 pack-bitmap.c               |  3 +-
 pack-revindex.c             | 13 ++++--
 pack-revindex.h             |  2 +-
 packfile.c                  | 18 ++++++--
 packfile.h                  | 86 ++++++++++++++++++++-----------------
 server-info.c               | 57 +++++++++++-------------
 t/t5319-multi-pack-index.sh | 29 ++++++++++---
 t/t5570-git-daemon.sh       |  1 +
 11 files changed, 172 insertions(+), 110 deletions(-)
10:  bead52a5dc =  1:  829e666ec9 packfile.h: drop extern from function declarations
 -:  ---------- >  2:  d46c0eafd0 pack-revindex: open index if necessary
 1:  3d1978ccaf =  3:  5b965de360 t5319: fix bogus cat-file argument
 2:  ce74312004 =  4:  32ef03e059 t5319: drop useless --buffer from cat-file
 3:  ea637c9cf1 <  -:  ---------- packfile: factor out .pack to .idx name conversion
 -:  ---------- >  5:  d87a2aa2bf midx: check both pack and index names for containment
 4:  4723e1b952 !  6:  c2bed91a3f packfile: check midx coverage with .idx rather than .pack

@@ -1,35 +1,40 @@
Author: Jeff King <[email protected]>

- packfile: check midx coverage with .idx rather than .pack
+ packfile: fix pack basename computation

- When we have a .midx that covers many packfiles, we try to avoid opening
- the .idx for those packfiles. However, there are a few problems with the
- filename comparison we use:
+ When we have a multi-pack-index that covers many packfiles, we try to
+ avoid opening the .idx for those packfiles. To do that we feed the pack
+ name to midx_contains_pack(). But that function wants to see only the
+ basename, which we compute using strrchr() to find the final slash. But
+ that leaves an extra "/" at the start of our string.

- - we ask midx_contains_pack() about the .pack name, not the .idx name.
- But it compares to the latter.
+ We can fix this by incrementing the pointer. That also raises the
+ question of what to do when the name does not have a '/' at all. This
+ should generally not happen (we always find files in "pack/"), but it
+ doesn't hurt to be defensive here.

- - we compute the basename of the pack using strrchr() to find the
- final slash. But that leaves an extra "/" at the start of our
- string; we need to advance past it.
-
- That also raises the question of what to do when the name does not
- have a slash at all. This should generally not happen (we always
- find files in "pack/"), but it doesn't hurt to be defensive here.
+ Let's wrap all of that up in a helper function and make it publicly
+ available, since a later patch will need to use it, too.

The tests don't notice because there's nothing about opening those .idx files that would cause us to give incorrect output. It's just a little slower. The new test checks this case by corrupting the covered .idx, and then making sure we don't complain about it.

+ We also have to tweak t5570, which intentionally corrupts a .idx file
+ and expects us to notice it. When run with GIT_TEST_MULTI_PACK_INDEX,
+ this will fail since we now will (correctly) not bother opening the .idx
+ at all. We can fix that by unconditionally dropping any midx that's
+ there, which ensures we'll have to read the .idx.
+
diff --git a/packfile.c b/packfile.c
--- a/packfile.c
+++ b/packfile.c
@@
#endif
}

-+static const char *pack_basename(struct packed_git *p) ++const char *pack_basename(struct packed_git *p)
+{
+ const char *ret = strrchr(p->pack_name, '/');
+ if (ret)
@@ -43,25 +48,31 @@
* Do not call this directly as this leaks p->pack_fd on error return;
* call open_packed_git() instead.
@@
- ssize_t read_result;
- const unsigned hashsz = the_hash_algo->rawsz;

-- if (!p->index_data) {
-+ if (!p->index_data && the_repository->objects->multi_pack_index) {
+ if (!p->index_data) {
struct multi_pack_index *m;
- const char *pack_name = strrchr(p->pack_name, '/');
-+ char *idx_name = pack_name_to_idx(pack_basename(p));
++ const char *pack_name = pack_basename(p);

for (m = the_repository->objects->multi_pack_index;
m; m = m->next) {
-- if (midx_contains_pack(m, pack_name))
-+ if (midx_contains_pack(m, idx_name))
- break;
- }
-+ free(idx_name);
+
+ diff --git a/packfile.h b/packfile.h
+ --- a/packfile.h
+ +++ b/packfile.h
+@@
+ */
+ char *sha1_pack_index_name(const unsigned char *sha1);

- if (!m && open_pack_index(p))
- return error("packfile %s index unavailable", p->pack_name);
++/*
++ * Return the basename of the packfile, omitting any containing directory ++ * (e.g., "pack-1234abcd[...].pack").
++ */
++const char *pack_basename(struct packed_git *p);
++
+ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path); +
+ typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len,

diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh --- a/t/t5319-multi-pack-index.sh
@@ -87,3 +98,15 @@
test_expect_success 'add more objects' '
for i in $(test_seq 6 10)
do
+
+ diff --git a/t/t5570-git-daemon.sh b/t/t5570-git-daemon.sh
+ --- a/t/t5570-git-daemon.sh
+ +++ b/t/t5570-git-daemon.sh
+@@
+ test_expect_success 'fetch notices corrupt idx' '
+ cp -R "$GIT_DAEMON_DOCUMENT_ROOT_PATH"/repo_pack.git "$GIT_DAEMON_DOCUMENT_ROOT_PATH"/repo_bad2.git &&
+ (cd "$GIT_DAEMON_DOCUMENT_ROOT_PATH"/repo_bad2.git &&
++ rm -f objects/pack/multi-pack-index &&
+ p=$(ls objects/pack/pack-*.idx) &&
+ chmod u+w $p &&
+ printf %0256d 0 | dd of=$p bs=256 count=1 seek=1 conv=notrunc

 5:  0bf3fb7758 =  7:  15a7d0c690 http: simplify parsing of remote objects/info/packs
 6:  3f711f74f1 =  8:  aee572065c server-info: fix blind pointer arithmetic
 7:  5a77637699 =  9:  a9d226960a server-info: simplify cleanup in parse_pack_def()
 8:  cbc7b6fa71 = 10:  388e7bf2be server-info: use strbuf to read old info/packs file
 9:  9601aa240c = 11:  76ca72f4bf server-info: drop nr_alloc struct member
11:  23ff6ba3dc ! 12:  8753c537a7 server-info: drop objdirlen pointer arithmetic

@@ -18,36 +18,6 @@
future-proofs us, and should hopefully be more obviously safe to somebody reading the code.

- diff --git a/packfile.c b/packfile.c
- --- a/packfile.c
- +++ b/packfile.c
-@@
- #endif
- }
-
--static const char *pack_basename(struct packed_git *p)
-+const char *pack_basename(struct packed_git *p)
- {
- const char *ret = strrchr(p->pack_name, '/');
- if (ret)
-
- diff --git a/packfile.h b/packfile.h
- --- a/packfile.h
- +++ b/packfile.h
-@@
- */
- char *sha1_pack_index_name(const unsigned char *sha1);
-
-+/*
-+ * Return the basename of the packfile, omitting any containing directory -+ * (e.g., "pack-1234abcd[...].pack").
-+ */
-+const char *pack_basename(struct packed_git *p);
-+
- struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path); -
- typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len, -
diff --git a/server-info.c b/server-info.c
--- a/server-info.c
+++ b/server-info.c
12: 3f9cff4360 = 13: 96c5a26a99 update_info_refs(): drop unused force parameter