Multi-pack-index: Fix "too many file descriptors" bug

Thanks to Jeff H for finding the problem with the multi-pack-index regarding many packs. Specifically: if we open too many packs, the close_one_pack() method cannot find the packs from the multi-pack-index to close.

Jeff already fixed the problem explicitly in 'git multi-pack-index verify' which would hit this issue 100% of the time we had 2000+ packs. This issue could still happen in 'git rev-list --all --objects' if there are enough packs containing commits and trees. This series fixes the issue.

The basic solution is to add packs from the multi-pack-index into the packed_git struct as they are opened. To avoid performance issues, add a multi_pack_index bit to the packed_git struct. Midx-aware algorithms can then ignore those packs.

There was a very subtle issue that happens during a 'git repack': we clear the multi-pack-index after possibly reading some packs from it, thus leaving some packs in the packed_git struct but having a NULL multi_pack_index in the object store. This informs the change to close_midx().

I'm based on a recent 'master' commit that contains the following three branches due to nearby changes causing conflicts if I pick only Jeff's change as a base:

jh/midx-verify-too-many-packs jk/server-info-rabbit-hole bc/hash-transition-16

Thanks, -Stolee

Derrick Stolee (2):
midx: pass a repository pointer
midx: add packs to packed_git linked list

 builtin/multi-pack-index.c |  2 +-
 builtin/pack-objects.c     |  2 +-
 midx.c                     | 42 +++++++++++++++++++++++++-------------
 midx.h                     |  7 ++++---
 object-store.h             |  9 ++------
 packfile.c                 | 30 ++++++++-------------------
 sha1-name.c                |  6 ++++++
 7 files changed, 51 insertions(+), 47 deletions(-)

base-commit: 83232e38648b51abbcbdb56c94632b6906cc85a6
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-182%2Fderrickstolee%2Fmany-packs-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-182/derrickstolee/many-packs-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/182

gitgitgadget