Skip to content

Avoid expensive 'repack -ad' in gc --auto

Nguyễn Thái Ngọc Duy git 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest5 patches 20 repliesIn review
  2. v11 patch 3 repliesSuperseded

Changes in v2 from the cover letter

- the core idea remains the same, mem_have is increased to half total memory though. I figure including the whole mmap'd pack in the memory estimation may be a bit much, which is why I make this change.
- no creating .keep files temporarily
- the config key is renamed gc.bigBasePackThreshold (named after core.bigFileThreshold)
- note that if you set gc.bigFileThreshold, then normal gc (without --auto) can trigger this mode too.
- you can also control this with --[no-]keep-base-pack
- documents and tests
- some more progress output improvements

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/5fixup! Add a test showing that 'git repack' throws away grafted-away parentst/t7700-repack.sh+1 -11 reply
2/5repack: add --keep-pack optionDocumentation/git-pack-objects.txt Documentation/git-repack.txt builtin/pack-objects.c +2 more+78 -32 replies
3/5gc --auto: exclude base pack if not enough mem to "repack -ad"Documentation/config.txt Documentation/git-gc.txt builtin/gc.c +5 more+204 -713 replies
4/5pack-objects: show some progress when counting kept objectsbuiltin/pack-objects.c+6 -63 replies
5/5pack-objects: display progress in get_object_details()builtin/pack-objects.c+6 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 0/5] Avoid expensive 'repack -ad' in gc --auto

I'm pretty happy with this now. Like v1, this is about not touching the giant base pack when doing background gc. This saves about 2/3 of memory, which in turn should improve performance if you're under memory pressure.

v2 changes:

- the core idea remains the same, mem_have is increased to half total memory though. I figure including the whole mmap'd pack in the memory estimation may be a bit much, which is why I make this change.
- no creating .keep files temporarily
- the config key is renamed gc.bigBasePackThreshold (named after core.bigFileThreshold)
- note that if you set gc.bigFileThreshold, then normal gc (without --auto) can trigger this mode too.
- you can also control this with --[no-]keep-base-pack
- documents and tests
- some more progress output improvements

I'm _not_ doing external rev-list in this series though. I found out that we have added more and more stuff in the internal rev-list code path over the year and simply running

git rev-list .... | git pack-objects

will break stuff (the test suite first for example). I will do it because it does help. But it will take some time.

PS. This conflicts with sb/packfiles-in-repository on 'pu' because I introduced new references to the global variable "packed_git" and prepare_packed_git(). Resolving this should be simple though:

- drop prepare_packed_git()
- replace packed_git with get_packed_git(the_repository)

Nguyễn Thái Ngọc Duy (5):
fixup! Add a test showing that 'git repack' throws away grafted-away
parents
repack: add --keep-pack option
gc --auto: exclude base pack if not enough mem to "repack -ad"
pack-objects: show some progress when counting kept objects
pack-objects: display progress in get_object_details()

 Documentation/config.txt           |   7 ++
 Documentation/git-gc.txt           |  13 +++
 Documentation/git-pack-objects.txt |   4 +
 Documentation/git-repack.txt       |   4 +
 builtin/gc.c                       | 153 +++++++++++++++++++++++++++--
 builtin/pack-objects.c             |  51 ++++++++--
 builtin/repack.c                   |  23 ++++-
 config.mak.uname                   |   1 +
 git-compat-util.h                  |   4 +
 pack-objects.h                     |   2 +
 t/t6500-gc.sh                      |  29 ++++++
 t/t7700-repack.sh                  |  21 +++-
 12 files changed, 295 insertions(+), 17 deletions(-)
Signature · 1 line
2.16.2.784.gb291bd247e