Skip to contentSkip to conversation

git-repack: Outof memory

DISCUSSION Dongsheng Song 4 participants 6 messages Jun 15, 2016 4 messages not in the archive

lore
Share & tools
lore

Dongsheng Song

author
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all:

I'm import from subversion. The problem appears to be git-repack phase using too many memory:

$ git-repack -a -f -d --window=64 --depth=64
Generating pack...
Done counting 123497 objects.
Deltifying 123497 objects.
  24% (29677/123497) done
$ top
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3572 www-data  18   0 2591m 1.9g  528 R   13 94.8  81:48.98 git-pack-object
$ free
             total       used       free     shared    buffers     cached
Mem:       2076308    2029824      46484          0       2760       6800
- -/+ buffers/cache:    2020264      56044
Swap:      1028152     684032     344120

How to compute memory usage of git-repack ?

Thanks and regards,

Dongsheng

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)

iD8DBQFFDKVh90pbDJCgbHoRAlweAJ45DhTXI+bb+nb2Y+JlbIBoFusK8wCgk0U2 XcUc9K/chYiUYq3ZLychzcU=
=NzXf
-----END PGP SIGNATURE-----

#1 lore raw
Quoted 8 lines · Dongsheng Song

I'm import from subversion. The problem appears to be git-repack phase using too many memory:

$ git-repack -a -f -d --window=64 --depth=64
Generating pack...
Done counting 123497 objects.
Deltifying 123497 objects.
  24% (29677/123497) done
Quoted 4 lines
$ top
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3572 www-data  18   0 2591m 1.9g  528 R   13 94.8  81:48.98 git-pack-object

*ouch* You are probably running out of address space if you are on a 32 bit architecture. A 2.5 GiB virtual address space is pretty close to the maximum allowed on most OSes.

The code that I'm sitting on but haven't yet completed rebasing onto current Git would probably help here.

Do you have any existing .pack files in .git/objects/pack? How big are they and their corresponding .idx files?

git-repack will need to mmap every .pack and .idx in .git/objects/pack, plus it needs working memory for each object (123,497 of 'em) but as I recall its pretty frugal on its per-object allocation. It can easily work with as many as 2 million objects on a 32 bit system, assuming the .pack and .idx files aren't too large.

Signature · 1 line
Shawn.

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 10 lines · Dongsheng Song

finished, thanks a lot.

$ ls -l .git/objects/pack/
total 1675466
-rw-r--r-- 1 www-data www-data    2964992 Sep 17 19:17
pack-b09c57c25e4459f6365b5d27139abfd93bf1c86f.idx
-rw-r--r-- 1 www-data www-data 1711037142 Sep 17 19:17
pack-b09c57c25e4459f6365b5d27139abfd93bf1c86f.pack

If the pack files larger than 2.5g, how can I repack it on i686 ?

*youch* A 1.5 GiB pack file?

Your files apparently do not delta compress very well. I fear that you are going to bump up against address space limitations soon on 32 bit systems. Then you will bump up against the 4 GiB pack file size limit. Which means you will need to use several packs and avoid the '-a' flag when calling git-repack.

Nico was suggesting using the default window size (rather than --window=64) as larger windows requires more memory during repack. But you may also need the mmap window code I'm working on. I better hurry up and get that into Junio's testing branches. :-)

Signature · 1 line
Shawn.
#7 lore raw
Quoted 5 lines · Shawn Pearce

... I fear that
you are going to bump up against address space limitations soon on 32 bit systems. Then you will bump up against the 4 GiB pack file size limit. Which means you will need to use several packs and avoid the '-a' flag when calling git-repack.

I presume that you are hinting that we would need to update git-repack so that it is still useful without --all.

Which means "pack-objects --unpacked=active-pack" would need to be pushed out so that git-repack can be updated to do the "archived ones and repacking the active pack" we talked about earlier.

I think pack-objects --unpacked=active-pack is ready, so I'll push it out.

#8 lore raw
Quoted 10 lines · Junio C Hamano
Quoted 5 lines · Shawn Pearce

... I fear that
you are going to bump up against address space limitations soon on 32 bit systems. Then you will bump up against the 4 GiB pack file size limit. Which means you will need to use several packs and avoid the '-a' flag when calling git-repack.

I presume that you are hinting that we would need to update git-repack so that it is still useful without --all.

Yes. :-)

Apparently the Porcelain level is still open for debate. Someone just needs to post a reasonable implementation to get discussion (or lack there of) going. I'm too tired right now to try to roll something up right, even though it looks to be rather simple.

As it is I'm having a difficult time navigating in vi to write this email and normally vi navigation is as easy as breathing for me...

Quoted 2 lines

I think pack-objects --unpacked=active-pack is ready, so I'll push it out.

Thanks; I noticed that in your "What's in git.git" announcement.

Signature · 1 line
Shawn.
#9 lore raw
In reply to Dongsheng Song
Quoted 9 lines · Dongsheng Song

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all:

I'm import from subversion. The problem appears to be git-repack phase using too many memory:

$ git-repack -a -f -d --window=64 --depth=64

Try with that --window=64 argument removed.

Nicolas