git doesn't like big files when pushing
I have a mail archive stored with git, in mbox form, and I made some changes to a few of the files and checked them back in.
That worked fine, but when I went to push the stuff to my server, I got the following errors:
$ git push origin updating 'refs/heads/master' from 490badd9bec9ada3a21be275c97fb2a3a390f49e to 16be8985abc8a9c89ad2cc8f46a0d8e9786e832f Generating pack... Done counting 8 objects. Deltifying 8 objects. fatal: Out of memory, malloc failed fatal: early EOF unpack unpacker exited with error code ng refs/heads/master n/a (unpacker error)
And here are the file sizes of the files that were changed:
$ ls -lh linux-usb-devel.save.200* -rw-r--r-- 1 greg users 41M Jan 6 14:30 linux-usb-devel.save.2001 -rw-r--r-- 1 greg users 80M Jan 6 14:30 linux-usb-devel.save.2002 -rw-r--r-- 1 greg users 74M Jan 6 14:30 linux-usb-devel.save.2003 -rw-r--r-- 1 greg users 99M Jan 6 14:30 linux-usb-devel.save.2004 -rw-r--r-- 1 greg users 89M Jan 6 14:30 linux-usb-devel.save.2005
So, am I just foolish for trying to use git for this? Should I just go back to using rsync to back stuff like this up with?
thanks,
greg k-h
Quoted 6 lines · Greg KH
I have a mail archive stored with git, in mbox form, and I made some changes to a few of the files and checked them back in.
That worked fine, but when I went to push the stuff to my server, I got the following errors:
Quoted 11 lines
$ git push origin updating 'refs/heads/master' from 490badd9bec9ada3a21be275c97fb2a3a390f49e to 16be8985abc8a9c89ad2cc8f46a0d8e9786e832f Generating pack... Done counting 8 objects. Deltifying 8 objects. fatal: Out of memory, malloc failed fatal: early EOF unpack unpacker exited with error code ng refs/heads/master n/a (unpacker error)
Oh, and I'm using:
$ git --version git version 1.2.3.g8c2f
if that helps or not.
thanks,
greg k-h
Quoted 19 lines · Greg KH
Quoted 6 lines · Greg KH
I have a mail archive stored with git, in mbox form, and I made some changes to a few of the files and checked them back in.
That worked fine, but when I went to push the stuff to my server, I got the following errors:
Quoted 11 lines
$ git push origin updating 'refs/heads/master' from 490badd9bec9ada3a21be275c97fb2a3a390f49e to 16be8985abc8a9c89ad2cc8f46a0d8e9786e832f Generating pack... Done counting 8 objects. Deltifying 8 objects. fatal: Out of memory, malloc failed fatal: early EOF unpack unpacker exited with error code ng refs/heads/master n/a (unpacker error)
Quoted 3 lines
Oh, and I'm using:
$ git --version git version 1.2.3.g8c2f
Hm, 1.2.4.g6177 seems better, it's still trying to pack things, after about 10 minutes, but at least it isn't dying yet. I'll let you know if it finishes properly or not...
thanks,
greg k-h
Quoted 4 lines · Greg KH
Quoted 2 lines · Greg KH
I have a mail archive stored with git, in mbox form, and I made some changes to a few of the files and checked them back in.
...
Ouch. Running out of memory while deltifying sounds really bad.
Quoted 5 lines
Oh, and I'm using:
$ git --version git version 1.2.3.g8c2fif that helps or not.
It doen't, since I do not have 8c2fXXXX commit ;-).
I suspect "git push --thin origin" might help, if you are on my "master" branch:
diff-tree a79a276... (from 2245be3...)
Author: Junio C Hamano <[email protected]>
Date: Mon Feb 20 00:09:41 2006 -0800
Add git-push --thin.
Maybe we would want to make this default before it graduates to the master branch, but in the meantime to help testing things, this allows you to say "git push --thin destination".
:100755 100755 706db99... 73dcf06... M git-push.sh
Quoted 9 lines · Junio C Hamano
Quoted 4 lines · Greg KH
Quoted 2 lines · Greg KH
I have a mail archive stored with git, in mbox form, and I made some changes to a few of the files and checked them back in.
...
Ouch. Running out of memory while deltifying sounds really bad.
Quoted 7 lines
Quoted 5 lines
Oh, and I'm using:
$ git --version git version 1.2.3.g8c2fif that helps or not.
It doen't, since I do not have 8c2fXXXX commit ;-).
Heh, that's a merge on my tree, sorry. My tree only has one change to the git-format-patch.sh script.
858cbfbabe4ede5f5eba32041eb7448319e53e2a is the most recent commit from your tree.
Quoted 7 lines
I suspect "git push --thin origin" might help, if you are on my "master" branch:
diff-tree a79a276... (from 2245be3...)
Author: Junio C Hamano <[email protected]>
Date: Mon Feb 20 00:09:41 2006 -0800
Quoted 7 lines
Add git-push --thin.
Maybe we would want to make this default before it graduates to the master branch, but in the meantime to help testing things, this allows you to say "git push --thin destination".
Signed-off-by Junio C Hamano <[email protected]>
Will try that. I eventually gave up on the last push when it ran for 45 minutes at full cpu usage, and X got killed by the OOM killer in the kernel for some reason...
thanks,
greg k-h
Quoted 14 lines · Greg KH
Quoted 13 lines · Junio C Hamano
I suspect "git push --thin origin" might help, if you are on my "master" branch:
diff-tree a79a276... (from 2245be3...)
Author: Junio C Hamano <[email protected]>
Date: Mon Feb 20 00:09:41 2006 -0800Add git-push --thin.
Maybe we would want to make this default before it graduates to the master branch, but in the meantime to help testing things, this allows you to say "git push --thin destination".
Quoted 5 lines
Quoted 1 line
Signed-off-by Junio C Hamano <[email protected]>Will try that. I eventually gave up on the last push when it ran for 45 minutes at full cpu usage, and X got killed by the OOM killer in the kernel for some reason...
Nice, this worked!
Now what's the odds that when I pull from the server to another box these same objects, the server will have the same problem as git-push did?
will go try that now...
thanks,
greg k-h
Quoted 21 lines · Greg KH
Quoted 14 lines · Greg KH
Quoted 13 lines · Junio C Hamano
I suspect "git push --thin origin" might help, if you are on my "master" branch:
diff-tree a79a276... (from 2245be3...)
Author: Junio C Hamano <[email protected]>
Date: Mon Feb 20 00:09:41 2006 -0800Add git-push --thin.
Maybe we would want to make this default before it graduates to the master branch, but in the meantime to help testing things, this allows you to say "git push --thin destination".
Quoted 5 lines
Quoted 1 line
Signed-off-by Junio C Hamano <[email protected]>Will try that. I eventually gave up on the last push when it ran for 45 minutes at full cpu usage, and X got killed by the OOM killer in the kernel for some reason...
Quoted 5 lines
Nice, this worked!
Now what's the odds that when I pull from the server to another box these same objects, the server will have the same problem as git-push did?
Ok, no problem there either.
thanks for the pointer to that option.
greg k-h
Quoted 3 lines · Greg KH
Ok, no problem there either.
thanks for the pointer to that option.
Now that it has been tested twice, it's about time it becomes the default ;-)
cheers,
martin
Quoted 4 lines · Greg KH
That worked fine, but when I went to push the stuff to my server, I got the following errors:
Quoted 9 lines
$ git push origin updating 'refs/heads/master' from 490badd9bec9ada3a21be275c97fb2a3a390f49e to 16be8985abc8a9c89ad2cc8f46a0d8e9786e832f Generating pack... Done counting 8 objects. Deltifying 8 objects. fatal: Out of memory, malloc failed fatal: early EOF
Gaah. We probably have a memory leak somewhere, and it just normally doesn't much matter.
Git does want to keep the "window" of the objects it packs in memory while packing (it would be really costly to read them in one at a time, over and over again), but it should hopefully not really not need tons more memory than that. Since the window is normally 10, and you only have 8 objects, it really wants to have all eight in memory, but it shouldn't need a whole lot more.
But maybe it's really the case that you can't fit those 8 objects in memory. One option (which might also solve some of the performance issues) is to make the window be based on object _size_ rather than just be a fixed number (ie with an 80MB object, you'd only try a couple of objects around it, not the full window).
Linus
Hi,
Quoted 1 line · Linus Torvalds
But maybe it's really the case that you can't fit those 8 objects in memory. One option (which might also solve some of the performance issues) is to make the window be based on object _size_ rather than just be a fixed number (ie with an 80MB object, you'd only try a couple of objects around it, not the full window).
Well, doesn't the pack process involve keeping objects of similar size in memory next to each other? My suspicion is that the system is running out of virtual address space because almost every malloc() will attempt to get an entirely new block because the one that was just freed is a few bytes too small.
Simon