git-p4 out of memory for very large repository
Hello,
Has anyone actually gotten git-p4 to clone a large Perforce repository? I have one codebase in particular that gets to about 67%, then consistently gets get-fast-import (and often times a few other processes) killed by the OOM killer.
I've found some patches out there that claim to resolve this, but they're all for versions of git-p4.py from several years ago. Not only will they not apply cleanly, but as far as I can tell the issues that these patches are meant to address aren't in the current version, anyway.
Any suggestions would be greatly appreciated.
Thanks,
Corey
Quoted 3 lines · Corey Thompson
Hello,
Has anyone actually gotten git-p4 to clone a large Perforce repository?
Yes. I've cloned repos with a couple of Gig of files.
Quoted 3 lines
I have one codebase in particular that gets to about 67%, then consistently gets get-fast-import (and often times a few other processes) killed by the OOM killer.
What size is this codebase? Which version and platform of git are you using?
Maybe it's a regression, or perhaps you've hit some new, previously unknown size limit?
Thanks
Luke
Quoted 15 lines
I've found some patches out there that claim to resolve this, but they're all for versions of
git-p4.pyfrom several years ago. Not only will they not apply cleanly, but as far as I can tell the issues that these patches are meant to address aren't in the current version, anyway.Any suggestions would be greatly appreciated.
Thanks,
Corey
--
To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Luke Diamand
Quoted 3 lines · Corey Thompson
Hello,
Has anyone actually gotten git-p4 to clone a large Perforce repository?
Yes. I've cloned repos with a couple of Gig of files.
Quoted 3 lines
I have one codebase in particular that gets to about 67%, then consistently gets get-fast-import (and often times a few other processes) killed by the OOM killer.
Quoted 25 lines
What size is this codebase? Which version and platform of git are you using?
Maybe it's a regression, or perhaps you've hit some new, previously unknown size limit?
Thanks
LukeQuoted 15 lines
I've found some patches out there that claim to resolve this, but they're all for versions of
git-p4.pyfrom several years ago. Not only will they not apply cleanly, but as far as I can tell the issues that these patches are meant to address aren't in the current version, anyway.Any suggestions would be greatly appreciated.
Thanks,
Corey
--
To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Sorry, I guess I could have included more details in my original post. Since then, I have also made an attempt to clone another (slightly more recent) branch, and at last had success. So I see this does indeed work, it just seems to be very unhappy with one particular branch.
So, here are a few statistics I collected on the two branches.
branch-that-fails:
total workspace disk usage (current head): 12GB
68 files over 20MB
largest three being about 118MB
branch-that-clones:
total workspace disk usage (current head): 11GB
22 files over 20MB
largest three being about 80MB
I suspect that part of the problem here might be that my company likes to submit very large binaries into our repo (.tar.gzs, pre-compiled third party binaries, etc.).
Is there any way I can clone this in pieces? The best I've come up with is to clone only up to a change number just before it tends to fail, and then rebase to the latest. My clone succeeded, but the rebase still runs out of memory. It would be great if I could specify a change number to rebase up to, so that I can just take this thing a few hundred changes at a time.
Thanks,
Corey
Quoted 28 lines · Corey Thompson
Sorry, I guess I could have included more details in my original post. Since then, I have also made an attempt to clone another (slightly more recent) branch, and at last had success. So I see this does indeed work, it just seems to be very unhappy with one particular branch.
So, here are a few statistics I collected on the two branches.
branch-that-fails:
total workspace disk usage (current head): 12GB
68 files over 20MB
largest three being about 118MBbranch-that-clones:
total workspace disk usage (current head): 11GB
22 files over 20MB
largest three being about 80MBI suspect that part of the problem here might be that my company likes to submit very large binaries into our repo (.tar.gzs, pre-compiled third party binaries, etc.).
Is there any way I can clone this in pieces? The best I've come up with is to clone only up to a change number just before it tends to fail, and then rebase to the latest. My clone succeeded, but the rebase still runs out of memory. It would be great if I could specify a change number to rebase up to, so that I can just take this thing a few hundred changes at a time.
Quoted 2 lines
Thanks,
Corey
And I still haven't told you anything about my platform or git version...
This is on Fedora Core 11, with git 1.8.3.4 built from the github repo (117eea7e).
I think I've cloned files as large as that or larger. If you just want to clone this and move on, perhaps you just need a bit more memory? What's the size of your physical memory and swap partition? Per process memory limit?
On 23 Aug 2013 12:59, "Corey Thompson" <[email protected]> wrote:
Quoted 37 lines · Corey Thompson
Quoted 28 lines · Corey Thompson
Sorry, I guess I could have included more details in my original post. Since then, I have also made an attempt to clone another (slightly more recent) branch, and at last had success. So I see this does indeed work, it just seems to be very unhappy with one particular branch.
So, here are a few statistics I collected on the two branches.
branch-that-fails:
total workspace disk usage (current head): 12GB
68 files over 20MB
largest three being about 118MBbranch-that-clones:
total workspace disk usage (current head): 11GB
22 files over 20MB
largest three being about 80MBI suspect that part of the problem here might be that my company likes to submit very large binaries into our repo (.tar.gzs, pre-compiled third party binaries, etc.).
Is there any way I can clone this in pieces? The best I've come up with is to clone only up to a change number just before it tends to fail, and then rebase to the latest. My clone succeeded, but the rebase still runs out of memory. It would be great if I could specify a change number to rebase up to, so that I can just take this thing a few hundred changes at a time.
Quoted 2 lines
Thanks,
CoreyAnd I still haven't told you anything about my platform or git version...
This is on Fedora Core 11, with git 1.8.3.4 built from the github repo (
117eea7e).
Quoted 5 lines · Luke Diamand
I think I've cloned files as large as that or larger. If you just want to clone this and move on, perhaps you just need a bit more memory? What's the size of your physical memory and swap partition? Per process memory limit?
The machine has 32GB of memory, so I hope that should be more than sufficient!
$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 268288 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Admittedly I don't typically look at ulimit, so please excuse me if I interpret this wrong, but I feel like this is indicating that the only artificial limit in place is a maximum of 64kB mlock()'d memory.
Thanks,
Corey
[email protected] wrote on Fri, 23 Aug 2013 07:48 -0400:
Quoted 11 lines · Luke Diamand
Quoted 10 lines · Corey Thompson
Quoted 3 lines
Hello,
Has anyone actually gotten git-p4 to clone a large Perforce repository?
Yes. I've cloned repos with a couple of Gig of files.
Quoted 3 lines
I have one codebase in particular that gets to about 67%, then consistently gets get-fast-import (and often times a few other processes) killed by the OOM killer.
[..]
Quoted 21 lines
Sorry, I guess I could have included more details in my original post. Since then, I have also made an attempt to clone another (slightly more recent) branch, and at last had success. So I see this does indeed work, it just seems to be very unhappy with one particular branch.
So, here are a few statistics I collected on the two branches.
branch-that-fails:
total workspace disk usage (current head): 12GB
68 files over 20MB
largest three being about 118MBbranch-that-clones:
total workspace disk usage (current head): 11GB
22 files over 20MB
largest three being about 80MBI suspect that part of the problem here might be that my company likes to submit very large binaries into our repo (.tar.gzs, pre-compiled third party binaries, etc.).
Quoted 6 lines
Is there any way I can clone this in pieces? The best I've come up with is to clone only up to a change number just before it tends to fail, and then rebase to the latest. My clone succeeded, but the rebase still runs out of memory. It would be great if I could specify a change number to rebase up to, so that I can just take this thing a few hundred changes at a time.
Modern git, including your version, do "streaming" reads from p4, so the git-p4 python process never even holds a whole file's worth of data. You're seeing git-fast-import die, it seems. It will hold onto the entire file contents. But just one, not the entire repo. How big is the single largest file?
You can import in pieces. See the change numbers like this:
p4 changes -m 1000 //depot/big/...
p4 changes -m 1000 //depot/big/...@<some-old-change>Import something far enough back in history so that it seems to work:
git p4 clone --destination=big //depot/big@60602
cd bigSync up a bit at a time:
git p4 sync @60700
git p4 sync @60800
...I don't expect this to get around the problem you describe, however. Sounds like there is one gigantic file that is causing git-fast-import to fill all of memory. You will at least isolate the change.
There are options to git-fast-import to limit max pack size and to cause it to skip importing files that are too big, if that would help.
You can also use a client spec to hide the offending files from git.
Can you watch with "top"? Hit "M" to sort by memory usage, and see how big the processes get before falling over.
-- Pete
Quoted 32 lines · Pete Wyckoff
Modern git, including your version, do "streaming" reads from p4, so the git-p4 python process never even holds a whole file's worth of data. You're seeing git-fast-import die, it seems. It will hold onto the entire file contents. But just one, not the entire repo. How big is the single largest file?
You can import in pieces. See the change numbers like this:
p4 changes -m 1000 //depot/big/... p4 changes -m 1000 //depot/big/...@<some-old-change>Import something far enough back in history so that it seems to work:
git p4 clone --destination=big //depot/big@60602 cd bigSync up a bit at a time:
git p4 sync @60700 git p4 sync @60800 ...I don't expect this to get around the problem you describe, however. Sounds like there is one gigantic file that is causing git-fast-import to fill all of memory. You will at least isolate the change.
There are options to git-fast-import to limit max pack size and to cause it to skip importing files that are too big, if that would help.
Quoted 7 lines
You can also use a client spec to hide the offending files from git.
Can you watch with "top"? Hit "M" to sort by memory usage, and see how big the processes get before falling over.
-- Pete
You are correct that git-fast-import is killed by the OOM killer, but I was unclear about which process was malloc()ing so much memory that the OOM killer got invoked (as other completely unrelated processes usually also get killed when this happens).
Unless there's one gigantic file in one change that gets removed by another change, I don't think that's the problem; as I mentioned in another email, the machine has 32GB physical memory and the largest single file in the current head is only 118MB. Even if there is a very large transient file somewhere in the history, I seriously doubt it's tens of gigabytes in size.
I have tried watching it with top before, but it takes several hours before it dies. I haven't been able to see any explosion of memory usage, even within the final hour, but I've never caught it just before it dies, either. I suspect that whatever the issue is here, it happens very quickly.
If I'm unable to get through this today using the incremental p4 sync method you described, I'll try running a full-blown clone overnight with top in batch mode writing to a log file to see whether it catches anything.
Thanks again,
Corey
Quoted 18 lines · Corey Thompson
You are correct that git-fast-import is killed by the OOM killer, but I was unclear about which process was malloc()ing so much memory that the OOM killer got invoked (as other completely unrelated processes usually also get killed when this happens).
Unless there's one gigantic file in one change that gets removed by another change, I don't think that's the problem; as I mentioned in another email, the machine has 32GB physical memory and the largest single file in the current head is only 118MB. Even if there is a very large transient file somewhere in the history, I seriously doubt it's tens of gigabytes in size.
I have tried watching it with top before, but it takes several hours before it dies. I haven't been able to see any explosion of memory usage, even within the final hour, but I've never caught it just before it dies, either. I suspect that whatever the issue is here, it happens very quickly.
Quoted 7 lines
If I'm unable to get through this today using the incremental p4 sync method you described, I'll try running a full-blown clone overnight with top in batch mode writing to a log file to see whether it catches anything.
Thanks again,
Corey
Unforunately I have not made much progress. The incremental sync method fails with the output pasted below. The change I specified is only one change number above where that repo was cloned...
So I tried a 'git p4 rebase' overnight with top running, and as I feared I did not see anything out of the ordinary. git, git-fast-import, and git-p4 all hovered under 1.5% MEM the entire time, right up until death. The last entry in my log shows git-fast-import at 0.8%, with git and git-p4 at 0.0% and 0.1%, respectively. I could try again with a more granular period, but I feel like this method is ultimately a goose chase.
Corey
$ git p4 sync //path/to/some/branch@505859
Doing initial import of //path/to/some/branch/ from revision @505859 into refs/remotes/p4/master
fast-import failed: warning: Not updating refs/remotes/p4/master (new tip 29ef6ff25f1448fa2f907d22fd704594dc8769bd does not contain d477672be5ac6a00cc9175ba2713d5395660e840)
git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects: 165000
Total objects: 69 ( 232434 duplicates )
blobs : 45 ( 209904 duplicates 40 deltas of 42 attempts)
trees : 23 ( 22530 duplicates 0 deltas of 23 attempts)
commits: 1 ( 0 duplicates 0 deltas of 0 attempts)
tags : 0 ( 0 duplicates 0 deltas of 0 attempts)
Total branches: 1 ( 1 loads )
marks: 1024 ( 0 unique )
atoms: 105170
Memory total: 24421 KiB
pools: 17976 KiB
objects: 6445 KiB
---------------------------------------------------------------------
pack_report: getpagesize() = 4096
pack_report: core.packedGitWindowSize = 33554432
pack_report: core.packedGitLimit = 268435456
pack_report: pack_used_ctr = 4371
pack_report: pack_mmap_calls = 124
pack_report: pack_open_windows = 8 / 9
pack_report: pack_mapped = 268435456 / 268435456
---------------------------------------------------------------------[email protected] wrote on Wed, 28 Aug 2013 11:41 -0400:
Quoted 24 lines · Corey Thompson
Quoted 23 lines
You are correct that git-fast-import is killed by the OOM killer, but I was unclear about which process was malloc()ing so much memory that the OOM killer got invoked (as other completely unrelated processes usually also get killed when this happens).
Unless there's one gigantic file in one change that gets removed by another change, I don't think that's the problem; as I mentioned in another email, the machine has 32GB physical memory and the largest single file in the current head is only 118MB. Even if there is a very large transient file somewhere in the history, I seriously doubt it's tens of gigabytes in size.
I have tried watching it with top before, but it takes several hours before it dies. I haven't been able to see any explosion of memory usage, even within the final hour, but I've never caught it just before it dies, either. I suspect that whatever the issue is here, it happens very quickly.
If I'm unable to get through this today using the incremental p4 sync method you described, I'll try running a full-blown clone overnight with top in batch mode writing to a log file to see whether it catches anything.
Quoted 6 lines
Quoted 2 lines
Thanks again,
CoreyUnforunately I have not made much progress. The incremental sync method fails with the output pasted below. The change I specified is only one change number above where that repo was cloned...
I usually just do "git p4 sync @505859". The error message below crops up when things get confused. Usually after a previous error. I tend to destroy the repo and try again. Sorry I don't can't explain better what's happening here. It's not a memory issue; it reports only 24 MB used.
Quoted 7 lines
So I tried a 'git p4 rebase' overnight with top running, and as I feared I did not see anything out of the ordinary. git, git-fast-import, and git-p4 all hovered under 1.5% MEM the entire time, right up until death. The last entry in my log shows git-fast-import at 0.8%, with git and git-p4 at 0.0% and 0.1%, respectively. I could try again with a more granular period, but I feel like this method is ultimately a goose chase.
Bizarre. There is no good explanation why memory usage would go up to 32 GB (?) within one top interval (3 sec ?). My theory about one gigantic object is debunked: you have only the 118 MB one. Perhaps there's some container or process memory limit, as Luke guessed, but it's not obvious here.
The other big hammer is "strace". If you're still interested in playing with this, you could do:
strace -vf -tt -s 200 -o /tmp/strace.out git p4 clone ....
and hours later, see if something suggests itself toward the end of that output file.
-- Pete
I guess you could try changing the OOM score for git-fast-import.
change /proc/<pid>/oomadj.
I think a value of -31 would make it very unlikely to be killed.
Quoted 61 lines · Pete Wyckoff
[email protected] wrote on Wed, 28 Aug 2013 11:41 -0400:
Quoted 24 lines · Corey Thompson
Quoted 23 lines
You are correct that git-fast-import is killed by the OOM killer, but I was unclear about which process was malloc()ing so much memory that the OOM killer got invoked (as other completely unrelated processes usually also get killed when this happens).
Unless there's one gigantic file in one change that gets removed by another change, I don't think that's the problem; as I mentioned in another email, the machine has 32GB physical memory and the largest single file in the current head is only 118MB. Even if there is a very large transient file somewhere in the history, I seriously doubt it's tens of gigabytes in size.
I have tried watching it with top before, but it takes several hours before it dies. I haven't been able to see any explosion of memory usage, even within the final hour, but I've never caught it just before it dies, either. I suspect that whatever the issue is here, it happens very quickly.
If I'm unable to get through this today using the incremental p4 sync method you described, I'll try running a full-blown clone overnight with top in batch mode writing to a log file to see whether it catches anything.
Quoted 6 lines
Quoted 2 lines
Thanks again,
CoreyUnforunately I have not made much progress. The incremental sync method fails with the output pasted below. The change I specified is only one change number above where that repo was cloned...
I usually just do "git p4 sync @505859". The error message below crops up when things get confused. Usually after a previous error. I tend to destroy the repo and try again. Sorry I don't can't explain better what's happening here. It's not a memory issue; it reports only 24 MB used.
Quoted 7 lines
So I tried a 'git p4 rebase' overnight with top running, and as I feared I did not see anything out of the ordinary. git, git-fast-import, and git-p4 all hovered under 1.5% MEM the entire time, right up until death. The last entry in my log shows git-fast-import at 0.8%, with git and git-p4 at 0.0% and 0.1%, respectively. I could try again with a more granular period, but I feel like this method is ultimately a goose chase.
Bizarre. There is no good explanation why memory usage would go up to 32 GB (?) within one top interval (3 sec ?). My theory about one gigantic object is debunked: you have only the 118 MB one. Perhaps there's some container or process memory limit, as Luke guessed, but it's not obvious here.
The other big hammer is "strace". If you're still interested in playing with this, you could do:
strace -vf -tt -s 200 -o /tmp/strace.out git p4 clone ....and hours later, see if something suggests itself toward the end of that output file.
-- Pete
Quoted 29 lines · Luke Diamand
I guess you could try changing the OOM score for git-fast-import.
change /proc/<pid>/oomadj.
I think a value of -31 would make it very unlikely to be killed.
Quoted 21 lines · Pete Wyckoff
I usually just do "git p4 sync @505859". The error message below crops up when things get confused. Usually after a previous error. I tend to destroy the repo and try again. Sorry I don't can't explain better what's happening here. It's not a memory issue; it reports only 24 MB used.
Bizarre. There is no good explanation why memory usage would go up to 32 GB (?) within one top interval (3 sec ?). My theory about one gigantic object is debunked: you have only the 118 MB one. Perhaps there's some container or process memory limit, as Luke guessed, but it's not obvious here.
The other big hammer is "strace". If you're still interested in playing with this, you could do:
strace -vf -tt -s 200 -o /tmp/strace.out git p4 clone ....and hours later, see if something suggests itself toward the end of that output file.
-- Pete
Finally, I claim success! Unfortunately I did not try either of the OOM score or strace suggestions - sorry! After spending so much time on this, I've gotten to the point that I'm more interested in getting it to work than in figuring out why the direct approach isn't working; it sounds like you're both pretty confident that git is working as it should, and I don't maintain the system I'm doing this on so I don't doubt that there might be some artificial limit or other quirk here that we just aren't seeing.
Anyway, what I found is that Pete's incremental method does work, I just have to know how to do it properly! This is what I WAS doing to generate the error message I pasted several posts ago:
git clone //path/to/branch@<begin>,<stage1>
cd branch
git sync //path/to/branch@<stage2>
# ERROR!
# (I also tried //path/to/branch@<stage1+1>,<stage2>, same error)
Eventually what happened is that I downloaded the free 20-user p4d, set up a very small repository with only 4 changes, and started some old fashioned trial-and-error. Here's what I should have been doing all along:
git clone //path/to/branch@<begin>,<stage1>
cd branch
git sync //path/to/branch@<begin>,<stage2>
git sync //path/to/branch@<begin>,<stage3>
# and so on...
And syncing a few thousand changes every day over the course of the past week, my git repo is finally up to the Perforce HEAD. So I suppose ultimately this was my own misunderstanding, partly because when you begin your range at the original first change number the output looks suspiciously like it's importing changes again that it's already imported. Maybe this is all documented somewhere, and if it is I just failed to find it.
Thanks to both of you for all your help!
Corey
[email protected] wrote on Fri, 06 Sep 2013 15:03 -0400:
Quoted 38 lines
Finally, I claim success! Unfortunately I did not try either of the OOM score or strace suggestions - sorry! After spending so much time on this, I've gotten to the point that I'm more interested in getting it to work than in figuring out why the direct approach isn't working; it sounds like you're both pretty confident that git is working as it should, and I don't maintain the system I'm doing this on so I don't doubt that there might be some artificial limit or other quirk here that we just aren't seeing.
Anyway, what I found is that Pete's incremental method does work, I just have to know how to do it properly! This is what I WAS doing to generate the error message I pasted several posts ago:
git clone //path/to/branch@<begin>,<stage1>
cd branch
git sync //path/to/branch@<stage2>
# ERROR!
# (I also tried //path/to/branch@<stage1+1>,<stage2>, same error)Eventually what happened is that I downloaded the free 20-user p4d, set up a very small repository with only 4 changes, and started some old fashioned trial-and-error. Here's what I should have been doing all along:
git clone //path/to/branch@<begin>,<stage1>
cd branch
git sync //path/to/branch@<begin>,<stage2>
git sync //path/to/branch@<begin>,<stage3># and so on...And syncing a few thousand changes every day over the course of the past week, my git repo is finally up to the Perforce HEAD. So I suppose ultimately this was my own misunderstanding, partly because when you begin your range at the original first change number the output looks suspiciously like it's importing changes again that it's already imported. Maybe this is all documented somewhere, and if it is I just failed to find it.
Quoted 1 line
Thanks to both of you for all your help!
That you got it to work is the most important thing. Amazing all the effort you put into it; a lesser hacker would have walked away much earlier.
The changes don't overlap. If you give it a range that includes changes already synced, git-p4 makes sure to start only at the lowest change it has not yet seen. I'll see if I can update the docs somewhere.
-- Pete