State of Perforce importing.
I'd like to track a lot of code living in a Perforce repository, so I've been playing with 'git-p4.py'. Is the one in the contrib/fast-import directory the latest version, or is there a better place.
So far, it is having a couple of problems:
- The commit comment is empty. It doesn't seem to grab the Perforce description, and the user seems to be <a@b>.
- Every revision seems to check every file out of Perforce. This means that for the directory I want, every revision is going to take about 20 minutes.
Before I start working on it, I'd like to make sure I'm working on the latest code, though.
Thanks,
David
Quoted 3 lines · On Monday 17 September 2007 21:30:28 David Brown wrote:
I'd like to track a lot of code living in a Perforce repository, so I've been playing with '
git-p4.py'. Is the one in the contrib/fast-import directory the latest version, or is there a better place.
This is indeed the latest version (on contrib/fast-import).
Quoted 4 lines
So far, it is having a couple of problems:
- The commit comment is empty. It doesn't seem to grab the Perforce description, and the user seems to be <a@b>.
This may be a problem with the python output of perforce. Can you run the following command?
git-p4 debug change <a change number in your depot>
That should print a dictionary that has a 'desc' field containing the commit comment/log and a 'user' field that has the perforce user name.
Quoted 3 lines
- Every revision seems to check every file out of Perforce. This means that for the directory I want, every revision is going to take about
20 minutes.
For every revision only every _changed_ file is retrieved (using p4 print //path/file#revision).
Simon
Quoted 4 lines · David Brown
I'd like to track a lot of code living in a Perforce repository, so I've been playing with '
git-p4.py'. Is the one in the contrib/fast-import directory the latest version, or is there a better place.
Quoted 8 lines
So far, it is having a couple of problems:
- The commit comment is empty. It doesn't seem to grab the Perforce description, and the user seems to be <a@b>.
- Every revision seems to check every file out of Perforce. This means that for the directory I want, every revision is going to take about 20 minutes.
An additional problem:
- git-p4 doesn't preserve the execute permission bit from Perforce.
David
Quoted 10 lines · On Wednesday 19 September 2007 01:37:49 David Brown wrote:
Quoted 9 lines · David Brown
I'd like to track a lot of code living in a Perforce repository, so I've been playing with '
git-p4.py'. Is the one in the contrib/fast-import directory the latest version, or is there a better place.So far, it is having a couple of problems:
- The commit comment is empty. It doesn't seem to grab the Perforce description, and the user seems to be <a@b>.
Quoted 7 lines
Quoted 3 lines
- Every revision seems to check every file out of Perforce. This means that for the directory I want, every revision is going to take about
20 minutes.An additional problem:
- git-p4 doesn't preserve the execute permission bit from Perforce.
Hmm, can you paste the output of
p4 fstat //path/in/depot/to/file/that/is/imported/incorrectly
? I'm interested in the type of the file that p4 reports.
FWIW it works for me ;-)
Thanks,
Simon
Quoted 9 lines · Simon Hausmann
Quoted 3 lines
An additional problem:
- git-p4 doesn't preserve the execute permission bit from Perforce.
Hmm, can you paste the output of
p4 fstat //path/in/depot/to/file/that/is/imported/incorrectly
? I'm interested in the type of the file that p4 reports.
headType kxtext
so the problem is that the git-p4 is only looking for an 'x' at the start. According to 'p4 help filetypes', we need to use execute for any of:
cxtext, kxtext, uxbinary, and the others that start with 'x'.
I think it would be sufficient to check the first or second character for an 'x'. I'll make a change and give it a try later today.
David
PATCH 1/1 Detect exec bit in more cases.
git-p4 was missing the execute bit setting if the file had other attribute bits set.
--- contrib/fast-import/git-p4 | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
contrib/fast-import/git-p4 +4 -1
@@ -932,7 +932,10 @@ class P4Sync(Command): data = file['data'] mode = "644" - if file["type"].startswith("x"): + if (file["type"].startswith("x") or + file["type"].startswith("cx") or + file["type"].startswith("kx") or + file["type"].startswith("ux")): mode = "755" elif file["type"] == "symlink": mode = "120000"
Signature · 1 line
1.5.3.1
PATCH 1/1 Detect exec bit in more cases.
git-p4 was missing the execute bit setting if the file had other attribute bits set.
--- contrib/fast-import/git-p4 | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-)
contrib/fast-import/git-p4 +9 -1
@@ -63,6 +63,14 @@ def system(cmd): if os.system(cmd) != 0: die("command failed: %s" % cmd) +def isP4Exec(kind): + """Determine if a Perforce 'kind' should have execute permission + + 'p4 help filetypes' gives a list of the types. If it starts with 'x', + or x follows one of a few letters. Otherwise, if there is an 'x' after + a plus sign, it is also executable""" + return (re.search(r"(^[cku]?x)|\+.*x", kind) != None) + def p4CmdList(cmd, stdin=None, stdin_mode='w+b'): cmd = "p4 -G %s" % cmd if verbose:
@@ -932,7 +940,7 @@ class P4Sync(Command): data = file['data'] mode = "644" - if file["type"].startswith("x"): + if isP4Exec(file["type"]): mode = "755" elif file["type"] == "symlink": mode = "120000"
Signature · 1 line
1.5.3
Re: [PATCH] [git-p4] Detect exec bit in more cases.
Quoted 5 lines · David Brown
git-p4 was missing the execute bit setting if the file had other attribute bits set.
---
contrib/fast-import/git-p4 | 10 +++++++++-1 files changed, 9 insertions(+), 1 deletions(-)
I've tested this patch on our fairly large P4 repo, and at least the tip exactly matches the files that P4 finds. So, it at least should be better. git-p4 still has some problems with case-insensitive servers.
Dave
Re: [PATCH] [git-p4] Detect exec bit in more cases.
In reply to David BrownQuoted 12 lines · On Wednesday 19 September 2007 20:15:03 David Brown wrote:
git-p4 was missing the execute bit setting if the file had other attribute bits set.
--- contrib/fast-import/git-p4 | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)contrib/fast-import/git-p4 +0
-0@@ -932,7 +932,10 @@ class P4Sync(Command): data = file['data']
contrib/fast-import/git-p4 class P4Sync(Command): · line 941
Quoted 10 lines
contrib/fast-import/git-p4 +4
-1mode = "644" - if file["type"].startswith("x"): + if (file["type"].startswith("x") or + file["type"].startswith("cx") or + file["type"].startswith("kx") or + file["type"].startswith("ux")): mode = "755" elif file["type"] == "symlink": mode = "120000"
I'm fine with this, so unless you find a better way:
Simon
Re: [PATCH] [git-p4] Detect exec bit in more cases.
Quoted 14 lines · Simon Hausmann
Quoted 13 lines · On Wednesday 19 September 2007 20:15:03 David Brown wrote:
git-p4 was missing the execute bit setting if the file had other attribute bits set.
--- contrib/fast-import/git-p4 | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)contrib/fast-import/git-p4 +0
-0@@ -932,7 +932,10 @@ class P4Sync(Command): data = file['data']
Quoted 11 lines
contrib/fast-import/git-p4 class P4Sync(Command): · line 941
Quoted 9 lines
contrib/fast-import/git-p4 +4
-1mode = "644" - if file["type"].startswith("x"): + if (file["type"].startswith("x") or + file["type"].startswith("cx") or + file["type"].startswith("kx") or + file["type"].startswith("ux")): mode = "755" elif file["type"] == "symlink": mode = "120000"I'm fine with this, so unless you find a better way:
Well, I just tested it, and it still doesn't work, so I need to take some time and try to figure out what is happening.
I'm sometimes getting back 'xtext', and sometimes things like 'text+mx' back from perforce, so I need to read up, and really figure out what to look for.
David
Re: [PATCH] [git-p4] Detect exec bit in more cases.
Quoted 5 lines · David Brown
Quoted 4 lines · Simon Hausmann
Quoted 2 lines · On Wednesday 19 September 2007 20:15:03 David Brown wrote:
git-p4 was missing the execute bit setting if the file had other attribute bits set.
Quoted 8 lines
Quoted 1 line
I'm fine with this, so unless you find a better way:
Well, I just tested it, and it still doesn't work, so I need to take some time and try to figure out what is happening.
I'm sometimes getting back 'xtext', and sometimes things like 'text+mx' back from perforce, so I need to read up, and really figure out what to look for.
The output of "git p4 filetypes" was enough for me when I wrote my p4 front-end to fast-import; I never did read the p4 manual.
Have fun,
Signature · 1 line
Dana L. How [email protected] +1 650 804 5991 cell
Re: [PATCH] [git-p4] Detect exec bit in more cases.
Quoted 11 lines · Dana How
Quoted 10 lines · David Brown
Quoted 5 lines · Simon Hausmann
Quoted 2 lines · On Wednesday 19 September 2007 20:15:03 David Brown wrote:
git-p4 was missing the execute bit setting if the file had other attribute bits set.
I'm fine with this, so unless you find a better way:
Well, I just tested it, and it still doesn't work, so I need to take some time and try to figure out what is happening.
Quoted 7 lines
Quoted 3 lines
I'm sometimes getting back 'xtext', and sometimes things like 'text+mx' back from perforce, so I need to read up, and really figure out what to look for.
The output of "git p4 filetypes" was enough for me when I wrote my p4 front-end to fast-import; I never did read the p4 manual.
I have a patch based on what 'p4 help filetypes'. I'm running it now on a large test repository, and I'll compare the results when it is done. Provided it works, I'll send a new patch in a little bit.
David
Re: [PATCH] [git-p4] Detect exec bit in more cases.
In reply to Simon HausmannQuoted 9 lines · Simon Hausmann
Quoted 3 lines · On Wednesday 19 September 2007 20:15:03 David Brown wrote:
git-p4 was missing the execute bit setting if the file had other attribute bits set.
---I'm fine with this, so unless you find a better way:
Acked-by Simon Hausmann <[email protected]>
I sent out an improved version of this patch yesterday <[email protected]> that I'd like to get approved. I guess I'm not quite sure what happens at this point with a patch.
Thanks,
David
Re: [PATCH] [git-p4] Detect exec bit in more cases.
Quoted 7 lines · David Brown
Quoted 5 lines · Simon Hausmann
Quoted 3 lines · On Wednesday 19 September 2007 20:15:03 David Brown wrote:
git-p4 was missing the execute bit setting if the file had other attribute bits set.
---
Quoted 8 lines
Quoted 3 lines
I'm fine with this, so unless you find a better way:
Acked-by Simon Hausmann <[email protected]>I sent out an improved version of this patch yesterday <[email protected]> that I'd like to get approved. I guess I'm not quite sure what happens at this point with a patch.
I still have that *768445* message as "the last one proposed as better than previous ones" in my mbox.
Simon?
Simon Hausmann
Acked-byRe: [PATCH] [git-p4] Detect exec bit in more cases.
Quoted 16 lines · On Friday 21 September 2007 00:53:52 Junio C Hamano wrote:
Quoted 6 lines · David Brown
Quoted 5 lines · Simon Hausmann
Quoted 3 lines · On Wednesday 19 September 2007 20:15:03 David Brown wrote:
git-p4 was missing the execute bit setting if the file had other attribute bits set.
---Quoted 8 lines
Quoted 3 lines
I'm fine with this, so unless you find a better way:
Acked-by Simon Hausmann <[email protected]>I sent out an improved version of this patch yesterday <[email protected]> that I'd like to get approved. I guess I'm not quite sure what happens at this point with a patch.
Quoted 4 lines
I still have that *768445* message as "the last one proposed as better than previous ones" in my mbox.
Simon?
Indeed, the new improved version is much better :)
Simon
Re: [PATCH] [git-p4] Detect exec bit in more cases.
Quoted 17 lines · Simon Hausmann
Quoted 16 lines · On Friday 21 September 2007 00:53:52 Junio C Hamano wrote:
Quoted 14 lines · David Brown
Quoted 8 lines · Simon Hausmann
Quoted 3 lines · On Wednesday 19 September 2007 20:15:03 David Brown wrote:
git-p4 was missing the execute bit setting if the file had other attribute bits set.
---I'm fine with this, so unless you find a better way:
Acked-by Simon Hausmann <[email protected]>I sent out an improved version of this patch yesterday <[email protected]> that I'd like to get approved. I guess I'm not quite sure what happens at this point with a patch.
Quoted 8 lines
Quoted 4 lines
I still have that *768445* message as "the last one proposed as better than previous ones" in my mbox.
Simon?
Indeed, the new improved version is much better :)
Acked-by Simon Hausmann <[email protected]>
Thanks. This should go to 'maint' (part of v1.5.3.3) right?
Re: [PATCH] [git-p4] Detect exec bit in more cases.
Quoted 22 lines · Junio C Hamano
Quoted 20 lines · Simon Hausmann
Quoted 19 lines · On Friday 21 September 2007 00:53:52 Junio C Hamano wrote:
Quoted 6 lines · David Brown
Quoted 5 lines · Simon Hausmann
Quoted 3 lines · On Wednesday 19 September 2007 20:15:03 David Brown wrote:
git-p4 was missing the execute bit setting if the file had other attribute bits set.
---Quoted 8 lines
Quoted 3 lines
I'm fine with this, so unless you find a better way:
Acked-by Simon Hausmann <[email protected]>I sent out an improved version of this patch yesterday <[email protected]> that I'd like to get approved. I guess I'm not quite sure what happens at this point with a patch.
I still have that *768445* message as "the last one proposed as better than previous ones" in my mbox.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Simon?
Indeed, the new improved version is much better :)
Acked-by Simon Hausmann <[email protected]>Thanks. This should go to 'maint' (part of v1.5.3.3) right?
Sounds good by me. I've been using it on a few other repos, and haven't had any problems.
David
Re: [PATCH] [git-p4] Detect exec bit in more cases.
Quoted 24 lines · On Friday 21 September 2007 23:24:43 David Brown wrote:
Quoted 23 lines · Junio C Hamano
Quoted 22 lines · Simon Hausmann
Quoted 20 lines · On Friday 21 September 2007 00:53:52 Junio C Hamano wrote:
Quoted 6 lines · David Brown
Quoted 5 lines · Simon Hausmann
Quoted 3 lines · On Wednesday 19 September 2007 20:15:03 David Brown wrote:
git-p4 was missing the execute bit setting if the file had other attribute bits set.
---Quoted 8 lines
Quoted 3 lines
I'm fine with this, so unless you find a better way:
Acked-by Simon Hausmann <[email protected]>I sent out an improved version of this patch yesterday <[email protected]> that I'd like to get approved. I guess I'm not quite sure what happens at this point with a patch.
I still have that *768445* message as "the last one proposed as better than previous ones" in my mbox.
Simon?
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Indeed, the new improved version is much better :)
Acked-by Simon Hausmann <[email protected]>Thanks. This should go to 'maint' (part of v1.5.3.3) right?
Sounds good by me. I've been using it on a few other repos, and haven't had any problems.
Agreed.
Simon
Quoted 14 lines · David Brown
Quoted 9 lines · Simon Hausmann
Quoted 3 lines
An additional problem:
- git-p4 doesn't preserve the execute permission bit from Perforce.
Hmm, can you paste the output of
p4 fstat //path/in/depot/to/file/that/is/imported/incorrectly
? I'm interested in the type of the file that p4 reports.
headType kxtext
Quoted 7 lines
so the problem is that the git-p4 is only looking for an 'x' at the start. According to 'p4 help filetypes', we need to use execute for any of:
cxtext, kxtext, uxbinary, and the others that start with 'x'.
I think it would be sufficient to check the first or second character for an 'x'. I'll make a change and give it a try later today.
These are the old file types. If you read the output of p4 help filetypes, the new way of specifying this is with file type modifiers. Therefore, you also have things like text+x.
- Reece
Quoted 6 lines · Reece Dunn
Quoted 2 lines
I think it would be sufficient to check the first or second character for an 'x'. I'll make a change and give it a try later today.
These are the old file types. If you read the output of
p4 help filetypes, the new way of specifying this is with file type modifiers. Therefore, you also have things like text+x.
So my patch I just sent may not be sufficient. Thing is, we set the file type as 'text+x' and it comes back as xtext, so I'm not sure if P4 ever gives out text+x or if that is just available as a new way of specifying them.
David
Quoted 5 lines · David Brown
Quoted 3 lines · Reece Dunn
Quoted 2 lines
I think it would be sufficient to check the first or second character for an 'x'. I'll make a change and give it a try later today.
Quoted 8 lines
Quoted 3 lines
These are the old file types. If you read the output of
p4 help filetypes, the new way of specifying this is with file type modifiers. Therefore, you also have things like text+x.So my patch I just sent may not be sufficient. Thing is, we set the file type as 'text+x' and it comes back as xtext, so I'm not sure if P4 ever gives out text+x or if that is just available as a new way of specifying them.
I'm not sure. The Perforce help says that xtext and its variants are there for backward compatibility. If you are running an older server with a new client, or the other way around, they may be doing a map from text+x to xtext so that the old version can work properly. This is just speculation, though.
I don't know enough to say what Perforce is doing. I find it strange that it is reporting xtext, when you specified text+x.
Have you tried a combination that is not supported? Is xunicode supported, in which case you could try unicode+x (if you have a file that you can experiment with)?
- Reece
Quoted 4 lines · David Brown
An additional problem:
- git-p4 doesn't preserve the execute permission bit from Perforce.
FWIW I found that bit on bit 9 of the 'file type' flag in the db, which is the third column in the "db.rev" table. It's used to come up with the silly names like "text" vs "xtext" (difference? well, one's executable of course).
Sam.
Quoted 10 lines · Sam Vilain
Quoted 4 lines · David Brown
An additional problem:
- git-p4 doesn't preserve the execute permission bit from Perforce.
FWIW I found that bit on bit 9 of the 'file type' flag in the db, which is the third column in the "db.rev" table. It's used to come up with the silly names like "text" vs "xtext" (difference? well, one's executable of course).
It does come back in the 'kind' field when it asks the client for the file type. I'll look into using that information to set the execute bit in the mode it sends off.
Dave
Quoted 3 lines · David Brown
I'd like to track a lot of code living in a Perforce repository, so I've been playing with '
git-p4.py'. Is the one in the contrib/fast-import directory the latest version, or is there a better place.
I'm pretty close to giving a newer one a spin, that actually imports from the raw perforce back-end files without needing the perforce server. I am hoping that this should give a very clean import and will be very fast and efficient, sending files that share ancestry to gfi in sequence so that the on-the-fly delta system works.
If you're interested, take a look at
http://utsl.gen.nz/gitweb/?p=git-p4raw;a=summary. Expect the commands that say "WIP" to be rebased :-). It requires Postgres - I haven't yet re-written the SQL queries that step outside of MySQL's little box.
It could possibly be adapted to use the p4 client (though I'd expect that to be relatively slow per-revision), and possibly be extended to be bidirectional as all of the upstream change number information is recorded, a la git-svn.
Sam.
Quoted 5 lines · Sam Vilain
I'm pretty close to giving a newer one a spin, that actually imports from the raw perforce back-end files without needing the perforce server. I am hoping that this should give a very clean import and will be very fast and efficient, sending files that share ancestry to gfi in sequence so that the on-the-fly delta system works.
Unfortunately, this isn't something I'm going to be able to use. The Perforce server will remain live, and resides on a machine I don't have access to.
Quoted 4 lines
It could possibly be adapted to use the p4 client (though I'd expect that to be relatively slow per-revision), and possibly be extended to be bidirectional as all of the upstream change number information is recorded, a la git-svn.
I was able to get 'git-p4' to work a lot better by using @all, but it still has some problems, at least bad interactions with P4.
- It doesn't use any client spec. Our P4 server space is a complete mismash and has to be fixed up to get a sane directory layout. For example, some revisions have hundred-MB tar files sitting in the root directory and I don't want that in the repo. I also need to exclude directories, and in some cases completely rearrange the directory layout.
- Our P4 server is set to be case insensitive. 'git-p4' ignores paths that come back from the server that are specified using a different case. Unfortunately, this means that a handful of files just get randomly dropped from each revision.
I tried importing a client path instead of a depot path, but the names that come back from 'p4 files' are depot based so none ever match. I end up with a nice revision history of entirely empty trees.
I'm probably going to end up writing an importer that uses an actual client workspace to let Perforce do the client mapping. I'm also going to have to put some work into some code to clean up the log messages, since most of our changes have as a first line "New Features:", which makes for a rather uninformative shortlog.
But, I did learn about 'p4 -G' from git-p4 so that will help in getting information from the repository.
Thanks,
David
Quoted 11 lines · David Brown
Quoted 5 lines · Sam Vilain
I'm pretty close to giving a newer one a spin, that actually imports from the raw perforce back-end files without needing the perforce server. I am hoping that this should give a very clean import and will be very fast and efficient, sending files that share ancestry to gfi in sequence so that the on-the-fly delta system works.
Unfortunately, this isn't something I'm going to be able to use. The Perforce server will remain live, and resides on a machine I don't have access to.
I use git-p4 in the same way. The best approach would be to have both tools and to use whichever one best matches your needs.
Quoted 7 lines
Quoted 4 lines
It could possibly be adapted to use the p4 client (though I'd expect that to be relatively slow per-revision), and possibly be extended to be bidirectional as all of the upstream change number information is recorded, a la git-svn.
I was able to get 'git-p4' to work a lot better by using @all, but it still has some problems, at least bad interactions with P4.
I have also seen this.
Quoted 6 lines
- It doesn't use any client spec. Our P4 server space is a complete mismash and has to be fixed up to get a sane directory layout. For example, some revisions have hundred-MB tar files sitting in the root directory and I don't want that in the repo. I also need to exclude directories, and in some cases completely rearrange the directory layout.
The directory exclusion you could do the other way, if git-p4 supported multiple directory paths.
The main issues with using client workspaces is that they require you to use p4 sync, whereas git-p4 uses p4 print and that they may change as the repository changes, but Perforce does not track these changes.
That said, something like what workspaces are doing (allowing you to specify multiple paths and where they are to go) would be useful.
Quoted 4 lines
- Our P4 server is set to be case insensitive. 'git-p4' ignores paths that come back from the server that are specified using a different case. Unfortunately, this means that a handful of files just get randomly dropped from each revision.
It is worse than this. If you have:
p4 integrate foo Foo
p4 delete foo
then git-p4 will completely remove the file foo from the repository! I reported this a while back, but did not get a reply.
This is something I want to fix, as doing an '@all', rebase or sync is broken in this case when dealing with renamed files as above when importing a repository on a case insensitive system.
The alternative is to do the importing from Perforce on a Linux machine and then clone/pull/rebase from it on a Windows machine.
Quoted 3 lines
I tried importing a client path instead of a depot path, but the names that come back from 'p4 files' are depot based so none ever match. I end up with a nice revision history of entirely empty trees.
Ideally, git-p4 should bail out here with an error about the path needing to be specified as a depot path.
Quoted 2 lines
I'm probably going to end up writing an importer that uses an actual client workspace to let Perforce do the client mapping.
As eluded to above, I want to extend git-p4 to support a workspace-like importer map file to use instead of a specific depot path (with a single depot path being supported as well for backward compatibility and simplicity if the repository you are importing from has a simple layout).
There is no need to create yet another Perforce importing tool, git-p4 works well in most cases. If we focus on improving git-p4, extending it to support the functionality mentioned here, fix the issues that there are with it, then that will be more beneficial to the community as they will not have to learn another tool with a different set of bugs and issues.
Quoted 4 lines
I'm also going to have to
put some work into some code to clean up the log messages, since most of our changes have as a first line "New Features:", which makes for a rather uninformative shortlog.
I would not do that. It is a good idea to keep the original log messages, even if it does make for an uninformative shortlog. Look at some of the CVS/SVN imported logs!
- Reece
Quoted 4 lines · Reece Dunn
The main issues with using client workspaces is that they require you to use
p4 sync, whereas git-p4 usesp4 printand that they may change as the repository changes, but Perforce does not track these changes.
Unfortunately, we have one project that heavily abuses P4 client specs. For every release, someone creates a >900 line client spec and labels the files in it. Those are the versions that need to get checked in, and without rewriting much of what P4 does, I'm going to have to let P4 do the syncing and checking out.
Quoted 3 lines
I would not do that. It is a good idea to keep the original log messages, even if it does make for an uninformative shortlog. Look at some of the CVS/SVN imported logs!
I think what I want then is something to filter between 'git log' and 'git shortlog' that would find a summary line in the commit message and copy it to the top. It wouldn't change the history, but clean it up for shortlog's purpose.
David
Quoted 7 lines · David Brown
Quoted 4 lines · Reece Dunn
The main issues with using client workspaces is that they require you to use
p4 sync, whereas git-p4 usesp4 printand that they may change as the repository changes, but Perforce does not track these changes.
Quoted 5 lines
Unfortunately, we have one project that heavily abuses P4 client specs. For every release, someone creates a >900 line client spec and labels the files in it. Those are the versions that need to get checked in, and without rewriting much of what P4 does, I'm going to have to let P4 do the syncing and checking out.
If you can get a hold of the "checkpoint" and "journal" files, you could probably throw the client spec data into a few Pg tables, chuck a couple of constraints on it to confirm that it works the way you thought, and then get the information on what's where using a SQL query. The file images themselves can come from wherever, it doesn't really matter because there are MD5 hashes in the data tables you can use to confirm you got the right file.
I haven't looked at importing the client spec because it's not important for the project I'm importing. But I'd be happy to provide pointers.
Quoted 7 lines
Quoted 3 lines
I would not do that. It is a good idea to keep the original log messages, even if it does make for an uninformative shortlog. Look at some of the CVS/SVN imported logs!
I think what I want then is something to filter between 'git log' and 'git shortlog' that would find a summary line in the commit message and copy it to the top. It wouldn't change the history, but clean it up for shortlog's purpose.
Sounds like a job for a templating git-log porcelain.
Sam.
Quoted 7 lines · Sam Vilain
If you can get a hold of the "checkpoint" and "journal" files, you could probably throw the client spec data into a few Pg tables, chuck a couple of constraints on it to confirm that it works the way you thought, and then get the information on what's where using a SQL query. The file images themselves can come from wherever, it doesn't really matter because there are MD5 hashes in the data tables you can use to confirm you got the right file.
In my instance, I don't have an account on the P4 server, so I'm going to have to deal with this through P4's normal client.
I don't have much confidence that P4 really knows what it is doing when it comes to integrate, so I'm not much worried about branches. But, I do want to accurately get the history of a particular branch.
So far, the only thing that isn't working is the execute bit doesn't get set on files that should have it.
Dave
Quoted 6 lines · Reece Dunn
There is no need to create yet another Perforce importing tool, git-p4 works well in most cases. If we focus on improving git-p4, extending it to support the functionality mentioned here, fix the issues that there are with it, then that will be more beneficial to the community as they will not have to learn another tool with a different set of bugs and issues.
I like my approach; it's clean and I think shows a tasteful level of distrust towards the sanity and integrity of the data held by Perforce.
Actually it really helped me understand what was really going on;
because the information as displayed by for instance "p4 integrate" is a lot more confusing than the underlying tables (IMHO).
Sam.
Quoted 8 lines · Sam Vilain
Quoted 6 lines · Reece Dunn
There is no need to create yet another Perforce importing tool, git-p4 works well in most cases. If we focus on improving git-p4, extending it to support the functionality mentioned here, fix the issues that there are with it, then that will be more beneficial to the community as they will not have to learn another tool with a different set of bugs and issues.
Quoted 5 lines
I like my approach; it's clean and I think shows a tasteful level of distrust towards the sanity and integrity of the data held by Perforce.
Actually it really helped me understand what was really going on;
because the information as displayed by for instance "p4 integrate" is a lot more confusing than the underlying tables (IMHO).
I agree. What I wasn't clear about in that paragraph, but had eluded to in other comments in that email, is that having both git-p4 and git-p4raw is a good thing as they operate on two differing use cases. What I was referring to there is to have another equivalent of git-p4 that interfaced using the p4 client.
- Reece
I had to import quite a big Perforce depot too. And after some struggle it went fine.
For case mismatch: http://kb.perforce.com/AdminTasks/SuperuserTasks/CrossPlatfor..erMigration, bullet 10. It requires server access. For a@b and for excluding one p4 path during migration you could use my quick-and-dirty fix (attached).
You can easily extend it to include and exclude multiple paths. Then it becomes as flexible as p4 client mapping.
- Dmitry
----- Original Message ----- From: "David Brown" <[email protected]>
Newsgroups: gmane.comp.version-control.git
To: "Sam Vilain" <[email protected]>
Cc: "Git" <[email protected]>
Sent: Tuesday, 18 September 2007 8:49
Subject: Re: State of Perforce importing.
Quoted 42 lines · Sam Vilain
Quoted 6 lines
I'm pretty close to giving a newer one a spin, that actually imports from the raw perforce back-end files without needing the perforce server. I am hoping that this should give a very clean import and will be very fast and efficient, sending files that share ancestry to gfi in sequence so that the on-the-fly delta system works.
Unfortunately, this isn't something I'm going to be able to use. The Perforce server will remain live, and resides on a machine I don't have access to.
Quoted 4 lines
It could possibly be adapted to use the p4 client (though I'd expect that to be relatively slow per-revision), and possibly be extended to be bidirectional as all of the upstream change number information is recorded, a la git-svn.
I was able to get 'git-p4' to work a lot better by using @all, but it still has some problems, at least bad interactions with P4.
- It doesn't use any client spec. Our P4 server space is a complete mismash and has to be fixed up to get a sane directory layout. For example, some revisions have hundred-MB tar files sitting in the root directory and I don't want that in the repo. I also need to exclude directories, and in some cases completely rearrange the directory layout.
- Our P4 server is set to be case insensitive. 'git-p4' ignores paths that come back from the server that are specified using a different case. Unfortunately, this means that a handful of files just get randomly dropped from each revision.
I tried importing a client path instead of a depot path, but the names that come back from 'p4 files' are depot based so none ever match. I end up with a nice revision history of entirely empty trees.
I'm probably going to end up writing an importer that uses an actual client workspace to let Perforce do the client mapping. I'm also going to have to put some work into some code to clean up the log messages, since most of our changes have as a first line "New Features:", which makes for a rather uninformative shortlog.
Quoted 6 lines
But, I did learn about 'p4 -G' from git-p4 so that will help in getting information from the repository.
Thanks,
David
Froma089b02239c3bc310956964d61075b093d26549fMon Sep 17 00:00:00 2001
From: Dmitry Kakurin <[email protected]>
Date: Sun, 9 Sep 2007 13:58:12 -0700
Subject: [PATCH] git-p4: Added --exclude option and use P4 client name in commits
--- contrib/fast-import/git-p4 | 19 +++++++++++++++---- 1 files changed, 15 insertions(+), 4 deletions(-)
contrib/fast-import/git-p4 +15 -4
@@ -769,6 +769,7 @@ class P4Sync(Command): self.keepRepoPath = False self.depotPaths = None self.p4BranchesInGit = [] + self.cloneExclude = None if gitConfig("git-p4.syncFromOrigin") == "false": self.syncWithOrigin = False
@@ -779,8 +780,12 @@ class P4Sync(Command): while commit.has_key("depotFile%s" % fnum): path = commit["depotFile%s" % fnum] - found = [p for p in self.depotPaths - if path.startswith (p)] + if self.cloneExclude and path.startswith( self.cloneExclude ): + found = False + else: + found = [p for p in self.depotPaths + if path.startswith (p)] + if not found: fnum = fnum + 1 continue
@@ -905,6 +910,8 @@ class P4Sync(Command): else: committer = "%s <a@b> %s %s" % (author, epoch, self.tz) + committer = "%s <%s@%s> %s %s" % (author, details["user"], details["client"], epoch, self.tz) + self.gitStream.write("committer %s\n" % committer) self.gitStream.write("data <<EOT\n")
@@ -1540,10 +1547,14 @@ class P4Clone(P4Sync): P4Sync.__init__(self) self.description = "Creates a new git repository and imports from Perforce into it" self.usage = "usage: %prog [options] //depot/path[@revRange]" - self.options.append( + self.options += [ optparse.make_option("--destination", dest="cloneDestination", action='store', default=None, - help="where to leave result of the clone")) + help="where to leave result of the clone"), + optparse.make_option("--exclude", dest="cloneExclude", + action='store', default=None, + help="exclude depot path") + ] self.cloneDestination = None self.needsGit = False
Signature · 1 line
1.5.3.mingw.1.1.g01e3a1