git-p4 clone @all error

Hi,

(I am a French student, sorry for my English.)

So, i want import my perforce projet on my server git.

perforce my project tree :

depot
dev_data
mainline
release_1.0
release_1.0.0

my command is :

git-p4 clone -v --detect-branches //depot@all /home/user/projets/deport

The problem :

Importing revision 7727 (100%)Traceback (most recent call last):
File "/usr/bin/git-p4", line 3183, in <module>

    main()

File "/usr/bin/git-p4", line 3177, in main
if not cmd.run(args):
File "/usr/bin/git-p4", line 3048, in run
if not P4Sync.run(self, depotPaths):
File "/usr/bin/git-p4", line 2911, in run
self.importChanges(changes)
File "/usr/bin/git-p4", line 2618, in importChanges
self.initialParent)
File "/usr/bin/git-p4", line 2198, in commit
epoch = details["time"]
KeyError: 'time'

if i make a p4 sync //depot/...#head on my perforce server i've this error :
Librarian checkout depot/mainline/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/api429decryption.txt failed. open for read: depot/mainline/xxxxxxxxxxxxxxxxxx/api429decryption.txt,v: Le fichier spcifi est introuvable.

My p4 clone can't checking out files after importing revision..

I hope I was clear ...

Thanks for your help.

--
View this message in context: http://git.661346.n2.nabble.com/git-p4-clone-all-error-tp7570219.html
Sent from the git mailing list archive at Nabble.com.

up

--
View this message in context: http://git.661346.n2.nabble.com/git-p4-clone-all-error-tp7570219p7570343.html
Sent from the git mailing list archive at Nabble.com.

Hi,

Sorry, forgot to reply-to-all, here is my response again:

Are you permanently converting a project, or are you planning to continue submitting to perforce with git-p4?

I have seen similar bugs myself when using the --detect-branches option. The branch detection in git-p4 is flaky anyway: it is limited what it can handle, and it used to require correct perforce branch specs at least, so I would recommend not using it unless you know what it is doing under the hood.

Instead I would just clone a single branch at a time (drop the --detect-branches) and work on that.

I do this even in the rare cases when I need more than one perforce branch in the same git repo - there are other ways to achieve the same thing.

- Thomas

[email protected] wrote on Wed, 31 Oct 2012 15:01 +0100:

I hope --detect-branches is unrelated to any problems related to p4 describe, because it is used identically both with and without the option. But you never know.

It relies on heuristics because the git idea of branches doesn't have a direct analog in p4, even though p4 users will organize their files and directories in ways that suggest branches. The way it works is described in the BRANCH DETECTION part of the git-p4.1 man page.

The man page also explains how you can use a config variable to explicitly define the branch relationships. This might help:

git-p4.branchList::
List of branches to be imported when branch detection is enabled. Each entry should be a pair of branch names separated by a colon (:). This example declares that both branchA and branchB were created from main:

git config git-p4.branchList main:branchA
git config --add git-p4.branchList main:branchB

It still only works for branches at the same "level" of the depot path.

Yep, that should alway works.

		-- Pete

Hi,

Here is my import :

Importing from //depot@all into XXXXX
Initialized empty Git repository in
/home/arthur/projets_git/XXXXX/XXXXX/.git/
Importing revision 4258 (43%)
Importing new branch depot/DEV_DATA

Resuming with change 4258
Importing revision 5828 (63%)
Importing new branch depot/RELEASE_1.0

Resuming with change 5828
Importing revision 7720 (88%)
Importing new branch depot/RELEASE_1.0.0

Resuming with change 7720
Importing revision 8588 (100%)
Updated branches: DEV_DATA RELEASE_1.0 MAINLINE/02_SubSystem/10_ARINC_429 MAINLINE RELEASE_1.0.0
fast-import failed: error: unable to resolve reference refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429: Not a directory
error: Unable to lock refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429
error: unable to resolve reference refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429: Not a directory
error: Unable to lock refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429
error: unable to resolve reference refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429: Not a directory
error: Unable to lock refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429
error: unable to resolve reference refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429: Not a directory
error: Unable to lock refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429
error: unable to resolve reference refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429: Not a directory
error: Unable to lock refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429
error: unable to resolve reference refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429: Not a directory
error: Unable to lock refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429
error: unable to resolve reference refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429: Not a directory
error: Unable to lock refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429 git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects: 170000
Total objects: 169644 ( 195421 duplicates )

      blobs  :        92182 (    186294 duplicates      74565 deltas)
      trees  :        70889 (      9127 duplicates      57686 deltas)
      commits:         6573 (         0 duplicates          0 deltas)
      tags   :            0 (         0 duplicates          0 deltas)
Total branches:           8 (         8 loads     )
      marks:           1024 (         0 unique    )
      atoms:          19603
Memory total:         19217 KiB
       pools:         12576 KiB

objects: 6640 KiB
---------------------------------------------------------------------
pack_report: getpagesize() = 4096
pack_report: core.packedGitWindowSize = 33554432
pack_report: core.packedGitLimit = 268435456
pack_report: pack_used_ctr = 19803
pack_report: pack_mmap_calls = 65
pack_report: pack_open_windows = 10 / 11
pack_report: pack_mapped = 257715823 / 268009874
---------------------------------------------------------------------

Import crash after importing revision, my import have not files

--
View this message in context: http://git.661346.n2.nabble.com/git-p4-clone-all-error-tp7570219p7570575.html
Sent from the git mailing list archive at Nabble.com.

[email protected] wrote on Mon, 05 Nov 2012 02:02 -0800:

Thanks, that is interesting output.

Adding Vitor who knows his way around the branch detection code. The first mail by Arthur is here, which focuses more on the backtrace from "p4 describe" errors, but also suggests something is dodgy in branch detection:

    http://thread.gmane.org/gmane.comp.version-control.git/208693

I'm not sure if this is a --verbose run, though, as I don't see any chatter like "branch is xxx". So I'm not sure if the auto-parent detection code got run.

The "unable to resolve reference" comes from refs.c, which is when fast-import calls update_branch() on each of the changed branches as it is finishing.

Vitor, what could git-p4 have done to create a branch name called "refs/remotes/p4/depot/MAINLINE/02_SubSystem/10_ARINC_429"? Any other questions for Arthur?

		-- Pete

Hi Arthur,

Could you please share with us your git-p4.branchList configuration? Would it also be possible to describe a bit more on your depot structure? I'm particularly concerned about its integration history.

I suspect the error you are experiencing is connected to a wrong configuration. Please take into account that each entry must be in the form of origin:destination, where "origin" is the folder used as the source of the integration to "destination" when first created.

Vitor

Thanks for your support,

If i get latest révision on Perforce i have this errors :

/ 590 errors reported
Librarian checkout
depot/mainline/02_subsystem/10_arinc_429/00_cots/01_bsp_aim/original files/api429decryption.txt failed.
open for read:
depot/mainline/02_subsystem/10_arinc_429/00_cots/01_bsp_aim/original files/api429decryption.txt,v: Le fichier spcifi est introuvable.
Librarian checkout
depot/mainline/02_subsystem/10_arinc_429/00_cots/01_bsp_aim/original files/api429linuxrelnotes.pdf failed.
open for read:
depot/mainline/02_subsystem/10_arinc_429/00_cots/01_bsp_aim/original files/api429linuxrelnotes.pdf,d\1.1150: Le chemin d'accs spcifi est introuvable.
Librarian checkout
depot/mainline/02_subsystem/10_arinc_429/00_cots/01_bsp_aim/original files/lien vers readme.txt failed.

    .../

Perforce can't find the files because they were removed. (option obliterate...)
So the error on the Perforce clone may come from this.

How i can show git-p4.branchList ?

My depot structure :

MAINLINE
DEV_DATA
RELEASE_1.0
RELEASE_1.0.0

--
View this message in context: http://git.661346.n2.nabble.com/git-p4-clone-all-error-tp7570219p7570622.html
Sent from the git mailing list archive at Nabble.com.

root@Srv-git:/home/arthur/projets_git# git config  git-p4.branchList 
main:MAINLINE
error: More than one value for the key git-p4.branchlist: MAINLINE:DEV_DATA
error: More than one value for the key git-p4.branchlist:
MAINLINE:RELEASE_1.0
error: More than one value for the key git-p4.branchlist:
MAINLINE:RELEASE_1.0.0

--
View this message in context: http://git.661346.n2.nabble.com/git-p4-clone-all-error-tp7570219p7570630.html
Sent from the git mailing list archive at Nabble.com.

To be honest, I don't know how git-p4 behaves when obliterate is used.

This tells me you did not setup the branchList configuration. In your ~/.gitconfig please add the following:

[git-p4]
branchList = DEV_DATA:RELEASE_1.0
branchList = DEV_DATA:RELEASE_1.0.0

This assumes that the RELEASE branches are integrated from DEV_DATA. After the import is successful you can move the [git-p4] from ~/.gitconfig to .git/config of the git repository.

Please make sure you keep in CC, because I'm not subscribed to the mailing list.

Vitor

In ~/.gitconfig i've :

[user]
name = Arthur
email = [email protected]
[git-p4]
branchList = MAINLINE:DEV_DATA
branchList = MAINLINE:RELEASE_1.0
branchList = MAINLINE:RELEASE_1.0.0

it's ok ?
So :

/# git config git-p4.branchList MAINLINE:DEV_DATA
error: More than one value for the key git-p4.branchlist:
MAINLINE:RELEASE_1.0
error: More than one value for the key git-p4.branchlist:
MAINLINE:RELEASE_1.0.0/

--
View this message in context: http://git.661346.n2.nabble.com/git-p4-clone-all-error-tp7570219p7570631.html
Sent from the git mailing list archive at Nabble.com.

You need to run "git config --get-all git-p4.branchList" to avoid those warnings.

How does "git p4 clone //depot@all" behave with these options?

Vitor

P.S. - Please keep me in Cc:

[email protected] wrote on Tue, 30 Oct 2012 03:44 -0700:

This isn't the --detect-branches code path, as far as I can tell. Line 2618, importChanges, is the not-detect case. But either way, I suspect that p4 returns results we weren't expecting, and this is just the symptom of that problem.

I've seen this before at our site. If you search for "librarian checout" you'll see lots of messages suggesting there's some sort of problem with the p4 server.

I'm going to work on a patch to git-p4 to make it print a more useful diagnostic in the case of your error above. Can you make this problem occur reliably?

		-- Pete

Group the two calls to "p4 describe" into a new helper function, and try to validate the p4 results. The current behavior when p4 describe fails is to die with a python backtrace. The new behavior will print the full response.

---

Arthur and Matt, you've both had intermittent "p4 describe" errors. I've not been able to repeat this or come up with a possible root cause. But it is clear that the error handling in this area is weak.

Can you continue using git-p4 with this patch applied? If it fails again, at least we'll get some interesting output.

This is appropriate for upstream too, since it should do no harm and might flush out a bug at some point. It includes "-s" on both p4 describe calls now as Matt suggested.

		-- Pete

 git-p4.py | 35 ++++++++++++++++++++++++++---------
 1 file changed, 26 insertions(+), 9 deletions(-)

index 882b1bb..e51a081 100755
@@ -169,6 +169,29 @@ def p4_reopen(type, f):
 def p4_move(src, dest):
     p4_system(["move", "-k", wildcard_encode(src), wildcard_encode(dest)])
 
+def p4_describe(change):
+    """Make sure it returns a valid result by checking for
+       the presence of field "time".  Return a dict of the
+       results."""
+
+    ds = p4CmdList(["describe", "-s", str(change)])
+    if len(ds) != 1:
+        die("p4 describe -s %d did not return 1 result: %s" % (change, str(ds)))
+
+    d = ds[0]
+
+    if "p4ExitCode" in d:
+        die("p4 describe -s %d exited with %d: %s" % (change, d["p4ExitCode"],
+                                                      str(d)))
+    if "code" in d:
+        if d["code"] == "error":
+            die("p4 describe -s %d returned error code: %s" % (change, str(d)))
+
+    if "time" not in d:
+        die("p4 describe -s %d returned no \"time\": %s" % (change, str(d)))
+
+    return d
+
 #
 # Canonicalize the p4 type and return a tuple of the
 # base type, plus any modifiers.  See "p4 help filetypes"

@@ -2543,7 +2566,7 @@ class P4Sync(Command, P4UserMap):
     def importChanges(self, changes):
         cnt = 1
         for change in changes:
-            description = p4Cmd(["describe", str(change)])
+            description = p4_describe(change)
             self.updateOptionDict(description)
 
             if not self.silent:

@@ -2667,14 +2690,8 @@ class P4Sync(Command, P4UserMap):
 
         # Use time from top-most change so that all git p4 clones of
         # the same p4 repo have the same commit SHA1s.
-        res = p4CmdList("describe -s %d" % newestRevision)
-        newestTime = None
-        for r in res:
-            if r.has_key('time'):
-                newestTime = int(r['time'])
-        if newestTime is None:
-            die("\"describe -s\" on newest change %d did not give a time")
-        details["time"] = newestTime
+        res = p4_describe(newestRevision)
+        details["time"] = res["time"]
 
         self.updateOptionDict(details)
         try:

1.7.12.1.457.g468b3ef

I tried this week to find the commit that caused the problem, but there were too many to sift through so I haven't managed to see it again either