git-p4

I have a git-p4 question: I work in a Perforce shop and am doing Linux kernel work, I need to share that work with colleagues who see the world as a Perforce place. The kernel I have came from Linus' tree and has a lot of history. When I try to do my first a "git p4 submit" it chokes as it looks back in the entire git history until it fails looking for the ancestor of the first commit (linux-2.6.12-rc2!), I think it is looking for the last time it did a git-p4 submit so it knows how far back to go--but it has never done a submit in this new relationship between p4 and git. There is plenty of git history that is not reflected in p4, and I don't want it in p4, I just want new work in p4.

I fear that git-p4 is for git people to contribute to bits natively p4-homed code, not this case where the code is natively git-homed code and it is the p4 people who will be contributing bits.

My attempt at a work around was this:

- create a director on the p4 side, and from the p4 side submit the
files that match my latest git submit.

- sync with git-p4

- try to submit a file with git-p4...and that fails as it runs all the
way back through the history. (Thank goodness it didn't succeed in submitting kernel activity since 2005!)

I was hoping I could merge the p4/master with master to force a stopping point in the git-p4 submit, but that doesn't work, it is searching through git history to find the stopping point. What might I put in the git history to force the stop (or maybe make a change to git-p4 to help my plight along)?

Thanks,

-kb, the Kent who has always hated Perforce, alas.

El Fri, 10 Sep 2010 15:54:16 -0400
Kent Borg <[email protected]> escribió:

[ CCing git mailing list. Looks like a better place to ask this question]

2010/9/10 Alejandro Riveira Fernández <[email protected]>:

Yes, you are right in that git-p4 looks for the last point in history where p4 and git were "in sync".

This is also correct, I'd say.

This is how I would do it too...

Well, when you did the "sync with git-p4", did it create a branch in refs/remotes/p4/master or something like that? I think that's generally how it solves the issue of knowing what to submit to p4; On the git side you have a number of commits that you _don't_ want to submit, and the most recent of these commits should have a log message that ends with:

[git-p4: depot-paths = "//Path/To/Your/Project/": change = 30049]

.. where the 'change' number is the number of the perforce changelist you synced using git-p4.

So - work that you want to submit to p4 should be rebased on top of such a commit. Then it should work to do git-p4 submit.

-Tor Arvid-

That doesn't work. I am suspecting there is a tangle in my branches.

Firing up a Python session and manually running the git-p4 findUpstreamBranchPoint() I get back something reminiscent of the commit I made at your suggestion:

['remotes/p4/master', {'change': '160991', 'depot-paths':
['//depot/imgeng/sw/inferno/kernel/linux-2.6.33-quatro']}]

I thought it looked reasonable, but when I try to run the git p4 summit, at line 811 in git-p4 it is doing:

  git rev-list --no-merges remotes/p4/master..master

And then reversing the results and trying to apply them all, starting with 1da177e (which is Linux-2.16.12-rc), and again it fails. (Again, thank goodness it failed at that.) How is git rev-list supposed to get a limited list back?

Anyone spot my silliness? Next I'm going to see if I can create a normal (and working) case for comparison. (As a newbie git is one thing, trying to understand git-p4 at the same time makes it that much harder.)

Thanks a bunch,

-kb, the Kent who Monday morning (UTC-0400) would love to casually say "Oh, yes, git-p4 is working great, both directions, no problem.".

2010/9/12 Kent Borg <[email protected]>:

Ok, so if you call this on the cmd line, it should output sha1's on all commits that will be submitted (in reverse order) to p4. If it doesn't, this may well be a good place to dig for a solution.

Say that you have only one commit to submit to p4. We'll call it X.

The parent of X should then be the commit that remotes/p4/master is pointing to. If it isn't, you can try to:

1) git checkout -b my_p4_branch p4/master
2) git cherry-pick X # we'll call this cherry-picked commit Y
3) git rev-list --no-merges remotes/p4/master..my_p4_branch
* if it lists Y and only Y, try
4) git p4 submit

-Tor Arvid-

It does output the commits it wants to submit. In my broken case far too many.

I created a simple working case to see what working looks like, and the exact same command outputs just the one commit. Looking more carefully at my gitg pictures in the good and bad cases, I realize that I in the broken case I probably don't have the graph I need.

Good case:

(master)
|
(p4/HEAD) (p4/master)
|
initial test import from existing depot

Bad case:

(master) trivial change I am trying to submit
|
(tmp) commit with the "[git-p4: depot-paths = "//depot/[...blah
blah...]": change = 160991]" string at the end of the description
|
Merge remote branch 'p4/master'

   | \
   |  (p4/HEAD)  (p4/master)  a commit that is real work
   |   |
   |   more real work
   |   |
   |   more real work
   |   |
   Merge remote branch 'p4/master'
   |  \|
   some commit in my thrashing about
   |   |
   |   more real work
   |   |
   .   .
   .   .
   .   .
   |   |
   |   initial import of no-history linux sources that I put in manually

via Perforce
|
.
.
.
|
Linux-2.6.12-rc2

I think my understanding of merges and rebases needs more depth...and I think I have mangled branches.

I tried a checkout of master and a "git rebase remotes/p4/master" and that produced thousands of conflicts. Was that due to my initial Linux sources put in on the Perforce side?

How do I untangle myself here?

I think I am about to be saved, thanks so much in advance for that!

-kb

2010/9/12 Kent Borg <[email protected]>:

<snip>

Hmm.. Let's define a "commit that contains that [git-p4:
depot-paths=...] stuff" as a git/perforce commit. Then I would say that p4/master should always point to the most recent git/perforce commit. In your "Bad case", it seems that it doesn't do that, because the most recent git/perforce commit is (tmp).

When messing about with git-p4 I have sometimes messed up that p4/master ref myself. I can normally fix it by setting it manually: In git, all existing branches are simply files with a sha1 in them (or symbolic refs that point to other refs, like "ref:
refs/heads/master"). You should thus be able to put the sha1 of your (tmp) commit into refs/remotes/p4/master.

You could try that and see if git rev-list gives the expected result.

Well, when you do that, git will look back until it finds a common ancestor commit of master and p4/master. I am guessing, that a common ancestor for those branches does not exist in your case... If my guess is correct, git will go back to the beginning of master (to linux 2.6.12-rc2) and try to apply each of those commits on top of p4/master. Naturally, that will cause a world of pain and conflicts ;)

-Tor Arvid-

Um, I maybe just did that before I saw your e-mail.

I edited:

.git/info/refs
.git/ORIG_HEAD
.git/packed/refs

...and put in the sha1 of the "[git-p4:"-commit.

When I now do:

  $ git rev-list --no-merges remotes/p4/master..master

I get just the one commit that I hoped to get!

...did I just fix it?

I don't have time to try the "git p4 submit" now, I am going to be late in meeting my wife. Tomorrow morning...will it work?

Thanks so very much for your help,

-kb

2010/9/12 Kent Borg <[email protected]>:

My guess: Yes! :)

I'm optimistic :)

-TA-

Um, not so great.

Two problems:

- I was on the wrong branch this morning when I did the "git p4 submit" (stupid me),

- I realize that even if I had done that right, part of my history is a big lump of Linux kernel files with no history but a sync out of Perforce.

So it seems what I need to do is get on a clean branch that has good history and cherry-pick all the bits I want off the branch with the bad ancestry onto the clean branch. (And get a good "[git-p4: depot-paths:
..." in place.)

Then...just point p4/HEAD and p4/master and master to the newest commit on that new clean branch. Voila!

Possibly one of my last stupid questions on this: How do I do that? "p4/master" looks like a remote, but "git remote" doesn't seem to know about it. In gitg "master" is green, just like a branch I might create, so I figure I can delete and re-create it where I want, but "p4/master" and "p4/HEAD" is blue. What are these blue things? How do I move them to a different commit?

Thanks,

-kb

I don't know if I understand what you mean here... If "git diff p4/master.." gives output equivalent to what you would want to submit to p4, then that seems correct to me.

Well, I don't know gitg, but I can guess that it shows branches as blue if they are files created inside refs/remotes/*. This is the case for p4/master and p4/HEAD. They are placed there because normally, users should not update these refs, nor should they directly do checkouts of these branches (but your case is not exactly that normal).

git remote, shows remotes as defined in your .git/config file (the [remote "someremotename"] sections). git-p4 does not need nor create such entries.

-Tor Arvid-

I am confused trying to understand what kind of beast the p4/master is.

I think my fundamental problem is that I need p4/master to point at a branch with a real git history, not a sync from Perforce (which has no git history).

I tried pointing p4/master at a clean branch in my repository with "git update-ref refs/remotes/p4/master ...", but every time I tried a "git p4 sync --verbose" it sprang back to pointing at the old commit (the one without a full git history).

So I figured I have brokenness I might be able to leave behind if I tried making a new "git p4 clone", doing a "git remote add ..." to the branch I like in my old git repository, and trying it again...but I get the same result.

After my update-ref I grep in .git for the commit hash that keeps grabbing my p4/master...and that hash is only in
logs/refs/remotes/p4/master. I do a "git p4 sync" and the hash
reappears in refs/remotes/p4/master. Where is it coming from?

How do I change where p4/master points? (Put another way, how can I have a complete Linux history in git but only new work in Perforce?)

Thanks,

-kb, the Kent who is getting a lot smarter about git in the last few days, but for whom git-p4 is not yet in control.

Well, then I think you are a bit confused ;)

The p4/master branch is git's view of your p4 history. So p4/master points to the most recent git/perforce commit. An important side point here, is that if you have another remote (which you do in your case) that is a pure git remote that knows nothing about p4, then the p4/master branch and the origin/master branch will be disjoint.

Ehm, thinking about this some more, I wonder if this whole endeavor might be continually painful...

Say that you clone Linus' kernel tree with git, and that commit X is the master where you will start your work.

You then import your tree at commit X into p4, and sync it back using git-p4. That git-p4 sync will give you a commit X' - Such that checking out X or X' will yield the same checkout tree, but they will have differing commit IDs... Do you have a strategy if you:

1) Commit something, Y', on top of X' in git (and syncs with p4).
2) Someone else commits Z on top of X, which you get when you do git fetch.

What do you do in p4? You can't be sure that it'll work to apply Z on top of Y', nor the other way around...

It kind of seems like you want to have two different repositories, and you want these to be equal (no one is the "one true repository"), and this presents problems in the scenario I described above, right?

I think maybe draw a few branch trees for git and p4/git-p4 and your own work on a good old piece of paper... It often helps to clear the head and think freshly (is that a word?).

-Tor Arvid-

That I know is true, but I am making progress.

- I have "git p4 rebase"-d changes from p4 world out to git. More than once even.

- I have "git p4 submit"-ted changes from git back into p4 world. Again, more than once.

- I can pull and push from/to this git repository to my primary git
repository.

Yes.

That, I think I fixed! The first commit on the p4/master branch used to be a sync from p4, but after surgery on branch references (correct term?) my gateway git-p4 repository's p4/master branch now has history all the way back to the beginning of time in the git universe (back to good ol' Linux 2.6.12-rc2). The recent commits have git-p4 comments that mark the matching p4 changesets. I am not sure exactly how I did it, but it seems that doing a "git-p4 rebase" instead of "git-p4 sync" made my surgery work.

One odd thing that had me worried was seeing the git side of the gateway repository show a single history back and then show a short split history and then a single history, flopping as I ran transactions through it. I am not sure what was going on, but I think git-p4 is doing an amend of the last commit to put its notes in the message, and if I have anything newer hanging from that commit this is a very bad thing. I am still worried but less so as long as I behave myself about not expecting it to make amendments to anything but the newest commits.

Part of the consideration is to simply be very aware of those "[git-p4:
..." notes and decide where this should propagate to and design the workflow accordingly. (lkml probably won't want to see p4 notations...)

But anyway, I seem to have git-p4 working in both directions, with a complete beginning-of-time history on the git side.

Tor Arvid: I owe you a beer (or whatever you drink when someone offers you a beer), how often do you visit Boston?

Thanks for everyone's patience with a newbie,

-kb

This is true. git-p4 does rebase (which usually rewrites history) the active branch as the last step when you do git-p4 submit. So, as you say, it is important to be aware of this.

If HEAD points to X, and you do git-p4 submit, then if you have another branch YY on top of X, you may want to checkout YY and do git rebase X' (where X' is what git-p4 produces after it amends its [git-p4: ...] stuff).

Good stuff! Congrats :)

Appreciated :) Well, its been ~9 years, so maybe I should go again soon :) Otherwise, say hello whenever you're in Oslo ;)

-TA-