Working with remotes; cloning remote references

Hi all,

We're a Subversion shop moving to git, and our git-fu is still pretty weak. So I'd appreciate feedback on the setup I'm going to describe, in addition to the specific issue I'm going to raise.

Our products involve extensive customization of various externally-maintained code bases (mainly in FreeBSD, but also in a few other projects too). We'd like to track the various external repositories, to be able to bring their updates into our code base. We don't normally push our own customizations back upstream (yes, not The Way Things Should Be, please turn down the flamethrowers).

So we want our main code tree to be composed of our own code and an assortment of modified external code:

main/
external1/
external2/
ourstuffA/
ourstuffB/

Our general approach for supporting this (remember, git novices here) is to keep a local git mirror of each external code repository (say, with git-svn for the FreeBSD tree). Then we incorporate the local mirrors into our main repo, each under its own directory.

I've looked at two ways to do this: submodules and subtree merges, and the latter looks better to me. The main complication with submodules is branching and tagging. We need to branch or tag the entire main code tree, including all the external code bases. With submodules that seems like it would involve several steps to touch all the external mirror repos as well as the main repo. Another complication is that we do a lot of our work (50% or more) in the external code trees, and having to update the main repo's submodule references as that code changes seems rather fragile.

OTOH, a subtree merge nicely puts the external code into a specific spot in our main tree, lets that spot get updated with a single command (git pull -s subtree External2 master), and still let us branch and tag with single commands in the main repo. Also, the changes we make to externally-based code are plain old git pushes to the main repo. Clean and simple. (BTW, what does braid add beyond just doing subtree merges directly?)

So, first the general question: Anything braindead about the above? Any better ways to do this?

Now for the specific issue, which has to do with managing the remotes defined in the main repository. The subtree-merge approach calls for the mirrored external repos to be "git remote add"-ed to the main repo.
But clones of that repo don't include the remote references (using git
1.5.4.3 for my testing here).

I think that makes sense in most cases: Usually a developer will only want to clone the main repo and work with what's in there, ad not want to bother with the external references. But when it comes time to pull in updates from the external mirror repos the only place where that can be done is on the main repo where the original "git remote add" commands were issued.

That means merge work has to be done on the main repository's shared host (not good) and that testing merged code has to involve committing changes on the main shared repo (also not good, even in a branch) and testing them on a different host (still no goodness here).

From this point of view (and please correct me if I'm wrong), it would be good if "git clone" had an option to include a repo's remote references in the clone. The clone's origin reference would point to the original repo as usual, but all the other remotes would be in the clone.

Pushing and pulling would then also update (non-origin) the remote references.

Does this make sense? Am I missing something here? (And thanks for reading this far!)

Marc

"git clone" doesn't have this option, but you can turn it on immediately after with something similar to:
git config --add remote.origin.fetch +refs/remotes/*:refs/remotes/* (which I use for fanning-out my git-svn repos)

See "git help fetch" for further documentation.

Peter Harris

Thanks for the pointer (and the quick reply).

That doesn't seem to be what I'm looking for, though -- perhaps I'm missing something? The above puts the remotes in the .git/refs/remotes directory, but the .git/config file doesn't have them.

More specifically, if I clone the main repository and run the above and then fetch, then AFAICT I'm still not linked to the external mirrors in the clone. In particular, I can't refer to an external project in the subtree pull:

	$ git pull -s subtree external2 master
	fatal: 'external2': unable to chdir or not a git archive

(Where 'external2' is the name of the remote in the main repo.) Also, the only remote in the config file is still just the origin.

(As an aside, the above "git config --add" incantation causes problems if run inside a clone of a clone:

	$ git clone main clone-of-main
	$ git clone clone-of-main clone-of-clone-of-main
	$ cd clone-of-clone-of-main
	$ git config --add remote.origin.fetch \
		+refs/remotes/*:refs/remotes/*
	$ git pull
	fatal: refs/remotes/origin/master tracks both 
refs/remotes/origin/master and refs/heads/master
)

You did say "something similar" in your reply -- am I not seeing something obvious?

Marc

Ah. I believe I misunderstood what you wanted. Perhaps you want "git remote add", then? Unfortunately, you have to know what your remotes are outside of git for this. Perhaps a script in the root of your repository you can run to set this up after the initial clone?

I seem to recall some discussion of allowing a .gitconfig to be in repositories (similar to .gitignore), but the idea was shot down for security reasons.

I said "something similar" because you probably actually want git config --add remote.$remote.fetch +refs/remotes/*:refs/remotes/$remote/* (or other names of your own choosing) so that multiple remote remotes don't stomp on each other.

Unless you want to be able to address the remotes by name, in which case you want "git remote add" instead (see above).

Peter Harris

I think we're converging onto the same track.

Yes, I do want to use "git remote add". My point is basically that, having done various git-remote-adds in the main repository, I'd like to avoid having to redo them in a clone of that repository.

A script would work, sure, but to me this seems like something git should handle for me. If I have to re-establish my connections to the remotes whenever I want to pull in updates, then I don't see much point in keeping the remotes defined in any git repo.

I think I can understand why that would be undesirable, but I don't know if something like that would be necessary for what I'm talking about.

Marc

Marc Branchaud venit, vidit, dixit 17.10.2008 00:09:

I don't think there is a direct gittish way for transferring the remote config from one repo to a clone (other than copying what git submodule does with .gitmodules etc.).

Would it be sufficient for you if a clone could trigger the main repo to update its remotes (i.e. git remote update)?

Michael

Thanks for joining the thread, Michael!

That's my impression, too, so I'm suggesting that the clone/push/pull commands get an option to work with remotes in config.

Hmmm, I'm not sure... How would the overall merge-in-changes-from-upstream process work in that case?

Say the main repo has the code for "ThingOne" merged under a top-level thing-one/ directory:

main/$ git remote add -f ThingOne git://thing/ThingOne.git
main/$ git merge -s ours --no-commit ThingOne/master main/$ git read-tree --prefix=thing-one/ -i ThingOne/master main/$ git commit -m "Merged ThingOne into /thing-one/"

Then the ThingOne folks update their code, so we want to incorporate their changes into our version of their code. I think you're suggesting that we might make a clone of the main repo then trigger a "git remote update" in the clone's origin:

clone/$ make-origin-do-git-remote-update

What's not clear to me is how to proceed from here. At this point I'm hoping there's some way I could do some work in the clone to merge the changes the ThingOne folks did into our code. I'm too new to git to know if there is a way forward -- is there?

(With the subtree merge pattern I'd run "git pull -s subtree ThingOne master" in the main repo -- is there an equivalent achievable in the clone if the main has done a "git remote update"?)

Thanks,

Marc

Marc Branchaud venit, vidit, dixit 17.10.2008 16:44:

Sure, there's no need to mention it.

Well, maybe there is, given the recent discussion: Everyone can contribute (with or without C/perl/whatever fu) and thus "pay back" for free software, even by "only" trying to answer questions. But it requires participation, combining efforts, rather than splitting off and not returning... Uh, sorry, your question:

I thought your problem was "transporting" the remote branches from main (which tracks external) to clone, without having the remote config for external on the clones. In that case being able to trigger "remote update" on main from the clones would help.

"pull -s strategy repo master" does a fetch followed by "merge -s strategy repomaster", where repomaster is the ref for master on repo. So, if you got that branch (repomaster=ThingOne/master) by cloning from main you can do the merge (subtree or other) on your clone, even without the remote repo config for ThingOne on clone.

I guess you want to do those merges on several clones, and main, and push everything back to main. I don't know how well that mixes, but I don't expect problems, only unexpected ones ;)

Cheers,
Michael

I'm afraid I'm having trouble translating what you're saying into actual git commands (or are you proposing some new git functionality?). How would I get the ThingOne/master branch into the clone?

After some more thought I realized that the clone can just pull directly from the ThingOne repository:

clone/$ git pull -s subtree git://thing/ThingOne.git master

(I'm still getting used to git's ability to match commit IDs from anywhere -- it's magic! :) )

This goes a long way to where we want to be, in that we don't have to do our merging work in the original main repository.

It would be nice, though, if the clone were able to use the main repository's definition of the ThingOne remote. I can think of some plausible scenarios where a person could get confused about which repo/branch they're supposed to pull. It's easy to recover from that kind of mistake, but there'd be less chance of a mistake if one could tell git to "pull from X as defined in the origin repository".

And actually, git's remote functionality feels a bit crippled if clones can't make some use of the origin's remotes. Is there a reason for keeping remote definitions out of a clone?

Marc

Marc Branchaud venit, vidit, dixit 17.10.2008 21:50:

Sorry for being cryptic. What I meant was: The clones don't need the full remote config of main in order to get main's remote branches for the ThingOne remote. Say, main stores the remote tracking branches for ThingOne in "refs/remotes/ThingOne", using an appropriate remote config. Then a clone could use the refspec "refs/remotes/ThingOne/*:refs/remotes/ThingOne/*" (when fetching from main) in order to fetch those branches, without having a remote config for ThingOne on the clone. Concretely:

git config remote.main.fetch '+refs/remotes/ThingOne/*:refs/remotes/ThingOne/*'

on a clone which has a remote config "main" for the main repo.

I thought that's what you were trying to avoid...

... but this explains everything ;)

I think the approach I outlined above could solve this: main prepares everything to be pulled under refs/remotes/ThingOne, and clones (i.e. clone users) are told to fetch with a refspec like above. This way they get a copy of main's remote tracking branches for ThingOne. (Maybe you use one branch only, then the refspec is even simpler.)

Say A and B are working on a project C. Then, typically, A is interested in B's work on C, i.e. some of B's local branches, but not on B's remote
tracking branches: A tracks a "central" C already, just like B does,
and remote tracking branches don't carry any "value adding" by the cloner, they're just a local unmodified copy of the remote.

But you can always add a refspec like above.

Michael

Thanks for the explicit instructions, but I think I'm still missing something. Here's exactly what I'm trying, just to make sure I'm not doing something odd.

First I set up the remote in the main repo:

main/$ git remote add -f ThingOne git://thing/ThingOne.git
main/$ git merge -s ours --no-commit ThingOne/master main/$ git read-tree --prefix=thing-one/ -u ThingOne/master main/$ git commit -m "Adding ThingOne"

Then I make a clone, fetch main's refs to ThingOne, and try to pull in changes from ThingOne by using those refs:

$ git clone /path/to/main clone
$ cd clone
clone/$ git config remote.origin.fetch \
        '+refs/remotes/ThingOne/*:refs/remotes/ThingOne/*' 

clone/$ git fetch
From /path/to/main/
* [new branch] ThingOne/master -> ThingOne/master

clone/$ git pull -s subtree git://thing/ThingOne.git ThingOne/master
fatal: Couldn't find remote ref ThingOne/master

The config & fetch commands indeed add a .git/refs/remotes/ThingOne/master file to the clone, but I'm missing the magic words for how to use that ref.

Also:

I can appreciate that, but the approach leads to two consequences that I suggest should be avoidable:

1. A and B both have to set up the references to C themselves. A can't just piggyback on whatever B has already set up. (This is the impetus for my original message.)

2. Suppose B is just a repository that's being shared between A and D -- i.e. there's no B entity doing any work directly in B. In this case if A (or D) wants to change B's reference to C (say, to track a new branch in C), they can't: B has to be changed directly, and nothing can be done in a clone to accomplish this.

I admit I'm picking at nits here -- obviously someone is able to access the B repo and do whatever needs doing. I just feel that there are some situations where you want the origin's remotes in your clone, and some where you don't, and git should let you decide.

Marc

Marc Branchaud venit, vidit, dixit 10/20/08 18:50:

If you want to fetch main's local branches also, use option "--add" here so that you don't override the default fetch refspec (forgot last time, sorry).

because on ThingOne there is no such ref.

I thought you wanted to avoid pulling directly from ThingOne to clone? If you pull directly you might as well set up the same remote config as on main: for the correct pull line you need to know the same info as for the correct remote config.

git fetch
git merge -s subtree remotes/ThingOne/master

should do the trick. If that works you can set up things so that pulling from origin (pulling when you're in your integration branch) does that merge automatically, using branch.integrationbranch.remote=origin, branch.integrationbranch.merge=remotes/ThingOne/master (untested ;) ).

To be clear: The idea here is that main decides which ThingOne branch to store in remotes/ThingOne/master and where to get it from; clones always pull that one.

Well, it let's you decide: It tracks local branches by default, and using additional "git config" you can track remotes as well. You can also use the "--mirror" option to "git clone" or "git remote add", but that has other side effects.

Cheers,
Michael

Okay, got it.

Ah, I see (part of) our disconnect here -- sorry for not explaining it clearly before!

Yes, indeed, I do want to pull directly from ThingOne into the clone.

The scenario is that there's a bunch of us sharing the main repo, and
when some upstream changes happen in ThingOne I'd like for one of us to
be able to clone the main repo, pull the changes from ThingOne into the clone (fixing any conflicts; remember that we have our own changes in ThingOne's code), then push the merged changes back to main for everyone else to grab.

Yes, I see that -- it's why I'm basically satisfied with being able to do

clone/$ git pull -s subtree /path/to/ThingOne master

from the clone, and all I'm doing now is kvetching about having to remember the location and branch of ThingOne when that was already configured in main (and thanks for being patient with my kvetching!).

AFAICT that only refers to the clone's local branch for the ThingOne repo. The above git-config command doesn't add the ThingOne repo's URL to the clone, so I'm still stuck having to use that URL directly to pull changes from ThingOne into the clone. (And when I use the URL directly, I have to use a branch name that's defined in the ThingOne repo. I'd like to also be able to use whatever branch name got set up in main when "git remote add" was run.)

I think that "where to get it from" part is what I'm going on about. There doesn't seem to be a way for the main repo to tell the clone where the ThingOne repo is, so that the clone can pull in ThingOne changes directly.

I think we're mis-communicating, mainly because I'm not yet able to express things well in git-speak. Let me give it another stab...

I believe git lets you track the origin's _branches_ not the origin's _remotes_. I don't think --mirror does what I'm looking for, because (side effects aside) it only deals with branches, not remotes.

I find myself getting confused, and I think it's because the files in .git/refs/remotes/ are indeed tracking branches on remote repositories.
So I think our conversation gets a bit circular because our ideas of a
"remote" differ.

"git remote add" does two things (maybe more?): It adds a [remote] section to the .git/config file, and it adds a branch reference in .git/refs/remotes/. I think what I'd like is for the clone to be able to obtain both these things from the origin. The reason I think it's useful is that it would let the clone pull directly from the origin's remote repositories, without having to directly specify the remote repository's URL and branch name.

Fundamentally, I'm looking to do exactly

clone/$ git pull -s subtree /path/to/ThingOne master

i.e. pull stuff from one of main's remotes directly into the clone. But I want to replace the "/path/to/ThingOne master" part with something that means "use whatever URL and branch name was defined in the origin for this remote".

My questions are: Am I right in thinking this is desirable? Is there already some way to do this? If not, is it something worth implementing? (I'm happy to roll up my own sleeves here...)

I hope that clarifies things. Sorry for taking so long to get here!

Marc

Marc Branchaud venit, vidit, dixit 10/21/08 17:17:

Yes, I think there are multiple uses:

- a remote repository (referenced to by an URL)
- a remote config (as created by git remote add)
- a remote branch (a branch in your local repo which is a copy of a branch in a remote repo; stored under refs/remotes, never to be modified locally)
- a (remote) tracking branch (a local branch which is set up to pull from a remote branch by default)

Only the first of these 4 is something resides "remotely". Of course, "remote URL" could be a path on the local file system or even ".".

Yes.

I've got the strong impression you desire it...

Seriously, it seems desirable in cases where the remote URL changes frequently; say, because the upstream maintainer (maintainer of ThingOne) gets hit by a bus frequently, or walks the wrong streets in LA at the wrong time of the day. (I guess my seriousness ended with the ";".) Clones would follow those changes transparently then (if that feature existed).

None that I know of.

First you should decide whether it is worth for you. Comments on the list tend to kick in once people see a proposed implementation. A viable strategy could be, mimicking git submodule behaviour in part:

- Implement "git remote export reponame" which takes an existing remote config from .git/config, writes it to .gitremotes and checks in (or
better: stages for commit) the file .gitremotes [you would use this on main]

- Implement "git remote import" which reads the file .gitremotes and adds remote config to .git/config. [you would use this on clones]

- Change "git remote update" to take updates to .gitremotes into account before doing its usual routine (perhaps based on a config with default off, or a command line option, or better both)

Downside is that .gitremotes is tracked would show up as a file in the repo, but I can't come up with a better way which is as simple as the above. .gitremotes could be stored in a specially named branch, though.

Don't worry. I take partial blame ;)
And thanks for trying to match up your workflow with git.

Cheers,
Michael

(Remote) tracking branches are actually what you called remote branches, at least according to the git glossary. But I wonder, what is the right term for a branch that has the --track setup for pull?

On #git I usually fall back to some variation of "a branch that is configured for 'git pull'" or something similarly verbose (and maybe that's even partially wrong/inaccurate/incomplete?). And I always try to stick to saying "remote tracking branch" and not just "tracking branch" (as the glossary does) to avoid confusion as best as I can. But that feels quite suboptimal.

So, is there some term that describes a local branch that has been configured for "git pull"?

Thanks,
Björn

Björn Steinbrink venit, vidit, dixit 22.10.2008 18:13:

Ooops, I'm sorry. I went by the name of the "--track" option and was sure that that "git help branch" used the name "tracking branch" for the branches created with "--track", but I was wrong. So:

"(remote) tracking branch" is a local branch stored under refs/remotes/ which is a "copy" of a branch on a remote repository.

That leaves open:

- What does "remote branch" mean, if it means anything at all? It could be used for a branch in a remote repository, i.e. the other side of fetch/push refspec (remote branch:tracking branch).

- How to name a local branch created with --track off of a (remote) tracking branch? Local tracking branch? Downstream/work/modification branch?

I think that linguistically, the confusion comes from using the noun as well as the adjective "remote". As an adjective: "remote something" cleary is something residing remotely. As a noun it's the config added by "git remote add".

Note that the glossary doesn't define remote at all. I'd volunteer changing that once the discussion reaches a consensus.

My suggestion would be:

remote (noun): A configuration as created by "git remote add" which points to a remote repository (using a URL) and sets up refspecs for fetching and/or pulling. Note that the URL may point to a local filesystem or even ".". See also remote (adjective).

remote (adjective): Anything residing in another repository. See also remote (noun).

local: Sometimes used to emphasize things residing "locally", i.e. in the repository at hand, as opposed to "remotely".

tracking branch: A branch tracking a (remote) branch on a remote repository, i.e. a local branch stored under refs/remotes/ which is a "copy" of a branch on a remote repository; typically created by "git clone" or "git remote add/update".

remote tracking branch: Synonymous with tracking branch. [remote is a noun, an object for track here]

[adjective to be found] branch: A local branch which is set up to pull or rebase automatically from a tracking branch. Used for local modifications to remote branches. [I'm tempted to use local tracking branch here, but that would just add to the confusion.]

Come to think of it: "Off-track" branch would be the ideal name for a local branch created off of a (remote) tracking branch, using --track to specify the track where things go off. But I'm not sure everyone would like the implied pun...

Cheers,
Michael

I prefer to say "the branch on the remote" there, but that's just to avoid confusion with "remote tracking branch".

You can also have a local branch that "--track"s another local branch, so those names look a bit suboptimal to me, but I don't have any better ideas either :-/ For a second I thought about "pulling branch" but that doesn't really describe it either, I guess...

Those look good, to me at least.

"merge or rebase", and instead of "tracking" it should be "another" I think. Again, because you can also --track a local branch.

Same here, it doesn't apply to "remote branches" only.

Haha :-)

thanks,
Björn

So I just happened to use "git remote show origin" and that uses "remote branch" as you described it:

* remote origin
URL: git://git.kernel.org/pub/scm/git/git.git
Remote branch merged with 'git pull' while on branch master
master
Tracked remote branches

    html
    maint
    ...

JFYI
Björn

Er, "remote" is an adjective there... :)

I say there's no need for an adjective here, as this is just a plain old branch that git-branch creates by default (right?).

What's needed is an adjective for when git-branch is given the --no-track option (or when branch.automergesetup is false). "Non-tracking branch" perhaps?

Marc

Hm? It's a "branch that tracks (a branch of) a remote". Looks like a noun to me.

By default, no. It depends on the branch.automergesetup configuration and whether you create the branch from a remote tracking branch, a local branch or some other committish.

autosetupmerge = false
----
git branch foo origin/foo --> --no-track
git branch foo master --> --no-track
git branch foo origin/foo^0 --> --no-track

autosetupmerge = true
----
git branch foo origin/foo --> --track
git branch foo master --> --no-track
git branch foo origin/foo^0 --> --no-track

autosetupmerge = always
----
git branch foo origin/foo --> --track
git branch foo master --> --track
git branch foo origin/foo^0 --> --no-track

Isn't that easily confused with "[remote] tracking branch"? Especially since the "remote" is optional...

Björn

Hi,

I agree; I do not think that you need an adjective here.

I'd also recommend thinking about the branches as "local" and "remote". Not anything with "tracking" or "trailing" or somesuch. When you really look at it from the non-technical view point, even the branches you have in refs/remotes/*/ are the remote branches; the fact that you have a local cached version of them is just a minor implementation detail.

And as to the "--track" thing? I'd not make that big a deal out of it. At most, I'd say that a certain local branch happens to follow a certain remote branch, but that can change over time, right? (Just change branch.<name>.merge...)

Ciao,
Dscho

I'm still trying to figure that out...

That downside is a bit disappointing. I might as well just make "git remote export" simply generate a script of "git remote add" commands based on the contents of .git/config, and check that script in. Then I could run the script in a clone to recreate the origin's remotes.

It also seems awkward to have an export step in the origin repository. I don't really see a need for an export step (except as an artifact of the above implementation).

It seems to me that this would be more natural if our hypothetical "git remote import <X>" could just grab the remotes from repository <X> (or the origin, if <X> is unspecified). I assume that would involve lower-level changes than what you described, but to me it seems like the more usable approach. (But then I know nothing of Git's internals, so maybe this kind of change would be too much work?)

Marc

Marc Branchaud venit, vidit, dixit 27.10.2008 20:54:

Yes, if you feel okay about running a script repeatedly which may change due to being versioned; rather than running a script/command which uses versioned input.

I had the impression that you want to configure as much as possible on the "central", and have clones follow automatically. It's very likely that you want your clones to see only a subset of central's remotes. Both (individually) imply that there has to be a step on central which determines which (if any) central remotes appear in clones automatically.

Feel free to hack the protocol and remote commands... Implementing anything which exposes "server's" .git/config on the client without interaction on the server side will most probably be rejected, though. My suggestions were meant to be a minimal effort attempt within and following (see submodules) current infrastructure. I guess now it's up to you to pick the approach you deem most appropriate in your scenario and follow through with it.

Michael

Either way it's a two-step process: "git pull; git remote import" vs. "git pull; ./import-remotes.sh".

I actually don't care to limit what the clones can see. I understand the reasoning behind such control, but in my case it just doesn't apply.

Point taken, but I think I will start with no origin-side controls, since I have no need for them anyway. If/when I cobble together a patch hopefully at that point we'll be able to have a wider discussion about such controls.

Many thanks for taking the time to discuss this. Now I've got to roll up my sleeves!

(BTW, any pointers to descriptions or overviews of git's code base?)

Marc