Auto update submodules after merge and reset
Does anyone have a pointer to a thread/discussion that explains why git submodules are not auto updated when the superproject is updated (merge, reset, etc) by default?
Assuming a simple and default setup where submodule update policy is set to "checkout". It seems that the default and sane behavior should be to update (checkout) corresponding submodule commit to track the superproject.
I can't seem to find convincing explanation why it's not the case :). Having to manually update submodules after pull or reset has been error prone and confusing for the devs I work with.
I'm thinking about adding a config option that would enable automatic submodule update but wanted to see if there is some fundamental reason why it would not be accepted.
Thanx
Max
Am 30.11.2011 01:55, schrieb Max Krasnyansky:
Quoted 2 lines
Does anyone have a pointer to a thread/discussion that explains why git submodules are not auto updated when the superproject is updated (merge, reset, etc) by default?
This is because in current git a submodules work tree gets only updated when running "git submodule update". A default auto update wouldn't be backwards compatible (and some users really like it the way it is now). I'm working on a patch series to teach Git to optionally update the submodules work trees on checkout, reset merge and so on, but I'm not there yet.
Quoted 3 lines
Assuming a simple and default setup where submodule update policy is set to "checkout". It seems that the default and sane behavior should be to update (checkout) corresponding submodule commit to track the superproject.
I agree, but we should decide about a sane default when the feature is there. In the first version it will be off by default, so people can make up their minds about breaking backwards compatibility.
Quoted 2 lines
I can't seem to find convincing explanation why it's not the case :). Having to manually update submodules after pull or reset has been error prone and confusing for the devs I work with.
Yes, we even had some mis-merges because of that.
Quoted 2 lines
I'm thinking about adding a config option that would enable automatic submodule update but wanted to see if there is some fundamental reason why it would not be accepted.
I think adding something like an "submodule.autoupdate" config makes lots of sense, but IMO it should affect all work tree updating porcelain commands, not just merge.
Hi Jens,
Quoted 2 lines · Jens Lehmann
I'm working on a patch series to teach Git to optionally update the submodules work trees on checkout, reset merge and so on, but I'm not there yet. [SNIP]
Sorry for not replying right away.
Everything you suggested sounds great. We're on the same page (config option, etc). How far along are you? Do you have a tree I could pull from to play with things? I could help with testing, bug fixes and/or implementing parts of it. Let me know.
For now I implemented automatic submodules update using 'post-merge' hook. But obviously it does not handle reset and things. I'm thinking of adding 'post-reset' and 'pre-merge' that would be useful for this and maybe other things.
Thanx
Max
Am 06.12.2011 02:06, schrieb Max Krasnyansky:
Quoted 6 lines · Jens Lehmann
Quoted 1 line
I'm working on a patch series to teach Git to optionally update the submodules work trees on checkout, reset merge and so on, but I'm not there yet.
Everything you suggested sounds great. We're on the same page (config option, etc). How far along are you? Do you have a tree I could pull from to play with things? I could help with testing, bug fixes and/or implementing parts of it. Let me know.
Great to hear that! Please see my GitHub repo for the current state:
https://github.com/jlehmann/git-submod-enhancements
It has two interesting branches:
git-checkout-recurse-submodules:
This was my first attempt to tell unpack_trees() to checkout submodules and works quite well. Porcelain checks out submodules by default while plumbing learned the --recurse-submodules option to do that (and git gui and gitk use that option so stuff like "Revert Changes" does work on submodules :-). I use it at work for some time and it works quite well, but doesn't handle new or deleted submodules. And unfortunately the way I added the flag to control submodule checkout doesn't allow to add a per-submodule configuration option.
recursive_submodule_checkout:
This is where new development happens. I added the basic infrastructure to have global and per-submodule configuration controlling the checkout and ported the unpack_trees() changes from git-checkout-recurse-submodules here. I also added removal and creation of submodules based on the now moved gitdir. This branch has rudimentary tests but still needs quite some work.
I expect to have some time around the end of year to move things forward. It'd be cool if you could check the current state, after that we can decide how to move the topic forward together.
Quoted 3 lines
For now I implemented automatic submodules update using 'post-merge' hook. But obviously it does not handle reset and things. I'm thinking of adding 'post-reset' and 'pre-merge' that would be useful for this and maybe other things.
I doubt hooks can be more than a band aid for submodule checkout. I thought about doing that too and came to the conclusion it will only handle some of the issues. And you'll have to provide a real life use case to get a new hook accepted ;-)
Quoted 8 lines · Jens Lehmann
Am 30.11.2011 01:55, schrieb Max Krasnyansky:
I'm working on a patch series to teach Git to optionally update the submodules work trees on checkout, reset merge and so on, but I'm not there yet.Quoted 3 lines
I'm thinking about adding a config option that would enable automatic submodule update but wanted to see if there is some fundamental reason why it would not be accepted.
Because there is no good way to do so. It would be fine when you just track the submodules "read-only", but if you are actually working on submodules, it is a bad idea to always get a detached HEAD. It is also a bad idea to merge or rebase on the currently checkedout branch. Because if you are working on a maint branch in the submodule and then you checkout a pu branch in the superproject, because you have forgotten that maint branch in the submodule then all the proposed updates go to the maintenance branch -> bad. So auto-update is not easy. But below I describe an idea that might solve these issues and help auto-udpate to work in a sane way.
Quoted 3 lines
I think adding something like an "submodule.autoupdate" config makes lots of sense, but IMO it should affect all work tree updating porcelain commands, not just merge.
I was thinking about submodule integration and had the idea to bind a submodule to the superproject by having special references in the submodule like refs/super/master, refs/super/featureX... So these references are like tracking branches for the refs/heads/* of the superproject.
If you have tracking branches, the supermodule can just update the corresponding branch. If this branch is currently checkedout and the work area is clean, then the work area is updated, too. If there is currently a local branch or a diffent super-branch checked out then the working area should be considered "detached" from the superproject and not updated.
With this concept you could even switch branches in the superproject and the attached submodules follow - still having no detached HEAD. When you want to do some local work on the submodule you checkout a local branch and merge back into the super branch later. The head of that super branch might have changed by the update procedure meanwhile, but that is fine, then you just have a merge instead of a fast-forward.
Another nice feature would be a recursive commit. So all changed index files in the _attached_ submodules would first be committed in their submodules and then the superproject commits too - all with one command. Currently it feels a little bit like CVS - commit one file(submodule), commit the other file(submodule) and then apply a label(commit the superproject) to keep the changes together.
If the submodule is not attached the commit in the superproject can still detect changes that have been made to the corresponding tracking branch and pick these up.
As a summary: Tracking submodule branches in the superproject instead of only the current HEAD of the submodule gives you more freedom to install sane auto-update procedures. Even though it will raise a lot of detailed questions like "should the refs/super/* be pushed/pulled when syncing the submodule repositories".
Am 07.12.2011 10:07, schrieb Andreas T.Auer:
Quoted 7 lines
Quoted 5 lines · Jens Lehmann
Am 30.11.2011 01:55, schrieb Max Krasnyansky:
I'm working on a patch series to teach Git to optionally update the submodules work trees on checkout, reset merge and so on, but I'm not there yet.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
I'm thinking about adding a config option that would enable automatic submodule update but wanted to see if there is some fundamental reason why it would not be accepted.
Because there is no good way to do so. It would be fine when you just track the submodules "read-only", but if you are actually working on submodules, it is a bad idea to always get a detached HEAD.
YMMV. We get along *really* well with this because all developers know that if they want to hack on a submodule, they have to create a branch in there first (and if they forget to do that, git status and friends will tell them). What bugs us is that submodule HEADs don't follow what is checked out (or merged, or reset ...) in the superproject. We had some really nasty mismerges because of that, so we need the option to enable it.
Quoted 2 lines
It is also a bad idea to
merge or rebase on the currently checkedout branch.
As I'm no user of update=merge|rebase, I have no first hand experience on that. But people do use those settings, no?
Quoted 4 lines
Because if you are
working on a maint branch in the submodule and then you checkout a pu branch in the superproject, because you have forgotten that maint branch in the submodule then all the proposed updates go to the maintenance branch -> bad.
Nope, checkout will fail and not do anything as it will detect changes in the submodule to be updated by the checkout (just as it would do with a regular file).
Quoted 1 line
So auto-update is not easy.
No, it is, and it works really well. But it might not fit your use case.
Quoted 11 lines
But below I describe an idea that might solve these issues and help auto-udpate to work in a sane way.
Quoted 3 lines
I think adding something like an "submodule.autoupdate" config makes lots of sense, but IMO it should affect all work tree updating porcelain commands, not just merge.
I was thinking about submodule integration and had the idea to bind a submodule to the superproject by having special references in the submodule like refs/super/master, refs/super/featureX... So these references are like tracking branches for the refs/heads/* of the superproject.
Having stuff in the submodule reference branches in the superproject sounds upside down, as a superproject has (and should have) zero knowledge about the superproject (as it could have many different of them).
Quoted 5 lines
If you have tracking branches, the supermodule can just update the corresponding branch. If this branch is currently checkedout and the work area is clean, then the work area is updated, too. If there is currently a local branch or a diffent super-branch checked out then the working area should be considered "detached" from the superproject and not updated.
This sounds a lot like the "follow branch tip" model we discussed recently (which could be configured via .gitmodules), but I'm not sure you really are in the same boat here.
Quoted 4 lines
With this concept you could even switch branches in the superproject and the attached submodules follow - still having no detached HEAD. When you want to do some local work on the submodule you checkout a local branch and merge back into the super branch later.
You lost me here. How can you merge a submodule branch into one of the superproject?
Quoted 11 lines
The head of that super branch might have
changed by the update procedure meanwhile, but that is fine, then you just have a merge instead of a fast-forward.Another nice feature would be a recursive commit. So all changed index files in the _attached_ submodules would first be committed in their submodules and then the superproject commits too - all with one command. Currently it feels a little bit like CVS - commit one file(submodule), commit the other file(submodule) and then apply a label(commit the superproject) to keep the changes together.
Quoted 7 lines
If the submodule is not attached the commit in the superproject can still detect changes that have been made to the corresponding tracking branch and pick these up.
As a summary: Tracking submodule branches in the superproject instead of only the current HEAD of the submodule gives you more freedom to install sane auto-update procedures.
But we would want to have a deterministic update procedure, no? (And what has more freedom than a detached HEAD? ;-)
Quoted 3 lines
Even though it will raise a lot of detailed
questions like "should the refs/super/* be pushed/pulled when syncing the submodule repositories".
I doubt that is a good idea, as that might conflict with the same submodule sitting in a different superproject. But I'm interested to hear how you want to solve that.
Quoted 16 lines · On 07.12.2011 23:23 Jens Lehmann wrote:
Am 07.12.2011 10:07, schrieb Andreas T.Auer:
Quoted 7 lines
Quoted 5 lines · Jens Lehmann
Am 30.11.2011 01:55, schrieb Max Krasnyansky: I'm working on a patch series to teach Git to optionally update the submodules work trees on checkout, reset merge and so on, but I'm not there yet.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
I'm thinking about adding a config option that would enable automatic submodule update but wanted to see if there is some fundamental reason why it would not be accepted.
Because there is no good way to do so. It would be fine when you just track the submodules "read-only", but if you are actually working on submodules, it is a bad idea to always get a detached HEAD.
Quoted 4 lines
YMMV. We get along *really* well with this because all developers know that if they want to hack on a submodule, they have to create a branch in there first (and if they forget to do that, git status and friends will tell them).
Sorry, my fault. I was answering to the question why auto-update is not the default, but replied to the wrong text block. (I should have heeded the note to self about the coffee in the morning ;-) ) Having the config option is fine, of course. But it is not easy to choose a good default auto-update method, because you need different workflows for different submodules/users .
Quoted 5 lines
What bugs us is that submodule HEADs don't follow what is checked out (or merged, or reset ...) in the superproject. We had some really nasty mismerges because of that, so we need the option to enable it.
Full ack. Using the auto-update method "disabled" is a bad choice, too. ;-)
Quoted 9 lines
Quoted 4 lines
Because if you are working on a maint branch in the submodule and then you checkout a pu branch in the superproject, because you have forgotten that maint branch in the submodule then all the proposed updates go to the maintenance branch -> bad.
Nope, checkout will fail and not do anything as it will detect changes in the submodule to be updated by the checkout (just as it would do with a regular file).
Without auto-update you can easily checkout the pu branch in the superproject. And when you execute
git submodule update --merge
the pu referenced commit of the submodule will be merged into the currently checkedout maint branch of the submodule without warning unless you have merge conflicts. And when auto-update is just running git submodule update automatically it would act as I described.
But you are right, with auto-update the submodule's HEAD can be checked against the old gitlink before it is changed. If doing it in two steps it is not possible to have this check.
Quoted 7 lines
Quoted 6 lines
I was thinking about submodule integration and had the idea to bind a submodule to the superproject by having special references in the submodule like refs/super/master, refs/super/featureX... So these references are like tracking branches for the refs/heads/* of the superproject.
Quoted 5 lines
Having stuff in the submodule reference branches in the superproject
sounds upside down, as a superproject has (and should have) zero
knowledge about the superproject (as it could have many different of them).
My viewpoint is that I have a big project that is divided into submodules because not all developerss need all parts of the project. Therefore I wanted something that uses submodules as separate repos, but from the users viewpoint it should be as if the submodules are just subdirectories. It would include that diffs of submodules are not shown as a summary of commit messages but as a diff of the sources. And from that perspective it makes more sense to have tracking branches in the submodule that are owned by the superproject. In the first thought these tracking refs were meant to be readonly in the submodule and only updatable from the superproject, but then I thought the possibility of detaching and re-attaching is nice, too. One thing I've forgot to mention: the refs/super/* are not SHA1-refs to the superproject (that would be stupid indeed), but they contain the corresponding gitlink-SHA1 from the revision referenced by refs/heads/*. So when you have a detached HEAD after auto-update you would simply "git checkout -B super/<superproject-branchname>" in the submodule, with the difference that it shouldn't update refs/heads/super/*, but refs/super/* so that these branches can be treated specially.
Quoted 10 lines
Quoted 6 lines
If you have tracking branches, the supermodule can just update the corresponding branch. If this branch is currently checkedout and the work area is clean, then the work area is updated, too. If there is currently a local branch or a diffent super-branch checked out then the working area should be considered "detached" from the superproject and not updated.
This sounds a lot like the "follow branch tip" model we discussed recently (which could be configured via .gitmodules), but I'm not sure you really are in the same boat here.
When I understood that correctly it was just a configuration to what branch should be automatically checked out in the submodule. This seems to be too complicated IMO, because when you have different branches in the superproject then you may want to have different branches in the submodules, too, but you would need to configure that submodule branch in .gitmodules for each branch separately. I.e. in the master branch the .gitmodule may contain "master", in the maint branch the .gitmodules may have "maint" as the branch to follow. I do want to follow the tip of the branch, if the superproject has that currently checked out. If the superproject checks out a tagged version for a rebuild, then the submodule should not follow the tip, but should get a detached HEAD of the corresponding commit, just as the superproject. When the superproject goes back to the branch, the submodule should go back to its tracking branch.
Quoted 9 lines
Quoted 6 lines
With this concept you could even switch branches in the superproject and the attached submodules follow - still having no detached HEAD. When you want to do some local work on the submodule you checkout a local branch and merge back into the super branch later.
You lost me here. How can you merge a submodule branch into one of the superproject?
It wouldn't work, if the super/* branch would contain a superproject's SHA-1, that is right. But as explained above, it points to a commit of the submodule.
Quoted 3 lines
But we would want to have a deterministic update procedure, no? (And what has more freedom than a detached HEAD? ;-)__
I think my proposal would be deterministic.
And everything where you can commit to has more freedom than a detached HEAD
Quoted 8 lines
Quoted 4 lines
Even though it will raise a lot of detailed questions like "should the refs/super/* be pushed/pulled when syncing the submodule repositories".
I doubt that is a good idea, as that might conflict with the same submodule sitting in a different superproject. But I'm interested to hear how you want to solve that.
The first answer to my question was "yes, you need to transfer the refs or you get unreferenced objects" and "no, you can't transfer the refs, because they are owned by the superproject, not the submodule."
But binding a submodule to a superproject makes perfect sense if it is _one_ project that is split into submodules. In that case you only have one superproject for a submodule and for that purpose it would be good workflow. It is even nice to see which commits in the submodule belong to what branches in the superproject or to what release version (so tracking superproject tags would make sense, too). If you have a submodule that has more than one superproject but these are well-defined, it could be solved using refspecs (e.g. refs/super/foo/* for one and refs/super/bar/* for the other superproject), but currently I can't think of a context where this makes sense.
Of course there are other types of submodules, so using refs/super/* wouldn't be a good default variant for auto-update either. E.g. if you use a 3rdParty lib, then the detached HEAD is fine, because usually you don't touch it except when you switch to a new version from time to time.
Quoted 13 lines · On Thu, Dec 8, 2011 at 4:13 AM, <[email protected]> wrote:
Quoted 10 lines · On 07.12.2011 23:23 Jens Lehmann wrote:
Quoted 6 lines
If you have tracking branches, the supermodule can just update the corresponding branch. If this branch is currently checkedout and the work area is clean, then the work area is updated, too. If there is currently a local branch or a diffent super-branch checked out then the working area should be considered "detached" from the superproject and not updated.
This sounds a lot like the "follow branch tip" model we discussed recently (which could be configured via .gitmodules), but I'm not sure you really are in the same boat here.
Quoted 8 lines
When I understood that correctly it was just a configuration to what branch should be automatically checked out in the submodule. This seems to be too complicated IMO, because when you have different branches in the superproject then you may want to have different branches in the submodules, too, but you would need to configure that submodule branch in .gitmodules for each branch separately. I.e. in the master branch the .gitmodule may contain "master", in the maint branch the .gitmodules may have "maint" as the branch to follow.
Yes, but maybe you can update this information in the .gitmodules file easily with a command. Maybe it could be something simpler than "git sync-gitmodules-branches", but that is essentially what it would do:
it would save the current branch in each submodule as the "tracked" branch in the .gitmodules file.
The advantages to this, I think, are that
1. Your "submodule A follows branch X" information is explicit in the .gitmodules file and so it is not hidden when I examine your patch. (It sounds to me like the refs/super/* branches would necessarily be hard to find since the refs/ hierarchy is usually meta data about local and remote branches. Maybe I should think about tags and notes more, though.)
2. When you change to "submodule A now follows branch Y", this information can be unambiguously saved in the commit where it occurred rather than tucked away, again, in refs/super/*.
The disadvantage, maybe, are that you must now use 'git submodule sync' or something like that to put any .gitmodules changes into effect.
Or maybe that is an advantage. How often will this branch tracking change?
I like where you are going, though. But I have trouble following your meaning when you toss around words like "ref" and "HEAD" and "branch" and "super-branch". Maybe we can set up a strawman repo (virtually or on github) where you can explain what happens now and what you wish would happen instead.
For example, I have some repos like this:
super
+--subA +--subB
I wish I could do this:
cd super && checkout master
to get this:
super (master)
+--subA (master)
+--subB (master)Or, if I have SubB on super/'master' tracking 'foo', I could get this:
super (master)
+--subA (master)
+--subB (foo)And I wish these commands would work as if this was all one repository:
cd super && git diff master maint
cd super && git grep foo
cd subA && git grep foo -- :/
cd super && git status cd subA && git status
But I wonder what this would do:
cd super && git remote -v &&
cd subA && git remote -v
Quoted 6 lines
I do want to follow the tip of the branch, if the superproject has that currently checked out. If the superproject checks out a tagged version for a rebuild, then the submodule should not follow the tip, but should get a detached HEAD of the corresponding commit, just as the superproject. When the superproject goes back to the branch, the submodule should go back to its tracking branch.
Now this makes sense. I want the same thing. I want to preserve history on "old" commits, but I want to "advance to tip" on "new" commits.
The trouble, I think, is in telling the difference between "old" and "new". I think it means there is a switch, like --auto-follow (or --no-auto-follow for the alternate if core.auto_follow is set). But having a config option as the default is likely to break lots of scripts.
So maybe I need a new command that does this:
git checkout master && git submodule foreach 'git checkout master'
Maybe it's called "git checkout master --recurse-submodules". But I seem to recall this is already a non-starter for some reason, and anyway it doesn't solve the "variant branches in some submodules" problem.
Which brings us back to .gitmodules.
Quoted 14 lines
Quoted 8 lines
Quoted 5 lines
With this concept you could even switch branches in the superproject and the attached submodules follow - still having no detached HEAD. When you want to do some local work on the submodule you checkout a local branch and merge back into the super branch later.
You lost me here. How can you merge a submodule branch into one of the superproject?
It wouldn't work, if the super/* branch would contain a superproject's SHA-1, that is right. But as explained above, it points to a commit of the submodule.
Quoted 2 lines
Quoted 5 lines
Quoted 2 lines
But we would want to have a deterministic update procedure, no? (And what has more freedom than a detached HEAD? ;-)__
I think my proposal would be deterministic.
And everything where you can commit to has more freedom than a detached HEAD
You can commit to a detached HEAD. I do it all the time. The trick is in not switching away from a detached HEAD with your local commits still on it. :-)
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Even though it will raise a lot of detailed questions like "should the refs/super/* be pushed/pulled when syncing the submodule repositories".
I doubt that is a good idea, as that might conflict with the same submodule sitting in a different superproject. But I'm interested to hear how you want to solve that.
Quoted 6 lines
The first answer to my question was "yes, you need to transfer the refs or you get unreferenced objects" and "no, you can't transfer the refs, because they are owned by the superproject, not the submodule."
But binding a submodule to a superproject makes perfect sense if it is _one_ project that is split into submodules. In that case you only have one superproject for a submodule and for that purpose it would be good workflow.
This is not useful to me, though. Sorry.
Quoted 7 lines
It is even nice to see which commits in the submodule belong to what branches in the superproject or to what release version (so tracking superproject tags would make sense, too). If you have a submodule that has more than one superproject but these are well-defined, it could be solved using refspecs (e.g. refs/super/foo/* for one and refs/super/bar/* for the other superproject), but currently I can't think of a context where this makes sense.
I can, but this does put the cart before the horse. The submodule is subservient to the super project in all my setups. The submodule is not aware who owns him. He is a bit like the DAG in reverse. He knows one direction only (children), not the other (parents).
Phil
Quoted 30 lines · On 10.12.2011 00:57 Phil Hord wrote:
Quoted 13 lines · On Thu, Dec 8, 2011 at 4:13 AM,<[email protected]> wrote:
Quoted 11 lines · On 07.12.2011 23:23 Jens Lehmann wrote:
Quoted 7 lines
If you have tracking branches, the supermodule can just update the corresponding branch. If this branch is currently checkedout and the work area is clean, then the work area is updated, too. If there is currently a local branch or a diffent super-branch checked out then the working area should be considered "detached" from the superproject and not updated.
This sounds a lot like the "follow branch tip" model we discussed recently (which could be configured via .gitmodules), but I'm not sure you really are in the same boat here.
Quoted 9 lines
When I understood that correctly it was just a configuration to what branch should be automatically checked out in the submodule. This seems to be too complicated IMO, because when you have different branches in the superproject then you may want to have different branches in the submodules, too, but you would need to configure that submodule branch in .gitmodules for each branch separately. I.e. in the master branch the .gitmodule may contain "master", in the maint branch the .gitmodules may have "maint" as the branch to follow.
Yes, but maybe you can update this information in the .gitmodules file easily with a command. Maybe it could be something simpler than "git sync-gitmodules-branches", but that is essentially what it would do:
it would save the current branch in each submodule as the "tracked" branch in the .gitmodules file.
Quoted 9 lines
The advantages to this, I think, are that
1. Your "submodule A follows branch X" information is explicit in the .gitmodules file and so it is not hidden when I examine your patch. (It sounds to me like the refs/super/* branches would necessarily be hard to find since the refs/ hierarchy is usually meta data about local and remote branches. Maybe I should think about tags and notes more, though.)
Branches can be seen as "dynamic data" that can easily be updated, renamed or even deleted, if a branch is merged into another.
On the other hand .gitmodules can be seen as "static data" because it is committed to the object database, so if you checkout an old revision, you could get a version of the .gitmodules that refers to a branch, which existed at that time, but was deleted meanwhile.
Quoted 4 lines
2. When you change to "submodule A now follows branch Y", this information can be unambiguously saved in the commit where it occurred rather than tucked away, again, in refs/super/*.
If you place a reference in refs/super/ it will be displayed by gitk currently, so it is not really hidden.
Quoted 5 lines
The disadvantage, maybe, are that you must now use 'git submodule sync' or something like that to put any .gitmodules changes into effect.
Or maybe that is an advantage. How often will this branch tracking change?
It depends on your use case. In mine it will change quite often.
Quoted 10 lines
For example, I have some repos like this:
super
+--subA +--subBI wish I could do this:
cd super&& checkout masterto get this:
Quoted 9 lines
super (master) +--subA (master) +--subB (master)Or, if I have SubB on super/'master' tracking 'foo', I could get this:
super (master) +--subA (master) +--subB (foo)
No, the branch super/master always follows the master of the superproject. That's why it is called super/, because it contains the branchnames from the supermodule's namespace. The normal "local" submodule branches are in refs/heads/*. The references in refs/super can easily be created "on the fly" by the superproject, so they are not really properties of the submodules. It is a little bit like a cookie jar ;-).
Quoted 8 lines
Quoted 8 lines
I do want to follow the tip of the branch, if the superproject has that currently checked out. If the superproject checks out a tagged version for a rebuild, then the submodule should not follow the tip, but should get a detached HEAD of the corresponding commit, just as the superproject. When the superproject goes back to the branch, the submodule should go back to its tracking branch.
Quoted 5 lines
Now this makes sense. I want the same thing. I want to preserve history on "old" commits, but I want to "advance to tip" on "new" commits.
The trouble, I think, is in telling the difference between "old" and "new".
My approach says: Just like the superproject. If it checks out an old commit, do that, if it checks out the branch, follow.
Quoted 8 lines
So maybe I need a new command that does this:
git checkout master&& git submodule foreach 'git checkout master'Maybe it's called "git checkout master --recurse-submodules". But I seem to recall this is already a non-starter for some reason, and anyway it doesn't solve the "variant branches in some submodules" problem.
I don't know that problem, but maybe it is because the master branch of the submodule is not corresponding to the master branch of the superproject, which is a common use case, when external modules are used with different release cycles. For that reason I chose to use a different namespace in refs/super/master instead of that maybe existing refs/heads/master of the submodule.
Quoted 5 lines
You can commit to a detached HEAD. I do it all the time. The trick is in not switching away from a detached HEAD with your local commits still on it. :-)
Yes. And you can't push it, it can't be fetched, etc. So it really shouldn't be used that way, but you can do a lot of things you shouldn't do in git.
Quoted 10 lines
Quoted 7 lines
The first answer to my question was "yes, you need to transfer the refs or you get unreferenced objects" and "no, you can't transfer the refs, because they are owned by the superproject, not the submodule."
But binding a submodule to a superproject makes perfect sense if it is _one_ project that is split into submodules. In that case you only have one superproject for a submodule and for that purpose it would be good workflow.This is not useful to me, though. Sorry.
It is useful in huge projects.
Quoted 8 lines
Quoted 8 lines
It is even nice to see which commits in the submodule belong to what branches in the superproject or to what release version (so tracking superproject tags would make sense, too). If you have a submodule that has more than one superproject but these are well-defined, it could be solved using refspecs (e.g. refs/super/foo/* for one and refs/super/bar/* for the other superproject), but currently I can't think of a context where this makes sense.
Quoted 6 lines
I can, but this does put the cart before the horse. The submodule is subservient to the super project in all my setups. The submodule is not aware who owns him. He is a bit like the DAG in reverse. He knows one direction only (children), not the other (parents).
In the setup I have in mind, the submodules are not subservient to the superproject, but a part of the whole project.
Quoted 10 lines · Andreas T.Auer
Quoted 10 lines
Quoted 9 lines
It is even nice to see which commits in the submodule belong to what branches in the superproject or to what release version (so tracking superproject tags would make sense, too). If you have a submodule that has
more than one superproject but these are well-defined, it could be solved using refspecs (e.g. refs/super/foo/* for one and refs/super/bar/* for the
other superproject), but currently I can't think of a context where this makes sense.
Quoted 7 lines
Quoted 4 lines
I can, but this does put the cart before the horse. The submodule is subservient to the super project in all my setups. The submodule is not aware who owns him. He is a bit like the DAG in reverse. He knows one direction only (children), not the other (parents).
In the setup I have in mind, the submodules are not subservient to the superproject, but a part of the whole project.
I see that. I have a similar project with about 20 submodules. None of them are useful on their own; they are logical divisions of a large project.
Architecturally, submodules are oblivious of their super-projects in all other respects. Changing the architectural underpinnings should be a last resort, I think.
Phil
Quoted 8 lines · On 10.12.2011 00:57 Phil Hord wrote:
On Thu, Dec 8, 2011 at 4:13 AM,
<[email protected]> wrote:Yes, but maybe you can update this information in the .gitmodules file easily with a command. Maybe it could be something simpler than "git sync-gitmodules-branches", but that is essentially what it would do: it would save the current branch in each submodule as the "tracked" branch in the .gitmodules file.
Ok, I have read a better description of the "floating submodule" model now, so it is a different use case and somehow it makes sense. In that case there are probably just a few branches that you would like to follow, maybe an "unstable" for the newest development or "stable" for the current release or some maintenance branches.
Quoted 9 lines
Now this makes sense. I want the same thing. I want to preserve history on "old" commits, but I want to "advance to tip" on "new" commits.
The trouble, I think, is in telling the difference between "old" and
"new". I think it means there is a switch, like --auto-follow (or
--no-auto-follow for the alternate if core.auto_follow is set). But
having a config option as the default is likely to break lots of
scripts.
In my use case the branches on the submodules follow the superproject and (mostly) versions that are committed there, it just adds the possibility to keep on working without checking out a branch after an update and without colliding with existing branchnames in the submodule.
The other use case wants to follow the commits of that other submodule without checking the corresponding gitlinks into the superproject. But wouldn't it also make sense here to define actually a mapping in the .gitmodule that says "if the branch 'develop' is checkedout in the supermodule then with every submodule update automatically pull the newest 'unstable' commit from the submodule"? Or for "master" follow "stable" or for the "maint" branch follow updates in the "bugfixes" branch.
For example
[submodule "commonlib"]
update = heads develop:unstable master:stable maint:bugfixes
So whenever a defined branch is checked out in the superproject the mapped branch will be checked out in the submodule ("new" commit), but if a (e.g. tagged) commit is checked out ("old" commit) then the gitlink in the superproject is used to check out the referenced commit in the submodule.
In http://thread.gmane.org/gmane.comp.version-control.git/183837 was discussed whether the gitlink in the superproject should be set to all-zero if updates follow the tip or maybe use the SHA1 of the commit when the submodule was added. I think the gitlink should be updated everytime when a new commit in the superproject is created.
Am 11.12.2011 22:15, schrieb Andreas T.Auer:
Quoted 18 lines · On 10.12.2011 00:57 Phil Hord wrote:
Quoted 8 lines
On Thu, Dec 8, 2011 at 4:13 AM,
<[email protected]> wrote:Yes, but maybe you can update this information in the .gitmodules file easily with a command. Maybe it could be something simpler than "git sync-gitmodules-branches", but that is essentially what it would do: it would save the current branch in each submodule as the "tracked" branch in the .gitmodules file.
Ok, I have read a better description of the "floating submodule" model now, so it is a different use case and somehow it makes sense. In that case there are probably just a few branches that you would like to follow, maybe an "unstable" for the newest development or "stable" for the current release or some maintenance branches.
Quoted 4 lines
Now this makes sense. I want the same thing. I want to preserve history on "old" commits, but I want to "advance to tip" on "new" commits.
Quoted 7 lines
Quoted 5 lines
The trouble, I think, is in telling the difference between "old" and
"new". I think it means there is a switch, like --auto-follow (or
--no-auto-follow for the alternate if core.auto_follow is set). But
having a config option as the default is likely to break lots of
scripts.In my use case the branches on the submodules follow the superproject and (mostly) versions that are committed there, it just adds the possibility to keep on working without checking out a branch after an update and without colliding with existing branchnames in the submodule.
Using superproject branch names in the submodules make no sense for a lot of use cases.
Quoted 6 lines
The other use case wants to follow the commits of that other submodule without checking the corresponding gitlinks into the superproject. But wouldn't it also make sense here to define actually a mapping in the .gitmodule that says "if the branch 'develop' is checkedout in the supermodule then with every submodule update automatically pull the newest 'unstable' commit from the submodule"? Or for "master" follow "stable" or for the "maint" branch follow updates in the "bugfixes" branch.
For example
[submodule "commonlib"]
update = heads develop:unstable master:stable maint:bugfixes
Having that configured with "branch=unstable", "branch=stable" etc. in .gitmodules on the superprojects branches would be simpler and achieve the same functionality.
Quoted 1 line
So whenever a defined branch is checked out in the superproject the mapped branch will be checked out in the submodule ("new" commit), but if a (e.g. tagged) commit is checked out ("old" commit) then the gitlink in the superproject is used to check out the referenced commit in the submodule.
I think checkout should only use the submodule commit recorded in the superproject and a subsequent "git submodule update" should be needed to update the submodule to tip. Otherwise you record SHA-1 but still won't be able to bisect ...
Quoted 1 line
In http://thread.gmane.org/gmane.comp.version-control.git/183837 was discussed whether the gitlink in the superproject should be set to all-zero if updates follow the tip or maybe use the SHA1 of the commit when the submodule was added. I think the gitlink should be updated everytime when a new commit in the superproject is created.
Nope, only when "git submodule update" is run. Otherwise you'll spray the history with submodule updates totally unrelated to the commits in the superproject, which is rather confusing.
Quoted 10 lines · On 12.12.2011 23:39 Jens Lehmann wrote:
Am 11.12.2011 22:15, schrieb Andreas T.Auer:
Quoted 6 lines
In my use case the branches on the submodules follow the superproject and (mostly) versions that are committed there, it just adds the possibility to keep on working without checking out a branch after an update and without colliding with existing branchnames in the submodule.
Using superproject branch names in the submodules make no sense for a lot of use cases.
The most useful workflows for some use cases make no sense for a lot of other use cases. That is why configuration options are useful, right? There is no one-size-fits-all. It surely won't make sense for independent submodules.
Quoted 9 lines
Quoted 9 lines
The other use case wants to follow the commits of that other submodule without checking the corresponding gitlinks into the superproject. But wouldn't it also make sense here to define actually a mapping in the .gitmodule that says "if the branch 'develop' is checkedout in the supermodule then with every submodule update automatically pull the newest 'unstable' commit from the submodule"? Or for "master" follow "stable" or for the "maint" branch follow updates in the "bugfixes" branch.
Quoted 8 lines
Quoted 4 lines
For example
[submodule "commonlib"] update = heads develop:unstable master:stable maint:bugfixes
Having that configured with "branch=unstable", "branch=stable" etc. in .gitmodules on the superprojects branches would be simpler and achieve the same functionality.
Yes, this has been my first thought also, but there is also a good point to keep the .gitmodules stable, or you always have to change the file when branches change their names. So when the maint branch of version 1.3 become an archive branch and the new maint is on 1.4, which was the master before then you have to change the .gitmodules on these branches. I.e. .gitmodules of 1.4 have "stable" and must have "bugfixes" now and .gitmodules of 1.3 has "bugfixes" and must remove the floating completely. I'm not sure that this can always be solved with "easy" merging and therefore it is probably not really simpler, when you have to do this for every new release. Or what do you think?
Quoted 10 lines
Quoted 5 lines
So whenever a defined branch is checked out in the superproject the mapped branch will be checked out in the submodule ("new" commit), but if a (e.g. tagged) commit is checked out ("old" commit) then the gitlink in the superproject is used to check out the referenced commit in the submodule.
I think checkout should only use the submodule commit recorded in the superproject and a subsequent "git submodule update" should be needed to update the submodule to tip. Otherwise you record SHA-1 but still won't be able to bisect ...
bisect would leave the branch and therefore uses the recorded SHA1 for the submodule checkout instead of the tip. "follow-the-tip" should only work if the superproject follows the tip. If you configure auto-update on checkout you would not expect that a separate git submodule update has a different behavior.
Quoted 10 lines
Quoted 6 lines
In http://thread.gmane.org/gmane.comp.version-control.git/183837
was discussed whether the gitlink in the superproject should be set to all-zero if updates follow the tip or maybe use the SHA1 of the commit when the submodule was added. I think the gitlink should be updated everytime when a new commit in the superproject is created.Nope, only when "git submodule update" is run. Otherwise you'll spray the history with submodule updates totally unrelated to the commits in the superproject, which is rather confusing.
Of course, committing a new version to the superproject should not trigger pulling in a new version for the submodule or an automatic jump to the tip of the submodule. I just meant a normal manual "commit -a" behavior. Putting a 0{40} hash in the gitlink or only the hash of the submodule, when it first was added would be a special treatment that is neither needed nor wanted.
Am 13.12.2011 10:45, schrieb Andreas T.Auer:
Quoted 13 lines · On 12.12.2011 23:39 Jens Lehmann wrote:
Quoted 12 lines
Am 11.12.2011 22:15, schrieb Andreas T.Auer:
Quoted 11 lines
The other use case wants to follow the commits of that other submodule without checking the corresponding gitlinks into the superproject. But wouldn't it also make sense here to define actually a mapping in the .gitmodule that says "if the branch 'develop' is checkedout in the supermodule then with every submodule update automatically pull the newest 'unstable' commit from the submodule"? Or for "master" follow "stable" or for the "maint" branch follow updates in the "bugfixes" branch.
For example
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
[submodule "commonlib"] update = heads develop:unstable master:stable maint:bugfixes
Having that configured with "branch=unstable", "branch=stable" etc. in .gitmodules on the superprojects branches would be simpler and achieve the same functionality.
Yes, this has been my first thought also, but there is also a good point to keep the .gitmodules stable, or you always have to change the file when branches change their names. So when the maint branch of version 1.3 become an archive branch and the new maint is on 1.4, which was the master before then you have to change the .gitmodules on these branches. I.e. .gitmodules of 1.4 have "stable" and must have "bugfixes" now and .gitmodules of 1.3 has "bugfixes" and must remove the floating completely. I'm not sure that this can always be solved with "easy" merging and therefore it is probably not really simpler, when you have to do this for every new release. Or what do you think?
I never rename branches, so I do not concur ;-) And I think the .gitmodules file could benefit from a special merge driver being aware of the format and some subtleties (like not just adding a "branch" setting but rather creating a merge conflict) anyways. So I'd prefer to keep it simple and just use the .gitmodules we already have which can be different in different branches.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
So whenever a defined branch is checked out in the superproject the mapped branch will be checked out in the submodule ("new" commit), but if a (e.g. tagged) commit is checked out ("old" commit) then the gitlink in the superproject is used to check out the referenced commit in the submodule.
Quoted 6 lines
Quoted 4 lines
I think checkout should only use the submodule commit recorded in the superproject and a subsequent "git submodule update" should be needed to update the submodule to tip. Otherwise you record SHA-1 but still won't be able to bisect ...
bisect would leave the branch and therefore uses the recorded SHA1 for the submodule checkout instead of the tip. "follow-the-tip" should only work if the superproject follows the tip.
If you follow a tip there won't be any new SHA-1s recorded during that following so you could not do a bisect and expect the submodule to be what the developer had when doing the commits, no?
Quoted 1 line
If you configure auto-update on checkout you would not expect that a separate git submodule update has a different behavior.
Sure you do, when auto-update on checkout is active "git submodule update" becomes a no-op for the exact submodule model, as "git checkout" will do all the work "git submodule update" did before.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
In http://thread.gmane.org/gmane.comp.version-control.git/183837
was discussed whether the gitlink in the superproject should be set to all-zero if updates follow the tip or maybe use the SHA1 of the commit when the submodule was added. I think the gitlink should be updated everytime when a new commit in the superproject is created.
Quoted 5 lines
Quoted 3 lines
Nope, only when "git submodule update" is run. Otherwise you'll spray the history with submodule updates totally unrelated to the commits in the superproject, which is rather confusing.
Of course, committing a new version to the superproject should not trigger pulling in a new version for the submodule or an automatic jump to the tip of the submodule. I just meant a normal manual "commit -a" behavior. Putting a 0{40} hash in the gitlink or only the hash of the submodule, when it first was added would be a special treatment that is neither needed nor wanted.
I don't get that, what SHA-1 do you want to put into the gitlink? I understand that floating is not about updating the SHA-1 for the submodule each commit, right?
Quoted 24 lines · On 13.12.2011 22:44 Jens Lehmann wrote:
Am 13.12.2011 10:45, schrieb Andreas T.Auer:
Quoted 11 lines · On 12.12.2011 23:39 Jens Lehmann wrote:
Quoted 9 lines
Am 11.12.2011 22:15, schrieb Andreas T.Auer:
Quoted 4 lines
For example
[submodule "commonlib"] update = heads develop:unstable master:stable maint:bugfixes
Having that configured with "branch=unstable", "branch=stable" etc. in .gitmodules on the superprojects branches would be simpler and achieve the same functionality.
Quoted 11 lines
Yes, this has been my first thought also, but there is also a good point to keep the .gitmodules stable, or you always have to change the file when branches change their names. So when the maint branch of version 1.3 become an archive branch and the new maint is on 1.4, which was the master before then you have to change the .gitmodules on these branches. I.e. .gitmodules of 1.4 have "stable" and must have "bugfixes" now and .gitmodules of 1.3 has "bugfixes" and must remove the floating completely. I'm not sure that this can always be solved with "easy" merging and therefore it is probably not really simpler, when you have to do this for every new release. Or what do you think?
Quoted 1 line
I never rename branches, so I do not concur ;-)
Well, maybe you don't, but Junio does something like that. There is a maint-1.7.7 where maint has been before and maint jumped to master.
Quoted 4 lines
And I think the .gitmodules file could benefit from a special merge driver being aware of the format and some subtleties (like not just adding a "branch" setting but rather creating a merge conflict) anyways.
If that would work it would be fine, but you would still have to create a new commit, when maint jumps to master and you need to update the .gitmodules to be a maint .gitmodules.
Quoted 2 lines
So I'd prefer to keep it simple and just use the .gitmodules we already have which can be different in different branches.
I agree that the .gitmodule format would be simpler, but I'd prefer the .gitmodule to be a little bit more complex, but more stable.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
So whenever a defined branch is checked out in the superproject the mapped branch will be checked out in the submodule ("new" commit), but if a (e.g. tagged) commit is checked out ("old" commit) then the gitlink in the superproject is used to check out the referenced commit in the submodule.
I think checkout should only use the submodule commit recorded in the superproject and a subsequent "git submodule update" should be needed to update the submodule to tip. Otherwise you record SHA-1 but still won't be able to bisect ...
Quoted 7 lines
Quoted 3 lines
bisect would leave the branch and therefore uses the recorded SHA1 for the submodule checkout instead of the tip. "follow-the-tip" should only work if the superproject follows the tip.
If you follow a tip there won't be any new SHA-1s recorded during that following so you could not do a bisect and expect the submodule to be what the developer had when doing the commits, no?
If you never commit something to the superproject, you wouldn't get SHA1s recorded, that's right. But when you commit something to the superproject, why shouldn't the current submodule SHA1 be stored? Floating is about _ignoring_ the recorded SHA1 in _some_ cases, not about disabling the recording. So you can bisect to the bad superproject commit. If you suspect a bad submodule commit causing the problem then you could still bisect the submodule commits between the recorded SHA1s.
Quoted 7 lines · Andreas T.Auer
Quoted 4 lines · On 13.12.2011 22:44 Jens Lehmann wrote:
If you follow a tip there won't be any new SHA-1s recorded during that following so you could not do a bisect and expect the submodule to be what the developer had when doing the commits, no?
Quoted 7 lines
If you never commit something to the superproject, you wouldn't get SHA1s recorded, that's right. But when you commit something to the superproject, why shouldn't the current submodule SHA1 be stored? Floating is about _ignoring_ the recorded SHA1 in _some_ cases, not about disabling the recording. So you can bisect to the bad superproject commit. If you suspect a bad submodule commit causing the problem then you could still bisect the submodule commits between the recorded SHA1s.
To me this question is one of the more problematic aspects of floating
submodules: When to commit submodule SHA1s.
Andreas suggests always committing them whenever a submodule's tip is moved.
I don't think this is practical because commits end up with extra changes
that likely have nothing to do with the commit's topic. This makes it difficult to merge (or even cherry-pick) a commit elsewhere without also picking up an unwanted submodule update.
Jens suggests never committing submodule SHA1s. This makes it difficult to restore the submodules to the state they were in when a super-repo commit was made. To me this is unacceptable -- if commits don't encompass the entire state of the repo (including its submodules) then they're pretty much useless. Well, maybe commits themselves don't need to do that, but I sure need some way to restore the entire state of the repo.
I'm not sure there's a good or easy answer here. I suspect that it should always be up to the user whether or not submodule changes are included in a commit. The way git currently works, that means that a floating checkout would always have a modified status when in fact it isn't really dirty but merely smudged.
Wish I could propose a solution, but all I have are questions!
M.
Quoted 6 lines · Jens Lehmann
Am 11.12.2011 22:15, schrieb Andreas T.Auer:
Quoted 1 line
In http://thread.gmane.org/gmane.comp.version-control.git/183837 was discussed whether the gitlink in the superproject should be set to all-zero if updates follow the tip or maybe use the SHA1 of the commit when the submodule was added. I think the gitlink should be updated everytime when a new commit in the superproject is created.
Nope, only when "git submodule update" is run. Otherwise you'll spray the history with submodule updates totally unrelated to the commits in the superproject, which is rather confusing.
And this is why my superproject is a makefile, a .gitmodules file and a bunch of gitlinks. We only use it to track the advancement of submodule activity.
So yes, I want my superproject's gitlinks to update automatically. I just don't know a smart way to make that happen.
Phil
Am 13.12.2011 00:43, schrieb Phil Hord:
Quoted 11 lines · Jens Lehmann
Quoted 6 lines
Am 11.12.2011 22:15, schrieb Andreas T.Auer:
Quoted 1 line
In http://thread.gmane.org/gmane.comp.version-control.git/183837 was discussed whether the gitlink in the superproject should be set to all-zero if updates follow the tip or maybe use the SHA1 of the commit when the submodule was added. I think the gitlink should be updated everytime when a new commit in the superproject is created.
Nope, only when "git submodule update" is run. Otherwise you'll spray the history with submodule updates totally unrelated to the commits in the superproject, which is rather confusing.
And this is why my superproject is a makefile, a .gitmodules file and a bunch of gitlinks. We only use it to track the advancement of submodule activity.
Which is fine. Did you think about having a branch where only the submodules are updated (and built and tested) and committed to by a buildbot when everything is fine? You could then merge that branch whenever you want up-to-date submodules and have the reproducibility of the exact model while being able to "float" along the updates of that branch?
I think it always boils down to this: Either commit new gitlinks, so the submodules get updated in a reproducible manner, or don't use gitlinks at all so the submodules can float wherever they want.
Quoted 2 lines
So yes, I want my superproject's gitlinks to update automatically. I just don't know a smart way to make that happen.
Yup, that has been the endpoint of all discussions about that topic so far. And until someone comes up with a smart way to make that happen, I would rather not put something half-baked into git.
(But please tell us when you found a smart way to do that! ;-)