make submodule commands robust to having non-submodule Subprojects

wishlist? Yaroslav Halchenko 5 participants 13 messages Sep 15, 2016 – Sep 16, 2016

Share & tools

Yaroslav Halchenko

author

2016-09-15 13:03 UTC #0 lore raw

NB echos some questions of mine a few days back on IRC about Subprojects and submodules

If e.g. you just 'git add' a subdirectory which is a git repository, git adds it as a subproject but doesn't initiate any entry in .gitmodules since it is the job done by submodule and git core itself is agnostic of those beasts.

But having then this "Subproject"s which aren't registered as submodules (and I haven't found any other use for them besides being a submodule) brakes "git submodule" commands, e.g.

$> git submodule
cc6a09ac06c13cf06b4f4c8b54cda9a535e4e385 ds000001 (2.0.0+4) 0a9f3b66e06a2137311a537b7377c336f1fb30ad ds000002 (1.0.0-3-g0a9f3b6) 9da7e4f4221699915645ac2003298c6aba2db109 ds000003 (1.1.0+4) fe16cacb5cb9b4d53c50e498298fab182500e147 ds000005 (2.0.0+3) 6898d99ff3ba26880183ed3672a458a7fcde1737 ds000006 (2.0.0+2) bbd10f634fe87e9d5853df3a891edbdb18cda7f9 ds000007 (2.0.0+3) 138e6730193c0585a69b8baf5b9d7a4439e83ecc ds000008 (2.0.0+2) ddf3a4cf7ce51a01a664e6faff4b8334b8414b1f ds000009 (2.0.1+1) 7fa73b4df8166dba950c7dc07c3f8cdd50fca313 ds000011 (1.0.0-5-g7fa73b4)
fatal: no submodule mapping found in .gitmodules for path 'ds000017

which then stops without even looking at other submodules.

I think it would be more logical to make it a 'warning:' not a 'fatal:' and proceed.

Thank you for consideration

Signature · 5 lines

Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Stefan Beller

2016-09-15 16:05 UTC #1 lore raw

Quoted 24 lines · Yaroslav Halchenko

NB echos some questions of mine a few days back on IRC about Subprojects and submodules
If e.g. you just 'git add' a subdirectory which is a git repository, git adds it as a subproject but doesn't initiate any entry in .gitmodules since it is the job done by submodule and git core itself is agnostic of those beasts.
But having then this "Subproject"s which aren't registered as submodules (and I haven't found any other use for them besides being a submodule) brakes "git submodule" commands, e.g.
$> git submodule
cc6a09ac06c13cf06b4f4c8b54cda9a535e4e385 ds000001 (2.0.0+4) 0a9f3b66e06a2137311a537b7377c336f1fb30ad ds000002 (1.0.0-3-g0a9f3b6) 9da7e4f4221699915645ac2003298c6aba2db109 ds000003 (1.1.0+4) fe16cacb5cb9b4d53c50e498298fab182500e147 ds000005 (2.0.0+3) 6898d99ff3ba26880183ed3672a458a7fcde1737 ds000006 (2.0.0+2) bbd10f634fe87e9d5853df3a891edbdb18cda7f9 ds000007 (2.0.0+3) 138e6730193c0585a69b8baf5b9d7a4439e83ecc ds000008 (2.0.0+2) ddf3a4cf7ce51a01a664e6faff4b8334b8414b1f ds000009 (2.0.1+1) 7fa73b4df8166dba950c7dc07c3f8cdd50fca313 ds000011 (1.0.0-5-g7fa73b4)
fatal: no submodule mapping found in .gitmodules for path 'ds000017

Quoted 4 lines

which then stops without even looking at other submodules.
I think it would be more logical to make it a 'warning:' not a 'fatal:' and proceed.

So maybe we would want to introduce a switch
--existing-but-unconfigure-gitlinks=(warn|ignore)
as well as
git config submodule.existing-but-unconfigured (warn|ignore) for a more permanent solution?

Quoted 8 lines

Thank you for consideration
--
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Yaroslav Halchenko

author

2016-09-15 16:40 UTC #2 lore raw

Quoted 8 lines · Stefan Beller

Quoted 2 lines
I think it would be more logical to make it a 'warning:' not a 'fatal:' and proceed.
So maybe we would want to introduce a switch
--existing-but-unconfigure-gitlinks=(warn|ignore)
as well as
git config submodule.existing-but-unconfigured (warn|ignore) for a more permanent solution?

possibly ignorant question: is gitlink === Subproject or a
Subproject is a kinda of a gitlink and there are other gitlinks which aren't Subprojects? ;)

Signature · 5 lines

Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Stefan Beller

2016-09-15 17:29 UTC #3 lore raw

Quoted 12 lines · Yaroslav Halchenko

Quoted 8 lines · Stefan Beller
Quoted 2 lines
I think it would be more logical to make it a 'warning:' not a 'fatal:' and proceed.
So maybe we would want to introduce a switch
--existing-but-unconfigure-gitlinks=(warn|ignore)
as well as
git config submodule.existing-but-unconfigured (warn|ignore) for a more permanent solution?

Quoted 4 lines

possibly ignorant question: is gitlink === Subproject or a
Subproject is a kinda of a gitlink and there are other gitlinks which aren't Subprojects? ;)

gitlink is the internal name (just like a file is called blob, or a directory is called tree; there is no file system equivalent for e.g. commits, or gitlinks)

gitlinks as a basic building block only points at a sha1 to be part of the repository.

Submodules use gitlinks to point at (usually) different projects, e.g. a library.

Subprojects in Git is a loose term, it could mean submodule or part of your repo merged in via the git-subtree command. Or a sub project can be just a loose repository inside your repository.

gitlinks do not necessarily need to track other projects; it's "just a pointer" to a specific version of a repository.

So to come back to your question: yeah In that response I meant submodule==gitlink, though naming the internal data structure to be more explicit. (What if we invent a new way to do submodules not using gitlinks? Then that config option may be less confusing and backwards compatible. That is unlikely though. So I guess we'll settle with a different name)

Junio C Hamano

2016-09-15 18:02 UTC #4 lore raw

In reply to Stefan Beller

Quoted 19 lines · Stefan Beller

Quoted 18 lines · Yaroslav Halchenko
If e.g. you just 'git add' a subdirectory which is a git repository, git adds it as a subproject but doesn't initiate any entry in .gitmodules since it is the job done by submodule and git core itself is agnostic of those beasts.
...
$> git submodule
cc6a09ac06c13cf06b4f4c8b54cda9a535e4e385 ds000001 (2.0.0+4) 0a9f3b66e06a2137311a537b7377c336f1fb30ad ds000002 (1.0.0-3-g0a9f3b6) 9da7e4f4221699915645ac2003298c6aba2db109 ds000003 (1.1.0+4) fe16cacb5cb9b4d53c50e498298fab182500e147 ds000005 (2.0.0+3) 6898d99ff3ba26880183ed3672a458a7fcde1737 ds000006 (2.0.0+2) bbd10f634fe87e9d5853df3a891edbdb18cda7f9 ds000007 (2.0.0+3) 138e6730193c0585a69b8baf5b9d7a4439e83ecc ds000008 (2.0.0+2) ddf3a4cf7ce51a01a664e6faff4b8334b8414b1f ds000009 (2.0.1+1) 7fa73b4df8166dba950c7dc07c3f8cdd50fca313 ds000011 (1.0.0-5-g7fa73b4)
fatal: no submodule mapping found in .gitmodules for path 'ds000017

Quoted 4 lines

Quoted 4 lines
which then stops without even looking at other submodules.
I think it would be more logical to make it a 'warning:' not a 'fatal:' and proceed.

Making "git submodule" listing to continue from that point may be one thing, but do we have a sensible way in "git submodule add" to allow the user to recover from this condition? That is, "git add" is a right way to tell the core level that there is a gitlink, but as Yaroslav correctly observed in the early part of his message, having that gitlink alone is not good enough for the world view of "git submodule" that sits at higher-layer. And the usual way to tell the submodule layer that there is a submodule is with "git submodule add", but

	$ git init top
        $ cd top
        $ git commit --allow-empty -m 'initial in top'
        $ git init sub
        $ git -C sub commit --allow-empty -m 'initial in sub'

        $ git add sub
	$ git submodule
        fatal: no submodule mapping found in .gitmodules for path 'sub'

	$ git submodule add ./sub sub
        'sub' already exists in the index
	$ git submodule add -f ./sub sub
        'sub' already exists in the index

I highly suspect that the user will then get stuck at this point, after trying to "submodule add" and then even attempting to force it.

I think that is a more pressing thing to address. Once we make it easier for the user to bring a half-initialized submodule properly into the world view of the submodule subsystem, we would have to worry about the reported failure case even less and you do not need to pile on workaround options to let things continue in a state that is half-broken (that is, in a state that is perfectly sane to the core layer, but is not liked by the submodule layer).

Yaroslav Halchenko

author

2016-09-15 18:13 UTC #5 lore raw

Quoted 21 lines · Junio C Hamano

Quoted 4 lines
Quoted 4 lines
which then stops without even looking at other submodules.
I think it would be more logical to make it a 'warning:' not a 'fatal:' and proceed.
Making "git submodule" listing to continue from that point may be one thing, but do we have a sensible way in "git submodule add" to allow the user to recover from this condition? That is, "git add" is a right way to tell the core level that there is a gitlink, but as Yaroslav correctly observed in the early part of his message, having that gitlink alone is not good enough for the world view of "git submodule" that sits at higher-layer. And the usual way to tell the submodule layer that there is a submodule is with "git submodule add", but
	$ git init top
        $ cd top
        $ git commit --allow-empty -m 'initial in top'
        $ git init sub
        $ git -C sub commit --allow-empty -m 'initial in sub'

Quoted 8 lines

        $ git add sub
	$ git submodule
        fatal: no submodule mapping found in .gitmodules for path 'sub'

	$ git submodule add ./sub sub
        'sub' already exists in the index
	$ git submodule add -f ./sub sub
        'sub' already exists in the index

FWIW
I could have sworn that I have tried to 'submodule add' it and it worked... but pragmatically I just did edit .gitmodules, added the record for it, committed it, and then iirc git submodule update --init which seemed to make git happy... FTR:

$> git submodule add ./sub ./sub
'sub' already exists in the index

$> git submodule add ./sub/ ./sub/
'sub' already exists in the index

$> vim .gitmodules

$> git add .gitmodules
cached/staged changes:

 .gitmodules | 4 ++++
 sub         | 1 +

$> git submodule update --init
Submodule 'sub' (/tmp/111/top/sub) registered for path 'sub' cached/staged changes:

 .gitmodules | 4 ++++
 sub         | 1 +

$> git commit -m 'added finally'
[master aa6d912] added finally

 2 files changed, 5 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 sub

$> git submodule
6f574b298ef51aebd36daafad450a3e38802ca03 sub (heads/master)

Quoted 11 lines

I highly suspect that the user will then get stuck at this point, after trying to "submodule add" and then even attempting to force it.
I think that is a more pressing thing to address. Once we make it easier for the user to bring a half-initialized submodule properly into the world view of the submodule subsystem, we would have to worry about the reported failure case even less and you do not need to pile on workaround options to let things continue in a state that is half-broken (that is, in a state that is perfectly sane to the core layer, but is not liked by the submodule layer).

do you foresee any unpleasant side-effects from above manual editing .gitmodules/submodule update --init ?

Signature · 5 lines

Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Stefan Beller

2016-09-15 18:16 UTC #6 lore raw

Quoted 3 lines · Yaroslav Halchenko

do you foresee any unpleasant side-effects from above manual editing .gitmodules/submodule update --init ?

I think that is fine, but un(der)documented. So you have to figure it out from experience what to do exactly.

Junio C Hamano

2016-09-15 18:29 UTC #7 lore raw

In reply to Yaroslav Halchenko

Quoted 2 lines · Yaroslav Halchenko

do you foresee any unpleasant side-effects from above manual editing .gitmodules/submodule update --init ?

I do not think so; you essentially did what a canned command we should have had should have done by hand because of a lack of such a feature.

Stefan Beller

2016-09-15 18:15 UTC #8 lore raw

In reply to Junio C Hamano

Quoted 7 lines · Junio C Hamano

I think that is a more pressing thing to address. Once we make it easier for the user to bring a half-initialized submodule properly into the world view of the submodule subsystem, we would have to worry about the reported failure case even less and you do not need to pile on workaround options to let things continue in a state that is half-broken (that is, in a state that is perfectly sane to the core layer, but is not liked by the submodule layer).

Heh, I see.

When searching around the net, some people use half initialized submodules intentionally, e.g. I'll store some private keys in sub and I publish the superproject asking for collaborators to my new shiny webbased thing. The submodule containing the private keys never leaves my hard drive, hence no .gitmodules entry is necessary.

Not sure I agree with such a setup, but people use it.

So how about this fictional work flow:

         $ git init top
         $ cd top
         $ git commit --allow-empty -m 'initial in top'
         $ git init sub
         $ git -C sub commit --allow-empty -m 'initial in sub'
         $ git add sub
        You added a gitlink, but no corresponding entry in
        .gitmodules is found. This is fine for gits core functionality, but
        the submodule command gets confused by this unless you add 'sub'
        to your .gitmodules via `git submodule add --already-in-tree \
        --reuse-submodules-origin-as-URL sub`. Alternatively you can make this
        message disappear by configuring advice.gitlinkPitfalls.

         $ git submodule
         ... similar advice goes here...
         fatal: no submodule mapping found in .gitmodules for path 'sub'

         $ git submodule add --fixup-modules-file ./sub sub
         Adding .gitmodule entry only for `sub` to use `git -C remote
show origin` as URL.

         # user is happy now.

Junio C Hamano

2016-09-15 18:28 UTC #9 lore raw

Quoted 4 lines · Stefan Beller

When searching around the net, some people use half initialized submodules intentionally,...
Not sure I agree with such a setup, but people use it.

In such a top-level project, people would not use "git submodule" command, would they? I do not think anybody in this thread was pushing to forbid such a use, and it may be perfectly fine if "git submodule" does not work for such a gitlink; after all such a subdirectory is not even meant to be a submodule.

Quoted 14 lines

So how about this fictional work flow:

         $ git init top
         $ cd top
         $ git commit --allow-empty -m 'initial in top'
         $ git init sub
         $ git -C sub commit --allow-empty -m 'initial in sub'
         $ git add sub
        You added a gitlink, but no corresponding entry in
        .gitmodules is found. This is fine for gits core functionality, but
        the submodule command gets confused by this unless you add 'sub'
        to your .gitmodules via `git submodule add --already-in-tree \
        --reuse-submodules-origin-as-URL sub`. Alternatively you can make this
        message disappear by configuring advice.gitlinkPitfalls.

I am not sure if I agree with that direction.

If the trend in Git community collectively these days is to make usage of submodules easier and smoother, I'd imagine that you would want to teach "git add" that was given a submodule to "git submodule add" instead by default, with an option "git add --no-gitmodules sub" to disable it, or something like that.

Quoted 3 lines

         $ git submodule add --fixup-modules-file ./sub sub
         Adding .gitmodule entry only for `sub` to use `git -C remote
show origin` as URL.

I agree that a feature like this is needed regardless of what happens at "git add" time.

Heiko Voigt

2016-09-16 14:12 UTC #10 lore raw

Quoted 24 lines · Junio C Hamano

Quoted 14 lines · Stefan Beller
So how about this fictional work flow:
         $ git init top
         $ cd top
         $ git commit --allow-empty -m 'initial in top'
         $ git init sub
         $ git -C sub commit --allow-empty -m 'initial in sub'
         $ git add sub
        You added a gitlink, but no corresponding entry in
        .gitmodules is found. This is fine for gits core functionality, but
        the submodule command gets confused by this unless you add 'sub'
        to your .gitmodules via `git submodule add --already-in-tree \
        --reuse-submodules-origin-as-URL sub`. Alternatively you can make this
        message disappear by configuring advice.gitlinkPitfalls.
I am not sure if I agree with that direction.
If the trend in Git community collectively these days is to make usage of submodules easier and smoother, I'd imagine that you would want to teach "git add" that was given a submodule to "git submodule add" instead by default, with an option "git add --no-gitmodules sub" to disable it, or something like that.

Quoted 6 lines

Quoted 3 lines
         $ git submodule add --fixup-modules-file ./sub sub
         Adding .gitmodule entry only for `sub` to use `git -C remote
show origin` as URL.
I agree that a feature like this is needed regardless of what happens at "git add" time.

How about just

git submodule add <submodulepath>

? I remember back in the days when I started with submodules thats the way I imagined submodules would work:

1. clone the submodule into a directory
2. git submodule add it
3. git commit everything

Because that how you basically work with files. So instead of adding another option I would rather like to autodetect that:

* its a relative path inside this repo that is passed to 'git submodule add'
* there is no .gitmodules entry
* and no .git/config
==> create those from a remote in the submodule

Corner cases:

* If there is more than one remote we could tell the user to use an option to specify which one to use.
* Barf in case there is no remote (not adding the submodule except -f is used).
* If the gitlink is already there but no .gitmodules entry, 'git submodule add' will just add the entry as if it was initially added.

Instead of giving an error message that the submodule is already added we could actually be nicer to the user and try to fix things for him instead.

Cheers Heiko

Jacob Keller

2016-09-16 15:40 UTC #11 lore raw

Quoted 29 lines · Heiko Voigt

How about just
git submodule add <submodulepath>
? I remember back in the days when I started with submodules thats the way I imagined submodules would work:
1. clone the submodule into a directory
2. git submodule add it
3. git commit everything
Because that how you basically work with files. So instead of adding another option I would rather like to autodetect that:
* its a relative path inside this repo that is passed to 'git submodule add'
* there is no .gitmodules entry
* and no .git/config
==> create those from a remote in the submodule
Corner cases:
* If there is more than one remote we could tell the user to use an option to specify which one to use.
* Barf in case there is no remote (not adding the submodule except -f is used).
* If the gitlink is already there but no .gitmodules entry, 'git submodule add' will just add the entry as if it was initially added.

Quoted 4 lines

Instead of giving an error message that the submodule is already added we could actually be nicer to the user and try to fix things for him instead.

This makes sense to me. Possibly we could warn in this case, so that the user knows that something was "off" but I don't think we should be failing here...

Regards,
Jake

Quoted 1 line

Cheers Heiko

Junio C Hamano

2016-09-16 18:29 UTC #12 lore raw

In reply to Heiko Voigt

Quoted 12 lines · Heiko Voigt

Quoted 10 lines · Junio C Hamano
If the trend in Git community collectively these days is to make usage of submodules easier and smoother, I'd imagine that you would want to teach "git add" that was given a submodule to "git submodule add" instead by default, with an option "git add --no-gitmodules sub" to disable it, or something like that.
Quoted 3 lines
         $ git submodule add --fixup-modules-file ./sub sub
         Adding .gitmodule entry only for `sub` to use `git -C remote
show origin` as URL.

Quoted 6 lines

Quoted 2 lines
I agree that a feature like this is needed regardless of what happens at "git add" time.
How about just
git submodule add <submodulepath>

When I said "a feature like this is needed", I didn't care about exact syntax. I am not sure how often people need the "fixup", what kind of causes there are that they need the "fixup", and what the distribution of vaious causes would be like. If the _ONLY_ kind of fixup necessary is "I meant to say 'git submodule add ./path path' but I said 'git add path' instead", then I think it makes sense to teach "submodule add" that the form "git submodule add <path>" is a short-pand for "git submodule add ./<path> <path>". I am not sure if we want to _ignore_ a gitlink that is already in the index unconditionally, i.e. if it is a good idea to let the second one override the first one

	git submodule add $URL sub &&
        git submodule add sub

in this sequence, though.

Quoted 7 lines

? I remember back in the days when I started with submodules thats the way I imagined submodules would work:
1. clone the submodule into a directory
2. git submodule add it
3. git commit everything

Quoted 8 lines

Because that how you basically work with files. So instead of adding another option I would rather like to autodetect that:
* its a relative path inside this repo that is passed to 'git submodule add'
* there is no .gitmodules entry
* and no .git/config
==> create those from a remote in the submodule

In other words, I agree with the general direction but I'd add another condition to the above three, i.e.

* and there is no gitlink for that path in the index yet.