Mirror repositories for submodules
Hi,
Would a contribution to add mirror repositories as alternate submodule sources be considered for inclusion? Some projects have mirror repositories on other hosting services, and may have bandwidth limits on their primary hosting service. Being able to indicate mirror repositories for where to check for updates and sources for submodules when doing git clone --recurse-submodules https://my.repo or git submodule update --init --recursive would be helpful when there is a timeout.
Regards,
Benson
Quoted 8 lines · Benson Muite
Would a contribution to add mirror repositories as alternate submodule sources be considered for inclusion? Some projects have mirror repositories on other hosting services, and may have bandwidth limits on their primary hosting service. Being able to indicate mirror repositories for where to check for updates and sources for submodules when doing
git clone --recurse-submodules https://my.repoorgit submodule update --init --recursivewould be helpful when there is a timeout.
I do not see why such a "oh, the repository at $URL1 seems to be down, but we know $URL2 serves the equivalent information, so let's go there instead" feature has to be limited to submodule use case.
So, no, I do not think a contribution to add mirror repositories as alternate submodule sources should be considered for inclusion, as it artificially limits usefulness of the feature. A feature to add mirror repositories as alternate sources might be worth considering, though.
Hi,
Quoted 5 lines · Junio C Hamano
So, no, I do not think a contribution to add mirror repositories as alternate submodule sources should be considered for inclusion, as it artificially limits usefulness of the feature. A feature to add mirror repositories as alternate sources might be worth considering, though.
This is relevant to the Debian use case: we run a git server that archives git trees for Debian packages, and ideally the objects on this server should be identical to what you get from upstream projects.
This is a big problem for archiving projects that use submodules, because we cannot alter the reference URLs.
Cloning from our server will, depending on what upstream uses, either a relative URL (which will go to our server, but we have little control over what the name part of the repository base URL is going to be), or an absolute URL that instructs clients to pull from another place, which conflicts with our goal to have a self-contained archive.
The idea posited earlier, to have a "repository identity" that remains the same across forks and clones, is somewhat appealing, but the best idea I can come up with is generating some kind of repository UUID, and adding a symlink -- not a great design because it pollutes outside the repo:
$ mkdir myproject
$ cd myproject
$ git init
$ ls -l ..
lrwxrwxrwx 1 simon simon 9 Jun 4 14:05
12345678-9abc-def0-1234-56789abcdef0.git -> myproject
drwxrwxr-x 2 simon simon 40 Jun 4 14:04 myprojectOn the other hand, this can be used to construct a stable relative submodule URL.
Making the symlinks optional would require keeping a list of local clones and their UUIDs, and resolving them.
I don't like that design, but as I said it's the best idea I have for now.
I also fully expect that Debian's servers will be used by a lot of people outside the project as soon as it becomes a convenient fallback, in the same way people are pulling .orig.tar.gz archives from Debian mirrors, so we need to make it easy to set up a mirror, to allow this to scale.
Simon
Quoted 19 lines · Simon Richter
Cloning from our server will, depending on what upstream uses, either a relative URL (which will go to our server, but we have little control over what the name part of the repository base URL is going to be), or an absolute URL that instructs clients to pull from another place, which conflicts with our goal to have a self-contained archive.
The idea posited earlier, to have a "repository identity" that remains the same across forks and clones, is somewhat appealing, but the best idea I can come up with is generating some kind of repository UUID, and adding a symlink -- not a great design because it pollutes outside the repo:
$ mkdir myproject $ cd myproject $ git init $ ls -l .. lrwxrwxrwx 1 simon simon 9 Jun 4 14:05 12345678-9abc-def0-1234-56789abcdef0.git -> myproject drwxrwxr-x 2 simon simon 40 Jun 4 14:04 myproject
Quoted 2 lines
On the other hand, this can be used to construct a stable relative submodule URL.
Here's a thought experiment. What if you put the UUID into a URL, like:
repoid://123456789.git
Then your in-repo .gitconfig would point to that repo id and be consistent. Of course you need some way to tell Git how to retrieve repoid:// URLs. You could do so with a custom remote helper (git-remote-repoid), but presumably that helper is eventually going to end up going over one of the normal Git protocols.
So we just need to tell Git how to resolve repo id URLs into concrete URLs. And indeed, we have url.*.insteadOf to do rewriting already. So for example, you can add a submodule but convert it into a uuid like this:
$ git submodule add https://github.com/git/git.git $ git config -f .gitmodules submodule.git.url https://github.com/git/git.git $ git config -f .gitmodules submodule.git.url repoid://123456789.git $ git commit -am 'add submodule with magic repoid'
Now if somebody else comes along and clones it naively, the repo uuid is not useful to git by itself:
$ git clone --recurse-submodules repo Submodule 'git' (repoid://123456789.git) registered for path 'git' Cloning into '/home/peff/tmp/repo/git'... fatal: transport 'repoid' not allowed fatal: clone of 'repoid://123456789.git' into submodule path '/home/peff/tmp/repo/git' failed
But imagine that "somehow" they have learned that 123456789.git can be found at some URL. You can do this:
git -c url.https://github.com/git/git.git.insteadOf=repoid://123456789.git \ clone --recurse-submodules repo.git
which would clone from the original URL. Or you could even imagine that they have a cache of repositories named by uuid, and then:
git -c url.https://my/cache/.insteadOf=repoid:// ...
would rewrite all repoid://'s automatically.
The use of "-c" here is mostly for illustration. It is a per-command config, so when you later try to update the submodule, you'd run into the same problem. Probably you'd want to stuff your mapping into on-disk config (either ~/.gitconfig, or if you have a lot of them, perhaps some file included from there).
It would be nice if you could use "git clone -c" (note "-c" as an option to "clone", not to "git") to set a permanent per-repo config variable. But sadly the URL rewriting happens in the submodule repository, not the parent. So it has to be a per-user setting.
Now, all of that said, do we still need uuids at all? If the canonical submodule name is https://github.com/git/git.git, then anybody can just rewrite that locally in the same way using url.*.insteadOf config. And I think this is a pretty standard way of using submodules. E.g., you might rewrite https:// into ssh:// if you prefer that protocol. Or point to a local server if it's faster for you.
Which makes me wonder if I am missing something about the original request that started this thread. But it sounds to me like it is just asking for the existing URL-rewriting feature.
-Peff
Hi,
Quoted 2 lines · Jeff King
Here's a thought experiment. What if you put the UUID into a URL, like:
repoid://123456789.git
Yes, that's the idea, except I would want to use a relative URL, like
../123456789.git
This could solve the "naive cloning" problem, because it creates an expectation that the submodules can be found on the same server, or in a nearby path.
I'm aware that this is *also* bad for decentralization, because it makes it easier to use one of the big forges where the repositories for often-used submodules are are already likely to be present, but it plays into our use case, where we want to share the repositories for often-used subprojects.
Quoted 3 lines
Now, all of that said, do we still need uuids at all? If the canonical submodule name is https://github.com/git/git.git, then anybody can just rewrite that locally in the same way using url.*.insteadOf config.
Yes, but we'd then need a mechanism for a server to indicate "for cloning, you should use these 'insteadOf' settings, which is a massive can of worms from a security standpoint.
I also don't think these canonical URLs can ever be stable if they refer to infrastructure that is not under the control of the maintainer -- it would tie the project identity to the hosting provider, and increase the inertia to overcome for moves (such as the current exodus from github and gitlab towards codeberg).
Quoted 3 lines
Which makes me wonder if I am missing something about the original request that started this thread. But it sounds to me like it is just asking for the existing URL-rewriting feature.
The original mail has a similar problem as we do in Debian, and as my employer has: CI jobs should exclusively talk to in-house infrastructure, because continuously cloning repositories for each build is bad for the environment.
The common goal is that a naive clone should get submodules from a local server, ideally without us having to write some tool to make an initial checkout, enumerate submodules, create insteadOf settings, clone first layer of submodules, enumerate second layer, ...
Simon
Quoted 7 lines · Simon Richter
Hi,
Quoted 2 lines · Jeff King
Here's a thought experiment. What if you put the UUID into a URL, like:
repoid://123456789.git
Quoted 7 lines
Yes, that's the idea, except I would want to use a relative URL, like
../123456789.gitThis could solve the "naive cloning" problem, because it creates an expectation that the submodules can be found on the same server, or in a nearby path.
I see. I forgot that we allowed relative submodule URLs.
Quoted 8 lines
Quoted 3 lines
Now, all of that said, do we still need uuids at all? If the canonical submodule name is https://github.com/git/git.git, then anybody can just rewrite that locally in the same way using url.*.insteadOf config.
Yes, but we'd then need a mechanism for a server to indicate "for cloning, you should use these 'insteadOf' settings, which is a massive can of worms from a security standpoint.
Quoted 5 lines
I also don't think these canonical URLs can ever be stable if they refer to infrastructure that is not under the control of the maintainer -- it would tie the project identity to the hosting provider, and increase the inertia to overcome for moves (such as the current exodus from github and gitlab towards codeberg).
From your description I was assuming the cloner had to always specify insteadOf (which they find out about "somehow").
If they're not, then your choice of canonical URL is effectively trading off some cases for others. In the scenario you care about, you assume that the submodules are hosted relative to the superproject, so clients can usually get what they need without further config. The server operator and the superproject repo coordinate on the names.
But in many decentralized cases, there's no URL or administrative relationship between the superproject and the submodules. They might happen to be on the same server, but even that falls down if the superproject is mirrored elsewhere. So using some canonical name which works in practice _now_ is usually the best we can do.
Quoted 4 lines
The common goal is that a naive clone should get submodules from a local server, ideally without us having to write some tool to make an initial checkout, enumerate submodules, create insteadOf settings, clone first layer of submodules, enumerate second layer, ...
You shouldn't need to do the recursive enumeration if you set up the inteadOf ahead of time. You don't know which insteadOf settings you'll want, but you can feed the whole possible mapping. How you get that mapping is unspecified, but if you are mirroring the submodules already on your local infrastructure, then whatever process does that can also output the mapping.
Just to be clear, I'm not trying to dismiss what you're going for. I'm looking at this from the lens of Git developers: how do existing Git features fit into this space, and which features are missing that might assist in a generalized way.
-Peff
Quoted 86 lines · Jeff King
Quoted 19 lines · Simon Richter
Cloning from our server will, depending on what upstream uses, either a relative URL (which will go to our server, but we have little control over what the name part of the repository base URL is going to be), or an absolute URL that instructs clients to pull from another place, which conflicts with our goal to have a self-contained archive.
The idea posited earlier, to have a "repository identity" that remains the same across forks and clones, is somewhat appealing, but the best idea I can come up with is generating some kind of repository UUID, and adding a symlink -- not a great design because it pollutes outside the repo:
$ mkdir myproject $ cd myproject $ git init $ ls -l .. lrwxrwxrwx 1 simon simon 9 Jun 4 14:05 12345678-9abc-def0-1234-56789abcdef0.git -> myproject drwxrwxr-x 2 simon simon 40 Jun 4 14:04 myprojectQuoted 2 lines
On the other hand, this can be used to construct a stable relative submodule URL.
Here's a thought experiment. What if you put the UUID into a URL, like:
repoid://123456789.git
Then your in-repo .gitconfig would point to that repo id and be consistent. Of course you need some way to tell Git how to retrieve repoid:// URLs. You could do so with a custom remote helper (git-remote-repoid), but presumably that helper is eventually going to end up going over one of the normal Git protocols.
So we just need to tell Git how to resolve repo id URLs into concrete URLs. And indeed, we have url.*.insteadOf to do rewriting already. So for example, you can add a submodule but convert it into a uuid like this:
$ git submodule add https://github.com/git/git.git $ git config -f .gitmodules submodule.git.url https://github.com/git/git.git $ git config -f .gitmodules submodule.git.url repoid://123456789.git $ git commit -am 'add submodule with magic repoid'Now if somebody else comes along and clones it naively, the repo uuid is not useful to git by itself:
$ git clone --recurse-submodules repo Submodule 'git' (repoid://123456789.git) registered for path 'git' Cloning into '/home/peff/tmp/repo/git'... fatal: transport 'repoid' not allowed fatal: clone of 'repoid://123456789.git' into submodule path '/home/peff/tmp/repo/git' failedBut imagine that "somehow" they have learned that 123456789.git can be found at some URL. You can do this:
git -c url.https://github.com/git/git.git.insteadOf=repoid://123456789.git \ clone --recurse-submodules repo.gitwhich would clone from the original URL. Or you could even imagine that they have a cache of repositories named by uuid, and then:
git -c url.https://my/cache/.insteadOf=repoid:// ...
would rewrite all repoid://'s automatically.
The use of "-c" here is mostly for illustration. It is a per-command config, so when you later try to update the submodule, you'd run into the same problem. Probably you'd want to stuff your mapping into on-disk config (either ~/.gitconfig, or if you have a lot of them, perhaps some file included from there).
It would be nice if you could use "git clone -c" (note "-c" as an option to "clone", not to "git") to set a permanent per-repo config variable. But sadly the URL rewriting happens in the submodule repository, not the parent. So it has to be a per-user setting.
Now, all of that said, do we still need uuids at all? If the canonical submodule name is https://github.com/git/git.git, then anybody can just rewrite that locally in the same way using url.*.insteadOf config. And I think this is a pretty standard way of using submodules. E.g., you might rewrite https:// into ssh:// if you prefer that protocol. Or point to a local server if it's faster for you.
Quoted 4 lines
Which makes me wonder if I am missing something about the original request that started this thread. But it sounds to me like it is just asking for the existing URL-rewriting feature.
The problem is that one might have multiple repositories, submodules may themselves have submodules. Typically a primary development organization will have its own host, but may also have mirrors on other services which maybe more convenient for others to use. A recursive clone could give upto 20 repositories not all of which are maintained by the same organization. URL-rewriting each of them can be inefficient, especially when the upstream maintains the mirror repositories and can indicate that in the source repositories.
Quoted 1 line
-Peff
Quoted 42 lines · Simon Richter
Hi,
Quoted 5 lines · Junio C Hamano
So, no, I do not think a contribution to add mirror repositories as alternate submodule sources should be considered for inclusion, as it artificially limits usefulness of the feature. A feature to add mirror repositories as alternate sources might be worth considering, though.
This is relevant to the Debian use case: we run a git server that archives git trees for Debian packages, and ideally the objects on this server should be identical to what you get from upstream projects.
This is a big problem for archiving projects that use submodules, because we cannot alter the reference URLs.
Cloning from our server will, depending on what upstream uses, either a relative URL (which will go to our server, but we have little control over what the name part of the repository base URL is going to be), or an absolute URL that instructs clients to pull from another place, which conflicts with our goal to have a self-contained archive.
The idea posited earlier, to have a "repository identity" that remains the same across forks and clones, is somewhat appealing, but the best idea I can come up with is generating some kind of repository UUID, and adding a symlink -- not a great design because it pollutes outside the repo:
$ mkdir myproject $ cd myproject $ git init $ ls -l .. lrwxrwxrwx 1 simon simon 9 Jun 4 14:05 12345678-9abc-def0-1234-56789abcdef0.git -> myproject drwxrwxr-x 2 simon simon 40 Jun 4 14:04 myprojectOn the other hand, this can be used to construct a stable relative submodule URL.
Making the symlinks optional would require keeping a list of local clones and their UUIDs, and resolving them.
Quoted 8 lines
I don't like that design, but as I said it's the best idea I have for now.
I also fully expect that Debian's servers will be used by a lot of people outside the project as soon as it becomes a convenient fallback, in the same way people are pulling .orig.tar.gz archives from Debian mirrors, so we need to make it easy to set up a mirror, to allow this to scale.
For submodules, the metadata consists of the url of the repository to clone from. One could have a list of absolute URLs. The default would be to assume that the URLs are tried in order, and if a URL times out, the next one would be tried. One may want to change the default ordering as a user setting, or do a ping test to get obtain content from the closest repository.
As an example, for linphone-desktop, the first part of the .gitmodules file contains:
[submodule "linphone-sdk"]
path = external/linphone-sdk
url = https://gitlab.linphone.org/BC/public/linphone-sdk.git
[submodule "external/google/gn"]
This could be updated to
[submodule "linphone-sdk"]
path = external/linphone-sdk
url = https://gitlab.linphone.org/BC/public/linphone-sdk.git
url = https://github.com/BelledonneCommunications/linphone-sdk.git
[submodule "external/google/gn"]
Quoted 7 lines · Benson Muite
For submodules, the metadata consists of the url of the repository to clone from. One could have a list of absolute URLs. The default would be to assume that the URLs are tried in order, and if a URL times out, the next one would be tried. One may want to change the default ordering as a user setting, or do a ping test to get obtain content from the closest repository.
Another idea is for the client to attempt in a random order as a kind of load balancing.
Quoted 9 lines
As an example, for linphone-desktop, the first part of the .gitmodules file contains:
[submodule "linphone-sdk"]
path = external/linphone-sdk
url = https://gitlab.linphone.org/BC/public/linphone-sdk.git
[submodule "external/google/gn"]
Quoted 7 lines
This could be updated to
[submodule "linphone-sdk"]
path = external/linphone-sdk
url = https://gitlab.linphone.org/BC/public/linphone-sdk.git
url = https://github.com/BelledonneCommunications/linphone-sdk.git
[submodule "external/google/gn"]
Relevant to Junio's earlier comment about aiming for a general solution:
It looks like the remote URL configuration already supports multiple URLs per a single entry. It's just that git only considers the first in the list for fetching, and will push to all.
Your proposed change to .gitmodules may be used to initialize the list of URLs for a cloned submodule's (single) initial remote? At this point, any kind of mirror management logic could generically operate on each remote's URL list.
From git-config(1):
remote.<name>.url
The URL of a remote repository. See git-fetch(1) or git-push(1). A configured remote can have multiple URLs; in this case the first is used for fetching, and all are used for pushing (assuming no remote.<name>.pushurl is defined). Setting this key to the empty string clears the list of urls, allowing you to override earlier config.
From gitmodules(5):
submodule.<name>.url
Defines a URL from which the submodule repository can be cloned. This may be either an absolute URL ready to be passed to git-clone(1) or (if it begins with ./ or ../) a location relative to the superproject’s origin repository.
Note that the submodule format seems to explicitly support only a single value right now. Of course, I am assuming that the implementation matches the documentation.
Quoted 36 lines · Simon Richter
Hi,
Quoted 5 lines · Junio C Hamano
So, no, I do not think a contribution to add mirror repositories as alternate submodule sources should be considered for inclusion, as it artificially limits usefulness of the feature. A feature to add mirror repositories as alternate sources might be worth considering, though.
This is relevant to the Debian use case: we run a git server that archives git trees for Debian packages, and ideally the objects on this server should be identical to what you get from upstream projects.
This is a big problem for archiving projects that use submodules, because we cannot alter the reference URLs.
Cloning from our server will, depending on what upstream uses, either a relative URL (which will go to our server, but we have little control over what the name part of the repository base URL is going to be), or an absolute URL that instructs clients to pull from another place, which conflicts with our goal to have a self-contained archive.
The idea posited earlier, to have a "repository identity" that remains the same across forks and clones, is somewhat appealing, but the best idea I can come up with is generating some kind of repository UUID, and adding a symlink -- not a great design because it pollutes outside the repo:
$ mkdir myproject $ cd myproject $ git init $ ls -l .. lrwxrwxrwx 1 simon simon 9 Jun 4 14:05 12345678-9abc-def0-1234-56789abcdef0.git -> myproject drwxrwxr-x 2 simon simon 40 Jun 4 14:04 myproject
Quoted 8 lines
On the other hand, this can be used to construct a stable relative submodule URL.
Making the symlinks optional would require keeping a list of local clones and their UUIDs, and resolving them.
I don't like that design, but as I said it's the best idea I have for now.
For submodules, the metadata consists of the url of the repository to clone from. One could have a list of absolute URLs. The default would be to assume that the URLs are tried in order, and if a URL times out, the next one would be tried. One may want to change the default ordering as a user setting, or do a ping test to get obtain content from the closest repository.
As an example, for linphone-desktop, the first part of the .gitmodules file contains:
[submodule "linphone-sdk"]
path = external/linphone-sdk
url = https://gitlab.linphone.org/BC/public/linphone-sdk.git
[submodule "external/google/gn"]
This could be updated to
[submodule "linphone-sdk"]
path = external/linphone-sdk
url = https://gitlab.linphone.org/BC/public/linphone-sdk.git
url = https://github.com/BelledonneCommunications/linphone-sdk.git
[submodule "external/google/gn"]
Quoted 7 lines
I also fully expect that Debian's servers will be used by a lot of people outside the project as soon as it becomes a convenient fallback, in the same way people are pulling .orig.tar.gz archives from Debian mirrors, so we need to make it easy to set up a mirror, to allow this to scale.
Simon
Hi,
Quoted 7 lines · Benson Muite
Quoted 2 lines · Simon Richter
On the other hand, this can be used to construct a stable relative submodule URL.
For submodules, the metadata consists of the url of the repository to clone from.
That is precisely what precludes mirroring: if I clone and republish a repository, people can clone from that repository, but will still fetch submodules from the URLs listed in the .gitmodules file.
If that is a relative URL, then all is (mostly) well: they will also ask my mirror server for the submodule, and all I have to do is make it available.
If it is an absolute URL, then I need a side channel to communicate to the client "you can also get this repository from me." This could, for example, generate an insteadOf config, but that would be a horrible hack that becomes unmanageable pretty quickly (updates? security implications?)
Hence this thread: is there a way to represent submodules so that their identity is independent from the hosting location -- and this ties into the other thread from last week, giving projects a stable identity that follows them through clones (or, if someone is using a forge, forks).
The download location for a project is project metadata that lives outside the project view of time, but it is expressed as (versioned) data in git, in the .gitmodules file, so if hosting for a project changes, projects referring to them must either rewrite all of their history, accept that old versions will no longer be buildable because they contain a broken link, or expect people/CI to manually generate insteadOf entries.
So the problem here is that we are treating metadata as data.
Simon
Quoted 15 lines · Junio C Hamano
Quoted 8 lines · Benson Muite
Would a contribution to add mirror repositories as alternate submodule sources be considered for inclusion? Some projects have mirror repositories on other hosting services, and may have bandwidth limits on their primary hosting service. Being able to indicate mirror repositories for where to check for updates and sources for submodules when doing
git clone --recurse-submodules https://my.repoorgit submodule update --init --recursivewould be helpful when there is a timeout.I do not see why such a "oh, the repository at $URL1 seems to be down, but we know $URL2 serves the equivalent information, so let's go there instead" feature has to be limited to submodule use case.
Quoted 5 lines
So, no, I do not think a contribution to add mirror repositories as alternate submodule sources should be considered for inclusion, as it artificially limits usefulness of the feature. A feature to add mirror repositories as alternate sources might be worth considering, though.
Thanks for the feedback. This was motivated by problems when trying to recursively clone, but a more general solution is also fine.
Quoted 15 lines · Junio C Hamano
Quoted 8 lines · Benson Muite
Would a contribution to add mirror repositories as alternate submodule sources be considered for inclusion? Some projects have mirror repositories on other hosting services, and may have bandwidth limits on their primary hosting service. Being able to indicate mirror repositories for where to check for updates and sources for submodules when doing
git clone --recurse-submodules https://my.repoorgit submodule update --init --recursivewould be helpful when there is a timeout.I do not see why such a "oh, the repository at $URL1 seems to be down, but we know $URL2 serves the equivalent information, so let's go there instead" feature has to be limited to submodule use case.
Quoted 5 lines
So, no, I do not think a contribution to add mirror repositories as alternate submodule sources should be considered for inclusion, as it artificially limits usefulness of the feature. A feature to add mirror repositories as alternate sources might be worth considering, though.
Thanks for the feedback. This was motivated by problems when trying to recursively clone, but a more general solution is also fine.