Creating own hierarchies under $GITDIR/refs ?
Hi,
in the context of an ongoing discussion on the Emacs developer list of converting the Bzr repository of Emacs, one question (with different approaches) is where to put the information regarding preexisting Bazaar revision numbers and bug tracker ids: those are not present in the current Git mirror.
Putting them in the commit messages would require a full history rewrite, and if some are missed in the process, this cannot be fixed afterwards.
So I mused: refs/heads contains branches, refs/tags contains tags. The respective information would likely easily enough be stored in refs/bzr and refs/bugs and in that manner would not pollute the "ordinary" tag and branch spaces, rendering "git tag" and/or "git branch" output mostly unusable. I tested creating such a directory and entries and indeed references like bzr/39005 then worked.
However, cloning from the repository did not copy those directories and references, so without modification, this scheme would not work for cloned repositories.
Are there some measures one can take/configure in the parent repository such that (named or all) additional directories inside of $GITDIR/refs would get cloned along with the rest?
It would definitely open viable options for dealing with mirrors and/or repository migrations in general.
Signature · 1 line
David Kastrup
Quoted 6 lines · David Kastrup
So I mused: refs/heads contains branches, refs/tags contains tags. The respective information would likely easily enough be stored in refs/bzr and refs/bugs and in that manner would not pollute the "ordinary" tag and branch spaces, rendering "git tag" and/or "git branch" output mostly unusable. I tested creating such a directory and entries and indeed references like bzr/39005 then worked.
Yes. The names "refs/tags" and "refs/heads" are special by convention, and there is no reason you cannot have other hierarchies (and indeed, we already have "refs/notes" and "refs/remotes" as common hierarchies).
Quoted 3 lines
However, cloning from the repository did not copy those directories and references, so without modification, this scheme would not work for cloned repositories.
Correct. Anyone who wants them will have to ask for them manually, like:
git config --add remote.origin.fetch '+refs/bzr/*:refs/bzr/*'
after which any "git fetch" will retrieve them.
Quoted 3 lines
Are there some measures one can take/configure in the parent repository such that (named or all) additional directories inside of $GITDIR/refs would get cloned along with the rest?
No. It is up to the client to decide which parts of the ref namespace they want to fetch. The server only advertises what it has, and the client selects from that.
Others mentioned that refs were never really intended to scale to one-per-commit. We serve some repositories with tens of thousands of refs from GitHub, and it does work. On the backend, we even have some repos in the hundreds of thousands (but these are not client facing). Most of the pain points (like O(n^2) loops) have been ironed out, but the two big ones are still:
- server ref advertisement lists _all_ refs at the start of the conversation. So, e.g.,
git fetch git://github.com/Homebrew/homebrew.git
sends 2MB of advertisement just so a client can find out "nope, nothing to fetch".
- the packed-refs storage is rather monolithic. Reading a value from it currently requires parsing the whole file. Likewise, deleting a ref requires rewriting the whole file.
So what you are proposing will work, but do note that there is a cost.
-Peff
Quoted 9 lines · David Kastrup
in the context of an ongoing discussion on the Emacs developer list of converting the Bzr repository of Emacs, one question (with different approaches) is where to put the information regarding preexisting Bazaar revision numbers and bug tracker ids: those are not present in the current Git mirror.
Putting them in the commit messages would require a full history rewrite, and if some are missed in the process, this cannot be fixed afterwards.
What do you need them for? Perhaps putting everything in a file, maybe sorted by SHA-1, would suffice? It should not be too hard to write a script to map bug tracker id to a commit id. The file is for past commits only. New commits can contain these info in their messages.
Signature · 1 line
Duy
Quoted 7 lines · Duy Nguyen
Quoted 6 lines · David Kastrup
in the context of an ongoing discussion on the Emacs developer list of converting the Bzr repository of Emacs, one question (with different approaches) is where to put the information regarding preexisting Bazaar revision numbers and bug tracker ids: those are not present in the current Git mirror.
Quoted 5 lines
Quoted 3 lines
Putting them in the commit messages would require a full history rewrite, and if some are missed in the process, this cannot be fixed afterwards.
What do you need them for?
Resolving references typically found in commit messages. Also establishing correlation to bug issue numbers.
Quoted 3 lines
Perhaps putting everything in a file, maybe sorted by SHA-1, would suffice? It should not be too hard to write a script to map bug tracker id to a commit id.
We are not talking about "it should not be too hard". We are talking about "obvious and reliable enough to render a complete history rewrite pointless".
Quoted 3 lines
The file is for past commits only.
New commits can contain these info in their messages.
If it's not forgotten. Experience shows that things like issue numbers have a tendency to be omitted, and then they stay missing.
At any rate, this is exactly the kind of stuff that tags are useful for, except that using them for all that would render the "tag space" overcrowded.
Rest assured that the "standard" answers have been beat to death in the Emacs developer list thread several times over.
So I'm more interested in getting actual answers dealing with the question I have asked rather than suggestions for questions that would be easier to answer.
Since Git has a working facility for references that is catered to do exactly this kind of mapping and already _does_, it seems like a convenient path to explore.
It apparently even already works with --decorate:
commit c92b1fb3ad8514f08fc4cec531211717955a5c29 (tag: release/2.19.1-1, origin/release/unstable, tag: refs/bzr/r15000)
Author: Phil Holmes <[email protected]> Date: Sun Jan 19 15:01:48 2014 +0000
Release: update news.
Signature · 1 line
David Kastrup
Quoted 4 lines · David Kastrup
Quoted 1 line · Duy Nguyen
The file is for past commits only.
Quoted 8 lines
Quoted 1 line
New commits can contain these info in their messages.
If it's not forgotten. Experience shows that things like issue numbers have a tendency to be omitted, and then they stay missing.
At any rate, this is exactly the kind of stuff that tags are useful for, except that using them for all that would render the "tag space" overcrowded.
Actually, I would say this is exactly the sort of thing notes are for.
git.git uses them to map commits back to mailing list discussions:
git fetch git://github.com/gitster/git +refs/notes/amlog:refs/notes/amlog &&
git log --notes=amlogSee also notes.displayRef in git-config(1).
Notes aren't fetch by default, but it's not hard for those interested to add a remote.*.fetch line to their config.
Quoted 10 lines · John Keeping
Quoted 9 lines · David Kastrup
Quoted 3 lines · Duy Nguyen
The file is for past commits only.
New commits can contain these info in their messages.
If it's not forgotten. Experience shows that things like issue numbers have a tendency to be omitted, and then they stay missing.
Quoted 7 lines
Quoted 3 lines
At any rate, this is exactly the kind of stuff that tags are useful for, except that using them for all that would render the "tag space" overcrowded.
Actually, I would say this is exactly the sort of thing notes are for.
git.git uses them to map commits back to mailing list discussions:
But that's the wrong direction. What is needed in the Emacs case is mapping the Bazaar reference numbers (and bug numbers) to commits.
While it is true that the history rewriting approach would not deliver this either (short of git log --grep with suitable patterns), I was looking for something less of a crutch here.
Quoted 2 lines
Notes aren't fetch by default, but it's not hard for those interested to add a remote.*.fetch line to their config.
If we are talking about measures everybody has to actively take before getting access to functionality, this does not cross the convenience threshold making it a solution preferred over others. But it's probably feasible to configure a fetch line doing this that will get cloned when first cloning a repository. That's not too hot for people with existing repositories, but since we are talking about a migration from Bazaar anyway, Git users currently are so by choice and so might be more willing to update their configuration if it helps with avoiding a fully new clone.
Signature · 1 line
David Kastrup
Quoted 16 lines · David Kastrup
Quoted 14 lines · John Keeping
Quoted 12 lines · David Kastrup
Quoted 3 lines · Duy Nguyen
The file is for past commits only.
New commits can contain these info in their messages.
If it's not forgotten. Experience shows that things like issue numbers have a tendency to be omitted, and then they stay missing.
At any rate, this is exactly the kind of stuff that tags are useful for, except that using them for all that would render the "tag space" overcrowded.
Quoted 6 lines
Quoted 3 lines
Actually, I would say this is exactly the sort of thing notes are for.
git.git uses them to map commits back to mailing list discussions:
But that's the wrong direction. What is needed in the Emacs case is mapping the Bazaar reference numbers (and bug numbers) to commits.
Ah, OK. I hadn't quite read carefully enough.
I actually wonder if you could do this with notes and git-grep; for example:
git grep -l keeping.me.uk refs/notes/amlog |
sed -e 's/.*://' -e 's!/!!g'
That should be relatively efficient since you're only looking at the current notes tree.
Quoted 4 lines
While it is true that the history rewriting approach would not deliver this either (short of git log --grep with suitable patterns), I was looking for something less of a crutch here.
Quoted 8 lines
Quoted 2 lines
Notes aren't fetch by default, but it's not hard for those interested to add a remote.*.fetch line to their config.
If we are talking about measures everybody has to actively take before getting access to functionality, this does not cross the convenience threshold making it a solution preferred over others. But it's probably feasible to configure a fetch line doing this that will get cloned when first cloning a repository.
I'm assuming you'll need some form of tool (at least a script) to manipulate this feature; it wouldn't be too hard for that to set this up the first time it's run.
Quoted 3 lines · David Kastrup
Since Git has a working facility for references that is catered to do exactly this kind of mapping and already _does_, it seems like a convenient path to explore.
It will not scale. If you make those refs available for cloning/fetching, all of them will be advertised first thing when git starts negotiate. Imagine thousands of refs (and keep increasing) sent to the receiver at the beginning of every connection. Something like "reverse git-notes" may transfer more efficiently. Or we need to improve git protocol to handle massive refs better, something that's been discussed for a while without any outcome.
Signature · 1 line
Duy
Quoted 9 lines · Duy Nguyen
Quoted 3 lines · David Kastrup
Since Git has a working facility for references that is catered to do exactly this kind of mapping and already _does_, it seems like a convenient path to explore.
It will not scale. If you make those refs available for cloning/fetching, all of them will be advertised first thing when git starts negotiate. Imagine thousands of refs (and keep increasing) sent to the receiver at the beginning of every connection.
In current LilyPond repository:
git tag|wc
969 969 15161
In current Emacs mirror:
git tag|wc
1202 1202 15729
In current Git repository:
git tag|wc
498 498 4820
Quoted 3 lines
Something like "reverse git-notes" may transfer more efficiently. Or we need to improve git protocol to handle massive refs better, something that's been discussed for a while without any outcome.
I think that even disregarding special use of references, _existing_ practice would already appear to warrant being able to deal with thousands of refs in a reasonable manner.
It's a reasonable expectation to have a tag per (potentially intermediate) release or release candidate. For any project publishing reproducible daily snapshots, the threshold of 1000 will get reached within few years.
Of course, it is relevant information to know that right _now_ references will not scale. But that does not seem like a defensible long-term perspective.
Signature · 1 line
David Kastrup
Quoted 8 lines · John Keeping
I actually wonder if you could do this with notes and git-grep; for example:
git grep -l keeping.me.uk refs/notes/amlog |
sed -e 's/.*://' -e 's!/!!g'That should be relatively efficient since you're only looking at the current notes tree.
I added notes handling to gitifyhg and would search it similar to this. Since gitifyhg is two-way, I could not modify the commits. Later, when we converted several repositories (up to 50k commits/80 MB), I appended
Hg-commit: $Hg_commit_hash
to all the commit messages. This way it shows up on the web interface, users don't have to obtain the notes specially, and "git log --grep" works naturally. I think it's worth considering this simple solution; existing Git users won't mind recloning once.
Quoted 3 lines · David Kastrup
Are there some measures one can take/configure in the parent repository such that (named or all) additional directories inside of $GITDIR/refs would get cloned along with the rest?
$ git config --add remote.orgin.fetch '+refs/notes/*:refs/notes/*'
Andreas.
Signature · 3 lines
Andreas Schwab, [email protected] GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."