Skip to contentSkip to conversation

Git fast-import : Warning duplicated ref: refs/remotes/origin/master

PATCH In review Jérémie NIKAES 5 participants 15 messages Jun 15, 2016 Documentation

lore
Share & tools
lore

Jérémie NIKAES

author
#0 lore raw

Hi,
I am currently working on the git-mediawiki gateway project. This project uses fast-import to get data from mediawiki. For each revision from mediawiki, we generate a chunk of fast-export stream looking like this :

commit refs/remotes/origin/master
mark :<int>
commiter <user> <address> <timestamp> +0000
data <sizeofcomment>
<comment>
M 644 inline <title>.wiki
data <sizeoffile>
<content>

At the end of a git clone mediawiki::http://ourwikitest, everything looks perfectly fine, including the git log, but a "Warning duplicated
ref: refs/remotes/origin/master" is thrown. And indeed, it is duplicated in .git/packedrefs

A git gc solves everything, but I wondered if there was a way to fix this or if this was a known bug.

Thank you,
Regards,

--
Jérémie Nikaes

#1 lore raw

Hi Jérémie,

Quoted 2 lines · Jérémie NIKAES

At the end of a git clone mediawiki::http://ourwikitest, everything looks perfectly fine

I just wanted to say, thank you for working on this[1].

This code answers some questions I had about the remote helper protocol, among other things.

Quoted 2 lines

a "Warning duplicated
ref: refs/remotes/origin/master" is thrown.

To end the suspense: I assume this was fixed by 9055039c (Bug fix for pulls with multiple revisions pulled, 2011-05-29).

Cheers,
Jonathan

[1] git://github.com/Bibzball/Git-Mediawiki.git

author
#2 lore raw

Jonathan,

Quoted 4 lines

I just wanted to say, thank you for working on this[1].

This code answers some questions I had about the remote helper protocol, among other things.

Glad we can help !

Quoted 6 lines
Quoted 3 lines

a "Warning duplicated
ref: refs/remotes/origin/master" is thrown.

To end the suspense: I assume this was fixed by 9055039c (Bug fix for pulls with multiple revisions pulled, 2011-05-29).

No, I actually still have the problem. The version commited on github is pulling revisions straight to refs/heads/master which is gross After discussing this matter with our teacher Matthieu Moy, I wanted to change this to refs/remotes/origin/master but then this warning gets thrown.

Does anyone have an idea ?

Thanks,
Cheers,

Signature · 1 line
Jérémie Nikaes
#3 lore raw
Quoted 4 lines · Jérémie NIKAES
Quoted 4 lines
Quoted 2 lines · Jérémie NIKAES

a "Warning duplicated
ref: refs/remotes/origin/master" is thrown.

[...]

Quoted 5 lines

No, I actually still have the problem. The version commited on github is pulling revisions straight to refs/heads/master which is gross After discussing this matter with our teacher Matthieu Moy, I wanted to change this to refs/remotes/origin/master but then this warning gets thrown.

Could you push a branch exhibiting the problem? The reason I ask is that searching with

git grep -F -e 'duplicated ref' origin/master

in git.git yields no hits for me, so I wanted to reproduce this to get the exact error message or figure out which program is printing it.

Thanks.

author
#4 lore raw
Quoted 11 lines

Could you push a branch exhibiting the problem?  The reason I ask is that searching with

       git grep -F -e 'duplicated ref' origin/master

in git.git yields no hits for me, so I wanted to reproduce this to get the exact error message or figure out which program is printing it.

Thanks.

Ok I just committed my changes to the master branch, feel free to pull and try it out.

Thanks for your help

Signature · 1 line
Jérémie Nikaes
#5 lore raw

PATCH 1/1 Documentation: do not misinterpret refspecs as bold text (Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master)

Quoted 2 lines · Jérémie NIKAES

Ok I just committed my changes to the master branch, feel free to pull and try it out.

Nice. Now I get

| $ git fetch --all
[...]

| Fetching & writing export data...
| 1/2: Revision n°7 of New_page
| 2/2: Revision n°8 of New_page
| 
| error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
| From mediawiki::http://localhost/mediawiki
|  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
| error: Could not fetch origin

which means that the transport machinery thought the helper was going to be fetching directly to "master". I suspect you will want a 'refspec' capability like

	refspec refs/heads/*:refs/mediawiki/${remotename}/*

to fix this.

Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map private ref names into normal names, 2009-11-18). What namespace should a helper use when asked to fetch to FETCH_HEAD without a remote name, like

git fetch mediawiki::testwiki

?

-- >8 --
Subject: Documentation: do not misinterpret refspecs as bold text

In v1.7.3.3~2 (Documentation: do not misinterpret pull refspec as bold text, 2010-12-03) many uses of asterisks in expressions like "refs/heads/*:refs/svn/origin/branches/*" were escaped as {asterisk} to avoid being treated as delimiters for bold text, but these two were missed.

Signed-off-by Jonathan Nieder <[email protected]>
Signed-off-by Junio C Hamano <[email protected]>
---
 Documentation/git-remote-helpers.txt |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
Documentation/git-remote-helpers.txt +3 -3
index 87cd11f..58f6ad4 100644
@@ -181,11 +181,11 @@ CAPABILITIES
 	When using the import command, expect the source ref to have
 	been written to the destination ref. The earliest applicable
 	refspec takes precedence. For example
-	"refs/heads/*:refs/svn/origin/branches/*" means that, after an
-	"import refs/heads/name", the script has written to
+	"refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}" means
+	that, after an "import refs/heads/name", the script has written to
 	refs/svn/origin/branches/name. If this capability is used at
 	all, it must cover all refs reported by the list command; if
-	it is not used, it is effectively "*:*"
+	it is not used, it is effectively "{asterisk}:{asterisk}"
 
 REF LIST ATTRIBUTES
 -------------------
Signature · 1 line
1.7.5.3
#6 lore raw

remote helpers: best practices for using the "refspec" capability

Quoted 13 lines · Jonathan Nieder
| error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
| From mediawiki::http://localhost/mediawiki
|  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
| error: Could not fetch origin

which means that the transport machinery thought the helper was going to be fetching directly to "master". I suspect you will want a 'refspec' capability like

	refspec refs/heads/*:refs/mediawiki/${remotename}/*

to fix this.

Quoted 8 lines

Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map private ref names into normal names, 2009-11-18). What namespace should a helper use when asked to fetch to FETCH_HEAD without a remote name, like

git fetch mediawiki::testwiki

?

Actually cc-ing this time. Sorry for the noise.

#7 lore raw

Re: remote helpers: best practices for using the "refspec" capability

Quoted 15 lines · Jonathan Nieder
Quoted 13 lines · Jonathan Nieder
| error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
| From mediawiki::http://localhost/mediawiki
|  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
| error: Could not fetch origin

which means that the transport machinery thought the helper was going to be fetching directly to "master". I suspect you will want a 'refspec' capability like

	refspec refs/heads/*:refs/mediawiki/${remotename}/*

to fix this.

Quoted 8 lines
Quoted 8 lines

Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map private ref names into normal names, 2009-11-18). What namespace should a helper use when asked to fetch to FETCH_HEAD without a remote name, like

git fetch mediawiki::testwiki

?

The main purpose of having the private ref names is to support incremental imports, where you obviously need to identify the sha1 of the last thing you imported, and you may have included sufficient machine-readable information in your imported commit messages to know what the foreign state was that generated that sha1. (And you need a namespace because you may have multiple refs that you imported.) I'm not sure that it makes much sense to do incremental imports without a remote name, since that's generally an operation you're not planning do repeatedly.

But the purpose of the namespacing is to be able to continue the correct incremental import, so it would make sense to do some arbitrary transformation to make your url be a valid ref directory, and use that.

It would make sense to add support for a namespace where fast-import can write whatever it wants, and it'll get discarded after the fetch is done, if it's the case that anyone can stand not having incremental imports.

-Daniel
*This .sig left intentionally blank*

#8 lore raw

Re: remote helpers: best practices for using the "refspec" capability

Heya,

Quoted 7 lines · Daniel Barkalow

But the purpose of the namespacing is to be able to continue the correct incremental import, so it would make sense to do some arbitrary transformation to make your url be a valid ref directory, and use that.

It would make sense to add support for a namespace where fast-import can write whatever it wants, and it'll get discarded after the fetch is done, if it's the case that anyone can stand not having incremental imports.

I remember a patch series by someone else a lot more that _disabled_ creating of refs during fast-import, and instead allowed for specifying what refs to update with which values at the end of the import.

Signature · 3 lines
Cheers,

Sverre Rabbelier
#9 lore raw
In reply to Jérémie NIKAES
Quoted 5 lines · Jérémie NIKAES

No, I actually still have the problem. The version commited on github is pulling revisions straight to refs/heads/master which is gross After discussing this matter with our teacher Matthieu Moy, I wanted to change this to refs/remotes/origin/master but then this warning gets thrown.

The following fixes it for me.

-- >8 --
Subject: use refs/mediawiki/<remote> namespace for private refs

Unless the "refspec" capability is used, the remote helper machinery expects remote helpers to write remote refs/heads/master ref to refs/heads/master. That's not very good since it scribbles on top of whatever local changes the user might have.

So specify a refspec that puts the refs elsewhere. For now this only works when fetching from a named remote, like so:

git remote add origin mediawiki::testwiki
git fetch origin

Attempting to fetch directly from a URL with a command like "git fetch mediawiki::testwiki" produces an "Invalid refspec" error.

---
 git-remote-mediawiki |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
git-remote-mediawiki +6 -4
index af66b58..7a3fdd3 100755
@@ -7,6 +7,7 @@ use Storable qw(freeze thaw);
 use DateTime::Format::ISO8601;
 use Encode qw(encode_utf8);
 
+my $remotename = $ARGV[0];
 my $url = $ARGV[1];
 
 print STDERR "$url\n";

@@ -62,6 +63,7 @@ while ($loop) {
 
 sub mw_capabilities {
 #	print STDOUT "fetch\n";
+	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
 	print STDOUT "import\n";
 	print STDOUT "list\n";
 	print STDOUT "option\n";

@@ -91,7 +93,7 @@ sub mw_import {
 	
 	sub get_last_revision {
 		# Get last commit sha1
-		my $commit_sha1 = `git rev-parse refs/remotes/origin/master 2>/dev/null`;
+		my $commit_sha1 = `git rev-parse refs/mediawiki/$remotename/master 2>/dev/null`;
 
 		# Get note regarding that commit
 		chomp($commit_sha1);

@@ -202,15 +204,15 @@ sub mw_import {
 		
 		# If it's a clone and the first revision, initalize this ref
 		if ($fetch_from == 1 && $n == 1) {
-			#print "reset refs/remotes/origin/master\n";
+			#print "reset refs/mediawiki/$remotename/master\n";
 		}
-		print "commit refs/remotes/origin/master\n";
+		print "commit refs/mediawiki/$remotename/master\n";
 		print "mark :$n\n";
 		print "committer $user <$user\@$wiki_name> ", $dt->epoch, " +0000\n";
 		print "data ", bytes::length(encode_utf8($comment)), "\n", encode_utf8($comment);
 		# If it's not a clone, needs to know where to start from
 		if ($fetch_from != 1 && $n == 1) {
-			print "from refs/remotes/origin/master^0\n";
+			print "from refs/mediawiki/$remotename/master^0\n";
 		}
 		print "M 644 inline $title.wiki\n";
 		print "data ", bytes::length(encode_utf8($content)), "\n", encode_utf8($content);
Signature · 1 line
1.7.5.3
#10 lore raw
Quoted 9 lines · Jonathan Nieder
Quoted 5 lines · Jérémie NIKAES

No, I actually still have the problem. The version commited on github is pulling revisions straight to refs/heads/master which is gross After discussing this matter with our teacher Matthieu Moy, I wanted to change this to refs/remotes/origin/master but then this warning gets thrown.

The following fixes it for me.

It seems it does, thanks!

clone does not give the warning anymore, and incremental pull work.

git-remote-mediawiki while ($loop) { · line 66

Quoted 3 lines

sub mw_capabilities {
# print STDOUT "fetch\n";

git-remote-mediawiki +1 -0
+	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";

Is this "mediawiki" comming from the URL (mediawiki::...), or is it just a convention you've set?

We've tried with refspec refs/heads/*:refs/remotes/origin/*, but without success.

Do I understand correctly that the "pull" is done in 3 stages:

1) import into refs/mediawiki/origin

2) fetch the imported ref into refs/remotes/origin

3) and merge as usual into the current branch

?

If so, I don't understand why the distinction between 1) and 2) is necessary, and why not to fetch directly into refs/remotes/origin/. IOW, in which case is refs/remotes/origin/master different from refs/mediawiki/origin/master for example?

Thanks,

Signature · 2 lines
Matthieu Moy
http://www-verimag.imag.fr/~moy/
#11 lore raw

Hi,

Quoted 11 lines · Matthieu Moy

git-remote-mediawiki while ($loop) { · line 66

Quoted 3 lines · Jonathan Nieder

sub mw_capabilities {
# print STDOUT "fetch\n";

git-remote-mediawiki +1 -0
+	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";

Is this "mediawiki" comming from the URL (mediawiki::...), or is it just a convention you've set?

We've tried with refspec refs/heads/*:refs/remotes/origin/*, but without success.

Yes, I stole the convention for the namespace name from the commit message to v1.7.0-rc0~62^2~19 (Allow helper to map private ref names into normal names, 2009-11-18).

Quoted 6 lines

Do I understand correctly that the "pull" is done in 3 stages:

1) import into refs/mediawiki/origin

2) fetch the imported ref into refs/remotes/origin

Quoted 8 lines

3) and merge as usual into the current branch

?

If so, I don't understand why the distinction between 1) and 2) is necessary, and why not to fetch directly into refs/remotes/origin/. IOW, in which case is refs/remotes/origin/master different from refs/mediawiki/origin/master for example?

That's a good question. One answer is that fetching to elsewhere allows the transport machinery to take care of rejecting non-fast-forward fetches and pretty-printing the ref update information:

From mediawiki::http://localhost/mediawiki

    479ee21..accd2c1  master     -> origin/master

I think in the long term, importing to no ref at all as Sverre suggested and teaching the transport-helper to do the ref update would make more sense.

Regards,
Jonathan

author
#12 lore raw
Quoted 12 lines
Quoted 6 lines · Matthieu Moy

Jonathan Nieder <[email protected]> writes:

If so, I don't understand why the distinction between 1) and 2) is necessary, and why not to fetch directly into refs/remotes/origin/. IOW, in which case is refs/remotes/origin/master different from refs/mediawiki/origin/master for example?

That's a good question.  One answer is that fetching to elsewhere allows the transport machinery to take care of rejecting non-fast-forward fetches and pretty-printing the ref update information:

Quoted 6 lines

 From mediawiki::http://localhost/mediawiki
   479ee21..accd2c1  master     -> origin/master

I think in the long term, importing to no ref at all as Sverre suggested and teaching the transport-helper to do the ref update would make more sense.

Well yes that would indeed make more sense. It feels weird to have to do it like this.
Anyways, thanks a lot again for your help, a rfc patch should come relatively quickly with the project in its current state.

Regards,

Signature · 1 line
Jérémie Nikaes
author
#14 lore raw
In reply to Matthieu Moy

2011/5/30 Matthieu Moy <[email protected]>:

Quoted 10 lines
Quoted 8 lines · Jonathan Nieder
Quoted 5 lines · Jérémie NIKAES

No, I actually still have the problem. The version commited on github is pulling revisions straight to refs/heads/master which is gross After discussing this matter with our teacher Matthieu Moy, I wanted to change this to refs/remotes/origin/master but then this warning gets thrown.

Quoted 6 lines
Quoted 1 line

The following fixes it for me.

It seems it does, thanks!

clone does not give the warning anymore, and incremental pull work.

Yes ! It definitely does ! Works like a charm.

Quoted 18 lines
Quoted 3 lines

 sub mw_capabilities {
 #    print STDOUT "fetch\n";
+     print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";

Is this "mediawiki" comming from the URL (mediawiki::...), or is it just a convention you've set?

We've tried with refspec refs/heads/*:refs/remotes/origin/*, but without success.

Do I understand correctly that the "pull" is done in 3 stages:

1) import into refs/mediawiki/origin

2) fetch the imported ref into refs/remotes/origin

3) and merge as usual into the current branch

Quoted 8 lines

?

If so, I don't understand why the distinction between 1) and 2) is necessary, and why not to fetch directly into refs/remotes/origin/. IOW, in which case is refs/remotes/origin/master different from refs/mediawiki/origin/master for example?

Thanks,

We just committed this change to our master branch on github. We actually have the same questions as Matthieu. We don't really understand why we have to use another ref to make it work.

Huge thanks to you,

Signature · 1 line
Jérémie Nikaes