importing multi-project svn repositories
What's the recommended way to import the full history from an svn repository organized as a group of projects? E.g., as described at
http://svnbook.red-bean.com/nightly/en/
svn.reposadmin.planning.html#svn.reposadmin.projects.chooselayout:
/
calc/
trunk/
tags/
branches/
calendar/
trunk/
tags/
branches/
spreadsheet/
trunk/
tags/
branches/
…
I'd like to import calc at the top level, put calc/tags/foo in git's tags/calc/foo and calc/branches/baz in git's heads/calc/baz. Ditto for calendar, spreadsheet, etc.
I suspect there's a way to use git-svnimport repeatedly with appropriate editing of .git/svn2git.
thanks,
dave h
Quoted 1 line · David Hanson
I'd like to import calc at the top level, put calc/tags/foo in git's tags/calc/foo and calc/branches/baz in git's heads/calc/baz. Ditto for calendar, spreadsheet, etc.
Try git-svn rather than git-svnimport. The latter, AFAIK, is no longer really maintained. The former will do what you want pretty easily. You just run it like
git svn clone --prefix=calc/ -T trunk -t tags -b branches \
http://svn/repo/url/calc gitrepo
and it'll create a git repository called "gitrepo" with the calc project's trunk/tags/branches as git tags. Then do
git svn clone --prefix=spreadsheet/ -T trunk -t tags -b branches \
http://svn/repo/url/spreadsheet gitrepo
and it will add the spreadsheet branches to the same repo. The "--prefix" option is needed to keep the branch namespaces from overlapping. git-svn will make an attempt to figure out the correct history of the branches and tags relative to the trunk so they look like sane git branches.
If you want to be able to check out calc and spreadsheet simultaneously, then you probably need to use git's submodule support, which is kind of a work in progress at the moment.
-Steve
Quoted 7 lines · Steven Grimm
Quoted 3 lines · David Hanson
I'd like to import calc at the top level, put calc/tags/foo in git's tags/calc/foo and calc/branches/baz in git's heads/calc/baz. Ditto for calendar, spreadsheet, etc.
Try git-svn rather than git-svnimport. The latter, AFAIK, is no longer really maintained. ...
Thanks for the tip. I was running git 1.5.0, which didn't support "clone" to git-svn. I upgraded to 1.5.1.3 and followed your suggestions.
If I decide to flush the svn repository in favor of git, I'll probably do some manual rearrangement so that commands like "git tag -l" gives useful results.
thanks,
dave h
Deprecate git-svnimport?
Quoted 2 lines · Steven Grimm
Try git-svn rather than git-svnimport. The latter, AFAIK, is no longer really maintained. ...
I said that because in the time I've been using git and reading this list, I think I've seen a grand total of one patch to git-svnimport, a ton of them to git-svn, and an occasional new user posting about git-svnimport failing to do something that git-svn would have done if they'd used it instead.
It looks like git-svnimport hasn't been updated to the 1.5.x remote branch naming convention (it still calls the svn head tracking branch "origin" if its manpage is to be believed).
Originally it looked like git-svn was what you used if you wanted to track just one svn branch bidirectionally, and git-svnimport was how you tracked multiple svn branches read-only. But now that git-svn has good handling of multi-branch svn repositories, I wonder if git-svnimport still serves a purpose. Should it perhaps be deprecated and removed in a future release? Or is it still useful for things that git-svn doesn't handle?
-Steve
minimize_url in git-svn?
I was trying to run git-svn against this:
https://repo.socialtext.net:8999/svn/socialtext/trunk
This is an open source project [*1*] and the trunk is supposed to be readable by everybody, but it seems that anything outside that area needs authentication. If I mimick the example in git-svn.txt manual page to clone from there, it creates trunk, trunk/.git, and then asks for password:
$ URL=https://repo.socialtext.net:8999/svn/socialtext/trunk $ git-svn clone $URL Authentication realm: <https://repo.socialtext.net:8999> Auth for SVN Password for 'junio': ^C
I've narrowed it down to this part of git-svn. If I tell it not to bother "minimiz"ing the URL, it seems to import without stepping outside of the URL it was given.
git-svn.perl +2 -1
@@ -1038,7 +1038,8 @@ } $self->{repo_id} = $existing; } else { - my $min_url = Git::SVN::Ra->new($url)->minimize_url; + my $ra = Git::SVN::Ra->new($url); + my $min_url = $url; # $ra->minimize_url; $existing = find_existing_remote($min_url, $r); if ($existing) { unless ($no_write) {
Two and half questions.
* What does minimize do, and why is it necessary?
* The resulting git-svn remote tracking branch (and 'master') seems to check out fine, but I do not know what damage the hack to avoid minimizing is causing. Are there any? I see many 0{40} lines in trunk/.git/svn/git-svn/.rev_db.* file, and also many lines in unhandled.log file (+empty_dir, +file_prop, and +dir_prop). Are these something to worry about?
* Assuming there aren't any damage, or maybe some damage that would cause minor decreased functionality/interoperability, would it perhaps make sense to optionally allow skipping the minimizing to avoid this problem? Would it make sense, or is the setting at socialtext site too esoteric and it isn't worth to worry about?
[Footnote]
*1* http://www.socialtext.net/stoss/index.cgi?developing_with_a_dev_env
Re: minimize_url in git-svn?
Quoted 22 lines · Junio C Hamano
I was trying to run git-svn against this:
https://repo.socialtext.net:8999/svn/socialtext/trunk
This is an open source project [*1*] and the trunk is supposed to be readable by everybody, but it seems that anything outside that area needs authentication. If I mimick the example in git-svn.txt manual page to clone from there, it creates trunk, trunk/.git, and then asks for password:
$ URL=https://repo.socialtext.net:8999/svn/socialtext/trunk $ git-svn clone $URL Authentication realm: <https://repo.socialtext.net:8999> Auth for SVN Password for 'junio': ^CI've narrowed it down to this part of git-svn. If I tell it not to bother "minimiz"ing the URL, it seems to import without stepping outside of the URL it was given.
git-svn.perl line 1045
Quoted 9 lines
git-svn.perl +2
-1@@ -1038,7 +1038,8 @@ } $self->{repo_id} = $existing; } else { - my $min_url = Git::SVN::Ra->new($url)->minimize_url; + my $ra = Git::SVN::Ra->new($url); + my $min_url = $url; # $ra->minimize_url; $existing = find_existing_remote($min_url, $r); if ($existing) { unless ($no_write) {
That should be fine.
Quoted 3 lines
Two and half questions.
* What does minimize do, and why is it necessary?
I try to connect to the root (or closer to the root) of the repository. This allows branches and tags to be tracked more effectively without needing reconnects. There's a reparent function in SVN 1.4, but it doesn't work correctly with svn:// repos last I checked (1.4.3)
Quoted 7 lines
* The resulting git-svn remote tracking branch (and 'master') seems to check out fine, but I do not know what damage the hack to avoid minimizing is causing. Are there any? I see many 0{40} lines in trunk/.git/svn/git-svn/.rev_db.* file, and also many lines in unhandled.log file (+empty_dir, +file_prop, and +dir_prop). Are these something to worry about?
Nope. unhandled.log is strictly informational.
.rev_db is offset-based database. Revision numbers to git commits can be looked up using (SVN revision * 41).
If the project has really high revision numbers (like gcc) or lots of tags, it's a space-killer. I've been meaning to add an optional SQLite alternative to .rev_db for people tracking those projects. Patches welcome :)
Quoted 6 lines
* Assuming there aren't any damage, or maybe some damage that would cause minor decreased functionality/interoperability, would it perhaps make sense to optionally allow skipping the minimizing to avoid this problem? Would it make sense, or is the setting at socialtext site too esoteric and it isn't worth to worry about?
It *should* be automatically detecting the highest level up it can access and stop there. In your case, there's obviously something broken in my code :(
I've definitely tested this as working against Seth Falcon's hedgehog repo (URL is somewhere in the archives). I also setup a test repository somewhere that I can double-check against.
Quoted 3 lines
[Footnote]
*1* http://www.socialtext.net/stoss/index.cgi?developing_with_a_dev_env
I'll try to take a look at that in the next few days. I also have segfaults to fix that I haven't gotten to :(
Signature · 1 line
Eric Wong
Re: minimize_url in git-svn?
Quoted 2 lines · Eric Wong
.rev_db is offset-based database. Revision numbers to git commits can be looked up using (SVN revision * 41).
Ah, that explains why the file looks so sparse. Their trunk/ seems to be somewhat inactive and activities elsewhere in the branch/ namespace we cannot view may advance SVN revisions.
Quoted 10 lines
Quoted 6 lines
* Assuming there aren't any damage, or maybe some damage that would cause minor decreased functionality/interoperability, would it perhaps make sense to optionally allow skipping the minimizing to avoid this problem? Would it make sense, or is the setting at socialtext site too esoteric and it isn't worth to worry about?
It *should* be automatically detecting the highest level up it can access and stop there. In your case, there's obviously something broken in my code :(
Perhaps, but if the user has the proper user identity to access the full site, I suspect you would be better off getting the credential and connect to the higher level. In other words, the definition of "the highest level up it can access" depends on the user and what the user is trying to do (i.e. a user who does have an account with that project may not be interested in getting tags/ nor branches/, in which case the user may want to be able to say "I am not interested in uplevel" upfront, so that the program does not even have to ask for the password. On the other hand the user may want to access tags/ and branches/ in which case the user would appreciate the current behaviour, asking for the password and go up after authenticated).
Quoted 10 lines
I've definitely tested this as working against Seth Falcon's hedgehog repo (URL is somewhere in the archives). I also setup a test repository somewhere that I can double-check against.
Quoted 3 lines
[Footnote]
*1* http://www.socialtext.net/stoss/index.cgi?developing_with_a_dev_env
I'll try to take a look at that in the next few days. I also have segfaults to fix that I haven't gotten to :(
Thanks. If I type <Return> instead of ^C in the transcript at the beginning of my message, I get a segfault. It may or may not be the same one you have been chasing.
PATCH 1/1 git-svn: don't attempt to minimize URLs by default
In reply to Eric WongFor tracking branches and tags, git-svn prefers to connect to the root of the repository or at least the level that houses branches and tags as well as trunk. However, users that are accustomed to tracking a single directory have no use for this feature.
As pointed out by Junio, users may not have permissions to connect to connect to a higher-level path in the repository.
While the current minimize_url() function detects lack of permissions to certain paths _after_ successful logins, it cannot effectively determine if it is trying to access a login-only portion of a repo when the user expects to connect to a part where anonymous access is allowed.
For people used to the git-svnimport switches of --trunk, --tags, --branches, they'll already pass the repository root (or root+subdirectory), so minimize URL isn't of too much use to them, either.
For people *not* used to git-svnimport, git-svn also supports:
git svn init --minimize-url \
--trunk http://repository-root/foo/trunk \
--branches http://repository-root/foo/branches \
--tags http://repository-root/foo/tags
And this is where the new --minimize-url command-line switch comes in to allow for this behavior to continue working.
--- git-svn.perl | 5 +++-- t/t9100-git-svn-basic.sh | 2 +- t/t9104-git-svn-follow-parent.sh | 13 +++++++------ t/t9105-git-svn-commit-diff.sh | 2 +- t/t9110-git-svn-use-svm-props.sh | 7 ++++--- t/t9111-git-svn-use-svnsync-props.sh | 6 +++--- 6 files changed, 19 insertions(+), 16 deletions(-)
git-svn.perl +3 -2
@@ -80,6 +80,7 @@ my %icv; my %init_opts = ( 'template=s' => \$_template, 'shared:s' => \$_shared, 'trunk|T=s' => \$_trunk, 'tags|t=s' => \$_tags, 'branches|b=s' => \$_branches, 'prefix=s' => \$_prefix, + 'minimize-url|m' => \$Git::SVN::_minimize_url, 'no-metadata' => sub { $icv{noMetadata} = 1 }, 'use-svm-props' => sub { $icv{useSvmProps} = 1 }, 'use-svnsync-props' => sub { $icv{useSvnsyncProps} = 1 },
@@ -820,7 +821,7 @@ use strict; use warnings; use vars qw/$default_repo_id $default_ref_id $_no_metadata $_follow_parent $_repack $_repack_flags $_use_svm_props $_head - $_use_svnsync_props $no_reuse_existing/; + $_use_svnsync_props $no_reuse_existing $_minimize_url/; use Carp qw/croak/; use File::Path qw/mkpath/; use File::Copy qw/copy/;
@@ -1037,7 +1038,7 @@ sub init_remote_config { "[svn-remote \"$existing\"]\n"; } $self->{repo_id} = $existing; - } else { + } elsif ($_minimize_url) { my $min_url = Git::SVN::Ra->new($url)->minimize_url; $existing = find_existing_remote($min_url, $r); if ($existing) {
t/t9100-git-svn-basic.sh +1 -1
@@ -229,7 +229,7 @@ test_expect_failure 'exit if init-ing a would clobber a URL' " test_expect_success \ 'init allows us to connect to another directory in the same repo' " - git-svn init -i bar $svnrepo/bar && + git-svn init --minimize-url -i bar $svnrepo/bar && git config --get svn-remote.svn.fetch \ '^bar:refs/remotes/bar$' && git config --get svn-remote.svn.fetch \
t/t9104-git-svn-follow-parent.sh +7 -6
@@ -28,7 +28,7 @@ test_expect_success 'initialize repo' " " test_expect_success 'init and fetch a moved directory' " - git-svn init -i thunk $svnrepo/thunk && + git-svn init --minimize-url -i thunk $svnrepo/thunk && git-svn fetch -i thunk && test \"\`git-rev-parse --verify refs/remotes/thunk@2\`\" \ = \"\`git-rev-parse --verify refs/remotes/thunk~1\`\" &&
@@ -68,7 +68,8 @@ test_expect_success 'follow larger parent' " echo hi > import/trunk/thunk/bump/thud/file && svn import -m 'import a larger parent' import $svnrepo/larger-parent && svn cp -m 'hi' $svnrepo/larger-parent $svnrepo/another-larger && - git-svn init -i larger $svnrepo/another-larger/trunk/thunk/bump/thud && + git-svn init --minimize-url -i larger \ + $svnrepo/another-larger/trunk/thunk/bump/thud && git-svn fetch -i larger && git-rev-parse --verify refs/remotes/larger && git-rev-parse --verify \
@@ -90,14 +91,14 @@ test_expect_success 'follow higher-level parent' " cd .. svn mkdir -m 'new glob at top level' $svnrepo/glob && svn mv -m 'move blob down a level' $svnrepo/blob $svnrepo/glob/blob && - git-svn init -i blob $svnrepo/glob/blob && + git-svn init --minimize-url -i blob $svnrepo/glob/blob && git-svn fetch -i blob " test_expect_success 'follow deleted directory' " svn mv -m 'bye!' $svnrepo/glob/blob/hi $svnrepo/glob/blob/bye && svn rm -m 'remove glob' $svnrepo/glob && - git-svn init -i glob $svnrepo/glob && + git-svn init --minimize-url -i glob $svnrepo/glob && git-svn fetch -i glob && test \"\`git cat-file blob refs/remotes/glob:blob/bye\`\" = hi && test \"\`git ls-tree refs/remotes/glob | wc -l \`\" -eq 1
@@ -127,7 +128,7 @@ test_expect_success 'follow-parent avoids deleting relevant info' " poke native/t/c.t && svn commit -m 'reorg test' && cd .. && - git-svn init -i r9270-t \ + git-svn init --minimize-url -i r9270-t \ $svnrepo/r9270/trunk/subversion/bindings/swig/perl/native/t && git-svn fetch -i r9270-t && test \`git rev-list r9270-t | wc -l\` -eq 2 &&
@@ -137,7 +138,7 @@ test_expect_success 'follow-parent avoids deleting relevant info' " test_expect_success "track initial change if it was only made to parent" " svn cp -m 'wheee!' $svnrepo/r9270/trunk $svnrepo/r9270/drunk && - git-svn init -i r9270-d \ + git-svn init --minimize-url -i r9270-d \ $svnrepo/r9270/drunk/subversion/bindings/swig/perl/native/t && git-svn fetch -i r9270-d && test \`git rev-list r9270-d | wc -l\` -eq 3 &&
t/t9105-git-svn-commit-diff.sh +1 -1
@@ -33,7 +33,7 @@ test_expect_success 'test the commit-diff command' " test_expect_success 'commit-diff to a sub-directory (with git-svn config)' " svn import -m 'sub-directory' import $svnrepo/subdir && - git-svn init $svnrepo/subdir && + git-svn init --minimize-url $svnrepo/subdir && git-svn fetch && git-svn commit-diff -r3 '$prev' '$head' && svn cat $svnrepo/subdir/readme > readme.2 &&
t/t9110-git-svn-use-svm-props.sh +4 -3
@@ -9,9 +9,10 @@ test_description='git-svn useSvmProps test' test_expect_success 'load svm repo' " svnadmin load -q $rawsvnrepo < ../t9110/svm.dump && - git-svn init -R arr -i bar $svnrepo/mirror/arr && - git-svn init -R argh -i dir $svnrepo/mirror/argh && - git-svn init -R argh -i e $svnrepo/mirror/argh/a/b/c/d/e && + git-svn init --minimize-url -R arr -i bar $svnrepo/mirror/arr && + git-svn init --minimize-url -R argh -i dir $svnrepo/mirror/argh && + git-svn init --minimize-url -R argh -i e \ + $svnrepo/mirror/argh/a/b/c/d/e && git-config svn.useSvmProps true && git-svn fetch --all "
t/t9111-git-svn-use-svnsync-props.sh +3 -3
@@ -9,9 +9,9 @@ test_description='git-svn useSvnsyncProps test' test_expect_success 'load svnsync repo' " svnadmin load -q $rawsvnrepo < ../t9111/svnsync.dump && - git-svn init -R arr -i bar $svnrepo/bar && - git-svn init -R argh -i dir $svnrepo/dir && - git-svn init -R argh -i e $svnrepo/dir/a/b/c/d/e && + git-svn init --minimize-url -R arr -i bar $svnrepo/bar && + git-svn init --minimize-url -R argh -i dir $svnrepo/dir && + git-svn init --minimize-url -R argh -i e $svnrepo/dir/a/b/c/d/e && git-config svn.useSvnsyncProps true && git-svn fetch --all "
Signature · 1 line
Eric Wong
Re: minimize_url in git-svn?
In reply to Junio C HamanoHi,
Quoted 10 lines · Junio C Hamano
I was trying to run git-svn against this:
https://repo.socialtext.net:8999/svn/socialtext/trunk
This is an open source project [*1*] and the trunk is supposed to be readable by everybody, but it seems that anything outside that area needs authentication. If I mimick the example in git-svn.txt manual page to clone from there, it creates trunk, trunk/.git, and then asks for password:
Quoted 4 lines
$ URL=https://repo.socialtext.net:8999/svn/socialtext/trunk $ git-svn clone $URL Authentication realm: <https://repo.socialtext.net:8999> Auth for SVN Password for 'junio': ^C
FWIW I encountered the same problem with tailor.
Ciao,
Dscho