git-svn and a nested branches folder

Hi,

I'm having some trouble with using git-svn to fetch a repository, and I think it's because the repository doesn't store branches as a flat list directly under the 'branches' directory.

Basically, we have a structure like this:

|

+-trunk
+-tags
+-branches

+ category-a
+ branch-a
+ branch-b
+ category-b
+ branch-c
+ branch-d

etc. category-a and category-b are simple directories created using svn mkdir. The branches are created using svn cp.

It helps us to organise the branches better, but the rationale is besides the point. The problem is that git-svn seems to want to treat category-a and category-b as branches, which isn't right at all. As a result, git-svn seems to skip most (if not all) revisions that occur in these directories and creates a lot of entries in unhandled.log.

I've also encountered an index corruption in one of the .git/svn/<branch>/index files which I think it probably related.

I've had a quick look at the source myself, but perl isn't my strong point. What I think it should do is something like recurse down the tree from the directory given looking for folders that copy from trunk (or some other branch that already exists). That would work perfectly well for the default flat branch storage method as well as the one we use.

The only other problem is in branch naming, which could clash if you only use the outer-most directory name, so I'd suggest something that involves concatenating the folders in the path relative to 'branches' to keep them unique (if git can handle slashes in branch names then all the better).

Quoted 1 line · Russ Brown

Hi,

Hi Russ,

Quoted 17 lines

I'm having some trouble with using git-svn to fetch a repository, and I think it's because the repository doesn't store branches as a flat list directly under the 'branches' directory.

Basically, we have a structure like this:

|

+-trunk
+-tags
+-branches

+ category-a
+ branch-a
+ branch-b
+ category-b
+ branch-c
+ branch-d

Quoted 8 lines

etc. category-a and category-b are simple directories created using svn mkdir. The branches are created using svn cp.

It helps us to organise the branches better, but the rationale is besides the point. The problem is that git-svn seems to want to treat category-a and category-b as branches, which isn't right at all. As a result, git-svn seems to skip most (if not all) revisions that occur in these directories and creates a lot of entries in unhandled.log.

This is a known problem with the way git-svn handles branches and tags. Nested branch (and tags) structures aren't supported with globbing and so you can't have more than one branches/tags specification in your config.

Quoted 2 lines

I've also encountered an index corruption in one of the .git/svn/<branch>/index files which I think it probably related.

Not sure about that. git-svn should detect and attempt to fix index corruptions (which happened for me since I interrupt git-svn quite often when working on it).

Quoted 5 lines

I've had a quick look at the source myself, but perl isn't my strong point. What I think it should do is something like recurse down the tree from the directory given looking for folders that copy from trunk (or some other branch that already exists). That would work perfectly well for the default flat branch storage method as well as the one we use.

The globbing functionality of branches in git-svn is some of the ugliest code I've ever written in my life. Somehow I got it working for the simple general cases but I've been afraid to touch it for a while...

Quoted 5 lines

The only other problem is in branch naming, which could clash if you only use the outer-most directory name, so I'd suggest something that involves concatenating the folders in the path relative to 'branches' to keep them unique (if git can handle slashes in branch names then all the better).

As Peter suggested, disable globbing for branches and use explicit fetch refspecs for now...

Quoted 14 lines · Eric Wong

Quoted 12 lines · Russ Brown

Basically, we have a structure like this:

|

+-trunk
+-tags
+-branches

+ category-a
+ branch-a
+ branch-b
+ category-b
+ branch-c
+ branch-d

Quoted 9 lines

Quoted 5 lines

The only other problem is in branch naming, which could clash if you only use the outer-most directory name, so I'd suggest something that involves concatenating the folders in the path relative to 'branches' to keep them unique (if git can handle slashes in branch names then all the better).

As Peter suggested, disable globbing for branches and use explicit fetch refspecs for now...

I've actually knocked up a rough script which generates a list of refspec lines for you given a repo URL, trunk reference and branches directory. It uses svn log -v --xml and pipes it through a couple of XSLT templates, and basically looks for all copies that copy from trunk (recursively: so it includes branches of branches too). I can post it to the list if you'd find it useful or interesting.

It's generating output that looks sensible to me, but the results aren't quite what I'd expected. I'll paste a sample in here in case there's anything obvious someone might spot that I've missed

# This line was generated by git-svn init, and I kept it fetch = all/trunk:refs/remotes/trunk

# These lines generated by my tool, dirnames replaced for security reasons:

fetch = branches/folder/projecta:refs/remotes/svn/folder/projecta fetch = branches/folder/projectb:refs/remotes/svn/folder/projectb fetch = branches/folder/projectc:refs/remotes/svn/folder/projectc fetch = branches/folder/projectd:refs/remotes/svn/folder/projectd fetch = branches/folder/projecte:refs/remotes/svn/folder/projecte fetch = branches/folder/projectf:refs/remotes/svn/folder/projectf fetch = branches/folder/projectg:refs/remotes/svn/folder/projectg

git branch -a doesn't list any of those branches after fetch completes. Looking back at the output from fetch, all revisions applied were to trunk.

Anything wrong with those fetch lines?

From your tree diagram, it seemed that trunk/ and branches/ were at

Thanks for your time.

Quoted 50 lines · Russ Brown

Quoted 14 lines · Eric Wong

Quoted 12 lines · Russ Brown

Basically, we have a structure like this:

|

+-trunk
+-tags
+-branches

+ category-a
+ branch-a
+ branch-b
+ category-b
+ branch-c
+ branch-d

Quoted 8 lines

Quoted 5 lines

The only other problem is in branch naming, which could clash if you only use the outer-most directory name, so I'd suggest something that involves concatenating the folders in the path relative to 'branches' to keep them unique (if git can handle slashes in branch names then all the better).

As Peter suggested, disable globbing for branches and use explicit fetch refspecs for now...

I've actually knocked up a rough script which generates a list of refspec lines for you given a repo URL, trunk reference and branches directory. It uses svn log -v --xml and pipes it through a couple of XSLT templates, and basically looks for all copies that copy from trunk (recursively: so it includes branches of branches too). I can post it to the list if you'd find it useful or interesting.

It's generating output that looks sensible to me, but the results aren't quite what I'd expected. I'll paste a sample in here in case there's anything obvious someone might spot that I've missed

# This line was generated by git-svn init, and I kept it fetch = all/trunk:refs/remotes/trunk

# These lines generated by my tool, dirnames replaced for security reasons:

fetch = branches/folder/projecta:refs/remotes/svn/folder/projecta fetch = branches/folder/projectb:refs/remotes/svn/folder/projectb fetch = branches/folder/projectc:refs/remotes/svn/folder/projectc fetch = branches/folder/projectd:refs/remotes/svn/folder/projectd fetch = branches/folder/projecte:refs/remotes/svn/folder/projecte fetch = branches/folder/projectf:refs/remotes/svn/folder/projectf fetch = branches/folder/projectg:refs/remotes/svn/folder/projectg

git branch -a doesn't list any of those branches after fetch completes. Looking back at the output from fetch, all revisions applied were to trunk.

Quoted 1 line

Anything wrong with those fetch lines?

From your tree diagram, it seemed that trunk/ and branches/ were at the same depth in your SVN repository. However, in your generated fetch lines they all started with "branches/" in front, yet your trunk fetch line had "all/" in front of trunk, so maybe prefixing the generated ones with "all/" helps?

Quoted 1 line

Thanks for your time.

No problem.

I'm having some trouble with using git-svn to fetch a repository, and I think it's because the repository doesn't store branches as a flat list directly under the 'branches' directory.

Basically, we have a structure like this:

|

+-trunk
+-tags
+-branches

+ category-a
+ branch-a
+ branch-b
+ category-b
+ branch-c
+ branch-d

etc. category-a and category-b are simple directories created using svn mkdir. The branches are created using svn cp.

It helps us to organise the branches better, but the rationale is besides the point. The problem is that git-svn seems to want to treat category-a and category-b as branches, which isn't right at all. As a result, git-svn seems to skip most (if not all) revisions that occur in these directories and creates a lot of entries in unhandled.log.

Quoted 20 lines · Russ Brown

I'm having some trouble with using git-svn to fetch a repository, and I think it's because the repository doesn't store branches as a flat list directly under the 'branches' directory.

Basically, we have a structure like this:

|

+-trunk
+-tags
+-branches

+ category-a
+ branch-a
+ branch-b
+ category-b
+ branch-c
+ branch-d

etc. category-a and category-b are simple directories created using svn mkdir. The branches are created using svn cp.

Quoted 6 lines

It helps us to organise the branches better, but the rationale is besides the point. The problem is that git-svn seems to want to treat category-a and category-b as branches, which isn't right at all. As a result, git-svn seems to skip most (if not all) revisions that occur in these directories and creates a lot of entries in unhandled.log.

So what did you specify in your .git/config file regarding the svn structure?

Quoted 29 lines · David Kastrup

Quoted 26 lines · Russ Brown

I'm having some trouble with using git-svn to fetch a repository, and I think it's because the repository doesn't store branches as a flat list directly under the 'branches' directory.

Basically, we have a structure like this:

|

+-trunk
+-tags
+-branches

+ category-a
+ branch-a
+ branch-b
+ category-b
+ branch-c
+ branch-d

etc. category-a and category-b are simple directories created using svn mkdir. The branches are created using svn cp.

It helps us to organise the branches better, but the rationale is besides the point. The problem is that git-svn seems to want to treat category-a and category-b as branches, which isn't right at all. As a result, git-svn seems to skip most (if not all) revisions that occur in these directories and creates a lot of entries in unhandled.log.

Quoted 2 lines

So what did you specify in your .git/config file regarding the svn structure?

I specified the 'branches' directory, but that's because earlier in the life of the repo we did just do the flat branch layout, but decided to make it more structured once that got unwieldy.

Is it possible to specify more than one folder for the branches option?

Quoted 29 lines · Russ Brown

Quoted 28 lines · David Kastrup

Quoted 26 lines · Russ Brown

I'm having some trouble with using git-svn to fetch a repository, and I think it's because the repository doesn't store branches as a flat list directly under the 'branches' directory.

Basically, we have a structure like this:

|

+-trunk
+-tags
+-branches

+ category-a
+ branch-a
+ branch-b
+ category-b
+ branch-c
+ branch-d

etc. category-a and category-b are simple directories created using svn mkdir. The branches are created using svn cp.

It helps us to organise the branches better, but the rationale is besides the point. The problem is that git-svn seems to want to treat category-a and category-b as branches, which isn't right at all. As a result, git-svn seems to skip most (if not all) revisions that occur in these directories and creates a lot of entries in unhandled.log.

Quoted 5 lines

Quoted 2 lines

So what did you specify in your .git/config file regarding the svn structure?

I specified the 'branches' directory, but that's because earlier in the life of the repo we did just do the flat branch layout, but decided to make it more structured once that got unwieldy.

Cough, cough. _What_ did you specify in your .git/config file regarding the svn structure? Please quote the section.

Quoted 2 lines

Is it possible to specify more than one folder for the branches option?

It is possible to adapt the config section to the actual layout. If not otherwise, by starting with
git svn init
with a clean slate, editing the config file, and only then actually fetching stuff.

However, git-svn will not magically start guessing that you changed your structure around. You have to edit the configuration appropriately.

Quoted 31 lines · David Kastrup

Quoted 29 lines · Russ Brown

Quoted 28 lines · David Kastrup

Quoted 26 lines · Russ Brown

I'm having some trouble with using git-svn to fetch a repository, and I think it's because the repository doesn't store branches as a flat list directly under the 'branches' directory.

Basically, we have a structure like this:

|

+-trunk
+-tags
+-branches

+ category-a
+ branch-a
+ branch-b
+ category-b
+ branch-c
+ branch-d

etc. category-a and category-b are simple directories created using svn mkdir. The branches are created using svn cp.

It helps us to organise the branches better, but the rationale is besides the point. The problem is that git-svn seems to want to treat category-a and category-b as branches, which isn't right at all. As a result, git-svn seems to skip most (if not all) revisions that occur in these directories and creates a lot of entries in unhandled.log.

Quoted 9 lines

Quoted 5 lines

Quoted 2 lines

So what did you specify in your .git/config file regarding the svn structure?

I specified the 'branches' directory, but that's because earlier in the life of the repo we did just do the flat branch layout, but decided to make it more structured once that got unwieldy.

Cough, cough. _What_ did you specify in your .git/config file regarding the svn structure? Please quote the section.

Erm, sorry.

[svn-remote "svn"]
url = svn://svn.<name>.com
fetch = trunk:refs/remotes/trunk
branches = branches/*:refs/remotes/*
tags = tags/*:refs/remotes/tags/*

(URL changed in case it annoys my employers)

I didn't write this by hand: it was generated by git-svn init.

Quoted 48 lines · Russ Brown

Quoted 31 lines · David Kastrup

Quoted 29 lines · Russ Brown

Quoted 28 lines · David Kastrup

Quoted 26 lines · Russ Brown

I'm having some trouble with using git-svn to fetch a repository, and I think it's because the repository doesn't store branches as a flat list directly under the 'branches' directory.

Basically, we have a structure like this:

|

+-trunk
+-tags
+-branches

+ category-a
+ branch-a
+ branch-b
+ category-b
+ branch-c
+ branch-d

etc. category-a and category-b are simple directories created using svn mkdir. The branches are created using svn cp.

It helps us to organise the branches better, but the rationale is besides the point. The problem is that git-svn seems to want to treat category-a and category-b as branches, which isn't right at all. As a result, git-svn seems to skip most (if not all) revisions that occur in these directories and creates a lot of entries in unhandled.log.

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

So what did you specify in your .git/config file regarding the svn structure?

I specified the 'branches' directory, but that's because earlier in the life of the repo we did just do the flat branch layout, but decided to make it more structured once that got unwieldy.

Cough, cough. _What_ did you specify in your .git/config file regarding the svn structure? Please quote the section.

Erm, sorry.

[svn-remote "svn"]
url = svn://svn.<name>.com
fetch = trunk:refs/remotes/trunk
branches = branches/*:refs/remotes/*
tags = tags/*:refs/remotes/tags/*

Quoted 4 lines

(URL changed in case it annoys my employers)

I didn't write this by hand: it was generated by git-svn init.

Try something like this:

[svn-remote "svn"]

	# trunk
	fetch = trunk:refs/remotes/trunk

	# branches
	fetch = branches/category-a/branch_a:refs/remotes/svn/branch_a
	fetch = branches/category-a/branch_b:refs/remotes/svn/branch_b
	fetch = branches/category-b/branch_c:refs/remotes/svn/branch_c

	# tags
	tags = tags/*:refs/remotes/tags/*

(Not sure if wildcards will work here, but I'm sure you could experiment and try
it out :-)

-Peter