help moving boost.org to git
I have a question about the best approach to take for refactoring a large svn project into git. The project, boost.org, is a collection of C++ libraries (>100) that are mostly independent. (There may be cross-library dependencies, but we plan to handle that at a higher level.) After the move to git, we'd like each library to be in its own git repository. Boost can then be a stitching-together of these, using submodules or something (opinions welcome). It's an old project with lots of history that we don't want to lose. The naive approach of simply forking into N repositories for the N libraries and deleting the unwanted files in each is unworkable because we'll end up with all the history duplicated everywhere ... >100 repositories, each larger than 100Mb.
So, what are the options? Can I somehow delete from each repository the history that is irrelevant? Is these some feature of git I don't know about that can solve this problem for us?
(Caveat: I'm new to git and still getting up to speed. An acceptable answer is: go off an learn about feature X and come back to us.)
At boost, We've already discussed a few possible approaches. Feel free to comment and/or criticize any of the solutions suggested here:
http://github.com/ryppl/ryppl/issues#issue/4
Signature · 3 lines
Eric Niebler BoostPro Computing http://www.boostpro.com
Am 7/5/2010 16:16, schrieb Eric Niebler:
Quoted 6 lines
I have a question about the best approach to take for refactoring a large svn project into git. The project, boost.org, is a collection of C++ libraries (>100) that are mostly independent. (There may be cross-library dependencies, but we plan to handle that at a higher level.) After the move to git, we'd like each library to be in its own git repository.
You could use svn2git: http://gitorious.org/svn2git
KDE uses it to split its SVN repository into pieces. The tool is driven by a "ruleset" that specifies SVN subdirectories and revision numbers that make up a module.
Signature · 2 lines
"Atomic objects are neither active nor radioactive." -- Programming Languages -- C++, Final Committee Draft (Doc.N3092)
Quoted 8 lines · Johannes Sixt
Am 7/5/2010 16:16, schrieb Eric Niebler:
Quoted 6 lines
I have a question about the best approach to take for refactoring a large svn project into git. The project, boost.org, is a collection of C++ libraries (>100) that are mostly independent. (There may be cross-library dependencies, but we plan to handle that at a higher level.) After the move to git, we'd like each library to be in its own git repository.
Quoted 4 lines
You could use svn2git: http://gitorious.org/svn2git
KDE uses it to split its SVN repository into pieces. The tool is driven by a "ruleset" that specifies SVN subdirectories and revision numbers that make up a module.
I'm off to learn about filter-branch, tree-filter and svn2git. Thanks for the suggestions. More questions to come, I'm sure.
Signature · 3 lines
Eric Niebler BoostPro Computing http://www.boostpro.com
Heya,
Quoted 2 lines · Eric Niebler
I'm off to learn about filter-branch, tree-filter and svn2git. Thanks for the suggestions. More questions to come, I'm sure.
Also have a look at git-subtree.
Signature · 3 lines
Cheers, Sverre Rabbelier
Hi,
Quoted 8 lines · Johannes Sixt
Am 7/5/2010 16:16, schrieb Eric Niebler:
Quoted 6 lines
I have a question about the best approach to take for refactoring a large svn project into git. The project, boost.org, is a collection of C++ libraries (>100) that are mostly independent. (There may be cross-library dependencies, but we plan to handle that at a higher level.) After the move to git, we'd like each library to be in its own git repository.
Quoted 4 lines
You could use svn2git: http://gitorious.org/svn2git
KDE uses it to split its SVN repository into pieces. The tool is driven by a "ruleset" that specifies SVN subdirectories and revision numbers that make up a module.
I'm also involved in moving a large SVN project to git (the mono project). I have found and fixed several issues with svn2git
git://gitorious.org/~harinath/svn2git/rrh-svn2git.git
- Hari
Quoted 12 lines · Eric Niebler
I have a question about the best approach to take for refactoring a large svn project into git. The project, boost.org, is a collection of C++ libraries (>100) that are mostly independent. (There may be cross-library dependencies, but we plan to handle that at a higher level.) After the move to git, we'd like each library to be in its own git repository. Boost can then be a stitching-together of these, using submodules or something (opinions welcome). It's an old project with lots of history that we don't want to lose. The naive approach of simply forking into N repositories for the N libraries and deleting the unwanted files in each is unworkable because we'll end up with all the history duplicated everywhere ... >100 repositories, each larger than 100Mb.
Quoted 4 lines
So, what are the options? Can I somehow delete from each repository the history that is irrelevant? Is these some feature of git I don't know about that can solve this problem for us?
You're probably looking for git-filter-branch. This tool can be used with the --subdirectory-filter option to filter out a specific subdirectory to it's own branch. Or if the project isn't split into subdirectories, you can use the --tree-filter option to filter specific files if you want.
See http://www.kernel.org/pub/software/scm/git/docs/git-filter-branch.html
for details
Signature · 1 line
Erik "kusma" Faye-Lund
Quoted 11 lines · Eric Niebler
I have a question about the best approach to take for refactoring a large svn project into git. The project, boost.org, is a collection of C++ libraries (>100) that are mostly independent. (There may be cross-library dependencies, but we plan to handle that at a higher level.) After the move to git, we'd like each library to be in its own git repository. Boost can then be a stitching-together of these, using submodules or something (opinions welcome). It's an old project with lots of history that we don't want to lose. The naive approach of simply forking into N repositories for the N libraries and deleting the unwanted files in each is unworkable because we'll end up with all the history duplicated everywhere ... >100 repositories, each larger than 100Mb.
If the libraries are not independent (i.e. some commits are across multiple libraries), submodules will give you some interesting challenges to put it mildly.
The current boost 1.43 is 29344 files, is this all there is? This should fit eaily into a single repository. The Linux kernel is much larger, and that is sort of the canonical single repo git project. I _strongly_ recommend that you go for a single repo if you can make it work.
If you manage to create a single git repo with the history you want, it is trivial to split out separate repositories of subdirectories later (and those repos will then be comparatively small). git subtree allegedly automates this process more or less (I have not used it, but have heard good things about it). What about having a single "master repository", and then using subtree to create single-library repos for the library developers if they want a smaller repo to play around in?
Quoted 3 lines
So,, what are the options? Can I somehow delete from each repository the history that is irrelevant? Is these some feature of git I don't know about that can solve this problem for us?
How do you define "irrelevant"? Do you only require enough history for git annotate/blame to give correct results? Or does this only refer to multiple repositories sharing the same ancient history?
Quoted 4 lines
At boost, We've already discussed a few possible approaches. Feel free to comment and/or criticize any of the solutions suggested here:
It is unclear from the discussion if you will change to git, or use git in addition to svn? This will have some impact on how to go about this.
- Finn Arne
Quoted 13 lines · Finn Arne Gangstad
Quoted 11 lines · Eric Niebler
I have a question about the best approach to take for refactoring a large svn project into git. The project, boost.org, is a collection of C++ libraries (>100) that are mostly independent. (There may be cross-library dependencies, but we plan to handle that at a higher level.) After the move to git, we'd like each library to be in its own git repository. Boost can then be a stitching-together of these, using submodules or something (opinions welcome). It's an old project with lots of history that we don't want to lose. The naive approach of simply forking into N repositories for the N libraries and deleting the unwanted files in each is unworkable because we'll end up with all the history duplicated everywhere ... >100 repositories, each larger than 100Mb.
Quoted 3 lines
If the libraries are not independent (i.e. some commits are across multiple libraries), submodules will give you some interesting challenges to put it mildly.
You have correctly assessed the situation. There *are* cross-library commits in our history. What are the implications of this for modularlization?
Quoted 1 line
The current boost 1.43 is 29344 files, is this all there is?
Yes.
Quoted 5 lines
This
should fit eaily into a single repository. The Linux kernel is much larger, and that is sort of the canonical single repo git project. I _strongly_ recommend that you go for a single repo if you can make it work.
It does fit into one repo, but that doesn't meet our needs for the future. Users want to install and build library X and its dependencies, not all of boost. This is increasingly becoming a problem as boost grows. Imagine if a perl programmer had to download all of CPAN to use or hack on any one perl module. Or if contributing to CPAN meant getting the whole shebang, history and all. I'm sure even in the Linux kernel, not *every* third-party driver is maintained in the master git repo.
We are aiming to make boost a clearing-house for C++ libraries (like CPAN, or PyPi for python), turning the official boost distribution into little more than a well-tested collection of the libraries that have passed our peer-review and regression test process.
In fact, the modularization has already been done, and work is well underway on the infrastructure to support dependency tracking. But the modularization is not history-preserving and needs to be redone.
Quoted 7 lines
If you manage to create a single git repo with the history you want, it is trivial to split out separate repositories of subdirectories later (and those repos will then be comparatively small). git subtree allegedly automates this process more or less (I have not used it, but have heard good things about it). What about having a single "master repository", and then using subtree to create single-library repos for the library developers if they want a smaller repo to play around in?
This sounds like it might be ok, but I need to research it.
Quoted 7 lines
Quoted 3 lines
So,, what are the options? Can I somehow delete from each repository the history that is irrelevant? Is these some feature of git I don't know about that can solve this problem for us?
How do you define "irrelevant"? Do you only require enough history for git annotate/blame to give correct results? Or does this only refer to multiple repositories sharing the same ancient history?
If multiple repositories share the same ancient history, wouldn't that give git annotate/blame enough information? Sorry, git newbie here.
Quoted 8 lines
Quoted 4 lines
At boost, We've already discussed a few possible approaches. Feel free to comment and/or criticize any of the solutions suggested here:
It is unclear from the discussion if you will change to git, or use git in addition to svn? This will have some impact on how to go about this.
The plan is to move to git. However, we don't expect this to happen overnight, so a way to continue to pull changes from a svn mirror while the new git repositories are being set up would be ideal.
Signature · 3 lines
Eric Niebler BoostPro Computing http://www.boostpro.com
(note: on this mailing list, you shouldn't drop names from the cc:
line when replying to a thread)
Quoted 7 lines · Eric Niebler
Quoted 5 lines · Finn Arne Gangstad
This
should fit eaily into a single repository. The Linux kernel is much larger, and that is sort of the canonical single repo git project. I _strongly_ recommend that you go for a single repo if you can make it work.
Quoted 7 lines
It does fit into one repo, but that doesn't meet our needs for the future. Users want to install and build library X and its dependencies, not all of boost. This is increasingly becoming a problem as boost grows. Imagine if a perl programmer had to download all of CPAN to use or hack on any one perl module. Or if contributing to CPAN meant getting the whole shebang, history and all. I'm sure even in the Linux kernel, not *every* third-party driver is maintained in the master git repo.
Actually, that's mostly not true; there are a few third-party drivers that don't make it into the core Linux repo, but that's mostly because they haven't been accepted by the kernel maintainers for whatever reason (often quality or duplication, I guess). The goal for the vast majority of Linux drivers is indeed to get merged into the Linux core.
...and it works pretty well, all things considered. It's certainly not the only way to do it for every project, but it's actually a pretty good way. The kernel repo history runs to hundreds of megs nowadays, but on a modern Internet connection that's not a big deal. And then you never have to worry about downloading more modules later.
You also never have versioning problems.
Quoted 4 lines
We are aiming to make boost a clearing-house for C++ libraries (like CPAN, or PyPi for python), turning the official boost distribution into little more than a well-tested collection of the libraries that have passed our peer-review and regression test process.
Of course you will want to have some kind of really excellent versioned dependency fetching system (exactly like CPAN or PyPi or ruby gems) if you want this to be nice. git's submodules stuff is almost certainly not going to add any features you need/want. On the other hand, cloning a separate git repo is pretty easy to write your CPAN-like script around.
Quoted 3 lines
In fact, the modularization has already been done, and work is well underway on the infrastructure to support dependency tracking. But the modularization is not history-preserving and needs to be redone.
If your code doesn't move too many files around, then splitting out the history is pretty easy with git-subtree (a tool I wrote that's not part of git):
git subtree split --prefix=/path/to/subdir
And you get a new history for just that subdir. That might do exactly what you want. It also works iteratively, so you can export your history from svn, then re-export the changes as they occur over time.
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
So,, what are the options? Can I somehow delete from each repository the history that is irrelevant? Is these some feature of git I don't know about that can solve this problem for us?
How do you define "irrelevant"? Do you only require enough history for git annotate/blame to give correct results? Or does this only refer to multiple repositories sharing the same ancient history?
If multiple repositories share the same ancient history, wouldn't that give git annotate/blame enough information? Sorry, git newbie here.
Yes, it would. But how much of the ancient history do you want? If you want all of it, you don't save any space in your repo.
Quoted 3 lines
The plan is to move to git. However, we don't expect this to happen overnight, so a way to continue to pull changes from a svn mirror while the new git repositories are being set up would be ideal.
This isn't too hard to do; you just need some scripts around git-svn and git-subtree (or whatever tool you use to do the splitting). We've done this at work for a couple of years now and it's working fine.
The confusing part is taking *submissions* back through both channels.
If you value your sanity, you probably want to only allow submissions
back via svn while you're running the two in parallel; but that makes git's added features a lot less useful, so you probably want to run in parallel for only a short time.
Have fun,
Avery
Quoted 2 lines · Avery Pennarun
(note: on this mailing list, you shouldn't drop names from the cc:
line when replying to a thread)
Noted, thanks.
Quoted 16 lines · Eric Niebler
Quoted 7 lines · Finn Arne Gangstad
Quoted 5 lines
This
should fit eaily into a single repository. The Linux kernel is much larger, and that is sort of the canonical single repo git project. I _strongly_ recommend that you go for a single repo if you can make it work.Quoted 7 lines
It does fit into one repo, but that doesn't meet our needs for the future. Users want to install and build library X and its dependencies, not all of boost. This is increasingly becoming a problem as boost grows. Imagine if a perl programmer had to download all of CPAN to use or hack on any one perl module. Or if contributing to CPAN meant getting the whole shebang, history and all. I'm sure even in the Linux kernel, not *every* third-party driver is maintained in the master git repo.
Quoted 2 lines
Actually, that's mostly not true; there are a few third-party drivers that don't make it into the core Linux repo
<snip discussion showing my ignorance of Linux's repository structure>
Thanks for the correction. The CPAN/PyPi analogy is still apt.
Quoted 11 lines
Quoted 4 lines
We are aiming to make boost a clearing-house for C++ libraries (like CPAN, or PyPi for python), turning the official boost distribution into little more than a well-tested collection of the libraries that have passed our peer-review and regression test process.
Of course you will want to have some kind of really excellent versioned dependency fetching system (exactly like CPAN or PyPi or ruby gems) if you want this to be nice. git's submodules stuff is almost certainly not going to add any features you need/want. On the other hand, cloning a separate git repo is pretty easy to write your CPAN-like script around.
Indeed, we are stealing the work of the python guys. Pip does most of what we want. They've graciously been accepting our patches so it happily clones git repos in order to satisfy dependencies now. It is some kind of really excellent! :-)
Quoted 8 lines
Quoted 3 lines
In fact, the modularization has already been done, and work is well underway on the infrastructure to support dependency tracking. But the modularization is not history-preserving and needs to be redone.
If your code doesn't move too many files around, then splitting out the history is pretty easy with git-subtree (a tool I wrote that's not part of git):
Quoted 5 lines
git subtree split --prefix=/path/to/subdirAnd you get a new history for just that subdir. That might do exactly what you want. It also works iteratively, so you can export your history from svn, then re-export the changes as they occur over time.
This looks like it here:
http://github.com/apenwarr/git-subtree
I'll have to read the docs. Thanks for the tip.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
So,, what are the options? Can I somehow delete from each repository the history that is irrelevant? Is these some feature of git I don't know about that can solve this problem for us?
How do you define "irrelevant"? Do you only require enough history for git annotate/blame to give correct results? Or does this only refer to multiple repositories sharing the same ancient history?
Quoted 5 lines
Quoted 2 lines
If multiple repositories share the same ancient history, wouldn't that give git annotate/blame enough information? Sorry, git newbie here.
Yes, it would. But how much of the ancient history do you want? If you want all of it, you don't save any space in your repo.
Repos, plural. We'd save space because the history wouldn't be duplicated in each one. Right? Or else I'm confused and this something that will become clear after I understand what git subtree does.
Right now, the other boost developers are pushing for a solution that uses grafts. I'm fuzzy on what they are exactly, but it seems that we'd freeze a svn mirror and have anybody interested in history put grafts in their local repository pointing back at the mirror. I don't know enough yet to say what the pros/cons of this approach might be wrt git subtree.
Quoted 7 lines
Quoted 3 lines
The plan is to move to git. However, we don't expect this to happen overnight, so a way to continue to pull changes from a svn mirror while the new git repositories are being set up would be ideal.
This isn't too hard to do; you just need some scripts around git-svn and git-subtree (or whatever tool you use to do the splitting). We've done this at work for a couple of years now and it's working fine.
Cool.
Quoted 5 lines
The confusing part is taking *submissions* back through both channels. If you value your sanity, you probably want to only allow submissions back via svn while you're running the two in parallel; but that makes git's added features a lot less useful, so you probably want to run in parallel for only a short time.
Oh my! I don't think we'd open the git repositories for changes until after we close down svn. This problem is hard enough.
Signature · 3 lines
Eric Niebler BoostPro Computing http://www.boostpro.com
Quoted 11 lines · Eric Niebler
Quoted 6 lines · Avery Pennarun
Quoted 2 lines · Eric Niebler
If multiple repositories share the same ancient history, wouldn't that give git annotate/blame enough information? Sorry, git newbie here.
Yes, it would. But how much of the ancient history do you want? If you want all of it, you don't save any space in your repo.
Repos, plural. We'd save space because the history wouldn't be duplicated in each one. Right? Or else I'm confused and this something that will become clear after I understand what git subtree does.
The statement "multiple repositories share the same ancient history" above is the part that's confusing. If you use a tool like git-subtree or git-filter-branch, you're actually generating a "new history" based on the original history. The "new history" obviously contains fewer files than the original, which would take less space. But if you want multiple repositories to "share the same ancient history" you can't rewrite it, and thus you aren't saving any space in any one repo.
I'm assuming you want to rewrite history to save space (since that's what this thread is about). And git annotate/blame will work as long as your rewritten history contains all the files you care about in that repo.
Quoted 5 lines
Right now, the other boost developers are pushing for a solution that uses grafts. I'm fuzzy on what they are exactly, but it seems that we'd freeze a svn mirror and have anybody interested in history put grafts in their local repository pointing back at the mirror. I don't know enough yet to say what the pros/cons of this approach might be wrt git subtree.
The primary advantage of grafts is that you can do something easy *right now* and then fix it all up later. eg. if you screw up your history extraction and do it better later, you can just re-graft it and you're done.
A secondary advantage of grafts is that cloning the "primary" repository will be tiny since it doesn't have much ancient history.
A disadvantage of grafts is that each user has to deal with grafts in his cloned repo, and unless he does, things like 'git log' and 'git blame' won't show anything from the grafted history. Supposedly 'git replace' was designed to help with this issue, but I've never used it so I don't know for sure.
And of course, grafts don't actually do any history rewriting for you.
You could split out a subtree's history and then graft it on, but the
splitting process is still the same as it would be without grafts. The alternative would be to *not* rewrite history, just keep the entire history of the whole project in one place, and graft it on if you really need it. That's actually pretty clean (and accurately reflects exactly what *really happened*, which is a nice feature to have in a vcs history), but you'll then never have a single repo of just one subproject with the entire history of that subproject. That latter turns out to not actually be very important in practice, so you might want to do it.
Quoted 8 lines
Quoted 5 lines
The confusing part is taking *submissions* back through both channels. If you value your sanity, you probably want to only allow submissions back via svn while you're running the two in parallel; but that makes git's added features a lot less useful, so you probably want to run in parallel for only a short time.
Oh my! I don't think we'd open the git repositories for changes until after we close down svn. This problem is hard enough.
It can be done, and I've done it :) But you're wise to avoid that situation.
Have fun,
Avery
Quoted 13 lines · Avery Pennarun
Quoted 11 lines · Eric Niebler
Quoted 6 lines · Avery Pennarun
Quoted 2 lines · Eric Niebler
If multiple repositories share the same ancient history, wouldn't that give git annotate/blame enough information? Sorry, git newbie here.
Yes, it would. But how much of the ancient history do you want? If you want all of it, you don't save any space in your repo.
Repos, plural. We'd save space because the history wouldn't be duplicated in each one. Right? Or else I'm confused and this something that will become clear after I understand what git subtree does.
Quoted 8 lines
The statement "multiple repositories share the same ancient history" above is the part that's confusing. If you use a tool like git-subtree or git-filter-branch, you're actually generating a "new history" based on the original history. The "new history" obviously contains fewer files than the original, which would take less space. But if you want multiple repositories to "share the same ancient history" you can't rewrite it, and thus you aren't saving any space in any one repo.
I think I have reached understanding! Thank you. It *would* save if I pull down, say, 100 of these new repos+ancient history because git would just store the ancient history locally once. I'm also guessing git is smart enough to avoid /downloading/ the ancient history 100x.
Quoted 4 lines
I'm assuming you want to rewrite history to save space (since that's what this thread is about). And git annotate/blame will work as long as your rewritten history contains all the files you care about in that repo.
Right. I now understand that, too.
Quoted 10 lines
Quoted 5 lines
Right now, the other boost developers are pushing for a solution that uses grafts. I'm fuzzy on what they are exactly, but it seems that we'd freeze a svn mirror and have anybody interested in history put grafts in their local repository pointing back at the mirror. I don't know enough yet to say what the pros/cons of this approach might be wrt git subtree.
The primary advantage of grafts is that you can do something easy *right now* and then fix it all up later. eg. if you screw up your history extraction and do it better later, you can just re-graft it and you're done.
How does one screw up the history extraction, if one is not doing any fancy history rewriting (in this scenario)? Be there dragons?
Quoted 2 lines
A secondary advantage of grafts is that cloning the "primary" repository will be tiny since it doesn't have much ancient history.
Right. Only those who ask for it will pay for it. And only developers will have need of it, and not all developers at that.
Quoted 5 lines
A disadvantage of grafts is that each user has to deal with grafts in his cloned repo, and unless he does, things like 'git log' and 'git blame' won't show anything from the grafted history. Supposedly 'git replace' was designed to help with this issue, but I've never used it so I don't know for sure.
I'll add it to the list of things to learn about.
Quoted 11 lines
And of course, grafts don't actually do any history rewriting for you. You could split out a subtree's history and then graft it on, but the splitting process is still the same as it would be without grafts. The alternative would be to *not* rewrite history, just keep the entire history of the whole project in one place, and graft it on if you really need it. That's actually pretty clean (and accurately reflects exactly what *really happened*, which is a nice feature to have in a vcs history), but you'll then never have a single repo of just one subproject with the entire history of that subproject. That latter turns out to not actually be very important in practice, so you might want to do it.
That's starting to sound pretty good.
Thanks,
Signature · 3 lines
Eric Niebler BoostPro Computing http://www.boostpro.com
Quoted 8 lines · Eric Niebler
Quoted 4 lines · Avery Pennarun
The primary advantage of grafts is that you can do something easy *right now* and then fix it all up later. eg. if you screw up your history extraction and do it better later, you can just re-graft it and you're done.
How does one screw up the history extraction, if one is not doing any fancy history rewriting (in this scenario)? Be there dragons?
Well, "rewriting history" necessarily involves changing things about the permanent record. Every time you change things, you have a potential to change them incorrectly. So in general, not rewriting is less error-prone than rewriting :)
Specifically, with a tool like git-subtree, it only really works if a particular subproject has always existed in the same subdir of your repo since it started. If the subdir was ever renamed, or if some of the files were previously part of one subdir but then moved around, git-subtree doesn't (currently) know how to deal with that. git-filter-branch can do anything you want, but you have to teach it how, which is obviously even *more* error prone.
Things are also a little messy if you have some kind of top-level directory with build infrastructure shared by all the subdirs. Does the top-level Makefile have a list of the subdirs it needs to build? If so, there's no way to extract only a subset of true history that will still build correctly - it'll be looking for directories that you explicitly removed. You could update the Makefiles programmatically in every single revision, but that's starting to get extremely messy... and your history stops representing what *real life* really looked like at the time.
If your subdirs haven't been moving around (which sounds like that might be the case for you), and you don't have any top-level files that you care about, rewriting might turn out to be straightforward. You could also make the decision on a subdir-by-subdir basis, I guess.
Have fun,
Avery
On 7/6/2010 2:13 PM, Avery Pennarun wrote:
<snip>
Quoted 5 lines
Specifically, with a tool like git-subtree, it only really works if a particular subproject has always existed in the same subdir of your repo since it started. If the subdir was ever renamed, or if some of the files were previously part of one subdir but then moved around, git-subtree doesn't (currently) know how to deal with that.
Bah! Yes, directories have moved around in our svn repro. :-( In particular, we've had cases where libraries in boost began life as sub-projects of a different library and then got spun off.
Quoted 2 lines
git-filter-branch can do anything you want, but you have to teach it how, which is obviously even *more* error prone.
I can only imagine.
Quoted 3 lines
Things are also a little messy if you have some kind of top-level directory with build infrastructure shared by all the subdirs. Does the top-level
Makefilehave a list of the subdirs it needs to build?
Bah! Yes, the build, the docs and the test infrastructure all currently share files across our submodules-to-be. Surely other projects have encountered this problem before, right? (KDE, I'm looking in your direction.)
Quoted 6 lines
If so, there's no way to extract only a subset of true history that will still build correctly - it'll be looking for directories that you explicitly removed. You could update the Makefiles programmatically in every single revision, but that's starting to get extremely messy... and your history stops representing what *real life* really looked like at the time.
I see what you mean.
Quoted 4 lines
If your subdirs haven't been moving around (which sounds like that might be the case for you), and you don't have any top-level files that you care about, rewriting might turn out to be straightforward. You could also make the decision on a subdir-by-subdir basis, I guess.
More evidence that the fancy filter/branch/subtree/svn2git/whatever utilities aren't going to get us where we'd like to be. A simple conversion and grafts look like the only workable approach.
Quoted 1 line
Have fun,
Having heaps! Thanks,
Signature · 3 lines
Eric Niebler BoostPro Computing http://www.boostpro.com
Quoted 2 lines · Eric Niebler <eric <at> boostpro.com> writes:
We are aiming to make boost a clearing-house for C++ libraries (like CPAN, or PyPi for python),
Clarification: that's our goal for Ryppl, not Boost.
Quoted 3 lines
turning the official boost distribution into
little more than a well-tested collection of the libraries that have passed our peer-review and regression test process.
Exactly right.
--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
Quoted 6 lines · Dave Abrahams
Quoted 2 lines · Eric Niebler <eric <at> boostpro.com> writes:
We are aiming to make boost a clearing-house for C++ libraries (like CPAN, or PyPi for python),
Clarification: that's our goal for Ryppl, not Boost.
By the way, could you please add information about Ryppl to Git Wiki?
https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools
Thanks in advance
Signature · 3 lines
Jakub Narebski Poland ShadeHawk on #git
At Tue, 06 Jul 2010 01:51:00 -0700 (PDT),
Quoted 5 lines · Jakub Narebski
Quoted 2 lines · Dave Abrahams
Eric Niebler <eric <at> boostpro.com> writes:
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
We are aiming to make boost a clearing-house for C++ libraries (like CPAN, or PyPi for python),
Clarification: that's our goal for Ryppl, not Boost.
By the way, could you please add information about Ryppl to Git Wiki?
https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools
We'd be happy to, but: have you read the document at ryppl.org, and do you really think it's appropriate? Even though ryppl is still very alpha?
[BTW, I can't reach that page right now; it's just forever “waiting for reply...”]
Signature · 3 lines
Dave Abrahams BoostPro Computing http://www.boostpro.com
You have found the core issue with svn/git: svn allows you to have a large repo with everything (and atomic commits across it) and to have users check out parts of the repo separately. git does not, because the svn separate checkouts model only works with a remote repository that you don't keep a copy of. With git, cloning the repo gets you the whole thing.
One thought is that you may want to separate how you organize boost sources in git and how you release them. It's possible to have a single git repo for all libraries and have atomic commits but then create distfiles for each library separately.
git becomes a bit slow when repositories get really large (although other tools are not any better - the problem is in the sheer number of vnode ops necessary for the semantics). I have a repo with all of NetBSD's "src", "xsrc" and "pkgsrc" in it, and "git status" can take several seconds because it is calling stat on 230K files. With only 23K files, things should be ok.
My advice (which is not really about git) is to figure out whether you want:
A) a set of interrelated libraries on which you will allow atomic commits that change interfaces/usage in multiple libraries
or
B) a set of independent libaries which have commits to separate libraries, and for which you insist that each library have an API and ABI compatiblity story, so that even when upgraded other libraries can continue to use it.
For A, you probably want one git repo, much as you have one svn repo now. For B, multiple git repos are the right answer.
Quoted 7 lines · Greg Troxel
You have found the core issue with svn/git: svn allows you to have a large repo with everything (and atomic commits across it) and to have users check out parts of the repo separately. git does not, because the svn separate checkouts model only works with a remote repository that you don't keep a copy of. With git, cloning the repo gets you the whole thing.
Makes sense.
Quoted 6 lines
One thought is that you may want to separate how you organize boost sources in git and how you release them. It's possible to have a single git repo for all libraries and have atomic commits but then create distfiles for each library separately.
git becomes a bit slow when ...
<snip>
It can't get any worse than svn. We haven't run into any perf problems with git yet. That's not our primary concern.
Quoted 8 lines
My advice (which is not really about git) is to figure out whether you want:
A) a set of interrelated libraries on which you will allow atomic commits that change interfaces/usage in multiple libraries
or
Quoted 8 lines
B) a set of independent libaries which have commits to separate libraries, and for which you insist that each library have an API and ABI compatiblity story, so that even when upgraded other libraries can continue to use it.
For A, you probably want one git repo, much as you have one svn repo now. For B, multiple git repos are the right answer.
I'll take B FTW! :-) The idea is to open up and distribute C++ library development. Versioned dependency tracking will be handled at a higher level with per-project metadata and a tool (pip) that resolves dependencies. API compatibility is handled with peer review and regression testing. ABI compatibility is not an issue because we're distributing source code.
Signature · 3 lines
Eric Niebler BoostPro Computing http://www.boostpro.com