bzr-fastimport plugin, yet another Bazaar import option]
Quoted 6 lines · Ian Clatworthy
FYI. I thought you'd be interested in this as it's inspired by and based on git-fast-import. You can download the Python source from
https://code.launchpad.net/bzr-fastimport/. In particular, there's a Python parser of the stream format included that may be useful to the Git community or other VCS communities. The fast-import-info and fast-import-filter commands might also be useful to others.
This is interesting. I'm not a Python guy, but the info and filter commands do look like they could be useful beyond the Bazaar community.
Michael Haggerty of cvs2svn has spent a good amount of time creating a git-fast-import backend to cvs2svn. Given that cvs2svn is one of the few tools that can read some of the really strange real world CVS trees its good to be able to leverage that work for other systems (SVN, Git, and now Bazaar).
Quoted 6 lines
BTW, you might want to either extend the specification (a little) or fix git-fast-export so they match. :-) See
doc/notes.txtunder
http://bazaar.launchpad.net/~bzr/bzr-fastimport/fastimport.dev/files for details. For example, running git-fast-export on 64-bit Hardy Heron produces file modes longer than permitted if the spec was strictly interpreted.
We may need to take a small hammer to git-fast-export and fix its output. Generating long mode strings like your notes suggest is incorrect.
The fast-import format is very strict, to avoid any sort of ambiguous behavior and implicit data corruption during import. This is one reason we don't use "auto format detection" for dates and instead require that the frontend tell us what date format it is using, and stick to only that format thoughout the stream. Its also a reason why we only support a limited number of date formats.
File modes in Git are very limited. We really only want symlinks or regular files with permissions of 644 and 755. Everything else is bogus. We also now have the S_IFGITLINK mode to deal with but fast-import does not currently support it.
Quoted 8 lines
Looking forward, I'd probably like to extend the spec to support some Bazaar-specific features, e.g. versioning of directories without files inside them. If you have a preferred way of me doing this or would like to work on it together when that time comes, please let me know. To keep backwards compatibility, the first option that springs to mind is using specially marked comments for stuff like this, e.g.
##bzr:: blah blah blah
Technically its valid for a Git tree to contain an empty subdirectory, but that directory would disappear if the user tries to make a commit on top of it due to the current limitations of the index file.
So git-fast-import could actually allow the frontend to create an empty directory in the stream format, and record that correctly in Git. Its just that building on top of that may cause the directory to disappear. :)
If we are heading in the direction of making this a common stream format I'd like to try and work it out such that any additional extensions aren't VCS specific, at least as much as we can avoid it. That way exports from a source into this format can be loaded into any VCS that recognizes it, and have little or no loss. So yes, I am interested in trying to work with you and anyone else who wants to extend the format further.
Quoted 5 lines
Finally, thanks for writing git-fast-import and the associated documentation. It's well done. If you have any thoughts on the various front-ends available, I'm interested in hearing them. As well as saving me time vs testing lots of them, your thoughts will give us things to keep in mind when developing bzr-fastexport soon.
I think the fast-import documentation is the longest chunk of docs we have in git, at least as a single manpage. :-)
The git-p4 importer in the git.git contrib/ directory is probably the most well known and most widely used frontend. It is also an incremental tool, and I know a number of Qt developers use it to mirror the Qt Perforce tree into Git.
The t/t9300-fast-import.sh test script contains a number of tests for git-fast-import. The test cases themselves (in terms of the stream it feeds in) may be of some use to you as it covers most of the currently recognized stream format.
Quoted 12 lines
-------- Original Message --------
Subject: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option
Date: Fri, 22 Feb 2008 01:23:34 +1000
From: Ian Clatworthy <[email protected]>
To: Bazaar <[email protected]>, [email protected]I'm pleased to announce bzr-fastimport, a plugin useful for loading data exported by a large number of foreign VCS tools. Places to start are:
* the Launchpad page - https://launchpad.net/bzr-fastimport
* the Wiki page - http://bazaar-vcs.org/BzrFastImport.
Quoted 8 lines
Please note that this is not yet production quality but seems to be working well enough to be useful for a large number of projects.
I would *greatly* appreciate testing, feedback and improvements. In particular, I'm using this for migrating the OpenOffice.org repository (76K files and 500K revisions) into Bazaar from Subversion, so I'd really like some help with testing out and enhancing the existing Subversion front-ends.
Heh. OOo is _huge_. I think the best import into Git thus far is taking up about 1.5G of disk space once fully repacked. I don't recall how they did the import, but coming from SVN I think they used git-svn, which is not based on git-fast-import.
What frontend are you using to go from SVN -> fast-import?
Quoted 8 lines · Shawn O. Pearce
Quoted 6 lines · Ian Clatworthy
FYI. I thought you'd be interested in this as it's inspired by and based on git-fast-import. You can download the Python source from
https://code.launchpad.net/bzr-fastimport/. In particular, there's a Python parser of the stream format included that may be useful to the Git community or other VCS communities. The fast-import-info and fast-import-filter commands might also be useful to others.
Quoted 8 lines
This is interesting. I'm not a Python guy, but the info and filter commands do look like they could be useful beyond the Bazaar community.
Michael Haggerty of cvs2svn has spent a good amount of time creating a git-fast-import backend to cvs2svn. Given that cvs2svn is one of the few tools that can read some of the really strange real world CVS trees its good to be able to leverage that work for other systems (SVN, Git, and now Bazaar).
/me opens bigs ears and eyes: does this mean that we have an
incremental importer of CVS based on git-fast-import ? I mean I'm really interested into that, as git-cvsimport is really broken with the glibc CVS tree, and as the glibc CVSROOT is rsync-able, an incremental importer that has access to the CVSROOT RCS files is probably the most efficient way.
Is there any link you can provide to me about these new features of
cvs2svn ?
Signature · 3 lines
·O· Pierre Habouzit ··O [email protected] OOO http://www.madism.org
Quoted 6 lines · Pierre Habouzit
/me opens bigs ears and eyes: does this mean that we have an
incremental importer of CVS based on git-fast-import ? I mean I'm really interested into that, as git-cvsimport is really broken with the glibc CVS tree, and as the glibc CVSROOT is rsync-able, an incremental importer that has access to the CVSROOT RCS files is probably the most efficient way.
In the repository I convert (PostgreSQL), I'm using the ruby fromcvs/togit converter, which has worked well, because git-cvsimport doesn't work.
I actually found the problem with the PostgreSQL CVS repository - it is a TAG, which seems to have some cyclic dependencies which throws cvsps into a loop. Unfortuntely, I have neither time nor energy to be able to look into fixing cvsps, especially since fromcvs "just works" on it.
http://mid.gmane.org/[email protected]
I don't know what the problem with the glibc CVSROOT, but if it's the same, that might be something to look at.
Note that fromcvs doesn't import tags (that's probably why it didn't have any trouble with the PostgreSQL CVS) but that doesn't bother me, since CVS tags carry none of the authority git tags do, and the git commit ids provide a stable way to refer to particular commits anyways.
a.
Signature · 3 lines
Aidan Van Dyk Create like a god, [email protected] command like a king, http://www.highrise.ca/ work like a slave.
Quoted 9 lines · Aidan Van Dyk
Quoted 6 lines · Pierre Habouzit
/me opens bigs ears and eyes: does this mean that we have an
incremental importer of CVS based on git-fast-import ? I mean I'm really interested into that, as git-cvsimport is really broken with the glibc CVS tree, and as the glibc CVSROOT is rsync-able, an incremental importer that has access to the CVSROOT RCS files is probably the most efficient way.
Quoted 3 lines
In the repository I convert (PostgreSQL), I'm using the ruby fromcvs/togit converter, which has worked well, because git-cvsimport doesn't work.
Well, last time I tried, it exploded miserably (big fat OOM) because
glibc CVS repository comes back to 1984 or so, and has a very nasty big fat Changelog with literally thousands of modifications.
Signature · 3 lines
·O· Pierre Habouzit ··O [email protected] OOO http://www.madism.org
Quoted 3 lines · Pierre Habouzit
Well, last time I tried, it exploded miserably (big fat OOM) because
glibc CVS repository comes back to 1984 or so, and has a very nasty big fat Changelog with literally thousands of modifications.
<me run="prepare to">
When importing a repository like that wouldn't it make sense to nuke the redundant ChangeLog,v?
</me run="far away">
Signature · 3 lines
Aidan Van Dyk Create like a god, [email protected] command like a king, http://www.highrise.ca/ work like a slave.
Quoted 9 lines · Pierre Habouzit
Quoted 5 lines · Shawn O. Pearce
Michael Haggerty of cvs2svn has spent a good amount of time creating a git-fast-import backend to cvs2svn. Given that cvs2svn is one of the few tools that can read some of the really strange real world CVS trees its good to be able to leverage that work for other systems (SVN, Git, and now Bazaar).
/me opens bigs ears and eyes: does this mean that we have an
incremental importer of CVS based on git-fast-import ?
cvs2svn is robust and uses git-fast-import, but it is *not* incremental.
Incremental conversion would be fun but it would be a lot of work to
implement in such a way that it works reliably.
Michael
cvs2svn, was Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]
In reply to Pierre HabouzitHi,
Quoted 2 lines · Pierre Habouzit
Is there any link you can provide to me about these new features of
cvs2svn ?
http://article.gmane.org/gmane.comp.version-control.git/74461/match=cvs2svn
Hth,
Dscho
Quoted 7 lines · Shawn O. Pearce
If we are heading in the direction of making this a common stream format I'd like to try and work it out such that any additional extensions aren't VCS specific, at least as much as we can avoid it. That way exports from a source into this format can be loaded into any VCS that recognizes it, and have little or no loss. So yes, I am interested in trying to work with you and anyone else who wants to extend the format further.
Excellent. That sounds the right way to go. I'll contact you if and when I want to add stuff.
Quoted 6 lines
Heh. OOo is _huge_. I think the best import into Git thus far is taking up about 1.5G of disk space once fully repacked. I don't recall how they did the import, but coming from SVN I think they used git-svn, which is not based on git-fast-import.
What frontend are you using to go from SVN -> fast-import?
The pack file in the Git clone I have is 2.4G. I thought that was large but it's quite small compared to the 82G svn dump that creates a 55G svn repo!
I'm using svn-fast-export.c currently. I'd rather enhance the Python one but my Subversion binding knowledge is slim and there's a bug wrt "too many open files" that causes it to crash almost immediately on the OOo repo. It's not obvious to me how to fix that unfortunately. It worked fine for Wordpress OTOH.
The svn-all-fast-export tool sounds interesting but is completely undocumented to my knowledge.
Ian C.
Hi,
Quoted 7 lines · Ian Clatworthy
Quoted 5 lines · Shawn O. Pearce
Heh. OOo is _huge_. I think the best import into Git thus far is taking up about 1.5G of disk space once fully repacked. I don't recall how they did the import, but coming from SVN I think they used git-svn, which is not based on git-fast-import.
Quoted 5 lines
Quoted 1 line
What frontend are you using to go from SVN -> fast-import?
The pack file in the Git clone I have is 2.4G. I thought that was large but it's quite small compared to the 82G svn dump that creates a 55G svn repo!
The 2.4G have been compressed (loss-lessly ;-) to less than 1.5G. Unlike other SCMs, git has transparent access to the object database, which means that we can actually repack _expensively_ for a better compression.
So yes, the Git clone you have _is_ 2.4G, but that size is not the best size you _can_ have.
Ciao,
Dscho
Hi,
Quoted 9 lines · Shawn O. Pearce
Quoted 6 lines · Ian Clatworthy
BTW, you might want to either extend the specification (a little) or fix git-fast-export so they match. :-) See
doc/notes.txtunder
http://bazaar.launchpad.net/~bzr/bzr-fastimport/fastimport.dev/files
for details. For example, running git-fast-export on 64-bit Hardy Heron produces file modes longer than permitted if the spec was strictly interpreted.
Quoted 3 lines
We may need to take a small hammer to git-fast-export and fix its output. Generating long mode strings like your notes suggest is incorrect.
Indeed. Raising the issue with the original author of git-fast-export would not have hurt either.
Ciao,
Dscho