vcs-svn: prepare for (implement?) incremental import
cover vcs-svn: prepare for (implement?) incremental import
Hi,
Using David's "ls" command we can eliminate the in-memory repo_tree and rely on the target repository for information about old revs. This means all state that needs to persist between svn-fe runs is either in the target repo or in the marks file, so in theory this should allow incremental imports already (I haven't tried it).
Caveats:
Not split up as nicely as it ought to be. In particular, the last patch should be split up at least into one patch to modify git fast-import, another for the vcs-svn lib. Perhaps it could be split finer than that.
There is a potential deadlock, marked with NEEDSWORK.
Error checking is not so great. Error states tend to result in deadlock rather than a nicer error.
In particular, we have no way to distinguish between a missing (nonsense) path and an empty directory.
There's a little blind alley in the first few patches --- cat_mark and the function to apply deltas to an old rev do not survive to the end of the series. I kept that because (1) I am lazy and (2) it serves as a quick intro to use of the "ls" command.
Probably there are all sorts of bugs and unclear code. I haven't tested beyond running the test suite.
You can find the series alone at
git://repo.or.cz/git/jrn.git refs/topics/db/vcs-svn-incremental
and merged with other topics at
git://repo.or.cz/git/jrn.git vcs-svn-pu
Requires the db/text-delta and db/fast-import-blob-access topics, available from the same place.
Any thoughts would be welcome.
Jonathan Nieder (10):
vcs-svn: use higher mark numbers for blobs
vcs-svn: save marks for imported commits
vcs-svn: introduce cat_mark function to retrieve a marked blob
vcs-svn: make apply_delta caller retrieve preimage
vcs-svn: split off function to export result from delta application
vcs-svn: do not rely on marks for old blobs
vcs-svn: split off function to make 'ls' requests
vcs-svn: prepare to eliminate repo_tree structure
vcs-svn: simplifications for repo_modify_path et al
vcs-svn: eliminate repo_tree structure
cache.h | 2 + fast-import.c | 42 +++++-- t/t9010-svn-fe.sh | 55 +++++---- vcs-svn/fast_export.c | 168 +++++++++++++++++++------ vcs-svn/fast_export.h | 22 +++- vcs-svn/repo_tree.c | 333 ++++--------------------------------------------- vcs-svn/repo_tree.h | 6 +- vcs-svn/string_pool.c | 2 +- vcs-svn/string_pool.h | 2 +- vcs-svn/svndump.c | 89 +++++++++---- 10 files changed, 301 insertions(+), 420 deletions(-)
Signature · 1 line
1.7.2.4
PATCH 1/10 vcs-svn: use higher mark numbers for blobs
Prepare to use mark :5 for the commit corresponding to r5 (and so on).
1 billion seems sufficiently high for blob marks to avoid conflicting with rev marks, while still leaving room for 3 billion blobs. Such high mark numbers cause trouble with ancient fast-import versions, but this topic cannot support git fast-import versions before 1.7.4 (which introduces the cat-blob command) anyway.
--- vcs-svn/repo_tree.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
vcs-svn/repo_tree.c +1 -1
@@ -298,7 +298,7 @@ void repo_commit(uint32_t revision, uint32_t author, char *log, uint32_t uuid,static void mark_init (void ) {uint32_t i; - mark =0 ; + mark =1024 *1024 *1024 ;for (i =0 ; i < dent_pool.size; i++)if (!repo_dirent_is_dir (dent_pointer (i)) &&dent_pointer (i)->content_offset > mark)
Signature · 1 line
1.7.2.4
PATCH 2/10 vcs-svn: save marks for imported commits
In reply to Jonathan Nieder--- vcs-svn/fast_export.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
vcs-svn/fast_export.c +1 -0
@@ -61,6 +61,7 @@ void fast_export_commit(uint32_t revision, uint32_t author, char *log, *gitsvnline ='\0' ; }printf ("commit refs/heads/master\n" ); +printf ("mark :%" PRIu32"\n" , revision); printf("committer %s <%s@%s> %ld +0000\n", ~author ? pool_fetch(author) : "nobody", ~author ? pool_fetch(author) : "nobody",
Signature · 1 line
1.7.2.4
Re: [PATCH 02/10] vcs-svn: save marks for imported commits
(pruned cc because reviving an old thread)
Hi,
Quoted 3 lines · Jonathan Nieder
[Subject: vcs-svn: save marks for imported commits]
Signed-off-by Jonathan Nieder <[email protected]>
That's a lousy commit message. A version committed later (that eventually found its way to David's repository) explains:
This way, a person can use
svnadmin dump $path |
svn-fe |
git fast-import --relative-marks --export-marks=svn-revs
to get a list of what commit corresponds to each svn revision (plus some irrelevant blob names) in .git/info/fast-import/svn-revs.
In other words, this is the first half of a two-way mapping between svn and git commit names.
It should be possible to build a mapping the other way like so:
awk '
BEGIN {
print "commit refs/notes/svn-id";
printf "committer ";
system("git var GIT_COMMITTER_IDENT");
print "data <<EOT";
print "Automatically generated commits-to-revs mapping.";
print "EOT";
}
{
num = 0 + substr($1, 2);
commitname = $2;
if (num < 1024 * 1024 * 1024) {
print "N inline " commitname
print "data <<EOT";
print "r" num;
print "EOT";
}
}
' .git/info/fast-import/svn-revs |
git fast-importPATCH 3/10 vcs-svn: introduce cat_mark function to retrieve a marked blob
In reply to Jonathan Nieder---
A blind alley. But it demonstrates how this works.
vcs-svn/fast_export.c | 23 +++++++++++++++-------- 1 files changed, 15 insertions(+), 8 deletions(-)
vcs-svn/fast_export.c +15 -8
@@ -116,6 +116,19 @@ static const char *get_response_line(void)return response_line.buf; } +static off_t cat_mark (uint32_t mark) +{ +const char *response; +off_t length = length; + +printf ("cat-blob :%" PRIu32"\n" , mark); +fflush (stdout); + response =get_response_line (); +if (parse_cat_response_line (response, &length)) +die ("invalid cat-blob response: %s" , response); +return length; +} +static long apply_delta (uint32_t mark,off_t len,struct line_buffer *input,uint32_t old_mark,uint32_t old_mode) {
@@ -126,14 +139,8 @@ static long apply_delta(uint32_t mark, off_t len, struct line_buffer *input,if (init_postimage () || !(out =buffer_tmpfile_rewind (&postimage)))die ("cannot open temporary file for blob retrieval" ); -if (old_mark) { -const char *response; -printf ("cat-blob :%" PRIu32"\n" , old_mark); -fflush (stdout); - response =get_response_line (); -if (parse_cat_response_line (response, &preimage_len)) -die ("invalid cat-blob response: %s" , response); - } +if (old_mark) + preimage_len =cat_mark (old_mark);if (old_mode ==REPO_MODE_LNK ) {strbuf_addstr (&preimage.buf,"link " ); preimage_len +=strlen ("link " );
Signature · 1 line
1.7.2.4
PATCH 4/10 vcs-svn: make apply_delta caller retrieve preimage
In reply to Jonathan NiederThe preimage argument to apply_delta is currently a mark, but some callers might want to use a preimage named by sha1 or by revision number and path instead. Let the caller take care of that.
The preimage_len argument represents the length of the preimage that will appear followed by a newline in the REPORT_FD stream, or -1 to just use an empty preimage. apply_delta is renamed to delta_apply so callers that have not been updated can be detected at compile time.
---
The special-cased behavior of -1 won't need to survive.
vcs-svn/fast_export.c | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-)
vcs-svn/fast_export.c +8 -8
@@ -129,25 +129,23 @@ static off_t cat_mark(uint32_t mark)return length; } -static long apply_delta (uint32_t mark,off_t len,struct line_buffer *input, -uint32_t old_mark,uint32_t old_mode) +static long delta_apply (uint32_t mark,off_t len,struct line_buffer *input, +off_t preimage_len,uint32_t old_mode) {long ret; -off_t preimage_len =0 ;struct view preimage = {REPORT_FILENO ,0 ,STRBUF_INIT };FILE *out;if (init_postimage () || !(out =buffer_tmpfile_rewind (&postimage)))die ("cannot open temporary file for blob retrieval" ); -if (old_mark) - preimage_len =cat_mark (old_mark);if (old_mode ==REPO_MODE_LNK ) {strbuf_addstr (&preimage.buf,"link " ); - preimage_len +=strlen ("link " ); +if (preimage_len >=0 ) + preimage_len +=strlen ("link " ); }if (svndiff0_apply (input, len, &preimage, out))die ("cannot apply delta" ); -if (old_mark) { +if (preimage_len >=0 ) {/* Read the remainder of preimage and trailing newline. */ if (move_window (&preimage, preimage_len,1 ))die ("cannot seek to end of input" );
@@ -180,7 +178,9 @@ void fast_export_blob_delta(uint32_t mode, uint32_t mark,long postimage_len;if (len >maximum_signed_value_of_type (off_t))die ("enormous delta" ); - postimage_len =apply_delta (mark, (off_t ) len, input, old_mark, old_mode); + postimage_len =delta_apply (mark, (off_t ) len, input, + old_mark ?cat_mark (old_mark) :-1 , + old_mode);if (mode ==REPO_MODE_LNK ) {buffer_skip_bytes (&postimage,strlen ("link " )); postimage_len -=strlen ("link " );
Signature · 1 line
1.7.2.4
PATCH 5/10 vcs-svn: split off function to export result from delta application
In reply to Jonathan NiederIn this time of heavy experimentation, we will need multiple fast_export_blob_delta variants. Make this easier by factoring out the common part.
---
Also not needed, except perhaps as code cleanup.
vcs-svn/fast_export.c | 20 +++++++++++++------- 1 files changed, 13 insertions(+), 7 deletions(-)
vcs-svn/fast_export.c +13 -7
@@ -159,6 +159,18 @@ static long delta_apply(uint32_t mark, off_t len, struct line_buffer *input,return ret; } +static void record_postimage (uint32_t mark,uint32_t mode, +long postimage_len) +{ +if (mode ==REPO_MODE_LNK ) { +buffer_skip_bytes (&postimage,strlen ("link " )); + postimage_len -=strlen ("link " ); + } +printf ("blob\nmark :%" PRIu32"\ndata %ld\n" , mark, postimage_len); +buffer_copy_bytes (&postimage, postimage_len); +fputc ('\n' , stdout); +} +void fast_export_blob (uint32_t mode,uint32_t mark,uint32_t len,struct line_buffer *input) {if (mode ==REPO_MODE_LNK ) {
@@ -181,11 +193,5 @@ void fast_export_blob_delta(uint32_t mode, uint32_t mark, postimage_len =delta_apply (mark, (off_t ) len, input, old_mark ?cat_mark (old_mark) :-1 , old_mode); -if (mode ==REPO_MODE_LNK ) { -buffer_skip_bytes (&postimage,strlen ("link " )); - postimage_len -=strlen ("link " ); - } -printf ("blob\nmark :%" PRIu32"\ndata %ld\n" , mark, postimage_len); -buffer_copy_bytes (&postimage, postimage_len); -fputc ('\n' , stdout); +record_postimage (mark, mode, postimage_len); }
Signature · 1 line
1.7.2.4
PATCH 6/10 vcs-svn: do not rely on marks for old blobs
In reply to Jonathan NiederRetrieve old blobs by name and revision number from fast-import. One step closer to bounded memory usage in svn-fe.
---
Superfluous except that it shows how to parse 'ls' responses. A demo.
vcs-svn/fast_export.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++ vcs-svn/fast_export.h | 3 ++ vcs-svn/string_pool.c | 2 +- vcs-svn/string_pool.h | 2 +- vcs-svn/svndump.c | 6 +++++ 5 files changed, 64 insertions(+), 2 deletions(-)
vcs-svn/fast_export.c +53 -0
@@ -88,6 +88,21 @@ static int ends_with(const char *s, size_t len, const char *suffix)return !memcmp (s + len - suffixlen, suffix, suffixlen); } +static int parse_ls_response_line (const char *line,struct strbuf *objnam) +{ +const char *end = line +strlen (line); +const char *name, *tab; + +if (end - line <strlen ("100644 blob " )) +return error ("ls response too short: %s" , line); + name = line +strlen ("100644 blob " ); + tab =memchr (name,'\t' , end - name); +if (!tab) +return error ("ls response does not contain tab: %s" , line); +strbuf_add (objnam, name, tab - name); +return 0 ; +} +static int parse_cat_response_line (const char *header,off_t *len) {size_t headerlen =strlen (header);
@@ -129,6 +144,31 @@ static off_t cat_mark(uint32_t mark)return length; } +static off_t cat_from_rev (uint32_t rev,const uint32_t *path) +{ +const char *response; +off_t length = length; +struct strbuf blob_name =STRBUF_INIT ; + +/* ls :5 "path/to/old/file" */ +printf ("ls :%" PRIu32" \"" , rev); +pool_print_seq (REPO_MAX_PATH_DEPTH , path,'/' , stdout); +printf ("\"\n" ); +fflush (stdout); + + response =get_response_line (); +if (parse_ls_response_line (response, &blob_name)) +die ("invalid ls response: %s" , response); + +printf ("cat-blob %s\n" , blob_name.buf); +fflush (stdout); + response =get_response_line (); +if (parse_cat_response_line (response, &length)) +die ("invalid cat-blob response: %s" , response); +strbuf_release (&blob_name); +return length; +} +static long delta_apply (uint32_t mark,off_t len,struct line_buffer *input,off_t preimage_len,uint32_t old_mode) {
@@ -195,3 +235,16 @@ void fast_export_blob_delta(uint32_t mode, uint32_t mark, old_mode);record_postimage (mark, mode, postimage_len); } + +void fast_export_blob_delta_rev (uint32_t mode,uint32_t mark,uint32_t old_mode, +uint32_t old_rev,const uint32_t *old_path, +uint32_t len,struct line_buffer *input) +{ +long postimage_len; +if (len >maximum_signed_value_of_type (off_t)) +die ("enormous delta" ); + postimage_len =delta_apply (mark, (off_t ) len, input, +cat_from_rev (old_rev, old_path), + old_mode); +record_postimage (mark, mode, postimage_len); +}
vcs-svn/fast_export.h +3 -0
@@ -13,5 +13,8 @@ void fast_export_blob(uint32_t mode, uint32_t mark, uint32_t len,void fast_export_blob_delta (uint32_t mode,uint32_t mark,uint32_t old_mode,uint32_t old_mark,uint32_t len,struct line_buffer *input); +void fast_export_blob_delta_rev (uint32_t mode,uint32_t mark,uint32_t old_mode, +uint32_t old_rev,const uint32_t *old_path, +uint32_t len,struct line_buffer *input);#endif
vcs-svn/string_pool.c +1 -1
@@ -65,7 +65,7 @@ uint32_t pool_tok_r(char *str, const char *delim, char **saveptr)return token ?pool_intern (token) : ~0 ; } -void pool_print_seq (uint32_t len,uint32_t *seq,char delim,FILE *stream) +void pool_print_seq (uint32_t len,const uint32_t *seq,char delim,FILE *stream) {uint32_t i;for (i =0 ; i < len && ~seq[i]; i++) {
vcs-svn/string_pool.h +1 -1
@@ -4,7 +4,7 @@uint32_t pool_intern (const char *key);const char *pool_fetch (uint32_t entry);uint32_t pool_tok_r (char *str,const char *delim,char **saveptr); -void pool_print_seq (uint32_t len,uint32_t *seq,char delim,FILE *stream); +void pool_print_seq (uint32_t len,const uint32_t *seq,char delim,FILE *stream);uint32_t pool_tok_seq (uint32_t sz,uint32_t *seq,const char *delim,char *str);void pool_reset (void );
vcs-svn/svndump.c +6 -0
@@ -259,6 +259,12 @@ static void handle_node(void)fast_export_blob (node_ctx.type, mark, node_ctx.textLength, &input);return ; } +if (node_ctx.srcRev) { +fast_export_blob_delta_rev (node_ctx.type, mark, old_mode, + node_ctx.srcRev, node_ctx.src, + node_ctx.textLength, &input); +return ; + }fast_export_blob_delta (node_ctx.type, mark, old_mode, old_mark, node_ctx.textLength, &input); }
Signature · 1 line
1.7.2.4
PATCH 7/10 vcs-svn: split off function to make 'ls' requests
In reply to Jonathan Nieder---ls_from_rev will survive; cat_from_rev will not.
vcs-svn/fast_export.c | 16 ++++++++++------ 1 files changed, 10 insertions(+), 6 deletions(-)
vcs-svn/fast_export.c +10 -6
@@ -80,6 +80,15 @@ void fast_export_commit(uint32_t revision, uint32_t author, char *log,printf ("progress Imported commit %" PRIu32".\n\n" , revision); } +static void ls_from_rev (uint32_t rev,const uint32_t *path) +{ +/* ls :5 "path/to/old/file" */ +printf ("ls :%" PRIu32" \"" , rev); +pool_print_seq (REPO_MAX_PATH_DEPTH , path,'/' , stdout); +printf ("\"\n" ); +fflush (stdout); +} +static int ends_with (const char *s,size_t len,const char *suffix) {const size_t suffixlen =strlen (suffix);
@@ -150,12 +159,7 @@ static off_t cat_from_rev(uint32_t rev, const uint32_t *path)off_t length = length;struct strbuf blob_name =STRBUF_INIT ; -/* ls :5 "path/to/old/file" */ -printf ("ls :%" PRIu32" \"" , rev); -pool_print_seq (REPO_MAX_PATH_DEPTH , path,'/' , stdout); -printf ("\"\n" ); -fflush (stdout); - +ls_from_rev (rev, path); response =get_response_line ();if (parse_ls_response_line (response, &blob_name))die ("invalid ls response: %s" , response);
Signature · 1 line
1.7.2.4
PATCH 8/10 vcs-svn: prepare to eliminate repo_tree structure
In reply to Jonathan NiederCurrently svn-fe processes each commit in two stages: first decide on the correct content for all paths and export the relevant blobs, then export a commit with the result.
But we can keep less state and simplify svn-fe a great deal by doing exporting the commit in one stage: use 'inline' blobs for each path and remember nothing. This way, the repo_tree structure could be eliminated, and we would get support for incremental imports 'for free'.
Reorganize handle_node() along these lines. This is just a code cleanup; the functional change to repo_tree will come later.
--- vcs-svn/svndump.c | 32 +++++++++++++++++++++++--------- 1 files changed, 23 insertions(+), 9 deletions(-)
vcs-svn/svndump.c +23 -9
@@ -201,11 +201,12 @@ static void handle_node(void)uint32_t mark =0 , old_mode, old_mark;const uint32_t type = node_ctx.type;const int have_props = node_ctx.propLength !=LENGTH_UNKNOWN ; +const int have_text = node_ctx.textLength !=LENGTH_UNKNOWN ; -if (node_ctx.textLength !=LENGTH_UNKNOWN ) +if (have_text) mark =next_blob_mark ();if (node_ctx.action ==NODEACT_DELETE ) { -if (mark || have_props || node_ctx.srcRev) +if (have_text || have_props || node_ctx.srcRev)die ("invalid dump: deletion node has " "copyfrom info, text, or properties" );return repo_delete (node_ctx.dst);
@@ -219,8 +220,13 @@ static void handle_node(void)if (node_ctx.action ==NODEACT_ADD ) node_ctx.action =NODEACT_CHANGE ; } -if (mark && type ==REPO_MODE_DIR ) +if (have_text && type ==REPO_MODE_DIR )die ("invalid dump: directories cannot have text attached" ); + +/* +* Find old content (old_mark) and decide on the new content (mark) +* and mode (node_ctx.type). +*/ if (node_ctx.action ==NODEACT_CHANGE && !~*node_ctx.dst) {if (type !=REPO_MODE_DIR )die ("invalid dump: root of tree is not a regular file" );
@@ -228,7 +234,9 @@ static void handle_node(void) }else if (node_ctx.action ==NODEACT_CHANGE ) {uint32_t mode; old_mark =repo_read_path (node_ctx.dst); - mode =repo_modify_path (node_ctx.dst,0 , mark); +if (!have_text) + mark = old_mark; + mode =repo_modify_path (node_ctx.dst,0 ,0 );if (!mode)die ("invalid dump: path to be modified is missing" );if (mode ==REPO_MODE_DIR && type !=REPO_MODE_DIR )
@@ -237,23 +245,29 @@ static void handle_node(void)die ("invalid dump: cannot modify a file into a directory" ); node_ctx.type = mode; }else if (node_ctx.action ==NODEACT_ADD ) { -if (!mark && type !=REPO_MODE_DIR ) +if (!have_text && type !=REPO_MODE_DIR )die ("invalid dump: adds node without text" ); -repo_add (node_ctx.dst, type, mark); old_mark =0 ; }else {die ("invalid dump: Node-path block lacks Node-action" ); } + +/* +* Adjust mode to reflect properties. +*/ old_mode = node_ctx.type;if (have_props) {if (!node_ctx.prop_delta) node_ctx.type = type;if (node_ctx.propLength)read_props (); -if (node_ctx.type != old_mode) -repo_modify_path (node_ctx.dst, node_ctx.type, mark); } -if (!mark) + +/* +* Save the result. +*/ +repo_add (node_ctx.dst, node_ctx.type, mark); +if (!have_text)return ;if (!node_ctx.text_delta) {fast_export_blob (node_ctx.type, mark, node_ctx.textLength, &input);
Signature · 1 line
1.7.2.4
PATCH v2 1/1 vcs-svn: prepare to eliminate repo_tree structure
Date: Fri, 10 Dec 2010 04:28:06 -0600
Currently svn-fe processes each commit in two stages: first decide on the correct content for all paths and export the relevant blobs, then export a commit with the result.
But we can keep less state and simplify svn-fe a great deal by doing exporting the commit in one stage: use 'inline' blobs for each path and remember nothing. This way, the repo_tree structure could be eliminated, and we would get support for incremental imports 'for free'.
Reorganize handle_node() along these lines. This is just a code cleanup; the functional change to repo_tree will come later.
Backported by David Barr to apply without text delta support.
---
Hi,
Jonathan Nieder wrote:
[...]
Review static void handle_node(void) · line 239
Quoted 8 lines
(quoted diff) +3
-1@@ -228,7 +234,9 @@ static void handle_node(void) } else if (node_ctx.action == NODEACT_CHANGE) { uint32_t mode; old_mark = repo_read_path(node_ctx.dst); - mode = repo_modify_path(node_ctx.dst, 0, mark); + if (!have_text) + mark = old_mark; + mode = repo_modify_path(node_ctx.dst, 0, 0);
[...]
Backported so we can merge it a little sooner (thanks!). It's very similar to the original patch except there is no old_mark --- the previous text at a path is useless until we learn to patch it (which is a topic for another day).
David, may I have your sign-off on this series (the vcs-svn-incremental branch)?
vcs-svn/svndump.c | 33 +++++++++++++++++++++++---------- 1 files changed, 23 insertions(+), 10 deletions(-)
vcs-svn/svndump.c +23 -10
@@ -201,13 +201,14 @@ static void handle_node(void)uint32_t mark =0 ;const uint32_t type = node_ctx.type;const int have_props = node_ctx.propLength !=LENGTH_UNKNOWN ; +const int have_text = node_ctx.textLength !=LENGTH_UNKNOWN ;if (node_ctx.text_delta)die ("text deltas not supported" ); -if (node_ctx.textLength !=LENGTH_UNKNOWN ) +if (have_text) mark =next_blob_mark ();if (node_ctx.action ==NODEACT_DELETE ) { -if (mark || have_props || node_ctx.srcRev) +if (have_text || have_props || node_ctx.srcRev)die ("invalid dump: deletion node has " "copyfrom info, text, or properties" );return repo_delete (node_ctx.dst);
@@ -221,13 +222,20 @@ static void handle_node(void)if (node_ctx.action ==NODEACT_ADD ) node_ctx.action =NODEACT_CHANGE ; } -if (mark && type ==REPO_MODE_DIR ) +if (have_text && type ==REPO_MODE_DIR )die ("invalid dump: directories cannot have text attached" ); + +/* +* Decide on the new content (mark) and mode (node_ctx.type). +*/ if (node_ctx.action ==NODEACT_CHANGE && !~*node_ctx.dst) {if (type !=REPO_MODE_DIR )die ("invalid dump: root of tree is not a regular file" ); }else if (node_ctx.action ==NODEACT_CHANGE ) { -uint32_t mode =repo_modify_path (node_ctx.dst,0 , mark); +uint32_t mode; +if (!have_text) + mark =repo_read_path (node_ctx.dst); + mode =repo_modify_path (node_ctx.dst,0 ,0 );if (!mode)die ("invalid dump: path to be modified is missing" );if (mode ==REPO_MODE_DIR && type !=REPO_MODE_DIR )
@@ -236,22 +244,27 @@ static void handle_node(void)die ("invalid dump: cannot modify a file into a directory" ); node_ctx.type = mode; }else if (node_ctx.action ==NODEACT_ADD ) { -if (!mark && type !=REPO_MODE_DIR ) +if (!have_text && type !=REPO_MODE_DIR )die ("invalid dump: adds node without text" ); -repo_add (node_ctx.dst, type, mark); }else {die ("invalid dump: Node-path block lacks Node-action" ); } + +/* +* Adjust mode to reflect properties. +*/ if (have_props) { -const uint32_t old_mode = node_ctx.type;if (!node_ctx.prop_delta) node_ctx.type = type;if (node_ctx.propLength)read_props (); -if (node_ctx.type != old_mode) -repo_modify_path (node_ctx.dst, node_ctx.type, mark); } -if (mark) + +/* +* Save the result. +*/ +repo_add (node_ctx.dst, node_ctx.type, mark); +if (have_text)fast_export_blob (node_ctx.type, mark, node_ctx.textLength, &input); }
Signature · 1 line
1.7.4.1.91.g15e19.dirty
Re: [PATCH v2] vcs-svn: prepare to eliminate repo_tree structure
Hi,
Quoted 2 lines
David, may I have your sign-off on this series (the vcs-svn-incremental branch)?
Absolutely, I've updated my branch with the correct sign-offs.
Please pull
git://github.com/barrbrain/git.git vcs-svn-incremental
for the complete series of back-ported patches to support incremental imports for vcs-svn.
--
David Barr.
PATCH 9/10 vcs-svn: simplifications for repo_modify_path et al
In reply to Jonathan NiederRestrict repo_modify_path API to functions that are actually needed. That is:
- decouple reading the mode and content of dirents from other operations
- remove repo_modify_path. It was only used to read the mode from dirents.
- remove the ability to use repo_read_mode on a missing path. The existing code only errored out in that case, anyway.
--- vcs-svn/repo_tree.c | 25 +++++++++---------------- vcs-svn/repo_tree.h | 4 ++-- vcs-svn/svndump.c | 4 +--- 3 files changed, 12 insertions(+), 21 deletions(-)
vcs-svn/repo_tree.c +9 -16
@@ -166,7 +166,15 @@ uint32_t repo_read_path(uint32_t *path)return content_offset; } -uint32_t repo_copy (uint32_t revision,uint32_t *src,uint32_t *dst) +uint32_t repo_read_mode (const uint32_t *path) +{ +struct repo_dirent *dent =repo_read_dirent (active_commit, path); +if (dent ==NULL ) +die ("invalid dump: path to be modified is missing" ); +return dent->mode; +} + +void repo_copy (uint32_t revision,uint32_t *src,uint32_t *dst) {uint32_t mode =0 , content_offset =0 ;struct repo_dirent *src_dent;
@@ -176,7 +184,6 @@ uint32_t repo_copy(uint32_t revision, uint32_t *src, uint32_t *dst) content_offset = src_dent->content_offset;repo_write_dirent (dst, mode, content_offset,0 ); } -return mode; }void repo_add (uint32_t *path,uint32_t mode,uint32_t blob_mark)
@@ -184,20 +191,6 @@ void repo_add(uint32_t *path, uint32_t mode, uint32_t blob_mark)repo_write_dirent (path, mode, blob_mark,0 ); } -uint32_t repo_modify_path (uint32_t *path,uint32_t mode,uint32_t blob_mark) -{ -struct repo_dirent *src_dent; - src_dent =repo_read_dirent (active_commit, path); -if (!src_dent) -return 0 ; -if (!blob_mark) - blob_mark = src_dent->content_offset; -if (!mode) - mode = src_dent->mode; -repo_write_dirent (path, mode, blob_mark,0 ); -return mode; -} -void repo_delete (uint32_t *path) {repo_write_dirent (path,0 ,0 ,1 );
vcs-svn/repo_tree.h +2 -2
@@ -12,10 +12,10 @@#define REPO_MAX_PATH_DEPTH 1000uint32_t next_blob_mark (void ); -uint32_t repo_copy (uint32_t revision,uint32_t *src,uint32_t *dst); +void repo_copy (uint32_t revision,uint32_t *src,uint32_t *dst);void repo_add (uint32_t *path,uint32_t mode,uint32_t blob_mark); -uint32_t repo_modify_path (uint32_t *path,uint32_t mode,uint32_t blob_mark);uint32_t repo_read_path (uint32_t *path); +uint32_t repo_read_mode (const uint32_t *path);void repo_delete (uint32_t *path);void repo_commit (uint32_t revision,uint32_t author,char *log,uint32_t uuid,uint32_t url,long unsigned timestamp);
vcs-svn/svndump.c +1 -3
@@ -236,9 +236,7 @@ static void handle_node(void) old_mark =repo_read_path (node_ctx.dst);if (!have_text) mark = old_mark; - mode =repo_modify_path (node_ctx.dst,0 ,0 ); -if (!mode) -die ("invalid dump: path to be modified is missing" ); + mode =repo_read_mode (node_ctx.dst);if (mode ==REPO_MODE_DIR && type !=REPO_MODE_DIR )die ("invalid dump: cannot modify a directory into a file" );if (mode !=REPO_MODE_DIR && type ==REPO_MODE_DIR )
Signature · 1 line
1.7.2.4
PATCH 10/10 vcs-svn: eliminate repo_tree structure
In reply to Jonathan NiederRely on fast-import for information about previous revs.
This requires always setting up backward flow of information, even for v2 dumps. On the plus side, it simplifies the code by quite a bit and opens the door to further simplifications.
---
That's the end of the series. Sorry for its rough state; just thought it would be useful to get this out early so other tools can be built on it. No doubt it is terribly buggy and ugly. Reports of both sorts of problem would be greatly appreciated.
cache.h | 2 + fast-import.c | 42 +++++-- t/t9010-svn-fe.sh | 55 +++++---- vcs-svn/fast_export.c | 173 +++++++++++++++------------ vcs-svn/fast_export.h | 25 +++-- vcs-svn/repo_tree.c | 310 +++---------------------------------------------- vcs-svn/repo_tree.h | 2 +- vcs-svn/svndump.c | 73 +++++++----- 8 files changed, 239 insertions(+), 443 deletions(-)
cache.h +2 -0
@@ -678,6 +678,8 @@ static inline void hashclr(unsigned char *hash)#define EMPTY_TREE_SHA1_BIN \ "\x4b\x82\x5d\xc6\x42\xcb\x6e\xb9\xa0\x60" \ "\xe5\x4b\xf8\xd6\x92\x88\xfb\xee\x49\x04" +#define EMPTY_BLOB_SHA1_HEX \ + "e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"int git_mkstemp (char *path,size_t n,const char *template);
fast-import.c +29 -13
58 lines not on this page · raw message
t/t9010-svn-fe.sh +33 -22
147 lines not on this page · raw message
vcs-svn/fast_export.c +96 -77
257 lines not on this page · raw message
vcs-svn/fast_export.h +15 -10
@@ -3,18 +3,23 @@#include "line_buffer.h" +/* Output routines */ void fast_export_delete (uint32_t depth,uint32_t *path); -void fast_export_modify (uint32_t depth,uint32_t *path,uint32_t mode, -uint32_t mark); -void fast_export_commit (uint32_t revision,uint32_t author,char *log, +void fast_export_modify (uint32_t depth,const uint32_t *path,uint32_t mode, +const char *dataref); +void fast_export_begin_commit (uint32_t revision,uint32_t author,char *log,uint32_t uuid,uint32_t url,unsigned long timestamp); -void fast_export_blob (uint32_t mode,uint32_t mark,uint32_t len, -struct line_buffer *input); -void fast_export_blob_delta (uint32_t mode,uint32_t mark, -uint32_t old_mode,uint32_t old_mark, -uint32_t len,struct line_buffer *input); -void fast_export_blob_delta_rev (uint32_t mode,uint32_t mark,uint32_t old_mode, -uint32_t old_rev,const uint32_t *old_path, +void fast_export_end_commit (uint32_t revision); +void fast_export_empty_blob (void ); +void fast_export_data (uint32_t mode,uint32_t len,struct line_buffer *input); +void fast_export_delta (const uint32_t *path,uint32_t mode,uint32_t old_mode, +const char *dataref,uint32_t len,struct line_buffer *input); +/* Input routines */ +void fast_export_ls_rev (uint32_t rev,const uint32_t *path, +uint32_t *mode_out,struct strbuf *dataref_out); +void fast_export_ls (const uint32_t *path, +uint32_t *mode_out,struct strbuf *dataref_out); +#endif
vcs-svn/repo_tree.c +19 -291
340 lines not on this page · raw message
vcs-svn/repo_tree.h +1 -1
@@ -14,7 +14,7 @@uint32_t next_blob_mark (void );void repo_copy (uint32_t revision,uint32_t *src,uint32_t *dst);void repo_add (uint32_t *path,uint32_t mode,uint32_t blob_mark); -uint32_t repo_read_path (uint32_t *path); +const char *repo_read_path (uint32_t *path);uint32_t repo_read_mode (const uint32_t *path);void repo_delete (uint32_t *path); void repo_commit(uint32_t revision, uint32_t author, char *log, uint32_t uuid,
vcs-svn/svndump.c +44 -29
155 lines not on this page · raw message
Signature · 1 line
1.7.2.4
[PATCH 13/10] vcs-svn: use mark from previous import for parent commit
In reply to Jonathan NiederFrom: David Barr <[email protected]>
Date: Sun, 12 Dec 2010 13:41:38 +1100
Subject: [PATCH] vcs-svn: use mark from previous import for parent commit
--- vcs-svn/fast_export.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
vcs-svn/fast_export.c +1 -1
@@ -77,7 +77,7 @@ void fast_export_begin_commit(uint32_t revision, uint32_t author, char *log, log, gitsvnline);if (!first_commit_done) {if (revision >1 ) -printf ("from refs/heads/master^0\n" ); +printf ("from :%" PRIu32"\n" , revision -1 ); first_commit_done =1 ; } }
Signature · 1 line
1.7.3.2.846.gf4b062
Re: [PATCH 13/10] vcs-svn: use mark from previous import for parent commit
Quoted 1 line · David Michael Barr
Subject: [PATCH] vcs-svn: use mark from previous import for parent commit
[...]
vcs-svn/fast_export.c void fast_export_begin_commit(uint32_t revision, uint32_t · line 80
Quoted 7 lines
+++
b/vcs-svn/fast_export.cvcs-svn/fast_export.c +1
-1@@ -77,7 +77,7 @@ void fast_export_begin_commit(uint32_t revision, uint32_t log, gitsvnline);if (!first_commit_done) {if (revision >1 ) -printf ("from refs/heads/master^0\n" ); +printf ("from :%" PRIu32"\n" , revision -1 );
This deals more sanely with attempts to continue an import starting at the wrong revision.
Example: if I try
import () {
rm -f backflow
mkfifo backflow
svn-fe 3<backflow |
git fast-import --cat-blob-fd=3 \
--relative-marks \
${1+--import-marks=svnrevs} \
--export-marks=svnrevs \
3>backflow
}
svnrdump -r0:100 $url | import
svnrdump -r100:200 $url | import continuethen svn-fe should correctly re-import r100 the second time, instead of trying to apply the same deltas twice. If I try
svnrdump -r0:100 $url | import svnrdump -r102:200 $url | import continue
then the second command should error out.
Thanks, queued.
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): 20101211184654.GA17464@burratino
[RFC/PATCH] fast-import: treat filemodify with empty tree as delete
Quoted 3 lines · Jonathan Nieder
Maybe fast-import should make the check itself, to avoid writing trees with empty directories that would be hard to re-create with "git write-tree".
Like this, maybe?
-- 8< --
Subject: fast-import: treat filemodify with empty tree as delete
Traditionally, git trees do not contain entries for empty subdirectories. Generally speaking, subtrees are not created or destroyed explicitly; instead, they automatically appear when needed to hold regular files, symlinks, and submodules.
v1.7.3-rc0~75^2 (Teach fast-import to import subtrees named by tree id, 2010-06-30) changed that, by allowing an empty subtree to be included in a fast-import stream explicitly:
M 040000 4b825dc642cb6eb9a060e54bf8d69288fbee4904 subdir
That was unintentional. Better and more closely analogous to "git read-tree --prefix" to treat such an input line as a request to delete ("to empty") subdir.
---
Tests use the "ls" command from vcs-svn-pu. The actual change would apply cleanly to master or maint, though.
fast-import.c | 10 ++++ t/t9300-fast-import.sh | 107 ++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 109 insertions(+), 8 deletions(-)
fast-import.c +10 -0
@@ -2196,6 +2196,16 @@ static void file_change_m(struct branch *b) p = uq.buf; } +/* +* Git does not track empty, non-toplevel directories. +*/ +if (S_ISDIR (mode) && + !memcmp (sha1, (unsigned char *)EMPTY_TREE_SHA1_BIN ,20 ) && + *p) { +tree_content_remove (&b->branch_tree, p,NULL ); +return ; + } +if (S_ISGITLINK (mode)) {if (inline_data) die("Git links cannot be specified 'inline': %s",
t/t9300-fast-import.sh +99 -8
125 lines not on this page · raw message
Signature · 1 line
1.7.2.4
PATCH 1/1 vcs-svn: avoid git-isms in fast-import stream
In reply to a message not in the archiveQuoted 9 lines · Jonathan Nieder
I am not totally happy with the result, since it adds another git-specific detail to svn-fe (the first was hardcoding the empty blob sha1 instead of using
blob mark :0 data 0).
Maybe this would help?
-- 8< --
Subject: vcs-svn: avoid git-isms in fast-import stream
Current svn-fe is not likely to work without change with other fast-import backends, but don't let that stop us from trying:
- instead of suppressing copies of empty trees, let the backend decide what to do with them;
- use a mark instead of hard-coding git's name for the empty blob.
However, we do not include commands in the stream for new empty directories, since no syntax is documented for that yet.
--- vcs-svn/fast_export.c | 8 +------- vcs-svn/svndump.c | 2 +- 2 files changed, 2 insertions(+), 8 deletions(-)
vcs-svn/fast_export.c +1 -7
@@ -40,12 +40,6 @@ void fast_export_delete(uint32_t depth, const uint32_t *path)void fast_export_modify (uint32_t depth,const uint32_t *path,uint32_t mode,const char *dataref) { -/* Git does not track empty directories. */ -if (S_ISDIR (mode) && !strcmp (dataref,EMPTY_TREE_SHA1_HEX )) { -fast_export_delete (depth, path); -return ; - } -/* Mode must be 100644, 100755, 120000, or 160000. */ printf ("M %06" PRIo32" %s " , mode, dataref);pool_print_seq (depth, path,'/' , stdout);
@@ -255,7 +249,7 @@ void fast_export_data(uint32_t mode, uint32_t len, struct line_buffer *input)void fast_export_empty_blob (void ) { -printf ("blob\ndata 0\n\n" ); +printf ("blob\nmark :0\ndata 0\n\n" ); } void fast_export_delta(const uint32_t *path, uint32_t mode, uint32_t old_mode,
vcs-svn/svndump.c +1 -1
@@ -244,7 +244,7 @@ static void handle_node(void)if (type ==REPO_MODE_DIR ) old_data =NULL ;else if (have_text) - old_data =EMPTY_BLOB_SHA1_HEX ; + old_data =":0" ;else die ("invalid dump: adds node without text" ); }else {
Signature · 1 line
1.7.2.4
[PATCH 12/10] vcs-svn: quote paths correctly for ls command
In reply to Jonathan NiederFrom: David Barr <[email protected]>
Date: Sun, 12 Dec 2010 03:59:31 +1100
Subject: [PATCH] vcs-svn: quote paths correctly for ls command
This bug was found while importing rev 601865 of ASF.
--- vcs-svn/fast_export.c | 4 ++-- vcs-svn/string_pool.c | 11 +++++++++++ vcs-svn/string_pool.h | 1 + 3 files changed, 14 insertions(+), 2 deletions(-)
vcs-svn/fast_export.c +2 -2
@@ -100,7 +100,7 @@ static void ls_from_rev(uint32_t rev, const uint32_t *path) {/* ls :5 "path/to/old/file" */ printf ("ls :%" PRIu32" \"" , rev); -pool_print_seq (REPO_MAX_PATH_DEPTH , path,'/' , stdout); +pool_print_seq_q (REPO_MAX_PATH_DEPTH , path,'/' , stdout);printf ("\"\n" );fflush (stdout); }
@@ -149,7 +149,7 @@ void fast_export_ls(const uint32_t *path, uint32_t *mode, struct strbuf *dataref) { printf("ls \""); -pool_print_seq (REPO_MAX_PATH_DEPTH , path,'/' , stdout); +pool_print_seq_q (REPO_MAX_PATH_DEPTH , path,'/' , stdout);printf ("\"\n" );fflush (stdout);parse_ls_response (get_response_line (), mode, dataref);
vcs-svn/string_pool.c +11 -0
@@ -4,6 +4,8 @@*/ #include "git-compat-util.h" +#include "strbuf.h" +#include "quote.h" #include "trp.h" #include "obj_pool.h" #include "string_pool.h"
@@ -75,6 +77,15 @@ void pool_print_seq(uint32_t len, const uint32_t *seq, char delim, FILE *stream) } } +void pool_print_seq_q (uint32_t len,const uint32_t *seq,char delim,FILE *stream) +{ +uint32_t i; +for (i =0 ; i < len && ~seq[i]; i++) { +quote_c_style (pool_fetch (seq[i]),NULL , stream,1 ); +if (i < len -1 && ~seq[i +1 ]) +fputc (delim, stream); + } +}uint32_t pool_tok_seq (uint32_t sz,uint32_t *seq,const char *delim,char *str) {char *context =NULL ;
vcs-svn/string_pool.h +1 -0
@@ -5,6 +5,7 @@ uint32_t pool_intern(const char *key);const char *pool_fetch (uint32_t entry);uint32_t pool_tok_r (char *str,const char *delim,char **saveptr);void pool_print_seq (uint32_t len,const uint32_t *seq,char delim,FILE *stream); +void pool_print_seq_q (uint32_t len,const uint32_t *seq,char delim,FILE *stream);uint32_t pool_tok_seq (uint32_t sz,uint32_t *seq,const char *delim,char *str);void pool_reset (void );
Signature · 1 line
1.7.3.2.846.gf4b062
PATCH 1/1 vcs-svn: quote all paths passed to fast-import
Filenames with linefeeds or double quotes need to be quoted if fast-import is not to misinterpret them.
--- vcs-svn/fast_export.c | 13 ++++++------- 1 files changed, 6 insertions(+), 7 deletions(-)
vcs-svn/fast_export.c +6 -7
@@ -31,19 +31,18 @@ static int init_postimage(void)void fast_export_delete (uint32_t depth,const uint32_t *path) { -putchar ('D' ); -putchar (' ' ); -pool_print_seq (depth, path,'/' , stdout); -putchar ('\n' ); +printf ("D \"" ); +pool_print_seq_q (depth, path,'/' , stdout); +printf ("\"\n" ); }void fast_export_modify (uint32_t depth,const uint32_t *path,uint32_t mode,const char *dataref) {/* Mode must be 100644, 100755, 120000, or 160000. */ -printf ("M %06" PRIo32" %s " , mode, dataref); -pool_print_seq (depth, path,'/' , stdout); -putchar ('\n' ); +printf ("M %06" PRIo32" %s \"" , mode, dataref); +pool_print_seq_q (depth, path,'/' , stdout); +printf ("\"\n" ); }static char gitsvnline[MAX_GITSVN_LINE_LEN ];
Signature · 1 line
1.7.2.4
v2 cover vcs-svn: incremental import
In reply to Jonathan NiederHi again,
Quoted 2 lines · Jonathan Nieder
Using David's "ls" command we can eliminate the in-memory
repo_treeand rely on the target repository for information about old revs.
Here's a reroll. Aside from the aspects already mentioned (which avoid a dependency on the mostly orthogonal topic of support for text deltas), the original patch #10 has been split into smaller, more easily digestible pieces.
Most of the credit for this incarnation of the series belongs to David, who heroically streamlined it and untangled it from other topics.
Patch 1 changes the mark numbers for blobs to be ridiculously high, to make room for memorable commit marks (:1 for r1, :2 for r2, etc). Patch 2 brings those commit marks into existence, as mentioned before.
Patches 3-5 simplify the repo-tree API somewhat. They are somewhat minimal; patches on top of this series offering further simplification would be very welcome.
Patch 6 is a bit sneaky. We want svn-fe's output to change from
<import blob> <import blob> ... <import blob> <import commit, using blobs>
to
<commit header> M 100644 inline one/path <import blob> M 100644 inline another/path ... <commit footer (progress update)>
since the latter allows svn-fe to maintain much less state. But that's a big change, so patch 6 introduces a stepping stone on the way there:
<comment that will become commit header> <import blob>
...
<import commit; this will become the commit footer>
That paves the way for patches 7-11, which teach svn-fe to rely on the fast-import backend for information about previously imported blobs, at long last.
The visible effects should be:
- svn-fe _requires_ a backchannel from the fast-import backend now. You can't do
svn-fe <dump >stream && fast-import <stream
in two steps any more.
- Given one dump that picks up where another left off, svn-fe can continue the import. Use
git fast-import --relative-marks \ --import-marks-if-exists=svn-revs \ --export-marks=svn-revs \ --cat-blob-fd=3 3>backchannel
for both imports.
I'm not happy about the loss of usability but I'm happy about the gain in functionality. A good next step might be to build a simple remote helper to make this comfortable to use.
Thoughts? Improvements? Complaints? Despite the deficiencies just mentioned I'm tempted to push this out soon. Feedback in either direction would be welcome.
David Barr (3):
vcs-svn: set up channel to read fast-import cat-blob response
vcs-svn: quote paths correctly for ls command
vcs-svn: use mark from previous import for parent commit
Jonathan Nieder (9):
vcs-svn: use higher mark numbers for blobs
vcs-svn: save marks for imported commits
vcs-svn: introduce repo_read_path to check the content at a path
vcs-svn: handle_node: use repo_read_path
vcs-svn: simplify repo_modify_path and repo_copy
vcs-svn: add a comment before each commit
vcs-svn: allow input errors to be detected promptly
vcs-svn: eliminate repo_tree structure
vcs-svn: handle filenames with dq correctly
contrib/svn-fe/svn-fe.txt | 6 +- t/t9010-svn-fe.sh | 217 +++++++++++++++++++------ vcs-svn/fast_export.c | 145 +++++++++++++++-- vcs-svn/fast_export.h | 39 +++-- vcs-svn/line_buffer.c | 5 + vcs-svn/line_buffer.h | 1 + vcs-svn/repo_tree.c | 386 ++++++++------------------------------------- vcs-svn/repo_tree.h | 5 +- vcs-svn/string_pool.c | 13 ++- vcs-svn/string_pool.h | 3 +- vcs-svn/svndump.c | 106 +++++++++---- 11 files changed, 490 insertions(+), 436 deletions(-)
rewrite vcs-svn/fast_export.h (75%)
rewrite vcs-svn/repo_tree.c (96%)
PATCH v2 1/12 vcs-svn: use higher mark numbers for blobs
Date: Fri, 10 Dec 2010 04:21:35 -0600
Prepare to use mark :5 for the commit corresponding to r5 (and so on).
1 billion seems sufficiently high for blob marks to avoid conflicting with rev marks, while still leaving room for 3 billion blobs. Such high mark numbers cause trouble with ancient fast-import versions, but this topic cannot support git fast-import versions before 1.7.4 (which introduces the cat-blob command) anyway.
---
Unchanged except for a tiny tweak in the working of the change description.
vcs-svn/repo_tree.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
vcs-svn/repo_tree.c +1 -1
@@ -289,7 +289,7 @@ void repo_commit(uint32_t revision, uint32_t author, char *log, uint32_t uuid,static void mark_init (void ) {uint32_t i; - mark =0 ; + mark =1024 *1024 *1024 ;for (i =0 ; i < dent_pool.size; i++)if (!repo_dirent_is_dir (dent_pointer (i)) &&dent_pointer (i)->content_offset > mark)
Signature · 1 line
1.7.4.1
Re: [PATCH 01/12] vcs-svn: use higher mark numbers for blobs
Quoted 9 lines · Jonathan Nieder
Date: Fri, 10 Dec 2010 04:21:35 -0600
Prepare to use mark :5 for the commit corresponding to r5 (and so on).
1 billion seems sufficiently high for blob marks to avoid conflicting with rev marks, while still leaving room for 3 billion blobs. Such high mark numbers cause trouble with ancient fast-import versions, but this topic cannot support git fast-import versions before 1.7.4 (which introduces the cat-blob command) anyway.
Hmm, 1G+3G split? Will we have HIGHMEM option someday? ;-)
How confident are you that you will never need more than two classes later and you will never need to split the larger space again?
If you are not, and if the topic is to introduce incompatible output, would it be wiser to be even more forward looking and introduce different classes of marks with a backward incompatible syntax, perhaps like using ":\d+" for anything, and using ":[a-zA-Z0-9]+:\d+" for some application specific "class" of objects that is specifed by the [a-zA-Z0-9]+ part?
Re: [PATCH 01/12] vcs-svn: use higher mark numbers for blobs
Hi,
Quoted 10 lines · Junio C Hamano
Hmm, 1G+3G split? Will we have HIGHMEM option someday? ;-)
How confident are you that you will never need more than two classes later and you will never need to split the larger space again?
If you are not, and if the topic is to introduce incompatible output, would it be wiser to be even more forward looking and introduce different classes of marks with a backward incompatible syntax, perhaps like using ":\d+" for anything, and using ":[a-zA-Z0-9]+:\d+" for some application specific "class" of objects that is specifed by the [a-zA-Z0-9]+ part?
That sounds very sensible (and I'd be happy to see something like that).
In this particular case a later patch ("vcs-svn: eliminate repo_tree structure") gets rid of the blob marks so the split is temporary. Perhaps a paragraph added to the change description would clear it up.
A later patch will eliminate the blob marks altogether.
For the "vcs-svn: eliminate repo_tree" patch:
Rely on fast-import for information about previous revs.
This requires always setting up backward flow of information, even for v2 dumps. On the plus side:
- No more need to include blobs in the marks table.
- Given one dump that picks up where another left off, svn-fe can continue the import. Use
git fast-import --relative-marks \
--export-marks=svn-revs \
--cat-blob-fd=3 3>backchannel
for the first import and
git fast-import --relative-marks \
--import-marks=svn-revs \
--export-marks=svn-revs \
--cat-blob-fd=3 3>backchannel
for later ones.
- It simplifies the code by quite a bit and opens the door to further simplifications.
Thanks for some clarity.
Jonathan
PATCH v2 2/12 vcs-svn: save marks for imported commits
In reply to Jonathan NiederDate: Thu, 9 Dec 2010 18:57:13 -0600
This way, a person can use
svnadmin dump $path | svn-fe | git fast-import --relative-marks --export-marks=svn-revs
to get a list of what commit corresponds to each svn revision (plus some irrelevant blob names) in .git/info/fast-import/svn-revs.
---
From David's tree.
vcs-svn/fast_export.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
vcs-svn/fast_export.c +1 -0
@@ -45,6 +45,7 @@ void fast_export_commit(uint32_t revision, uint32_t author, char *log, *gitsvnline ='\0' ; }printf ("commit refs/heads/master\n" ); +printf ("mark :%" PRIu32"\n" , revision); printf("committer %s <%s@%s> %ld +0000\n", ~author ? pool_fetch(author) : "nobody", ~author ? pool_fetch(author) : "nobody",
Signature · 1 line
1.7.4.1
PATCH v2 3/12 vcs-svn: introduce repo_read_path to check the content at a path
In reply to Jonathan NiederDate: Sat, 20 Nov 2010 13:25:28 -0600
The repo_tree structure remembers, for each path in each revision, a mode (regular file, executable, symlink, or directory) and content (blob mark or directory structure). Maintaining a second copy of all this information when it's already in the target repository is wasteful, it does not persist between svn-fe invocations, and most importantly, there is no convenient way to transfer it from one machine to another. So it would be nice to get rid of it.
As a first step, let's change the repo_tree API to match fast-import's read commands more closely. Currently to read the mode for a path, one uses
repo_modify_path(path, new_mode, new_content);
which changes the mode and content as a side effect. There is no function to read the content at a path; add one.
--- vcs-svn/repo_tree.c | 12 +++++++++++- vcs-svn/repo_tree.h | 1 + 2 files changed, 12 insertions(+), 1 deletions(-)
vcs-svn/repo_tree.c +11 -1
@@ -87,7 +87,8 @@ static struct repo_dir *repo_clone_dir(struct repo_dir *orig_dir)return dir_pointer (new_o); } -static struct repo_dirent *repo_read_dirent (uint32_t revision,uint32_t *path) +static struct repo_dirent *repo_read_dirent (uint32_t revision, +const uint32_t *path) {uint32_t name =0 ;struct repo_dirent *key =dent_pointer (dent_alloc (1 ));
@@ -157,6 +158,15 @@ static void repo_write_dirent(uint32_t *path, uint32_t mode,dent_remove (&dir_pointer (parent_dir_o)->entries, dent); } +uint32_t repo_read_path (const uint32_t *path) +{ +uint32_t content_offset =0 ; +struct repo_dirent *dent =repo_read_dirent (active_commit, path); +if (dent !=NULL ) + content_offset = dent->content_offset; +return content_offset; +} +uint32_t repo_copy (uint32_t revision,uint32_t *src,uint32_t *dst) {uint32_t mode =0 , content_offset =0 ;
vcs-svn/repo_tree.h +1 -0
@@ -15,6 +15,7 @@ uint32_t next_blob_mark(void);uint32_t repo_copy (uint32_t revision,uint32_t *src,uint32_t *dst);void repo_add (uint32_t *path,uint32_t mode,uint32_t blob_mark);uint32_t repo_modify_path (uint32_t *path,uint32_t mode,uint32_t blob_mark); +uint32_t repo_read_path (const uint32_t *path);void repo_delete (uint32_t *path);void repo_commit (uint32_t revision,uint32_t author,char *log,uint32_t uuid,uint32_t url,long unsigned timestamp);
Signature · 1 line
1.7.4.1
PATCH v2 4/12 vcs-svn: handle_node: use repo_read_path
In reply to Jonathan NiederDate: Fri, 10 Dec 2010 04:28:06 -0600
svn-fe processes each commit in two stages: first decide on the correct content for all paths and export the relevant blobs, then export a commit with the result.
We can keep less state and simplify svn-fe a great deal by doing exporting the commit in one stage: use 'inline' blobs for each path and remember nothing. This way, the repo_tree structure could be eliminated, and we would get support for incremental imports 'for free'.
Reorganize handle_node() along these lines. This is just a code cleanup; the functional changes to repo_tree and handle_revision will come later.
[db: backported to apply without text delta support]
--- vcs-svn/svndump.c | 33 +++++++++++++++++++++++---------- 1 files changed, 23 insertions(+), 10 deletions(-)
vcs-svn/svndump.c +23 -10
@@ -201,13 +201,14 @@ static void handle_node(void)uint32_t mark =0 ;const uint32_t type = node_ctx.type;const int have_props = node_ctx.propLength !=LENGTH_UNKNOWN ; +const int have_text = node_ctx.textLength !=LENGTH_UNKNOWN ;if (node_ctx.text_delta)die ("text deltas not supported" ); -if (node_ctx.textLength !=LENGTH_UNKNOWN ) +if (have_text) mark =next_blob_mark ();if (node_ctx.action ==NODEACT_DELETE ) { -if (mark || have_props || node_ctx.srcRev) +if (have_text || have_props || node_ctx.srcRev)die ("invalid dump: deletion node has " "copyfrom info, text, or properties" );return repo_delete (node_ctx.dst);
@@ -221,13 +222,20 @@ static void handle_node(void)if (node_ctx.action ==NODEACT_ADD ) node_ctx.action =NODEACT_CHANGE ; } -if (mark && type ==REPO_MODE_DIR ) +if (have_text && type ==REPO_MODE_DIR )die ("invalid dump: directories cannot have text attached" ); + +/* +* Decide on the new content (mark) and mode (node_ctx.type). +*/ if (node_ctx.action ==NODEACT_CHANGE && !~*node_ctx.dst) {if (type !=REPO_MODE_DIR )die ("invalid dump: root of tree is not a regular file" ); }else if (node_ctx.action ==NODEACT_CHANGE ) { -uint32_t mode =repo_modify_path (node_ctx.dst,0 , mark); +uint32_t mode; +if (!have_text) + mark =repo_read_path (node_ctx.dst); + mode =repo_modify_path (node_ctx.dst,0 ,0 );if (!mode)die ("invalid dump: path to be modified is missing" );if (mode ==REPO_MODE_DIR && type !=REPO_MODE_DIR )
@@ -236,22 +244,27 @@ static void handle_node(void)die ("invalid dump: cannot modify a file into a directory" ); node_ctx.type = mode; }else if (node_ctx.action ==NODEACT_ADD ) { -if (!mark && type !=REPO_MODE_DIR ) +if (!have_text && type !=REPO_MODE_DIR )die ("invalid dump: adds node without text" ); -repo_add (node_ctx.dst, type, mark); }else {die ("invalid dump: Node-path block lacks Node-action" ); } + +/* +* Adjust mode to reflect properties. +*/ if (have_props) { -const uint32_t old_mode = node_ctx.type;if (!node_ctx.prop_delta) node_ctx.type = type;if (node_ctx.propLength)read_props (); -if (node_ctx.type != old_mode) -repo_modify_path (node_ctx.dst, node_ctx.type, mark); } -if (mark) + +/* +* Save the result. +*/ +repo_add (node_ctx.dst, node_ctx.type, mark); +if (have_text)fast_export_blob (node_ctx.type, mark, node_ctx.textLength, &input); }
Signature · 1 line
1.7.4.1
PATCH v2 5/12 vcs-svn: simplify repo_modify_path and repo_copy
In reply to Jonathan NiederDate: Fri, 10 Dec 2010 00:53:54 -0600
Restrict the repo_tree API to functions that are actually needed.
- decouple reading the mode and content of dirents from other operations.
- remove repo_modify_path. It is only used to read the mode from dirents.
- remove the ability to use repo_read_mode on a missing path. The existing code only errors out in that case, anyway.
--- vcs-svn/repo_tree.c | 27 ++++++++++----------------- vcs-svn/repo_tree.h | 4 ++-- vcs-svn/svndump.c | 4 +--- 3 files changed, 13 insertions(+), 22 deletions(-)
vcs-svn/repo_tree.c +10 -17
@@ -106,7 +106,7 @@ static struct repo_dirent *repo_read_dirent(uint32_t revision,return dent; } -static void repo_write_dirent (uint32_t *path,uint32_t mode, +static void repo_write_dirent (const uint32_t *path,uint32_t mode,uint32_t content_offset,uint32_t del) {uint32_t name, revision, dir_o = ~0 , parent_dir_o = ~0 ;
@@ -167,7 +167,15 @@ uint32_t repo_read_path(const uint32_t *path)return content_offset; } -uint32_t repo_copy (uint32_t revision,uint32_t *src,uint32_t *dst) +uint32_t repo_read_mode (const uint32_t *path) +{ +struct repo_dirent *dent =repo_read_dirent (active_commit, path); +if (dent ==NULL ) +die ("invalid dump: path to be modified is missing" ); +return dent->mode; +} + +void repo_copy (uint32_t revision,const uint32_t *src,const uint32_t *dst) {uint32_t mode =0 , content_offset =0 ;struct repo_dirent *src_dent;
@@ -177,7 +185,6 @@ uint32_t repo_copy(uint32_t revision, uint32_t *src, uint32_t *dst) content_offset = src_dent->content_offset;repo_write_dirent (dst, mode, content_offset,0 ); } -return mode; }void repo_add (uint32_t *path,uint32_t mode,uint32_t blob_mark)
@@ -185,20 +192,6 @@ void repo_add(uint32_t *path, uint32_t mode, uint32_t blob_mark)repo_write_dirent (path, mode, blob_mark,0 ); } -uint32_t repo_modify_path (uint32_t *path,uint32_t mode,uint32_t blob_mark) -{ -struct repo_dirent *src_dent; - src_dent =repo_read_dirent (active_commit, path); -if (!src_dent) -return 0 ; -if (!blob_mark) - blob_mark = src_dent->content_offset; -if (!mode) - mode = src_dent->mode; -repo_write_dirent (path, mode, blob_mark,0 ); -return mode; -} -void repo_delete (uint32_t *path) {repo_write_dirent (path,0 ,0 ,1 );
vcs-svn/repo_tree.h +2 -2
@@ -12,10 +12,10 @@#define REPO_MAX_PATH_DEPTH 1000uint32_t next_blob_mark (void ); -uint32_t repo_copy (uint32_t revision,uint32_t *src,uint32_t *dst); +void repo_copy (uint32_t revision,const uint32_t *src,const uint32_t *dst);void repo_add (uint32_t *path,uint32_t mode,uint32_t blob_mark); -uint32_t repo_modify_path (uint32_t *path,uint32_t mode,uint32_t blob_mark);uint32_t repo_read_path (const uint32_t *path); +uint32_t repo_read_mode (const uint32_t *path);void repo_delete (uint32_t *path);void repo_commit (uint32_t revision,uint32_t author,char *log,uint32_t uuid,uint32_t url,long unsigned timestamp);
vcs-svn/svndump.c +1 -3
@@ -235,9 +235,7 @@ static void handle_node(void)uint32_t mode;if (!have_text) mark =repo_read_path (node_ctx.dst); - mode =repo_modify_path (node_ctx.dst,0 ,0 ); -if (!mode) -die ("invalid dump: path to be modified is missing" ); + mode =repo_read_mode (node_ctx.dst);if (mode ==REPO_MODE_DIR && type !=REPO_MODE_DIR )die ("invalid dump: cannot modify a directory into a file" );if (mode !=REPO_MODE_DIR && type ==REPO_MODE_DIR )
Signature · 1 line
1.7.4.1
PATCH v2 6/12 vcs-svn: add a comment before each commit
In reply to Jonathan NiederDate: Tue, 4 Jan 2011 21:53:33 -0600
Current svn-fe produces output like this:
blob mark :7382321 data 5 hello blob mark :7382322 data 5 Hello commit mark :3 [...] M 100644 :7382321 hello.c M 100644 :7382322 hello2.c
This means svn-fe has to keep track of the paths modified in each commit and the corresponding marks, instead of dealing with each file as it arrives in input and then forgetting about it. A better strategy would be to use inline blobs:
commit mark :3 [...] M 100644 inline hello.c data 5 hello
[...]
As a first step towards that, teach svn-fe to notice when the collection of blobs for each commit starts and write a comment ("# commit 3.") there.
--- vcs-svn/fast_export.c | 5 +++++ vcs-svn/fast_export.h | 1 + vcs-svn/svndump.c | 29 ++++++++++++++++++++++------- 3 files changed, 28 insertions(+), 7 deletions(-)
vcs-svn/fast_export.c +5 -0
@@ -30,6 +30,11 @@ void fast_export_modify(uint32_t depth, uint32_t *path, uint32_t mode,putchar ('\n' ); } +void fast_export_begin_commit (uint32_t revision) +{ +printf ("# commit %" PRIu32".\n" , revision); +} +static char gitsvnline[MAX_GITSVN_LINE_LEN ]; void fast_export_commit(uint32_t revision, uint32_t author, char *log, uint32_t uuid, uint32_t url,
vcs-svn/fast_export.h +1 -0
@@ -6,6 +6,7 @@void fast_export_delete (uint32_t depth,uint32_t *path);void fast_export_modify (uint32_t depth,uint32_t *path,uint32_t mode,uint32_t mark); +void fast_export_begin_commit (uint32_t revision);void fast_export_commit (uint32_t revision,uint32_t author,char *log,uint32_t uuid,uint32_t url,unsigned long timestamp); void fast_export_blob(uint32_t mode, uint32_t mark, uint32_t len,
vcs-svn/svndump.c +22 -7
@@ -20,9 +20,11 @@#define NODEACT_CHANGE 1#define NODEACT_UNKNOWN 0 -#define DUMP_CTX 0 -#define REV_CTX 1 -#define NODE_CTX 2 +/* States: */ +#define DUMP_CTX 0/* dump metadata */ +#define REV_CTX 1/* revision metadata */ +#define NODE_CTX 2/* node metadata */ +#define INTERNODE_CTX 3/* between nodes */ #define LENGTH_UNKNOWN (~0)#define DATE_RFC2822_LEN 31
@@ -267,7 +269,14 @@ static void handle_node(void) node_ctx.textLength, &input); } -static void handle_revision (void ) +static void begin_revision (void ) +{ +if (!rev_ctx.revision)/* revision 0 gets no git commit. */ +return ; +fast_export_begin_commit (rev_ctx.revision); +} + +static void end_revision (void ) {if (rev_ctx.revision) repo_commit(rev_ctx.revision, rev_ctx.author, rev_ctx.log,
@@ -301,13 +310,17 @@ void svndump_read(const char *url) }else if (key == keys.revision_number) {if (active_ctx ==NODE_CTX )handle_node (); +if (active_ctx ==REV_CTX ) +begin_revision ();if (active_ctx !=DUMP_CTX ) -handle_revision (); +end_revision (); active_ctx =REV_CTX ;reset_rev_ctx (atoi (val)); }else if (key == keys.node_path) {if (active_ctx ==NODE_CTX )handle_node (); +if (active_ctx ==REV_CTX ) +begin_revision (); active_ctx =NODE_CTX ;reset_node_ctx (val); }else if (key == keys.node_kind) {
@@ -349,7 +362,7 @@ void svndump_read(const char *url)read_props (); }else if (active_ctx ==NODE_CTX ) {handle_node (); - active_ctx =REV_CTX ; + active_ctx =INTERNODE_CTX ; }else {fprintf (stderr,"Unexpected content length header: %" PRIu32"\n" , len);buffer_skip_bytes (&input, len);
@@ -358,8 +371,10 @@ void svndump_read(const char *url) }if (active_ctx ==NODE_CTX )handle_node (); +if (active_ctx ==REV_CTX ) +begin_revision ();if (active_ctx !=DUMP_CTX ) -handle_revision (); +end_revision (); }int svndump_init (const char *filename)
Signature · 1 line
1.7.4.1
PATCH v2 7/12 vcs-svn: allow input errors to be detected promptly
In reply to Jonathan NiederDate: Sun, 10 Oct 2010 21:51:21 -0500
The line_buffer library silently flags input errors until buffer_deinit time; unfortunately, by that point usually errno is invalid. Expose the error flag so callers can check for and report errors early for easy debugging.
some_error_prone_operation(...);
if (buffer_ferror(buf))
return error("input error: %s", strerror(errno));--- vcs-svn/line_buffer.c | 5 +++++ vcs-svn/line_buffer.h | 1 + 2 files changed, 6 insertions(+), 0 deletions(-)
vcs-svn/line_buffer.c +5 -0
@@ -59,6 +59,11 @@ long buffer_tmpfile_prepare_to_read(struct line_buffer *buf)return pos; } +int buffer_ferror (struct line_buffer *buf) +{ +return ferror (buf->infile); +} +int buffer_read_char (struct line_buffer *buf) {return fgetc (buf->infile);
vcs-svn/line_buffer.h +1 -0
@@ -21,6 +21,7 @@ int buffer_tmpfile_init(struct line_buffer *buf);FILE *buffer_tmpfile_rewind (struct line_buffer *buf);/* prepare to write. */ long buffer_tmpfile_prepare_to_read (struct line_buffer *buf); +int buffer_ferror (struct line_buffer *buf);char *buffer_read_line (struct line_buffer *buf);char *buffer_read_string (struct line_buffer *buf,uint32_t len);int buffer_read_char (struct line_buffer *buf);
Signature · 1 line
1.7.4.1
PATCH v2 8/12 vcs-svn: set up channel to read fast-import cat-blob response
In reply to Jonathan NiederFrom: David Barr <[email protected]>
Date: Sat, 5 Mar 2011 13:30:23 +1100
Set up some plumbing: teach the svndump lib to pass a file descriptor number to the fast_export lib, representing where cat-blob/ls responses can be read from, and add a get_response_line helper function to the fast_export lib to read a line from that file.
Unfortunately this means that svn-fe needs file descriptor 3 to be redirected from somewhere (preferrably the cat-blob stream of a fast-import backend); otherwise it will fail:
$ svndump <path> | svn-fe fatal: cannot read from file descriptor 3: Bad file descriptor
For the moment, "svn-fe 3</dev/null" works as a workaround but it will not work for very long. A fast-import backend that can retrieve old commits is needed in order to be able to fulfill svn "Node-copyfrom-rev" requests that refer to revs from a previous run.
[jn: with new change description]
--- contrib/svn-fe/svn-fe.txt | 6 ++- t/t9010-svn-fe.sh | 118 +++++++++++++++++++++++++------------------- vcs-svn/fast_export.c | 28 +++++++++++ vcs-svn/fast_export.h | 4 ++ vcs-svn/svndump.c | 5 ++ 5 files changed, 109 insertions(+), 52 deletions(-)
contrib/svn-fe/svn-fe.txt +5 -1
@@ -7,7 +7,11 @@ svn-fe - convert an SVN "dumpfile" to a fast-import stream SYNOPSIS -------- -svnadmin dump --incremental REPO | svn-fe [url] | git fast-import +[verse] +mkfifo backchannel && +svnadmin dump --incremental REPO | + svn-fe [url] 3<backchannel | + git fast-import --cat-blob-fd=3 3>backchannel DESCRIPTION -----------
t/t9010-svn-fe.sh +67 -51
297 lines not on this page · raw message
vcs-svn/fast_export.c +28 -0
@@ -12,6 +12,24 @@#define MAX_GITSVN_LINE_LEN 4096static uint32_t first_commit_done; +static struct line_buffer report_buffer =LINE_BUFFER_INIT ; + +void fast_export_init (int fd) +{ +if (buffer_fdinit (&report_buffer, fd)) +die_errno ("cannot read from file descriptor %d" , fd); +} + +void fast_export_deinit (void ) +{ +if (buffer_deinit (&report_buffer)) +die_errno ("error closing fast-import feedback stream" ); +} + +void fast_export_reset (void ) +{ +buffer_reset (&report_buffer); +}void fast_export_delete (uint32_t depth,uint32_t *path) {
@@ -69,6 +87,16 @@ void fast_export_commit(uint32_t revision, uint32_t author, char *log,printf ("progress Imported commit %" PRIu32".\n\n" , revision); } +static const char *get_response_line (void ) +{ +const char *line =buffer_read_line (&report_buffer); +if (line) +return line; +if (buffer_ferror (&report_buffer)) +die_errno ("error reading from fast-import" ); +die ("unexpected end of fast-import feedback" ); +} +void fast_export_blob (uint32_t mode,uint32_t mark,uint32_t len,struct line_buffer *input) {if (mode ==REPO_MODE_LNK ) {
vcs-svn/fast_export.h +4 -0
@@ -3,6 +3,10 @@#include "line_buffer.h" +void fast_export_init (int fd); +void fast_export_deinit (void ); +void fast_export_reset (void ); +void fast_export_delete (uint32_t depth,uint32_t *path);void fast_export_modify (uint32_t depth,uint32_t *path,uint32_t mode,uint32_t mark);
vcs-svn/svndump.c +5 -0
@@ -14,6 +14,8 @@#include "obj_pool.h" #include "string_pool.h" +#define REPORT_FILENO 3 +#define NODEACT_REPLACE 4#define NODEACT_DELETE 3#define NODEACT_ADD 2
@@ -382,6 +384,7 @@ int svndump_init(const char *filename)if (buffer_init (&input, filename))return error ("cannot open %s: %s" , filename,strerror (errno));repo_init (); +fast_export_init (REPORT_FILENO );reset_dump_ctx (~0 );reset_rev_ctx (0 );reset_node_ctx (NULL );
@@ -392,6 +395,7 @@ int svndump_init(const char *filename)void svndump_deinit (void ) {log_reset (); +fast_export_deinit ();repo_reset ();reset_dump_ctx (~0 );reset_rev_ctx (0 );
@@ -405,6 +409,7 @@ void svndump_deinit(void)void svndump_reset (void ) {log_reset (); +fast_export_reset ();buffer_reset (&input);repo_reset ();reset_dump_ctx (~0 );
Signature · 1 line
1.7.4.1
PATCH v2 9/12 vcs-svn: eliminate repo_tree structure
In reply to Jonathan NiederDate: Fri, 10 Dec 2010 04:00:55 -0600
Rely on fast-import for information about previous revs.
This requires always setting up backward flow of information, even for v2 dumps. On the plus side, it simplifies the code by quite a bit and opens the door to further simplifications.
[db: adjusted to support final version of the cat-blob patch] [jn: avoiding hard-coding git's name for the empty tree for
portability to other backends]
--- vcs-svn/fast_export.c | 108 ++++++++++++-- vcs-svn/fast_export.h | 44 +++--- vcs-svn/repo_tree.c | 389 ++++++++----------------------------------------- vcs-svn/repo_tree.h | 2 +- vcs-svn/string_pool.c | 2 +- vcs-svn/string_pool.h | 2 +- vcs-svn/svndump.c | 53 +++++-- 7 files changed, 222 insertions(+), 378 deletions(-)
rewrite vcs-svn/fast_export.h (65%)
rewrite vcs-svn/repo_tree.c (95%)
vcs-svn/fast_export.c +94 -14
@@ -8,6 +8,7 @@#include "line_buffer.h" #include "repo_tree.h" #include "string_pool.h" +#include "strbuf.h" #define MAX_GITSVN_LINE_LEN 4096
@@ -31,7 +32,7 @@ void fast_export_reset(void)buffer_reset (&report_buffer); } -void fast_export_delete (uint32_t depth,uint32_t *path) +void fast_export_delete (uint32_t depth,const uint32_t *path) {putchar ('D' );putchar (' ' );
@@ -39,22 +40,27 @@ void fast_export_delete(uint32_t depth, uint32_t *path)putchar ('\n' ); } -void fast_export_modify (uint32_t depth,uint32_t *path,uint32_t mode, -uint32_t mark) +static void fast_export_truncate (uint32_t depth,const uint32_t *path,uint32_t mode) +{ +fast_export_modify (depth, path, mode,"inline" ); +printf ("data 0\n\n" ); +} + +void fast_export_modify (uint32_t depth,const uint32_t *path,uint32_t mode, +const char *dataref) {/* Mode must be 100644, 100755, 120000, or 160000. */ -printf ("M %06" PRIo32" :%" PRIu32" " , mode, mark); +if (!dataref) { +fast_export_truncate (depth, path, mode); +return ; + } +printf ("M %06" PRIo32" %s " , mode, dataref);pool_print_seq (depth, path,'/' , stdout);putchar ('\n' ); } -void fast_export_begin_commit (uint32_t revision) -{ -printf ("# commit %" PRIu32".\n" , revision); -} -static char gitsvnline[MAX_GITSVN_LINE_LEN ]; -void fast_export_commit (uint32_t revision,uint32_t author,char *log, +void fast_export_begin_commit (uint32_t revision,uint32_t author,char *log,uint32_t uuid,uint32_t url,unsigned long timestamp) {
@@ -81,12 +87,31 @@ void fast_export_commit(uint32_t revision, uint32_t author, char *log,printf ("from refs/heads/master^0\n" ); first_commit_done =1 ; } -repo_diff (revision -1 , revision); -fputc ('\n' , stdout); +} +void fast_export_end_commit (uint32_t revision) +{printf ("progress Imported commit %" PRIu32".\n\n" , revision); } +static void ls_from_rev (uint32_t rev,uint32_t depth,const uint32_t *path) +{ +/* ls :5 path/to/old/file */ +printf ("ls :%" PRIu32" " , rev); +pool_print_seq (depth, path,'/' , stdout); +putchar ('\n' ); +fflush (stdout); +} + +static void ls_from_active_commit (uint32_t depth,const uint32_t *path) +{ +/* ls "path/to/file" */ +printf ("ls \"" ); +pool_print_seq (depth, path,'/' , stdout); +printf ("\"\n" ); +fflush (stdout); +} +static const char *get_response_line (void ) {const char *line =buffer_read_line (&report_buffer);
@@ -97,14 +122,69 @@ static const char *get_response_line(void)die ("unexpected end of fast-import feedback" ); } -void fast_export_blob (uint32_t mode,uint32_t mark,uint32_t len,struct line_buffer *input) +void fast_export_data (uint32_t mode,uint32_t len,struct line_buffer *input) {if (mode ==REPO_MODE_LNK ) {/* svn symlink blobs start with "link " */ buffer_skip_bytes (input,5 ); len -=5 ; } -printf ("blob\nmark :%" PRIu32"\ndata %" PRIu32"\n" , mark, len); +printf ("data %" PRIu32"\n" , len);buffer_copy_bytes (input, len);fputc ('\n' , stdout); } + +static int parse_ls_response (const char *response,uint32_t *mode, +struct strbuf *dataref) +{ +const char *tab; +const char *response_end; + +assert (response); + response_end = response +strlen (response); + +if (*response =='m' ) {/* Missing. */ + errno =ENOENT ; +return -1 ; + } + +/* Mode. */ +if (response_end - response <strlen ("100644" ) || + response[strlen ("100644" )] !=' ' ) +die ("invalid ls response: missing mode: %s" , response); + *mode =0 ; +for (; *response !=' ' ; response++) { +char ch = *response; +if (ch <'0' || ch >'7' ) +die ("invalid ls response: mode is not octal: %s" , response); + *mode *=8 ; + *mode += ch -'0' ; + } + +/* ' blob ' or ' tree ' */ +if (response_end - response <strlen (" blob " ) || + (response[1 ] !='b' && response[1 ] !='t' )) +die ("unexpected ls response: not a tree or blob: %s" , response); + response +=strlen (" blob " ); + +/* Dataref. */ + tab =memchr (response,'\t' , response_end - response); +if (!tab) +die ("invalid ls response: missing tab: %s" , response); +strbuf_add (dataref, response, tab - response); +return 0 ; +} + +int fast_export_ls_rev (uint32_t rev,uint32_t depth,const uint32_t *path, +uint32_t *mode,struct strbuf *dataref) +{ +ls_from_rev (rev, depth, path); +return parse_ls_response (get_response_line (), mode, dataref); +} + +int fast_export_ls (uint32_t depth,const uint32_t *path, +uint32_t *mode,struct strbuf *dataref) +{ +ls_from_active_commit (depth, path); +return parse_ls_response (get_response_line (), mode, dataref); +}
vcs-svn/fast_export.h +25 -19
44 lines not on this page · raw message
vcs-svn/repo_tree.c +64 -325
389 lines not on this page · raw message
vcs-svn/repo_tree.h +1 -1
@@ -14,7 +14,7 @@uint32_t next_blob_mark (void );void repo_copy (uint32_t revision,const uint32_t *src,const uint32_t *dst);void repo_add (uint32_t *path,uint32_t mode,uint32_t blob_mark); -uint32_t repo_read_path (const uint32_t *path); +const char *repo_read_path (const uint32_t *path);uint32_t repo_read_mode (const uint32_t *path);void repo_delete (uint32_t *path); void repo_commit(uint32_t revision, uint32_t author, char *log, uint32_t uuid,
vcs-svn/string_pool.c +1 -1
@@ -65,7 +65,7 @@ uint32_t pool_tok_r(char *str, const char *delim, char **saveptr)return token ?pool_intern (token) : ~0 ; } -void pool_print_seq (uint32_t len,uint32_t *seq,char delim,FILE *stream) +void pool_print_seq (uint32_t len,const uint32_t *seq,char delim,FILE *stream) {uint32_t i;for (i =0 ; i < len && ~seq[i]; i++) {
vcs-svn/string_pool.h +1 -1
@@ -4,7 +4,7 @@uint32_t pool_intern (const char *key);const char *pool_fetch (uint32_t entry);uint32_t pool_tok_r (char *str,const char *delim,char **saveptr); -void pool_print_seq (uint32_t len,uint32_t *seq,char delim,FILE *stream); +void pool_print_seq (uint32_t len,const uint32_t *seq,char delim,FILE *stream);uint32_t pool_tok_seq (uint32_t sz,uint32_t *seq,const char *delim,char *str);void pool_reset (void );
vcs-svn/svndump.c +36 -17
124 lines not on this page · raw message
Signature · 1 line
1.7.4.1
PATCH v2 10/12 vcs-svn: quote paths correctly for ls command
In reply to Jonathan NiederFrom: David Barr <[email protected]>
Date: Sun, 12 Dec 2010 03:59:31 +1100
This bug was found while importing rev 601865 of ASF.
[jn: with test]
--- t/t9010-svn-fe.sh | 99 +++++++++++++++++++++++++++++++++++++++++++++++++ vcs-svn/fast_export.c | 2 +- vcs-svn/string_pool.c | 11 +++++ vcs-svn/string_pool.h | 1 + 4 files changed, 112 insertions(+), 1 deletions(-)
t/t9010-svn-fe.sh +99 -0
@@ -270,6 +270,105 @@ test_expect_success PIPE 'directory with files' ' test_cmp hi directory/file2 ' +test_expect_success PIPE 'branch name with backslash' ' + reinit_git && + sort <<-\EOF >expect.branch-files && + trunk/file1 + trunk/file2 + "branches/UpdateFOPto094\\/file1" + "branches/UpdateFOPto094\\/file2" + EOF + + echo hi >hi && + echo hello >hello && + { + properties \ + svn:author [email protected] \ + svn:date "1999-02-02T00:01:02.000000Z" \ + svn:log "add directory with some files in it" && + echo PROPS-END + } >props.setup && + { + properties \ + svn:author [email protected] \ + svn:date "2007-12-06T21:38:34.000000Z" \ + svn:log "Updating fop to .94 and adjust fo-stylesheets" && + echo PROPS-END + } >props.branch && + { + cat <<-EOF && + SVN-fs-dump-format-version: 3 + + Revision-number: 1 + EOF + echo Prop-content-length: $(wc -c <props.setup) && + echo Content-length: $(wc -c <props.setup) && + echo && + cat props.setup && + cat <<-\EOF && + + Node-path: trunk + Node-kind: dir + Node-action: add + Prop-content-length: 10 + Content-length: 10 + + PROPS-END + + Node-path: branches + Node-kind: dir + Node-action: add + Prop-content-length: 10 + Content-length: 10 + + PROPS-END + + Node-path: trunk/file1 + Node-kind: file + Node-action: add + EOF + text_no_props hello && + cat <<-\EOF && + Node-path: trunk/file2 + Node-kind: file + Node-action: add + EOF + text_no_props hi && + cat <<-\EOF && + + Revision-number: 2 + EOF + echo Prop-content-length: $(wc -c <props.branch) && + echo Content-length: $(wc -c <props.branch) && + echo && + cat props.branch && + cat <<-\EOF + + Node-path: branches/UpdateFOPto094\ + Node-kind: dir + Node-action: add + Node-copyfrom-rev: 1 + Node-copyfrom-path: trunk + + Node-kind: dir + Node-action: add + Prop-content-length: 34 + Content-length: 34 + + K 13 + svn:mergeinfo + V 0 + + PROPS-END + EOF + } >branch.dump && + try_dump branch.dump && + + git ls-tree -r --name-only HEAD | + sort >actual.branch-files && + test_cmp expect.branch-files actual.branch-files +' + test_expect_success PIPE 'node without action' ' reinit_git && cat >inaction.dump <<-\EOF &&
vcs-svn/fast_export.c +1 -1
@@ -107,7 +107,7 @@ static void ls_from_active_commit(uint32_t depth, const uint32_t *path) {/* ls "path/to/file" */ printf ("ls \"" ); -pool_print_seq (depth, path,'/' , stdout); +pool_print_seq_q (depth, path,'/' , stdout);printf ("\"\n" );fflush (stdout); }
vcs-svn/string_pool.c +11 -0
@@ -4,6 +4,7 @@*/ #include "git-compat-util.h" +#include "quote.h" #include "trp.h" #include "obj_pool.h" #include "string_pool.h"
@@ -75,6 +76,16 @@ void pool_print_seq(uint32_t len, const uint32_t *seq, char delim, FILE *stream) } } +void pool_print_seq_q (uint32_t len,const uint32_t *seq,char delim,FILE *stream) +{ +uint32_t i; +for (i =0 ; i < len && ~seq[i]; i++) { +quote_c_style (pool_fetch (seq[i]),NULL , stream,1 ); +if (i < len -1 && ~seq[i +1 ]) +fputc (delim, stream); + } +} +uint32_t pool_tok_seq (uint32_t sz,uint32_t *seq,const char *delim,char *str) {char *context =NULL ;
vcs-svn/string_pool.h +1 -0
@@ -5,6 +5,7 @@ uint32_t pool_intern(const char *key);const char *pool_fetch (uint32_t entry);uint32_t pool_tok_r (char *str,const char *delim,char **saveptr);void pool_print_seq (uint32_t len,const uint32_t *seq,char delim,FILE *stream); +void pool_print_seq_q (uint32_t len,const uint32_t *seq,char delim,FILE *stream);uint32_t pool_tok_seq (uint32_t sz,uint32_t *seq,const char *delim,char *str);void pool_reset (void );
Signature · 1 line
1.7.4.1
PATCH v2 11/12 vcs-svn: handle filenames with dq correctly
In reply to Jonathan NiederDate: Sat, 11 Dec 2010 17:08:51 -0600
Quote paths passed to fast-import so filenames with double quotes are not misinterpreted.
One might imagine this could help with filenames with newlines, too, but svn does not allow those.
--- vcs-svn/fast_export.c | 19 +++++++++---------- 1 files changed, 9 insertions(+), 10 deletions(-)
vcs-svn/fast_export.c +9 -10
@@ -34,10 +34,9 @@ void fast_export_reset(void)void fast_export_delete (uint32_t depth,const uint32_t *path) { -putchar ('D' ); -putchar (' ' ); -pool_print_seq (depth, path,'/' , stdout); -putchar ('\n' ); +printf ("D \"" ); +pool_print_seq_q (depth, path,'/' , stdout); +printf ("\"\n" ); }static void fast_export_truncate (uint32_t depth,const uint32_t *path,uint32_t mode)
@@ -54,9 +53,9 @@ void fast_export_modify(uint32_t depth, const uint32_t *path, uint32_t mode,fast_export_truncate (depth, path, mode);return ; } -printf ("M %06" PRIo32" %s " , mode, dataref); -pool_print_seq (depth, path,'/' , stdout); -putchar ('\n' ); +printf ("M %06" PRIo32" %s \"" , mode, dataref); +pool_print_seq_q (depth, path,'/' , stdout); +printf ("\"\n" ); }static char gitsvnline[MAX_GITSVN_LINE_LEN ];
@@ -97,9 +96,9 @@ void fast_export_end_commit(uint32_t revision)static void ls_from_rev (uint32_t rev,uint32_t depth,const uint32_t *path) {/* ls :5 path/to/old/file */ -printf ("ls :%" PRIu32" " , rev); -pool_print_seq (depth, path,'/' , stdout); -putchar ('\n' ); +printf ("ls :%" PRIu32" \"" , rev); +pool_print_seq_q (depth, path,'/' , stdout); +printf ("\"\n" );fflush (stdout); }
Signature · 1 line
1.7.4.1
PATCH v2 12/12 vcs-svn: use mark from previous import for parent commit
In reply to Jonathan NiederFrom: David Barr <[email protected]>
Date: Sun, 12 Dec 2010 13:41:38 +1100
With this patch, overlapping incremental imports work.
---
That's the end of the series.
Bug reports, suggestions, improvements, welcome, of course. The documentation in contrib/svn-fe/svn-fe.txt is probably out of date now.
Have fun :)
Jonathan
vcs-svn/fast_export.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
vcs-svn/fast_export.c +1 -1
@@ -83,7 +83,7 @@ void fast_export_begin_commit(uint32_t revision, uint32_t author, char *log, log, gitsvnline);if (!first_commit_done) {if (revision >1 ) -printf ("from refs/heads/master^0\n" ); +printf ("from :%" PRIu32"\n" , revision -1 ); first_commit_done =1 ; } }
Signature · 1 line
1.7.4.1
Re: [PATCH v2 00/12] vcs-svn: incremental import
In reply to Jonathan NiederHeya,
Quoted 2 lines · Jonathan Nieder
Patch 1 changes the mark numbers for blobs to be ridiculously high, to make room for memorable commit marks (:1 for r1, :2 for r2, etc).
How high is rediculously high? There are repositories with a lot of commits, will they have enough room? Ditto for repositories with a lot of blobs, will _those_ have enough room? What about the git fast-import side, what about this statement:
"Marks are stored in a sparse array, using 1 pointer (4 bytes or 8 bytes, depending on pointer size) per mark. Although the array is sparse, frontends are still strongly encouraged to use marks between 1 and n, where n is the total number of marks required for this import."
Signature · 3 lines
Cheers, Sverre Rabbelier
Re: [PATCH v2 00/12] vcs-svn: incremental import
Quoted 6 lines · Sverre Rabbelier
Quoted 2 lines · Jonathan Nieder
Patch 1 changes the mark numbers for blobs to be ridiculously high, to make room for memorable commit marks (:1 for r1, :2 for r2, etc).
How high is rediculously high?
2^30. Later in the series (vcs-svn: eliminate repo_tree structure) the blob marks are eliminated altogether.
Quoted 4 lines
"Marks are stored in a sparse array, using 1 pointer (4 bytes or 8 bytes, depending on pointer size) per mark. Although the array is sparse, frontends are still strongly encouraged to use marks between 1 and n, where n is the total number of marks required for this import."
I assume that is mostly meant to make it easier to write alternate backends.