Bug with git svn fetch? "error:too many matches for svn-remote.svn.added-placeholder"
Hello,
I'm running a git svn fetch of a large svn repo and it is grinding to a halt with thousands of "error: too many matches for svn-remote.svn.added-placeholder". My .git/config has grown to ~50000 lines of added-placeholders.
I also had to up my linux file limit because I have ~2500 open files (tempfiles in .git)
I'm doing a onetime migration so I don't really care about having the added-placeholder entries but I do want to keep empty directories.
Any thoughts for a workaround?
Thanks!
PATCH 1/1 config: arbitrary number of matches for --unset and --replace-all
git-config used a static match array to hold the matches we want to unset/replace when using --unset or --replace-all. Use a variable-sized array instead.
This in particular fixes the symptoms git-svn had when storing large numbers of svn-remote.*.added-placeholder entries in the config file.
While the tests are rather more paranoid than just --unset and --replace-all, the other operations already worked. Indeed git-svn's usage only breaks the first time *after* creating so many entries, when it wants to unset and re-add them all.
---
This should fix it, though I haven't actually tested with such a funny use-case, nor written a proper git-svn test for it. My analysis about the failure mode is from briefly looking at the code.
git-svn should probably be changed to use another way of storing these, as git-config is not a very efficient database, but I'm leaving that for someone who cares about SVN. (Note that it's also much harder as it'll need a migration plan.)
config.c | 19 ++++++++++----- t/t1303-wacky-config.sh | 64 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 6 deletions(-)
config.c +13 -6
@@ -1197,15 +1197,14 @@ int git_config(config_fn_t fn, void *data)* Find all the stuff for git_config_set() below. */ -#define MAX_MATCHES 512 -static struct {int baselen;char *key;int do_not_match;regex_t *value_regex;int multi_replace; -size_t offset[MAX_MATCHES ]; +size_t *offset; +unsigned int offset_alloc;enum {START ,SECTION_SEEN ,SECTION_END_SEEN ,KEY_SEEN } state;int seen; } store;
@@ -1228,11 +1227,11 @@ static int store_aux(const char *key, const char *value, void *cb)if (matches (key, value)) {if (store.seen ==1 && store.multi_replace ==0 ) {warning ("%s has multiple values" , key); - }else if (store.seen >=MAX_MATCHES ) { -error ("too many matches for %s" , key); -return 1 ; } +ALLOC_GROW (store.offset, store.seen+1 , + store.offset_alloc); + store.offset[store.seen] = cf->do_ftell (cf); store.seen++; }
@@ -1260,11 +1259,15 @@ static int store_aux(const char *key, const char *value, void *cb)* Do not increment matches: this is no match, but we * just made sure we are in the desired section. */ +ALLOC_GROW (store.offset, store.seen+1 , + store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf);/* fallthru */ case SECTION_END_SEEN :case START :if (matches (key, value)) { +ALLOC_GROW (store.offset, store.seen+1 , + store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf); store.state =KEY_SEEN ; store.seen++;
@@ -1272,6 +1275,9 @@ static int store_aux(const char *key, const char *value, void *cb)if (strrchr (key,'.' ) - key == store.baselen && !strncmp (key, store.key, store.baselen)) { store.state =SECTION_SEEN ; +ALLOC_GROW (store.offset, + store.seen+1 , + store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf); } }
@@ -1570,6 +1576,7 @@ int git_config_set_multivar_in_file(const char *config_filename, } } +ALLOC_GROW (store.offset,1 , store.offset_alloc); store.offset[0 ] =0 ; store.state =START ; store.seen =0 ;
t/t1303-wacky-config.sh +64 -0
85 lines not on this page · raw message
Signature · 1 line
1.8.5.rc0.346.g150976e
Re: [PATCH] config: arbitrary number of matches for --unset and --replace-all
Quoted 7 lines · Thomas Rast
t/t1303-wacky-config.sh +0
-0@@ -47,4 +58,57 @@ test_expect_success 'do not crash on special long config line' ' check section.key "$LONG_VALUE" '
t/t1303-wacky-config.sh test_expect_success 'do not crash on special long config line' ' · line 69
Quoted 10 lines
t/t1303-wacky-config.sh +9
-0+setup_many() { + setup && + # This time we want the newline so that we can tack on more + # entries. + echo >>.git/config && + # Semi-efficient way of concatenating 5^5 = 3125 lines. Note + # that because 'setup' already put one line, this means 3126 + # entries for section.key in the config file. + cat >5to1 <<EOF
Broken &&-chain.
Quoted 47 lines
t/t1303-wacky-config.sh +44
-1+ key = foo + key = foo + key = foo + key = foo + key = foo +EOF + cat 5to1 5to1 5to1 5to1 5to1 >5to2 && # 25 + cat 5to2 5to2 5to2 5to2 5to2 >5to3 && # 125 + cat 5to3 5to3 5to3 5to3 5to3 >5to4 && # 635 + cat 5to4 5to4 5to4 5to4 5to4 >>.git/config # 3125 +} + +test_expect_success 'get many entries' ' + setup_many && + git config --get-all section.key >actual && + test_line_count = 3126 actual +' + +test_expect_success 'get many entries by regex' ' + setup_many && + git config --get-regexp "sec.*ke." >actual && + test_line_count = 3126 actual +' + +test_expect_success 'add and replace one of many entries' ' + setup_many && + git config --add section.key bar && + check_regex section.key "b.*r" bar && + git config section.key beer "b.*r" && + check_regex section.key "b.*r" beer +' + +test_expect_success 'replace many entries' ' + setup_many && + git config --replace-all section.key bar && + check section.key bar +' + +test_expect_success 'unset many entries' ' + setup_many && + git config --unset-all section.key && + test_must_fail git config section.key +' + test_done --1.8.5.rc0.346.g150976e
PATCH 1/1 config: arbitrary number of matches for --unset and --replace-all
git-config used a static match array to hold the matches we want to unset/replace when using --unset or --replace-all. Use a variable-sized array instead.
This in particular fixes the symptoms git-svn had when storing large numbers of svn-remote.*.added-placeholder entries in the config file.
While the tests are rather more paranoid than just --unset and --replace-all, the other operations already worked. Indeed git-svn's usage only breaks the first time *after* creating so many entries, when it wants to unset and re-add them all.
---
Quoted 2 lines · Eric Sunshine
Quoted 1 line · Thomas Rast
+setup_many() {
[...]
Quoted 3 lines
Quoted 1 line
+ cat >5to1 <<EOF
Broken &&-chain.
Oops, thanks for catching.
config.c | 19 ++++++++++----- t/t1303-wacky-config.sh | 64 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 6 deletions(-)
config.c +13 -6
@@ -1197,15 +1197,14 @@ int git_config(config_fn_t fn, void *data)* Find all the stuff for git_config_set() below. */ -#define MAX_MATCHES 512 -static struct {int baselen;char *key;int do_not_match;regex_t *value_regex;int multi_replace; -size_t offset[MAX_MATCHES ]; +size_t *offset; +unsigned int offset_alloc;enum {START ,SECTION_SEEN ,SECTION_END_SEEN ,KEY_SEEN } state;int seen; } store;
@@ -1228,11 +1227,11 @@ static int store_aux(const char *key, const char *value, void *cb)if (matches (key, value)) {if (store.seen ==1 && store.multi_replace ==0 ) {warning ("%s has multiple values" , key); - }else if (store.seen >=MAX_MATCHES ) { -error ("too many matches for %s" , key); -return 1 ; } +ALLOC_GROW (store.offset, store.seen+1 , + store.offset_alloc); + store.offset[store.seen] = cf->do_ftell (cf); store.seen++; }
@@ -1260,11 +1259,15 @@ static int store_aux(const char *key, const char *value, void *cb)* Do not increment matches: this is no match, but we * just made sure we are in the desired section. */ +ALLOC_GROW (store.offset, store.seen+1 , + store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf);/* fallthru */ case SECTION_END_SEEN :case START :if (matches (key, value)) { +ALLOC_GROW (store.offset, store.seen+1 , + store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf); store.state =KEY_SEEN ; store.seen++;
@@ -1272,6 +1275,9 @@ static int store_aux(const char *key, const char *value, void *cb)if (strrchr (key,'.' ) - key == store.baselen && !strncmp (key, store.key, store.baselen)) { store.state =SECTION_SEEN ; +ALLOC_GROW (store.offset, + store.seen+1 , + store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf); } }
@@ -1570,6 +1576,7 @@ int git_config_set_multivar_in_file(const char *config_filename, } } +ALLOC_GROW (store.offset,1 , store.offset_alloc); store.offset[0 ] =0 ; store.state =START ; store.seen =0 ;
t/t1303-wacky-config.sh +64 -0
85 lines not on this page · raw message
Signature · 1 line
1.8.5.rc1.353.g06ba152
Re: [PATCH] config: arbitrary number of matches for --unset and --replace-all
Quoted 7 lines · Thomas Rast
git-config used a static match array to hold the matches we want to unset/replace when using --unset or --replace-all. Use a variable-sized array instead.
This in particular fixes the symptoms git-svn had when storing large numbers of svn-remote.*.added-placeholder entries in the config file.
Quoted 7 lines
While the tests are rather more paranoid than just --unset and --replace-all, the other operations already worked. Indeed git-svn's usage only breaks the first time *after* creating so many entries, when it wants to unset and re-add them all.
Reported-by Jess Hottenstein <[email protected]>Signed-off-by Thomas Rast <[email protected]>
This looks good to me.
I agree with your earlier assessment that adding tons of config keys is probably a bad idea. It's not just slow when looking up those keys, but it also slows down every single git operation (which might read config many times, if it is a script). Still, we should at least do the right thing in the face of such config.
Quoted 6 lines
config.c +2
-0@@ -1260,11 +1259,15 @@ static int store_aux(const char *key, const char *value, void *cb)* Do not increment matches: this is no match, but we * just made sure we are in the desired section. */ +ALLOC_GROW (store.offset, store.seen+1 , + store.offset_alloc);
config.c static int store_aux(const char *key, const char *value, void *cb) · line 1273
Quoted 10 lines
config.c +2
-0store.offset[store.seen] = cf->do_ftell (cf);/* fallthru */ case SECTION_END_SEEN :case START :if (matches (key, value)) { +ALLOC_GROW (store.offset, store.seen+1 , + store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf); store.state =KEY_SEEN ; store.seen++;
This code is weird to follow because of the fall-throughs. I do not think you have introduced any bugs with your patch, but it seems weird to me that we set the offset at the top of the hunk. If we hit the conditional in the bottom half, we do actually increment storer.seen, but only _after_ having overwritten the value from above (with the same value, no less).
But if we do not follow that code path, we may end up here:
config.c static int store_aux(const char *key, const char *value, void *cb) · line 1283
Quoted 10 lines
config.c +3
-0@@ -1272,6 +1275,9 @@ static int store_aux(const char *key, const char *value, void *cb)if (strrchr (key,'.' ) - key == store.baselen && !strncmp (key, store.key, store.baselen)) { store.state =SECTION_SEEN ; +ALLOC_GROW (store.offset, + store.seen+1 , + store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf); } }
where we overwrite it again, but do not update store.seen. Or we may trigger neither, and leave the function with our offset stored, but store.seen not incremented.
So it seems odd to me that we would set the offset, but in some cases never actually increment our counter. AFAICT, we do not ever access those values. The reader limits itself to "i < store.seen", which makes sense. And the writers always call a fresh ftell before incrementing store.seen. But maybe I am missing something.
Anyway, it is neither here nor there for your patch. Just something odd I noticed while reading the code.
Quoted 10 lines
t/t1303-wacky-config.sh +10
-0+setup_many() { + setup && + # This time we want the newline so that we can tack on more + # entries. + echo >>.git/config && + # Semi-efficient way of concatenating 5^5 = 3125 lines. Note + # that because 'setup' already put one line, this means 3126 + # entries for section.key in the config file. + cat >5to1 <<EOF && + key = foo
t/t1303-wacky-config.sh test_expect_success 'do not crash on special long config line' ' · line 80
Quoted 10 lines
t/t1303-wacky-config.sh +10
-0+ key = foo + key = foo + key = foo + key = foo +EOF + cat 5to1 5to1 5to1 5to1 5to1 >5to2 && # 25 + cat 5to2 5to2 5to2 5to2 5to2 >5to3 && # 125 + cat 5to3 5to3 5to3 5to3 5to3 >5to4 && # 635 + cat 5to4 5to4 5to4 5to4 5to4 >>.git/config # 3125 +}
You could make this even more semi-efficient by just storing the end result in 5to5, and then copying it into place rather than doing the whole dance for each test. I doubt it matters that much, though.
-Peff
Re: [PATCH] config: arbitrary number of matches for --unset and --replace-all
Quoted 9 lines · Jeff King
This code is weird to follow because of the fall-throughs. I do not think you have introduced any bugs with your patch, but it seems weird to me that we set the offset at the top of the hunk. If we hit the conditional in the bottom half, we do actually increment storer.seen, but only _after_ having overwritten the value from above (with the same value, no less).
But if we do not follow that code path, we may end up here:
Quoted 14 lines
config.c static int store_aux(const char *key, const char *value, void *cb) · line 1283
Quoted 10 lines
config.c +3
-0@@ -1272,6 +1275,9 @@ static int store_aux(const char *key, const char *value, void *cb)if (strrchr (key,'.' ) - key == store.baselen && !strncmp (key, store.key, store.baselen)) { store.state =SECTION_SEEN ; +ALLOC_GROW (store.offset, + store.seen+1 , + store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf); } }where we overwrite it again, but do not update store.seen. Or we may trigger neither, and leave the function with our offset stored, but store.seen not incremented.
It's doubly strange that we write in this hunk without any protection against overflow. I was too lazy to think about it long enough to come up with a possible example that triggers this, and instead just put in the defensive ALLOC_GROW(). But if you can trigger it, it will probably cause the algorithm to go off the rails because it overwrote store.state and possibly even store.seen.
Signature · 2 lines
Thomas Rast [email protected]
PATCH 1/1 fixup! config: arbitrary number of matches for --unset and --replace-all
In reply to Thomas Rast---
* I'll squash this to tr/config-multivalue-lift-max in preparation for merging it to 'master',which should happen by the end of this week.
Thanks.
config.c | 8 ++++---- t/t1303-wacky-config.sh | 14 +++++++------- 2 files changed, 11 insertions(+), 11 deletions(-)
config.c +4 -4
@@ -1192,7 +1192,7 @@ static int store_aux(const char *key, const char *value, void *cb)warning ("%s has multiple values" , key); } -ALLOC_GROW (store.offset, store.seen+1 , +ALLOC_GROW (store.offset, store.seen +1 , store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf);
@@ -1222,14 +1222,14 @@ static int store_aux(const char *key, const char *value, void *cb)* Do not increment matches: this is no match, but we * just made sure we are in the desired section. */ -ALLOC_GROW (store.offset, store.seen+1 , +ALLOC_GROW (store.offset, store.seen +1 , store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf);/* fallthru */ case SECTION_END_SEEN :case START :if (matches (key, value)) { -ALLOC_GROW (store.offset, store.seen+1 , +ALLOC_GROW (store.offset, store.seen +1 , store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf); store.state =KEY_SEEN ;
@@ -1239,7 +1239,7 @@ static int store_aux(const char *key, const char *value, void *cb) !strncmp(key, store.key, store.baselen)) { store.state =SECTION_SEEN ;ALLOC_GROW (store.offset, - store.seen+1 , + store.seen +1 , store.offset_alloc); store.offset[store.seen] = cf->do_ftell (cf); }
t/t1303-wacky-config.sh +7 -7
@@ -66,13 +66,13 @@ setup_many() { # Semi-efficient way of concatenating 5^5 = 3125 lines. Note # that because 'setup' already put one line, this means 3126 # entries for section.key in the config file. - cat >5to1 <<EOF - key = foo - key = foo - key = foo - key = foo - key = foo -EOF + cat >5to1 <<-\EOF && + key = foo + key = foo + key = foo + key = foo + key = foo + EOF cat 5to1 5to1 5to1 5to1 5to1 >5to2 && # 25 cat 5to2 5to2 5to2 5to2 5to2 >5to3 && # 125 cat 5to3 5to3 5to3 5to3 5to3 >5to4 && # 635
Signature · 1 line
1.8.5.1-402-gdd8f092
Re: [PATCH] fixup! config: arbitrary number of matches for --unset and --replace-all
Quoted 4 lines · Junio C Hamano
---
* I'll squash this to tr/config-multivalue-lift-max in preparation for merging it to 'master',which should happen by the end of this week.
Yeah, all makes sense to me. Thanks.
-Peff