wildmatch precompilation interface
cover wildmatch precompilation interface
My recently landed wildmatch test series was in preperation for some more major wildmatch changes.
Here's another series that prepares for bigger changes in wildmatch, it adds an interface to pre-compile the patterns. Right now there's no point in doing this, and it's harmless since none of the codepaths are that performance sensitive, but down the line this'll save us time as we'll be able to skip re-parsing the pattern each time with a better wildmatch backend.
Ævar Arnfjörð Bjarmason (2):
wildmatch: add interface for precompiling wildmatch() patterns
wildmatch: use the new precompiling wildmatch()
builtin/name-rev.c | 7 ++++++- builtin/replace.c | 7 ++++--- config.c | 8 ++++++-- refs.c | 7 ++++--- wildmatch.c | 25 +++++++++++++++++++++++++ wildmatch.h | 11 +++++++++++ 6 files changed, 56 insertions(+), 9 deletions(-)
Signature · 1 line
2.15.1.424.g9478a66081
PATCH 1/2 wildmatch: add interface for precompiling wildmatch() patterns
Add the scaffolding necessary for precompiling wildmatch() patterns.
There is currently no point in doing this with the wildmatch() function we have now, since it can't make any use of precompiling the pattern.
But adding this interface and making use of it will make it easy to refactor the wildmatch() function to parse the pattern into opcodes as some glob() implementations do, or to drop an alternate wildmatch() backend in which trades parsing slowness for faster matching, such as the PCRE v2 conversion function that understands the wildmatch() syntax.
It's very unlikely that we'll remove the wildmatch() function as a convenience wrapper even if we end up requiring a separate compilation step in some future implementation. There are a lot of one-shot wildmatches in the codebase, in that case most likely wildmatch() will be kept around as a shorthand for wildmatch_{compile,match,free}().
I modeled this interface on the PCRE v2 interface. I didn't go with a glob(3) & globfree(3)-like interface because that would require every wildmatch() user to pass a dummy parameter, which I got rid of in 55d3426929 ("wildmatch: remove unused wildopts parameter", 2017-06-22).
--- wildmatch.c | 25 +++++++++++++++++++++++++ wildmatch.h | 11 +++++++++++ 2 files changed, 36 insertions(+)
wildmatch.c +25 -0
@@ -276,3 +276,28 @@ int wildmatch(const char *pattern, const char *text, unsigned int flags) {return dowild ((const uchar *)pattern, (const uchar *)text, flags); } + +struct wildmatch_compiled *wildmatch_compile (const char *pattern, +unsigned int flags) +{ +struct wildmatch_compiled *wildmatch_compiled =xmalloc ( +sizeof (struct wildmatch_compiled )); + wildmatch_compiled->pattern =xstrdup (pattern); + wildmatch_compiled->flags = flags; + +return wildmatch_compiled; +} + +int wildmatch_match (struct wildmatch_compiled *wildmatch_compiled, +const char *text) +{ +return wildmatch (wildmatch_compiled->pattern, text, + wildmatch_compiled->flags); +} + +void wildmatch_free (struct wildmatch_compiled *wildmatch_compiled) +{ +if (wildmatch_compiled) +free ((void *)wildmatch_compiled->pattern); +free (wildmatch_compiled); +}
wildmatch.h +11 -0
@@ -10,5 +10,16 @@#define WM_ABORT_ALL -1#define WM_ABORT_TO_STARSTAR -2 +struct wildmatch_compiled { +const char *pattern; +unsigned int flags; +}; +int wildmatch (const char *pattern,const char *text,unsigned int flags); +struct wildmatch_compiled *wildmatch_compile (const char *pattern, +unsigned int flags); +int wildmatch_match (struct wildmatch_compiled *wildmatch_compiled, +const char *text); +void wildmatch_free (struct wildmatch_compiled *wildmatch_compiled); +#endif
Signature · 1 line
2.15.1.424.g9478a66081
PATCH 2/2 wildmatch: use the new precompiling wildmatch()
In reply to Ævar Arnfjörð BjarmasonMake some limited use of the wildmatch() interface for "precompiling" patterns, although the current implementation does not do this yet.
The main hot codepath in dir.c is not being touched yet, but some other invocations where we repeatedly match the same glob against multiple strings have been migrated.
--- builtin/name-rev.c | 7 ++++++- builtin/replace.c | 7 ++++--- config.c | 8 ++++++-- refs.c | 7 ++++--- 4 files changed, 20 insertions(+), 9 deletions(-)
builtin/name-rev.c +6 -1
@@ -128,14 +128,19 @@ static void name_rev(struct commit *commit,static int subpath_matches (const char *path,const char *filter) {const char *subpath = path; +struct wildmatch_compiled *wildmatch_compiled = +wildmatch_compile (filter,0 );while (subpath) { -if (!wildmatch (filter, subpath,0 )) +if (!wildmatch_match (wildmatch_compiled, subpath)) { +wildmatch_free (wildmatch_compiled);return subpath - path; + } subpath =strchr (subpath,'/' );if (subpath) subpath++; } +wildmatch_free (wildmatch_compiled);return -1 ; }
builtin/replace.c +4 -3
@@ -32,7 +32,7 @@ enum replace_format { };struct show_data { -const char *pattern; +struct wildmatch_compiled *wildmatch_compiled;enum replace_format format; };
@@ -41,7 +41,7 @@ static int show_reference(const char *refname, const struct object_id *oid, {struct show_data *data = cb_data; -if (!wildmatch (data->pattern, refname,0 )) { +if (!wildmatch_match (data->wildmatch_compiled, refname)) {if (data->format ==REPLACE_FORMAT_SHORT )printf ("%s\n" , refname);else if (data->format ==REPLACE_FORMAT_MEDIUM )
@@ -70,7 +70,7 @@ static int list_replace_refs(const char *pattern, const char *format)if (pattern ==NULL ) pattern ="*" ; - data.pattern = pattern; + data.wildmatch_compiled =wildmatch_compile (pattern,0 );if (format ==NULL || *format =='\0' || !strcmp (format,"short" )) data.format =REPLACE_FORMAT_SHORT ;
@@ -84,6 +84,7 @@ static int list_replace_refs(const char *pattern, const char *format) format);for_each_replace_ref (show_reference, (void *)&data); +wildmatch_free (data.wildmatch_compiled);return 0 ; }
config.c +6 -2
@@ -210,6 +210,7 @@ static int include_by_gitdir(const struct config_options *opts,int ret =0 , prefix;const char *git_dir;int already_tried_absolute =0 ; +struct wildmatch_compiled *wildmatch_compiled =NULL ;if (opts->git_dir) git_dir = opts->git_dir;
@@ -237,8 +238,10 @@ static int include_by_gitdir(const struct config_options *opts, goto done; } - ret = !wildmatch (pattern.buf + prefix, text.buf + prefix, - icase ?WM_CASEFOLD :0 ); +if (!wildmatch_compiled) + wildmatch_compiled =wildmatch_compile (pattern.buf + prefix, + icase ?WM_CASEFOLD :0 ); + ret = !wildmatch_match (wildmatch_compiled, text.buf + prefix);if (!ret && !already_tried_absolute) {/*
@@ -257,6 +260,7 @@ static int include_by_gitdir(const struct config_options *opts, done:strbuf_release (&pattern);strbuf_release (&text); +wildmatch_free (wildmatch_compiled);return ret; }
refs.c +4 -3
@@ -213,7 +213,7 @@ char *resolve_refdup(const char *refname, int resolve_flags,/* The argument to filter_refs */ struct ref_filter { -const char *pattern; +struct wildmatch_compiled *code;each_ref_fn *fn;void *cb_data; };
@@ -291,7 +291,7 @@ static int filter_refs(const char *refname, const struct object_id *oid, {struct ref_filter *filter = (struct ref_filter *)data; -if (wildmatch (filter->pattern, refname,0 )) +if (wildmatch_match (filter->code, refname))return 0 ;return filter->fn (refname, oid, flags, filter->cb_data); }
@@ -454,12 +454,13 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,strbuf_addch (&real_pattern,'*' ); } - filter.pattern = real_pattern.buf; + filter.code =wildmatch_compile (real_pattern.buf,0 ); filter.fn = fn; filter.cb_data = cb_data; ret =for_each_ref (filter_refs, &filter);strbuf_release (&real_pattern); +wildmatch_free (filter.code);return ret; }
Signature · 1 line
2.15.1.424.g9478a66081
Re: [PATCH 1/2] wildmatch: add interface for precompiling wildmatch() patterns
In reply to Ævar Arnfjörð BjarmasonQuoted 37 lines · Ævar Arnfjörð Bjarmason
Add the scaffolding necessary for precompiling
wildmatch()patterns.There is currently no point in doing this with the
wildmatch()function we have now, since it can't make any use of precompiling the pattern.But adding this interface and making use of it will make it easy to refactor the
wildmatch()function to parse the pattern into opcodes as someglob()implementations do, or to drop an alternatewildmatch()backend in which trades parsing slowness for faster matching, such as the PCRE v2 conversion function that understands thewildmatch()syntax.It's very unlikely that we'll remove the
wildmatch()function as a convenience wrapper even if we end up requiring a separate compilation step in some future implementation. There are a lot of one-shot wildmatches in the codebase, in that case most likelywildmatch()will be kept around as a shorthand for wildmatch_{compile,match,free}().I modeled this interface on the PCRE v2 interface. I didn't go with a glob(3) & globfree(3)-like interface because that would require every
wildmatch()user to pass a dummy parameter, which I got rid of in55d3426929("wildmatch: remove unused wildopts parameter", 2017-06-22).Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- wildmatch.c | 25 +++++++++++++++++++++++++ wildmatch.h | 11 +++++++++++ 2 files changed, 36 insertions(+)wildmatch.c +0
-0
wildmatch.c int wildmatch(const char *pattern, const char *text, unsigned int flags) · line 284
Quoted 9 lines
wildmatch.c +6
-0@@ -276,3 +276,28 @@ int wildmatch(const char *pattern, const char *text, unsigned int flags) {return dowild ((const uchar *)pattern, (const uchar *)text, flags); } + +struct wildmatch_compiled *wildmatch_compile (const char *pattern, +unsigned int flags) +{ +struct wildmatch_compiled *wildmatch_compiled =xmalloc ( +sizeof (struct wildmatch_compiled ));
struct wildmatch_compiled *data = xmalloc(sizeof(*data));
?
It shortens the line a bit. We already use WM_ prefix for wildmatch flags, perhaps we can use it for wildmatch structs too (e.g. wm_compiled instead)
Quoted 7 lines
wildmatch.c +7
-0+ wildmatch_compiled->pattern =xstrdup (pattern); + wildmatch_compiled->flags = flags; + +return wildmatch_compiled; +} + +int wildmatch_match(struct wildmatch_compiled *wildmatch_compiled,
wildmatch.c int wildmatch(const char *pattern, const char *text, unsigned int flags) · line 301
Quoted 10 lines
wildmatch.c +10
-0+ const char *text) +{ +return wildmatch (wildmatch_compiled->pattern, text, + wildmatch_compiled->flags); +} + +void wildmatch_free (struct wildmatch_compiled *wildmatch_compiled) +{ +if (wildmatch_compiled) +free ((void *)wildmatch_compiled->pattern);
Why do make pattern type "const char *" then remove "const" with typecast here? Why not just "char *" in wildmatch_compiled?
If the reason is to avoid other users from peeking in and modifying it, then perhaps you can move struct wildmatch_compiled to wildmatch.c and keep it an opaque struct pointer.
Quoted 26 lines
wildmatch.c +2
-0+free (wildmatch_compiled); +}wildmatch.h +11
-1@@ -10,5 +10,16 @@#define WM_ABORT_ALL -1#define WM_ABORT_TO_STARSTAR -2 +struct wildmatch_compiled { +const char *pattern; +unsigned int flags; +}; +int wildmatch (const char *pattern,const char *text,unsigned int flags); +struct wildmatch_compiled *wildmatch_compile (const char *pattern, +unsigned int flags); +int wildmatch_match (struct wildmatch_compiled *wildmatch_compiled, +const char *text); +void wildmatch_free (struct wildmatch_compiled *wildmatch_compiled); +#endif --2.15.1.424.g9478a66081
Signature · 1 line
Duy
Re: [PATCH 2/2] wildmatch: use the new precompiling wildmatch()
In reply to Ævar Arnfjörð BjarmasonQuoted 13 lines · Ævar Arnfjörð Bjarmason
config.c +1
-0@@ -210,6 +210,7 @@ static int include_by_gitdir(const struct config_options *opts,int ret =0 , prefix;const char *git_dir;int already_tried_absolute =0 ; +struct wildmatch_compiled *wildmatch_compiled =NULL ;if (opts->git_dir) git_dir = opts->git_dir;
config.c static int include_by_gitdir(const struct config_options *opts, · line 244
Quoted 9 lines
config.c +4
-2@@ -237,8 +238,10 @@ static int include_by_gitdir(const struct config_options *opts, goto done; } - ret = !wildmatch (pattern.buf + prefix, text.buf + prefix, - icase ?WM_CASEFOLD :0 ); +if (!wildmatch_compiled) + wildmatch_compiled =wildmatch_compile (pattern.buf + prefix, + icase ?WM_CASEFOLD :0 ); + ret = !wildmatch_match (wildmatch_compiled, text.buf + prefix);
This is a one-shot matching. Is it worth converting to the new interface?
Quoted 16 lines
config.c +1
-0if (!ret && !already_tried_absolute) {/* @@ -257,6 +260,7 @@ static int include_by_gitdir(const struct config_options *opts, done:strbuf_release (&pattern);strbuf_release (&text); +wildmatch_free (wildmatch_compiled);return ret; }refs.c +0
-0
refs.c char *resolve_refdup(const char *refname, int resolve_flags, · line 216
Quoted 5 lines
refs.c +1
-1@@ -213,7 +213,7 @@ char *resolve_refdup(const char *refname, int resolve_flags,/* The argument to filter_refs */ struct ref_filter { -const char *pattern; +struct wildmatch_compiled *code;
This actually makes me think if we should name this struct wildmatch_pattern.
Quoted 30 lines
refs.c +3
-3each_ref_fn *fn;void *cb_data; };@@ -291,7 +291,7 @@ static int filter_refs(const char *refname, const struct object_id *oid, {struct ref_filter *filter = (struct ref_filter *)data; -if (wildmatch (filter->pattern, refname,0 )) +if (wildmatch_match (filter->code, refname))return 0 ;return filter->fn (refname, oid, flags, filter->cb_data); }@@ -454,12 +454,13 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,strbuf_addch (&real_pattern,'*' ); } - filter.pattern = real_pattern.buf; + filter.code =wildmatch_compile (real_pattern.buf,0 ); filter.fn = fn; filter.cb_data = cb_data; ret =for_each_ref (filter_refs, &filter);strbuf_release (&real_pattern); +wildmatch_free (filter.code);return ret; } --2.15.1.424.g9478a66081
Signature · 1 line
Duy
Quoted 9 lines · Ævar Arnfjörð Bjarmason
My recently landed wildmatch test series was in preperation for some more major wildmatch changes.
Here's another series that prepares for bigger changes in wildmatch, it adds an interface to pre-compile the patterns. Right now there's no point in doing this, and it's harmless since none of the codepaths are that performance sensitive, but down the line this'll save us time as we'll be able to skip re-parsing the pattern each time with a better wildmatch backend.
I don't see any big problem with this, but should this be a standalone series? Some changes look harmless now, but I'd rather see the real precompile implementation to see how it impacts (or benefits) the converted call sites.
Signature · 1 line
Duy
Re: [PATCH 2/2] wildmatch: use the new precompiling wildmatch()
In reply to Duy NguyenOn Mon, Feb 26 2018, Duy Nguyen jotted:
Quoted 18 lines
Quoted 16 lines · Ævar Arnfjörð Bjarmason
config.c +1
-0@@ -210,6 +210,7 @@ static int include_by_gitdir(const struct config_options *opts,int ret =0 , prefix;const char *git_dir;int already_tried_absolute =0 ; +struct wildmatch_compiled *wildmatch_compiled =NULL ;if (opts->git_dir) git_dir = opts->git_dir;@@ -237,8 +238,10 @@ static int include_by_gitdir(const struct config_options *opts, goto done; }
Quoted 8 lines
config.c static int include_by_gitdir(const struct config_options *opts, · line 244
Quoted 6 lines
config.c +4
-2- ret = !wildmatch (pattern.buf + prefix, text.buf + prefix, - icase ?WM_CASEFOLD :0 ); +if (!wildmatch_compiled) + wildmatch_compiled =wildmatch_compile (pattern.buf + prefix, + icase ?WM_CASEFOLD :0 ); + ret = !wildmatch_match (wildmatch_compiled, text.buf + prefix);This is a one-shot matching.
It will match once *or* twice depending on how the goto in that codepath does.
Quoted 1 line
Is it worth converting to the new interface?
I moved it more as a showcase, to show how the interface looks like in common scenarios. It obviously won't be a performance bottleneck or anything like that for this specific codepath.
Quoted 17 lines
Quoted 17 lines
config.c +1
-0if (!ret && !already_tried_absolute) {/* @@ -257,6 +260,7 @@ static int include_by_gitdir(const struct config_options *opts, done:strbuf_release (&pattern);strbuf_release (&text); +wildmatch_free (wildmatch_compiled);return ret; }refs.c +0
-0@@ -213,7 +213,7 @@ char *resolve_refdup(const char *refname, int resolve_flags,
Quoted 6 lines
refs.c char *resolve_refdup(const char *refname, int resolve_flags, · line 216
Quoted 4 lines
refs.c +1
-1/* The argument to filter_refs */ struct ref_filter { -const char *pattern; +struct wildmatch_compiled *code;This actually makes me think if we should name this struct
wildmatch_pattern.
Yeah, that's a better name.
Quoted 30 lines
Quoted 30 lines
refs.c +3
-3each_ref_fn *fn;void *cb_data; };@@ -291,7 +291,7 @@ static int filter_refs(const char *refname, const struct object_id *oid, {struct ref_filter *filter = (struct ref_filter *)data; -if (wildmatch (filter->pattern, refname,0 )) +if (wildmatch_match (filter->code, refname))return 0 ;return filter->fn (refname, oid, flags, filter->cb_data); }@@ -454,12 +454,13 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,strbuf_addch (&real_pattern,'*' ); } - filter.pattern = real_pattern.buf; + filter.code =wildmatch_compile (real_pattern.buf,0 ); filter.fn = fn; filter.cb_data = cb_data; ret =for_each_ref (filter_refs, &filter);strbuf_release (&real_pattern); +wildmatch_free (filter.code);return ret; } --2.15.1.424.g9478a66081
Re: [PATCH 1/2] wildmatch: add interface for precompiling wildmatch() patterns
In reply to Duy NguyenOn Mon, Feb 26 2018, Duy Nguyen jotted:
Quoted 51 lines
Quoted 37 lines · Ævar Arnfjörð Bjarmason
Add the scaffolding necessary for precompiling
wildmatch()patterns.There is currently no point in doing this with the
wildmatch()function we have now, since it can't make any use of precompiling the pattern.But adding this interface and making use of it will make it easy to refactor the
wildmatch()function to parse the pattern into opcodes as someglob()implementations do, or to drop an alternatewildmatch()backend in which trades parsing slowness for faster matching, such as the PCRE v2 conversion function that understands thewildmatch()syntax.It's very unlikely that we'll remove the
wildmatch()function as a convenience wrapper even if we end up requiring a separate compilation step in some future implementation. There are a lot of one-shot wildmatches in the codebase, in that case most likelywildmatch()will be kept around as a shorthand for wildmatch_{compile,match,free}().I modeled this interface on the PCRE v2 interface. I didn't go with a glob(3) & globfree(3)-like interface because that would require every
wildmatch()user to pass a dummy parameter, which I got rid of in55d3426929("wildmatch: remove unused wildopts parameter", 2017-06-22).Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- wildmatch.c | 25 +++++++++++++++++++++++++ wildmatch.h | 11 +++++++++++ 2 files changed, 36 insertions(+)wildmatch.c +0
-0wildmatch.c int wildmatch(const char *pattern, const char *text, unsigned int flags) · line 284
Quoted 9 lines
wildmatch.c +6
-0@@ -276,3 +276,28 @@ int wildmatch(const char *pattern, const char *text, unsigned int flags) {return dowild ((const uchar *)pattern, (const uchar *)text, flags); } + +struct wildmatch_compiled *wildmatch_compile (const char *pattern, +unsigned int flags) +{ +struct wildmatch_compiled *wildmatch_compiled =xmalloc ( +sizeof (struct wildmatch_compiled ));struct
wildmatch_compiled*data = xmalloc(sizeof(*data));
Quoted 5 lines
?
It shortens the line a bit. We already use
WM_prefix for wildmatch flags, perhaps we can use it for wildmatch structs too (e.g.wm_compiledinstead)
Thanks, that's better.
Quoted 21 lines
Quoted 7 lines
wildmatch.c +7
-0+ wildmatch_compiled->pattern =xstrdup (pattern); + wildmatch_compiled->flags = flags; + +return wildmatch_compiled; +} + +int wildmatch_match(struct wildmatch_compiled *wildmatch_compiled,wildmatch.c int wildmatch(const char *pattern, const char *text, unsigned int flags) · line 301
Quoted 10 lines
wildmatch.c +10
-0+ const char *text) +{ +return wildmatch (wildmatch_compiled->pattern, text, + wildmatch_compiled->flags); +} + +void wildmatch_free (struct wildmatch_compiled *wildmatch_compiled) +{ +if (wildmatch_compiled) +free ((void *)wildmatch_compiled->pattern);Why do make pattern type "const char *" then remove "const" with typecast here? Why not just "char *" in
wildmatch_compiled?
Quoted 3 lines
If the reason is to avoid other users from peeking in and modifying it, then perhaps you can move struct
wildmatch_compiledtowildmatch.cand keep it an opaque struct pointer.
Yes, it's to indicate that "pattern" won't ever be modified. I think it's more readable / self documenting to have the compiler enforce that via const, even though it requires the cast to free it.
On Mon, Feb 26 2018, Duy Nguyen jotted:
Quoted 12 lines
Quoted 9 lines · Ævar Arnfjörð Bjarmason
My recently landed wildmatch test series was in preperation for some more major wildmatch changes.
Here's another series that prepares for bigger changes in wildmatch, it adds an interface to pre-compile the patterns. Right now there's no point in doing this, and it's harmless since none of the codepaths are that performance sensitive, but down the line this'll save us time as we'll be able to skip re-parsing the pattern each time with a better wildmatch backend.
Quoted 4 lines
I don't see any big problem with this, but should this be a standalone series? Some changes look harmless now, but I'd rather see the real precompile implementation to see how it impacts (or benefits) the converted call sites.
Let's drop this for now sinceyou're on the fence about it, I wasn't all that sure myself and thought I'd send it in for comments.
I don't have anything ready for submission from the rest of this series, but figured (if you/others didn't mind) that it might be easier to review the addition of the interface at first, but indeed, on second thought that doesn't make sense.
The state of what I have is:
1. <this tiny series>
2. <WIP add the wildmatch compile interface to more stuff, notably dir.c>
The dir.c use (and some tree-related stuff) are the real hot codepaths using wildmatch.
3. <WIP Replace the wildmatch backend with syntax-compatible PCRE>
This is somewhat of a mess right now. Preliminary tests reveal that the pathological case tested for by t/perf/p0100-globbing.sh is wildly faster, but that most regular matching is a bit slower, although that might be me being stupid with the interface.