grep: memory leak in PCRE2
cover grep: memory leak in PCRE2
This series fixes a small memory leak that was introduced when PCRE2 support was added, and that should be visible with t7813 and valgrind
Applies cleanly to maint and master but will conflict slightly with ab/no-kwset (currently in next) and most likely other changes introduced about this same codebase (ex: ab/pcre-jit-fixes) but hopefully the spreading on short simple commits helps with reviewing.
Carlo Marcelo Arenas Belón (3):
grep: make pcre1_tables version agnostic
grep: use pcre_tables also for PCRE2
grep: plug leak of pcre chartables in PCRE2
grep.c | 12 ++++++------ grep.h | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-)
Signature · 1 line
2.22.0
PATCH 1/3 grep: make pcre1_tables version agnostic
6d4b5747f0 ("grep: change internal *pcre* variable & function names to be *pcre1*", 2017-05-25), renamed most variables to be PCRE1 specific to give space to similarly named variables for PCRE2, but in this case the change wasn't needed as the types were compatible enough (const unsigned char* vs const uint8_t*) to be shared.
Revert that change, as 94da9193a6 ("grep: add support for PCRE v2", 2017-06-01) failed to create an equivalent PCRE2 version.
--- grep.c | 6 +++--- grep.h | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)
grep.c +3 -3
@@ -389,14 +389,14 @@ static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)if (opt->ignore_case) {if (has_non_ascii (p->pattern)) - p->pcre1_tables =pcre_maketables (); + p->pcre_tables =pcre_maketables (); options |=PCRE_CASELESS ; }if (is_utf8_locale () &&has_non_ascii (p->pattern)) options |=PCRE_UTF8 ; p->pcre1_regexp =pcre_compile (p->pattern, options, &error, &erroffset, - p->pcre1_tables); + p->pcre_tables);if (!p->pcre1_regexp)compile_regexp_failed (p, error);
@@ -462,7 +462,7 @@ static void free_pcre1_regexp(struct grep_pat *p) {pcre_free (p->pcre1_extra_info); } - pcre_free((void *)p->pcre1_tables); + pcre_free((void *)p->pcre_tables); }#else /* !USE_LIBPCRE1 */ static void compile_pcre1_regexp (struct grep_pat *p,const struct grep_opt *opt)
grep.h +1 -1
@@ -89,7 +89,7 @@ struct grep_pat {pcre *pcre1_regexp;pcre_extra *pcre1_extra_info;pcre_jit_stack *pcre1_jit_stack; -const unsigned char *pcre1_tables; +const unsigned char *pcre_tables;int pcre1_jit_on;pcre2_code *pcre2_pattern;pcre2_match_data *pcre2_match_data;
Signature · 1 line
2.22.0
Re: [PATCH 1/3] grep: make pcre1_tables version agnostic
Quoted 5 lines · Carlo Marcelo Arenas Belón
6d4b5747f0("grep: change internal *pcre* variable & function names to be *pcre1*", 2017-05-25), renamed most variables to be PCRE1 specific to give space to similarly named variables for PCRE2, but in this case the change wasn't needed as the types were compatible enough (const unsigned char* vs const uint8_t*) to be shared.
Both the v1 and v2 functions return const unsigned char *. I don't know where I got the uint8_t from. This makes more sense.
This series looks good to me. Thanks for the fix. Just one caveat:
The point of 6d4b5747f0 was not to only split out those variables we couldn't get away with re-using. Then I would have later re-used e.g. pcre1_jit_on & pcre2_jit_on as just pcre_jit_on. We could also do that now.
I think doing that & this part of the your changes makes things less readable. The two code branches we compile with ifdefs are mutually exclusive, so having the variables be unique helps with eyeballing / reasoning when changing the code.
Quoted 52 lines
Revert that change, as
94da9193a6("grep: add support for PCRE v2", 2017-06-01) failed to create an equivalent PCRE2 version.Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>--- grep.c | 6 +++--- grep.h | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)grep.c +3
-3@@ -389,14 +389,14 @@ static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)if (opt->ignore_case) {if (has_non_ascii (p->pattern)) - p->pcre1_tables =pcre_maketables (); + p->pcre_tables =pcre_maketables (); options |=PCRE_CASELESS ; }if (is_utf8_locale () &&has_non_ascii (p->pattern)) options |=PCRE_UTF8 ; p->pcre1_regexp =pcre_compile (p->pattern, options, &error, &erroffset, - p->pcre1_tables); + p->pcre_tables);if (!p->pcre1_regexp)compile_regexp_failed (p, error);@@ -462,7 +462,7 @@ static void free_pcre1_regexp(struct grep_pat *p) {pcre_free (p->pcre1_extra_info); } - pcre_free((void *)p->pcre1_tables); + pcre_free((void *)p->pcre_tables); }#else /* !USE_LIBPCRE1 */ static void compile_pcre1_regexp (struct grep_pat *p,const struct grep_opt *opt)grep.h +1
-1@@ -89,7 +89,7 @@ struct grep_pat {pcre *pcre1_regexp;pcre_extra *pcre1_extra_info;pcre_jit_stack *pcre1_jit_stack; -const unsigned char *pcre1_tables; +const unsigned char *pcre_tables;int pcre1_jit_on;pcre2_code *pcre2_pattern;pcre2_match_data *pcre2_match_data;
Re: [PATCH 1/3] grep: make pcre1_tables version agnostic
Quoted 8 lines · Ævar Arnfjörð Bjarmason
Quoted 5 lines · Carlo Marcelo Arenas Belón
6d4b5747f0("grep: change internal *pcre* variable & function names to be *pcre1*", 2017-05-25), renamed most variables to be PCRE1 specific to give space to similarly named variables for PCRE2, but in this case the change wasn't needed as the types were compatible enough (const unsigned char* vs const uint8_t*) to be shared.
reading this again, had to admit there is a fair amount of guessing on intent, but reading commits and the email discussion couldn't come up with a better explanation.
is the root cause for the bug different?, could it be that the pcre2 API was misunderstood? and you expected this pointer will be free together with the context? (as it is done when a cloned context with tables is used?)
Quoted 2 lines
Both the v1 and v2 functions return const unsigned char *. I don't know where I got the
uint8_tfrom. This makes more sense.
the type in PCRE2 is uint8_t, the documentation has a bug[1] almost every platform git cares for would have unsigned char = uint8_t though.
Quoted 4 lines
The point of
6d4b5747f0was not to only split out those variables we couldn't get away with re-using. Then I would have later re-used e.g.pcre1_jit_on&pcre2_jit_onas justpcre_jit_on. We could also do that now.
IMHO pcre_jit_on makes more sense as a bit, with local variables being used for the pcre*_config() call with the right type.(uint32_t != int)
Quoted 4 lines
I think doing that & this part of the your changes makes things less readable. The two code branches we compile with ifdefs are mutually exclusive, so having the variables be unique helps with eyeballing / reasoning when changing the code.
I thought that too until the typo[2] in the pcre?_jit_on variable (now in next) kind of
proved us wrong. Maybe the names are too similar?
either way, would you rather drop this patch and make a replica variable? should I rebase to next with Reviewed-By tags so that all other changes in flight that would conflict could be corrected?
Carlo
[1] https://bugs.exim.org/show_bug.cgi?id=2420
[2] https://public-inbox.org/git/[email protected]/
PATCH 2/3 grep: use pcre_tables also for PCRE2
In reply to Carlo Marcelo Arenas Belón94da9193a6 ("grep: add support for PCRE v2", 2017-06-01) added a local variable to keep track of the chartables (used when locale is not UTF-8 but non-ASCII characters are needed)
Remove that local variable in favor of the shared one within the grep structure and that can be shared by all functions.
--- grep.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
grep.c +2 -3
@@ -488,7 +488,6 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *optPCRE2_UCHAR errbuf[256 ];PCRE2_SIZE erroffset;int options =PCRE2_MULTILINE ; -const uint8_t *character_tables =NULL ;int jitret;int patinforet;size_t jitsizearg;
@@ -499,9 +498,9 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *optif (opt->ignore_case) {if (has_non_ascii (p->pattern)) { - character_tables =pcre2_maketables (NULL ); + p->pcre_tables =pcre2_maketables (NULL ); p->pcre2_compile_context =pcre2_compile_context_create (NULL ); -pcre2_set_character_tables (p->pcre2_compile_context, character_tables); +pcre2_set_character_tables (p->pcre2_compile_context, p->pcre_tables); } options |=PCRE2_CASELESS ; }
Signature · 1 line
2.22.0
PATCH 3/3 grep: plug leak of pcre chartables in PCRE2
In reply to Carlo Marcelo Arenas BelónJust as it is done with PCRE1, make sure that the allocated chartables get free at cleanup time.
This assumes no global context is used (NULL passed when created the tables), but will likely be updated in tandem if that ever changes.
--- grep.c | 1 + 1 file changed, 1 insertion(+)
grep.c +1 -0
@@ -604,6 +604,7 @@ static void free_pcre2_pattern(struct grep_pat *p)pcre2_match_data_free (p->pcre2_match_data);pcre2_jit_stack_free (p->pcre2_jit_stack);pcre2_match_context_free (p->pcre2_match_context); + free((void *)p->pcre_tables); }#else /* !USE_LIBPCRE2 */ static void compile_pcre2_pattern (struct grep_pat *p,const struct grep_opt *opt)
Signature · 1 line
2.22.0
Re: [PATCH 3/3] grep: plug leak of pcre chartables in PCRE2
Quoted 16 lines · Carlo Marcelo Arenas Belón
Just as it is done with PCRE1, make sure that the allocated chartables get free at cleanup time.
This assumes no global context is used (NULL passed when created the tables), but will likely be updated in tandem if that ever changes.
Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>--- grep.c | 1 + 1 file changed, 1 insertion(+)grep.c +0
-0
grep.c static void free_pcre2_pattern(struct grep_pat *p) · line 607
Quoted 4 lines
grep.c +1
-0@@ -604,6 +604,7 @@ static void free_pcre2_pattern(struct grep_pat *p)pcre2_match_data_free (p->pcre2_match_data);pcre2_jit_stack_free (p->pcre2_jit_stack);pcre2_match_context_free (p->pcre2_match_context); +free ((void *)p->pcre_tables);
Is the cast really needed? I'm rusty on the rules, removing it from the pcre_free() you might have copied this from produces a warning for me, but not for free() itself. This is on GCC 8.3.0. How about for you & what compiler(s)?
Re: [PATCH 3/3] grep: plug leak of pcre chartables in PCRE2
Quoted 6 lines · Ævar Arnfjörð Bjarmason
Quoted 1 line
+
free((void *)p->pcre_tables);Is the cast really needed? I'm rusty on the rules, removing it from the
pcre_free()you might have copied this from produces a warning for me, but not forfree()itself. This is on GCC 8.3.0. How about for you & what compiler(s)?
both will trigger warnings for the same reason (-Wincompatible-pointer-types-discards-qualifiers) with Apple LLVM version 10.0.1 (clang-1001.0.46.4)
gcc-9 in macOS triggers 2 "warnings"; one for discarding the const qualifier (-Wdiscarded-qualifiers)
and another for mismatching parameter to free():
note: expected 'void *' but argument is of type 'const uint8_t *' {aka 'const unsigned char *'}
Carlo
Re: [PATCH 3/3] grep: plug leak of pcre chartables in PCRE2
Am 28.07.19 um 03:41 schrieb Carlo Arenas:
Quoted 13 lines
Quoted 6 lines · Ævar Arnfjörð Bjarmason
Quoted 1 line
+
free((void *)p->pcre_tables);Is the cast really needed? I'm rusty on the rules, removing it from the
pcre_free()you might have copied this from produces a warning for me, but not forfree()itself. This is on GCC 8.3.0. How about for you & what compiler(s)?both will trigger warnings for the same reason (-Wincompatible-pointer-types-discards-qualifiers) with Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Quoted 6 lines
gcc-9 in macOS triggers 2 "warnings"; one for discarding the const qualifier (-Wdiscarded-qualifiers)
and another for mismatching parameter tofree():note: expected 'void *' but argument is of type 'const
uint8_t*' {aka 'const unsigned char *'}
Right: pcre_maketables() returns a const pointer, and you have to cast away this const'ness at some point if you want to free(3) that memory. Returning a non-const pointer would be more fitting, but I guess the idea is that users of the library are not supposed to change the contents of the table.
But wouldn't it be more correct to use pcre_free()? As long as we keep pcre_malloc() and pcre_free() at their default values it doesn't matter in practice, but using free(3) directly is a layering violation, no?
Perhaps just UNLEAK that thing? There is only a single way to build it and we can reuse it throughout the lifetime of the program, so there is no real need to clean it up before the OS does.
René
Re: [PATCH 3/3] grep: plug leak of pcre chartables in PCRE2
Quoted 19 lines · René Scharfe
Am 28.07.19 um 03:41 schrieb Carlo Arenas:
Quoted 17 lines
Quoted 6 lines · Ævar Arnfjörð Bjarmason
Quoted 1 line
+
free((void *)p->pcre_tables);Is the cast really needed? I'm rusty on the rules, removing it from the
pcre_free()you might have copied this from produces a warning for me, but not forfree()itself. This is on GCC 8.3.0. How about for you & what compiler(s)?both will trigger warnings for the same reason (-Wincompatible-pointer-types-discards-qualifiers) with Apple LLVM version 10.0.1 (clang-1001.0.46.4)
gcc-9 in macOS triggers 2 "warnings"; one for discarding the const qualifier (-Wdiscarded-qualifiers)
and another for mismatching parameter tofree():
Quoted 8 lines
Quoted 2 lines
note: expected 'void *' but argument is of type 'const
uint8_t*' {aka 'const unsigned char *'}Right:
pcre_maketables()returns a const pointer, and you have to cast away this const'ness at some point if you want to free(3) that memory. Returning a non-const pointer would be more fitting, but I guess the idea is that users of the library are not supposed to change the contents of the table.
note that this pointer was generated by pcre2_maketables() instead which is actually "const uint8_t *", but yes these tables are meant to be inmutable and that is why they are "cost".
there is even a compile time generated table for the C locale that is used internally and obviously shouldn't be free().
Quoted 3 lines
But wouldn't it be more correct to use
pcre_free()? As long as we keeppcre_malloc()andpcre_free()at their default values it doesn't matter in practice, but using free(3) directly is a layering violation, no?
yes, but that is the only option PCRE2 gives when not using a global context which is what the comment in the commit refers to.
FWIW pcre_free() doesn't exist anymore in PCRE2.
Quoted 3 lines
Perhaps just UNLEAK that thing? There is only a single way to build it and we can reuse it throughout the lifetime of the program, so there is no real need to clean it up before the OS does.
That would be a better fit if it would be created once in cmd_grep and then shared with all worker threads (which I thought would be nice to do in the future anyway), but this change was trying to be conservative and just to the minimum to close the leak.
Carlo
Re: [PATCH 3/3] grep: plug leak of pcre chartables in PCRE2
Am 30.07.19 um 02:08 schrieb Carlo Arenas:
Quoted 29 lines · René Scharfe
Quoted 19 lines
Am 28.07.19 um 03:41 schrieb Carlo Arenas:
Quoted 17 lines
Quoted 6 lines · Ævar Arnfjörð Bjarmason
Quoted 1 line
+
free((void *)p->pcre_tables);Is the cast really needed? I'm rusty on the rules, removing it from the
pcre_free()you might have copied this from produces a warning for me, but not forfree()itself. This is on GCC 8.3.0. How about for you & what compiler(s)?both will trigger warnings for the same reason (-Wincompatible-pointer-types-discards-qualifiers) with Apple LLVM version 10.0.1 (clang-1001.0.46.4)
gcc-9 in macOS triggers 2 "warnings"; one for discarding the const qualifier (-Wdiscarded-qualifiers)
and another for mismatching parameter tofree():Quoted 8 lines
Quoted 2 lines
note: expected 'void *' but argument is of type 'const
uint8_t*' {aka 'const unsigned char *'}Right:
pcre_maketables()returns a const pointer, and you have to cast away this const'ness at some point if you want to free(3) that memory. Returning a non-const pointer would be more fitting, but I guess the idea is that users of the library are not supposed to change the contents of the table.
Quoted 3 lines
note that this pointer was generated by
pcre2_maketables()instead which is actually "constuint8_t*", but yes these tables are meant to be inmutable and that is why they are "cost".
Only the const'ness matters in regards to the need for casting a pointer to feed to free(3).
Doing the cast in a library function or using an opaque pointer type instead of a fake const pointer would be nicer ways on the part of PCRE2 to keep callers from messing with the table. Forcing users to cast const away or leak is not very nice.. Nothing we can do about it, except perhaps adding a wish list item for PCRE3, I guess.
https://pcre.org/current/doc/html/pcre2_maketables.html says that pcre2_maketables() returns const unsigned char *, by the way. I don't get the PCRE2_SUFFIX business in pcre2.h ("Define macros that generate width-specific names from generic versions."), though.
Quoted 8 lines
Quoted 3 lines
But wouldn't it be more correct to use
pcre_free()? As long as we keeppcre_malloc()andpcre_free()at their default values it doesn't matter in practice, but using free(3) directly is a layering violation, no?yes, but that is the only option PCRE2 gives when not using a global context which is what the comment in the commit refers to.
FWIW
pcre_free()doesn't exist anymore in PCRE2.
OK, and while https://pcre.org/original/doc/html/pcreapi.html#TOC1
says that pcre_malloc() is used by pcre_maketables() (and thus the result should be passed to pcre_free() after use),
https://pcre.org/current/doc/html/pcre2_maketables.html says pcre2_maketables() uses malloc(3), so that pointer needs to go to free(3) at the end. Missed the second part in my earlier reply.
Quoted 8 lines
Quoted 3 lines
Perhaps just UNLEAK that thing? There is only a single way to build it and we can reuse it throughout the lifetime of the program, so there is no real need to clean it up before the OS does.
That would be a better fit if it would be created once in
cmd_grepand then shared with all worker threads (which I thought would be nice to do in the future anyway), but this change was trying to be conservative and just to the minimum to close the leak.
Sure. I wonder how sharing between threads would influence performance..
René
PATCH v2 1/1 grep: avoid leak of chartables in PCRE2
In reply to Carlo Marcelo Arenas Belón94da9193a6 ("grep: add support for PCRE v2", 2017-06-01) introduced a small memory leak visible with valgrind in t7813.
Complete the creation of a PCRE2 specific variable that was missing from the original change and free the generated table just like it is done for PCRE1.
The table cleanup use free as there is no global context defined when it was created (pcre2_maketables is passed a NULL pointer) but if that gets ever changed will need to be updated in tandem.
---
V2:
* better document why free is used as suggested by René
* avoid reusing PCRE1 variable for easy of maintenance (per Ævar)
grep.c | 7 ++++--- grep.h | 1 + 2 files changed, 5 insertions(+), 3 deletions(-)
grep.c +4 -3
@@ -488,7 +488,6 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *optPCRE2_UCHAR errbuf[256 ];PCRE2_SIZE erroffset;int options =PCRE2_MULTILINE ; -const uint8_t *character_tables =NULL ;int jitret;int patinforet;size_t jitsizearg;
@@ -499,9 +498,10 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *optif (opt->ignore_case) {if (has_non_ascii (p->pattern)) { - character_tables =pcre2_maketables (NULL ); + p->pcre2_tables =pcre2_maketables (NULL ); p->pcre2_compile_context =pcre2_compile_context_create (NULL ); -pcre2_set_character_tables (p->pcre2_compile_context, character_tables); +pcre2_set_character_tables (p->pcre2_compile_context, + p->pcre2_tables); } options |=PCRE2_CASELESS ; }
@@ -605,6 +605,7 @@ static void free_pcre2_pattern(struct grep_pat *p)pcre2_match_data_free (p->pcre2_match_data);pcre2_jit_stack_free (p->pcre2_jit_stack);pcre2_match_context_free (p->pcre2_match_context); + free((void *)p->pcre2_tables); }#else /* !USE_LIBPCRE2 */ static void compile_pcre2_pattern (struct grep_pat *p,const struct grep_opt *opt)
grep.h +1 -0
@@ -96,6 +96,7 @@ struct grep_pat {pcre2_compile_context *pcre2_compile_context;pcre2_match_context *pcre2_match_context;pcre2_jit_stack *pcre2_jit_stack; +const uint8_t *pcre2_tables;uint32_t pcre2_jit_on;kwset_t kws; unsigned fixed:1;
Signature · 1 line
2.22.0
Re: [PATCH v2] grep: avoid leak of chartables in PCRE2
Quoted 11 lines · Carlo Marcelo Arenas Belón
94da9193a6("grep: add support for PCRE v2", 2017-06-01) introduced a small memory leak visible with valgrind in t7813.Complete the creation of a PCRE2 specific variable that was missing from the original change and free the generated table just like it is done for PCRE1.
The table cleanup use free as there is no global context defined when it was created (
pcre2_maketablesis passed a NULL pointer) but if that gets ever changed will need to be updated in tandem.
Quoted 5 lines
Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>---
V2:
* better document why free is used as suggested by René
* avoid reusing PCRE1 variable for easy of maintenance (per Ævar)
Thanks; will queue.
Quoted 50 lines
grep.c | 7 ++++--- grep.h | 1 + 2 files changed, 5 insertions(+), 3 deletions(-)grep.c +4
-3@@ -488,7 +488,6 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *optPCRE2_UCHAR errbuf[256 ];PCRE2_SIZE erroffset;int options =PCRE2_MULTILINE ; -const uint8_t *character_tables =NULL ;int jitret;int patinforet;size_t jitsizearg;@@ -499,9 +498,10 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *optif (opt->ignore_case) {if (has_non_ascii (p->pattern)) { - character_tables =pcre2_maketables (NULL ); + p->pcre2_tables =pcre2_maketables (NULL ); p->pcre2_compile_context =pcre2_compile_context_create (NULL ); -pcre2_set_character_tables (p->pcre2_compile_context, character_tables); +pcre2_set_character_tables (p->pcre2_compile_context, + p->pcre2_tables); } options |=PCRE2_CASELESS ; }@@ -605,6 +605,7 @@ static void free_pcre2_pattern(struct grep_pat *p)pcre2_match_data_free (p->pcre2_match_data);pcre2_jit_stack_free (p->pcre2_jit_stack);pcre2_match_context_free (p->pcre2_match_context); + free((void *)p->pcre2_tables); }#else /* !USE_LIBPCRE2 */ static void compile_pcre2_pattern (struct grep_pat *p,const struct grep_opt *opt)grep.h +1
-0@@ -96,6 +96,7 @@ struct grep_pat {pcre2_compile_context *pcre2_compile_context;pcre2_match_context *pcre2_match_context;pcre2_jit_stack *pcre2_jit_stack; +const uint8_t *pcre2_tables;uint32_t pcre2_jit_on;kwset_t kws; unsigned fixed:1;
Re: [PATCH v2] grep: avoid leak of chartables in PCRE2
Junio
Thanks for fixing the conflicts in pu as well; apologize if I could had made it easier by doings things differently
Carlo
Re: [PATCH v2] grep: avoid leak of chartables in PCRE2
Quoted 2 lines · Carlo Arenas
Thanks for fixing the conflicts in pu as well; apologize if I could had made it easier by doings things differently
With many topics in flight, conflicts between topics happen all the time. Thanks for checking to catch a mismerge.