diff: add -I<regex> that ignores matching changes
cover diff: add -I<regex> that ignores matching changes
This patch series adds a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I option in standalone diff utilities and can be used e.g. to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
I personally found this feature handy quite a few times while reviewing patches and a quick web search yields some questions from other people looking for something like this, so I thought I would take a shot at submitting it upstream.
Please note that while this patch series in its current shape builds, seems to pass tests, and appears to work as I would expect it to, this is my first attempt at contributing to Git and I did come across a few stumbling blocks:
- Instead of adding an extra flag for 'xdl_opts', would it be better to make sure xpparam_t structures are always zero-initialized before xdl_diff() is called (see commit message for patch 1 for context)?
- If this patch series is accepted, should xdl_mark_ignorable() be renamed to something like xdl_mark_ignorable_blank() for slightly improved code clarity?
- Would a more thorough explanation of the option help or hurt?
- Given that hunk emitting rules are the same for the -I option proposed in this patch series as for --ignore-blank-lines, is it necessary to duplicate the (impressively extensive!) --inter-hunk-context tests in t/t4015-diff-whitespace.sh for -I? I decided against doing that for the time being (though I did add some basic tests for such scenarios in patch 2, see the "with -U1" test).
- Should tests be added in a separate commit? This is what I did as I thought it would help with readability, but...
Thanks in advance for taking a look!
Michał Kępień (2):
diff: add -I<regex> that ignores matching changes
t: add -I<regex> tests
Documentation/diff-options.txt | 3 + diff.c | 16 ++ diff.h | 2 + t/t4069-diff-ignore-regex.sh | 330 +++++++++++++++++++++++++++++++++ xdiff/xdiff.h | 2 + xdiff/xdiffi.c | 36 ++++ 6 files changed, 389 insertions(+) create mode 100755 t/t4069-diff-ignore-regex.sh
Signature · 1 line
2.28.0
PATCH 1/2 diff: add -I<regex> that ignores matching changes
Add a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I option in standalone diff utilities and can be used e.g. to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
Use the 'ignore' field of xdchange_t for marking a change as ignored or not. Since the same field is used by --ignore-blank-lines, identical hunk emitting rules apply for --ignore-blank-lines and -I. These two options can also be used together at the same time.
Apart from adding a new field to struct diff_options, also define a new flag, XDF_IGNORE_REGEX, for the 'xdl_opts' field of that structure. This is done because the xpparam_t structure (which is used for passing around the regular expression supplied to -I) is not zeroed out in all call stacks involving xdl_diff() and thus only performing a NULL check on xpp->ignore_regex could result in xdl_mark_ignorable_regex() treating garbage on the stack as a regular expression. As the 'flags' field of xpparam_t is initialized in all call stacks involving xdl_diff(), adding a flag check avoids that problem.
--- Documentation/diff-options.txt | 3 +++ diff.c | 16 +++++++++++++++ diff.h | 2 ++ xdiff/xdiff.h | 2 ++ xdiff/xdiffi.c | 36 ++++++++++++++++++++++++++++++++++ 5 files changed, 59 insertions(+)
Documentation/diff-options.txt +3 -0
@@ -687,6 +687,9 @@ endif::git-format-patch[] --ignore-blank-lines:: Ignore changes whose lines are all blank. +-I<regex>:: + Ignore changes whose all lines match <regex>. + --inter-hunk-context=<lines>:: Show the context between diff hunks, up to the specified number of lines, thereby fusing hunks that are close to each other.
diff.c +16 -0
@@ -3587,6 +3587,7 @@ static void builtin_diff(const char *name_a,if (header.len && !o->flags.suppress_diff_headers) ecbdata.header = &header; xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;
@@ -3716,6 +3717,7 @@ static void builtin_diffstat(const char *name_a, const char *name_b,memset (&xpp,0 ,sizeof (xpp));memset (&xecfg,0 ,sizeof (xecfg)); xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;
@@ -5203,6 +5205,17 @@ static int diff_opt_patience(const struct option *opt,return 0 ; } +static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; + +BUG_ON_OPT_NEG (unset); + options->xdl_opts |=XDF_IGNORE_REGEX ; + options->ignore_regex = arg; +return 0 ; +} +static int diff_opt_pickaxe_regex (const struct option *opt,const char *arg,int unset) {
@@ -5491,6 +5504,9 @@ static void prep_parse_options(struct diff_options *options)OPT_BIT_F (0 ,"ignore-blank-lines" , &options->xdl_opts,N_ ("ignore changes whose lines are all blank" ),XDF_IGNORE_BLANK_LINES ,PARSE_OPT_NONEG ), +OPT_CALLBACK_F ('I' ,NULL , options,N_ ("<regex>" ), +N_ ("ignore changes whose all lines match <regex>" ), +0 , diff_opt_ignore_regex),OPT_BIT (0 ,"indent-heuristic" , &options->xdl_opts,N_ ("heuristic to shift diff hunk boundaries for easy reading" ), XDF_INDENT_HEURISTIC),
diff.h +2 -0
@@ -234,6 +234,8 @@ struct diff_options {*/ const char *pickaxe; +const char *ignore_regex; +const char *single_follow;const char *a_prefix, *b_prefix;const char *line_prefix;
xdiff/xdiff.h +2 -0
@@ -40,6 +40,7 @@ extern "C" { XDF_IGNORE_CR_AT_EOL)#define XDF_IGNORE_BLANK_LINES (1 << 7) +#define XDF_IGNORE_REGEX (1 << 8)#define XDF_PATIENCE_DIFF (1 << 14)#define XDF_HISTOGRAM_DIFF (1 << 15)
@@ -78,6 +79,7 @@ typedef struct s_mmbuffer { typedef struct s_xpparam { unsigned long flags; + const char *ignore_regex; /* See Documentation/diff-options.txt. */ char **anchors;
xdiff/xdiffi.c +36 -0
@@ -1019,6 +1019,39 @@ static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) } } +static void +xdl_mark_ignorable_regex (xdchange_t *xscr,const xdfenv_t *xe, +const char *ignore_regex) +{ +xdchange_t *xch; +regex_t regex; + +if (regcomp (®ex, ignore_regex,REG_EXTENDED |REG_NEWLINE )) +die ("invalid regex: %s" , ignore_regex); + +for (xch = xscr; xch; xch = xch->next) { +regmatch_t regmatch; +xrecord_t **rec; +int ignore =1 ; +long i; + + rec = &xe->xdf1.recs[xch->i1]; +for (i =0 ; i < xch->chg1 && ignore; i++) + ignore = !regexec_buf (®ex, rec[i]->ptr, rec[i]->size, +1 , ®match,0 ); + + rec = &xe->xdf2.recs[xch->i2]; +for (i =0 ; i < xch->chg2 && ignore; i++) + ignore = !regexec_buf (®ex, rec[i]->ptr, rec[i]->size, +1 , ®match,0 ); + +/* +* Do not override --ignore-blank-lines. +*/ + xch->ignore |= ignore; + } +} +int xdl_diff (mmfile_t *mf1,mmfile_t *mf2,xpparam_t const *xpp,xdemitconf_t const *xecfg,xdemitcb_t *ecb) {xdchange_t *xscr;
@@ -1040,6 +1073,9 @@ int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,if (xpp->flags &XDF_IGNORE_BLANK_LINES )xdl_mark_ignorable (xscr, &xe, xpp->flags); +if ((xpp->flags &XDF_IGNORE_REGEX ) && xpp->ignore_regex) +xdl_mark_ignorable_regex (xscr, &xe, xpp->ignore_regex); +if (ef (&xe, xscr, ecb, xecfg) <0 ) {xdl_free_script (xscr);
Signature · 1 line
2.28.0
Quoted 9 lines · Michał Kępień
Apart from adding a new field to struct
diff_options, also define a new flag,XDF_IGNORE_REGEX, for the 'xdl_opts' field of that structure. This is done because thexpparam_tstructure (which is used for passing around the regular expression supplied to -I) is not zeroed out in all call stacks involvingxdl_diff()and thus only performing a NULL check onxpp->ignore_regexcould result inxdl_mark_ignorable_regex()treating garbage on the stack as a regular expression. As the 'flags' field ofxpparam_tis initialized in all call stacks involvingxdl_diff(), adding a flag check avoids that problem.
You mentioned in your cover letter about this, and I tend to agree that this feels like a hack, at least from the first glance. The next feature that wants to have an optional pointer in xpparam_t and have the code behave differently with the data pointed by it would need to waste another bit the same way. Do you already know (read:
I am not asking you to dig to find out immediately---just asking if you already know the answer) if there is an inherent reason why they cannot be memset(0) before use? It seems like a much better approach to ensure that we clear the structure. Doesn't existing anchors array share the same issue (at least anchors_nr must be cleared if there is no interesting anchors) already? IOW, should "git grep anchors_nr" be a valid way to identify _all_ places where you need to clear the ignore_regex field?
Documentation/diff-options.txt endif::git-format-patch[] · line 692
Quoted 3 lines
+-I<regex>::
Documentation/diff-options.txt +2
-0+ Ignore changes whose all lines match <regex>. +
The implementation seems to allow only one regex, but I suspect we'd want to mimic
$ printf "%s\n" a a a >test_a
$ printf "%s\n" b b b >test_b
$ diff -Ia test_a test_b
$ diff -Ib test_a test_b
$ diff -Ia -Ib test_a test_band until that happens, the explanation needs to say something about earlier <regex> being discarded when this option is given more than once.
diff.c static int diff_opt_patience(const struct option *opt, · line 5215
Quoted 12 lines
diff.c +8
-0@@ -5203,6 +5205,17 @@ static int diff_opt_patience(const struct option *opt,return 0 ; } +static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; + +BUG_ON_OPT_NEG (unset); + options->xdl_opts |=XDF_IGNORE_REGEX ; + options->ignore_regex = arg;
When given twice or more, the earlier value gets lost (it does not leak, luckily, though).
diff.c static int diff_opt_patience(const struct option *opt, · line 5217
Quoted 2 lines
diff.c +2
-0+return 0 ; +}
If we somehow can lose the use of XDF_IGNORE_REGEX bit, we do not have to have this as a callback. Instead, it can be OPT_STRING with the current semantics of "only the last one is used", or we can use OPT_STRING_LIST to keep all the expressions.
On the other hand, I wonder if it would be a valid approach to make the new field(s) in diff_options a "regex_t *ignore_regex" (and add an "int ignore_regex_nr" next to it, if we shoot for multiple -I options), instead of "const char *". For that, we would need a callback even without XDF_IGNORE_REGEX bit having to futz with xdl_opts field.
Doing so would give us a chance to compile and notice a malformed regular expression in diff.c, before it hits xdiff/ layer, which may or may not be a good thing.
Review static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) · line 1020
Quoted 3 lines
(quoted diff) +0
-0@@ -1019,6 +1019,39 @@ static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) } }
I agree with what you said in the cover letter that it would be a good idea to name the existing xdl_mark_ignorable() and the new one in such a way that they look similar and parallel, by renaming the former to xdl_mark_ignorable_lines or something.
xdiff/xdiffi.c static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) · line 1024
Quoted 3 lines
xdiff/xdiffi.c +3
-0+static void +xdl_mark_ignorable_regex (xdchange_t *xscr,const xdfenv_t *xe, +const char *ignore_regex)
I know some coding standard do chomp line immediately before the function name (I grew up with one), but that is not what this project uses, and judging from the surrounding code, it is not what the upstream xdiff source we borrowed uses, either.
xdiff/xdiffi.c static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) · line 1030
Quoted 6 lines
xdiff/xdiffi.c +6
-0+{ +xdchange_t *xch; +regex_t regex; + +if (regcomp (®ex, ignore_regex,REG_EXTENDED |REG_NEWLINE )) +die ("invalid regex: %s" , ignore_regex);
If we compile in diff.c layer and pass regex_t down here, we won't have to fail here this deep in the callchain.
Quoted 9 lines
xdiff/xdiffi.c +9
-0+for (xch = xscr; xch; xch = xch->next) { +regmatch_t regmatch; +xrecord_t **rec; +int ignore =1 ; +long i; + + rec = &xe->xdf1.recs[xch->i1]; +for (i =0 ; i < xch->chg1 && ignore; i++) + ignore = !regexec_buf(®ex, rec[i]->ptr, rec[i]->size,
xdiff/xdiffi.c static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) · line 1051
Quoted 10 lines
xdiff/xdiffi.c +10
-0+ 1, ®match, 0); + rec = &xe->xdf2.recs[xch->i2]; +for (i =0 ; i < xch->chg2 && ignore; i++) + ignore = !regexec_buf (®ex, rec[i]->ptr, rec[i]->size, +1 , ®match,0 ); + +/* +* Do not override --ignore-blank-lines. +*/ + xch->ignore |= ignore;
Well, you could optimize out the whole regexp matching by adding
if (xch->ignore) continue;
before the two loops try to find a non-matching line, no?
Quoted 7 lines
xdiff/xdiffi.c +3
-0+ } +} +int xdl_diff (mmfile_t *mf1,mmfile_t *mf2,xpparam_t const *xpp,xdemitconf_t const *xecfg,xdemitcb_t *ecb) {xdchange_t *xscr;
xdiff/xdiffi.c int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, · line 1081
Quoted 9 lines
xdiff/xdiffi.c +3
-0@@ -1040,6 +1073,9 @@ int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,if (xpp->flags &XDF_IGNORE_BLANK_LINES )xdl_mark_ignorable (xscr, &xe, xpp->flags); +if ((xpp->flags &XDF_IGNORE_REGEX ) && xpp->ignore_regex) +xdl_mark_ignorable_regex (xscr, &xe, xpp->ignore_regex); +if (ef (&xe, xscr, ecb, xecfg) <0 ) {xdl_free_script (xscr);
Thanks for an exciting read ;-)
Quoted 10 lines
Quoted 9 lines
Apart from adding a new field to struct
diff_options, also define a new flag,XDF_IGNORE_REGEX, for the 'xdl_opts' field of that structure. This is done because thexpparam_tstructure (which is used for passing around the regular expression supplied to -I) is not zeroed out in all call stacks involvingxdl_diff()and thus only performing a NULL check onxpp->ignore_regexcould result inxdl_mark_ignorable_regex()treating garbage on the stack as a regular expression. As the 'flags' field ofxpparam_tis initialized in all call stacks involvingxdl_diff(), adding a flag check avoids that problem.
Quoted 5 lines
You mentioned in your cover letter about this, and I tend to agree that this feels like a hack, at least from the first glance. The next feature that wants to have an optional pointer in
xpparam_tand have the code behave differently with the data pointed by it would need to waste another bit the same way.
Agreed.
Quoted 5 lines
Do you already know (read:
I am not asking you to dig to find out immediately---just asking if you already know the answer) if there is an inherent reason why they cannot be memset(0) before use? It seems like a much better approach to ensure that we clear the structure.
I do not know of any reason for xpparam_t structures to not be zero-initialized. In fact, they _are_ zero-initialized most of the time; AFAICT, there are just three places in the tree in which they are not.
Would you like me to address that in a separate patch in this patch series?
Quoted 5 lines
Doesn't existing
anchors array share the same issue (at leastanchors_nrmust be cleared if there is no interesting anchors) already? IOW, should "git grepanchors_nr" be a valid way to identify _all_ places where you need to clear theignore_regexfield?
Yes, the 'anchors' and 'anchors_nr' fields of xpparam_t are also affected by the same problem, but the symptoms are more benign in their case because these fields are only used in xdiff/xpatience.c, i.e. when XDF_PATIENCE_DIFF is set in 'flags' - and as I already mentioned in commit messages, that field is always initialized properly, so there is currently no practical possibility of stack garbage being interpreted as e.g. a pointer to the anchor array.
Quoted 7 lines
Documentation/diff-options.txt endif::git-format-patch[] · line 692
Quoted 3 lines
+-I<regex>::
Documentation/diff-options.txt +2
-0+ Ignore changes whose all lines match <regex>. +The implementation seems to allow only one regex, but I suspect we'd want to mimic
Quoted 5 lines
$ printf "%s\n" a a a >test_a
$ printf "%s\n" b b b >test_b
$ diff -Ia test_a test_b
$ diff -Ib test_a test_b
$ diff -Ia -Ib test_a test_bAh, sure. After skimming through various man pages available online, I was not sure whether all standalone versions of diff which support -I allow that switch to be used multiple times and I thought it would be better to start with the simplest thing possible. If you would rather have the above implemented immediately, I can sure try that in v2.
Quoted 3 lines
and until that happens, the explanation needs to say something about earlier <regex> being discarded when this option is given more than once.
Sorry, where do I look for that explanation? I tried to make -I behave similarly to -G and -S, each of which can be specified more than once, but the last value provided overrides the earlier ones - and I could not find any mention of that behavior in the docs. Please help?
Quoted 4 lines
Quoted 4 lines
diff.c +0
-0@@ -5203,6 +5205,17 @@ static int diff_opt_patience(const struct option *opt,return 0 ; }
Quoted 11 lines
diff.c static int diff_opt_patience(const struct option *opt, · line 5215
Quoted 8 lines
diff.c +8
-0+static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; + +BUG_ON_OPT_NEG (unset); + options->xdl_opts |=XDF_IGNORE_REGEX ; + options->ignore_regex = arg;When given twice or more, the earlier value gets lost (it does not leak, luckily, though).
Right, as I mentioned above, I just wanted to start with something simple that resembles -G/-S. I will revise this part in v2 if you would like to see support for multiple regular expressions in this patch series.
Quoted 7 lines
diff.c static int diff_opt_patience(const struct option *opt, · line 5217
Quoted 2 lines
diff.c +2
-0+return 0 ; +}If we somehow can lose the use of
XDF_IGNORE_REGEXbit, we do not have to have this as a callback. Instead, it can beOPT_STRINGwith the current semantics of "only the last one is used", or we can useOPT_STRING_LISTto keep all the expressions.
I see, thanks for the hints!
Quoted 10 lines
On the other hand, I wonder if it would be a valid approach to make the new field(s) in
diff_optionsa "regex_t*ignore_regex" (and add an "intignore_regex_nr" next to it, if we shoot for multiple -I options), instead of "const char *". For that, we would need a callback even withoutXDF_IGNORE_REGEXbit having to futz withxdl_optsfield.Doing so would give us a chance to compile and notice a malformed regular expression in
diff.c, before it hits xdiff/ layer, which may or may not be a good thing.
I have not thought about this approach before, but it sounds elegant to me, at least at first glance - option parsing code sounds like the right place for sanitizing user-provided parameters. Doubly so if we take the multiple -I options approach. I will try this in v2.
Quoted 8 lines
Review static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) · line 1020
Quoted 3 lines
(quoted diff) +0
-0@@ -1019,6 +1019,39 @@ static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) } }I agree with what you said in the cover letter that it would be a good idea to name the existing
xdl_mark_ignorable()and the new one in such a way that they look similar and parallel, by renaming the former toxdl_mark_ignorable_linesor something.
Then I will do that in v2. Separate patch?
Quoted 8 lines
xdiff/xdiffi.c static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) · line 1024
Quoted 3 lines
xdiff/xdiffi.c +3
-0+static void +xdl_mark_ignorable_regex (xdchange_t *xscr,const xdfenv_t *xe, +const char *ignore_regex)I know some coding standard do chomp line immediately before the function name (I grew up with one), but that is not what this project uses, and judging from the surrounding code, it is not what the upstream xdiff source we borrowed uses, either.
Oh, right, sorry - force of habit. I will fix this in v2.
Quoted 9 lines
xdiff/xdiffi.c static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) · line 1030
Quoted 6 lines
xdiff/xdiffi.c +6
-0+{ +xdchange_t *xch; +regex_t regex; + +if (regcomp (®ex, ignore_regex,REG_EXTENDED |REG_NEWLINE )) +die ("invalid regex: %s" , ignore_regex);If we compile in
diff.clayer and passregex_tdown here, we won't have to fail here this deep in the callchain.
Agreed - I already responded to this remark above.
Quoted 22 lines
Quoted 9 lines
xdiff/xdiffi.c +9
-0+for (xch = xscr; xch; xch = xch->next) { +regmatch_t regmatch; +xrecord_t **rec; +int ignore =1 ; +long i; + + rec = &xe->xdf1.recs[xch->i1]; +for (i =0 ; i < xch->chg1 && ignore; i++) + ignore = !regexec_buf(®ex, rec[i]->ptr, rec[i]->size,xdiff/xdiffi.c static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) · line 1051
Quoted 10 lines
xdiff/xdiffi.c +10
-0+ 1, ®match, 0); + rec = &xe->xdf2.recs[xch->i2]; +for (i =0 ; i < xch->chg2 && ignore; i++) + ignore = !regexec_buf (®ex, rec[i]->ptr, rec[i]->size, +1 , ®match,0 ); + +/* +* Do not override --ignore-blank-lines. +*/ + xch->ignore |= ignore;Well, you could optimize out the whole regexp matching by adding
Quoted 4 lines
if (xch->ignore) continue;before the two loops try to find a non-matching line, no?
Ah, of course, it looks so obvious now that you pointed it out :-) I guess I was copy-past^W^W^Wtrying to make xdl_mark_ignorable_regex() look similar to xdl_mark_ignorable(). I will use your suggestion in v2.
Quoted 10 lines
Quoted 10 lines
xdiff/xdiffi.c +3
-0+ } +} +int xdl_diff (mmfile_t *mf1,mmfile_t *mf2,xpparam_t const *xpp,xdemitconf_t const *xecfg,xdemitcb_t *ecb) {xdchange_t *xscr;@@ -1040,6 +1073,9 @@ int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,if (xpp->flags &XDF_IGNORE_BLANK_LINES )xdl_mark_ignorable (xscr, &xe, xpp->flags);
Quoted 8 lines
xdiff/xdiffi.c int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, · line 1081
Quoted 6 lines
xdiff/xdiffi.c +3
-0+if ((xpp->flags &XDF_IGNORE_REGEX ) && xpp->ignore_regex) +xdl_mark_ignorable_regex (xscr, &xe, xpp->ignore_regex); +if (ef (&xe, xscr, ecb, xecfg) <0 ) {xdl_free_script (xscr);Thanks for an exciting read ;-)
My pleasure ;-) And thanks for taking a look. Sorry about the long turnaround time, but unfortunately this is the best I can do at the moment.
Signature · 2 lines
Best regards, Michał Kępień
Quoted 10 lines · Michał Kępień
Quoted 2 lines
cannot be memset(0) before use? It seems like a much better approach to ensure that we clear the structure.
I do not know of any reason for
xpparam_tstructures to not be zero-initialized. In fact, they _are_ zero-initialized most of the time; AFAICT, there are just three places in the tree in which they are not.Would you like me to address that in a separate patch in this patch series?
Yes, as a preliminary clean-up patch before the real/fun stuff ;-)
Quoted 19 lines
Quoted 7 lines
Documentation/diff-options.txt endif::git-format-patch[] · line 692
Quoted 3 lines
+-I<regex>::
Documentation/diff-options.txt +2
-0+ Ignore changes whose all lines match <regex>. +The implementation seems to allow only one regex, but I suspect we'd want to mimic
Quoted 5 lines
$ printf "%s\n" a a a >test_a $ printf "%s\n" b b b >test_b $ diff -Ia test_a test_b $ diff -Ib test_a test_b $ diff -Ia -Ib test_a test_bAh, sure. After skimming through various man pages available online, I was not sure whether all standalone versions of diff which support -I allow that switch to be used multiple times and I thought it would be better to start with the simplest thing possible. If you would rather have the above implemented immediately, I can sure try that in v2.
Quoted 5 lines
Quoted 3 lines
and until that happens, the explanation needs to say something about earlier <regex> being discarded when this option is given more than once.
Sorry, where do I look for that explanation?
You wrote "Ignore changes whose all lines match <regex>"; I was suggesting that we also need "when given more than once, all but the last <regex> are ignored" after that sentence, because that is not what people who know how -I works in versions of "diff" that support it expect.
But I think it should be trivial to make it a list of patterns and try to match against them in a loop. So let's support multiple patterns from the get-go.
Quoted 4 lines
I have not thought about this approach before, but it sounds elegant to me, at least at first glance - option parsing code sounds like the right place for sanitizing user-provided parameters. Doubly so if we take the multiple -I options approach. I will try this in v2.
Sounds good.
Quoted 6 lines
Quoted 4 lines
I agree with what you said in the cover letter that it would be a good idea to name the existing
xdl_mark_ignorable()and the new one in such a way that they look similar and parallel, by renaming the former toxdl_mark_ignorable_linesor something.Then I will do that in v2. Separate patch?
Given that the function has only one caller, I think it is better done within the same patch. xdl_mark_ignorable() as the name of the function is not wrong per-se, so it does not deserve to be renamed standalone in a preliminary clean-up patch---there is nothing to be cleaned up. The fact that we introduce a similarly tasked function makes its current name less desirable, so adjusting to the new world order is better done as part of the same patch.
Quoted 3 lines
My pleasure ;-) And thanks for taking a look. Sorry about the long turnaround time, but unfortunately this is the best I can do at the moment.
Pleasure is mutual ;-)
We've survived without -I for 15 years. Even a few more months would not hurt anybody. Take time, aim for a quality job, and most importantly, have fun.
Thanks.
PATCH 2/2 t: add -I<regex> tests
In reply to Michał KępieńExercise the new 'git diff -I<regex>' option in various scenarios to ensure it behaves as expected.
--- t/t4069-diff-ignore-regex.sh | 330 +++++++++++++++++++++++++++++++++++ 1 file changed, 330 insertions(+) create mode 100755 t/t4069-diff-ignore-regex.sh
t/t4069-diff-ignore-regex.sh new file +330 -0
@@ -0,0 +1,330 @@ +#!/bin/sh + +test_description='Test diff -I<regex>' + +. ./test-lib.sh +. "$TEST_DIRECTORY"/diff-lib.sh + +test_expect_success setup ' + test_seq 20 >x && + git update-index --add x +' + +test_expect_success 'one line changed' ' + test_seq 20 | sed "s/10/100/" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -7,7 +7,7 @@ + 7 + 8 + 9 + -10 + +100 + 11 + 12 + 13 + EOF + compare_diff_patch expected plain && + + # Both old and new line match regex - ignore change + git diff -I "^10" >actual && + test_must_be_empty actual && + + # Only old line matches regex - do not ignore change + git diff -I "^10\$" >actual && + compare_diff_patch plain actual && + + # Only new line matches regex - do not ignore change + git diff -I "^100" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'one line removed' ' + test_seq 20 | sed "10d" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -7,7 +7,6 @@ + 7 + 8 + 9 + -10 + 11 + 12 + 13 + EOF + compare_diff_patch expected plain && + + # Removed line matches regex - ignore change + git diff -I "^10" >actual && + test_must_be_empty actual && + + # Removed line does not match regex - do not ignore change + git diff -I "^101" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'one line added' ' + test_seq 21 >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -18,3 +18,4 @@ + 18 + 19 + 20 + +21 + EOF + compare_diff_patch expected plain && + + # Added line matches regex - ignore change + git diff -I "^21" >actual && + test_must_be_empty actual && + + # Added line does not match regex - do not ignore change + git diff -I "^212" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'last two lines changed' ' + test_seq 20 | sed "s/19/21/; s/20/22/" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -16,5 +16,5 @@ + 16 + 17 + 18 + -19 + -20 + +21 + +22 + EOF + compare_diff_patch expected plain && + + # All changed lines match regex - ignore change + git diff -I "^[12]" >actual && + test_must_be_empty actual && + + # Not all changed lines match regex - do not ignore change + git diff -I "^2" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'two non-adjacent lines removed in the same hunk' ' + test_seq 20 | sed "1d; 3d" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,6 +1,4 @@ + -1 + 2 + -3 + 4 + 5 + 6 + EOF + compare_diff_patch expected plain && + + # Both removed lines match regex - ignore hunk + git diff -I "^[1-3]" >actual && + test_must_be_empty actual && + + # First removed line does not match regex - do not ignore hunk + git diff -I "^[2-3]" >actual && + compare_diff_patch plain actual && + + # Second removed line does not match regex - do not ignore hunk + git diff -I "^[1-2]" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'two non-adjacent lines removed in the same hunk, with -U1' ' + test_seq 20 | sed "1d; 3d" >x && + + # Get plain diff + git diff -U1 >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,4 +1,2 @@ + -1 + 2 + -3 + 4 + EOF + compare_diff_patch expected plain && + + # Both removed lines match regex - ignore hunk + git diff -U1 -I "^[1-3]" >actual && + test_must_be_empty actual && + + # First removed line does not match regex, but is out of context - ignore second change + git diff -U1 -I "^[2-3]" >actual && + cat >second-change-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,2 +1 @@ + -1 + 2 + EOF + compare_diff_patch second-change-ignored actual && + + # Second removed line does not match regex, but is out of context - ignore first change + git diff -U1 -I "^[1-2]" >actual && + cat >first-change-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -2,3 +1,2 @@ + 2 + -3 + 4 + EOF + compare_diff_patch first-change-ignored actual +' + +test_expect_success 'multiple hunks' ' + test_seq 20 | sed "1d; 20d" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,4 +1,3 @@ + -1 + 2 + 3 + 4 + @@ -17,4 +16,3 @@ + 17 + 18 + 19 + -20 + EOF + compare_diff_patch expected plain && + + # Ignore both hunks + git diff -I "^[12]" >actual && + test_must_be_empty actual && + + # Only ignore first hunk + git diff -I "^1" >actual && + cat >first-hunk-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -17,4 +16,3 @@ + 17 + 18 + 19 + -20 + EOF + compare_diff_patch first-hunk-ignored actual && + + # Only ignore second hunk + git diff -I "^2" >actual && + cat >second-hunk-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,4 +1,3 @@ + -1 + 2 + 3 + 4 + EOF + compare_diff_patch second-hunk-ignored actual +' + +test_expect_success 'multiple hunks, with --ignore-blank-lines' ' + echo >x && + test_seq 21 >>x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,3 +1,4 @@ + + + 1 + 2 + 3 + @@ -18,3 +19,4 @@ + 18 + 19 + 20 + +21 + EOF + compare_diff_patch expected plain && + + # -I does not override --ignore-blank-lines - ignore both hunks + git diff --ignore-blank-lines -I ^21 >actual && + test_must_be_empty actual +' + +test_expect_success 'diffstat' ' + test_seq 20 | sed "s/^5/0/p; s/^15/10/; 16d" >x && + + # Get plain diffstat + git diff --stat >actual && + cat >expected <<-EOF && + x | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + EOF + test_cmp expected actual && + + # Ignore both hunks + git diff --stat -I "^[0-5]" >actual && + test_must_be_empty actual && + + # Only ignore first hunk + git diff --stat -I "^[05]" >actual && + cat >expected <<-EOF && + x | 3 +-- + 1 file changed, 1 insertion(+), 2 deletions(-) + EOF + test_cmp expected actual && + + # Only ignore second hunk + git diff --stat -I "^1" >actual && + cat >expected <<-EOF && + x | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + EOF + test_cmp expected actual +' + +test_expect_success 'invalid regex' ' + >x && + git diff -I "^[1" 2>&1 | grep "invalid regex: " +' + +test_done
Signature · 1 line
2.28.0
Quoted 7 lines · Michał Kępień
This patch series adds a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I option in standalone diff utilities and can be used e.g. to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
I am uncomfortably excited to see a change to quite a low-level code of the diff machinery. It does not happen every day ;-)
v2 cover diff: add -I<regex> that ignores matching changes
In reply to Michał KępieńThis patch series adds a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I option in standalone diff utilities and can be used e.g. to ignore changes which only affect code comments or to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
Changes from v1:
- Add a new preliminary cleanup patch which ensures xpparam_t structures are always zero-initialized. (This was a prerequisite for the next change below.)
- Do not add a new 'xdl_opts' flag to check whether -I was used; instead, just check whether the array of regular expressions to match against is non-NULL.
- Enable the -I option to be used multiple times. As a consequence of this, regular expressions are now "pre-compiled" in the option's callback (and passed around as an array of regex_t structures) rather than deep down in xdiff code. Add test cases exercising use of multiple -I options in the same git invocation. Update documentation accordingly.
- Rename xdl_mark_ignorable() to xdl_mark_ignorable_lines(), to indicate that it is logically a "sibling" of xdl_mark_ignorable_regex() rather than its "parent".
- Optimize xdl_mark_ignorable_regex() by making it immediately skip changes already marked as ignored by xdl_mark_ignorable_lines().
- Fix coding style issue in the prototype part of the definition of xdl_mark_ignorable_regex().
- Add "/* see Documentation/diff-options.txt */" comments for the fields added to struct diff_options and xpparam_t, mimicking the comments used for 'anchors', 'anchors_nr', and 'anchors_alloc'.
- Revise commit log messages to reflect all of the above.
Michał Kępień (3):
merge-base, xdiff: zero out xpparam_t structures
diff: add -I<regex> that ignores matching changes
t: add -I<regex> tests
Documentation/diff-options.txt | 4 + builtin/merge-tree.c | 1 + diff.c | 23 ++ diff.h | 4 + t/t4069-diff-ignore-regex.sh | 426 +++++++++++++++++++++++++++++++++ xdiff/xdiff.h | 4 + xdiff/xdiffi.c | 47 +++- xdiff/xhistogram.c | 2 + xdiff/xpatience.c | 2 + 9 files changed, 511 insertions(+), 2 deletions(-) create mode 100755 t/t4069-diff-ignore-regex.sh
Signature · 1 line
2.28.0
PATCH v2 1/3 merge-base, xdiff: zero out xpparam_t structures
xpparam_t structures are usually zero-initialized before their specific fields are assigned to, but there are three locations in the tree where that does not happen. Add the missing memset() calls in order to make initialization of xpparam_t structures consistent tree-wide and to prevent stack garbage from being used as field values.
--- builtin/merge-tree.c | 1 + xdiff/xhistogram.c | 2 ++ xdiff/xpatience.c | 2 ++ 3 files changed, 5 insertions(+)
builtin/merge-tree.c +1 -0
@@ -109,6 +109,7 @@ static void show_diff(struct merge_list *entry)xdemitconf_t xecfg;xdemitcb_t ecb; +memset (&xpp,0 ,sizeof (xpp)); xpp.flags =0 ;memset (&xecfg,0 ,sizeof (xecfg)); xecfg.ctxlen =3 ;
xdiff/xhistogram.c +2 -0
@@ -235,6 +235,8 @@ static int fall_back_to_classic_diff(xpparam_t const *xpp, xdfenv_t *env, int line1, int count1, int line2, int count2) { xpparam_t xpparam; + +memset (&xpparam,0 ,sizeof (xpparam)); xpparam.flags = xpp->flags & ~XDF_DIFF_ALGORITHM_MASK ; return xdl_fall_back_diff(env, &xpparam,
xdiff/xpatience.c +2 -0
@@ -318,6 +318,8 @@ static int fall_back_to_classic_diff(struct hashmap *map, int line1, int count1, int line2, int count2) { xpparam_t xpp; + +memset (&xpp,0 ,sizeof (xpp)); xpp.flags = map->xpp->flags & ~XDF_DIFF_ALGORITHM_MASK ; return xdl_fall_back_diff(map->env, &xpp,
Signature · 1 line
2.28.0
Re: [PATCH v2 1/3] merge-base, xdiff: zero out xpparam_t structures
Hi Michał,
xdiff/xhistogram.c static int fall_back_to_classic_diff(xpparam_t const *xpp, xdfenv_t *env, · line 239
Quoted 10 lines · Michał Kępień
xdiff/xhistogram.c +2
-0@@ -235,6 +235,8 @@ static int fall_back_to_classic_diff(xpparam_t const *xpp, xdfenv_t *env, int line1, int count1, int line2, int count2) { xpparam_t xpparam; + +memset (&xpparam,0 ,sizeof (xpparam));
A slightly more elegant way to do this would be
xpparam_t xpparam = { 0l };Or even
xpparam_t xpparam = XPPARAM_T_INIT;
with
#define XPPARAM_T_INIT { 0l, NULL, 0 }in xdiff/xdiff.h.
Thanks,
Dscho
Quoted 21 lines
xdiff/xhistogram.c +0
-0xpparam.flags = xpp->flags & ~XDF_DIFF_ALGORITHM_MASK ; return xdl_fall_back_diff(env, &xpparam,xdiff/xpatience.c +2
-1@@ -318,6 +318,8 @@ static int fall_back_to_classic_diff(struct hashmap *map, int line1, int count1, int line2, int count2) { xpparam_t xpp; + +memset (&xpp,0 ,sizeof (xpp)); xpp.flags = map->xpp->flags & ~XDF_DIFF_ALGORITHM_MASK ; return xdl_fall_back_diff(map->env, &xpp, --2.28.0
Re: [PATCH v2 1/3] merge-base, xdiff: zero out xpparam_t structures
Quoted 10 lines · Johannes Schindelin
xdiff/xhistogram.c static int fall_back_to_classic_diff(xpparam_t const *xpp, xdfenv_t *env, · line 239
Quoted 3 lines
xdiff/xhistogram.c +2
-0xpparam_t xpparam; + +memset (&xpparam,0 ,sizeof (xpparam));A slightly more elegant way to do this would be
xpparam_t xpparam = { 0l };Or even
Quoted 7 lines
xpparam_t xpparam = XPPARAM_T_INIT;with
#define XPPARAM_T_INIT { 0l, NULL, 0 }in
xdiff/xdiff.h.
Let's not suggest any of the above.
I think we had a recent thread [*1*] on which we agreed that "{ 0 }" is the way to spell "a naturally zero" initialization like this one, if we were to spell it out.
TBH, I'd say that memset() is also OK as-is in this codebase as an established way (git grep 'memset([^,]*, 0, sizeof' gives too many hits).
Thanks.
[Reference]
Re: [PATCH v2 1/3] merge-base, xdiff: zero out xpparam_t structures
In reply to Michał Kępieńbuiltin/merge-tree.c static void show_diff(struct merge_list *entry) · line 115
Quoted 12 lines · Michał Kępień
builtin/merge-tree.c +1
-0@@ -109,6 +109,7 @@ static void show_diff(struct merge_list *entry)xdemitconf_t xecfg;xdemitcb_t ecb; +memset (&xpp,0 ,sizeof (xpp)); xpp.flags =0 ;memset (&xecfg,0 ,sizeof (xecfg)); xecfg.ctxlen =3 ;
The existing "xpp.flags=0" can then go away, and as Dscho hinted,
xpparam_t xpp = {
.flags = 0,
};may also be a valid way to write this. What it says is that we want the entirety of xpp to be reasonably initialized, but we do care that its .flags field to be exactly zero.
xdiff/xhistogram.c static int fall_back_to_classic_diff(xpparam_t const *xpp, xdfenv_t *env, · line 240
Quoted 11 lines
xdiff/xhistogram.c +2
-0@@ -235,6 +235,8 @@ static int fall_back_to_classic_diff(xpparam_t const *xpp, xdfenv_t *env, int line1, int count1, int line2, int count2) { xpparam_t xpparam; + +memset (&xpparam,0 ,sizeof (xpparam)); xpparam.flags = xpp->flags & ~XDF_DIFF_ALGORITHM_MASK ;
Likewise. Giving an initializer to the local variable def, e.g.
xpparam_t xpparam = {
.flags = xpp->flags & ~XDF_DIFF_ALGORITHM_MASK,
};would allow us to lose one line.
xdiff/xpatience.c static int fall_back_to_classic_diff(struct hashmap *map, · line 323
Quoted 12 lines
xdiff/xhistogram.c +0
-0return xdl_fall_back_diff(env, &xpparam,xdiff/xpatience.c +2
-0@@ -318,6 +318,8 @@ static int fall_back_to_classic_diff(struct hashmap *map, int line1, int count1, int line2, int count2) { xpparam_t xpp; + +memset (&xpp,0 ,sizeof (xpp)); xpp.flags = map->xpp->flags & ~XDF_DIFF_ALGORITHM_MASK ;
Likewise.
xdiff/xpatience.c static int fall_back_to_classic_diff(struct hashmap *map, · line 325
Quoted 1 line
xdiff/xpatience.c +0
-0return xdl_fall_back_diff(map->env, &xpp,
But the patch is good as-is, given especially the way how xecfg is cleared the same way in builtin/merge-tree.c::show_diff().
Will queue. Thanks.
Re: [PATCH v2 1/3] merge-base, xdiff: zero out xpparam_t structures
Junio, Johannes,
Thank you for your feedback.
Quoted 2 lines
But the patch is good as-is, given especially the way how xecfg is cleared the same way in builtin/merge-tree.c::show_diff().
I just wanted to explain that I initially planned to take the "= { 0 }" approach and I started checking whether all xpparam_t structures in the source tree are stack-allocated, but I quickly noticed that, as Junio pointed out, existing code showed an inclination towards memset(), so I settled for that instead.
Quoted 1 line
Will queue. Thanks.
I have a process question: since the other two patches in this patch series will need a v3, would you like me to keep bundling patch 1 in this series or should I only send revised version of patches 2 & 3 in v3?
Signature · 2 lines
Best regards, Michał Kępień
PATCH v2 2/3 diff: add -I<regex> that ignores matching changes
In reply to Michał KępieńAdd a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I option in standalone diff utilities and can be used e.g. to ignore changes which only affect code comments or to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
Use the 'ignore' field of xdchange_t for marking a change as ignored or not. Since the same field is used by --ignore-blank-lines, identical hunk emitting rules apply for --ignore-blank-lines and -I. These two options can also be used together in the same git invocation (they are complementary to each other).
Rename xdl_mark_ignorable() to xdl_mark_ignorable_lines(), to indicate that it is logically a "sibling" of xdl_mark_ignorable_regex() rather than its "parent".
--- Documentation/diff-options.txt | 4 +++ diff.c | 23 +++++++++++++++++ diff.h | 4 +++ xdiff/xdiff.h | 4 +++ xdiff/xdiffi.c | 47 ++++++++++++++++++++++++++++++++-- 5 files changed, 80 insertions(+), 2 deletions(-)
Documentation/diff-options.txt +4 -0
@@ -687,6 +687,10 @@ endif::git-format-patch[] --ignore-blank-lines:: Ignore changes whose lines are all blank. +-I<regex>:: + Ignore changes whose all lines match <regex>. This option may + be specified more than once. + --inter-hunk-context=<lines>:: Show the context between diff hunks, up to the specified number of lines, thereby fusing hunks that are close to each other.
diff.c +23 -0
@@ -3587,6 +3587,8 @@ static void builtin_diff(const char *name_a,if (header.len && !o->flags.suppress_diff_headers) ecbdata.header = &header; xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; + xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;
@@ -3716,6 +3718,8 @@ static void builtin_diffstat(const char *name_a, const char *name_b,memset (&xpp,0 ,sizeof (xpp));memset (&xecfg,0 ,sizeof (xecfg)); xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; + xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;
@@ -5203,6 +5207,22 @@ static int diff_opt_patience(const struct option *opt,return 0 ; } +static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; +regex_t *regex; + +BUG_ON_OPT_NEG (unset); + regex =xcalloc (1 ,sizeof (*regex)); +if (regcomp (regex, arg,REG_EXTENDED |REG_NEWLINE )) +die ("invalid regex: %s" , arg); +ALLOC_GROW (options->ignore_regex, options->ignore_regex_nr +1 , + options->ignore_regex_alloc); + options->ignore_regex[options->ignore_regex_nr++] = regex; +return 0 ; +} +static int diff_opt_pickaxe_regex (const struct option *opt,const char *arg,int unset) {
@@ -5491,6 +5511,9 @@ static void prep_parse_options(struct diff_options *options)OPT_BIT_F (0 ,"ignore-blank-lines" , &options->xdl_opts,N_ ("ignore changes whose lines are all blank" ),XDF_IGNORE_BLANK_LINES ,PARSE_OPT_NONEG ), +OPT_CALLBACK_F ('I' ,NULL , options,N_ ("<regex>" ), +N_ ("ignore changes whose all lines match <regex>" ), +0 , diff_opt_ignore_regex),OPT_BIT (0 ,"indent-heuristic" , &options->xdl_opts,N_ ("heuristic to shift diff hunk boundaries for easy reading" ), XDF_INDENT_HEURISTIC),
diff.h +4 -0
@@ -234,6 +234,10 @@ struct diff_options {*/ const char *pickaxe; +/* see Documentation/diff-options.txt */ +regex_t **ignore_regex; +size_t ignore_regex_nr, ignore_regex_alloc; +const char *single_follow;const char *a_prefix, *b_prefix;const char *line_prefix;
xdiff/xdiff.h +4 -0
@@ -79,6 +79,10 @@ typedef struct s_mmbuffer { typedef struct s_xpparam { unsigned long flags; + /* See Documentation/diff-options.txt. */ + regex_t **ignore_regex; + size_t ignore_regex_nr; + /* See Documentation/diff-options.txt. */ char **anchors; size_t anchors_nr;
xdiff/xdiffi.c +45 -2
65 lines not on this page · raw message
Signature · 1 line
2.28.0
Hi Michał,
Quoted 7 lines · Michał Kępień
diff.c +3
-0@@ -5203,6 +5207,22 @@ static int diff_opt_patience(const struct option *opt,return 0 ; } +static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{
diff.c static int diff_opt_patience(const struct option *opt, · line 5222
Quoted 10 lines
diff.c +10
-0+struct diff_options *options = opt->value; +regex_t *regex; + +BUG_ON_OPT_NEG (unset); + regex =xcalloc (1 ,sizeof (*regex)); +if (regcomp (regex, arg,REG_EXTENDED |REG_NEWLINE )) +die ("invalid regex: %s" , arg); +ALLOC_GROW (options->ignore_regex, options->ignore_regex_nr +1 , + options->ignore_regex_alloc); + options->ignore_regex[options->ignore_regex_nr++] = regex;
A slightly more elegant way would be to have ignore_regex have the type regex_t * and use ALLOC_GROW_BY() (which zeroes the newly-added elements automagically).
Quoted 8 lines
diff.c +3
-0+return 0 ; +} +static int diff_opt_pickaxe_regex (const struct option *opt,const char *arg,int unset) {
diff.c static void prep_parse_options(struct diff_options *options) · line 5519
Quoted 9 lines
diff.c +3
-0@@ -5491,6 +5511,9 @@ static void prep_parse_options(struct diff_options *options)OPT_BIT_F (0 ,"ignore-blank-lines" , &options->xdl_opts,N_ ("ignore changes whose lines are all blank" ),XDF_IGNORE_BLANK_LINES ,PARSE_OPT_NONEG ), +OPT_CALLBACK_F ('I' ,NULL , options,N_ ("<regex>" ), +N_ ("ignore changes whose all lines match <regex>" ), +0 , diff_opt_ignore_regex),OPT_BIT (0 ,"indent-heuristic" , &options->xdl_opts,N_ ("heuristic to shift diff hunk boundaries for easy reading" ), XDF_INDENT_HEURISTIC),
Are we releasing the ignore_regex anywhere?
Thanks,
Dscho
Quoted 106 lines
diff.h +4
-0@@ -234,6 +234,10 @@ struct diff_options {*/ const char *pickaxe; +/* see Documentation/diff-options.txt */ +regex_t **ignore_regex; +size_t ignore_regex_nr, ignore_regex_alloc; +const char *single_follow;const char *a_prefix, *b_prefix;const char *line_prefix;xdiff/xdiff.h +4
-0@@ -79,6 +79,10 @@ typedef struct s_mmbuffer { typedef struct s_xpparam { unsigned long flags; + /* See Documentation/diff-options.txt. */ + regex_t **ignore_regex; + size_t ignore_regex_nr; + /* See Documentation/diff-options.txt. */ char **anchors; size_t anchors_nr;xdiff/xdiffi.c +45
-3@@ -998,7 +998,7 @@ static int xdl_call_hunk_func(xdfenv_t *xe, xdchange_t *xscr, xdemitcb_t *ecb,return 0 ; } -static void xdl_mark_ignorable (xdchange_t *xscr,xdfenv_t *xe,long flags) +static void xdl_mark_ignorable_lines (xdchange_t *xscr,xdfenv_t *xe,long flags) {xdchange_t *xch;@@ -1019,6 +1019,46 @@ static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) } } +static int record_matches_regex (xrecord_t *rec,xpparam_t const *xpp) { +regmatch_t regmatch; +int i; + +for (i =0 ; i < xpp->ignore_regex_nr; i++) +if (!regexec_buf (xpp->ignore_regex[i], rec->ptr, rec->size,1 , + ®match,0 )) +return 1 ; + +return 0 ; +} + +static void xdl_mark_ignorable_regex (xdchange_t *xscr,const xdfenv_t *xe, +xpparam_t const *xpp) +{ +xdchange_t *xch; + +for (xch = xscr; xch; xch = xch->next) { +xrecord_t **rec; +int ignore =1 ; +long i; + +/* +* Do not override --ignore-blank-lines. +*/ +if (xch->ignore) +continue ; + + rec = &xe->xdf1.recs[xch->i1]; +for (i =0 ; i < xch->chg1 && ignore; i++) + ignore =record_matches_regex (rec[i], xpp); + + rec = &xe->xdf2.recs[xch->i2]; +for (i =0 ; i < xch->chg2 && ignore; i++) + ignore =record_matches_regex (rec[i], xpp); + + xch->ignore = ignore; + } +} +int xdl_diff (mmfile_t *mf1,mmfile_t *mf2,xpparam_t const *xpp,xdemitconf_t const *xecfg,xdemitcb_t *ecb) {xdchange_t *xscr;@@ -1038,7 +1078,10 @@ int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, }if (xscr) {if (xpp->flags &XDF_IGNORE_BLANK_LINES ) -xdl_mark_ignorable (xscr, &xe, xpp->flags); +xdl_mark_ignorable_lines (xscr, &xe, xpp->flags); + +if (xpp->ignore_regex) +xdl_mark_ignorable_regex (xscr, &xe, xpp);if (ef (&xe, xscr, ecb, xecfg) <0 ) { --2.28.0
Quoted 17 lines · Johannes Schindelin
Hi Michał,
Quoted 13 lines · Michał Kępień
diff.c +9
-0@@ -5203,6 +5207,22 @@ static int diff_opt_patience(const struct option *opt,return 0 ; } +static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; +regex_t *regex; + +BUG_ON_OPT_NEG (unset); + regex =xcalloc (1 ,sizeof (*regex)); +if (regcomp (regex, arg,REG_EXTENDED |REG_NEWLINE ))
Quoted 8 lines
diff.c static int diff_opt_patience(const struct option *opt, · line 5222
Quoted 4 lines
diff.c +4
-0+die ("invalid regex: %s" , arg); +ALLOC_GROW (options->ignore_regex, options->ignore_regex_nr +1 , + options->ignore_regex_alloc); + options->ignore_regex[options->ignore_regex_nr++] = regex;A slightly more elegant way would be to have
ignore_regexhave the typeregex_t *and useALLOC_GROW_BY()(which zeroes the newly-added elements automagically).
It may be "elegant", but we we know if it is "correct" on everybody's implementation of regex_t?
A struct like
struct foo {
char *str;
char in_place_buffer[10];
};where str points at in_place_buffer[] only when it needs to point at a very short string, and points at an allocated memory on heap, can not be safely copied and used, and an array of such a struct breaks when ALLOC_GROW_BY() needs to call realloc(). Keeping an array of pointers to regex_t and growing it would not break even for such a structure type.
Since we cannot rely on the implementation detail of regex_t on a single platform, I'd rather not to see anybody suggest such an "elegant" approach.
Thanks.
Hi Junio,
Quoted 45 lines · Junio C Hamano
Quoted 17 lines · Johannes Schindelin
Hi Michał,
Quoted 13 lines · Michał Kępień
diff.c +9
-0@@ -5203,6 +5207,22 @@ static int diff_opt_patience(const struct option *opt,return 0 ; } +static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; +regex_t *regex; + +BUG_ON_OPT_NEG (unset); + regex =xcalloc (1 ,sizeof (*regex)); +if (regcomp (regex, arg,REG_EXTENDED |REG_NEWLINE ))Quoted 8 lines
diff.c static int diff_opt_patience(const struct option *opt, · line 5222
Quoted 4 lines
diff.c +4
-0+die ("invalid regex: %s" , arg); +ALLOC_GROW (options->ignore_regex, options->ignore_regex_nr +1 , + options->ignore_regex_alloc); + options->ignore_regex[options->ignore_regex_nr++] = regex;A slightly more elegant way would be to have
ignore_regexhave the typeregex_t *and useALLOC_GROW_BY()(which zeroes the newly-added elements automagically).It may be "elegant", but we we know if it is "correct" on everybody's implementation of
regex_t?A struct like
struct foo { char *str; char in_place_buffer[10]; };where str points at in_place_buffer[] only when it needs to point at a very short string, and points at an allocated memory on heap, can not be safely copied and used, and an array of such a struct breaks when
ALLOC_GROW_BY()needs to callrealloc(). Keeping an array of pointers toregex_tand growing it would not break even for such a structure type.
Quoted 3 lines
Since we cannot rely on the implementation detail of
regex_ton a single platform, I'd rather not to see anybody suggest such an "elegant" approach.
True, such an implementation detail is totally conceivable.
Thanks for the sanity check.
Having said that, my suggestion to use ALLOC_GROW_BY() was triggered by the use of xcalloc(), and now that I think further about it, an xmalloc() should be plenty enough: regcomp() does not need that struct to be initialized to all zero.
Ciao,
Dscho
Quoted 3 lines · Johannes Schindelin
the use of
xcalloc(), and now that I think further about it, anxmalloc()should be plenty enough:regcomp()does not need that struct to be initialized to all zero.
Yup, I think it makes sense.
Quoted 5 lines
Quoted 3 lines
the use of
xcalloc(), and now that I think further about it, anxmalloc()should be plenty enough:regcomp()does not need that struct to be initialized to all zero.Yup, I think it makes sense.
Sure thing, I will switch to xmalloc() in v3. xcalloc()'s semantics ("allocate one structure of this size") appealed to me, but there is indeed no need to zero-initialize the allocated memory before passing it to regcomp().
Signature · 2 lines
Best regards, Michał Kępień
Quoted 2 lines · Michał Kępień
... xcalloc()'s semantics
("allocate one structure of this size") appealed to me, ...
FWIW, it appeals to me, too ;-) But if we are going to pass it to regcomp(), there is no point clearing it beforehand.
Hi Johannes,
Quoted 4 lines
Quoted 4 lines
diff.c +0
-0@@ -5491,6 +5511,9 @@ static void prep_parse_options(struct diff_options *options)OPT_BIT_F (0 ,"ignore-blank-lines" , &options->xdl_opts,N_ ("ignore changes whose lines are all blank" ), XDF_IGNORE_BLANK_LINES, PARSE_OPT_NONEG),
Quoted 8 lines
diff.c static void prep_parse_options(struct diff_options *options) · line 5519
Quoted 6 lines
diff.c +3
-0+OPT_CALLBACK_F ('I' ,NULL , options,N_ ("<regex>" ), +N_ ("ignore changes whose all lines match <regex>" ), +0 , diff_opt_ignore_regex),OPT_BIT (0 ,"indent-heuristic" , &options->xdl_opts,N_ ("heuristic to shift diff hunk boundaries for easy reading" ), XDF_INDENT_HEURISTIC),Are we releasing the
ignore_regexanywhere?
Oops, I tried to mimic what is done for 'anchors', and I failed to notice that apparently the elements of the options->anchors array are only free()'d when --patience is also used and the array pointer itself is never free()'d at all. Given this, I believe I need to fix diff_opt_ignore_regex() in patch 2 and also make sure that the memory allocated in diff_opt_anchored() gets properly released - in another preliminary clean-up patch?
At first glance, diff_flush() - specifically the part below the 'free_queue' label - looks like a sane place to free() things. Am I mistaken?
Signature · 2 lines
Best regards, Michał Kępień
Hi Michał,
Quoted 23 lines · Michał Kępień
Hi Johannes,
Quoted 4 lines
Quoted 4 lines
diff.c +0
-0@@ -5491,6 +5511,9 @@ static void prep_parse_options(struct diff_options *options)OPT_BIT_F (0 ,"ignore-blank-lines" , &options->xdl_opts,N_ ("ignore changes whose lines are all blank" ), XDF_IGNORE_BLANK_LINES, PARSE_OPT_NONEG),Quoted 8 lines
diff.c static void prep_parse_options(struct diff_options *options) · line 5519
Quoted 6 lines
diff.c +3
-0+OPT_CALLBACK_F ('I' ,NULL , options,N_ ("<regex>" ), +N_ ("ignore changes whose all lines match <regex>" ), +0 , diff_opt_ignore_regex),OPT_BIT (0 ,"indent-heuristic" , &options->xdl_opts,N_ ("heuristic to shift diff hunk boundaries for easy reading" ), XDF_INDENT_HEURISTIC),Are we releasing the
ignore_regexanywhere?Oops, I tried to mimic what is done for 'anchors', and I failed to notice that apparently the elements of the
options->anchorsarray are only free()'d when --patience is also used and the array pointer itself is never free()'d at all. Given this, I believe I need to fixdiff_opt_ignore_regex()in patch 2 and also make sure that the memory allocated indiff_opt_anchored()gets properly released - in another preliminary clean-up patch?
Quoted 3 lines
At first glance,
diff_flush()- specifically the part below the 'free_queue' label - looks like a sane place tofree()things. Am I mistaken?
Oh wow, from a cursory look it seems as if the diff machinery was not exactly careful with releasing memory. I might be mistaken, but if I am not, then this would deserve a separate patch series, methinks.
Ciao,
Dscho
Quoted 3 lines · Johannes Schindelin
Oh wow, from a cursory look it seems as if the diff machinery was not exactly careful with releasing memory. I might be mistaken, but if I am not, then this would deserve a separate patch series, methinks.
I wouldn't be surprised if newer parts of it is much less careful than the older parts of the machinery. Most callers of the diff machinery, including "log -p" that repeatedly generates patches, would however run just a single setup for the entire series of diff invocations before tearing it down AFIAR, so leaking the result of parsing the command line option may not be such a big deal to these callers. But some callers added recently may not follow the access pattern, in which case the machinery may have to be made more leakproof.
Thanks.
Hi Johannes,
Quoted 3 lines
Oh wow, from a cursory look it seems as if the diff machinery was not exactly careful with releasing memory. I might be mistaken, but if I am not, then this would deserve a separate patch series, methinks.
Oh, okay. Since that sounds like a non-trivial task, I should probably finish this patch series first, making sure it releases memory properly, and only then move on to looking for memory leaks in diff.c.
Signature · 2 lines
Best regards, Michał Kępień
Quoted 15 lines · Michał Kępień
Add a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I option in standalone diff utilities and can be used e.g. to ignore changes which only affect code comments or to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
Use the 'ignore' field of
xdchange_tfor marking a change as ignored or not. Since the same field is used by --ignore-blank-lines, identical hunk emitting rules apply for --ignore-blank-lines and -I. These two options can also be used together in the same git invocation (they are complementary to each other).
Quoted 5 lines
Rename
xdl_mark_ignorable()toxdl_mark_ignorable_lines(), to indicate that it is logically a "sibling" ofxdl_mark_ignorable_regex()rather than its "parent".Signed-off-by Michał Kępień <[email protected]>
Well explained.
Quoted 1 line
+-I<regex>::
Since we are mimicking other folks' feature, giving also the
--ignore-matching-lines=<regex>
synonym to that their users are familiar with would be a good idea, no?
Quoted 37 lines
Documentation/diff-options.txt +3
-0+ Ignore changes whose all lines match <regex>. This option may + be specified more than once. + --inter-hunk-context=<lines>:: Show the context between diff hunks, up to the specified number of lines, thereby fusing hunks that are close to each other.diff.c +9
-0@@ -3587,6 +3587,8 @@ static void builtin_diff(const char *name_a,if (header.len && !o->flags.suppress_diff_headers) ecbdata.header = &header; xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; + xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;@@ -3716,6 +3718,8 @@ static void builtin_diffstat(const char *name_a, const char *name_b,memset (&xpp,0 ,sizeof (xpp));memset (&xecfg,0 ,sizeof (xecfg)); xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; + xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;@@ -5203,6 +5207,22 @@ static int diff_opt_patience(const struct option *opt,return 0 ; } +static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; +regex_t *regex;
diff.c static int diff_opt_patience(const struct option *opt, · line 5224
Quoted 10 lines
diff.c +10
-0+ +BUG_ON_OPT_NEG (unset); + regex =xcalloc (1 ,sizeof (*regex)); +if (regcomp (regex, arg,REG_EXTENDED |REG_NEWLINE )) +die ("invalid regex: %s" , arg); +ALLOC_GROW (options->ignore_regex, options->ignore_regex_nr +1 , + options->ignore_regex_alloc); + options->ignore_regex[options->ignore_regex_nr++] = regex; +return 0 ; +}
Don't these parse-options callback functions have a way to tell the caller die instead of them themselves dying like this? Would it work better to "return error(... message ...)", or would that give two error messages? In anycase, this is end-user facing, so we'd want it to be localizable, e.g.
die(_("invalid regexp given to -I: '%s'", arg));probably.
diff.c static int diff_opt_patience(const struct option *opt, · line 5228
Quoted 3 lines
diff.c +0
-0static int diff_opt_pickaxe_regex (const struct option *opt,const char *arg,int unset) {
Other than that, nicely done.
Thanks.
Quoted 8 lines
Quoted 1 line
+-I<regex>::
Since we are mimicking other folks' feature, giving also the
--ignore-matching-lines=<regex>
synonym to that their users are familiar with would be a good idea, no?
Some diff implementations (e.g. the OpenBSD one) lack the long option and I was aiming for the "greatest common divisor" for all diff implementations, but sure, I do not see how adding the synonym could hurt. I will do that in v3.
Quoted 16 lines
Quoted 16 lines
diff.c +12
-0@@ -5203,6 +5207,22 @@ static int diff_opt_patience(const struct option *opt,return 0 ; } +static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; +regex_t *regex; + +BUG_ON_OPT_NEG (unset); + regex =xcalloc (1 ,sizeof (*regex)); +if (regcomp (regex, arg,REG_EXTENDED |REG_NEWLINE )) +die ("invalid regex: %s" , arg); +ALLOC_GROW (options->ignore_regex, options->ignore_regex_nr +1 , + options->ignore_regex_alloc);
Quoted 8 lines
diff.c static int diff_opt_patience(const struct option *opt, · line 5224
Quoted 3 lines
diff.c +3
-0+ options->ignore_regex[options->ignore_regex_nr++] = regex; +return 0 ; +}Don't these parse-options callback functions have a way to tell the caller die instead of them themselves dying like this? Would it work better to "return error(... message ...)", or would that give two error messages?
Yes, that would indeed look better, thanks - just one error message is generated when "return error(...)" is used. I just failed to notice that it is a thing. I will use it in v3.
Quoted 6 lines
In anycase, this is end-user facing, so we'd
want it to be localizable, e.g.die(_("invalid regexp given to -I: '%s'", arg));probably.
Absolutely, I will fix that in v3. I tried to mimic what other similar code in the tree does and a quick git grep 'die.*regex' yielded mostly non-localized strings, so I settled for one as well.
Signature · 2 lines
Best regards, Michał Kępień
diff.h struct diff_options { · line 237
Quoted 1 line · Michał Kępień
diff.h +1
-0+/* see Documentation/diff-options.txt */
This comment adds negative value.
If it were
/* "-I<regexp>" */
the readers won't have to switch to the file only to find out that the comment didn't tell them where in the file to look at.
Quoted 11 lines
diff.h +3
-0+regex_t **ignore_regex; +size_t ignore_regex_nr, ignore_regex_alloc; +const char *single_follow;const char *a_prefix, *b_prefix;const char *line_prefix;xdiff/xdiff.h +0
-0
xdiff/xdiff.h typedef struct s_mmbuffer { · line 82
Quoted 4 lines
xdiff/xdiff.h +1
-0@@ -79,6 +79,10 @@ typedef struct s_mmbuffer { typedef struct s_xpparam { unsigned long flags; + /* See Documentation/diff-options.txt. */
Likewise.
Quoted 6 lines
xdiff/xdiff.h +3
-0+regex_t **ignore_regex; +size_t ignore_regex_nr; +/* See Documentation/diff-options.txt. */ char **anchors;size_t anchors_nr;
Quoted 10 lines
diff.h struct diff_options { · line 237
Quoted 1 line
diff.h +1
-0+/* see Documentation/diff-options.txt */ This comment adds negative value.
If it were
/* "-I<regexp>" */the readers won't have to switch to the file only to find out that the comment didn't tell them where in the file to look at.
Agreed, thanks, I will fix this in v3.
Quoted 14 lines
Quoted 14 lines
diff.h +3
-0+regex_t **ignore_regex; +size_t ignore_regex_nr, ignore_regex_alloc; +const char *single_follow;const char *a_prefix, *b_prefix;const char *line_prefix;xdiff/xdiff.h +0
-0@@ -79,6 +79,10 @@ typedef struct s_mmbuffer { typedef struct s_xpparam { unsigned long flags;
Quoted 3 lines
xdiff/xdiff.h typedef struct s_mmbuffer { · line 82
Quoted 1 line
xdiff/xdiff.h +1
-0+/* See Documentation/diff-options.txt. */ Likewise.
I will fix this in v3.
Signature · 2 lines
Best regards, Michał Kępień
PATCH v2 3/3 t: add -I<regex> tests
In reply to Michał KępieńExercise the new -I<regex> diff option in various scenarios to ensure it behaves as expected.
--- t/t4069-diff-ignore-regex.sh | 426 +++++++++++++++++++++++++++++++++++ 1 file changed, 426 insertions(+) create mode 100755 t/t4069-diff-ignore-regex.sh
t/t4069-diff-ignore-regex.sh new file +426 -0
@@ -0,0 +1,426 @@ +#!/bin/sh + +test_description='Test diff -I<regex>' + +. ./test-lib.sh +. "$TEST_DIRECTORY"/diff-lib.sh + +test_expect_success setup ' + test_seq 20 >x && + git update-index --add x +' + +test_expect_success 'one line changed' ' + test_seq 20 | sed "s/10/100/" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -7,7 +7,7 @@ + 7 + 8 + 9 + -10 + +100 + 11 + 12 + 13 + EOF + compare_diff_patch expected plain && + + # Both old and new line match regex - ignore change + git diff -I "^10" >actual && + test_must_be_empty actual && + + # Both old and new line match some regex - ignore change + git diff -I "^10\$" -I "^100" >actual && + test_must_be_empty actual && + + # Only old line matches regex - do not ignore change + git diff -I "^10\$" >actual && + compare_diff_patch plain actual && + + # Only new line matches regex - do not ignore change + git diff -I "^100" >actual && + compare_diff_patch plain actual && + + # Only old line matches some regex - do not ignore change + git diff -I "^10\$" -I "^101" >actual && + compare_diff_patch plain actual && + + # Only new line matches some regex - do not ignore change + git diff -I "^11\$" -I "^100" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'one line removed' ' + test_seq 20 | sed "10d" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -7,7 +7,6 @@ + 7 + 8 + 9 + -10 + 11 + 12 + 13 + EOF + compare_diff_patch expected plain && + + # Removed line matches regex - ignore change + git diff -I "^10" >actual && + test_must_be_empty actual && + + # Removed line matches some regex - ignore change + git diff -I "^10" -I "^100" >actual && + test_must_be_empty actual && + + # Removed line does not match regex - do not ignore change + git diff -I "^101" >actual && + compare_diff_patch plain actual && + + # Removed line does not match any regex - do not ignore change + git diff -I "^100" -I "^101" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'one line added' ' + test_seq 21 >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -18,3 +18,4 @@ + 18 + 19 + 20 + +21 + EOF + compare_diff_patch expected plain && + + # Added line matches regex - ignore change + git diff -I "^21" >actual && + test_must_be_empty actual && + + # Added line matches some regex - ignore change + git diff -I "^21" -I "^22" >actual && + test_must_be_empty actual && + + # Added line does not match regex - do not ignore change + git diff -I "^212" >actual && + compare_diff_patch plain actual && + + # Added line does not match any regex - do not ignore change + git diff -I "^211" -I "^212" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'last two lines changed' ' + test_seq 20 | sed "s/19/21/; s/20/22/" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -16,5 +16,5 @@ + 16 + 17 + 18 + -19 + -20 + +21 + +22 + EOF + compare_diff_patch expected plain && + + # All changed lines match regex - ignore change + git diff -I "^[12]" >actual && + test_must_be_empty actual && + + # All changed lines match some regex - ignore change + git diff -I "^1" -I "^2" >actual && + test_must_be_empty actual && + + # Not all changed lines match regex - do not ignore change + git diff -I "^2" >actual && + compare_diff_patch plain actual && + + # Not all changed lines match some regex - do not ignore change + git diff -I "^2" -I "^3" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'two non-adjacent lines removed in the same hunk' ' + test_seq 20 | sed "1d; 3d" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,6 +1,4 @@ + -1 + 2 + -3 + 4 + 5 + 6 + EOF + compare_diff_patch expected plain && + + # Both removed lines match regex - ignore hunk + git diff -I "^[1-3]" >actual && + test_must_be_empty actual && + + # Both removed lines match some regex - ignore hunk + git diff -I "^1" -I "^3" >actual && + test_must_be_empty actual && + + # First removed line does not match regex - do not ignore hunk + git diff -I "^[2-3]" >actual && + compare_diff_patch plain actual && + + # First removed line does not match any regex - do not ignore hunk + git diff -I "^2" -I "^3" >actual && + compare_diff_patch plain actual && + + # Second removed line does not match regex - do not ignore hunk + git diff -I "^[1-2]" >actual && + compare_diff_patch plain actual && + + # Second removed line does not match any regex - do not ignore hunk + git diff -I "^1" -I "^2" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'two non-adjacent lines removed in the same hunk, with -U1' ' + test_seq 20 | sed "1d; 3d" >x && + + # Get plain diff + git diff -U1 >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,4 +1,2 @@ + -1 + 2 + -3 + 4 + EOF + compare_diff_patch expected plain && + + # Both removed lines match regex - ignore hunk + git diff -U1 -I "^[1-3]" >actual && + test_must_be_empty actual && + + # Both removed lines match some regex - ignore hunk + git diff -U1 -I "^1" -I "^3" >actual && + test_must_be_empty actual && + + # First removed line does not match regex, but is out of context - ignore second change + git diff -U1 -I "^[2-3]" >actual && + cat >second-change-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,2 +1 @@ + -1 + 2 + EOF + compare_diff_patch second-change-ignored actual && + + # First removed line does not match any regex, but is out of context - ignore second change + git diff -U1 -I "^2" -I "^3" >actual && + compare_diff_patch second-change-ignored actual && + + # Second removed line does not match regex, but is out of context - ignore first change + git diff -U1 -I "^[1-2]" >actual && + cat >first-change-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -2,3 +1,2 @@ + 2 + -3 + 4 + EOF + compare_diff_patch first-change-ignored actual && + + # Second removed line does not match any regex, but is out of context - ignore first change + git diff -U1 -I "^1" -I "^2" >actual && + compare_diff_patch first-change-ignored actual +' + +test_expect_success 'multiple hunks' ' + test_seq 20 | sed "1d; 20d" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,4 +1,3 @@ + -1 + 2 + 3 + 4 + @@ -17,4 +16,3 @@ + 17 + 18 + 19 + -20 + EOF + compare_diff_patch expected plain && + + # Ignore both hunks (single regex) + git diff -I "^[12]" >actual && + test_must_be_empty actual && + + # Ignore both hunks (multiple regexes) + git diff -I "^1" -I "^2" >actual && + test_must_be_empty actual && + + # Only ignore first hunk (single regex) + git diff -I "^1" >actual && + cat >first-hunk-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -17,4 +16,3 @@ + 17 + 18 + 19 + -20 + EOF + compare_diff_patch first-hunk-ignored actual && + + # Only ignore first hunk (multiple regexes) + git diff -I "^0" -I "^1" >actual && + compare_diff_patch first-hunk-ignored actual && + + # Only ignore second hunk (single regex) + git diff -I "^2" >actual && + cat >second-hunk-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,4 +1,3 @@ + -1 + 2 + 3 + 4 + EOF + compare_diff_patch second-hunk-ignored actual && + + # Only ignore second hunk (multiple regexes) + git diff -I "^2" -I "^3" >actual && + compare_diff_patch second-hunk-ignored actual +' + +test_expect_success 'multiple hunks, with --ignore-blank-lines' ' + echo >x && + test_seq 21 >>x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,3 +1,4 @@ + + + 1 + 2 + 3 + @@ -18,3 +19,4 @@ + 18 + 19 + 20 + +21 + EOF + compare_diff_patch expected plain && + + # -I does not override --ignore-blank-lines - ignore both hunks (single regex) + git diff --ignore-blank-lines -I "^21" >actual && + test_must_be_empty actual && + + # -I does not override --ignore-blank-lines - ignore both hunks (multiple regexes) + git diff --ignore-blank-lines -I "^21" -I "^12" >actual && + test_must_be_empty actual +' + +test_expect_success 'diffstat' ' + test_seq 20 | sed "s/^5/0/p; s/^15/10/; 16d" >x && + + # Get plain diffstat + git diff --stat >actual && + cat >expected <<-EOF && + x | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + EOF + test_cmp expected actual && + + # Ignore both hunks (single regex) + git diff --stat -I "^[0-5]" >actual && + test_must_be_empty actual && + + # Ignore both hunks (multiple regexes) + git diff --stat -I "^0" -I "^1" -I "^5" >actual && + test_must_be_empty actual && + + # Only ignore first hunk (single regex) + git diff --stat -I "^[05]" >actual && + cat >expected <<-EOF && + x | 3 +-- + 1 file changed, 1 insertion(+), 2 deletions(-) + EOF + test_cmp expected actual && + + # Only ignore first hunk (multiple regexes) + git diff --stat -I "^0" -I "^5" >actual && + test_cmp expected actual && + + # Only ignore second hunk (single regex) + git diff --stat -I "^1" >actual && + cat >expected <<-EOF && + x | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + EOF + test_cmp expected actual && + + # Only ignore second hunk (multiple regexes) + git diff --stat -I "^1" -I "^2" >actual && + test_cmp expected actual +' + +test_expect_success 'invalid regexes' ' + >x && + + # Single invalid regex + git diff -I "^[1" 2>&1 | grep "invalid regex: " && + + # Two regexes: first invalid, second valid + git diff -I "^[1" -I "^1" 2>&1 | grep "invalid regex: " && + + # Two invalid regexes + git diff -I "^[1" -I "^[2" 2>&1 | grep "invalid regex: " +' + +test_done
Signature · 1 line
2.28.0
Re: [PATCH v2 3/3] t: add -I<regex> tests
Hi Michał,
Quoted 2 lines · Michał Kępień
Exercise the new -I<regex> diff option in various scenarios to ensure it behaves as expected.
Excellent. I was actually looking for a test in patch 2/3 and almost commented about that.
Quoted 4 lines
Signed-off-by Michał Kępień <[email protected]>--- t/t4069-diff-ignore-regex.sh | 426 +++++++++++++++++++++++++++++++++++
Hmm. I wonder whether we could do with a much more concise test script. The test suite already takes a quite long time to run, which is not a laughing matter: we had issues in the past where contributors would skip running it because it took too long, and this test is sure to exacerbate that problem.
I could imagine, for example, that it would be plenty enough to do something like this instead:
-- snip --
t/t4013-diff-various.sh +21 -0
@@ -6,6 +6,7 @@ test_description='Various diff formatting options' . ./test-lib.sh +. "$TEST_DIRECTORY"/diff-lib.sh test_expect_success setup '
@@ -473,4 +474,24 @@ test_expect_success 'diff-tree --stdin with log formatting' ' test_cmp expect actual ' +test_expect_success '-I<regex>' ' + seq 50 >I.txt && + sed -e "s/13/ten and three/" -e "/7\$/d" <I.txt >J.txt && + test_must_fail git diff --no-index -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual && + cat >expect <<-\EOF && + diff --git a/I.txt b/J.txt + --- a/I.txt + +++ b/J.txt + @@ -34,7 +31,6 @@ + 34 + 35 + 36 + -37 + 38 + 39 + 40 + EOF + compare_diff_patch expect actual +' + test_done
-- snap --
Note how it tests various things in one go?
Thanks,
Dscho
P.S.: My main interest in the -I option is its use case for git range-diff in Git's own context: if you want to compare your patches to what entered the seen branch, there will _always_ be a difference because Junio adds their DCO. Something like this can help that:
git range-diff \ -I'^ Signed-off-by: Junio C Hamano <[email protected]>$' \ <my-patch-range> <the-equivalent-in-seen>
Quoted 439 lines
1 file changed, 426 insertions(+) create mode 100755 t/t4069-diff-ignore-regex.sht/t4069-diff-ignore-regex.sh new file +426
-1@@ -0,0 +1,426 @@ +#!/bin/sh + +test_description='Test diff -I<regex>' + +. ./test-lib.sh +. "$TEST_DIRECTORY"/diff-lib.sh + +test_expect_success setup ' + test_seq 20 >x && + git update-index --add x +' + +test_expect_success 'one line changed' ' + test_seq 20 | sed "s/10/100/" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -7,7 +7,7 @@ + 7 + 8 + 9 + -10 + +100 + 11 + 12 + 13 + EOF + compare_diff_patch expected plain && + + # Both old and new line match regex - ignore change + git diff -I "^10" >actual && + test_must_be_empty actual && + + # Both old and new line match some regex - ignore change + git diff -I "^10\$" -I "^100" >actual && + test_must_be_empty actual && + + # Only old line matches regex - do not ignore change + git diff -I "^10\$" >actual && + compare_diff_patch plain actual && + + # Only new line matches regex - do not ignore change + git diff -I "^100" >actual && + compare_diff_patch plain actual && + + # Only old line matches some regex - do not ignore change + git diff -I "^10\$" -I "^101" >actual && + compare_diff_patch plain actual && + + # Only new line matches some regex - do not ignore change + git diff -I "^11\$" -I "^100" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'one line removed' ' + test_seq 20 | sed "10d" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -7,7 +7,6 @@ + 7 + 8 + 9 + -10 + 11 + 12 + 13 + EOF + compare_diff_patch expected plain && + + # Removed line matches regex - ignore change + git diff -I "^10" >actual && + test_must_be_empty actual && + + # Removed line matches some regex - ignore change + git diff -I "^10" -I "^100" >actual && + test_must_be_empty actual && + + # Removed line does not match regex - do not ignore change + git diff -I "^101" >actual && + compare_diff_patch plain actual && + + # Removed line does not match any regex - do not ignore change + git diff -I "^100" -I "^101" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'one line added' ' + test_seq 21 >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -18,3 +18,4 @@ + 18 + 19 + 20 + +21 + EOF + compare_diff_patch expected plain && + + # Added line matches regex - ignore change + git diff -I "^21" >actual && + test_must_be_empty actual && + + # Added line matches some regex - ignore change + git diff -I "^21" -I "^22" >actual && + test_must_be_empty actual && + + # Added line does not match regex - do not ignore change + git diff -I "^212" >actual && + compare_diff_patch plain actual && + + # Added line does not match any regex - do not ignore change + git diff -I "^211" -I "^212" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'last two lines changed' ' + test_seq 20 | sed "s/19/21/; s/20/22/" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -16,5 +16,5 @@ + 16 + 17 + 18 + -19 + -20 + +21 + +22 + EOF + compare_diff_patch expected plain && + + # All changed lines match regex - ignore change + git diff -I "^[12]" >actual && + test_must_be_empty actual && + + # All changed lines match some regex - ignore change + git diff -I "^1" -I "^2" >actual && + test_must_be_empty actual && + + # Not all changed lines match regex - do not ignore change + git diff -I "^2" >actual && + compare_diff_patch plain actual && + + # Not all changed lines match some regex - do not ignore change + git diff -I "^2" -I "^3" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'two non-adjacent lines removed in the same hunk' ' + test_seq 20 | sed "1d; 3d" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,6 +1,4 @@ + -1 + 2 + -3 + 4 + 5 + 6 + EOF + compare_diff_patch expected plain && + + # Both removed lines match regex - ignore hunk + git diff -I "^[1-3]" >actual && + test_must_be_empty actual && + + # Both removed lines match some regex - ignore hunk + git diff -I "^1" -I "^3" >actual && + test_must_be_empty actual && + + # First removed line does not match regex - do not ignore hunk + git diff -I "^[2-3]" >actual && + compare_diff_patch plain actual && + + # First removed line does not match any regex - do not ignore hunk + git diff -I "^2" -I "^3" >actual && + compare_diff_patch plain actual && + + # Second removed line does not match regex - do not ignore hunk + git diff -I "^[1-2]" >actual && + compare_diff_patch plain actual && + + # Second removed line does not match any regex - do not ignore hunk + git diff -I "^1" -I "^2" >actual && + compare_diff_patch plain actual +' + +test_expect_success 'two non-adjacent lines removed in the same hunk, with -U1' ' + test_seq 20 | sed "1d; 3d" >x && + + # Get plain diff + git diff -U1 >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,4 +1,2 @@ + -1 + 2 + -3 + 4 + EOF + compare_diff_patch expected plain && + + # Both removed lines match regex - ignore hunk + git diff -U1 -I "^[1-3]" >actual && + test_must_be_empty actual && + + # Both removed lines match some regex - ignore hunk + git diff -U1 -I "^1" -I "^3" >actual && + test_must_be_empty actual && + + # First removed line does not match regex, but is out of context - ignore second change + git diff -U1 -I "^[2-3]" >actual && + cat >second-change-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,2 +1 @@ + -1 + 2 + EOF + compare_diff_patch second-change-ignored actual && + + # First removed line does not match any regex, but is out of context - ignore second change + git diff -U1 -I "^2" -I "^3" >actual && + compare_diff_patch second-change-ignored actual && + + # Second removed line does not match regex, but is out of context - ignore first change + git diff -U1 -I "^[1-2]" >actual && + cat >first-change-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -2,3 +1,2 @@ + 2 + -3 + 4 + EOF + compare_diff_patch first-change-ignored actual && + + # Second removed line does not match any regex, but is out of context - ignore first change + git diff -U1 -I "^1" -I "^2" >actual && + compare_diff_patch first-change-ignored actual +' + +test_expect_success 'multiple hunks' ' + test_seq 20 | sed "1d; 20d" >x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,4 +1,3 @@ + -1 + 2 + 3 + 4 + @@ -17,4 +16,3 @@ + 17 + 18 + 19 + -20 + EOF + compare_diff_patch expected plain && + + # Ignore both hunks (single regex) + git diff -I "^[12]" >actual && + test_must_be_empty actual && + + # Ignore both hunks (multiple regexes) + git diff -I "^1" -I "^2" >actual && + test_must_be_empty actual && + + # Only ignore first hunk (single regex) + git diff -I "^1" >actual && + cat >first-hunk-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -17,4 +16,3 @@ + 17 + 18 + 19 + -20 + EOF + compare_diff_patch first-hunk-ignored actual && + + # Only ignore first hunk (multiple regexes) + git diff -I "^0" -I "^1" >actual && + compare_diff_patch first-hunk-ignored actual && + + # Only ignore second hunk (single regex) + git diff -I "^2" >actual && + cat >second-hunk-ignored <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,4 +1,3 @@ + -1 + 2 + 3 + 4 + EOF + compare_diff_patch second-hunk-ignored actual && + + # Only ignore second hunk (multiple regexes) + git diff -I "^2" -I "^3" >actual && + compare_diff_patch second-hunk-ignored actual +' + +test_expect_success 'multiple hunks, with --ignore-blank-lines' ' + echo >x && + test_seq 21 >>x && + + # Get plain diff + git diff >plain && + cat >expected <<-EOF && + diff --git a/x b/x + --- a/x + +++ b/x + @@ -1,3 +1,4 @@ + + + 1 + 2 + 3 + @@ -18,3 +19,4 @@ + 18 + 19 + 20 + +21 + EOF + compare_diff_patch expected plain && + + # -I does not override --ignore-blank-lines - ignore both hunks (single regex) + git diff --ignore-blank-lines -I "^21" >actual && + test_must_be_empty actual && + + # -I does not override --ignore-blank-lines - ignore both hunks (multiple regexes) + git diff --ignore-blank-lines -I "^21" -I "^12" >actual && + test_must_be_empty actual +' + +test_expect_success 'diffstat' ' + test_seq 20 | sed "s/^5/0/p; s/^15/10/; 16d" >x && + + # Get plain diffstat + git diff --stat >actual && + cat >expected <<-EOF && + x | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + EOF + test_cmp expected actual && + + # Ignore both hunks (single regex) + git diff --stat -I "^[0-5]" >actual && + test_must_be_empty actual && + + # Ignore both hunks (multiple regexes) + git diff --stat -I "^0" -I "^1" -I "^5" >actual && + test_must_be_empty actual && + + # Only ignore first hunk (single regex) + git diff --stat -I "^[05]" >actual && + cat >expected <<-EOF && + x | 3 +-- + 1 file changed, 1 insertion(+), 2 deletions(-) + EOF + test_cmp expected actual && + + # Only ignore first hunk (multiple regexes) + git diff --stat -I "^0" -I "^5" >actual && + test_cmp expected actual && + + # Only ignore second hunk (single regex) + git diff --stat -I "^1" >actual && + cat >expected <<-EOF && + x | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + EOF + test_cmp expected actual && + + # Only ignore second hunk (multiple regexes) + git diff --stat -I "^1" -I "^2" >actual && + test_cmp expected actual +' + +test_expect_success 'invalid regexes' ' + >x && + + # Single invalid regex + git diff -I "^[1" 2>&1 | grep "invalid regex: " && + + # Two regexes: first invalid, second valid + git diff -I "^[1" -I "^1" 2>&1 | grep "invalid regex: " && + + # Two invalid regexes + git diff -I "^[1" -I "^[2" 2>&1 | grep "invalid regex: " +' + +test_done --2.28.0
Re: [PATCH v2 3/3] t: add -I<regex> tests
Hi Johannes,
Quoted 5 lines
Quoted 2 lines
Exercise the new -I<regex> diff option in various scenarios to ensure it behaves as expected.
Excellent. I was actually looking for a test in patch 2/3 and almost commented about that.
Right, I expressed my doubts in this area at the end of the cover letter for v1:
Quoted 2 lines
Quoted 2 lines
- Should tests be added in a separate commit? This is what I did as I thought it would help with readability, but...
I will be glad to follow any guidance provided, I just picked one of the two possible routes for v1.
Quoted 5 lines
Hmm. I wonder whether we could do with a much more concise test script. The test suite already takes a quite long time to run, which is not a laughing matter: we had issues in the past where contributors would skip running it because it took too long, and this test is sure to exacerbate that problem.
First, let me say that the goal of minimizing the run time of a test suite is close to my heart (it is an issue at my day job). Yet, I assumed that this new test would not be detrimental to test suite run times as it takes about half a second to run t4069-diff-ignore-regex.sh on my machine - and (I hope) its contents are in line with the "tests are the best documentation" proverb. That being said, I realize that the hosts used in various test environments may have different processing capabilities. I tried preparing something exhaustive and well-commented, so that it is clear what to expect from the new feature. Yet, if you would rather have me cut some things out, I am certainly not particularly attached to the tests from patch 3 and I will be glad to rip them out if that is the recommendation :-)
Quoted 43 lines
I could imagine, for example, that it would be plenty enough to do something like this instead:
-- snip --
t/t4013-diff-various.sh +21
-1@@ -6,6 +6,7 @@ test_description='Various diff formatting options' . ./test-lib.sh +. "$TEST_DIRECTORY"/diff-lib.sh test_expect_success setup '@@ -473,4 +474,24 @@ test_expect_success 'diff-tree --stdin with log formatting' ' test_cmp expect actual ' +test_expect_success '-I<regex>' ' + seq 50 >I.txt && + sed -e "s/13/ten and three/" -e "/7\$/d" <I.txt >J.txt && + test_must_fail git diff --no-index -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual && + cat >expect <<-\EOF && + diff --git a/I.txt b/J.txt + --- a/I.txt + +++ b/J.txt + @@ -34,7 +31,6 @@ + 34 + 35 + 36 + -37 + 38 + 39 + 40 + EOF + compare_diff_patch expect actual +' + test_done -- snap --
Quoted 1 line
Note how it tests various things in one go?
Right, neat, though this does not (yet) test:
- the interaction between -I and --ignore-blank-lines (this is visible in code coverage),
- whether the list of hunks emitted varies for different -U<n> values,
- diffstat with -I<regex>,
- invalid regular expressions.
Would you like me to add these tests to your proposal or to skip them, given that -I uses the same field for marking changes as ignored as --ignore-blank-lines does?
Quoted 8 lines
P.S.: My main interest in the-Ioption is its use case forgit range-diffin Git's own context: if you want to compare your patches to what entered theseenbranch, there will _always_ be a difference because Junio adds their DCO. Something like this can help that:git range-diff \ -I'^ Signed-off-by: Junio C Hamano <[email protected]>$' \ <my-patch-range> <the-equivalent-in-seen>
Right, makes sense, I have not thought of that use case.
Signature · 2 lines
Best regards, Michał Kępień
Re: [PATCH v2 3/3] t: add -I<regex> tests
Hi Michał,
Quoted 6 lines · Michał Kępień
Quoted 5 lines
Hmm. I wonder whether we could do with a much more concise test script. The test suite already takes a quite long time to run, which is not a laughing matter: we had issues in the past where contributors would skip running it because it took too long, and this test is sure to exacerbate that problem.
Quoted 6 lines
First, let me say that the goal of minimizing the run time of a test suite is close to my heart (it is an issue at my day job). Yet, I assumed that this new test would not be detrimental to test suite run times as it takes about half a second to run
t4069-diff-ignore-regex.shon my machine - and (I hope) its contents are in line with the "tests are the best documentation" proverb.
Sadly, the test is not quite as fast on Windows. I just ran this (on a not quite idle machine, admittedly) and it ended in this:
# passed all 11 test(s) 1..11
real 0m51.470s user 0m0.046s sys 0m0.015s
Yes, that's almost a minute.
Quoted 43 lines
Quoted 43 lines
I could imagine, for example, that it would be plenty enough to do something like this instead:
-- snip --
t/t4013-diff-various.sh +21
-1@@ -6,6 +6,7 @@ test_description='Various diff formatting options' . ./test-lib.sh +. "$TEST_DIRECTORY"/diff-lib.sh test_expect_success setup '@@ -473,4 +474,24 @@ test_expect_success 'diff-tree --stdin with log formatting' ' test_cmp expect actual ' +test_expect_success '-I<regex>' ' + seq 50 >I.txt && + sed -e "s/13/ten and three/" -e "/7\$/d" <I.txt >J.txt && + test_must_fail git diff --no-index -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual && + cat >expect <<-\EOF && + diff --git a/I.txt b/J.txt + --- a/I.txt + +++ b/J.txt + @@ -34,7 +31,6 @@ + 34 + 35 + 36 + -37 + 38 + 39 + 40 + EOF + compare_diff_patch expect actual +' + test_done -- snap --
Quoted 6 lines
Quoted 1 line
Note how it tests various things in one go?
Right, neat, though this does not (yet) test:
- the interaction between -I and --ignore-blank-lines (this is visible in code coverage),
Right. Any chance you can finagle that in, e.g. by yet another -e argument to the sed call?
Quoted 1 line
- whether the list of hunks emitted varies for different -U<n> values,
I am not worried about that.
Quoted 1 line
- diffstat with -I<regex>,
I am not worried about that, either, as diffstat consumes xdiff's output, therefore if one consumer works, another consumer will work, too.
Quoted 1 line
- invalid regular expressions.
Right, that should be super easy (and quick) to test.
Quoted 3 lines
Would you like me to add these tests to your proposal or to skip them, given that -I uses the same field for marking changes as ignored as --ignore-blank-lines does?
I'd say it depends how easily (read: in how small a test case or modifications to an existing test case) you can add a test for that interaction.
Thanks,
Dscho
Re: [PATCH v2 3/3] t: add -I<regex> tests
Quoted 4 lines · Johannes Schindelin
Quoted 1 line
- diffstat with -I<regex>,
I am not worried about that, either, as
diffstatconsumesxdiff's output, therefore if one consumer works, another consumer will work, too.
Careful. Such a "we know what happens in the code" transparent-box testing attitude is laying a minefield for later our selves.
As we learned in a recent bug in sequencer, some corners of implementation do the same thing in different codepaths as optimization. The really bad part of the story is that such an implementation detail can and will change over time, since that is the kind of thing we do when optimizing code.
In other words, we only know what happens in the current code. And automated tests protect us from the future, when done right. If written with too intimate knowledge of how the current code works, well, what are we really testing? It's a balancing act and there is no single "right" answer, but I'd draw the line on a bit more careful side than you are drawing here.
Thanks.
Re: [PATCH v2 3/3] t: add -I<regex> tests
In reply to Johannes SchindelinHi Johannes,
Quoted 10 lines
Quoted 6 lines
First, let me say that the goal of minimizing the run time of a test suite is close to my heart (it is an issue at my day job). Yet, I assumed that this new test would not be detrimental to test suite run times as it takes about half a second to run
t4069-diff-ignore-regex.shon my machine - and (I hope) its contents are in line with the "tests are the best documentation" proverb.Sadly, the test is not quite as fast on Windows. I just ran this (on a not quite idle machine, admittedly) and it ended in this:
Quoted 8 lines
# passed all 11 test(s) 1..11real 0m51.470s user 0m0.046s sys 0m0.015sYes, that's almost a minute.
Out of curiosity: is that under Cygwin? I have seen shell-based tests finishing in 15 *seconds* on Unix-like systems and in 15 *minutes* under Cygwin, which would be in line with your measurements provided above.
Quoted 7 lines
Quoted 4 lines
Right, neat, though this does not (yet) test:
- the interaction between -I and --ignore-blank-lines (this is visible in code coverage),
Right. Any chance you can finagle that in, e.g. by yet another
-eargument to thesedcall?
I will try in v3 (while also looking at what I can do for other missing -I<regex> tests I pointed out).
Signature · 2 lines
Best regards, Michał Kępień
Re: [PATCH v2 3/3] t: add -I<regex> tests
Hi Michał,
Quoted 17 lines · Michał Kępień
Quoted 17 lines
Quoted 6 lines
First, let me say that the goal of minimizing the run time of a test suite is close to my heart (it is an issue at my day job). Yet, I assumed that this new test would not be detrimental to test suite run times as it takes about half a second to run
t4069-diff-ignore-regex.shon my machine - and (I hope) its contents are in line with the "tests are the best documentation" proverb.Sadly, the test is not quite as fast on Windows. I just ran this (on a not quite idle machine, admittedly) and it ended in this:
# passed all 11 test(s) 1..11real 0m51.470s user 0m0.046s sys 0m0.015s
Quoted 5 lines
Quoted 1 line
Yes, that's almost a minute.
Out of curiosity: is that under Cygwin? I have seen shell-based tests finishing in 15 *seconds* on Unix-like systems and in 15 *minutes* under Cygwin, which would be in line with your measurements provided above.
Close enough: it's an MSYS2 Bash (and MSYS2 is a close fork of Cygwin). It is what Git for Windows uses.
Ciao,
Dscho
v3 cover diff: add -I<regex> that ignores matching changes
In reply to Michał KępieńThis patch series adds a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I/--ignore-matching-lines option in standalone diff utilities and can be used e.g. to ignore changes which only affect code comments or to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
Changes from v2:
- Add a long option for -I (--ignore-matching-lines) as it is commonplace in standalone diff utilities. Update documentation and commit log messages accordingly.
- Use xmalloc() instead of xcalloc() for allocating regex_t structures in diff_opt_ignore_regex().
- Ensure the memory allocated in diff_opt_ignore_regex() gets released.
- Use "return error(...)" instead of die() in the -I option callback. Make the relevant error message localizable.
- Drastically reduce the number of -I<regex> tests due to excessive run time of t/t4069-diff-ignore-regex.sh from v1/v2 on some platforms (notably Windows). Use a tweaked version of a test suggested by Johannes Schindelin (thanks!). Given its reduction in size, squash patch 3 (which contained the tests) into patch 2.
- Replace "see Documentation/diff-options.txt" with "-I<regex>" in the comments for the added structure fields, in order to make these comments more useful.
Changes from v1:
- Add a new preliminary cleanup patch which ensures xpparam_t structures are always zero-initialized. (This was a prerequisite for the next change below.)
- Do not add a new 'xdl_opts' flag to check whether -I was used; instead, just check whether the array of regular expressions to match against is non-NULL.
- Enable the -I option to be used multiple times. As a consequence of this, regular expressions are now "pre-compiled" in the option's callback (and passed around as an array of regex_t structures) rather than deep down in xdiff code. Add test cases exercising use of multiple -I options in the same git invocation. Update documentation accordingly.
- Rename xdl_mark_ignorable() to xdl_mark_ignorable_lines(), to indicate that it is logically a "sibling" of xdl_mark_ignorable_regex() rather than its "parent".
- Optimize xdl_mark_ignorable_regex() by making it immediately skip changes already marked as ignored by xdl_mark_ignorable_lines().
- Fix coding style issue in the prototype part of the definition of xdl_mark_ignorable_regex().
- Add "/* see Documentation/diff-options.txt */" comments for the fields added to struct diff_options and xpparam_t, mimicking the comments used for 'anchors', 'anchors_nr', and 'anchors_alloc'.
- Revise commit log messages to reflect all of the above.
Michał Kępień (2):
merge-base, xdiff: zero out xpparam_t structures
diff: add -I<regex> that ignores matching changes
Documentation/diff-options.txt | 5 ++++ builtin/merge-tree.c | 1 + diff.c | 28 ++++++++++++++++++++ diff.h | 4 +++ t/t4013-diff-various.sh | 33 ++++++++++++++++++++++++ xdiff/xdiff.h | 4 +++ xdiff/xdiffi.c | 47 ++++++++++++++++++++++++++++++++-- xdiff/xhistogram.c | 2 ++ xdiff/xpatience.c | 2 ++ 9 files changed, 124 insertions(+), 2 deletions(-)
Signature · 1 line
2.28.0
PATCH v3 1/2 merge-base, xdiff: zero out xpparam_t structures
xpparam_t structures are usually zero-initialized before their specific fields are assigned to, but there are three locations in the tree where that does not happen. Add the missing memset() calls in order to make initialization of xpparam_t structures consistent tree-wide and to prevent stack garbage from being used as field values.
--- builtin/merge-tree.c | 1 + xdiff/xhistogram.c | 2 ++ xdiff/xpatience.c | 2 ++ 3 files changed, 5 insertions(+)
builtin/merge-tree.c +1 -0
@@ -109,6 +109,7 @@ static void show_diff(struct merge_list *entry)xdemitconf_t xecfg;xdemitcb_t ecb; +memset (&xpp,0 ,sizeof (xpp)); xpp.flags =0 ;memset (&xecfg,0 ,sizeof (xecfg)); xecfg.ctxlen =3 ;
xdiff/xhistogram.c +2 -0
@@ -235,6 +235,8 @@ static int fall_back_to_classic_diff(xpparam_t const *xpp, xdfenv_t *env, int line1, int count1, int line2, int count2) { xpparam_t xpparam; + +memset (&xpparam,0 ,sizeof (xpparam)); xpparam.flags = xpp->flags & ~XDF_DIFF_ALGORITHM_MASK ; return xdl_fall_back_diff(env, &xpparam,
xdiff/xpatience.c +2 -0
@@ -318,6 +318,8 @@ static int fall_back_to_classic_diff(struct hashmap *map, int line1, int count1, int line2, int count2) { xpparam_t xpp; + +memset (&xpp,0 ,sizeof (xpp)); xpp.flags = map->xpp->flags & ~XDF_DIFF_ALGORITHM_MASK ; return xdl_fall_back_diff(map->env, &xpp,
Signature · 1 line
2.28.0
PATCH v3 2/2 diff: add -I<regex> that ignores matching changes
In reply to Michał KępieńAdd a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I/--ignore-matching-lines option in standalone diff utilities and can be used e.g. to ignore changes which only affect code comments or to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
Use the 'ignore' field of xdchange_t for marking a change as ignored or not. Since the same field is used by --ignore-blank-lines, identical hunk emitting rules apply for --ignore-blank-lines and -I. These two options can also be used together in the same git invocation (they are complementary to each other).
Rename xdl_mark_ignorable() to xdl_mark_ignorable_lines(), to indicate that it is logically a "sibling" of xdl_mark_ignorable_regex() rather than its "parent".
--- Documentation/diff-options.txt | 5 ++++ diff.c | 28 ++++++++++++++++++++ diff.h | 4 +++ t/t4013-diff-various.sh | 33 ++++++++++++++++++++++++ xdiff/xdiff.h | 4 +++ xdiff/xdiffi.c | 47 ++++++++++++++++++++++++++++++++-- 6 files changed, 119 insertions(+), 2 deletions(-)
Documentation/diff-options.txt +5 -0
@@ -687,6 +687,11 @@ endif::git-format-patch[] --ignore-blank-lines:: Ignore changes whose lines are all blank. +-I<regex>:: +--ignore-matching-lines=<regex>:: + Ignore changes whose all lines match <regex>. This option may + be specified more than once. + --inter-hunk-context=<lines>:: Show the context between diff hunks, up to the specified number of lines, thereby fusing hunks that are close to each other.
diff.c +28 -0
@@ -3587,6 +3587,8 @@ static void builtin_diff(const char *name_a,if (header.len && !o->flags.suppress_diff_headers) ecbdata.header = &header; xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; + xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;
@@ -3716,6 +3718,8 @@ static void builtin_diffstat(const char *name_a, const char *name_b,memset (&xpp,0 ,sizeof (xpp));memset (&xecfg,0 ,sizeof (xecfg)); xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; + xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;
@@ -5203,6 +5207,22 @@ static int diff_opt_patience(const struct option *opt,return 0 ; } +static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; +regex_t *regex; + +BUG_ON_OPT_NEG (unset); + regex =xmalloc (sizeof (*regex)); +if (regcomp (regex, arg,REG_EXTENDED |REG_NEWLINE )) +return error (_ ("invalid regex given to -I: '%s'" ), arg); +ALLOC_GROW (options->ignore_regex, options->ignore_regex_nr +1 , + options->ignore_regex_alloc); + options->ignore_regex[options->ignore_regex_nr++] = regex; +return 0 ; +} +static int diff_opt_pickaxe_regex (const struct option *opt,const char *arg,int unset) {
@@ -5491,6 +5511,9 @@ static void prep_parse_options(struct diff_options *options)OPT_BIT_F (0 ,"ignore-blank-lines" , &options->xdl_opts,N_ ("ignore changes whose lines are all blank" ),XDF_IGNORE_BLANK_LINES ,PARSE_OPT_NONEG ), +OPT_CALLBACK_F ('I' ,"ignore-matching-lines" , options,N_ ("<regex>" ), +N_ ("ignore changes whose all lines match <regex>" ), +0 , diff_opt_ignore_regex),OPT_BIT (0 ,"indent-heuristic" , &options->xdl_opts,N_ ("heuristic to shift diff hunk boundaries for easy reading" ), XDF_INDENT_HEURISTIC),
@@ -6405,6 +6428,11 @@ void diff_flush(struct diff_options *options)DIFF_QUEUE_CLEAR (q);if (options->close_file)fclose (options->file); +for (i =0 ; i < options->ignore_regex_nr; i++) { +regfree (options->ignore_regex[i]); +free (options->ignore_regex[i]); + } +free (options->ignore_regex);/* * Report the content-level differences with HAS_CHANGES;
diff.h +4 -0
@@ -234,6 +234,10 @@ struct diff_options {*/ const char *pickaxe; +/* -I<regex> */ +regex_t **ignore_regex; +size_t ignore_regex_nr, ignore_regex_alloc; +const char *single_follow;const char *a_prefix, *b_prefix;const char *line_prefix;
t/t4013-diff-various.sh +33 -0
@@ -6,6 +6,7 @@ test_description='Various diff formatting options' . ./test-lib.sh +. "$TEST_DIRECTORY"/diff-lib.sh test_expect_success setup '
@@ -473,4 +474,36 @@ test_expect_success 'diff-tree --stdin with log formatting' ' test_cmp expect actual ' +test_expect_success 'diff -I<regex>' ' + test_seq 50 >I.txt && + sed -e "s/13/ten and three/" -e "/7\$/d" <I.txt >J.txt && + echo >>J.txt && + + test_expect_code 1 git diff --no-index --ignore-blank-lines -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual && + cat >expect <<-\EOF && + diff --git a/I.txt b/J.txt + --- a/I.txt + +++ b/J.txt + @@ -34,7 +31,6 @@ + 34 + 35 + 36 + -37 + 38 + 39 + 40 + EOF + compare_diff_patch expect actual && + + test_expect_code 1 git diff --stat --no-index --ignore-blank-lines -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual && + cat >expect <<-\EOF && + I.txt => J.txt | 1 - + 1 file changed, 1 deletion(-) + EOF + test_cmp expect actual && + + test_expect_code 129 git diff --no-index --ignore-matching-lines="^[124-9]" --ignore-matching-lines="^[124-9" I.txt J.txt >output 2>&1 && + test_i18ngrep "invalid regex given to -I: " output +' + test_done
xdiff/xdiff.h +4 -0
@@ -79,6 +79,10 @@ typedef struct s_mmbuffer { typedef struct s_xpparam { unsigned long flags; + /* -I<regex> */ + regex_t **ignore_regex; + size_t ignore_regex_nr; + /* See Documentation/diff-options.txt. */ char **anchors; size_t anchors_nr;
xdiff/xdiffi.c +45 -2
65 lines not on this page · raw message
Signature · 1 line
2.28.0
Hi Michał
Thanks for working on this, it will be a useful addition. Unfortunately there's a use-after-free error see below
Quoted 101 lines · Michał Kępień
Add a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I/--ignore-matching-lines option in standalone diff utilities and can be used e.g. to ignore changes which only affect code comments or to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
Use the 'ignore' field of
xdchange_tfor marking a change as ignored or not. Since the same field is used by --ignore-blank-lines, identical hunk emitting rules apply for --ignore-blank-lines and -I. These two options can also be used together in the same git invocation (they are complementary to each other).Rename
xdl_mark_ignorable()toxdl_mark_ignorable_lines(), to indicate that it is logically a "sibling" ofxdl_mark_ignorable_regex()rather than its "parent".Signed-off-by Michał Kępień <[email protected]>--- Documentation/diff-options.txt | 5 ++++ diff.c | 28 ++++++++++++++++++++ diff.h | 4 +++ t/t4013-diff-various.sh | 33 ++++++++++++++++++++++++ xdiff/xdiff.h | 4 +++ xdiff/xdiffi.c | 47 ++++++++++++++++++++++++++++++++-- 6 files changed, 119 insertions(+), 2 deletions(-)Documentation/diff-options.txt +5
-0@@ -687,6 +687,11 @@ endif::git-format-patch[] --ignore-blank-lines:: Ignore changes whose lines are all blank. +-I<regex>:: +--ignore-matching-lines=<regex>:: + Ignore changes whose all lines match <regex>. This option may + be specified more than once. + --inter-hunk-context=<lines>:: Show the context between diff hunks, up to the specified number of lines, thereby fusing hunks that are close to each other.diff.c +23
-0@@ -3587,6 +3587,8 @@ static void builtin_diff(const char *name_a,if (header.len && !o->flags.suppress_diff_headers) ecbdata.header = &header; xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; + xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;@@ -3716,6 +3718,8 @@ static void builtin_diffstat(const char *name_a, const char *name_b,memset (&xpp,0 ,sizeof (xpp));memset (&xecfg,0 ,sizeof (xecfg)); xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; + xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;@@ -5203,6 +5207,22 @@ static int diff_opt_patience(const struct option *opt,return 0 ; } +static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; +regex_t *regex; + +BUG_ON_OPT_NEG (unset); + regex =xmalloc (sizeof (*regex)); +if (regcomp (regex, arg,REG_EXTENDED |REG_NEWLINE )) +return error (_ ("invalid regex given to -I: '%s'" ), arg); +ALLOC_GROW (options->ignore_regex, options->ignore_regex_nr +1 , + options->ignore_regex_alloc); + options->ignore_regex[options->ignore_regex_nr++] = regex; +return 0 ; +} +static int diff_opt_pickaxe_regex (const struct option *opt,const char *arg,int unset) {@@ -5491,6 +5511,9 @@ static void prep_parse_options(struct diff_options *options)OPT_BIT_F (0 ,"ignore-blank-lines" , &options->xdl_opts,N_ ("ignore changes whose lines are all blank" ),XDF_IGNORE_BLANK_LINES ,PARSE_OPT_NONEG ), +OPT_CALLBACK_F ('I' ,"ignore-matching-lines" , options,N_ ("<regex>" ), +N_ ("ignore changes whose all lines match <regex>" ), +0 , diff_opt_ignore_regex),OPT_BIT (0 ,"indent-heuristic" , &options->xdl_opts,N_ ("heuristic to shift diff hunk boundaries for easy reading" ), XDF_INDENT_HEURISTIC),
diff.c void diff_flush(struct diff_options *options) · line 6435
Quoted 8 lines
diff.c +5
-0@@ -6405,6 +6428,11 @@ void diff_flush(struct diff_options *options)DIFF_QUEUE_CLEAR (q);if (options->close_file)fclose (options->file); +for (i =0 ; i < options->ignore_regex_nr; i++) { +regfree (options->ignore_regex[i]); +free (options->ignore_regex[i]); + } +free (options->ignore_regex);
If I run git log -p -I foo then the address sanitizer reports
AddressSanitizer: heap-use-after-free xdiff/xdiffi.c:1027 in record_matches_regex
after it has printed the diff for the first commit. I think freeing the regex here is the cause of the problem.
Best Wishes
Phillip
Quoted 154 lines
diff.c +0
-0/* * Report the content-level differences with HAS_CHANGES; diff.h +4
-0@@ -234,6 +234,10 @@ struct diff_options {*/ const char *pickaxe; +/* -I<regex> */ +regex_t **ignore_regex; +size_t ignore_regex_nr, ignore_regex_alloc; +const char *single_follow;const char *a_prefix, *b_prefix;const char *line_prefix;t/t4013-diff-various.sh +33
-0@@ -6,6 +6,7 @@ test_description='Various diff formatting options' . ./test-lib.sh +. "$TEST_DIRECTORY"/diff-lib.sh test_expect_success setup '@@ -473,4 +474,36 @@ test_expect_success 'diff-tree --stdin with log formatting' ' test_cmp expect actual ' +test_expect_success 'diff -I<regex>' ' + test_seq 50 >I.txt && + sed -e "s/13/ten and three/" -e "/7\$/d" <I.txt >J.txt && + echo >>J.txt && + + test_expect_code 1 git diff --no-index --ignore-blank-lines -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual && + cat >expect <<-\EOF && + diff --git a/I.txt b/J.txt + --- a/I.txt + +++ b/J.txt + @@ -34,7 +31,6 @@ + 34 + 35 + 36 + -37 + 38 + 39 + 40 + EOF + compare_diff_patch expect actual && + + test_expect_code 1 git diff --stat --no-index --ignore-blank-lines -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual && + cat >expect <<-\EOF && + I.txt => J.txt | 1 - + 1 file changed, 1 deletion(-) + EOF + test_cmp expect actual && + + test_expect_code 129 git diff --no-index --ignore-matching-lines="^[124-9]" --ignore-matching-lines="^[124-9" I.txt J.txt >output 2>&1 && + test_i18ngrep "invalid regex given to -I: " output +' + test_donexdiff/xdiff.h +4
-0@@ -79,6 +79,10 @@ typedef struct s_mmbuffer { typedef struct s_xpparam { unsigned long flags; + /* -I<regex> */ + regex_t **ignore_regex; + size_t ignore_regex_nr; + /* See Documentation/diff-options.txt. */ char **anchors; size_t anchors_nr;xdiff/xdiffi.c +45
-2@@ -998,7 +998,7 @@ static int xdl_call_hunk_func(xdfenv_t *xe, xdchange_t *xscr, xdemitcb_t *ecb,return 0 ; } -static void xdl_mark_ignorable (xdchange_t *xscr,xdfenv_t *xe,long flags) +static void xdl_mark_ignorable_lines (xdchange_t *xscr,xdfenv_t *xe,long flags) {xdchange_t *xch;@@ -1019,6 +1019,46 @@ static void xdl_mark_ignorable(xdchange_t *xscr, xdfenv_t *xe, long flags) } } +static int record_matches_regex (xrecord_t *rec,xpparam_t const *xpp) { +regmatch_t regmatch; +int i; + +for (i =0 ; i < xpp->ignore_regex_nr; i++) +if (!regexec_buf (xpp->ignore_regex[i], rec->ptr, rec->size,1 , + ®match,0 )) +return 1 ; + +return 0 ; +} + +static void xdl_mark_ignorable_regex (xdchange_t *xscr,const xdfenv_t *xe, +xpparam_t const *xpp) +{ +xdchange_t *xch; + +for (xch = xscr; xch; xch = xch->next) { +xrecord_t **rec; +int ignore =1 ; +long i; + +/* +* Do not override --ignore-blank-lines. +*/ +if (xch->ignore) +continue ; + + rec = &xe->xdf1.recs[xch->i1]; +for (i =0 ; i < xch->chg1 && ignore; i++) + ignore =record_matches_regex (rec[i], xpp); + + rec = &xe->xdf2.recs[xch->i2]; +for (i =0 ; i < xch->chg2 && ignore; i++) + ignore =record_matches_regex (rec[i], xpp); + + xch->ignore = ignore; + } +} +int xdl_diff (mmfile_t *mf1,mmfile_t *mf2,xpparam_t const *xpp,xdemitconf_t const *xecfg,xdemitcb_t *ecb) {xdchange_t *xscr;@@ -1038,7 +1078,10 @@ int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, }if (xscr) {if (xpp->flags &XDF_IGNORE_BLANK_LINES ) -xdl_mark_ignorable (xscr, &xe, xpp->flags); +xdl_mark_ignorable_lines (xscr, &xe, xpp->flags); + +if (xpp->ignore_regex) +xdl_mark_ignorable_regex (xscr, &xe, xpp);if (ef (&xe, xscr, ecb, xecfg) <0 ) {
Quoted 12 lines · Phillip Wood
diff.c void diff_flush(struct diff_options *options) · line 6435
Quoted 9 lines
diff.c +5
-0@@ -6405,6 +6428,11 @@ void diff_flush(struct diff_options *options)DIFF_QUEUE_CLEAR (q);if (options->close_file)fclose (options->file); +for (i =0 ; i < options->ignore_regex_nr; i++) { +regfree (options->ignore_regex[i]); +free (options->ignore_regex[i]); + } +free (options->ignore_regex);If I run
git log -p -I foothen the address sanitizer reports
Quoted 5 lines
AddressSanitizer: heap-use-after-free
xdiff/xdiffi.c:1027inrecord_matches_regexafter it has printed the diff for the first commit. I think freeing the regex here is the cause of the problem.
Yes, this is a good location to clear the diff_queue, which is per-invocation. But diff_options->ignore_regex[] can and should be shared across multiple invocations of the diff machinery, as we are parsing upfront in the command line option parser just once to be used throughout the life of the program. It is wrong to free them early like this.
I really hate to suggest this, one common API call made into the diff machinery from various consumers, when they are absolutely done with diff_options, is probably diff_result_code(). Its purpose is to collect bits computed to participate in the overall result into the final result code and anybody who ran one or more diff and wants to learn the outcome must call it, so it is unlikely that callers that matter are missing a call to finalize their uses of the diff machinery. So if freeing .ignore_regex[] is really necessary, it could be used to tell us where to do so [*1*].
Unlike per-invocation resources that MUST be freed for repeated invocations of the diff machinery made in "git log" family, resources held for and repeatedly used during the entire session like this may not have to be freed, to be honest, though.
Thanks.
[Footnote]
*1* Adding calls to free() in diff_result_code() directly is
probably not a good idea. Some callers may not even need result code (e.g. "blame") and do not call it at all, but we may want to free the resource in diff_options that is not per-invocation. So if we were to do this, we probably need a new API function, perhaps diff_options_clear() or something, and call that from diff_result_code(), and then find callers that do not care about the result code and call diff_options_clear() directly from them.
If a caller does a single diff_setup_done(), makes repeated calls to the diff machinery, and calls diff_result_code() once per each iteration (e.g. imagine "git log -p" that detects the status for each commit), then having the diff_options_clear() call in diff_result_code() will break the machinery, so if we were to follow the outline given in the previous paragraph, we need to make sure there is no such caller. But I see that diff_result_code() does not clear the fields it looks at in the diff_options fields after it uses them, so the second and subsequent calls into the diff machinery by such a caller may already be broken (not in the "resource leakage" sense, but in the correctness sense).
First, thanks to Phillip for testing this.
Quoted 6 lines
Yes, this is a good location to clear the
diff_queue, which is per-invocation. But diff_options->ignore_regex[] can and should be shared across multiple invocations of the diff machinery, as we are parsing upfront in the command line option parser just once to be used throughout the life of the program. It is wrong to free them early like this.
Ugh, sorry, I am only now starting to see the big picture.
Quoted 10 lines
I really hate to suggest this, one common API call made into the diff machinery from various consumers, when they are absolutely done with
diff_options, is probablydiff_result_code(). Its purpose is to collect bits computed to participate in the overall result into the final result code and anybody who ran one or more diff and wants to learn the outcome must call it, so it is unlikely that callers that matter are missing a call to finalize their uses of the diff machinery. So if freeing .ignore_regex[] is really necessary, it could be used to tell us where to do so [*1*].
Quoted 4 lines
Unlike per-invocation resources that MUST be freed for repeated invocations of the diff machinery made in "git log" family, resources held for and repeatedly used during the entire session like this may not have to be freed, to be honest, though.
After reading your message, taking a closer look, and doing a few Valgrind runs, I tend to agree that perhaps there is no need to free the 'ignore_regex' array after all - it does *not* get allocated repeatedly during each invocation of the diff machinery and when Git exits, the OS will reclaim any memory allocated by the process anyway. On my laptop, a Valgrind run done for a -DSUPPRESS_ANNOTATED_LEAKS build claims that 'ignore_regex' is one of 1,300+ memory blocks still reachable at exit time, which amount to about 2.5 MB of memory in total. Given this, it feels slightly weird to single 'ignore_regex' out.
I will revert the diff_flush() changes in v4 unless somebody disagrees with the above reasoning.
Signature · 2 lines
Best regards, Michał Kępień
t/t4013-diff-various.sh test_expect_success 'diff-tree --stdin with log formatting' ' · line 482
Quoted 6 lines · Michał Kępień
t/t4013-diff-various.sh +6
-0+test_expect_success 'diff -I<regex>' ' + test_seq 50 >I.txt && + sed -e "s/13/ten and three/" -e "/7\$/d" <I.txt >J.txt && + echo >>J.txt && + + test_expect_code 1 git diff --no-index --ignore-blank-lines -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual &&
Please wrap an overly long line like this oned, perhaps like:
test_expect_code 1 git diff --no-index --ignore-blank-lines \ -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual &&
t/t4013-diff-various.sh test_expect_success 'diff-tree --stdin with log formatting' ' · line 505
Quoted 3 lines
t/t4013-diff-various.sh +2
-0+ test_expect_code 1 git diff --stat --no-index --ignore-blank-lines -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual && + test_expect_code 129 git diff --no-index --ignore-matching-lines="^[124-9]" --ignore-matching-lines="^[124-9" I.txt J.txt >output 2>&1 &&
Cramming unrelated tests into a single one made me puzzled, staring at this line for longer than necessary before realizing that this is an attempt to catch a malformed regexp. If this were in a separate test with its own title, e.g.
test_expect_success 'diff -I<regex>: setup' ' ... set up the sample input, perhaps sample history ... ... perhaps prepare the expected output in "expect" ... ' test_expect_success 'diff -I<regex> -p' ' git diff --I"ten.*e" ... >actual && test_cmp expect actual ' test_expect_success 'diff -I<regex> --stat' ' git diff --stat --I"ten.*e" ... >actual && test_cmp expect actual ' test_expect_success 'diff -I<regex>: detect malformed regex' test_must_fail git diff -I="[^124-9" ... 2>error && test_i18ngrep "invalid regex" error '
It would have been much easier to follow.
It also is curious that it only tests --no-index, which is a bolted on feature that may not exercise the codepath the users depend on working correctly. If this were tested with "git log -p" for a few commits, the early destruction of regexp may have been caught, especially with ASAN build.
Other than that, and also the premature destruction of regexp pointed out by Phillip already, looks reasonably done.
Thanks.
Quoted 11 lines
t/t4013-diff-various.sh test_expect_success 'diff-tree --stdin with log formatting' ' · line 482
Quoted 6 lines · Michał Kępień
t/t4013-diff-various.sh +6
-0+test_expect_success 'diff -I<regex>' ' + test_seq 50 >I.txt && + sed -e "s/13/ten and three/" -e "/7\$/d" <I.txt >J.txt && + echo >>J.txt && + + test_expect_code 1 git diff --no-index --ignore-blank-lines -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual &&Please wrap an overly long line like this oned, perhaps like:
Quoted 2 lines
test_expect_code 1 git diff --no-index --ignore-blank-lines \ -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual &&
Yeah, thanks, I was unsure what the line length limit for test code is. I will fix this in v4.
Quoted 11 lines
t/t4013-diff-various.sh test_expect_success 'diff-tree --stdin with log formatting' ' · line 505
Quoted 3 lines
t/t4013-diff-various.sh +2
-0+ test_expect_code 1 git diff --stat --no-index --ignore-blank-lines -I"ten.*e" -I"^[124-9]" I.txt J.txt >actual && + test_expect_code 129 git diff --no-index --ignore-matching-lines="^[124-9]" --ignore-matching-lines="^[124-9" I.txt J.txt >output 2>&1 &&Cramming unrelated tests into a single one made me puzzled, staring at this line for longer than necessary before realizing that this is an attempt to catch a malformed regexp. If this were in a separate test with its own title, e.g.
(...)
Quoted 7 lines
It would have been much easier to follow.
It also is curious that it only tests --no-index, which is a bolted on feature that may not exercise the codepath the users depend on working correctly. If this were tested with "git log -p" for a few commits, the early destruction of regexp may have been caught, especially with ASAN build.
Sorry, I think I got a bit carried away trying to use the test suggested by Johannes while also extending it with the missing bits. I will try to come up with something more balanced in v4.
Signature · 2 lines
Best regards, Michał Kępień
Quoted 8 lines · Michał Kępień
Quoted 6 lines
Cramming unrelated tests into a single one made me puzzled, staring at this line for longer than necessary before realizing that this is an attempt to catch a malformed regexp. If this were in a separate test with its own title, e.g.
It would have been much easier to follow.
... I will try
to come up with something more balanced in v4.
We want to avoid going overboard and doing a full test matrix.
But having it tested fully with one variant while testing just the basic for other variants may give us a good balance, e.g. "git log -p -I<regex>" is tested but we do not bother testing interactions with other options", while we try different combinations when testing "git diff --no-index -I<regex>", like multiple -I options etc.
Thanks.
Hi Michał,
Quoted 30 lines · Michał Kępień
This patch series adds a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I/--ignore-matching-lines option in standalone diff utilities and can be used e.g. to ignore changes which only affect code comments or to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
Changes from v2:
- Add a long option for -I (--ignore-matching-lines) as it is commonplace in standalone diff utilities. Update documentation and commit log messages accordingly.
- Use
xmalloc()instead ofxcalloc()for allocatingregex_tstructures indiff_opt_ignore_regex().- Ensure the memory allocated in
diff_opt_ignore_regex()gets released.- Use "return error(...)" instead of
die()in the -I option callback. Make the relevant error message localizable.- Drastically reduce the number of -I<regex> tests due to excessive run time of
t/t4069-diff-ignore-regex.shfrom v1/v2 on some platforms (notably Windows). Use a tweaked version of a test suggested by Johannes Schindelin (thanks!). Given its reduction in size, squash patch 3 (which contained the tests) into patch 2.
Quoted 3 lines
- Replace "see
Documentation/diff-options.txt" with "-I<regex>" in the comments for the added structure fields, in order to make these comments more useful.
Thank you for this diligent work! I looked over the patches and like them a lot!
Thanks,
Dscho
v4 cover diff: add -I<regex> that ignores matching changes
In reply to Michał KępieńThis patch series adds a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I/--ignore-matching-lines option in standalone diff utilities and can be used e.g. to ignore changes which only affect code comments or to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
Changes from v3:
- Do not release the memory allocated in diff_opt_ignore_regex() after all as the free() calls added in v2 were triggering use-after-free errors when the diff machinery was invoked multiple times throughout the lifetime of a single Git command (e.g. by "git log -p").
- Further test improvements: split various -I<regex> checks into multiple tests; add a check for "git log -p -I<regex>" to trigger the issue mentioned above; avoid using --no-index.
Changes from v2:
- Add a long option for -I (--ignore-matching-lines) as it is commonplace in standalone diff utilities. Update documentation and commit log messages accordingly.
- Use xmalloc() instead of xcalloc() for allocating regex_t structures in diff_opt_ignore_regex().
- Ensure the memory allocated in diff_opt_ignore_regex() gets released.
- Use "return error(...)" instead of die() in the -I option callback. Make sure the error message is localizable.
- Drastically reduce the number of -I<regex> tests due to excessive run time of t/t4069-diff-ignore-regex.sh from v1/v2 on some platforms (notably Windows). Use a tweaked version of a test suggested by Johannes Schindelin (thanks!). Squash patch 3 into patch 2.
- Replace "see Documentation/diff-options.txt" with "-I<regex>" in the comments for the added structure fields, in order to make these comments more useful.
Changes from v1:
- Add a new preliminary cleanup patch which ensures xpparam_t structures are always zero-initialized. (This was a prerequisite for the next change below.)
- Do not add a new 'xdl_opts' flag to check whether -I was used; instead, just check whether the array of regular expressions to match against is non-NULL.
- Enable the -I option to be used multiple times. As a consequence of this, regular expressions are now "pre-compiled" in the option's callback (and passed around as an array of regex_t structures) rather than deep down in xdiff code. Add test cases exercising use of multiple -I options in the same git invocation. Update documentation accordingly.
- Rename xdl_mark_ignorable() to xdl_mark_ignorable_lines(), to indicate that it is logically a "sibling" of xdl_mark_ignorable_regex() rather than its "parent".
- Optimize xdl_mark_ignorable_regex() by making it immediately skip changes already marked as ignored by xdl_mark_ignorable_lines().
- Fix coding style issue in the prototype part of the definition of xdl_mark_ignorable_regex().
- Add "/* see Documentation/diff-options.txt */" comments for the fields added to struct diff_options and xpparam_t, mimicking the comments used for 'anchors', 'anchors_nr', and 'anchors_alloc'.
- Revise commit log messages to reflect all of the above.
Michał Kępień (2):
merge-base, xdiff: zero out xpparam_t structures
diff: add -I<regex> that ignores matching changes
Documentation/diff-options.txt | 5 ++ builtin/merge-tree.c | 1 + diff.c | 23 +++++ diff.h | 4 + t/t4013-diff-various.sh | 41 +++++++++ t/t4013/diff.log_-IA_-IB_-I1_-I2_-p_master | 99 ++++++++++++++++++++++ xdiff/xdiff.h | 4 + xdiff/xdiffi.c | 47 +++++++++- xdiff/xhistogram.c | 2 + xdiff/xpatience.c | 2 + 10 files changed, 226 insertions(+), 2 deletions(-) create mode 100644 t/t4013/diff.log_-IA_-IB_-I1_-I2_-p_master
Signature · 1 line
2.28.0
PATCH v4 1/2 merge-base, xdiff: zero out xpparam_t structures
xpparam_t structures are usually zero-initialized before their specific fields are assigned to, but there are three locations in the tree where that does not happen. Add the missing memset() calls in order to make initialization of xpparam_t structures consistent tree-wide and to prevent stack garbage from being used as field values.
--- builtin/merge-tree.c | 1 + xdiff/xhistogram.c | 2 ++ xdiff/xpatience.c | 2 ++ 3 files changed, 5 insertions(+)
builtin/merge-tree.c +1 -0
@@ -109,6 +109,7 @@ static void show_diff(struct merge_list *entry)xdemitconf_t xecfg;xdemitcb_t ecb; +memset (&xpp,0 ,sizeof (xpp)); xpp.flags =0 ;memset (&xecfg,0 ,sizeof (xecfg)); xecfg.ctxlen =3 ;
xdiff/xhistogram.c +2 -0
@@ -235,6 +235,8 @@ static int fall_back_to_classic_diff(xpparam_t const *xpp, xdfenv_t *env, int line1, int count1, int line2, int count2) { xpparam_t xpparam; + +memset (&xpparam,0 ,sizeof (xpparam)); xpparam.flags = xpp->flags & ~XDF_DIFF_ALGORITHM_MASK ; return xdl_fall_back_diff(env, &xpparam,
xdiff/xpatience.c +2 -0
@@ -318,6 +318,8 @@ static int fall_back_to_classic_diff(struct hashmap *map, int line1, int count1, int line2, int count2) { xpparam_t xpp; + +memset (&xpp,0 ,sizeof (xpp)); xpp.flags = map->xpp->flags & ~XDF_DIFF_ALGORITHM_MASK ; return xdl_fall_back_diff(map->env, &xpp,
Signature · 1 line
2.28.0
PATCH v4 2/2 diff: add -I<regex> that ignores matching changes
In reply to Michał KępieńAdd a new diff option that enables ignoring changes whose all lines (changed, removed, and added) match a given regular expression. This is similar to the -I/--ignore-matching-lines option in standalone diff utilities and can be used e.g. to ignore changes which only affect code comments or to look for unrelated changes in commits containing a large number of automatically applied modifications (e.g. a tree-wide string replacement). The difference between -G/-S and the new -I option is that the latter filters output on a per-change basis.
Use the 'ignore' field of xdchange_t for marking a change as ignored or not. Since the same field is used by --ignore-blank-lines, identical hunk emitting rules apply for --ignore-blank-lines and -I. These two options can also be used together in the same git invocation (they are complementary to each other).
Rename xdl_mark_ignorable() to xdl_mark_ignorable_lines(), to indicate that it is logically a "sibling" of xdl_mark_ignorable_regex() rather than its "parent".
--- Documentation/diff-options.txt | 5 ++ diff.c | 23 +++++ diff.h | 4 + t/t4013-diff-various.sh | 41 +++++++++ t/t4013/diff.log_-IA_-IB_-I1_-I2_-p_master | 99 ++++++++++++++++++++++ xdiff/xdiff.h | 4 + xdiff/xdiffi.c | 47 +++++++++- 7 files changed, 221 insertions(+), 2 deletions(-) create mode 100644 t/t4013/diff.log_-IA_-IB_-I1_-I2_-p_master
Documentation/diff-options.txt +5 -0
@@ -687,6 +687,11 @@ endif::git-format-patch[] --ignore-blank-lines:: Ignore changes whose lines are all blank. +-I<regex>:: +--ignore-matching-lines=<regex>:: + Ignore changes whose all lines match <regex>. This option may + be specified more than once. + --inter-hunk-context=<lines>:: Show the context between diff hunks, up to the specified number of lines, thereby fusing hunks that are close to each other.
diff.c +23 -0
@@ -3587,6 +3587,8 @@ static void builtin_diff(const char *name_a,if (header.len && !o->flags.suppress_diff_headers) ecbdata.header = &header; xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; + xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;
@@ -3716,6 +3718,8 @@ static void builtin_diffstat(const char *name_a, const char *name_b,memset (&xpp,0 ,sizeof (xpp));memset (&xecfg,0 ,sizeof (xecfg)); xpp.flags = o->xdl_opts; + xpp.ignore_regex = o->ignore_regex; + xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; xecfg.ctxlen = o->context;
@@ -5203,6 +5207,22 @@ static int diff_opt_patience(const struct option *opt,return 0 ; } +static int diff_opt_ignore_regex (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; +regex_t *regex; + +BUG_ON_OPT_NEG (unset); + regex =xmalloc (sizeof (*regex)); +if (regcomp (regex, arg,REG_EXTENDED |REG_NEWLINE )) +return error (_ ("invalid regex given to -I: '%s'" ), arg); +ALLOC_GROW (options->ignore_regex, options->ignore_regex_nr +1 , + options->ignore_regex_alloc); + options->ignore_regex[options->ignore_regex_nr++] = regex; +return 0 ; +} +static int diff_opt_pickaxe_regex (const struct option *opt,const char *arg,int unset) {
@@ -5491,6 +5511,9 @@ static void prep_parse_options(struct diff_options *options)OPT_BIT_F (0 ,"ignore-blank-lines" , &options->xdl_opts,N_ ("ignore changes whose lines are all blank" ),XDF_IGNORE_BLANK_LINES ,PARSE_OPT_NONEG ), +OPT_CALLBACK_F ('I' ,"ignore-matching-lines" , options,N_ ("<regex>" ), +N_ ("ignore changes whose all lines match <regex>" ), +0 , diff_opt_ignore_regex),OPT_BIT (0 ,"indent-heuristic" , &options->xdl_opts,N_ ("heuristic to shift diff hunk boundaries for easy reading" ), XDF_INDENT_HEURISTIC),
diff.h +4 -0
@@ -234,6 +234,10 @@ struct diff_options {*/ const char *pickaxe; +/* -I<regex> */ +regex_t **ignore_regex; +size_t ignore_regex_nr, ignore_regex_alloc; +const char *single_follow;const char *a_prefix, *b_prefix;const char *line_prefix;
t/t4013-diff-various.sh +41 -0
57 lines not on this page · raw message
t/t4013/diff.log_-IA_-IB_-I1_-I2_-p_master new file +99 -0
99 lines not on this page · raw message
xdiff/xdiff.h +4 -0
@@ -79,6 +79,10 @@ typedef struct s_mmbuffer { typedef struct s_xpparam { unsigned long flags; + /* -I<regex> */ + regex_t **ignore_regex; + size_t ignore_regex_nr; + /* See Documentation/diff-options.txt. */ char **anchors; size_t anchors_nr;
xdiff/xdiffi.c +45 -2
65 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH v2 1/2 diff: add an API for deferred freeing
In reply to Michał KępieńAdd a diff_free() function to free anything we may have allocated in the "diff_options" struct, and the ability to make calling it a noop by setting "no_free" in "diff_options".
This is required because when e.g. "git diff" is run we'll allocate things in that struct, use the diff machinery once, and then exit, but if we run e.g. "git log -p" we're going to re-use what we allocated across multiple diff_flush() calls, and only want to free things at the end.
We've thus ended up with features like the recently added "diff -I"[1] where we'll leak memory. As it turns out it could have simply used the pattern established in 6ea57703f6 (log: prepare log/log-tree to reuse the diffopt.close_file attribute, 2016-06-22).
Manually adding more such flags to things log_tree_commit() every time we need to allocate something would be tedious.
Let's instead move that fclose() code it to a new diff_free(), in anticipation of freeing more things in that function in follow-up commits. I'm renaming the "close_file" struct member to "fclose_file" for the ease of validating this, we can be certain that these are all the relevant callsites.
Some functions such as log_tree_commit() need an idiom of optionally retaining a previous "no_free", as they may either free the memory themselves, or their caller may do so. I'm keeping that idiom in log_show_early() even though I don't think it's currently called in this manner, since it also gets passed an existing "struct rev_info"..
1. 296d4a94e7 (diff: add -I<regex> that ignores matching changes, 2020-10-20)
--- builtin/add.c | 2 +- builtin/am.c | 6 ++++-- builtin/log.c | 27 ++++++++++++++------------- diff.c | 18 +++++++++++++----- diff.h | 14 +++++++++++++- log-tree.c | 10 ++++++---- wt-status.c | 2 +- 7 files changed, 52 insertions(+), 27 deletions(-)
builtin/add.c +1 -1
@@ -282,7 +282,7 @@ static int edit_patch(int argc, const char **argv, const char *prefix)if (out <0 )die (_ ("Could not open '%s' for writing." ), file); rev.diffopt.file =xfdopen (out,"w" ); - rev.diffopt.close_file =1 ; + rev.diffopt.fclose_file =1 ;if (run_diff_files (&rev,0 ))die (_ ("Could not write patch" ));
builtin/am.c +4 -2
@@ -1315,7 +1315,7 @@ static void write_commit_patch(const struct am_state *state, struct commit *comm rev_info.diffopt.flags.full_index =1 ; rev_info.diffopt.use_color =0 ; rev_info.diffopt.file = fp; - rev_info.diffopt.close_file =1 ; + rev_info.diffopt.fclose_file =1 ;/* log_tree_commit() sets .no_free=1 */ add_pending_object (&rev_info, &commit->object,"" );diff_setup_done (&rev_info.diffopt);log_tree_commit (&rev_info, commit);
@@ -1347,10 +1347,12 @@ static void write_index_patch(const struct am_state *state) rev_info.diffopt.output_format =DIFF_FORMAT_PATCH ; rev_info.diffopt.use_color =0 ; rev_info.diffopt.file = fp; - rev_info.diffopt.close_file =1 ; + rev_info.diffopt.fclose_file =1 ; + rev_info.diffopt.no_free =1 ;add_pending_object (&rev_info, &tree->object,"" );diff_setup_done (&rev_info.diffopt);run_diff_index (&rev_info,1 ); +diff_free (&rev_info.diffopt); }/**
builtin/log.c +14 -13
@@ -306,10 +306,11 @@ static struct itimerval early_output_timer;static void log_show_early (struct rev_info *revs,struct commit_list *list) { -int i = revs->early_output, close_file = revs->diffopt.close_file; +int i = revs->early_output;int show_header =1 ; +int no_free = revs->diffopt.no_free; - revs->diffopt.close_file =0 ; + revs->diffopt.no_free =0 ;sort_in_topological_order (&list, revs->sort_order);while (list && i) {struct commit *commit = list->item;
@@ -326,8 +327,8 @@ static void log_show_early(struct rev_info *revs, struct commit_list *list)case commit_ignore:break ;case commit_error: -if (close_file) -fclose (revs->diffopt.file); + revs->diffopt.no_free = no_free; +diff_free (&revs->diffopt);return ; } list = list->next;
@@ -335,8 +336,8 @@ static void log_show_early(struct rev_info *revs, struct commit_list *list)/* Did we already get enough commits for the early output? */ if (!i) { -if (close_file) -fclose (revs->diffopt.file); + revs->diffopt.no_free =0 ; +diff_free (&revs->diffopt);return ; }
@@ -400,7 +401,7 @@ static int cmd_log_walk(struct rev_info *rev) {struct commit *commit;int saved_nrl =0 ; -int saved_dcctc =0 , close_file = rev->diffopt.close_file; +int saved_dcctc =0 ;if (rev->early_output)setup_early_output ();
@@ -416,7 +417,7 @@ static int cmd_log_walk(struct rev_info *rev)* and HAS_CHANGES being accumulated in rev->diffopt, so be careful to * retain that state information if replacing rev->diffopt in this loop */ - rev->diffopt.close_file =0 ; + rev->diffopt.no_free =1 ;while ((commit =get_revision (rev)) !=NULL ) {if (!log_tree_commit (rev, commit) && rev->max_count >=0 )/*
@@ -441,8 +442,8 @@ static int cmd_log_walk(struct rev_info *rev) } rev->diffopt.degraded_cc_to_c = saved_dcctc; rev->diffopt.needed_rename_limit = saved_nrl; -if (close_file) -fclose (rev->diffopt.file); + rev->diffopt.no_free =0 ; +diff_free (&rev->diffopt);if (rev->diffopt.output_format &DIFF_FORMAT_CHECKDIFF && rev->diffopt.flags.check_failed) {
@@ -1952,18 +1953,18 @@ int cmd_format_patch(int argc, const char **argv, const char *prefix)if (rev.show_notes)load_display_notes (&rev.notes_opt); -if (use_stdout + rev.diffopt.close_file + !!output_directory >1 ) +if (use_stdout + rev.diffopt.fclose_file + !!output_directory >1 )die (_ ("--stdout, --output, and --output-directory are mutually exclusive" ));if (use_stdout) {setup_pager (); - }else if (rev.diffopt.close_file) { + }else if (rev.diffopt.fclose_file) {/* * The diff code parsed --output; it has already opened the * file, but but we must instruct it not to close after each * diff. */ - rev.diffopt.close_file =0 ; + rev.diffopt.no_free =1 ; }else {int saved;
diff.c +13 -5
@@ -5187,7 +5187,7 @@ static enum parse_opt_result diff_opt_output(struct parse_opt_ctx_t *ctx,BUG_ON_OPT_NEG (unset); path =prefix_filename (ctx->prefix, arg); options->file =xfopen (path,"w" ); - options->close_file =1 ; + options->fclose_file =1 ;if (options->use_color !=GIT_COLOR_ALWAYS ) options->use_color =GIT_COLOR_NEVER ;free (path);
@@ -6399,10 +6399,10 @@ void diff_flush(struct diff_options *options)* options->file to /dev/null should be safe, because we * aren't supposed to produce any output anyway. */ -if (options->close_file) +if (options->fclose_file)fclose (options->file); options->file =xfopen ("/dev/null" ,"w" ); - options->close_file =1 ; + options->fclose_file =1 ; options->color_moved =0 ;for (i =0 ; i < q->nr; i++) {struct diff_filepair *p = q->queue[i];
@@ -6433,8 +6433,7 @@ void diff_flush(struct diff_options *options) free_queue:free (q->queue);DIFF_QUEUE_CLEAR (q); -if (options->close_file) -fclose (options->file); +diff_free (options);/* * Report the content-level differences with HAS_CHANGES;
@@ -6449,6 +6448,15 @@ void diff_flush(struct diff_options *options) } } +void diff_free (struct diff_options *options) +{ +if (options->no_free) +return ; +if (options->fclose_file) +fclose (options->file); +} + +static int match_filter (const struct diff_options *options,const struct diff_filepair *p) { return (((p->status == DIFF_STATUS_MODIFIED) &&
diff.h +13 -1
@@ -49,7 +49,16 @@* - Once you finish feeding the pairs of files, call `diffcore_std()`. * This will tell the diffcore library to go ahead and do its work. * +* - The `diff_opt_parse()` etc. functions might allocate memory in +* `struct diff_options`. When running the API `N > 1` set `.no_free +* = 1` to make the `diff_free()` invoked by `diff_flush()` below a +* noop. +* * - Calling `diff_flush()` will produce the output. +* +* - If you set `.no_free = 1` before set it to `0` and call +* `diff_free()` again. If `.no_free = 1` was not set there's no +* need to call `diff_free()`, `diff_flush()` will call it. */ struct combine_diff_path ;
@@ -328,7 +337,7 @@ struct diff_options {void (*set_default)(struct diff_options *);FILE *file; -int close_file; +int fclose_file;#define OUTPUT_INDICATOR_NEW 0#define OUTPUT_INDICATOR_OLD 1
@@ -365,6 +374,8 @@ struct diff_options {struct repository *repo;struct option *parseopts; + +int no_free; };unsigned diff_filter_bit (char status);
@@ -559,6 +570,7 @@ void diffcore_fix_diff_index(void);int diff_queue_is_empty (void );void diff_flush (struct diff_options *); +void diff_free (struct diff_options *);void diff_warn_rename_limit (const char *varname,int needed,int degraded_cc);/* diff-raw status letters */
log-tree.c +6 -4
@@ -952,12 +952,14 @@ static int log_tree_diff(struct rev_info *opt, struct commit *commit, struct logint log_tree_commit (struct rev_info *opt,struct commit *commit) {struct log_info log; -int shown, close_file = opt->diffopt.close_file; +int shown; +/* maybe called by e.g. cmd_log_walk(), maybe stand-alone */ +int no_free = opt->diffopt.no_free; log.commit = commit; log.parent =NULL ; opt->loginfo = &log; - opt->diffopt.close_file =0 ; + opt->diffopt.no_free =1 ;if (opt->line_level_traverse)return line_log_print (opt, commit);
@@ -974,7 +976,7 @@ int log_tree_commit(struct rev_info *opt, struct commit *commit)fprintf (opt->diffopt.file,"\n%s\n" , opt->break_bar); opt->loginfo =NULL ;maybe_flush_or_die (opt->diffopt.file,"stdout" ); -if (close_file) -fclose (opt->diffopt.file); + opt->diffopt.no_free = no_free; +diff_free (&opt->diffopt);return shown; }
wt-status.c +1 -1
@@ -1052,7 +1052,7 @@ static void wt_longstatus_print_verbose(struct wt_status *s) rev.diffopt.rename_limit = s->rename_limit >=0 ? s->rename_limit : rev.diffopt.rename_limit; rev.diffopt.rename_score = s->rename_score >=0 ? s->rename_score : rev.diffopt.rename_score; rev.diffopt.file = s->fp; - rev.diffopt.close_file =0 ; + rev.diffopt.fclose_file =0 ;/* wt_status owns the s->fp */ /* * If we're not going to stdout, then we definitely don't * want color, since we are going to the commit message
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7
Re: [PATCH 1/2] diff: add an API for deferred freeing
Hi Ævar,
Quoted 3 lines · Ævar Arnfjörð Bjarmason
Add a
diff_free()function to free anything we may have allocated in the "diff_options" struct, and the ability to make calling it a noop by setting "no_free" in "diff_options".
Why do we need a no_free flag? Why not simply set the free()d (or fclose()d) attributes to NULL?
Quoted 11 lines
This is required because when e.g. "git diff" is run we'll allocate things in that struct, use the diff machinery once, and then exit, but if we run e.g. "git log -p" we're going to re-use what we allocated across multiple
diff_flush()calls, and only want to free things at the end.We've thus ended up with features like the recently added "diff -I"[1] where we'll leak memory. As it turns out it could have simply used the pattern established in
6ea57703f6(log: prepare log/log-tree to reuse the diffopt.close_file attribute, 2016-06-22).
Quoted 8 lines
Manually adding more such flags to things
log_tree_commit()every time we need to allocate something would be tedious.Let's instead move that
fclose()code it to a newdiff_free(), in anticipation of freeing more things in that function in follow-up commits. I'm renaming the "close_file" struct member to "fclose_file" for the ease of validating this, we can be certain that these are all the relevant callsites.
I like that idea a lot.
Quoted 25 lines
Some functions such as
log_tree_commit()need an idiom of optionally retaining a previous "no_free", as they may either free the memory themselves, or their caller may do so. I'm keeping that idiom inlog_show_early()even though I don't think it's currently called in this manner, since it also gets passed an existing "structrev_info"..1.
296d4a94e7(diff: add -I<regex> that ignores matching changes, 2020-10-20)Signed-off-by Ævar Arnfjörð Bjarmason <[email protected]>--- builtin/add.c | 2 +- builtin/am.c | 6 ++++-- builtin/log.c | 27 ++++++++++++++------------- diff.c | 18 +++++++++++++----- diff.h | 14 +++++++++++++- log-tree.c | 10 ++++++---- wt-status.c | 2 +- 7 files changed, 52 insertions(+), 27 deletions(-)builtin/add.c +0
-0
builtin/add.c static int edit_patch(int argc, const char **argv, const char *prefix) · line 285
Quoted 5 lines
builtin/add.c +1
-1@@ -282,7 +282,7 @@ static int edit_patch(int argc, const char **argv, const char *prefix)if (out <0 )die (_ ("Could not open '%s' for writing." ), file); rev.diffopt.file =xfdopen (out,"w" ); - rev.diffopt.close_file =1 ; + rev.diffopt.fclose_file =1 ;
This rename makes the patch unnecessarily tedious to review, and I do not even agree with leaking the implementation detail that we need to fclose() the file.
Let's just not?
Quoted 135 lines
builtin/add.c +0
-0if (run_diff_files (&rev,0 ))die (_ ("Could not write patch" ));builtin/am.c +4
-2@@ -1315,7 +1315,7 @@ static void write_commit_patch(const struct am_state *state, struct commit *comm rev_info.diffopt.flags.full_index =1 ; rev_info.diffopt.use_color =0 ; rev_info.diffopt.file = fp; - rev_info.diffopt.close_file =1 ; + rev_info.diffopt.fclose_file =1 ;/* log_tree_commit() sets .no_free=1 */ add_pending_object (&rev_info, &commit->object,"" );diff_setup_done (&rev_info.diffopt);log_tree_commit (&rev_info, commit);@@ -1347,10 +1347,12 @@ static void write_index_patch(const struct am_state *state) rev_info.diffopt.output_format =DIFF_FORMAT_PATCH ; rev_info.diffopt.use_color =0 ; rev_info.diffopt.file = fp; - rev_info.diffopt.close_file =1 ; + rev_info.diffopt.fclose_file =1 ; + rev_info.diffopt.no_free =1 ;add_pending_object (&rev_info, &tree->object,"" );diff_setup_done (&rev_info.diffopt);run_diff_index (&rev_info,1 ); +diff_free (&rev_info.diffopt); }/** builtin/log.c +14
-13@@ -306,10 +306,11 @@ static struct itimerval early_output_timer;static void log_show_early (struct rev_info *revs,struct commit_list *list) { -int i = revs->early_output, close_file = revs->diffopt.close_file; +int i = revs->early_output;int show_header =1 ; +int no_free = revs->diffopt.no_free; - revs->diffopt.close_file =0 ; + revs->diffopt.no_free =0 ;sort_in_topological_order (&list, revs->sort_order);while (list && i) {struct commit *commit = list->item;@@ -326,8 +327,8 @@ static void log_show_early(struct rev_info *revs, struct commit_list *list)case commit_ignore:break ;case commit_error: -if (close_file) -fclose (revs->diffopt.file); + revs->diffopt.no_free = no_free; +diff_free (&revs->diffopt);return ; } list = list->next;@@ -335,8 +336,8 @@ static void log_show_early(struct rev_info *revs, struct commit_list *list)/* Did we already get enough commits for the early output? */ if (!i) { -if (close_file) -fclose (revs->diffopt.file); + revs->diffopt.no_free =0 ; +diff_free (&revs->diffopt);return ; }@@ -400,7 +401,7 @@ static int cmd_log_walk(struct rev_info *rev) {struct commit *commit;int saved_nrl =0 ; -int saved_dcctc =0 , close_file = rev->diffopt.close_file; +int saved_dcctc =0 ;if (rev->early_output)setup_early_output ();@@ -416,7 +417,7 @@ static int cmd_log_walk(struct rev_info *rev)* and HAS_CHANGES being accumulated in rev->diffopt, so be careful to * retain that state information if replacing rev->diffopt in this loop */ - rev->diffopt.close_file =0 ; + rev->diffopt.no_free =1 ;while ((commit =get_revision (rev)) !=NULL ) {if (!log_tree_commit (rev, commit) && rev->max_count >=0 )/* @@ -441,8 +442,8 @@ static int cmd_log_walk(struct rev_info *rev) } rev->diffopt.degraded_cc_to_c = saved_dcctc; rev->diffopt.needed_rename_limit = saved_nrl; -if (close_file) -fclose (rev->diffopt.file); + rev->diffopt.no_free =0 ; +diff_free (&rev->diffopt);if (rev->diffopt.output_format &DIFF_FORMAT_CHECKDIFF && rev->diffopt.flags.check_failed) {@@ -1952,18 +1953,18 @@ int cmd_format_patch(int argc, const char **argv, const char *prefix)if (rev.show_notes)load_display_notes (&rev.notes_opt); -if (use_stdout + rev.diffopt.close_file + !!output_directory >1 ) +if (use_stdout + rev.diffopt.fclose_file + !!output_directory >1 )die (_ ("--stdout, --output, and --output-directory are mutually exclusive" ));if (use_stdout) {setup_pager (); - }else if (rev.diffopt.close_file) { + }else if (rev.diffopt.fclose_file) {/* * The diff code parsed --output; it has already opened the * file, but but we must instruct it not to close after each * diff. */ - rev.diffopt.close_file =0 ; + rev.diffopt.no_free =1 ; }else {int saved;diff.c +1
-1@@ -5187,7 +5187,7 @@ static enum parse_opt_result diff_opt_output(struct parse_opt_ctx_t *ctx,BUG_ON_OPT_NEG (unset); path =prefix_filename (ctx->prefix, arg); options->file =xfopen (path,"w" ); - options->close_file =1 ; + options->fclose_file =1 ;if (options->use_color !=GIT_COLOR_ALWAYS ) options->use_color =GIT_COLOR_NEVER ;free (path);
diff.c void diff_flush(struct diff_options *options) · line 6403
Quoted 6 lines
diff.c +1
-1@@ -6399,10 +6399,10 @@ void diff_flush(struct diff_options *options)* options->file to /dev/null should be safe, because we * aren't supposed to produce any output anyway. */ -if (options->close_file) +if (options->fclose_file)fclose (options->file);
And at this stage, we should set options->file = NULL.
Quoted 17 lines
diff.c +2
-3options->file =xfopen ("/dev/null" ,"w" ); - options->close_file =1 ; + options->fclose_file =1 ; options->color_moved =0 ;for (i =0 ; i < q->nr; i++) {struct diff_filepair *p = q->queue[i];@@ -6433,8 +6433,7 @@ void diff_flush(struct diff_options *options) free_queue:free (q->queue);DIFF_QUEUE_CLEAR (q); -if (options->close_file) -fclose (options->file); +diff_free (options);/* * Report the content-level differences with HAS_CHANGES;
diff.c void diff_flush(struct diff_options *options) · line 6456
Quoted 9 lines
diff.c +6
-0@@ -6449,6 +6448,15 @@ void diff_flush(struct diff_options *options) } } +void diff_free (struct diff_options *options) +{ +if (options->no_free) +return ; +if (options->fclose_file) +fclose (options->file);
And at this stage, we should set options->file = NULL.
Quoted 11 lines
diff.c +3
-0+} + +static int match_filter (const struct diff_options *options,const struct diff_filepair *p) { return (((p->status == DIFF_STATUS_MODIFIED) &&diff.h +0
-0
diff.h line 55
Quoted 7 lines
diff.h +4
-0@@ -49,7 +49,16 @@* - Once you finish feeding the pairs of files, call `diffcore_std()`. * This will tell the diffcore library to go ahead and do its work. * +* - The `diff_opt_parse()` etc. functions might allocate memory in +* `struct diff_options`. When running the API `N > 1` set `.no_free +* = 1` to make the `diff_free()` invoked by `diff_flush()` below a +* noop.
I have serious trouble parsing that last sentence. Would you mind rephrasing it?
diff.h line 61
Quoted 6 lines
diff.h +5
-0+* * - Calling `diff_flush()` will produce the output. +* +* - If you set `.no_free = 1` before set it to `0` and call +* `diff_free()` again. If `.no_free = 1` was not set there's no +* need to call `diff_free()`, `diff_flush()` will call it.
Again, as I mentioned before, the indicator whether things need to be released should be whether the attribute is NULL or not. And once released, ot should be set to NULL.
Other than that, it looks fine to me. And I definitely like the idea of introducing a function to release all of the stuff in struct diffopt.
Thanks,
Dscho
Quoted 77 lines
diff.h +4
-1*/ struct combine_diff_path ;@@ -328,7 +337,7 @@ struct diff_options {void (*set_default)(struct diff_options *);FILE *file; -int close_file; +int fclose_file;#define OUTPUT_INDICATOR_NEW 0#define OUTPUT_INDICATOR_OLD 1@@ -365,6 +374,8 @@ struct diff_options {struct repository *repo;struct option *parseopts; + +int no_free; };unsigned diff_filter_bit (char status);@@ -559,6 +570,7 @@ void diffcore_fix_diff_index(void);int diff_queue_is_empty (void );void diff_flush (struct diff_options *); +void diff_free (struct diff_options *);void diff_warn_rename_limit (const char *varname,int needed,int degraded_cc);/* diff-raw status letters */ log-tree.c +6
-4@@ -952,12 +952,14 @@ static int log_tree_diff(struct rev_info *opt, struct commit *commit, struct logint log_tree_commit (struct rev_info *opt,struct commit *commit) {struct log_info log; -int shown, close_file = opt->diffopt.close_file; +int shown; +/* maybe called by e.g. cmd_log_walk(), maybe stand-alone */ +int no_free = opt->diffopt.no_free; log.commit = commit; log.parent =NULL ; opt->loginfo = &log; - opt->diffopt.close_file =0 ; + opt->diffopt.no_free =1 ;if (opt->line_level_traverse)return line_log_print (opt, commit);@@ -974,7 +976,7 @@ int log_tree_commit(struct rev_info *opt, struct commit *commit)fprintf (opt->diffopt.file,"\n%s\n" , opt->break_bar); opt->loginfo =NULL ;maybe_flush_or_die (opt->diffopt.file,"stdout" ); -if (close_file) -fclose (opt->diffopt.file); + opt->diffopt.no_free = no_free; +diff_free (&opt->diffopt);return shown; }wt-status.c +1
-2@@ -1052,7 +1052,7 @@ static void wt_longstatus_print_verbose(struct wt_status *s) rev.diffopt.rename_limit = s->rename_limit >=0 ? s->rename_limit : rev.diffopt.rename_limit; rev.diffopt.rename_score = s->rename_score >=0 ? s->rename_score : rev.diffopt.rename_score; rev.diffopt.file = s->fp; - rev.diffopt.close_file =0 ; + rev.diffopt.fclose_file =0 ;/* wt_status owns the s->fp */ /* * If we're not going to stdout, then we definitely don't * want color, since we are going to the commit message -- 2.30.0.284.gd98b1dd5eaa7
Re: [PATCH 1/2] diff: add an API for deferred freeing
Quoted 10 lines · Johannes Schindelin
Hi Ævar,
Quoted 3 lines · Ævar Arnfjörð Bjarmason
Add a
diff_free()function to free anything we may have allocated in the "diff_options" struct, and the ability to make calling it a noop by setting "no_free" in "diff_options".Why do we need a
no_freeflag? Why not simply set thefree()d (orfclose()d) attributes toNULL?
Because we're calling the diff API N times, so we need to not have those be NULL while we're using them in a loop, and we need to not free() them, and then flip "no_free = 0" at the end and free() them.
Quoted 15 lines
builtin/add.c static int edit_patch(int argc, const char **argv, const char *prefix) · line 285
Quoted 10 lines
builtin/add.c +1
-1@@ -282,7 +282,7 @@ static int edit_patch(int argc, const char **argv, const char *prefix)if (out <0 )die (_ ("Could not open '%s' for writing." ), file); rev.diffopt.file =xfdopen (out,"w" ); - rev.diffopt.close_file =1 ; + rev.diffopt.fclose_file =1 ;This rename makes the patch unnecessarily tedious to review, and I do not even agree with leaking the implementation detail that we need to
fclose()the file.
Quoted 1 line
Let's just not?
Fair enough, I figured it would be easier for reviewers to reason about to be guaranteed to see all the uses of the flag, but I'll drop it.
Quoted 9 lines
builtin/add.c void diff_flush(struct diff_options *options) · line 6403
Quoted 7 lines
builtin/add.c +1
-1@@ -6399,10 +6399,10 @@ void diff_flush(struct diff_options *options)* options->file to /dev/null should be safe, because we * aren't supposed to produce any output anyway. */ -if (options->close_file) +if (options->fclose_file)fclose (options->file);And at this stage, we should set
options->file = NULL.
Sure, why not, but unless we get rid of the need for "close_file" there's not much point other than ease of inspection in a debugger...[cont].
Quoted 9 lines
builtin/add.c
Quoted 7 lines
[...]
builtin/add.c +6
-0+void diff_free (struct diff_options *options) +{ +if (options->no_free) +return ; +if (options->fclose_file) +fclose (options->file);And at this stage, we should set
options->file = NULL.
...ditto...[cont]
Quoted 13 lines
Quoted 13 lines
builtin/add.c +3
-0+} + +static int match_filter (const struct diff_options *options,const struct diff_filepair *p) { return (((p->status == DIFF_STATUS_MODIFIED) &&diff.h +0
-0@@ -49,7 +49,16 @@* - Once you finish feeding the pairs of files, call `diffcore_std()`. * This will tell the diffcore library to go ahead and do its work.
Quoted 8 lines
diff.h line 55
Quoted 5 lines
diff.h +4
-0* +* - The `diff_opt_parse()` etc. functions might allocate memory in +* `struct diff_options`. When running the API `N > 1` set `.no_free +* = 1` to make the `diff_free()` invoked by `diff_flush()` below a +* noop. I have serious trouble parsing that last sentence. Would you mind rephrasing it?
Willdo.
Quoted 7 lines
diff.h line 61
Quoted 6 lines
diff.h +5
-0+* * - Calling `diff_flush()` will produce the output. +* +* - If you set `.no_free = 1` before set it to `0` and call +* `diff_free()` again. If `.no_free = 1` was not set there's no +* need to call `diff_free()`, `diff_flush()` will call it.
Quoted 6 lines
Again, as I mentioned before, the indicator whether things need to be released should be whether the attribute is
NULLor not. And once released, ot should be set toNULL.Other than that, it looks fine to me. And I definitely like the idea of introducing a function to release all of the stuff in
struct diffopt.
[cont]...I don't think it's possible in anything like the current API to do that.
We don't want to "if (file) fclose(file)", instead we need an opt-in "if (close_file) fclose(file)". That's because usually the "file" is stdout. So close_file=1 is only set when we're opening a "real" file, /dev/null or whatever.
That along with the "no_free" flag as noted above means we need both a "no_free" and "close_file" flags.
Unless there's some way of re-arranging this that I'm missing.
Re: [PATCH 1/2] diff: add an API for deferred freeing
Hi Ævar,
Quoted 10 lines · Ævar Arnfjörð Bjarmason
Quoted 8 lines · Johannes Schindelin
Quoted 3 lines · Ævar Arnfjörð Bjarmason
Add a
diff_free()function to free anything we may have allocated in the "diff_options" struct, and the ability to make calling it a noop by setting "no_free" in "diff_options".Why do we need a
no_freeflag? Why not simply set thefree()d (orfclose()d) attributes toNULL?
Hmm. That was not even clear to me until I read this reply.
Doesn't this indicate that the closing is done at the wrong layer? If we want to call a function N times and only at the last iteration should it clean up our resources, doesn't that indicate that the clean-up should be pulled out from that function?
I thought that's exactly what you did, but I must have glanced over something obvious...
Ciao,
Dscho
Re: [PATCH 1/2] diff: add an API for deferred freeing
Quoted 15 lines · Johannes Schindelin
Quoted 10 lines · Ævar Arnfjörð Bjarmason
Quoted 8 lines · Johannes Schindelin
Quoted 3 lines · Ævar Arnfjörð Bjarmason
Add a
diff_free()function to free anything we may have allocated in the "diff_options" struct, and the ability to make calling it a noop by setting "no_free" in "diff_options".Why do we need a
no_freeflag? Why not simply set thefree()d (orfclose()d) attributes toNULL?Hmm. That was not even clear to me until I read this reply.
Quoted 7 lines
Doesn't this indicate that the closing is done at the wrong layer? If we want to call a function N times and only at the last iteration should it clean up our resources, doesn't that indicate that the clean-up should be pulled out from that function?
I thought that's exactly what you did, but I must have glanced over something obvious...
I think the issue then is that every caller must now be modified to do the clean up (which otherwise happens automatically when flushing the diff).
So I definitely agree that the current API is weird and backwards. But flipping it now may introduce new leaks in existing callers.
-Peff
PATCH v2 1/2 diff: add an API for deferred freeing
Add a diff_free() function to free anything we may have allocated in the "diff_options" struct, and the ability to make calling it a noop by setting "no_free" in "diff_options".
This is required because when e.g. "git diff" is run we'll allocate things in that struct, use the diff machinery once, and then exit.
But if we run e.g. "git log -p" we're going to re-use what we allocated across multiple diff_flush() calls, and only want to free things at the end.
We've thus ended up with features like the recently added "diff -I"[1] where we'll leak memory. As it turns out it could have simply used the pattern established in 6ea57703f6 (log: prepare log/log-tree to reuse the diffopt.close_file attribute, 2016-06-22).
Manually adding more such flags to things log_tree_commit() every time we need to allocate something would be tedious. Let's instead move that fclose() code it to a new diff_free(), in anticipation of freeing more things in that function in follow-up commits.
Some functions such as log_tree_commit() need an idiom of optionally retaining a previous "no_free", as they may either free the memory themselves, or their caller may do so. I'm keeping that idiom in log_show_early() for good measure, even though I don't think it's currently called in this manner. It also gets passed an existing "struct rev_info", so future callers may want to set the "no_free" flag.
This change is a bit hard to read because while the freeing pattern we're introducing isn't unusual, the "file" member is a special snowflake. We usually don't want to fclose() it. This is because "file" is usually stdout, in which case we don't want to fclose() it. We only want to opt-in to closing it when we e.g. open a file on the filesystem. Thus the opt-in "close_file" flag.
So the API in general just needs a "no_free" flag to defer freeing, but the "file" member still needs its "close_file" flag. This is made more confusing because while refactoring this code we could replace some "close_file=0" with "no_free=1", whereas others need to set both flags.
This is because there were some cases where an existing "close_file=0" meant "let's defer deallocation", and others where it meant "we don't want to close this file handle at all".
1. 296d4a94e7 (diff: add -I<regex> that ignores matching changes, 2020-10-20)
--- builtin/log.c | 23 ++++++++++++----------- diff.c | 20 ++++++++++++++++---- diff.h | 15 ++++++++++++++- log-tree.c | 10 ++++++---- 4 files changed, 48 insertions(+), 20 deletions(-)
builtin/log.c +12 -11
@@ -307,10 +307,11 @@ static struct itimerval early_output_timer;static void log_show_early (struct rev_info *revs,struct commit_list *list) { -int i = revs->early_output, close_file = revs->diffopt.close_file; +int i = revs->early_output;int show_header =1 ; +int no_free = revs->diffopt.no_free; - revs->diffopt.close_file =0 ; + revs->diffopt.no_free =0 ;sort_in_topological_order (&list, revs->sort_order);while (list && i) {struct commit *commit = list->item;
@@ -327,8 +328,8 @@ static void log_show_early(struct rev_info *revs, struct commit_list *list)case commit_ignore:break ;case commit_error: -if (close_file) -fclose (revs->diffopt.file); + revs->diffopt.no_free = no_free; +diff_free (&revs->diffopt);return ; } list = list->next;
@@ -336,8 +337,8 @@ static void log_show_early(struct rev_info *revs, struct commit_list *list)/* Did we already get enough commits for the early output? */ if (!i) { -if (close_file) -fclose (revs->diffopt.file); + revs->diffopt.no_free =0 ; +diff_free (&revs->diffopt);return ; }
@@ -401,7 +402,7 @@ static int cmd_log_walk(struct rev_info *rev) {struct commit *commit;int saved_nrl =0 ; -int saved_dcctc =0 , close_file = rev->diffopt.close_file; +int saved_dcctc =0 ;if (rev->early_output)setup_early_output ();
@@ -417,7 +418,7 @@ static int cmd_log_walk(struct rev_info *rev)* and HAS_CHANGES being accumulated in rev->diffopt, so be careful to * retain that state information if replacing rev->diffopt in this loop */ - rev->diffopt.close_file =0 ; + rev->diffopt.no_free =1 ;while ((commit =get_revision (rev)) !=NULL ) {if (!log_tree_commit (rev, commit) && rev->max_count >=0 )/*
@@ -442,8 +443,8 @@ static int cmd_log_walk(struct rev_info *rev) } rev->diffopt.degraded_cc_to_c = saved_dcctc; rev->diffopt.needed_rename_limit = saved_nrl; -if (close_file) -fclose (rev->diffopt.file); + rev->diffopt.no_free =0 ; +diff_free (&rev->diffopt);if (rev->diffopt.output_format &DIFF_FORMAT_CHECKDIFF && rev->diffopt.flags.check_failed) {
@@ -1955,7 +1956,7 @@ int cmd_format_patch(int argc, const char **argv, const char *prefix)* file, but but we must instruct it not to close after each * diff. */ - rev.diffopt.close_file =0 ; + rev.diffopt.no_free =1 ; }else {int saved;
diff.c +16 -4
@@ -6336,6 +6336,20 @@ static void diff_flush_patch_all_file_pairs(struct diff_options *o) } } +static void diff_free_file (struct diff_options *options) +{ +if (options->close_file) +fclose (options->file); +} + +void diff_free (struct diff_options *options) +{ +if (options->no_free) +return ; + +diff_free_file (options); +} +void diff_flush (struct diff_options *options) {struct diff_queue_struct *q = &diff_queued_diff;
@@ -6399,8 +6413,7 @@ void diff_flush(struct diff_options *options)* options->file to /dev/null should be safe, because we * aren't supposed to produce any output anyway. */ -if (options->close_file) -fclose (options->file); +diff_free_file (options); options->file =xfopen ("/dev/null" ,"w" ); options->close_file =1 ; options->color_moved =0 ;
@@ -6433,8 +6446,7 @@ void diff_flush(struct diff_options *options) free_queue:free (q->queue);DIFF_QUEUE_CLEAR (q); -if (options->close_file) -fclose (options->file); +diff_free (options);/* * Report the content-level differences with HAS_CHANGES;
diff.h +14 -1
@@ -49,7 +49,17 @@* - Once you finish feeding the pairs of files, call `diffcore_std()`. * This will tell the diffcore library to go ahead and do its work. * -* - Calling `diff_flush()` will produce the output. +* - Calling `diff_flush()` will produce the output, it will call +* `diff_free()` to free any resources, e.g. those allocated in +* `diff_opt_parse()`. +* +* - Set `.no_free = 1` before calling `diff_flush()` to defer the +* freeing of allocated memory in diff_options. This is useful when +* `diff_flush()` is being called in a loop, rather than as a +* one-off. When setting `.no_free = 1` you must ensure that +* `diff_free()` is called at the end, either by flipping the flag +* before the last `diff_flush()` call, or by flipping it before +* calling `diff_free()` yourself. */ struct combine_diff_path ;
@@ -365,6 +375,8 @@ struct diff_options {struct repository *repo;struct option *parseopts; + +int no_free; };unsigned diff_filter_bit (char status);
@@ -559,6 +571,7 @@ void diffcore_fix_diff_index(void);int diff_queue_is_empty (void );void diff_flush (struct diff_options *); +void diff_free (struct diff_options *);void diff_warn_rename_limit (const char *varname,int needed,int degraded_cc);/* diff-raw status letters */
log-tree.c +6 -4
@@ -958,12 +958,14 @@ static int log_tree_diff(struct rev_info *opt, struct commit *commit, struct logint log_tree_commit (struct rev_info *opt,struct commit *commit) {struct log_info log; -int shown, close_file = opt->diffopt.close_file; +int shown; +/* maybe called by e.g. cmd_log_walk(), maybe stand-alone */ +int no_free = opt->diffopt.no_free; log.commit = commit; log.parent =NULL ; opt->loginfo = &log; - opt->diffopt.close_file =0 ; + opt->diffopt.no_free =1 ;if (opt->line_level_traverse)return line_log_print (opt, commit);
@@ -980,7 +982,7 @@ int log_tree_commit(struct rev_info *opt, struct commit *commit)fprintf (opt->diffopt.file,"\n%s\n" , opt->break_bar); opt->loginfo =NULL ;maybe_flush_or_die (opt->diffopt.file,"stdout" ); -if (close_file) -fclose (opt->diffopt.file); + opt->diffopt.no_free = no_free; +diff_free (&opt->diffopt);return shown; }
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7
v2 cover diff: add an API for deferred freeing
In reply to Jeff KingI skipped the rename of the "close_file" flag, and updated the code & commit messages in response to Johannes's feedback on v1. Hopefully this is all better now.
Ævar Arnfjörð Bjarmason (2):
diff: add an API for deferred freeing
diff: plug memory leak from regcomp() on {log,diff} -I
builtin/log.c | 23 ++++++++++++----------- diff.c | 32 ++++++++++++++++++++++++++++---- diff.h | 15 ++++++++++++++- log-tree.c | 10 ++++++---- 4 files changed, 60 insertions(+), 20 deletions(-)
Range-diff:
1: 531fed77f4c ! 1: 045d3f72d15 diff: add an API for deferred freeing
@@ Commit message
by setting "no_free" in "diff_options".
This is required because when e.g. "git diff" is run we'll allocate
- things in that struct, use the diff machinery once, and then exit, but
- if we run e.g. "git log -p" we're going to re-use what we allocated
- across multiple diff_flush() calls, and only want to free things at
- the end.
+ things in that struct, use the diff machinery once, and then exit.
+
+ But if we run e.g. "git log -p" we're going to re-use what we
+ allocated across multiple diff_flush() calls, and only want to free
+ things at the end.
We've thus ended up with features like the recently added "diff -I"[1] where we'll leak memory. As it turns out it could have simply used the
@@ Commit message
the diffopt.close_file attribute, 2016-06-22).
Manually adding more such flags to things log_tree_commit() every time
- we need to allocate something would be tedious.
-
- Let's instead move that fclose() code it to a new diff_free(), in
- anticipation of freeing more things in that function in follow-up
- commits. I'm renaming the "close_file" struct member to "fclose_file"
- for the ease of validating this, we can be certain that these are all
- the relevant callsites.
+ we need to allocate something would be tedious. Let's instead move
+ that fclose() code it to a new diff_free(), in anticipation of freeing
+ more things in that function in follow-up commits.
Some functions such as log_tree_commit() need an idiom of optionally retaining a previous "no_free", as they may either free the memory themselves, or their caller may do so. I'm keeping that idiom in
- log_show_early() even though I don't think it's currently called in
- this manner, since it also gets passed an existing "struct rev_info"..
+ log_show_early() for good measure, even though I don't think it's
+ currently called in this manner. It also gets passed an existing
+ "struct rev_info", so future callers may want to set the "no_free"
+ flag.
+
+ This change is a bit hard to read because while the freeing pattern
+ we're introducing isn't unusual, the "file" member is a special
+ snowflake. We usually don't want to fclose() it. This is because
+ "file" is usually stdout, in which case we don't want to fclose()
+ it. We only want to opt-in to closing it when we e.g. open a file on
+ the filesystem. Thus the opt-in "close_file" flag.
+
+ So the API in general just needs a "no_free" flag to defer freeing,
+ but the "file" member still needs its "close_file" flag. This is made
+ more confusing because while refactoring this code we could replace
+ some "close_file=0" with "no_free=1", whereas others need to set both
+ flags.
+
+ This is because there were some cases where an existing "close_file=0"
+ meant "let's defer deallocation", and others where it meant "we don't
+ want to close this file handle at all".
1. 296d4a94e7 (diff: add -I<regex> that ignores matching changes, 2020-10-20)
- ## builtin/add.c ##
-@@ builtin/add.c: static int edit_patch(int argc, const char **argv, const char *prefix)
- if (out < 0)
- die(_("Could not open '%s' for writing."), file);
- rev.diffopt.file = xfdopen(out, "w");
-- rev.diffopt.close_file = 1;
-+ rev.diffopt.fclose_file = 1;
- if (run_diff_files(&rev, 0))
- die(_("Could not write patch"));
-
-
- ## builtin/am.c ##
-@@ builtin/am.c: static void write_commit_patch(const struct am_state *state, struct commit *comm
- rev_info.diffopt.flags.full_index = 1;
- rev_info.diffopt.use_color = 0;
- rev_info.diffopt.file = fp;
-- rev_info.diffopt.close_file = 1;
-+ rev_info.diffopt.fclose_file = 1; /* log_tree_commit() sets .no_free=1 */
- add_pending_object(&rev_info, &commit->object, "");
- diff_setup_done(&rev_info.diffopt);
- log_tree_commit(&rev_info, commit);
-@@ builtin/am.c: static void write_index_patch(const struct am_state *state)
- rev_info.diffopt.output_format = DIFF_FORMAT_PATCH;
- rev_info.diffopt.use_color = 0;
- rev_info.diffopt.file = fp;
-- rev_info.diffopt.close_file = 1;
-+ rev_info.diffopt.fclose_file = 1;
-+ rev_info.diffopt.no_free = 1;
- add_pending_object(&rev_info, &tree->object, "");
- diff_setup_done(&rev_info.diffopt);
- run_diff_index(&rev_info, 1);
-+ diff_free(&rev_info.diffopt);
- }
-
- /**
-
## builtin/log.c ##
@@ builtin/log.c: static struct itimerval early_output_timer;
@@ builtin/log.c: static int cmd_log_walk(struct rev_info *rev)
if (rev->diffopt.output_format & DIFF_FORMAT_CHECKDIFF &&
rev->diffopt.flags.check_failed) {
@@ builtin/log.c: int cmd_format_patch(int argc, const char **argv, const char *prefix)
- if (rev.show_notes)
- load_display_notes(&rev.notes_opt);
-
-- if (use_stdout + rev.diffopt.close_file + !!output_directory > 1)
-+ if (use_stdout + rev.diffopt.fclose_file + !!output_directory > 1)
- die(_("--stdout, --output, and --output-directory are mutually exclusive"));
-
- if (use_stdout) {
- setup_pager();
-- } else if (rev.diffopt.close_file) {
-+ } else if (rev.diffopt.fclose_file) {
- /*
- * The diff code parsed --output; it has already opened the
* file, but but we must instruct it not to close after each
* diff.
*/
@@ builtin/log.c: int cmd_format_patch(int argc, const char **argv, const char *pre
## diff.c ##
-@@ diff.c: static enum parse_opt_result diff_opt_output(struct parse_opt_ctx_t *ctx,
- BUG_ON_OPT_NEG(unset);
- path = prefix_filename(ctx->prefix, arg);
- options->file = xfopen(path, "w");
-- options->close_file = 1;
-+ options->fclose_file = 1;
- if (options->use_color != GIT_COLOR_ALWAYS)
- options->use_color = GIT_COLOR_NEVER;
- free(path);
+@@ diff.c: static void diff_flush_patch_all_file_pairs(struct diff_options *o)
+ }
+ }
+
++static void diff_free_file(struct diff_options *options)
++{
++ if (options->close_file)
++ fclose(options->file);
++}
++
++void diff_free(struct diff_options *options)
++{
++ if (options->no_free)
++ return;
++
++ diff_free_file(options);
++}
++
+ void diff_flush(struct diff_options *options)
+ {
+ struct diff_queue_struct *q = &diff_queued_diff;
@@ diff.c: void diff_flush(struct diff_options *options)
* options->file to /dev/null should be safe, because we
* aren't supposed to produce any output anyway.
*/
- if (options->close_file)
-+ if (options->fclose_file)
- fclose(options->file);
+- fclose(options->file);
++ diff_free_file(options);
options->file = xfopen("/dev/null", "w");
-- options->close_file = 1;
-+ options->fclose_file = 1;
+ options->close_file = 1;
options->color_moved = 0;
- for (i = 0; i < q->nr; i++) {
- struct diff_filepair *p = q->queue[i];
@@ diff.c: void diff_flush(struct diff_options *options)
free_queue:
free(q->queue);
@@ diff.c: void diff_flush(struct diff_options *options)
/*
* Report the content-level differences with HAS_CHANGES;
-@@ diff.c: void diff_flush(struct diff_options *options)
- }
- }
-
-+void diff_free(struct diff_options *options)
-+{
-+ if (options->no_free)
-+ return;
-+ if (options->fclose_file)
-+ fclose(options->file);
-+}
-+
-+
- static int match_filter(const struct diff_options *options, const struct diff_filepair *p)
- {
- return (((p->status == DIFF_STATUS_MODIFIED) &&
## diff.h ##
@@
* - Once you finish feeding the pairs of files, call diffcore_std().
* This will tell the diffcore library to go ahead and do its work. *
-+ * - The diff_opt_parse() etc. functions might allocate memory in -+ * struct diff_options. When running the API N > 1 set .no_free -+ * = 1 to make the diff_free() invoked by diff_flush() below a -+ * noop.
-+ *
- * - Calling diff_flush() will produce the output.
+- * - Calling diff_flush() will produce the output.
++ * - Calling diff_flush() will produce the output, it will call
++ * diff_free() to free any resources, e.g. those allocated in
++ * diff_opt_parse().
+ *
-+ * - If you set .no_free = 1 before set it to 0 and call
-+ * diff_free() again. If .no_free = 1 was not set there's no
-+ * need to call diff_free(), diff_flush() will call it.
++ * - Set .no_free = 1 before calling diff_flush() to defer the
++ * freeing of allocated memory in diff_options. This is useful when
++ * diff_flush() is being called in a loop, rather than as a
++ * one-off. When setting .no_free = 1 you must ensure that
++ * diff_free() is called at the end, either by flipping the flag
++ * before the last diff_flush() call, or by flipping it before
++ * calling diff_free() yourself.
*/
struct combine_diff_path;
@@ diff.h: struct diff_options {
- void (*set_default)(struct diff_options *);
-
- FILE *file;
-- int close_file;
-+ int fclose_file;
-
- #define OUTPUT_INDICATOR_NEW 0
- #define OUTPUT_INDICATOR_OLD 1
-@@ diff.h: struct diff_options {
struct repository *repo;
struct option *parseopts;
@@ log-tree.c: int log_tree_commit(struct rev_info *opt, struct commit *commit)
+ diff_free(&opt->diffopt);
return shown;
}
-
- ## wt-status.c ##
-@@ wt-status.c: static void wt_longstatus_print_verbose(struct wt_status *s)
- rev.diffopt.rename_limit = s->rename_limit >= 0 ? s->rename_limit : rev.diffopt.rename_limit;
- rev.diffopt.rename_score = s->rename_score >= 0 ? s->rename_score : rev.diffopt.rename_score;
- rev.diffopt.file = s->fp;
-- rev.diffopt.close_file = 0;
-+ rev.diffopt.fclose_file = 0; /* wt_status owns the s->fp */
- /*
- * If we're not going to stdout, then we definitely don't
- * want color, since we are going to the commit message
2: 7192cf01e71 ! 2: f571524e6d8 diff: plug memory leak from regcomp() on {log,diff} -I
@@ Commit message
At that time freeing the memory was somewhat tedious, but since it isn't anymore with the newly introduced diff_free() let's use it.
+ Let's retain the pattern for diff_free_file() and add a
+ diff_free_ignore_regex(), even though (unlike "diff_free_file") we
+ don't need to call it elsewhere. I think this'll make for more
+ readable code than gradually accumulating a giant diff_free()
+ function, sharing "int i" across unrelated code etc.
+
## diff.c ##
-@@ diff.c: void diff_flush(struct diff_options *options) +@@ diff.c: static void diff_free_file(struct diff_options *options)
+ fclose(options->file);
+ }
- void diff_free(struct diff_options *options)
- {
++static void diff_free_ignore_regex(struct diff_options *options) ++{
+ int i;
- if (options->no_free)
- return;
- if (options->fclose_file)
- fclose(options->file);
+
+ for (i = 0; i < options->ignore_regex_nr; i++) {
+ regfree(options->ignore_regex[i]);
+ free(options->ignore_regex[i]);
+ }
+ free(options->ignore_regex);
++}
++
+ void diff_free(struct diff_options *options)
+ {
+ if (options->no_free)
+ return;
+
+ diff_free_file(options);
++ diff_free_ignore_regex(options);
}
-
+ void diff_flush(struct diff_options *options)
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7
PATCH v2 2/2 diff: plug memory leak from regcomp() on {log,diff} -I
In reply to Jeff KingFix a memory leak in 296d4a94e7 (diff: add -I<regex> that ignores matching changes, 2020-10-20) by freeing the memory it allocates in the newly introduced diff_free(). See the previous commit for details on that.
This memory leak was intentionally introduced in 296d4a94e7, see the discussion on a previous iteration of it in
https://lore.kernel.org/git/[email protected]/
At that time freeing the memory was somewhat tedious, but since it isn't anymore with the newly introduced diff_free() let's use it.
Let's retain the pattern for diff_free_file() and add a diff_free_ignore_regex(), even though (unlike "diff_free_file") we don't need to call it elsewhere. I think this'll make for more readable code than gradually accumulating a giant diff_free() function, sharing "int i" across unrelated code etc.
--- diff.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
diff.c +12 -0
@@ -6342,12 +6342,24 @@ static void diff_free_file(struct diff_options *options)fclose (options->file); } +static void diff_free_ignore_regex (struct diff_options *options) +{ +int i; + +for (i =0 ; i < options->ignore_regex_nr; i++) { +regfree (options->ignore_regex[i]); +free (options->ignore_regex[i]); + } +free (options->ignore_regex); +} +void diff_free (struct diff_options *options) {if (options->no_free)return ;diff_free_file (options); +diff_free_ignore_regex (options); }void diff_flush (struct diff_options *options)
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7
PATCH v2 2/2 diff: plug memory leak from regcomp() on {log,diff} -I
In reply to Michał KępieńFix a memory leak in 296d4a94e7 (diff: add -I<regex> that ignores matching changes, 2020-10-20) by freeing the memory it allocates in the newly introduced diff_free(). See the previous commit for details on that.
This memory leak was intentionally introduced in 296d4a94e7, see the discussion on a previous iteration of it in
https://lore.kernel.org/git/[email protected]/
At that time freeing the memory was somewhat tedious, but since it isn't anymore with the newly introduced diff_free() let's use it.
--- diff.c | 7 +++++++ 1 file changed, 7 insertions(+)
diff.c +7 -0
@@ -6450,10 +6450,17 @@ void diff_flush(struct diff_options *options)void diff_free (struct diff_options *options) { +int i;if (options->no_free)return ;if (options->fclose_file)fclose (options->file); + +for (i =0 ; i < options->ignore_regex_nr; i++) { +regfree (options->ignore_regex[i]); +free (options->ignore_regex[i]); + } +free (options->ignore_regex); }
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7