diff: add diff.<driver>.process for external hunk providers
Subsystems (3)
cover diff: add diff.<driver>.process for external hunk providers
This series adds diff.<driver>.process, a long-running subprocess protocol that lets external tools provide hunks to git's diff and blame pipelines.
Over the past 18 years, git's diff pipeline accumulated many features that operate on hunks: word diff, function context, color-moved, indent heuristic, blame. External tools can replace the pipeline entirely (diff.<driver>.command) or select among builtin algorithms (diff.<driver>.algorithm), but there is no way for a tool to provide line-change information into the pipeline. Tools that understand code structure (tree-sitter parsers, format-aware analyzers, tools like Difftastic and Mergiraf) must bypass git's pipeline and lose access to everything downstream.
The protocol follows filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation, one tool invocation per git command. The tool receives file pairs and returns hunk descriptors that git feeds into the standard xdiff pipeline. All output features work normally.
Zero hunks with status=success means the tool considers the files equivalent. git diff shows no output for the file, and git blame skips the commit, attributing lines to earlier commits.
On error or tool crash, git falls back silently to the builtin diff algorithm. The feature is opt-in via diff.<driver>.process and .gitattributes; unconfigured files are unaffected.
The series includes git diff-process-normalize, a built-in tool that compares files line by line ignoring whitespace (same logic as "git diff -w" via xdiff_compare_lines):
[diff "cdiff"]
process = git diff-process-normalize
A whitespace-only boolean flag could serve this specific case. The subprocess protocol is more general, allowing any tool to participate without further changes to git.
Michael Montalbo (5):
xdiff: support external hunks via xpparam_t
userdiff: add diff.<driver>.process config
diff: add long-running diff process via diff.<driver>.process
blame: consult diff process for zero-hunk detection
diff-process-normalize: add built-in whitespace normalizer
Documentation/config/diff.adoc | 10 + Documentation/gitattributes.adoc | 58 +++++ Makefile | 2 + blame.c | 43 +++- builtin.h | 1 + builtin/diff-process-normalize.c | 143 ++++++++++ diff-process.c | 203 +++++++++++++++ diff-process.h | 28 ++ diff.c | 25 ++ git.c | 1 + t/t4080-diff-process.sh | 430 +++++++++++++++++++++++++++++++ userdiff.c | 7 + userdiff.h | 2 + xdiff-interface.c | 7 +- xdiff/xdiff.h | 13 + xdiff/xdiffi.c | 98 ++++++- 16 files changed, 1063 insertions(+), 8 deletions(-) create mode 100644 builtin/diff-process-normalize.c create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100755 t/t4080-diff-process.sh
base-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2120
Signature · 1 line
gitgitgadget
PATCH 1/5 xdiff: support external hunks via xpparam_t
From: Michael Montalbo <[email protected]>
Add two new xpparam_t fields (external_hunks, external_hunks_nr) that let callers supply pre-computed hunks. When set, xdl_diff() populates the changed[] arrays from these hunks instead of running the diff algorithm, then continues through compaction and emission as usual.
Validate supplied hunks before use: reject out-of-bounds line numbers, overlapping or out-of-order hunks, negative counts, and violations of the synchronization invariant (unchanged line counts must match between files). On validation failure, fall back to the builtin diff algorithm.
Skip trim_common_tail() in xdi_diff() when external hunks are present, since external hunks reference line numbers in the original content.
--- xdiff-interface.c | 7 +++- xdiff/xdiff.h | 13 +++++++ xdiff/xdiffi.c | 98 +++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 114 insertions(+), 4 deletions(-)
xdiff-interface.c +6 -1
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t coif (mf1->size >MAX_XDIFF_SIZE || mf2->size >MAX_XDIFF_SIZE )return -1 ; -if (!xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT )) +/* +* External hunks reference line numbers in the original content; +* trimming the tail would change line counts and invalidate them. +*/ +if (!xpp->external_hunks && + !xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT ))trim_common_tail (&a, &b);return xdl_diff (&a, &b, xpp, xecfg, xecb);
xdiff/xdiff.h +13 -0
@@ -78,6 +78,15 @@ typedef struct s_mmbuffer {long size; } mmbuffer_t; +/* +* Hunk descriptor for externally computed diffs. +* Line numbers are 1-based, matching unified diff convention. +*/ +struct xdl_hunk { +long old_start, old_count; +long new_start, new_count; +}; + typedef struct s_xpparam { unsigned long flags;
@@ -88,6 +97,10 @@ typedef struct s_xpparam {/* See Documentation/diff-options.adoc. */ char **anchors;size_t anchors_nr; + +/* Externally computed hunks: bypass the diff algorithm. */ +const struct xdl_hunk *external_hunks; +size_t external_hunks_nr; } xpparam_t; typedef struct s_xdemitcb {
xdiff/xdiffi.c +95 -3
111 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t
Quoted 14 lines · Michael Montalbo via GitGitGadget
xdiff/xdiffi.c +14
-0+/* +* Populate the changed[] arrays from externally supplied hunks, +* bypassing the diff algorithm. Validates that hunks are in order, +* non-overlapping, and within bounds. +* +* Returns 0 on success, -1 on validation failure. +*/ +static int xdl_populate_hunks_from_external (xdfenv_t *xe, +const struct xdl_hunk *hunks, +size_t nr_hunks) +{ +size_t i; +long j, prev_old_end =0 , prev_new_end =0 ; +long total_old =0 , total_new =0 ;
xdiff/xdiffi.c static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe, · line 1111
Quoted 10 lines
xdiff/xdiffi.c +10
-0+ +/* +* Clear changed[] arrays. xdl_prepare_env() may have dirtied +* them via xdl_cleanup_records(). The allocation is nrec + 2 +* elements; changed points one past the start (see xprepare.c). +*/ +memset (xe->xdf1.changed -1 ,0 , + (xe->xdf1.nrec +2 ) *sizeof (bool )); +memset (xe->xdf2.changed -1 ,0 , + (xe->xdf2.nrec +2 ) *sizeof (bool ));
This, especially the starting offset of -1, looks horrible. The internal layout of xdfenv_t might happen to match the way the above code expects, which is how xdl_prepare_ctx() may have give you, but it somehow feels brittle. I guess the assumption that changed[] does not point at the beginning of the allocated area (e.g., it is a no-no to free(xe->xdf1.changed) or realloc() it) is so pervasive that it cannot be helped. Sigh.
Quoted 13 lines
int xdl_diff(mmfile_t *mf1,
mmfile_t*mf2,xpparam_tconst *xpp,xdiff/xdiffi.c +4
-3xdemitconf_t const *xecfg, xdemitcb_t *ecb) { xdchange_t *xscr;xdfenv_t xe;emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff; -if (xdl_do_diff (mf1, mf2, xpp, &xe) <0 ) { - -return -1 ; +if (xpp->external_hunks) { +if (xdl_prepare_env (mf1, mf2, xpp, &xe) <0 ) +return -1 ; + if (xdl_populate_hunks_from_external(&xe,
xdiff/xdiffi.c static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe, · line 1183
Quoted 10 lines
xdiff/xdiffi.c +10
-0+ xpp->external_hunks, + xpp->external_hunks_nr) < 0) { +/* +* Invalid external hunks; fall back to the +* builtin diff algorithm. Re-runs +* xdl_prepare_env() via xdl_do_diff(). +*/ +xdl_free_env (&xe); +if (xdl_do_diff (mf1, mf2, xpp, &xe) <0 ) +return -1 ;
If the external tool keeps sending bogus hunks, silently falling back to what we would have done if there weren't any external stuff may be necessary to pleasantly keep using Git, but two and a half short comments here.
(1) "What we would have done" is exactly the same as what appears in the corresponding "else" block. Can we make sure that we do not have to keep updating both copies in the future with some code rearrangement?
(2) The writer of the external tool may want to see some trace of warning under certain flags when a failure of the tool forces the receiving end to fallback.
(3) If the tool throws too many broken replies, perhaps we want to disable it automatically?
Quoted 9 lines
xdiff/xdiffi.c +5
-0+ } + }else { +if (xdl_do_diff (mf1, mf2, xpp, &xe) <0 ) +return -1 ; } +if (xdl_change_compact (&xe.xdf1, &xe.xdf2, xpp->flags) <0 ||xdl_change_compact (&xe.xdf2, &xe.xdf1, xpp->flags) <0 ||xdl_build_script (&xe, &xscr) <0 ) {
Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t
Quoted 28 lines · Junio C Hamano
Quoted 14 lines · Michael Montalbo via GitGitGadget
xdiff/xdiffi.c +14
-0+/* +* Populate the changed[] arrays from externally supplied hunks, +* bypassing the diff algorithm. Validates that hunks are in order, +* non-overlapping, and within bounds. +* +* Returns 0 on success, -1 on validation failure. +*/ +static int xdl_populate_hunks_from_external (xdfenv_t *xe, +const struct xdl_hunk *hunks, +size_t nr_hunks) +{ +size_t i; +long j, prev_old_end =0 , prev_new_end =0 ; +long total_old =0 , total_new =0 ;xdiff/xdiffi.c static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe, · line 1111
Quoted 10 lines
xdiff/xdiffi.c +10
-0+ +/* +* Clear changed[] arrays. xdl_prepare_env() may have dirtied +* them via xdl_cleanup_records(). The allocation is nrec + 2 +* elements; changed points one past the start (see xprepare.c). +*/ +memset (xe->xdf1.changed -1 ,0 , + (xe->xdf1.nrec +2 ) *sizeof (bool )); +memset (xe->xdf2.changed -1 ,0 , + (xe->xdf2.nrec +2 ) *sizeof (bool ));
Quoted 8 lines
This, especially the starting offset of -1, looks horrible. The internal layout of
xdfenv_tmight happen to match the way the above code expects, which is howxdl_prepare_ctx()may have give you, but it somehow feels brittle. I guess the assumption that changed[] does not point at the beginning of the allocated area (e.g., it is a no-no tofree(xe->xdf1.changed)orrealloc()it) is so pervasive that it cannot be helped. Sigh.
Agreed it is ugly. I wanted to make sure the entire changed[] including sentinels were clear as a defensive measure for downstream callers (xdl_change_compact). I agree this results in something that is ugly and brittle, but in the end I thought it was superior to relying on the fact that upstream zeroes the entire changed[] array. Maybe if the comment was more explicit about why this is happening it would be helpful?
/*
* Clear changed[] arrays including sentinels.
* xdl_prepare_env() may have dirtied them via
* xdl_cleanup_records(), and xdl_change_compact() reads
* the sentinel at changed[-1] during backward scans. */
Quoted 29 lines
Quoted 13 lines
int xdl_diff(mmfile_t *mf1,
mmfile_t*mf2,xpparam_tconst *xpp,xdemitconf_t const *xecfg, xdemitcb_t *ecb) { xdchange_t *xscr; xdfenv_t xe; emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;xdiff/xdiffi.c +4
-3-if (xdl_do_diff (mf1, mf2, xpp, &xe) <0 ) { - -return -1 ; +if (xpp->external_hunks) { +if (xdl_prepare_env (mf1, mf2, xpp, &xe) <0 ) +return -1 ; + if (xdl_populate_hunks_from_external(&xe,xdiff/xdiffi.c static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe, · line 1183
Quoted 10 lines
xdiff/xdiffi.c +10
-0+ xpp->external_hunks, + xpp->external_hunks_nr) < 0) { +/* +* Invalid external hunks; fall back to the +* builtin diff algorithm. Re-runs +* xdl_prepare_env() via xdl_do_diff(). +*/ +xdl_free_env (&xe); +if (xdl_do_diff (mf1, mf2, xpp, &xe) <0 ) +return -1 ;If the external tool keeps sending bogus hunks, silently falling back to what we would have done if there weren't any external stuff may be necessary to pleasantly keep using Git, but two and a half short comments here.
Quoted 5 lines
(1) "What we would have done" is exactly the same as what appears in the corresponding "else" block. Can we make sure that we do not have to keep updating both copies in the future with some code rearrangement?
How about something like this:
if (xpp->external_hunks) {
if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
return -1;
if (xdl_populate_hunks_from_external(&xe,
xpp->external_hunks,
xpp->external_hunks_nr) == 0)
goto diff_done;
xdl_free_env(&xe);
} if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
return -1;
diff_done:
Quoted 4 lines
(2) The writer of the external tool may want to see some trace of warning under certain flags when a failure of the tool forces the receiving end to fallback.
In diff.c how about we emit a warning rather than a trace on fallback:
warning(_("diff process failed for '%s',"
" falling back to builtin diff"),
name_a);
Quoted 3 lines
(3) If the tool throws too many broken replies, perhaps we want to disable it automatically?
For the RFC I wanted to keep it simple, but I definitely agree. A configurable failure policy makes a lot of sense to me (e.g., disable after N failures).
Quoted 9 lines
Quoted 9 lines
xdiff/xdiffi.c +5
-0+ } + }else { +if (xdl_do_diff (mf1, mf2, xpp, &xe) <0 ) +return -1 ; } +if (xdl_change_compact (&xe.xdf1, &xe.xdf2, xpp->flags) <0 ||xdl_change_compact (&xe.xdf2, &xe.xdf1, xpp->flags) <0 ||xdl_build_script (&xe, &xscr) <0 ) {
Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t
Quoted 18 lines · Michael Montalbo
Quoted 9 lines
xdiff/xdiffi.c static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe, · line 1111
Quoted 8 lines
xdiff/xdiffi.c +8
-0+* Clear changed[] arrays. xdl_prepare_env() may have dirtied +* them via xdl_cleanup_records(). The allocation is nrec + 2 +* elements; changed points one past the start (see xprepare.c). +*/ +memset (xe->xdf1.changed -1 ,0 , + (xe->xdf1.nrec +2 ) *sizeof (bool )); +memset (xe->xdf2.changed -1 ,0 , + (xe->xdf2.nrec +2 ) *sizeof (bool ));Quoted 8 lines
This, especially the starting offset of -1, looks horrible. The internal layout of
xdfenv_tmight happen to match the way the above code expects, which is howxdl_prepare_ctx()may have give you, but it somehow feels brittle. I guess the assumption that changed[] does not point at the beginning of the allocated area (e.g., it is a no-no tofree(xe->xdf1.changed)orrealloc()it) is so pervasive that it cannot be helped. Sigh.
Quoted 7 lines
Agreed it is ugly. I wanted to make sure the entire changed[] including sentinels were clear as a defensive measure for downstream callers (
xdl_change_compact). I agree this results in something that is ugly and brittle, but in the end I thought it was superior to relying on the fact that upstream zeroes the entire changed[] array. Maybe if the comment was more explicit about why this is happening it would be helpful?
Perhaps make these memset() into calls to a helper function that is defined in xdiff/xprepare.c with a descriptive name and placed near where xdl_prepare_ctx() is. That way, the patch in question does not even have to expose the strangeness of changed[] (i.e., it has 2 more elements than it would normally contain to make the memory region for changed[-1] and changed[N] valid, and freeing it requires free(changed-1)) to the code path. It only needs to say "Hey, I am clearing changed[] arrays because of XXX" without having to say "by the way, the memory layout of changed[] is strange this way", the latter of which is not exactly of interest for readers of this code.
Quoted 6 lines
/*
* Clear changed[] arrays including sentinels.
*xdl_prepare_env()may have dirtied them via
*xdl_cleanup_records(), andxdl_change_compact()reads
* the sentinel at changed[-1] during backward scans. */
And this belongs in xdiff/xprepare.c near that new helper function.
Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t
Quoted 40 lines · Junio C Hamano
Quoted 18 lines · Michael Montalbo
Quoted 9 lines
xdiff/xdiffi.c static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe, · line 1111
Quoted 8 lines
xdiff/xdiffi.c +8
-0+* Clear changed[] arrays. xdl_prepare_env() may have dirtied +* them via xdl_cleanup_records(). The allocation is nrec + 2 +* elements; changed points one past the start (see xprepare.c). +*/ +memset (xe->xdf1.changed -1 ,0 , + (xe->xdf1.nrec +2 ) *sizeof (bool )); +memset (xe->xdf2.changed -1 ,0 , + (xe->xdf2.nrec +2 ) *sizeof (bool ));Quoted 8 lines
This, especially the starting offset of -1, looks horrible. The internal layout of
xdfenv_tmight happen to match the way the above code expects, which is howxdl_prepare_ctx()may have give you, but it somehow feels brittle. I guess the assumption that changed[] does not point at the beginning of the allocated area (e.g., it is a no-no tofree(xe->xdf1.changed)orrealloc()it) is so pervasive that it cannot be helped. Sigh.Quoted 7 lines
Agreed it is ugly. I wanted to make sure the entire changed[] including sentinels were clear as a defensive measure for downstream callers (
xdl_change_compact). I agree this results in something that is ugly and brittle, but in the end I thought it was superior to relying on the fact that upstream zeroes the entire changed[] array. Maybe if the comment was more explicit about why this is happening it would be helpful?Perhaps make these
memset()into calls to a helper function that is defined inxdiff/xprepare.cwith a descriptive name and placed near wherexdl_prepare_ctx()is. That way, the patch in question does not even have to expose the strangeness of changed[] (i.e., it has 2 more elements than it would normally contain to make the memory region for changed[-1] and changed[N] valid, and freeing it requires free(changed-1)) to the code path. It only needs to say "Hey, I am clearing changed[] arrays because of XXX" without having to say "by the way, the memory layout of changed[] is strange this way", the latter of which is not exactly of interest for readers of this code.
Quoted 8 lines
Quoted 6 lines
/*
* Clear changed[] arrays including sentinels.
*xdl_prepare_env()may have dirtied them via
*xdl_cleanup_records(), andxdl_change_compact()reads
* the sentinel at changed[-1] during backward scans. */And this belongs in
xdiff/xprepare.cnear that new helper function.
That sounds a lot nicer. Will update.
PATCH 2/5 userdiff: add diff.<driver>.process config
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add a new per-driver configuration key that specifies the command for a long-running diff process.
The name follows filter.<driver>.process: a long-running subprocess that stays alive across files within a single git invocation.
--- userdiff.c | 7 +++++++ userdiff.h | 2 ++ 2 files changed, 9 insertions(+)
userdiff.c +7 -0
@@ -499,6 +499,13 @@ int userdiff_config(const char *k, const char *v) drv->algorithm = drv->algorithm_owned;return ret; } +if (!strcmp (type,"process" )) { +int ret; +FREE_AND_NULL (drv->process_owned); + ret =git_config_string (&drv->process_owned, k, v); + drv->process = drv->process_owned; +return ret; + }return 0 ; }
userdiff.h +2 -0
@@ -31,6 +31,8 @@ struct userdiff_driver {char *textconv_owned;struct notes_cache *textconv_cache;int textconv_want_cache; +const char *process; +char *process_owned; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,
Signature · 1 line
gitgitgadget
PATCH 3/5 diff: add long-running diff process via diff.<driver>.process
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add support for external diff processes that communicate via the long-running process protocol (pkt-line over stdin/stdout).
A diff process is configured per userdiff driver:
[diff "cdiff"]
process = /path/to/diff-tool
The tool receives file pairs and returns hunks describing which lines changed. Git feeds these hunks into the standard xdiff pipeline, so all output features (word diff, function context, color) work normally.
The handshake negotiates version=1 and capability=hunks. Per-file requests send command=hunks, pathname, and both file contents as packetized data. The tool responds with hunk lines and a status packet. On error, git falls back to the builtin diff algorithm.
Zero hunks with status=success means the tool considers the files equivalent. Git skips diff output for that file entirely.
--- Documentation/config/diff.adoc | 8 + Documentation/gitattributes.adoc | 40 ++++ Makefile | 1 + diff-process.c | 203 +++++++++++++++++++ diff-process.h | 28 +++ diff.c | 25 +++ t/t4080-diff-process.sh | 338 +++++++++++++++++++++++++++++++ 7 files changed, 643 insertions(+) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100755 t/t4080-diff-process.sh
Documentation/config/diff.adoc +8 -0
@@ -218,6 +218,14 @@ endif::git-diff[] Set this option to `true` to make the diff driver cache the text conversion outputs. See linkgit:gitattributes[5] for details. +`diff.<driver>.process`:: + The command to run as a long-running diff process. + The tool communicates via the pkt-line protocol and returns + hunks that are fed into Git's diff and blame pipelines. + If the tool returns zero hunks, the file is treated as + unchanged for both diff output and blame attribution. + See linkgit:gitattributes[5] for details. + `diff.indentHeuristic`:: Set this option to `false` to disable the default heuristics that shift diff hunk boundaries to make patches easier to read.
Documentation/gitattributes.adoc +40 -0
@@ -821,6 +821,46 @@ NOTE: If `diff.<name>.command` is defined for path with the (see above), and adding `diff.<name>.algorithm` has no effect, as the algorithm is not passed to the external diff driver. +Using an external diff process +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An external tool can provide content-aware line matching by +setting `diff.<name>.process` to the command that runs +the tool. The tool is a long-running process that communicates via +the pkt-line protocol (see +linkgit:gitprotocol-long-running-process[5]). + +------------------------ +*.c diff=cdiff +------------------------ + +---------------------------------------------------------------- +[diff "cdiff"] + process = /path/to/diff-process-tool +---------------------------------------------------------------- + +The tool receives file pairs and returns hunk descriptors indicating +which lines changed. Git feeds these hunks into its standard diff +pipeline, so all output features (word diff, function context, +color) work normally. + +If the tool fails or returns an error, Git silently falls back to +the builtin diff algorithm. If the tool returns invalid hunks +(out of bounds, overlapping), Git also falls back silently. + +The handshake negotiates `version=1` and `capability=hunks`. +Per-file requests send `command=hunks` and `pathname=<path>`, +followed by the old and new file content as packetized data. +The tool responds with lines of the form +`hunk <old_start> <old_count> <new_start> <new_count>` +(1-based line numbers), a flush packet, and `status=success`. + +If the tool returns zero hunks with `status=success`, Git treats +the file as having no changes and produces no diff output. + +Tools should ignore unknown keys in the per-file request to +remain forward-compatible. + Defining a custom hunk-header ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Makefile +1 -0
@@ -1142,6 +1142,7 @@ LIB_OBJS += diff-delta.o LIB_OBJS += diff-merges.o LIB_OBJS += diff-lib.o LIB_OBJS += diff-no-index.o +LIB_OBJS += diff-process.o LIB_OBJS += diff.o LIB_OBJS += diffcore-break.o LIB_OBJS += diffcore-delta.o
diff-process.c new file +203 -0
203 lines not on this page · raw message
diff-process.h new file +28 -0
@@ -0,0 +1,28 @@ +#ifndef DIFF_PROCESS_H +#define DIFF_PROCESS_H + +struct userdiff_driver ; +struct xdl_hunk ; + +/* +* Query a diff process for hunks describing the changes +* between old_buf and new_buf. +* +* The backend is a long-running subprocess configured via +* diff.<driver>.process. It receives file content via +* pkt-line and returns hunks with 1-based line numbers. +* +* On success, sets *hunks_out and *nr_hunks_out to a newly allocated +* array (caller must free) and returns 0. +* +* On failure, returns -1. The caller should fall back to the +* builtin diff algorithm. +*/ +int diff_process_get_hunks (struct userdiff_driver *drv, +const char *path, +const char *old_buf,long old_size, +const char *new_buf,long new_size, +struct xdl_hunk **hunks_out, +size_t *nr_hunks_out); + +#endif /* DIFF_PROCESS_H */
diff.c +25 -0
@@ -25,7 +25,9 @@#include "utf8.h" #include "odb.h" #include "userdiff.h" +#include "diff-process.h" #include "submodule.h" +#include "trace2.h" #include "hashmap.h" #include "mem-pool.h" #include "merge-ll.h"
@@ -3991,6 +3993,7 @@ static void builtin_diff(const char *name_a,xpparam_t xpp;xdemitconf_t xecfg;struct emit_callback ecbdata; +struct xdl_hunk *ext_hunks =NULL ;unsigned ws_rule;const struct userdiff_funcname *pe;
@@ -4031,6 +4034,27 @@ static void builtin_diff(const char *name_a, xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; + +if (!o->ignore_driver_algorithm && + one->driver && one->driver->process) { +size_t ext_hunks_nr =0 ; +if (!diff_process_get_hunks ( + one->driver, name_a, + mf1.ptr, mf1.size, + mf2.ptr, mf2.size, + &ext_hunks, &ext_hunks_nr)) { +if (!ext_hunks_nr) + goto free_ab_and_return; + xpp.external_hunks = ext_hunks; + xpp.external_hunks_nr = ext_hunks_nr; + }else { +trace2_data_string ("diff" , + o->repo, +"diff-process-fallback" , + name_a); + } + } + xecfg.ctxlen = o->context; xecfg.interhunkctxlen = o->interhunkcontext; xecfg.flags =XDL_EMIT_FUNCNAMES ;
@@ -4111,6 +4135,7 @@ static void builtin_diff(const char *name_a, }else if (xdi_diff_outf (&mf1, &mf2,NULL , fn_out_consume, &ecbdata, &xpp, &xecfg))die ("unable to generate diff for %s" , one->path); +free (ext_hunks);if (o->word_diff)free_diff_words_data (&ecbdata);if (textconv_one)
t/t4080-diff-process.sh new file +338 -0
338 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH 4/5 blame: consult diff process for zero-hunk detection
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
When a diff process is configured via diff.<driver>.process, consult it during blame's per-commit diffing. If the process returns zero hunks for a commit's changes to a file, treat the commit as having no changes, causing blame to attribute lines to earlier commits.
The subprocess is long-running (one startup cost amortized across the blame traversal), but each commit in the file's history incurs a round-trip to the tool.
--- Documentation/gitattributes.adoc | 3 +++ blame.c | 43 +++++++++++++++++++++++++++++--- t/t4080-diff-process.sh | 32 ++++++++++++++++++++++++ 3 files changed, 74 insertions(+), 4 deletions(-)
Documentation/gitattributes.adoc +3 -0
@@ -857,6 +857,9 @@ The tool responds with lines of the form If the tool returns zero hunks with `status=success`, Git treats the file as having no changes and produces no diff output. +`git blame` also consults the diff process and skips commits +where it reports zero hunks, attributing lines to earlier commits +instead. Tools should ignore unknown keys in the per-file request to remain forward-compatible.
blame.c +39 -4
75 lines not on this page · raw message
t/t4080-diff-process.sh +32 -0
@@ -335,4 +335,36 @@ test_expect_success PYTHON 'diff process zero hunks suppresses diff output' ' test_must_be_empty actual ' +test_expect_success PYTHON 'blame skips commits with zero hunks from diff process' ' + cat >blame.c <<-\EOF && + int main(void) + { + return 0; + } + EOF + git add blame.c && + git commit -m "add blame.c" && + + cat >blame.c <<-\EOF && + int main(void) + { + return 0; + } + EOF + git add blame.c && + git commit -m "reformat blame.c" && + BLAME_COMMIT=$(git rev-parse --short HEAD) && + + # Without zero-hunk mode, blame attributes the change. + git blame blame.c >without && + grep "$BLAME_COMMIT" without && + + # With zero-hunk mode, the process considers the files equivalent + # and blame skips the reformat commit. + git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \ + blame blame.c >with && + ! grep "$BLAME_COMMIT" with +' + + test_done
Signature · 1 line
gitgitgadget
PATCH 5/5 diff-process-normalize: add built-in whitespace normalizer
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add git diff-process-normalize, a built-in diff process that detects whitespace-only changes. It compares files line by line using xdiff_compare_lines() with XDF_IGNORE_WHITESPACE (same logic as "git diff -w"). If all lines match, it returns zero hunks; otherwise it returns an error so git falls back to the builtin diff algorithm.
[diff "cdiff"]
process = git diff-process-normalize
Update documentation to describe zero-hunk behavior for diff and blame, and document the built-in normalize tool.
--- Documentation/config/diff.adoc | 2 + Documentation/gitattributes.adoc | 15 ++++ Makefile | 1 + builtin.h | 1 + builtin/diff-process-normalize.c | 143 +++++++++++++++++++++++++++++++ git.c | 1 + t/t4080-diff-process.sh | 60 +++++++++++++ 7 files changed, 223 insertions(+) create mode 100644 builtin/diff-process-normalize.c
Documentation/config/diff.adoc +2 -0
@@ -224,6 +224,8 @@ endif::git-diff[] hunks that are fed into Git's diff and blame pipelines. If the tool returns zero hunks, the file is treated as unchanged for both diff output and blame attribution. + Git provides `git diff-process-normalize` as a built-in + tool that detects whitespace-only changes. See linkgit:gitattributes[5] for details. `diff.indentHeuristic`::
Documentation/gitattributes.adoc +15 -0
@@ -861,6 +861,21 @@ the file as having no changes and produces no diff output. where it reports zero hunks, attributing lines to earlier commits instead. +Git ships with a built-in diff process, `git diff-process-normalize`, +that detects whitespace-only changes. Files whose only differences +are whitespace produce zero hunks; files with non-whitespace changes +fall back to the builtin diff algorithm. To use it: + +---------------------------------------------------------------- +[diff "cdiff"] + process = git diff-process-normalize +---------------------------------------------------------------- + +This is useful after running a code formatter: `git diff` shows +no output for files that only had whitespace changes, +`git blame` skips whitespace-only commits automatically without +requiring a `.git-blame-ignore-revs` file. + Tools should ignore unknown keys in the per-file request to remain forward-compatible.
Makefile +1 -0
@@ -1409,6 +1409,7 @@ BUILTIN_OBJS += builtin/diagnose.o BUILTIN_OBJS += builtin/diff-files.o BUILTIN_OBJS += builtin/diff-index.o BUILTIN_OBJS += builtin/diff-pairs.o +BUILTIN_OBJS += builtin/diff-process-normalize.o BUILTIN_OBJS += builtin/diff-tree.o BUILTIN_OBJS += builtin/diff.o BUILTIN_OBJS += builtin/difftool.o
builtin.h +1 -0
@@ -178,6 +178,7 @@ int cmd_diff_files(int argc, const char **argv, const char *prefix, struct reposint cmd_diff_index (int argc,const char **argv,const char *prefix,struct repository *repo);int cmd_diff (int argc,const char **argv,const char *prefix,struct repository *repo);int cmd_diff_pairs (int argc,const char **argv,const char *prefix,struct repository *repo); +int cmd_diff_process_normalize (int argc,const char **argv,const char *prefix,struct repository *repo);int cmd_diff_tree (int argc,const char **argv,const char *prefix,struct repository *repo);int cmd_difftool (int argc,const char **argv,const char *prefix,struct repository *repo);int cmd_env__helper (int argc,const char **argv,const char *prefix,struct repository *repo);
builtin/diff-process-normalize.c new file +143 -0
143 lines not on this page · raw message
git.c +1 -0
@@ -568,6 +568,7 @@ static struct cmd_struct commands[] = { { "diff-files", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT }, { "diff-index", cmd_diff_index, RUN_SETUP | NO_PARSEOPT }, { "diff-pairs", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT }, + { "diff-process-normalize", cmd_diff_process_normalize, NO_PARSEOPT }, { "diff-tree", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT }, { "difftool", cmd_difftool, RUN_SETUP_GENTLY }, { "fast-export", cmd_fast_export, RUN_SETUP },
t/t4080-diff-process.sh +60 -0
65 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Quoted 22 lines · Michael Montalbo via GitGitGadget
This series adds diff.<driver>.process, a long-running subprocess protocol that lets external tools provide hunks to git's diff and blame pipelines.
Over the past 18 years, git's diff pipeline accumulated many features that operate on hunks: word diff, function context, color-moved, indent heuristic, blame. External tools can replace the pipeline entirely (diff.<driver>.command) or select among builtin algorithms (diff.<driver>.algorithm), but there is no way for a tool to provide line-change information into the pipeline. Tools that understand code structure (tree-sitter parsers, format-aware analyzers, tools like Difftastic and Mergiraf) must bypass git's pipeline and lose access to everything downstream.
The protocol follows filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation, one tool invocation per git command. The tool receives file pairs and returns hunk descriptors that git feeds into the standard xdiff pipeline. All output features work normally.
Zero hunks with status=success means the tool considers the files equivalent. git diff shows no output for the file, and git blame skips the commit, attributing lines to earlier commits.
Quoted 7 lines
On error or tool crash, git falls back silently to the builtin diff algorithm. The feature is opt-in via diff.<driver>.process and .gitattributes; unconfigured files are unaffected.
The series includes git diff-process-normalize, a built-in tool that compares files line by line ignoring whitespace (same logic as "git diff -w" via
xdiff_compare_lines):
Interesting.
If the goal is purely to normalize content before comparison (e.g. stripping comments or canonicalizing formatting), we already have the textconv mechanism. While textconv is a "one-shot" per-file process, it is significantly simpler.
I suspect, however, that the primary focus here is to allow external tools to provide structural alignment (e.g. for AST- aware diffs like Difftastic or Mergiraf) without losing the original content in the display. Unlike textconv, which transforms the text the user sees, this protocol lets the display remain identical to the source while using a custom engine for the line-matching logic.
If that is the intent, it should be stated more explicitly in the documentation and commit messages. The "whitespace-normalize" demonstration in [PATCH 5/5] is misleading because it's exactly the case where textconv would be sufficient.
I am afraid that the use of a long-running subprocess for every diff/blame invocation adds significant complexity and overhead. In particular, wouldn't the blame implementation performs a round-trip to the subprocess for every commit in the history? Even with a persistent process, the overhead of serializing and deserializing the entire file content twice (old and new) for every commit could be prohibitive for large files or deep histories.
So, I dunno.
Quoted 47 lines · Junio C Hamano
Quoted 22 lines · Michael Montalbo via GitGitGadget
This series adds diff.<driver>.process, a long-running subprocess protocol that lets external tools provide hunks to git's diff and blame pipelines.
Over the past 18 years, git's diff pipeline accumulated many features that operate on hunks: word diff, function context, color-moved, indent heuristic, blame. External tools can replace the pipeline entirely (diff.<driver>.command) or select among builtin algorithms (diff.<driver>.algorithm), but there is no way for a tool to provide line-change information into the pipeline. Tools that understand code structure (tree-sitter parsers, format-aware analyzers, tools like Difftastic and Mergiraf) must bypass git's pipeline and lose access to everything downstream.
The protocol follows filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation, one tool invocation per git command. The tool receives file pairs and returns hunk descriptors that git feeds into the standard xdiff pipeline. All output features work normally.
Zero hunks with status=success means the tool considers the files equivalent. git diff shows no output for the file, and git blame skips the commit, attributing lines to earlier commits.
Quoted 7 lines
On error or tool crash, git falls back silently to the builtin diff algorithm. The feature is opt-in via diff.<driver>.process and .gitattributes; unconfigured files are unaffected.
The series includes git diff-process-normalize, a built-in tool that compares files line by line ignoring whitespace (same logic as "git diff -w" via
xdiff_compare_lines):Interesting.
If the goal is purely to normalize content before comparison (e.g. stripping comments or canonicalizing formatting), we already have the
textconvmechanism. Whiletextconvis a "one-shot" per-file process, it is significantly simpler.I suspect, however, that the primary focus here is to allow external tools to provide structural alignment (e.g. for AST- aware diffs like Difftastic or Mergiraf) without losing the original content in the display. Unlike
textconv, which transforms the text the user sees, this protocol lets the display remain identical to the source while using a custom engine for the line-matching logic.
Quoted 5 lines
If that is the intent, it should be stated more explicitly in the documentation and commit messages. The "whitespace-normalize" demonstration in [PATCH 5/5] is misleading because it's exactly the case where
textconvwould be sufficient.
Thank you for looking at this.
Yes, you have correctly identified the primary focus. My intention with the whitespace normalization example was to provide a kind of "hello world" diff process that would showcase how such a tool could interact with the pipeline further down (i.e., blame vs diff output). However, I do agree that it is a confusing example because it seems to clash with something textconv already provides. I will update messaging across the series to make the true intention of these changes more clear.
Quoted 9 lines
I am afraid that the use of a long-running subprocess for every diff/blame invocation adds significant complexity and overhead. In particular, wouldn't the
blameimplementation performs a round-trip to the subprocess for every commit in the history? Even with a persistent process, the overhead of serializing and deserializing the entire file content twice (old and new) for every commit could be prohibitive for large files or deep histories.So, I dunno.
I hear you on this point. Anecdotally, my measurement of running blame on diff.c:
Performance:
without process: 0.57s
with process: 0.67s
Blame attribution:
without process: 726 unique commits
with process: 721 unique commits (5 whitespace-only skipped)
Skipped commits:
0ea7d5b6f8 diff.c: fix some recent whitespace style violations 2775d92c53 diff.c: stylefix
4b25d091ba Fix a bunch of pointer declarations (codestyle) a6080a0a44 War on whitespace
eeefa7c90e Style fixes, add a space after if/for/while.
I was imagining we could potentially optimize performance by extending the protocol to enable passing OIDs as a capability so tools could read objects directly without needing any serialization/deserialization.
v2 cover diff: add diff.<driver>.process for external hunk providers
In reply to Michael Montalbo via GitGitGadgetThis series adds diff.<driver>.process, a long-running subprocess protocol that lets external tools provide hunks into git's diff and blame pipelines.
diff.<driver>.process opens the door to integrating content-aware logic that don't exist inside git: structural diff tools, format-aware analyzers, and other tools that understand the semantics of the content being tracked. Where diff.<driver>.command replaces the diff pipeline entirely, diff.<driver>.process feeds hunks into it, so all downstream features (word diff, function context, color-moved, stat, blame) work normally.
The protocol follows filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation, one tool invocation per git command. The tool receives file pairs and returns hunk descriptors that git feeds into the standard xdiff pipeline.
Zero hunks with status=success means the tool considers the files equivalent. git diff shows no output for the file, and git blame skips the commit, attributing lines to earlier commits.
On error or tool crash, git falls back silently to the builtin diff algorithm. The feature is opt-in via diff.<driver>.process and .gitattributes; unconfigured files are unaffected.
The blame integration calls the diff process for each commit in the file's history. The subprocess is long-running (one startup amortized across the traversal), but per-commit round-trips add latency. A natural follow-up would be a capability that sends blob OIDs instead of content, allowing tools that can read the object store directly to avoid the cost of serializing and deserializing blob content over the pipe for each file pair.
Changes since v1:
* Dropped the built-in diff-process-normalize tool since it obscured the main use case for the RFC and update series messaging accordingly.
* Encapsulated changed[] memory layout behind xdl_clear_changed() helper in xprepare.c.
* Restructured the external hunks path in xdl_diff() to fall through to the regular diff algorithm on validation failure instead of duplicating diff logic on fallback then returning.
* Changed subprocess error reporting from trace to warning.
* Fixed whitespace issues in the test's embedded Python.
* Changed instances of grep to test_grep in tests.
Michael Montalbo (4):
xdiff: support external hunks via xpparam_t
userdiff: add diff.<driver>.process config
diff: add long-running diff process via diff.<driver>.process
blame: consult diff process for zero-hunk detection
Documentation/config/diff.adoc | 8 + Documentation/gitattributes.adoc | 43 ++++ Makefile | 1 + blame.c | 43 +++- diff-process.c | 206 +++++++++++++++++ diff-process.h | 28 +++ diff.c | 23 ++ t/.gitattributes | 1 + t/t4080-diff-process.sh | 370 +++++++++++++++++++++++++++++++ userdiff.c | 7 + userdiff.h | 2 + xdiff-interface.c | 7 +- xdiff/xdiff.h | 13 ++ xdiff/xdiffi.c | 84 ++++++- xdiff/xprepare.c | 10 + xdiff/xprepare.h | 1 + 16 files changed, 840 insertions(+), 7 deletions(-) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100755 t/t4080-diff-process.sh
base-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/2120
Range-diff vs v1:
1: 8c0ea0bc07 ! 1: f887a7e2ba xdiff: support external hunks via xpparam_t
@@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
+ long j, prev_old_end = 0, prev_new_end = 0;
+ long total_old = 0, total_new = 0;
+
-+ /*
-+ * Clear changed[] arrays. xdl_prepare_env() may have dirtied
-+ * them via xdl_cleanup_records(). The allocation is nrec + 2
-+ * elements; changed points one past the start (see xprepare.c).
-+ */
-+ memset(xe->xdf1.changed - 1, 0,
-+ (xe->xdf1.nrec + 2) * sizeof(bool));
-+ memset(xe->xdf2.changed - 1, 0,
-+ (xe->xdf2.nrec + 2) * sizeof(bool));
++ xdl_clear_changed(&xe->xdf1);
++ xdl_clear_changed(&xe->xdf2);
+
+ for (i = 0; i < nr_hunks; i++) {
+ const struct xdl_hunk *h = &hunks[i];
@@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
+ /* Bounds check (1-based line numbers) */
+ if (h->old_count > 0 &&
+ (h->old_start < 1 ||
-+ h->old_start + h->old_count - 1 > xe->xdf1.nrec))
++ h->old_start + h->old_count - 1 > (long)xe->xdf1.nrec))
+ return -1;
+ if (h->new_count > 0 &&
+ (h->new_start < 1 ||
-+ h->new_start + h->new_count - 1 > xe->xdf2.nrec))
++ h->new_start + h->new_count - 1 > (long)xe->xdf2.nrec))
+ return -1;
+
+ /* Zero-count hunks: start must still be in [1, nrec+1] */
+ if (h->old_count == 0 &&
-+ (h->old_start < 1 || h->old_start > xe->xdf1.nrec + 1))
++ (h->old_start < 1 || h->old_start > (long)xe->xdf1.nrec + 1))
+ return -1;
+ if (h->new_count == 0 &&
-+ (h->new_start < 1 || h->new_start > xe->xdf2.nrec + 1))
++ (h->new_start < 1 || h->new_start > (long)xe->xdf2.nrec + 1))
+ return -1;
+
+ /* Ordering: no overlap with previous hunk */
@@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
- if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
--
-- return -1;
+ if (xpp->external_hunks) {
+ if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+ return -1;
+ if (xdl_populate_hunks_from_external(&xe,
+ xpp->external_hunks,
-+ xpp->external_hunks_nr) < 0) {
-+ /*
-+ * Invalid external hunks; fall back to the
-+ * builtin diff algorithm. Re-runs
-+ * xdl_prepare_env() via xdl_do_diff().
-+ */
-+ xdl_free_env(&xe);
-+ if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
-+ return -1;
-+ }
-+ } else {
-+ if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
-+ return -1;
- }
++ xpp->external_hunks_nr) == 0)
++ goto diff_done;
++ xdl_free_env(&xe);
++ }
+
++ if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
+ return -1;
+- }
++
++diff_done:
+
if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
xdl_build_script(&xe, &xscr) < 0) {
+
+ ## xdiff/xprepare.c ##
+@@ xdiff/xprepare.c: int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
+
+ return 0;
+ }
++
++/*
++ * Reset the changed[] array so that no lines are marked as changed.
++ * Also clears the sentinel slots at changed[-1] and changed[nrec]
++ * that xdl_change_compact() relies on during backward scans.
++ */
++void xdl_clear_changed(xdfile_t *xdf)
++{
++ memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
++}
+
+ ## xdiff/xprepare.h ##
+@@
+ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
+ xdfenv_t *xe);
+ void xdl_free_env(xdfenv_t *xe);
++void xdl_clear_changed(xdfile_t *xdf);
+
+
+
2: 3bc127c800 = 2: de6d85f9d7 userdiff: add diff.<driver>.process config
3: f9976fc6aa ! 3: c25647c6e5 diff: add long-running diff process via diff.<driver>.process
@@ Commit message
[diff "cdiff"]
process = /path/to/diff-tool
- The tool receives file pairs and returns hunks describing which
- lines changed. Git feeds these hunks into the standard xdiff
- pipeline, so all output features (word diff, function context,
- color) work normally.
+ The tool provides custom line-matching: it receives file pairs
+ and returns hunks that reference original line numbers. Unlike
+ textconv, which transforms the displayed content, the diff
+ output shows the actual file while the tool controls which
+ lines are marked as changed.
The handshake negotiates version=1 and capability=hunks. Per-file
requests send command=hunks, pathname, and both file contents as
packetized data. The tool responds with hunk lines and a status
- packet. On error, git falls back to the builtin diff algorithm.
+ packet. On error, git falls back to the builtin diff algorithm
+ with a warning.
- Zero hunks with status=success means the tool considers the files
- equivalent. Git skips diff output for that file entirely.
+ Zero hunks with status=success means the tool considers the
+ files equivalent. Git skips diff output for that file.
Signed-off-by: Michael Montalbo <[email protected]>
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
+An external tool can provide content-aware line matching by
+setting `diff.<name>.process` to the command that runs
+the tool. The tool is a long-running process that communicates via
-+the pkt-line protocol (see
-+linkgit:gitprotocol-long-running-process[5]).
++the pkt-line protocol (described in
++Documentation/technical/long-running-process-protocol.adoc).
+
+------------------------
+*.c diff=cdiff
@@ diff-process.c (new)
@@
+/*
+ * Diff process backend: communicates with a long-running external
-+ * tool via the pkt-line protocol to obtain content-aware hunks.
++ * tool via the pkt-line protocol to obtain custom line-matching
++ * results. Unlike textconv, which transforms the displayed content,
++ * hunks from a diff process reference original line numbers and
++ * the display shows the actual file content.
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
@@ diff.c
#include "userdiff.h"
+#include "diff-process.h"
#include "submodule.h"
-+#include "trace2.h"
#include "hashmap.h"
#include "mem-pool.h"
- #include "merge-ll.h"
@@ diff.c: static void builtin_diff(const char *name_a,
xpparam_t xpp;
xdemitconf_t xecfg;
@@ diff.c: static void builtin_diff(const char *name_a,
+ xpp.external_hunks = ext_hunks;
+ xpp.external_hunks_nr = ext_hunks_nr;
+ } else {
-+ trace2_data_string("diff",
-+ o->repo,
-+ "diff-process-fallback",
-+ name_a);
++ warning(_("diff process failed for '%s',"
++ " falling back to builtin diff"),
++ name_a);
+ }
+ }
+
@@ diff.c: static void builtin_diff(const char *name_a,
free_diff_words_data(&ecbdata);
if (textconv_one)
+ ## t/.gitattributes ##
+@@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
+ /t8005/*.txt eol=lf
+ /t9*/*.dump eol=lf
+ /t0040*.sh whitespace=-indent-with-non-tab
++/t4080-diff-process.sh whitespace=-indent-with-non-tab
+
## t/t4080-diff-process.sh (new) ##
@@
+#!/bin/sh
@@ t/t4080-diff-process.sh (new)
+ # appear as changed in the output.
+ git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+ diff boundary.c >actual &&
-+ grep "^-OLD5" actual &&
-+ grep "^-OLD6" actual &&
-+ grep "^+NEW5" actual &&
-+ grep "^+NEW6" actual &&
-+ ! grep "^-OLD9" actual &&
-+ ! grep "^-OLD10" actual &&
-+ ! grep "^+NEW9" actual &&
-+ ! grep "^+NEW10" actual
++ test_grep "^-OLD5" actual &&
++ test_grep "^-OLD6" actual &&
++ test_grep "^+NEW5" actual &&
++ test_grep "^+NEW6" actual &&
++ ! test_grep "^-OLD9" actual &&
++ ! test_grep "^-OLD10" actual &&
++ ! test_grep "^+NEW9" actual &&
++ ! test_grep "^+NEW10" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on tool error status' '
@@ t/t4080-diff-process.sh (new)
+ git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+ diff boundary.c >actual &&
+ # Fallback produces the full builtin diff (both change regions).
-+ grep "^-OLD5" actual &&
-+ grep "^+NEW5" actual &&
-+ grep "^-OLD9" actual &&
-+ grep "^+NEW9" actual &&
++ test_grep "^-OLD5" actual &&
++ test_grep "^+NEW5" actual &&
++ test_grep "^-OLD9" actual &&
++ test_grep "^+NEW9" actual &&
+ # Tool was contacted (it replied with error, not crash).
-+ grep "command=hunks pathname=boundary.c" backend.log
++ test_grep "command=hunks pathname=boundary.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process fallback on bad hunks' '
+ git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+ diff boundary.c >actual &&
-+ grep "^-OLD5" actual &&
-+ grep "^+NEW5" actual &&
-+ grep "^-OLD9" actual &&
-+ grep "^+NEW9" actual
++ test_grep "^-OLD5" actual &&
++ test_grep "^+NEW5" actual &&
++ test_grep "^-OLD9" actual &&
++ test_grep "^+NEW9" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on tool crash' '
+ git -c diff.cdiff.process="$BACKEND --mode=crash" \
+ diff boundary.c >actual &&
-+ grep "^-OLD5" actual &&
-+ grep "^+NEW5" actual &&
-+ grep "^-OLD9" actual &&
-+ grep "^+NEW9" actual
++ test_grep "^-OLD5" actual &&
++ test_grep "^+NEW5" actual &&
++ test_grep "^-OLD9" actual &&
++ test_grep "^+NEW9" actual
+'
+
+test_expect_success PYTHON 'diff process abort disables for session' '
@@ t/t4080-diff-process.sh (new)
+ git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+ diff -- abort1.c abort2.c >actual &&
+ # Both files should still produce diff output via fallback.
-+ grep "return 10" actual &&
-+ grep "return 20" actual &&
++ test_grep "return 10" actual &&
++ test_grep "return 20" actual &&
+ # The tool aborts on the first file and git clears its
+ # capability. The second file never contacts the tool,
+ # so the log should have exactly one entry, not two.
-+ grep "command=hunks" backend.log >matches &&
++ test_grep "command=hunks" backend.log >matches &&
+ test_line_count = 1 matches
+'
+
@@ t/t4080-diff-process.sh (new)
+ rm -f backend.log &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ diff -- multi1.c multi2.c >actual &&
-+ grep "return 10" actual &&
-+ grep "return 20" actual &&
-+ grep "pathname=multi1.c" backend.log &&
-+ grep "pathname=multi2.c" backend.log
++ test_grep "return 10" actual &&
++ test_grep "return 20" actual &&
++ test_grep "pathname=multi1.c" backend.log &&
++ test_grep "pathname=multi2.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process with --word-diff' '
@@ t/t4080-diff-process.sh (new)
+
+ git -c diff.cdiff.process="$BACKEND" \
+ diff --word-diff worddiff.c >actual &&
-+ grep "\[-1;-\]" actual &&
-+ grep "{+999;+}" actual
++ test_grep "\[-1;-\]" actual &&
++ test_grep "{+999;+}" actual
+'
+
+test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
+ rm -f backend.log &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ diff --diff-algorithm=patience worddiff.c >actual &&
-+ grep "return 999" actual &&
++ test_grep "return 999" actual &&
+ test_path_is_missing backend.log
+'
+
@@ t/t4080-diff-process.sh (new)
+ rm -f backend.log &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ log -1 -p -- logtest.c >actual &&
-+ grep "return 2" actual &&
-+ grep "command=hunks pathname=logtest.c" backend.log
++ test_grep "return 2" actual &&
++ test_grep "command=hunks pathname=logtest.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
4: 4e6ea6d518 ! 4: 39ff53acef blame: consult diff process for zero-hunk detection
@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks sup
+
+ # Without zero-hunk mode, blame attributes the change.
+ git blame blame.c >without &&
-+ grep "$BLAME_COMMIT" without &&
++ test_grep "$BLAME_COMMIT" without &&
+
+ # With zero-hunk mode, the process considers the files equivalent
+ # and blame skips the reformat commit.
+ git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
+ blame blame.c >with &&
-+ ! grep "$BLAME_COMMIT" with
++ ! test_grep "$BLAME_COMMIT" with
+'
+
+ 5: 8c7359b8a1 < -: ---------- diff-process-normalize: add built-in whitespace normalizer
Signature · 1 line
gitgitgadget
PATCH v2 1/4 xdiff: support external hunks via xpparam_t
From: Michael Montalbo <[email protected]>
Add two new xpparam_t fields (external_hunks, external_hunks_nr) that let callers supply pre-computed hunks. When set, xdl_diff() populates the changed[] arrays from these hunks instead of running the diff algorithm, then continues through compaction and emission as usual.
Validate supplied hunks before use: reject out-of-bounds line numbers, overlapping or out-of-order hunks, negative counts, and violations of the synchronization invariant (unchanged line counts must match between files). On validation failure, fall back to the builtin diff algorithm.
Skip trim_common_tail() in xdi_diff() when external hunks are present, since external hunks reference line numbers in the original content.
--- xdiff-interface.c | 7 +++- xdiff/xdiff.h | 13 ++++++++ xdiff/xdiffi.c | 84 +++++++++++++++++++++++++++++++++++++++++++++-- xdiff/xprepare.c | 10 ++++++ xdiff/xprepare.h | 1 + 5 files changed, 112 insertions(+), 3 deletions(-)
xdiff-interface.c +6 -1
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t coif (mf1->size >MAX_XDIFF_SIZE || mf2->size >MAX_XDIFF_SIZE )return -1 ; -if (!xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT )) +/* +* External hunks reference line numbers in the original content; +* trimming the tail would change line counts and invalidate them. +*/ +if (!xpp->external_hunks && + !xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT ))trim_common_tail (&a, &b);return xdl_diff (&a, &b, xpp, xecfg, xecb);
xdiff/xdiff.h +13 -0
@@ -78,6 +78,15 @@ typedef struct s_mmbuffer {long size; } mmbuffer_t; +/* +* Hunk descriptor for externally computed diffs. +* Line numbers are 1-based, matching unified diff convention. +*/ +struct xdl_hunk { +long old_start, old_count; +long new_start, new_count; +}; + typedef struct s_xpparam { unsigned long flags;
@@ -88,6 +97,10 @@ typedef struct s_xpparam {/* See Documentation/diff-options.adoc. */ char **anchors;size_t anchors_nr; + +/* Externally computed hunks: bypass the diff algorithm. */ +const struct xdl_hunk *external_hunks; +size_t external_hunks_nr; } xpparam_t; typedef struct s_xdemitcb {
xdiff/xdiffi.c +82 -2
98 lines not on this page · raw message
xdiff/xprepare.c +10 -0
@@ -432,3 +432,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,return 0 ; } + +/* +* Reset the changed[] array so that no lines are marked as changed. +* Also clears the sentinel slots at changed[-1] and changed[nrec] +* that xdl_change_compact() relies on during backward scans. +*/ +void xdl_clear_changed (xdfile_t *xdf) +{ +memset (xdf->changed -1 ,0 , (xdf->nrec +2 ) *sizeof (bool )); +}
xdiff/xprepare.h +1 -0
@@ -28,6 +28,7 @@int xdl_prepare_env (mmfile_t *mf1,mmfile_t *mf2,xpparam_t const *xpp,xdfenv_t *xe);void xdl_free_env (xdfenv_t *xe); +void xdl_clear_changed (xdfile_t *xdf);
Signature · 1 line
gitgitgadget
PATCH v2 2/4 userdiff: add diff.<driver>.process config
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add a new per-driver configuration key that specifies the command for a long-running diff process.
The name follows filter.<driver>.process: a long-running subprocess that stays alive across files within a single git invocation.
--- userdiff.c | 7 +++++++ userdiff.h | 2 ++ 2 files changed, 9 insertions(+)
userdiff.c +7 -0
@@ -499,6 +499,13 @@ int userdiff_config(const char *k, const char *v) drv->algorithm = drv->algorithm_owned;return ret; } +if (!strcmp (type,"process" )) { +int ret; +FREE_AND_NULL (drv->process_owned); + ret =git_config_string (&drv->process_owned, k, v); + drv->process = drv->process_owned; +return ret; + }return 0 ; }
userdiff.h +2 -0
@@ -31,6 +31,8 @@ struct userdiff_driver {char *textconv_owned;struct notes_cache *textconv_cache;int textconv_want_cache; +const char *process; +char *process_owned; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,
Signature · 1 line
gitgitgadget
PATCH v2 3/4 diff: add long-running diff process via diff.<driver>.process
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add support for external diff processes that communicate via the long-running process protocol (pkt-line over stdin/stdout).
A diff process is configured per userdiff driver:
[diff "cdiff"]
process = /path/to/diff-tool
The tool provides custom line-matching: it receives file pairs and returns hunks that reference original line numbers. Unlike textconv, which transforms the displayed content, the diff output shows the actual file while the tool controls which lines are marked as changed.
The handshake negotiates version=1 and capability=hunks. Per-file requests send command=hunks, pathname, and both file contents as packetized data. The tool responds with hunk lines and a status packet. On error, git falls back to the builtin diff algorithm with a warning.
Zero hunks with status=success means the tool considers the files equivalent. Git skips diff output for that file.
--- Documentation/config/diff.adoc | 8 + Documentation/gitattributes.adoc | 40 ++++ Makefile | 1 + diff-process.c | 206 +++++++++++++++++++ diff-process.h | 28 +++ diff.c | 23 +++ t/.gitattributes | 1 + t/t4080-diff-process.sh | 338 +++++++++++++++++++++++++++++++ 8 files changed, 645 insertions(+) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100755 t/t4080-diff-process.sh
Documentation/config/diff.adoc +8 -0
@@ -218,6 +218,14 @@ endif::git-diff[] Set this option to `true` to make the diff driver cache the text conversion outputs. See linkgit:gitattributes[5] for details. +`diff.<driver>.process`:: + The command to run as a long-running diff process. + The tool communicates via the pkt-line protocol and returns + hunks that are fed into Git's diff and blame pipelines. + If the tool returns zero hunks, the file is treated as + unchanged for both diff output and blame attribution. + See linkgit:gitattributes[5] for details. + `diff.indentHeuristic`:: Set this option to `false` to disable the default heuristics that shift diff hunk boundaries to make patches easier to read.
Documentation/gitattributes.adoc +40 -0
@@ -821,6 +821,46 @@ NOTE: If `diff.<name>.command` is defined for path with the (see above), and adding `diff.<name>.algorithm` has no effect, as the algorithm is not passed to the external diff driver. +Using an external diff process +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An external tool can provide content-aware line matching by +setting `diff.<name>.process` to the command that runs +the tool. The tool is a long-running process that communicates via +the pkt-line protocol (described in +Documentation/technical/long-running-process-protocol.adoc). + +------------------------ +*.c diff=cdiff +------------------------ + +---------------------------------------------------------------- +[diff "cdiff"] + process = /path/to/diff-process-tool +---------------------------------------------------------------- + +The tool receives file pairs and returns hunk descriptors indicating +which lines changed. Git feeds these hunks into its standard diff +pipeline, so all output features (word diff, function context, +color) work normally. + +If the tool fails or returns an error, Git silently falls back to +the builtin diff algorithm. If the tool returns invalid hunks +(out of bounds, overlapping), Git also falls back silently. + +The handshake negotiates `version=1` and `capability=hunks`. +Per-file requests send `command=hunks` and `pathname=<path>`, +followed by the old and new file content as packetized data. +The tool responds with lines of the form +`hunk <old_start> <old_count> <new_start> <new_count>` +(1-based line numbers), a flush packet, and `status=success`. + +If the tool returns zero hunks with `status=success`, Git treats +the file as having no changes and produces no diff output. + +Tools should ignore unknown keys in the per-file request to +remain forward-compatible. + Defining a custom hunk-header ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Makefile +1 -0
@@ -1142,6 +1142,7 @@ LIB_OBJS += diff-delta.o LIB_OBJS += diff-merges.o LIB_OBJS += diff-lib.o LIB_OBJS += diff-no-index.o +LIB_OBJS += diff-process.o LIB_OBJS += diff.o LIB_OBJS += diffcore-break.o LIB_OBJS += diffcore-delta.o
diff-process.c new file +206 -0
206 lines not on this page · raw message
diff-process.h new file +28 -0
@@ -0,0 +1,28 @@ +#ifndef DIFF_PROCESS_H +#define DIFF_PROCESS_H + +struct userdiff_driver ; +struct xdl_hunk ; + +/* +* Query a diff process for hunks describing the changes +* between old_buf and new_buf. +* +* The backend is a long-running subprocess configured via +* diff.<driver>.process. It receives file content via +* pkt-line and returns hunks with 1-based line numbers. +* +* On success, sets *hunks_out and *nr_hunks_out to a newly allocated +* array (caller must free) and returns 0. +* +* On failure, returns -1. The caller should fall back to the +* builtin diff algorithm. +*/ +int diff_process_get_hunks (struct userdiff_driver *drv, +const char *path, +const char *old_buf,long old_size, +const char *new_buf,long new_size, +struct xdl_hunk **hunks_out, +size_t *nr_hunks_out); + +#endif /* DIFF_PROCESS_H */
diff.c +23 -0
@@ -25,6 +25,7 @@#include "utf8.h" #include "odb.h" #include "userdiff.h" +#include "diff-process.h" #include "submodule.h" #include "hashmap.h" #include "mem-pool.h"
@@ -3991,6 +3992,7 @@ static void builtin_diff(const char *name_a,xpparam_t xpp;xdemitconf_t xecfg;struct emit_callback ecbdata; +struct xdl_hunk *ext_hunks =NULL ;unsigned ws_rule;const struct userdiff_funcname *pe;
@@ -4031,6 +4033,26 @@ static void builtin_diff(const char *name_a, xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; + +if (!o->ignore_driver_algorithm && + one->driver && one->driver->process) { +size_t ext_hunks_nr =0 ; +if (!diff_process_get_hunks ( + one->driver, name_a, + mf1.ptr, mf1.size, + mf2.ptr, mf2.size, + &ext_hunks, &ext_hunks_nr)) { +if (!ext_hunks_nr) + goto free_ab_and_return; + xpp.external_hunks = ext_hunks; + xpp.external_hunks_nr = ext_hunks_nr; + }else { +warning (_ ("diff process failed for '%s'," +" falling back to builtin diff" ), + name_a); + } + } + xecfg.ctxlen = o->context; xecfg.interhunkctxlen = o->interhunkcontext; xecfg.flags =XDL_EMIT_FUNCNAMES ;
@@ -4111,6 +4133,7 @@ static void builtin_diff(const char *name_a, }else if (xdi_diff_outf (&mf1, &mf2,NULL , fn_out_consume, &ecbdata, &xpp, &xecfg))die ("unable to generate diff for %s" , one->path); +free (ext_hunks);if (o->word_diff)free_diff_words_data (&ecbdata);if (textconv_one)
t/.gitattributes +1 -0
@@ -23,3 +23,4 @@ t[0-9][0-9][0-9][0-9]/* -whitespace /t8005/*.txt eol=lf /t9*/*.dump eol=lf /t0040*.sh whitespace=-indent-with-non-tab +/t4080-diff-process.sh whitespace=-indent-with-non-tab
t/t4080-diff-process.sh new file +338 -0
338 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process
diff-process.c line 45
Quoted 7 lines · Michael Montalbo via GitGitGadget
diff-process.c new file +7
-0+struct diff_subprocess { +struct subprocess_entry subprocess; +unsigned int supported_capabilities; +}; + +static int subprocess_map_initialized; +static struct hashmap subprocess_map;
Can we avoid introducing new global variables like these? Would "struct userdiff_driver" or "struct diff_options" be a good place to hang this hashmap, perhaps?
diff-process.c line 95
Quoted 8 lines
diff-process.c new file +8
-0+static int send_file_content (int fd,const char *buf,long size) +{ +int ret; + +if (size >0 ) + ret =write_packetized_from_buf_no_flush (buf, size, fd); +else + ret =0 ;
Shouldn't "size == -24" be flagged as an invalid input?
diff-process.c line 129
Quoted 9 lines
diff-process.c new file +8
-0+if (ret) +return ret; +return packet_flush_gently (fd); +} +static int parse_hunk_line (const char *line,struct xdl_hunk *hunk) +{ +... +}
This gives a silent error diagnosis, which is good for a lower level helper.
Quoted 9 lines
diff-process.c new file +9
-0+int diff_process_get_hunks (struct userdiff_driver *drv, +const char *path, +const char *old_buf,long old_size, +const char *new_buf,long new_size, +struct xdl_hunk **hunks_out, +size_t *nr_hunks_out) +{ +struct diff_subprocess *backend; +struct child_process *process;
diff-process.c line 149
Quoted 10 lines
diff-process.c new file +10
-0+int fd_in, fd_out; +struct strbuf status =STRBUF_INIT ; +struct xdl_hunk *hunks =NULL ; +struct xdl_hunk hunk; +size_t nr_hunks =0 , alloc_hunks =0 ; +int len; +char *line; + +if (!drv || !drv->process) +return -1 ;
A driver that does not define process is not an error; it is perfectly normal in the current world order where nobody has such an external process and even fi this patch lands, external processes are optional. So here "return -1" does not mean an error, and silent return is perfectly fine.
diff-process.c line 153
Quoted 3 lines
diff-process.c new file +3
-0+ backend =find_or_start_process (drv->process); +if (!backend) +return -1 ;
This is probably an error; the user specified drv->process, we either tried to find or start the process and failed. Isn't it an event that deserves to be reported in an error message?
Review
Quoted 2 lines
(quoted diff) +2
-0+ if (!(backend->supported_capabilities & CAP_HUNKS)) + return -1;
Backend started, but the "hunks" feature is not supported. Perhaps in a year or two, this external process protocol may have become so popular that it gained more capabilities, possibly making get_hunks obsolete. We may be looking at such an external process that uses other capabilities but not this one. This is not an error, so silent return is perfectly fine.
Quoted 35 lines
diff-process.c new file +35
-0+ process =subprocess_get_child_process (&backend->subprocess); + fd_in = process->in; + fd_out = process->out; + +/* Send request */ +if (packet_write_fmt_gently (fd_in,"command=hunks\n" ) || +packet_write_fmt_gently (fd_in,"pathname=%s\n" , path) || +packet_flush_gently (fd_in)) + goto error; + +/* Send old file content */ +if (send_file_content (fd_in, old_buf, old_size)) + goto error; + +/* Send new file content */ +if (send_file_content (fd_in, new_buf, new_size)) + goto error; + +/* Read hunks until flush packet */ +while ((len =packet_read_line_gently (fd_out,NULL , &line)) >=0 && + line) { +if (parse_hunk_line (line, &hunk) <0 ) + goto error; +ALLOC_GROW (hunks, nr_hunks +1 , alloc_hunks); + hunks[nr_hunks++] = hunk; + } +if (len <0 ) + goto error; + +/* Read status */ +if (subprocess_read_status (fd_out, &status)) + goto error; + +if (strcmp (status.buf,"success" )) { +if (!strcmp (status.buf,"abort" ))
diff-process.c line 202
Quoted 10 lines
diff-process.c new file +10
-0+ backend->supported_capabilities &= ~CAP_HUNKS ; + goto error; + } + + *hunks_out = hunks; + *nr_hunks_out = nr_hunks; +strbuf_release (&status); +return 0 ; + +error:
All exceptions that lead here look like events that should be reported to the end-user.
Quoted 9 lines
diff-process.h new file +8
-0+free (hunks); +strbuf_release (&status); +return -1 ; +} +/* +* Query a diff process for hunks describing the changes +* between old_buf and new_buf. +*
diff-process.h line 20
Quoted 10 lines
diff-process.h new file +10
-0+* The backend is a long-running subprocess configured via +* diff.<driver>.process. It receives file content via +* pkt-line and returns hunks with 1-based line numbers. +* +* On success, sets *hunks_out and *nr_hunks_out to a newly allocated +* array (caller must free) and returns 0. +* +* On failure, returns -1. The caller should fall back to the +* builtin diff algorithm. +*/
I do not agree with this. If it is a failure, the user should fix the external process (or disable). It shouldn't be hidden behind a fallback. As I left comments, in this round of implementation, there are conditions that returns -1 for soemthing that is not an error (i.e., not configured, or process not supporting the particular capability) *and* in those cases the caller should fall back as if nothing happened. But some error cases, the caller should't hide them.
Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process
Quoted 15 lines · Junio C Hamano
diff-process.c line 45
Quoted 7 lines · Michael Montalbo via GitGitGadget
diff-process.c new file +7
-0+struct diff_subprocess { +struct subprocess_entry subprocess; +unsigned int supported_capabilities; +}; + +static int subprocess_map_initialized; +static struct hashmap subprocess_map;Can we avoid introducing new global variables like these? Would "struct
userdiff_driver" or "structdiff_options" be a good place to hang this hashmap, perhaps?
Will clean this up.
Quoted 11 lines
diff-process.c line 95
Quoted 8 lines
diff-process.c new file +8
-0+static int send_file_content (int fd,const char *buf,long size) +{ +int ret; + +if (size >0 ) + ret =write_packetized_from_buf_no_flush (buf, size, fd); +else + ret =0 ;Shouldn't "size == -24" be flagged as an invalid input?
Will fix and do a broader audit of input validation and bounds checking.
Quoted 97 lines
diff-process.c line 129
Quoted 9 lines
diff-process.c new file +8
-0+if (ret) +return ret; +return packet_flush_gently (fd); +} +static int parse_hunk_line (const char *line,struct xdl_hunk *hunk) +{ +... +}This gives a silent error diagnosis, which is good for a lower level helper.
Quoted 9 lines
diff-process.c new file +9
-0+int diff_process_get_hunks (struct userdiff_driver *drv, +const char *path, +const char *old_buf,long old_size, +const char *new_buf,long new_size, +struct xdl_hunk **hunks_out, +size_t *nr_hunks_out) +{ +struct diff_subprocess *backend; +struct child_process *process;diff-process.c line 149
Quoted 10 lines
diff-process.c new file +10
-0+int fd_in, fd_out; +struct strbuf status =STRBUF_INIT ; +struct xdl_hunk *hunks =NULL ; +struct xdl_hunk hunk; +size_t nr_hunks =0 , alloc_hunks =0 ; +int len; +char *line; + +if (!drv || !drv->process) +return -1 ;A driver that does not define process is not an error; it is perfectly normal in the current world order where nobody has such an external process and even fi this patch lands, external processes are optional. So here "return -1" does not mean an error, and silent return is perfectly fine.
diff-process.c line 153
Quoted 3 lines
diff-process.c new file +3
-0+ backend =find_or_start_process (drv->process); +if (!backend) +return -1 ;This is probably an error; the user specified
drv->process, we either tried to find or start the process and failed. Isn't it an event that deserves to be reported in an error message?Review
Quoted 2 lines
(quoted diff) +2
-0+ if (!(backend->supported_capabilities & CAP_HUNKS)) + return -1;Backend started, but the "hunks" feature is not supported. Perhaps in a year or two, this external process protocol may have become so popular that it gained more capabilities, possibly making
get_hunksobsolete. We may be looking at such an external process that uses other capabilities but not this one. This is not an error, so silent return is perfectly fine.Quoted 40 lines
diff-process.c new file +40
-0+ process =subprocess_get_child_process (&backend->subprocess); + fd_in = process->in; + fd_out = process->out; + +/* Send request */ +if (packet_write_fmt_gently (fd_in,"command=hunks\n" ) || +packet_write_fmt_gently (fd_in,"pathname=%s\n" , path) || +packet_flush_gently (fd_in)) + goto error; + +/* Send old file content */ +if (send_file_content (fd_in, old_buf, old_size)) + goto error; + +/* Send new file content */ +if (send_file_content (fd_in, new_buf, new_size)) + goto error; + +/* Read hunks until flush packet */ +while ((len =packet_read_line_gently (fd_out,NULL , &line)) >=0 && + line) { +if (parse_hunk_line (line, &hunk) <0 ) + goto error; +ALLOC_GROW (hunks, nr_hunks +1 , alloc_hunks); + hunks[nr_hunks++] = hunk; + } +if (len <0 ) + goto error; + +/* Read status */ +if (subprocess_read_status (fd_out, &status)) + goto error; + +if (strcmp (status.buf,"success" )) { +if (!strcmp (status.buf,"abort" )) + backend->supported_capabilities &= ~CAP_HUNKS ; + goto error; + } + + *hunks_out = hunks;
Quoted 9 lines
diff-process.c line 202
Quoted 5 lines
diff-process.c new file +5
-0+ *nr_hunks_out = nr_hunks; +strbuf_release (&status); +return 0 ; + +error:All exceptions that lead here look like events that should be reported to the end-user.
Agreed on all points. I will restructure things so errors are flagged when appropriate (i.e., user specified a process but one was not found / couldn't start and exceptions) and non-errors are treated as they should be.
Quoted 20 lines
Quoted 9 lines
diff-process.h new file +8
-0+free (hunks); +strbuf_release (&status); +return -1 ; +} +/* +* Query a diff process for hunks describing the changes +* between old_buf and new_buf. +* diff-process.h line 20
Quoted 10 lines
diff-process.h new file +10
-0+* The backend is a long-running subprocess configured via +* diff.<driver>.process. It receives file content via +* pkt-line and returns hunks with 1-based line numbers. +* +* On success, sets *hunks_out and *nr_hunks_out to a newly allocated +* array (caller must free) and returns 0. +* +* On failure, returns -1. The caller should fall back to the +* builtin diff algorithm. +*/
Quoted 8 lines
I do not agree with this. If it is a failure, the user should fix the external process (or disable). It shouldn't be hidden behind a fallback. As I left comments, in this round of implementation, there are conditions that returns -1 for soemthing that is not an error (i.e., not configured, or process not supporting the particular capability) *and* in those cases the caller should fall back as if nothing happened. But some error cases, the caller should't hide them.
Will address in a follow-up.
Thank you for the feedback!
Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process
In reply to Michael Montalbo via GitGitGadgetQuoted 2 lines · Michael Montalbo via GitGitGadget
Zero hunks with status=success means the tool considers the files equivalent. Git skips diff output for that file.
Is "zero hunk" a common word or some random string you invented? If the latter, which is I am assuming it to be, you should define what it means at/before the first use. Here in the proposed log message, and ...
Quoted 21 lines
Signed-off-by Michael Montalbo <[email protected]>--- Documentation/config/diff.adoc | 8 + Documentation/gitattributes.adoc | 40 ++++ Makefile | 1 + diff-process.c | 206 +++++++++++++++++++ diff-process.h | 28 +++ diff.c | 23 +++ t/.gitattributes | 1 + t/t4080-diff-process.sh | 338 +++++++++++++++++++++++++++++++ 8 files changed, 645 insertions(+) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100755 t/t4080-diff-process.shDocumentation/config/diff.adoc +0
-0
Documentation/config/diff.adoc endif::git-diff[] · line 227
Quoted 10 lines
Documentation/config/diff.adoc +7
-0@@ -218,6 +218,14 @@ endif::git-diff[] Set this option to `true` to make the diff driver cache the text conversion outputs. See linkgit:gitattributes[5] for details. +`diff.<driver>.process`:: + The command to run as a long-running diff process. + The tool communicates via the pkt-line protocol and returns + hunks that are fed into Git's diff and blame pipelines. + If the tool returns zero hunks, the file is treated as + unchanged for both diff output and blame attribution. + See linkgit:gitattributes[5] for details.
... also here.
I do not know if you mean "the tool returns no hunks" (there is no "hunk <old_start> <old_count> <new_start> <new_count>" line passed from the tool over the protocol) or "the tool returns zero-hunk" (there is a special "zero-hunk" message to signal this particular condition sent over the protocol), and this description does not quite help disambiguating between the two.
If the former, then avoid "zero hunks" as it sounds like a noun with special meaning. Yes, we can say "tool returns one hunk", "tool returns 31 hunks", etc., so "tool returns zero hunks" may logically be correct, but "when the tool returns no hunks with status=success" is much less confusing, I think.
Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process
Quoted 52 lines · Junio C Hamano
Quoted 2 lines · Michael Montalbo via GitGitGadget
Zero hunks with status=success means the tool considers the files equivalent. Git skips diff output for that file.
Is "zero hunk" a common word or some random string you invented? If the latter, which is I am assuming it to be, you should define what it means at/before the first use. Here in the proposed log message, and ...
Quoted 21 lines
Signed-off-by Michael Montalbo <[email protected]>--- Documentation/config/diff.adoc | 8 + Documentation/gitattributes.adoc | 40 ++++ Makefile | 1 + diff-process.c | 206 +++++++++++++++++++ diff-process.h | 28 +++ diff.c | 23 +++ t/.gitattributes | 1 + t/t4080-diff-process.sh | 338 +++++++++++++++++++++++++++++++ 8 files changed, 645 insertions(+) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100755 t/t4080-diff-process.shDocumentation/config/diff.adoc +0
-0Documentation/config/diff.adoc endif::git-diff[] · line 227
Quoted 10 lines
Documentation/config/diff.adoc +7
-0@@ -218,6 +218,14 @@ endif::git-diff[] Set this option to `true` to make the diff driver cache the text conversion outputs. See linkgit:gitattributes[5] for details. +`diff.<driver>.process`:: + The command to run as a long-running diff process. + The tool communicates via the pkt-line protocol and returns + hunks that are fed into Git's diff and blame pipelines. + If the tool returns zero hunks, the file is treated as + unchanged for both diff output and blame attribution. + See linkgit:gitattributes[5] for details.... also here.
I do not know if you mean "the tool returns no hunks" (there is no "hunk <
old_start> <old_count> <new_start> <new_count>" line passed from the tool over the protocol) or "the tool returns zero-hunk" (there is a special "zero-hunk" message to signal this particular condition sent over the protocol), and this description does not quite help disambiguating between the two.
Quoted 5 lines
If the former, then avoid "zero hunks" as it sounds like a noun with special meaning. Yes, we can say "tool returns one hunk", "tool returns 31 hunks", etc., so "tool returns zero hunks" may logically be correct, but "when the tool returns no hunks with status=success" is much less confusing, I think.
Yes, "zero hunks" was my own invention and I see why it's confusing. Will update the messaging to use "no hunks" instead and do a broader sweep of the documentation to clarify the protocol and expected tool behavior.
PATCH v2 4/4 blame: consult diff process for zero-hunk detection
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
When a diff process is configured via diff.<driver>.process, consult it during blame's per-commit diffing. If the process returns zero hunks for a commit's changes to a file, treat the commit as having no changes, causing blame to attribute lines to earlier commits.
The subprocess is long-running (one startup cost amortized across the blame traversal), but each commit in the file's history incurs a round-trip to the tool.
--- Documentation/gitattributes.adoc | 3 +++ blame.c | 43 +++++++++++++++++++++++++++++--- t/t4080-diff-process.sh | 32 ++++++++++++++++++++++++ 3 files changed, 74 insertions(+), 4 deletions(-)
Documentation/gitattributes.adoc +3 -0
@@ -857,6 +857,9 @@ The tool responds with lines of the form If the tool returns zero hunks with `status=success`, Git treats the file as having no changes and produces no diff output. +`git blame` also consults the diff process and skips commits +where it reports zero hunks, attributing lines to earlier commits +instead. Tools should ignore unknown keys in the per-file request to remain forward-compatible.
blame.c +39 -4
75 lines not on this page · raw message
t/t4080-diff-process.sh +32 -0
@@ -335,4 +335,36 @@ test_expect_success PYTHON 'diff process zero hunks suppresses diff output' ' test_must_be_empty actual ' +test_expect_success PYTHON 'blame skips commits with zero hunks from diff process' ' + cat >blame.c <<-\EOF && + int main(void) + { + return 0; + } + EOF + git add blame.c && + git commit -m "add blame.c" && + + cat >blame.c <<-\EOF && + int main(void) + { + return 0; + } + EOF + git add blame.c && + git commit -m "reformat blame.c" && + BLAME_COMMIT=$(git rev-parse --short HEAD) && + + # Without zero-hunk mode, blame attributes the change. + git blame blame.c >without && + test_grep "$BLAME_COMMIT" without && + + # With zero-hunk mode, the process considers the files equivalent + # and blame skips the reformat commit. + git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \ + blame blame.c >with && + ! test_grep "$BLAME_COMMIT" with +' + + test_done
Signature · 1 line
gitgitgadget
v3 cover diff: add diff.<driver>.process for external hunk providers
In reply to Michael Montalbo via GitGitGadgetLanguage-aware diff tools (e.g., Difftastic) and format-specific analyzers can produce better line matching than Git's builtin diff algorithm, but diff.<driver>.command replaces Git's output entirely, losing downstream features like word diff, function context, color, and blame.
This series adds diff.<driver>.process, a long-running subprocess protocol that lets an external tool control which lines Git considers changed while Git handles all output formatting. The protocol follows filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation, one process per Git invocation.
The tool receives both file versions and returns changed regions (line ranges in the old and new file). Git validates and feeds them into the xdiff pipeline in place of the builtin diff algorithm. When the tool returns no hunks, Git treats the files as having no changes.
The first two patches add xdiff plumbing for externally supplied hunks and the diff.<driver>.process config key. Patch 3 refactors the subprocess API to separate process lifecycle from hashmap management, since the diff process stores its subprocess on the userdiff driver rather than in a hashmap. The main feature lands in patch 4. Patch 5 wires up bypass knobs (--no-ext-diff, format-patch). Patch 6 integrates with blame so the tool can declare commits as having no changes.
Changes since v2:
* Extracted subprocess_start_command() from subprocess_start() so the diff process can reuse the startup machinery without a hashmap (new patch 3).
* Subprocess stored on userdiff_driver, no global variables.
* Differentiated error handling: tool failure warns (with tool name), "not configured" is silent, abort is voluntary.
* Single public entry point: diff_process_fill_hunks() handles driver lookup, flag checks, subprocess management, and error reporting for both diff.c and blame.c.
* Rewrote gitattributes protocol section to match filter process convention with packet diagrams and fully specified hunk format.
* Split bypass knobs (--no-ext-diff, format-patch) into a separate commit.
* SIGPIPE protection, stricter input validation, const correctness.
* Changed "zero hunks" to "no hunks" throughout.
Michael Montalbo (6):
xdiff: support external hunks via xpparam_t
userdiff: add diff.<driver>.process config
sub-process: separate process lifecycle from hashmap management
diff: add long-running diff process via diff.<driver>.process
diff: bypass diff process with --no-ext-diff and in format-patch
blame: consult diff process for no-hunk detection
Documentation/config/diff.adoc | 5 + Documentation/diff-algorithm-option.adoc | 3 + Documentation/diff-options.adoc | 4 +- Documentation/gitattributes.adoc | 139 +++++ Makefile | 1 + blame.c | 40 +- builtin/log.c | 7 + diff-process.c | 288 ++++++++++ diff-process.h | 39 ++ diff.c | 29 +- diff.h | 5 + meson.build | 1 + sub-process.c | 29 +- sub-process.h | 9 +- t/.gitattributes | 1 + t/meson.build | 1 + t/t4080-diff-process.sh | 660 +++++++++++++++++++++++ userdiff.c | 7 + userdiff.h | 5 + xdiff-interface.c | 7 +- xdiff/xdiff.h | 13 + xdiff/xdiffi.c | 85 ++- xdiff/xprepare.c | 10 + xdiff/xprepare.h | 1 + 24 files changed, 1368 insertions(+), 21 deletions(-) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100755 t/t4080-diff-process.sh
base-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/2120
Range-diff vs v2:
1: f887a7e2ba ! 1: 13eb201d63 xdiff: support external hunks via xpparam_t
@@ Commit message
numbers, overlapping or out-of-order hunks, negative counts, and
violations of the synchronization invariant (unchanged line counts
must match between files). On validation failure, fall back to
- the builtin diff algorithm.
+ the builtin diff algorithm; this re-runs xdl_prepare_env() since
+ the first call may have dirtied the changed[] arrays.
Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
@@ xdiff/xdiff.h: typedef struct s_xpparam {
char **anchors;
size_t anchors_nr;
+
-+ /* Externally computed hunks: bypass the diff algorithm. */
-+ const struct xdl_hunk *external_hunks;
++ /* Externally computed hunks: bypass the diff algorithm. Owned by caller. */
++ struct xdl_hunk *external_hunks;
+ size_t external_hunks_nr;
} xpparam_t;
@@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
+ * Returns 0 on success, -1 on validation failure.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
-+ const struct xdl_hunk *hunks,
++ struct xdl_hunk *hunks,
+ size_t nr_hunks)
+{
+ size_t i;
+ long j, prev_old_end = 0, prev_new_end = 0;
+ long total_old = 0, total_new = 0;
+
++ /*
++ * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
++ * Clear them so only the external hunks are marked.
++ */
+ xdl_clear_changed(&xe->xdf1);
+ xdl_clear_changed(&xe->xdf2);
+
+ for (i = 0; i < nr_hunks; i++) {
-+ const struct xdl_hunk *h = &hunks[i];
++ struct xdl_hunk *h = &hunks[i];
+
+ if (h->old_count < 0 || h->new_count < 0)
+ return -1;
-+
-+ /* Bounds check (1-based line numbers) */
-+ if (h->old_count > 0 &&
-+ (h->old_start < 1 ||
-+ h->old_start + h->old_count - 1 > (long)xe->xdf1.nrec))
-+ return -1;
-+ if (h->new_count > 0 &&
-+ (h->new_start < 1 ||
-+ h->new_start + h->new_count - 1 > (long)xe->xdf2.nrec))
++ if (h->old_start < 1 || h->new_start < 1)
+ return -1;
+
-+ /* Zero-count hunks: start must still be in [1, nrec+1] */
-+ if (h->old_count == 0 &&
-+ (h->old_start < 1 || h->old_start > (long)xe->xdf1.nrec + 1))
++ /*
++ * Range must fit: start + count - 1 <= nrec,
++ * rewritten to avoid overflow. Same for both sides.
++ *
++ * When count is 0 (pure insert/delete) the check
++ * reduces to 0 > nrec - start + 1, which rejects
++ * start > nrec + 1 and allows start == nrec + 1
++ * (the position after the last line).
++ */
++ if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1)
+ return -1;
-+ if (h->new_count == 0 &&
-+ (h->new_start < 1 || h->new_start > (long)xe->xdf2.nrec + 1))
++ if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
+ return -1;
+
-+ /* Ordering: no overlap with previous hunk */
++ /* Ordering: no overlap with previous hunk (adjacent is OK) */
+ if (h->old_start < prev_old_end ||
+ h->new_start < prev_new_end)
+ return -1;
@@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
- }
+
+diff_done:
-+
if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
xdl_build_script(&xe, &xscr) < 0) { 2: de6d85f9d7 ! 2: 58f4763c63 userdiff: add diff.<driver>.process config
@@ Metadata
## Commit message ##
userdiff: add diff.<driver>.process config
- Add a new per-driver configuration key that specifies the command
- for a long-running diff process.
-
- The name follows filter.<driver>.process: a long-running subprocess
- that stays alive across files within a single git invocation.
+ Add the process field to struct userdiff_driver and teach the
+ config parser to populate it from diff.<driver>.process.
-: ---------- > 3: d6c833dd42 sub-process: separate process lifecycle from hashmap management
3: c25647c6e5 ! 4: d044fa0ee5 diff: add long-running diff process via diff.<driver>.process
@@ Commit message
process = /path/to/diff-tool
The tool provides custom line-matching: it receives file pairs
- and returns hunks that reference original line numbers. Unlike
- textconv, which transforms the displayed content, the diff
- output shows the actual file while the tool controls which
- lines are marked as changed.
+ and returns hunks that reference line numbers in the content.
+ When textconv is also configured, the tool receives the
+ textconv-transformed content. The tool controls which lines
+ are marked as changed while the display shows the file content.
+ Patch output features (word diff, function context, color) work
+ normally; summary formats like --stat use their own diff path
+ and are not affected.
The handshake negotiates version=1 and capability=hunks. Per-file
requests send command=hunks, pathname, and both file contents as
packetized data. The tool responds with hunk lines and a status
- packet. On error, git falls back to the builtin diff algorithm
- with a warning.
+ packet (success, error, or abort). On error, Git warns and falls
+ back to the builtin diff algorithm for that file. On abort, Git
+ silently falls back for the current file and stops sending further
+ requests to the tool for the remainder of the session.
- Zero hunks with status=success means the tool considers the
- files equivalent. Git skips diff output for that file.
+ When the tool returns no hunks followed by status=success, Git
+ treats the file as having no changes and produces no diff output.
+ This also means --exit-code reports no changes for that file.
+
+ The subprocess is stored on the userdiff_driver struct and
+ launched on first use. If the process fails to start, the
+ handshake fails, or a communication error occurs mid-stream,
+ the failure is cached on the driver to avoid retrying and
+ re-warning on every subsequent file.
+
+ diff_process_fill_hunks() is the sole public entry point. It
+ handles driver lookup, flag checks, subprocess management, and
+ error reporting, returning an enum that lets callers distinguish
+ "hunks populated" from "files equivalent" from "not applicable"
+ from "tool failure."
Signed-off-by: Michael Montalbo <[email protected]>
@@ Documentation/config/diff.adoc: endif::git-diff[]
conversion outputs. See linkgit:gitattributes[5] for details.
+`diff.<driver>.process`::
-+ The command to run as a long-running diff process.
-+ The tool communicates via the pkt-line protocol and returns
-+ hunks that are fed into Git's diff and blame pipelines.
-+ If the tool returns zero hunks, the file is treated as
-+ unchanged for both diff output and blame attribution.
++ The command to run as a long-running diff process that
++ provides hunks to Git's diff pipeline.
+ See linkgit:gitattributes[5] for details.
+
`diff.indentHeuristic`::
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
algorithm is not passed to the external diff driver.
+Using an external diff process
-+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-+
-+An external tool can provide content-aware line matching by
-+setting `diff.<name>.process` to the command that runs
-+the tool. The tool is a long-running process that communicates via
-+the pkt-line protocol (described in
++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
++
++If `diff.<name>.process` is defined, Git sends the old and new file
++content to an external tool and receives back a list of changed
++regions (pairs of line ranges in the old and new file). Git uses
++these instead of its builtin diff algorithm, but still controls
++all output formatting, so features like word diff, function context,
++color, and blame work normally. This is achieved by using the
++long-running process protocol (described in
+Documentation/technical/long-running-process-protocol.adoc).
++Unlike `diff.<name>.command`, which replaces Git's output entirely,
++the diff process feeds results back into the standard pipeline.
++
++First, in `.gitattributes`, assign the `diff` attribute for paths.
+
+------------------------
+*.c diff=cdiff
+------------------------
+
++Then, define a "diff.<name>.process" configuration to specify
++the diff process command.
++
+----------------------------------------------------------------
+[diff "cdiff"]
+ process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
-+The tool receives file pairs and returns hunk descriptors indicating
-+which lines changed. Git feeds these hunks into its standard diff
-+pipeline, so all output features (word diff, function context,
-+color) work normally.
-+
-+If the tool fails or returns an error, Git silently falls back to
-+the builtin diff algorithm. If the tool returns invalid hunks
-+(out of bounds, overlapping), Git also falls back silently.
-+
-+The handshake negotiates `version=1` and `capability=hunks`.
-+Per-file requests send `command=hunks` and `pathname=<path>`,
-+followed by the old and new file content as packetized data.
-+The tool responds with lines of the form
-+`hunk <old_start> <old_count> <new_start> <new_count>`
-+(1-based line numbers), a flush packet, and `status=success`.
-+
-+If the tool returns zero hunks with `status=success`, Git treats
-+the file as having no changes and produces no diff output.
-+
-+Tools should ignore unknown keys in the per-file request to
-+remain forward-compatible.
++When Git encounters the first file that needs to be diffed, it starts
++the process and performs the handshake. In the handshake, the welcome
++message sent by Git is "git-diff-client", only version 1 is supported,
++and the supported capability is "hunks" (the changed regions
++described below).
++
++For each file, Git sends a list of "key=value" pairs terminated with
++a flush packet, followed by the old and new file content as packetized
++data, each terminated with a flush packet. The pathname is relative
++to the repository root. When `diff.<name>.textconv` is also set,
++the tool receives the textconv-transformed content rather than the
++raw blob. Git does not send binary files to the diff process.
++
++-----------------------
++packet: git> command=hunks
++packet: git> pathname=path/file.c
++packet: git> 0000
++packet: git> OLD_CONTENT
++packet: git> 0000
++packet: git> NEW_CONTENT
++packet: git> 0000
++-----------------------
++
++The tool is expected to respond with zero or more hunk lines,
++a flush packet, and a status packet terminated with a flush packet.
++Each hunk line has the form:
++
++ `hunk <old_start> <old_count> <new_start> <new_count>`
++
++where `<old_start>` and `<old_count>` identify a range of lines in
++the old file, and `<new_start>` and `<new_count>` identify the
++replacement range in the new file. Start values are 1-based and
++counts are non-negative. Ranges must not extend beyond the end of
++the file. For example, `hunk 3 2 3 4` means that 2 lines starting
++at line 3 in the old file were replaced by 4 lines starting at
++line 3 in the new file. An `<old_count>` of 0 means no lines were
++removed (pure insertion); a `<new_count>` of 0 means no lines were
++added (pure deletion).
++
++Lines are delimited by newlines. A file `"foo\nbar\n"` and a
++file `"foo\nbar"` both have 2 lines.
++
++Hunks must be listed in order and must not overlap. Any line
++not covered by a hunk is treated as unchanged, so the total
++number of unchanged lines must be the same on both sides.
++For example, if the old file has 10 lines and the hunks cover
++4 of them (`old_count` values summing to 4), then 6 old lines
++are unchanged. The new file must also have exactly 6 lines
++not covered by hunks, so the `new_count` values must sum to
++`new_file_lines - 6`.
++
++-----------------------
++packet: git< hunk 1 3 1 5
++packet: git< hunk 10 2 12 2
++packet: git< 0000
++packet: git< status=success
++packet: git< 0000
++-----------------------
++
++If the tool responds with hunks and "success", Git marks those lines
++as changed and feeds them into the standard diff pipeline. Patch
++output features (word diff, function context, color) work normally.
++Note that `--stat` and other summary formats use their own diff path
++and are not affected by the diff process.
++
++If no hunk lines precede the flush, followed by "success", Git
++treats the files as having no changes: `git diff` produces no output
++and `git blame` skips the commit, attributing lines to earlier commits.
++
++-----------------------
++packet: git< 0000
++packet: git< status=success
++packet: git< 0000
++-----------------------
++
++If the tool returns invalid hunks (out of bounds, overlapping), Git
++silently falls back to the builtin diff algorithm.
++
++In case the tool cannot or does not want to process the content,
++it is expected to respond with an "error" status. Git warns and
++falls back to the builtin diff algorithm for this file. The tool
++remains available for subsequent files.
++
++-----------------------
++packet: git< 0000
++packet: git< status=error
++packet: git< 0000
++-----------------------
++
++In case the tool cannot or does not want to process the content as
++well as any future content for the lifetime of the Git process, it
++is expected to respond with an "abort" status. Git silently falls
++back to the builtin diff algorithm for this file and does not send
++further requests to the tool.
++
++-----------------------
++packet: git< 0000
++packet: git< status=abort
++packet: git< 0000
++-----------------------
++
++If the tool dies during the communication or does not adhere to the
++protocol then Git will stop the process and fall back to the builtin
++diff algorithm. Git warns once and does not restart the process for
++subsequent files.
++
++Tools should ignore unknown keys in the per-file request to remain
++forward-compatible. Future versions of Git may send additional
++`command=` values; tools that receive an unrecognized command should
++respond with `status=error` rather than terminating.
+
Defining a custom hunk-header
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ diff-process.c (new)
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain custom line-matching
-+ * results. Unlike textconv, which transforms the displayed content,
-+ * hunks from a diff process reference original line numbers and
-+ * the display shows the actual file content.
++ * results. The tool controls which lines are marked as changed
++ * while the display shows the file content (after any textconv
++ * transformation, if configured).
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
@@ diff-process.c (new)
+ * tool< ... / flush
+ * tool< status=success / flush
+ *
-+ * Zero hunks with status=success means the tool considers the
-+ * files equivalent. Git will skip the diff for that file.
++ * When the tool returns no hunks with status=success, it considers
++ * the files equivalent. Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
++#include "diff.h"
++#include "gettext.h"
++#include "repository.h"
++#include "sigchain.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
@@ diff-process.c (new)
+ unsigned int supported_capabilities;
+};
+
-+static int subprocess_map_initialized;
-+static struct hashmap subprocess_map;
-+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+ static int versions[] = { 1, 0 };
@@ diff-process.c (new)
+ { NULL, 0 }
+ };
+ struct diff_subprocess *entry =
-+ (struct diff_subprocess *)subprocess;
++ container_of(subprocess, struct diff_subprocess, subprocess);
+
-+ /* Uses dying pkt-line variant, same as convert.c filters. */
+ return subprocess_handshake(subprocess, "git-diff",
+ versions, NULL,
+ capabilities,
+ &entry->supported_capabilities);
+}
+
-+static struct diff_subprocess *find_or_start_process(const char *cmd)
++static struct diff_subprocess *get_or_launch_process(
++ struct userdiff_driver *drv)
+{
+ struct diff_subprocess *entry;
+
-+ if (!subprocess_map_initialized) {
-+ subprocess_map_initialized = 1;
-+ hashmap_init(&subprocess_map, cmd2process_cmp, NULL, 0);
-+ }
-+
-+ entry = (struct diff_subprocess *)
-+ subprocess_find_entry(&subprocess_map, cmd);
-+ if (entry)
-+ return entry;
++ if (drv->diff_subprocess)
++ return drv->diff_subprocess;
+
+ entry = xcalloc(1, sizeof(*entry));
-+ if (subprocess_start(&subprocess_map, &entry->subprocess,
-+ cmd, start_diff_process_fn)) {
++ if (subprocess_start_command(&entry->subprocess, drv->process,
++ start_diff_process_fn)) {
+ free(entry);
++ drv->diff_process_failed = 1;
+ return NULL;
+ }
+
++ drv->diff_subprocess = entry;
+ return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
-+ int ret;
++ int ret = 0;
+
++ if (size < 0)
++ return -1;
+ if (size > 0)
+ ret = write_packetized_from_buf_no_flush(buf, size, fd);
-+ else
-+ ret = 0;
+ if (ret)
+ return ret;
+ return packet_flush_gently(fd);
@@ diff-process.c (new)
+{
+ char *end;
+
-+ /* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
++ /*
++ * Format: "hunk <old_start> <old_count> <new_start> <new_count>"
++ * All numbers must be non-negative decimal with no leading
++ * whitespace or sign characters.
++ */
+ if (!skip_prefix(line, "hunk ", &line))
+ return -1;
+
++ if (!isdigit(*line))
++ return -1;
++ errno = 0;
+ hunk->old_start = strtol(line, &end, 10);
-+ if (end == line || *end != ' ')
++ if (errno || end == line || *end++ != ' ')
+ return -1;
+ line = end;
+
++ if (!isdigit(*line))
++ return -1;
++ errno = 0;
+ hunk->old_count = strtol(line, &end, 10);
-+ if (end == line || *end != ' ')
++ if (errno || end == line || *end++ != ' ')
+ return -1;
+ line = end;
+
++ if (!isdigit(*line))
++ return -1;
++ errno = 0;
+ hunk->new_start = strtol(line, &end, 10);
-+ if (end == line || *end != ' ')
++ if (errno || end == line || *end++ != ' ')
+ return -1;
+ line = end;
+
++ if (!isdigit(*line))
++ return -1;
++ errno = 0;
+ hunk->new_count = strtol(line, &end, 10);
-+ if (end == line || *end != '\0')
++ if (errno || end == line || *end != '\0')
+ return -1;
+
+ return 0;
+}
+
-+int diff_process_get_hunks(struct userdiff_driver *drv,
-+ const char *path,
-+ const char *old_buf, long old_size,
-+ const char *new_buf, long new_size,
-+ struct xdl_hunk **hunks_out,
-+ size_t *nr_hunks_out)
++static enum diff_process_result get_hunks(
++ struct userdiff_driver *drv,
++ const char *path,
++ const char *old_buf, long old_size,
++ const char *new_buf, long new_size,
++ struct xdl_hunk **hunks_out,
++ size_t *nr_hunks_out)
+{
+ struct diff_subprocess *backend;
+ struct child_process *process;
@@ diff-process.c (new)
+ int len;
+ char *line;
+
-+ if (!drv || !drv->process)
-+ return -1;
-+
-+ backend = find_or_start_process(drv->process);
++ backend = get_or_launch_process(drv);
+ if (!backend)
-+ return -1;
++ return DIFF_PROCESS_ERROR;
+
+ if (!(backend->supported_capabilities & CAP_HUNKS))
-+ return -1;
++ return DIFF_PROCESS_SKIP;
+
+ process = subprocess_get_child_process(&backend->subprocess);
+ fd_in = process->in;
+ fd_out = process->out;
+
++ sigchain_push(SIGPIPE, SIG_IGN);
++
+ /* Send request */
+ if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+ packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
+ packet_flush_gently(fd_in))
-+ goto error;
++ goto comm_error;
+
+ /* Send old file content */
+ if (send_file_content(fd_in, old_buf, old_size))
-+ goto error;
++ goto comm_error;
+
+ /* Send new file content */
+ if (send_file_content(fd_in, new_buf, new_size))
-+ goto error;
++ goto comm_error;
+
+ /* Read hunks until flush packet */
+ while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+ line) {
+ if (parse_hunk_line(line, &hunk) < 0)
-+ goto error;
++ goto comm_error;
+ ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+ hunks[nr_hunks++] = hunk;
+ }
+ if (len < 0)
-+ goto error;
++ goto comm_error;
+
+ /* Read status */
+ if (subprocess_read_status(fd_out, &status))
-+ goto error;
++ goto comm_error;
++
++ if (!strcmp(status.buf, "success")) {
++ *hunks_out = hunks;
++ *nr_hunks_out = nr_hunks;
++ strbuf_release(&status);
++ sigchain_pop(SIGPIPE);
++ return DIFF_PROCESS_OK;
++ }
+
-+ if (strcmp(status.buf, "success")) {
-+ if (!strcmp(status.buf, "abort"))
-+ backend->supported_capabilities &= ~CAP_HUNKS;
-+ goto error;
++ if (!strcmp(status.buf, "abort")) {
++ /*
++ * The tool voluntarily withdrew: stop sending requests
++ * but do not warn (this is not a failure).
++ */
++ backend->supported_capabilities &= ~CAP_HUNKS;
++ free(hunks);
++ strbuf_release(&status);
++ sigchain_pop(SIGPIPE);
++ return DIFF_PROCESS_SKIP;
+ }
+
-+ *hunks_out = hunks;
-+ *nr_hunks_out = nr_hunks;
++ /* status=error or unknown status */
++ free(hunks);
+ strbuf_release(&status);
-+ return 0;
-+
-+error:
++ sigchain_pop(SIGPIPE);
++ return DIFF_PROCESS_ERROR;
++
++comm_error:
++ /*
++ * Communication failure (broken pipe, malformed response).
++ * Tear down the process and mark as failed so we do not
++ * retry on every subsequent file.
++ */
++ drv->diff_process_failed = 1;
++ drv->diff_subprocess = NULL;
++ subprocess_stop_command(&backend->subprocess);
++ free(backend);
+ free(hunks);
+ strbuf_release(&status);
-+ return -1;
++ sigchain_pop(SIGPIPE);
++ return DIFF_PROCESS_ERROR;
++}
++
++enum diff_process_result diff_process_fill_hunks(
++ struct diff_options *diffopt,
++ const char *path,
++ const mmfile_t *file_a,
++ const mmfile_t *file_b,
++ xpparam_t *xpp)
++{
++ struct userdiff_driver *drv;
++ struct xdl_hunk *ext_hunks = NULL;
++ size_t nr = 0;
++ enum diff_process_result res;
++
++ if (!diffopt || !path)
++ return DIFF_PROCESS_SKIP;
++ if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
++ return DIFF_PROCESS_SKIP;
++
++ drv = userdiff_find_by_path(diffopt->repo->index, path);
++ if (!drv || !drv->process)
++ return DIFF_PROCESS_SKIP;
++ if (drv->diff_process_failed)
++ return DIFF_PROCESS_SKIP;
++
++ res = get_hunks(drv, path,
++ file_a->ptr, file_a->size,
++ file_b->ptr, file_b->size,
++ &ext_hunks, &nr);
++ if (res == DIFF_PROCESS_OK) {
++ if (!nr) {
++ free(ext_hunks);
++ return DIFF_PROCESS_EQUIVALENT;
++ }
++ xpp->external_hunks = ext_hunks;
++ xpp->external_hunks_nr = nr;
++ return DIFF_PROCESS_OK;
++ }
++ if (res == DIFF_PROCESS_ERROR) {
++ warning(_("diff process '%s' failed for '%s',"
++ " falling back to builtin diff"),
++ drv->process, path);
++ return DIFF_PROCESS_ERROR;
++ }
++ return DIFF_PROCESS_SKIP;
+}
## diff-process.h (new) ##
@@ diff-process.h (new)
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
-+struct userdiff_driver;
-+struct xdl_hunk;
++#include "xdiff/xdiff.h"
++
++struct diff_options;
++
++enum diff_process_result {
++ DIFF_PROCESS_ERROR = -1, /* tool failure: warned, fell back */
++ DIFF_PROCESS_OK = 0, /* hunks populated in xpp */
++ DIFF_PROCESS_SKIP, /* no process configured: use builtin */
++ DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
++};
+
+/*
-+ * Query a diff process for hunks describing the changes
-+ * between old_buf and new_buf.
++ * Consult the diff process configured for 'path' and populate
++ * xpp->external_hunks with the returned hunks.
+ *
-+ * The backend is a long-running subprocess configured via
-+ * diff.<driver>.process. It receives file content via
-+ * pkt-line and returns hunks with 1-based line numbers.
++ * Handles driver lookup, flag checks (--no-ext-diff,
++ * --diff-algorithm), subprocess management, and error reporting.
+ *
-+ * On success, sets *hunks_out and *nr_hunks_out to a newly allocated
-+ * array (caller must free) and returns 0.
++ * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
++ * The caller owns xpp->external_hunks and must free() it.
+ *
-+ * On failure, returns -1. The caller should fall back to the
-+ * builtin diff algorithm.
++ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks
++ * (files are considered identical); caller should skip diff/blame.
++ * Returns DIFF_PROCESS_SKIP when no process applies; caller
++ * should use the builtin diff algorithm.
++ * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
++ * caller should fall back to the builtin diff algorithm.
+ */
-+int diff_process_get_hunks(struct userdiff_driver *drv,
-+ const char *path,
-+ const char *old_buf, long old_size,
-+ const char *new_buf, long new_size,
-+ struct xdl_hunk **hunks_out,
-+ size_t *nr_hunks_out);
++enum diff_process_result diff_process_fill_hunks(
++ struct diff_options *diffopt,
++ const char *path,
++ const mmfile_t *file_a,
++ const mmfile_t *file_b,
++ xpparam_t *xpp);
+
+#endif /* DIFF_PROCESS_H */
@@ diff.c
#include "submodule.h"
#include "hashmap.h"
#include "mem-pool.h"
-@@ diff.c: static void builtin_diff(const char *name_a,
- xpparam_t xpp;
- xdemitconf_t xecfg;
- struct emit_callback ecbdata;
-+ struct xdl_hunk *ext_hunks = NULL;
- unsigned ws_rule;
- const struct userdiff_funcname *pe;
-
@@ diff.c: static void builtin_diff(const char *name_a,
xpp.ignore_regex_nr = o->ignore_regex_nr;
xpp.anchors = o->anchors;
xpp.anchors_nr = o->anchors_nr;
+
-+ if (!o->ignore_driver_algorithm &&
-+ one->driver && one->driver->process) {
-+ size_t ext_hunks_nr = 0;
-+ if (!diff_process_get_hunks(
-+ one->driver, name_a,
-+ mf1.ptr, mf1.size,
-+ mf2.ptr, mf2.size,
-+ &ext_hunks, &ext_hunks_nr)) {
-+ if (!ext_hunks_nr)
-+ goto free_ab_and_return;
-+ xpp.external_hunks = ext_hunks;
-+ xpp.external_hunks_nr = ext_hunks_nr;
-+ } else {
-+ warning(_("diff process failed for '%s',"
-+ " falling back to builtin diff"),
-+ name_a);
-+ }
++ if (diff_process_fill_hunks(o, name_a,
++ &mf1, &mf2, &xpp)
++ == DIFF_PROCESS_EQUIVALENT) {
++ if (textconv_one)
++ free(mf1.ptr);
++ if (textconv_two)
++ free(mf2.ptr);
++ goto free_ab_and_return;
+ }
+
xecfg.ctxlen = o->context;
@@ diff.c: static void builtin_diff(const char *name_a,
} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
&ecbdata, &xpp, &xecfg))
die("unable to generate diff for %s", one->path);
-+ free(ext_hunks);
++ free(xpp.external_hunks);
if (o->word_diff)
free_diff_words_data(&ecbdata);
if (textconv_one)
+ ## diff.h ##
+@@ diff.h: struct diff_flags {
+ */
+ unsigned allow_external;
+
++ /** Disables diff.<driver>.process. */
++ unsigned no_diff_process;
++
+ /**
+ * For communication between the calling program and the options parser;
+ * tell the calling program to signal the presence of difference using
+
+ ## meson.build ##
+@@ meson.build: libgit_sources = [
+ 'diff-merges.c',
+ 'diff-lib.c',
+ 'diff-no-index.c',
++ 'diff-process.c',
+ 'diff.c',
+ 'diffcore-break.c',
+ 'diffcore-delta.c',
+
## t/.gitattributes ##
@@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
/t8005/*.txt eol=lf
@@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
/t0040*.sh whitespace=-indent-with-non-tab
+/t4080-diff-process.sh whitespace=-indent-with-non-tab
+ ## t/meson.build ##
+@@ t/meson.build: integration_tests = [
+ 't4072-diff-max-depth.sh',
+ 't4073-diff-stat-name-width.sh',
+ 't4074-diff-shifted-matched-group.sh',
++ 't4080-diff-process.sh',
+ 't4100-apply-stat.sh',
+ 't4101-apply-nonl.sh',
+ 't4102-apply-rename.sh',
+
## t/t4080-diff-process.sh (new) ##
@@
+#!/bin/sh
@@ t/t4080-diff-process.sh (new)
+# whole-file - report all lines as changed (default)
+# fixed-hunk - always report hunk 5 2 5 2
+# bad-hunk - report out-of-bounds hunk 999 1 999 1
-+# zero-hunk - return zero hunks (files considered equivalent)
++# bad-sync - report hunk with mismatched unchanged totals
++# overlap - report two overlapping hunks
++# no-hunks - return no hunks (files considered equivalent)
+# error - return status=error for every request
+# abort - return status=abort for every request
+# crash - read one request then exit without responding
@@ t/t4080-diff-process.sh (new)
+ if cmd is None: sys.exit(0)
+ old = read_content()
+ new = read_content()
-+ log(f"command={cmd} pathname={pathname}")
++ old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
++ new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
++ log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
+
+ if mode == "error":
+ write_flush()
@@ t/t4080-diff-process.sh (new)
+ write_pkt("hunk 5 2 5 2")
+ elif mode == "bad-hunk":
+ write_pkt("hunk 999 1 999 1")
-+ elif mode == "zero-hunk":
++ elif mode == "bad-sync":
++ write_pkt("hunk 1 2 1 1")
++ elif mode == "overlap":
++ write_pkt("hunk 1 5 1 5")
++ write_pkt("hunk 3 2 3 2")
++ elif mode == "no-hunks":
+ pass
+ else:
-+ ol = len(old.split(b"\n"))
-+ nl = len(new.split(b"\n"))
++ ol = old.count(b"\n")
++ nl = new.count(b"\n")
+ write_pkt(f"hunk 1 {ol} 1 {nl}")
+ write_flush()
+ write_pkt("status=success")
@@ t/t4080-diff-process.sh (new)
+ setup_backend &&
+ echo "*.c diff=cdiff" >.gitattributes &&
+ git add .gitattributes &&
-+ git commit -m "initial"
-+'
+
-+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
++ # boundary.c: 10 lines, changes at 5-6 and 9-10.
++ # Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
+ cat >boundary.c <<-\EOF &&
+ line1
+ line2
@@ t/t4080-diff-process.sh (new)
+ OLD10
+ EOF
+ git add boundary.c &&
-+ git commit -m "add boundary.c" &&
+
++ # worddiff.c: single-line function, value changes 1 -> 999.
++ # Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
++ cat >worddiff.c <<-\EOF &&
++ int value(void) { return 1; }
++ EOF
++ git add worddiff.c &&
++
++ # newfile.c: single-line function, value changes 42 -> 99.
++ # Used by: new file, --exit-code, multiple drivers.
++ cat >newfile.c <<-\EOF &&
++ int new_func(void) { return 42; }
++ EOF
++ git add newfile.c &&
++
++ # logtest.c: single-line function for log/format-patch tests.
++ # Needs two commits so log -1 has a diff.
++ cat >logtest.c <<-\EOF &&
++ int logfunc(void) { return 1; }
++ EOF
++ git add logtest.c &&
++
++ # two.c/one.c: two-file pair for error/abort/startup-failure tests.
++ cat >one.c <<-\EOF &&
++ int first(void) { return 1; }
++ EOF
++ cat >two.c <<-\EOF &&
++ int second(void) { return 2; }
++ EOF
++ git add one.c two.c &&
++
++ git commit -m "initial" &&
++
++ # Second commit for logtest.c (so log -1 has something to show).
++ cat >logtest.c <<-\EOF &&
++ int logfunc(void) { return 2; }
++ EOF
++ git add logtest.c &&
++ git commit -m "change logtest.c" &&
++
++ # Working tree modifications (not committed).
+ cat >boundary.c <<-\EOF &&
+ line1
+ line2
@@ t/t4080-diff-process.sh (new)
+ NEW10
+ EOF
+
++ cat >worddiff.c <<-\EOF &&
++ int value(void) { return 999; }
++ EOF
++
++ cat >newfile.c <<-\EOF &&
++ int new_func(void) { return 99; }
++ EOF
++
++ cat >one.c <<-\EOF &&
++ int first(void) { return 10; }
++ EOF
++
++ cat >two.c <<-\EOF
++ int second(void) { return 20; }
++ EOF
++'
++
++#
++# Core behavior: the tool controls which lines are marked as changed.
++#
++
++test_expect_success PYTHON 'diff process hunk boundaries affect output' '
+ # The file has changes at lines 5-6 and 9-10, but fixed-hunk
+ # only reports lines 5-6 as changed. Lines 9-10 should not
+ # appear as changed in the output.
@@ t/t4080-diff-process.sh (new)
+ test_grep "^-OLD6" actual &&
+ test_grep "^+NEW5" actual &&
+ test_grep "^+NEW6" actual &&
-+ ! test_grep "^-OLD9" actual &&
-+ ! test_grep "^-OLD10" actual &&
-+ ! test_grep "^+NEW9" actual &&
-+ ! test_grep "^+NEW10" actual
++ test_grep ! "^-OLD9" actual &&
++ test_grep ! "^-OLD10" actual &&
++ test_grep ! "^+NEW9" actual &&
++ test_grep ! "^+NEW10" actual
+'
+
-+test_expect_success PYTHON 'diff process fallback on tool error status' '
++test_expect_success PYTHON 'diff process works with new file' '
+ rm -f backend.log &&
-+ git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
-+ diff boundary.c >actual &&
-+ # Fallback produces the full builtin diff (both change regions).
-+ test_grep "^-OLD5" actual &&
-+ test_grep "^+NEW5" actual &&
-+ test_grep "^-OLD9" actual &&
-+ test_grep "^+NEW9" actual &&
-+ # Tool was contacted (it replied with error, not crash).
-+ test_grep "command=hunks pathname=boundary.c" backend.log
++ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
++ diff -- newfile.c >actual 2>stderr &&
++ test_grep "return 99" actual &&
++ test_grep "pathname=newfile.c" backend.log &&
++ test_must_be_empty stderr
+'
+
-+test_expect_success PYTHON 'diff process fallback on bad hunks' '
-+ git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
-+ diff boundary.c >actual &&
-+ test_grep "^-OLD5" actual &&
-+ test_grep "^+NEW5" actual &&
-+ test_grep "^-OLD9" actual &&
-+ test_grep "^+NEW9" actual
++test_expect_success PYTHON 'diff process works with added file (empty old side)' '
++ cat >added.c <<-\EOF &&
++ int added(void) { return 1; }
++ EOF
++ git add added.c &&
++
++ rm -f backend.log &&
++ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
++ diff --cached -- added.c >actual 2>stderr &&
++ test_grep "added" actual &&
++ test_grep "pathname=added.c" backend.log &&
++ test_must_be_empty stderr
+'
+
-+test_expect_success PYTHON 'diff process fallback on tool crash' '
-+ git -c diff.cdiff.process="$BACKEND --mode=crash" \
-+ diff boundary.c >actual &&
-+ test_grep "^-OLD5" actual &&
-+ test_grep "^+NEW5" actual &&
-+ test_grep "^-OLD9" actual &&
-+ test_grep "^+NEW9" actual
++test_expect_success PYTHON 'diff process skipped for binary files' '
++ printf "\\0binary" >binary.c &&
++ git add binary.c &&
++ git commit -m "add binary" &&
++ printf "\\0changed" >binary.c &&
++
++ rm -f backend.log &&
++ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
++ diff -- binary.c >actual &&
++ test_grep "Binary files" actual &&
++ test_path_is_missing backend.log
+'
+
-+test_expect_success PYTHON 'diff process abort disables for session' '
-+ cat >abort1.c <<-\EOF &&
-+ int first(void) { return 1; }
-+ EOF
-+ cat >abort2.c <<-\EOF &&
-+ int second(void) { return 2; }
-+ EOF
-+ git add abort1.c abort2.c &&
-+ git commit -m "add abort files" &&
++test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
++ echo "not tracked by cdiff" >unmatched.txt &&
++ git add unmatched.txt &&
++ git commit -m "add unmatched.txt" &&
+
-+ cat >abort1.c <<-\EOF &&
-+ int first(void) { return 10; }
-+ EOF
-+ cat >abort2.c <<-\EOF &&
-+ int second(void) { return 20; }
-+ EOF
++ echo "modified" >unmatched.txt &&
+
+ rm -f backend.log &&
-+ git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
-+ diff -- abort1.c abort2.c >actual &&
-+ # Both files should still produce diff output via fallback.
-+ test_grep "return 10" actual &&
-+ test_grep "return 20" actual &&
-+ # The tool aborts on the first file and git clears its
-+ # capability. The second file never contacts the tool,
-+ # so the log should have exactly one entry, not two.
-+ test_grep "command=hunks" backend.log >matches &&
-+ test_line_count = 1 matches
++ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
++ diff -- unmatched.txt >actual &&
++ test_grep "modified" actual &&
++ test_path_is_missing backend.log
+'
+
-+test_expect_success PYTHON 'diff process handles multiple files' '
-+ cat >multi1.c <<-\EOF &&
-+ int one(void) { return 1; }
++test_expect_success PYTHON 'multiple drivers use separate processes' '
++ echo "*.h diff=hdiff" >>.gitattributes &&
++ git add .gitattributes &&
++
++ cat >multi.h <<-\EOF &&
++ int header(void) { return 1; }
+ EOF
-+ cat >multi2.c <<-\EOF &&
-+ int two(void) { return 2; }
++ git add multi.h &&
++ git commit -m "add multi.h" &&
++
++ cat >multi.h <<-\EOF &&
++ int header(void) { return 2; }
+ EOF
-+ git add multi1.c multi2.c &&
-+ git commit -m "add multi files" &&
+
-+ cat >multi1.c <<-\EOF &&
-+ int one(void) { return 10; }
++ rm -f backend-c.log backend-h.log &&
++ git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
++ -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
++ diff -- newfile.c multi.h >actual 2>stderr &&
++ test_grep "pathname=newfile.c" backend-c.log &&
++ test_grep "pathname=multi.h" backend-h.log &&
++ test_must_be_empty stderr
++'
++
++test_expect_success PYTHON 'diff process works alongside textconv' '
++ write_script uppercase-filter <<-\EOF &&
++ tr "a-z" "A-Z" <"$1"
+ EOF
-+ cat >multi2.c <<-\EOF &&
-+ int two(void) { return 20; }
++
++ cat >textconv.c <<-\EOF &&
++ hello world
++ EOF
++ git add textconv.c &&
++ git commit -m "add textconv.c" &&
++
++ cat >textconv.c <<-\EOF &&
++ goodbye world
+ EOF
+
+ rm -f backend.log &&
-+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
-+ diff -- multi1.c multi2.c >actual &&
-+ test_grep "return 10" actual &&
-+ test_grep "return 20" actual &&
-+ test_grep "pathname=multi1.c" backend.log &&
-+ test_grep "pathname=multi2.c" backend.log
++ git -c diff.cdiff.textconv="./uppercase-filter" \
++ -c diff.cdiff.process="$BACKEND --log=backend.log" \
++ diff -- textconv.c >actual 2>stderr &&
++ # The diff process receives textconv-transformed (uppercase) content.
++ test_grep "pathname=textconv.c" backend.log &&
++ test_grep "old=HELLO WORLD" backend.log &&
++ test_grep "new=GOODBYE WORLD" backend.log &&
++ test_must_be_empty stderr
+'
+
++#
++# Downstream features: word diff, log, equivalent files, exit code.
++#
++
+test_expect_success PYTHON 'diff process with --word-diff' '
-+ cat >worddiff.c <<-\EOF &&
-+ int value(void) { return 1; }
++ rm -f backend.log &&
++ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
++ diff --word-diff worddiff.c >actual 2>stderr &&
++ test_grep "\[-1;-\]" actual &&
++ test_grep "{+999;+}" actual &&
++ test_grep "pathname=worddiff.c" backend.log &&
++ test_must_be_empty stderr
++'
++
++test_expect_success PYTHON 'diff process works with git log -p' '
++ # With no-hunks mode, the tool says the files are equivalent,
++ # so log -p should show the commit but no diff content.
++ rm -f backend.log &&
++ git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
++ log -1 -p -- logtest.c >actual 2>stderr &&
++ test_grep "change logtest.c" actual &&
++ test_grep ! "return 2" actual &&
++ test_grep "command=hunks pathname=logtest.c" backend.log &&
++ test_must_be_empty stderr
++'
++
++test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
++ cat >nohunks.c <<-\EOF &&
++ int zero(void) { return 0; }
+ EOF
-+ git add worddiff.c &&
-+ git commit -m "add worddiff.c" &&
++ git add nohunks.c &&
++ git commit -m "add nohunks.c" &&
+
-+ cat >worddiff.c <<-\EOF &&
-+ int value(void) { return 999; }
++ cat >nohunks.c <<-\EOF &&
++ int zero(void) { return 999; }
+ EOF
+
-+ git -c diff.cdiff.process="$BACKEND" \
-+ diff --word-diff worddiff.c >actual &&
-+ test_grep "\[-1;-\]" actual &&
-+ test_grep "{+999;+}" actual
++ git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
++ diff nohunks.c >actual &&
++ test_must_be_empty actual
++'
++
++test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
++ git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
++ diff --exit-code nohunks.c
++'
++
++test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
++ test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
++ diff --exit-code newfile.c
+'
+
++#
++# Bypass mechanisms: flags and commands that skip the diff process.
++#
++
+test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
+ rm -f backend.log &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
@@ t/t4080-diff-process.sh (new)
+ test_path_is_missing backend.log
+'
+
-+test_expect_success PYTHON 'diff process works with git log -p' '
-+ cat >logtest.c <<-\EOF &&
-+ int logfunc(void) { return 1; }
-+ EOF
-+ git add logtest.c &&
-+ git commit -m "add logtest.c" &&
++test_expect_success PYTHON 'diff process not used by --stat' '
++ rm -f backend.log &&
++ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
++ diff --stat worddiff.c >actual &&
++ test_grep "worddiff.c" actual &&
++ test_path_is_missing backend.log
++'
+
-+ cat >logtest.c <<-\EOF &&
-+ int logfunc(void) { return 2; }
-+ EOF
-+ git add logtest.c &&
-+ git commit -m "change logtest.c" &&
++#
++# Error handling and fallback.
++#
+
++test_expect_success PYTHON 'diff process fallback on tool error status' '
+ rm -f backend.log &&
-+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
-+ log -1 -p -- logtest.c >actual &&
-+ test_grep "return 2" actual &&
-+ test_grep "command=hunks pathname=logtest.c" backend.log
++ git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
++ diff boundary.c >actual 2>stderr &&
++ # Fallback produces the full builtin diff (both change regions).
++ test_grep "^-OLD5" actual &&
++ test_grep "^+NEW5" actual &&
++ test_grep "^-OLD9" actual &&
++ test_grep "^+NEW9" actual &&
++ # Tool was contacted (it replied with error, not crash).
++ test_grep "command=hunks pathname=boundary.c" backend.log &&
++ test_grep "diff process.*failed" stderr
+'
+
-+test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
-+ cat >zerohunk.c <<-\EOF &&
-+ int zero(void) { return 0; }
++test_expect_success PYTHON 'diff process error keeps tool available for next file' '
++ rm -f backend.log &&
++ git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
++ diff -- one.c two.c >actual 2>stderr &&
++ # Unlike abort, error keeps the tool available: both files
++ # are sent to the tool (and both fall back).
++ test_grep "pathname=one.c" backend.log &&
++ test_grep "pathname=two.c" backend.log &&
++ test_grep "return 10" actual &&
++ test_grep "return 20" actual
++'
++
++test_expect_success PYTHON 'diff process abort disables for session' '
++ rm -f backend.log &&
++ git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
++ diff -- one.c two.c >actual &&
++ # Both files should still produce diff output via fallback.
++ test_grep "return 10" actual &&
++ test_grep "return 20" actual &&
++ # The tool aborts on the first file and git clears its
++ # capability. The second file never contacts the tool.
++ test_grep "pathname=one.c" backend.log &&
++ test_grep ! "pathname=two.c" backend.log
++'
++
++test_expect_success PYTHON 'diff process fallback on tool crash' '
++ git -c diff.cdiff.process="$BACKEND --mode=crash" \
++ diff boundary.c >actual 2>stderr &&
++ test_grep "^-OLD5" actual &&
++ test_grep "^+NEW5" actual &&
++ test_grep "^-OLD9" actual &&
++ test_grep "^+NEW9" actual &&
++ # Crash is a communication failure, so a warning is emitted.
++ test_grep "diff process.*failed" stderr
++'
++
++test_expect_success PYTHON 'diff process startup failure only warns once' '
++ git -c diff.cdiff.process="/nonexistent/tool" \
++ diff -- one.c two.c >actual 2>stderr &&
++ # Both files produce diff output via fallback.
++ test_grep "return 10" actual &&
++ test_grep "return 20" actual &&
++ # Sentinel prevents repeated warnings: only one, not one per file.
++ test_grep "diff process.*failed" stderr >warnings &&
++ test_line_count = 1 warnings
++'
++
++test_expect_success PYTHON 'diff process fallback on bad hunks' '
++ git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
++ diff boundary.c >actual 2>stderr &&
++ test_grep "^-OLD5" actual &&
++ test_grep "^+NEW5" actual &&
++ test_grep "^-OLD9" actual &&
++ test_grep "^+NEW9" actual &&
++ # Invalid hunks are caught by xdiff validation, not the
++ # protocol layer, so no warning is emitted.
++ test_must_be_empty stderr
++'
++
++test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
++ cat >synctest.c <<-\EOF &&
++ line1
++ line2
++ line3
+ EOF
-+ git add zerohunk.c &&
-+ git commit -m "add zerohunk.c" &&
++ git add synctest.c &&
++ git commit -m "add synctest.c" &&
+
-+ cat >zerohunk.c <<-\EOF &&
-+ int zero(void) { return 999; }
++ cat >synctest.c <<-\EOF &&
++ line1
++ changed
++ line3
+ EOF
+
-+ git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
-+ diff zerohunk.c >actual &&
-+ test_must_be_empty actual
++ # bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
++ # line as changed, leaving 1 unchanged old vs 2 unchanged new.
++ # The synchronization invariant fails and git falls back.
++ git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
++ diff synctest.c >actual 2>stderr &&
++ test_grep "changed" actual
++'
++
++test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
++ # boundary.c has 10 lines, so both hunks are in bounds
++ # but they overlap at lines 3-5, triggering the ordering check.
++ git -c diff.cdiff.process="$BACKEND --mode=overlap" \
++ diff boundary.c >actual 2>stderr &&
++ test_grep "NEW5" actual
+'
+
+test_done
+
+ ## userdiff.h ##
+@@
+
+ #include "notes-cache.h"
+
++struct diff_subprocess;
+ struct index_state;
+ struct repository;
+
+@@ userdiff.h: struct userdiff_driver {
+ int textconv_want_cache;
+ const char *process;
+ char *process_owned;
++ struct diff_subprocess *diff_subprocess;
++ unsigned diff_process_failed : 1;
+ };
+ enum userdiff_driver_type {
+ USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-: ---------- > 5: f4fd9aa682 diff: bypass diff process with --no-ext-diff and in format-patch
4: 39ff53acef ! 6: 370e766978 blame: consult diff process for zero-hunk detection
@@ Metadata
Author: Michael Montalbo <[email protected]>
## Commit message ##
- blame: consult diff process for zero-hunk detection
+ blame: consult diff process for no-hunk detection
When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing. If the process
- returns zero hunks for a commit's changes to a file, treat the
+ returns no hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.
+ The consultation happens at the pass_blame_to_parent() callsite
+ using diff_process_fill_hunks(), matching how builtin_diff() in
+ diff.c uses the same function. A new diff_hunks_xpp() variant
+ accepts a pre-populated xpparam_t for this callsite, while the
+ existing diff_hunks() retains its original signature and behavior.
+ The copy-detection callsite is unaffected since it does not use
+ the diff process.
+
The subprocess is long-running (one startup cost amortized
across the blame traversal), but each commit in the file's
history incurs a round-trip to the tool.
Signed-off-by: Michael Montalbo <[email protected]>
- ## Documentation/gitattributes.adoc ##
-@@ Documentation/gitattributes.adoc: The tool responds with lines of the form
-
- If the tool returns zero hunks with `status=success`, Git treats
- the file as having no changes and produces no diff output.
-+`git blame` also consults the diff process and skips commits
-+where it reports zero hunks, attributing lines to earlier commits
-+instead.
-
- Tools should ignore unknown keys in the per-file request to
- remain forward-compatible.
-
## blame.c ##
@@
#include "tag.h"
#include "trace2.h"
#include "blame.h"
+#include "diff-process.h"
-+#include "userdiff.h"
++#include "xdiff-interface.h"
#include "alloc.h"
#include "commit-slab.h"
#include "bloom.h"
@@ blame.c: static struct commit *fake_working_tree_commit(struct repository *r,
- static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
+
+-static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
- xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
-+ xdl_emit_hunk_consume_func_t hunk_func, void *cb_data,
-+ int xdl_opts, struct index_state *istate,
-+ const char *path)
++static int diff_hunks_xpp(mmfile_t *file_a, mmfile_t *file_b,
++ xdl_emit_hunk_consume_func_t hunk_func,
++ void *cb_data, xpparam_t *xpp)
{
- xpparam_t xpp = {0};
+- xpparam_t xpp = {0};
xdemitconf_t xecfg = {0};
xdemitcb_t ecb = {NULL};
-+ struct xdl_hunk *ext_hunks = NULL;
-+ int ret;
- xpp.flags = xdl_opts;
+- xpp.flags = xdl_opts;
xecfg.hunk_func = hunk_func;
ecb.priv = cb_data;
- return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
++ return xdi_diff(file_a, file_b, xpp, &xecfg, &ecb);
++}
+
-+ if (path && istate) {
-+ struct userdiff_driver *drv;
-+ drv = userdiff_find_by_path(istate, path);
-+ if (drv && drv->process) {
-+ size_t nr = 0;
-+ if (!diff_process_get_hunks(drv, path,
-+ file_a->ptr, file_a->size,
-+ file_b->ptr, file_b->size,
-+ &ext_hunks, &nr)) {
-+ if (!nr) {
-+ /*
-+ * Zero hunks: the diff process
-+ * considers these files equivalent.
-+ * Skip so blame looks past this
-+ * commit.
-+ */
-+ return 0;
-+ }
-+ xpp.external_hunks = ext_hunks;
-+ xpp.external_hunks_nr = nr;
-+ }
-+ }
-+ }
++static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
++ xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
++{
++ xpparam_t xpp = {0};
+
-+ ret = xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
-+ free(ext_hunks);
-+ return ret;
++ xpp.flags = xdl_opts;
++ return diff_hunks_xpp(file_a, file_b, hunk_func, cb_data, &xpp);
}
static const char *get_next_line(const char *start, const char *end)
+@@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
+ struct blame_origin *parent, int ignore_diffs)
+ {
+ mmfile_t file_p, file_o;
++ xpparam_t xpp = {0};
+ struct blame_chunk_cb_data d;
+ struct blame_entry *newdest = NULL;
+
@@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
&sb->num_read_blob, ignore_diffs);
sb->num_get_patch++;
- if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
-+ if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts,
-+ sb->revs->diffopt.repo->index, target->path))
- die("unable to generate diff (%s -> %s)",
- oid_to_hex(&parent->commit->object.oid),
- oid_to_hex(&target->commit->object.oid));
-@@ blame.c: static void find_copy_in_blob(struct blame_scoreboard *sb,
- * file_p partially may match that image.
- */
- memset(split, 0, sizeof(struct blame_entry [3]));
-- if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts))
-+ if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts,
-+ NULL, NULL))
- die("unable to generate diff (%s)",
- oid_to_hex(&parent->commit->object.oid));
- /* remainder, if any, all match the preimage */
+- die("unable to generate diff (%s -> %s)",
+- oid_to_hex(&parent->commit->object.oid),
+- oid_to_hex(&target->commit->object.oid));
++ xpp.flags = sb->xdl_opts;
++ /*
++ * If the diff process considers the files equivalent,
++ * skip the diff so blame looks past this commit.
++ */
++ if (diff_process_fill_hunks(&sb->revs->diffopt, target->path,
++ &file_p, &file_o, &xpp)
++ != DIFF_PROCESS_EQUIVALENT) {
++ if (diff_hunks_xpp(&file_p, &file_o, blame_chunk_cb,
++ &d, &xpp))
++ die("unable to generate diff (%s -> %s)",
++ oid_to_hex(&parent->commit->object.oid),
++ oid_to_hex(&target->commit->object.oid));
++ }
++ free(xpp.external_hunks);
+ /* The rest are the same as the parent */
+ blame_chunk(&d.dstq, &d.srcq, INT_MAX, d.offset, INT_MAX, 0,
+ parent, target, 0);
## t/t4080-diff-process.sh ##
-@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
- test_must_be_empty actual
+@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
+ test_grep "NEW5" actual
'
-+test_expect_success PYTHON 'blame skips commits with zero hunks from diff process' '
++#
++# Blame integration.
++#
++
++test_expect_success PYTHON 'blame uses tool-provided hunks' '
++ cat >blame-hunk.c <<-\EOF &&
++ line1
++ line2
++ line3
++ line4
++ original5
++ original6
++ line7
++ line8
++ line9
++ line10
++ EOF
++ git add blame-hunk.c &&
++ git commit -m "add blame-hunk.c" &&
++ ORIG=$(git rev-parse --short HEAD) &&
++
++ cat >blame-hunk.c <<-\EOF &&
++ line1
++ line2
++ line3
++ line4
++ changed5
++ changed6
++ line7
++ line8
++ changed9
++ changed10
++ EOF
++ git add blame-hunk.c &&
++ git commit -m "change blame-hunk.c" &&
++ CHANGE=$(git rev-parse --short HEAD) &&
++
++ # With fixed-hunk mode the tool reports only lines 5-6 as changed,
++ # so blame should attribute lines 9-10 to the original commit
++ # even though the builtin diff would show them as changed.
++ git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
++ blame blame-hunk.c >actual &&
++ sed -n "9p" actual >line9 &&
++ sed -n "10p" actual >line10 &&
++ test_grep "$ORIG" line9 &&
++ test_grep "$ORIG" line10 &&
++ sed -n "5p" actual >line5 &&
++ sed -n "6p" actual >line6 &&
++ test_grep "$CHANGE" line5 &&
++ test_grep "$CHANGE" line6
++'
++
++test_expect_success PYTHON 'blame skips commits with no hunks from diff process' '
+ cat >blame.c <<-\EOF &&
+ int main(void)
+ {
@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks sup
+ EOF
+ git add blame.c &&
+ git commit -m "add blame.c" &&
++ ORIG_COMMIT=$(git rev-parse --short HEAD) &&
+
+ cat >blame.c <<-\EOF &&
+ int main(void)
@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks sup
+ git commit -m "reformat blame.c" &&
+ BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
-+ # Without zero-hunk mode, blame attributes the change.
++ # Without no-hunks mode, blame attributes the change.
+ git blame blame.c >without &&
+ test_grep "$BLAME_COMMIT" without &&
+
-+ # With zero-hunk mode, the process considers the files equivalent
-+ # and blame skips the reformat commit.
-+ git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
++ # With no-hunks mode, the process considers the files equivalent
++ # and blame skips the reformat commit, attributing to the original.
++ git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+ blame blame.c >with &&
-+ ! test_grep "$BLAME_COMMIT" with
++ test_grep ! "$BLAME_COMMIT" with &&
++ test_grep "$ORIG_COMMIT" with
+'
+
++test_expect_success PYTHON 'blame --no-ext-diff bypasses diff process' '
++ rm -f backend.log &&
++ git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
++ blame --no-ext-diff blame.c >actual &&
++ # Without the process, blame attributes the reformat commit normally.
++ test_grep "$BLAME_COMMIT" actual &&
++ test_path_is_missing backend.log
++'
++
++test_expect_success PYTHON 'blame --no-ext-diff uses builtin hunks' '
++ # fixed-hunk mode would narrow blame to lines 5-6, but
++ # --no-ext-diff should bypass it and use the builtin diff.
++ rm -f backend.log &&
++ git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
++ blame --no-ext-diff blame-hunk.c >actual &&
++ # Builtin diff attributes lines 9-10 to the change commit.
++ sed -n "9p" actual >line9 &&
++ test_grep "$CHANGE" line9 &&
++ test_path_is_missing backend.log
++'
+
test_doneSignature · 1 line
gitgitgadget
PATCH v3 1/6 xdiff: support external hunks via xpparam_t
From: Michael Montalbo <[email protected]>
Add two new xpparam_t fields (external_hunks, external_hunks_nr) that let callers supply pre-computed hunks. When set, xdl_diff() populates the changed[] arrays from these hunks instead of running the diff algorithm, then continues through compaction and emission as usual.
Validate supplied hunks before use: reject out-of-bounds line numbers, overlapping or out-of-order hunks, negative counts, and violations of the synchronization invariant (unchanged line counts must match between files). On validation failure, fall back to the builtin diff algorithm; this re-runs xdl_prepare_env() since the first call may have dirtied the changed[] arrays.
Skip trim_common_tail() in xdi_diff() when external hunks are present, since external hunks reference line numbers in the original content.
--- xdiff-interface.c | 7 +++- xdiff/xdiff.h | 13 ++++++++ xdiff/xdiffi.c | 85 +++++++++++++++++++++++++++++++++++++++++++++-- xdiff/xprepare.c | 10 ++++++ xdiff/xprepare.h | 1 + 5 files changed, 113 insertions(+), 3 deletions(-)
xdiff-interface.c +6 -1
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t coif (mf1->size >MAX_XDIFF_SIZE || mf2->size >MAX_XDIFF_SIZE )return -1 ; -if (!xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT )) +/* +* External hunks reference line numbers in the original content; +* trimming the tail would change line counts and invalidate them. +*/ +if (!xpp->external_hunks && + !xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT ))trim_common_tail (&a, &b);return xdl_diff (&a, &b, xpp, xecfg, xecb);
xdiff/xdiff.h +13 -0
@@ -78,6 +78,15 @@ typedef struct s_mmbuffer {long size; } mmbuffer_t; +/* +* Hunk descriptor for externally computed diffs. +* Line numbers are 1-based, matching unified diff convention. +*/ +struct xdl_hunk { +long old_start, old_count; +long new_start, new_count; +}; + typedef struct s_xpparam { unsigned long flags;
@@ -88,6 +97,10 @@ typedef struct s_xpparam {/* See Documentation/diff-options.adoc. */ char **anchors;size_t anchors_nr; + +/* Externally computed hunks: bypass the diff algorithm. Owned by caller. */ +struct xdl_hunk *external_hunks; +size_t external_hunks_nr; } xpparam_t; typedef struct s_xdemitcb {
xdiff/xdiffi.c +83 -2
99 lines not on this page · raw message
xdiff/xprepare.c +10 -0
@@ -432,3 +432,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,return 0 ; } + +/* +* Reset the changed[] array so that no lines are marked as changed. +* Also clears the sentinel slots at changed[-1] and changed[nrec] +* that xdl_change_compact() relies on during backward scans. +*/ +void xdl_clear_changed (xdfile_t *xdf) +{ +memset (xdf->changed -1 ,0 , (xdf->nrec +2 ) *sizeof (bool )); +}
xdiff/xprepare.h +1 -0
@@ -28,6 +28,7 @@int xdl_prepare_env (mmfile_t *mf1,mmfile_t *mf2,xpparam_t const *xpp,xdfenv_t *xe);void xdl_free_env (xdfenv_t *xe); +void xdl_clear_changed (xdfile_t *xdf);
Signature · 1 line
gitgitgadget
PATCH v3 2/6 userdiff: add diff.<driver>.process config
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add the process field to struct userdiff_driver and teach the config parser to populate it from diff.<driver>.process.
--- userdiff.c | 7 +++++++ userdiff.h | 2 ++ 2 files changed, 9 insertions(+)
userdiff.c +7 -0
@@ -499,6 +499,13 @@ int userdiff_config(const char *k, const char *v) drv->algorithm = drv->algorithm_owned;return ret; } +if (!strcmp (type,"process" )) { +int ret; +FREE_AND_NULL (drv->process_owned); + ret =git_config_string (&drv->process_owned, k, v); + drv->process = drv->process_owned; +return ret; + }return 0 ; }
userdiff.h +2 -0
@@ -31,6 +31,8 @@ struct userdiff_driver {char *textconv_owned;struct notes_cache *textconv_cache;int textconv_want_cache; +const char *process; +char *process_owned; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,
Signature · 1 line
gitgitgadget
PATCH v3 3/6 sub-process: separate process lifecycle from hashmap management
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
subprocess_start() and subprocess_stop() couple two concerns:
managing a child process (setup, handshake, teardown) and managing a hashmap that indexes running processes by command string. The hashmap suits callers like convert.c where many files may share one filter process looked up by name, but callers that manage process lifetime through their own data structures do not need it.
Extract subprocess_start_command() and subprocess_stop_command() so callers can reuse the child process setup and handshake machinery without maintaining a hashmap. subprocess_start() and subprocess_stop() become thin wrappers that add hashmap operations on top.
No functional change for existing callers.
--- sub-process.c | 29 ++++++++++++++++++++++++----- sub-process.h | 9 ++++++++- 2 files changed, 32 insertions(+), 6 deletions(-)
sub-process.c +24 -5
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)return (len <0 ) ? len :0 ; } -void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry) +void subprocess_stop_command (struct subprocess_entry *entry) {if (!entry)return ;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry) entry->process.clean_on_exit =0 ;kill (entry->process.pid,SIGTERM );finish_command (&entry->process); +} + +void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry) +{ +if (!entry) +return ; +subprocess_stop_command (entry);hashmap_remove (hashmap, &entry->ent,NULL ); }
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)finish_command (process); } -int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd, +int subprocess_start_command (struct subprocess_entry *entry,const char *cmd,subprocess_start_fn startfn) {int err;
@@ -96,15 +103,27 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, coreturn err; } -hashmap_entry_init (&entry->ent,strhash (cmd)); - err =startfn (entry);if (err) {error ("initialization for subprocess '%s' failed" , cmd); -subprocess_stop (hashmap, entry); +subprocess_stop_command (entry);return err; } +return 0 ; +} + +int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd, +subprocess_start_fn startfn) +{ +int err; + + err =subprocess_start_command (entry, cmd, startfn); +if (err) { +return err; + } + +hashmap_entry_init (&entry->ent,strhash (cmd));hashmap_add (hashmap, &entry->ent);return 0 ; }
sub-process.h +8 -1
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,*/ typedef int (*subprocess_start_fn )(struct subprocess_entry *entry); -/* Start a subprocess and add it to the subprocess hashmap. */ +/* Start a subprocess and run the startfn (typically handshake). */ +int subprocess_start_command (struct subprocess_entry *entry,const char *cmd, +subprocess_start_fn startfn); + +/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */ int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd,subprocess_start_fn startfn); +/* Kill a subprocess. */ +void subprocess_stop_command (struct subprocess_entry *entry); +/* Kill a subprocess and remove it from the subprocess hashmap. */ void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry);
Signature · 1 line
gitgitgadget
PATCH v3 4/6 diff: add long-running diff process via diff.<driver>.process
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add support for external diff processes that communicate via the long-running process protocol (pkt-line over stdin/stdout).
A diff process is configured per userdiff driver:
[diff "cdiff"]
process = /path/to/diff-tool
The tool provides custom line-matching: it receives file pairs and returns hunks that reference line numbers in the content. When textconv is also configured, the tool receives the textconv-transformed content. The tool controls which lines are marked as changed while the display shows the file content. Patch output features (word diff, function context, color) work normally; summary formats like --stat use their own diff path and are not affected.
The handshake negotiates version=1 and capability=hunks. Per-file requests send command=hunks, pathname, and both file contents as packetized data. The tool responds with hunk lines and a status packet (success, error, or abort). On error, Git warns and falls back to the builtin diff algorithm for that file. On abort, Git silently falls back for the current file and stops sending further requests to the tool for the remainder of the session.
When the tool returns no hunks followed by status=success, Git treats the file as having no changes and produces no diff output. This also means --exit-code reports no changes for that file.
The subprocess is stored on the userdiff_driver struct and launched on first use. If the process fails to start, the handshake fails, or a communication error occurs mid-stream, the failure is cached on the driver to avoid retrying and re-warning on every subsequent file.
diff_process_fill_hunks() is the sole public entry point. It handles driver lookup, flag checks, subprocess management, and error reporting, returning an enum that lets callers distinguish "hunks populated" from "files equivalent" from "not applicable" from "tool failure."
--- Documentation/config/diff.adoc | 5 + Documentation/gitattributes.adoc | 139 ++++++++ Makefile | 1 + diff-process.c | 288 +++++++++++++++++ diff-process.h | 39 +++ diff.c | 13 + diff.h | 3 + meson.build | 1 + t/.gitattributes | 1 + t/meson.build | 1 + t/t4080-diff-process.sh | 538 +++++++++++++++++++++++++++++++ userdiff.h | 3 + 12 files changed, 1032 insertions(+) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100755 t/t4080-diff-process.sh
Documentation/config/diff.adoc +5 -0
@@ -218,6 +218,11 @@ endif::git-diff[] Set this option to `true` to make the diff driver cache the text conversion outputs. See linkgit:gitattributes[5] for details. +`diff.<driver>.process`:: + The command to run as a long-running diff process that + provides hunks to Git's diff pipeline. + See linkgit:gitattributes[5] for details. + `diff.indentHeuristic`:: Set this option to `false` to disable the default heuristics that shift diff hunk boundaries to make patches easier to read.
Documentation/gitattributes.adoc +139 -0
145 lines not on this page · raw message
Makefile +1 -0
@@ -1142,6 +1142,7 @@ LIB_OBJS += diff-delta.o LIB_OBJS += diff-merges.o LIB_OBJS += diff-lib.o LIB_OBJS += diff-no-index.o +LIB_OBJS += diff-process.o LIB_OBJS += diff.o LIB_OBJS += diffcore-break.o LIB_OBJS += diffcore-delta.o
diff-process.c new file +288 -0
288 lines not on this page · raw message
diff-process.h new file +39 -0
@@ -0,0 +1,39 @@ +#ifndef DIFF_PROCESS_H +#define DIFF_PROCESS_H + +#include "xdiff/xdiff.h" + +struct diff_options ; + +enum diff_process_result { +DIFF_PROCESS_ERROR =-1 ,/* tool failure: warned, fell back */ +DIFF_PROCESS_OK =0 ,/* hunks populated in xpp */ +DIFF_PROCESS_SKIP ,/* no process configured: use builtin */ +DIFF_PROCESS_EQUIVALENT ,/* tool says files are equivalent */ +}; + +/* +* Consult the diff process configured for 'path' and populate +* xpp->external_hunks with the returned hunks. +* +* Handles driver lookup, flag checks (--no-ext-diff, +* --diff-algorithm), subprocess management, and error reporting. +* +* Returns DIFF_PROCESS_OK when hunks are populated in xpp. +* The caller owns xpp->external_hunks and must free() it. +* +* Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks +* (files are considered identical); caller should skip diff/blame. +* Returns DIFF_PROCESS_SKIP when no process applies; caller +* should use the builtin diff algorithm. +* Returns DIFF_PROCESS_ERROR on tool failure (already warned); +* caller should fall back to the builtin diff algorithm. +*/ +enum diff_process_result diff_process_fill_hunks ( +struct diff_options *diffopt, +const char *path, +const mmfile_t *file_a, +const mmfile_t *file_b, +xpparam_t *xpp); + +#endif /* DIFF_PROCESS_H */
diff.c +13 -0
@@ -25,6 +25,7 @@#include "utf8.h" #include "odb.h" #include "userdiff.h" +#include "diff-process.h" #include "submodule.h" #include "hashmap.h" #include "mem-pool.h"
@@ -4031,6 +4032,17 @@ static void builtin_diff(const char *name_a, xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; + +if (diff_process_fill_hunks (o, name_a, + &mf1, &mf2, &xpp) + ==DIFF_PROCESS_EQUIVALENT ) { +if (textconv_one) +free (mf1.ptr); +if (textconv_two) +free (mf2.ptr); + goto free_ab_and_return; + } + xecfg.ctxlen = o->context; xecfg.interhunkctxlen = o->interhunkcontext; xecfg.flags =XDL_EMIT_FUNCNAMES ;
@@ -4111,6 +4123,7 @@ static void builtin_diff(const char *name_a, }else if (xdi_diff_outf (&mf1, &mf2,NULL , fn_out_consume, &ecbdata, &xpp, &xecfg))die ("unable to generate diff for %s" , one->path); +free (xpp.external_hunks);if (o->word_diff)free_diff_words_data (&ecbdata);if (textconv_one)
diff.h +3 -0
@@ -173,6 +173,9 @@ struct diff_flags {*/ unsigned allow_external; +/** Disables diff.<driver>.process. */ +unsigned no_diff_process; +/** * For communication between the calling program and the options parser; * tell the calling program to signal the presence of difference using
meson.build +1 -0
@@ -328,6 +328,7 @@ libgit_sources = [ 'diff-merges.c', 'diff-lib.c', 'diff-no-index.c', + 'diff-process.c', 'diff.c', 'diffcore-break.c', 'diffcore-delta.c',
t/.gitattributes +1 -0
@@ -23,3 +23,4 @@ t[0-9][0-9][0-9][0-9]/* -whitespace /t8005/*.txt eol=lf /t9*/*.dump eol=lf /t0040*.sh whitespace=-indent-with-non-tab +/t4080-diff-process.sh whitespace=-indent-with-non-tab
t/meson.build +1 -0
@@ -510,6 +510,7 @@ integration_tests = [ 't4072-diff-max-depth.sh', 't4073-diff-stat-name-width.sh', 't4074-diff-shifted-matched-group.sh', + 't4080-diff-process.sh', 't4100-apply-stat.sh', 't4101-apply-nonl.sh', 't4102-apply-rename.sh',
t/t4080-diff-process.sh new file +538 -0
538 lines not on this page · raw message
userdiff.h +3 -0
@@ -3,6 +3,7 @@#include "notes-cache.h" +struct diff_subprocess ;struct index_state ;struct repository ;
@@ -33,6 +34,8 @@ struct userdiff_driver {int textconv_want_cache;const char *process;char *process_owned; +struct diff_subprocess *diff_subprocess; + unsigned diff_process_failed : 1; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,
Signature · 1 line
gitgitgadget
Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process
Hi Michael,
I stumbled about this patch when it broke CI in Git for Windows, where we do _not_ use NO_PYTHON, even though Python is unavailable in the build/test CI jobs. The existing tests handle this situation gracefully, this here patch does not:
Quoted 6 lines · Michael Montalbo via GitGitGadget
t/t4080-diff-process.sh new file +0
-0
t/t4080-diff-process.sh line 9
Quoted 9 lines
t/t4080-diff-process.sh new file +9
-0@@ -0,0 +1,538 @@ +#!/bin/sh + +test_description='diff process via long-running process' + +. ./test-lib.sh + +if test_have_prereq PYTHON +then + PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
When neither python3 nor python are available (which is the case in the minimal Git for Windows SDK used in Git's CI runs), this fails under set -e. Before even running the first test case. Resulting in an unexpected TAP format error.
Now, we could "fix" this by imitating what lib-p4 does (see
https://github.com/dscho/git/commit/bd0b5570c744f678911a67a62da63f30655f20d8
which demonstrates that it is indeed a work-around on Windows):
-- snip --
t/t4080-diff-process.sh +3 -2
@@ -4,9 +4,10 @@ test_description='diff process via long-running process' . ./test-lib.sh -if test_have_prereq PYTHON +if ! test_have_prereq PYTHON || ! test -x "$PYTHON_PATH" then - PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python) + skip_all='python interpreter not available' + test_done fi #
-- snap --
Of course, this uncovers _another_ problem with the Python script: It uses Python3-only f"..." format strings, which cannot be handled by the Python2 to which the PYTHON_PATH variable in linux-TEST-vars points. So this requires _another follow-up (see also
https://github.com/dscho/git/commit/c12a9f4c80e5ce8db0fe370fac46fb45be2b775f):
-- snip --
t/t4080-diff-process.sh +5 -3
@@ -39,7 +39,8 @@ setup_backend () { def write_pkt(line): data = (line + "\n").encode() - sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data) + hdr = "{:04x}".format(len(data) + 4).encode() + sys.stdout.buffer.write(hdr + data) sys.stdout.buffer.flush() def write_flush():
@@ -98,7 +99,8 @@ setup_backend () { new = read_content() old_first = old.split(b"\n")[0].decode(errors="replace") if old else "" new_first = new.split(b"\n")[0].decode(errors="replace") if new else "" - log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}") + log("command={} pathname={} old={} new={}".format( + cmd, pathname, old_first, new_first)) if mode == "error": write_flush()
@@ -130,7 +132,7 @@ setup_backend () { else: ol = old.count(b"\n") nl = new.count(b"\n") - write_pkt(f"hunk 1 {ol} 1 {nl}") + write_pkt("hunk 1 {} 1 {}".format(ol, nl)) write_flush() write_pkt("status=success") write_flush()
-- snap --
And this is still not enough to make it work with Python2, see
https://github.com/dscho/git/actions/runs/27091523842/job/79955895737:
-- snip --
[...]
+ git -c diff.cdiff.process=./diff-process-backend --mode=fixed-hunk diff boundary.c Traceback (most recent call last):
File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 45, in <module>
assert read_pkt() == "git-diff-client"
File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 4, in read_pkt
hdr = sys.stdin.buffer.read(4)
AttributeError: 'file' object has no attribute 'buffer'
-- snap --
I have experienced similar patterns in my career, where a single decision required multiple follow-up fixes _just_ to avoid having to revert that decision. This kind of doubling down has never ended well.
Therefore I would like to take a step back, and ask: Is it _really_ a good idea to use Python here? Are we certain that we want to _require_ Python to run this test and skip it if Python isn't available (as is the case in the Windows-related parts of Git's very own CI) even if Python has nothing at all to do with the feature that is being tested?
I don't want to be doomed to repeat history, and we can very well learn e.g. from prior art in this very project, where the tests for the clean/smudge filters (which _also_ want to speak pkt-line over stdio) needlessly incurred Perl as a requirement to run the tests. It was Matheus's heroic work in 52917a998ef3a (t0021: implementation the rot13-filter.pl script in C, 2022-08-14) and 4d1d843be7a15 (tests: use the new C rot13-filter helper to avoid PERL prereq, 2022-08-14) that avoided that unnecessary prerequisite.
Likewise, there is test-tool pkt-line intended for driving the pkt-line protocol via simple shell scripts.
So the conscious project direction has been: fold pkt-line test backends into test-tool and drop the scripting-language prereq. Reintroducing the same shape in Python would walk this back.
Patrick's careful effort in 27bd8ee311719 (Merge branch 'ps/fewer-perl', 2025-04-29) has been another clear sign that the Git project is actively _removing_ scripting-language dependencies from the build and test infrastructure, not adding new ones.
The clear prior art in Git's own tests for what t4080 wants to do, as of today, is t/helper/test-rot13-filter.c, which could be imitated here instead of (re-)introducing a dependency on a scripting language other than Unix shell in Git's test suite.
The PYTHON prereq exists in exactly five files today, all git p4 related (where Python is an intrinsic prerequisite given that git-p4.py _is_ written in Python): t/lib-git-p4.sh, t/t9802-git-p4-filetype.sh, t/t9810-git-p4-rcs.sh, t/t9835-git-p4-metadata-encoding-python2.sh, and t/t9836-git-p4-metadata-encoding-python3.sh.
After 7cdbff14d482 (remove merge-recursive-old, 2006-11-20), this here patch would be the first one, after almost 20 years, to re-introduce Python as a dependency outside git p4.
And it would also be the first ever to embed a Python script as a heredoc:
Quoted 121 lines
t/t4080-diff-process.sh new file +121
-0+fi + +# +# A single parametric diff process. +# Usage: diff-process-backend --mode=<mode> [--log=<path>] +# +# Modes: +# whole-file - report all lines as changed (default) +# fixed-hunk - always report hunk 5 2 5 2 +# bad-hunk - report out-of-bounds hunk 999 1 999 1 +# bad-sync - report hunk with mismatched unchanged totals +# overlap - report two overlapping hunks +# no-hunks - return no hunks (files considered equivalent) +# error - return status=error for every request +# abort - return status=abort for every request +# crash - read one request then exit without responding +# +setup_backend () { + cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF + import sys, os + + def read_pkt(): + hdr = sys.stdin.buffer.read(4) + if len(hdr) < 4: return None + length = int(hdr, 16) + if length == 0: return "" + data = sys.stdin.buffer.read(length - 4) + return data.decode().rstrip("\n") + + def write_pkt(line): + data = (line + "\n").encode() + sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data) + sys.stdout.buffer.flush() + + def write_flush(): + sys.stdout.buffer.write(b"0000") + sys.stdout.buffer.flush() + + def read_content(): + chunks = [] + while True: + hdr = sys.stdin.buffer.read(4) + if len(hdr) < 4: break + length = int(hdr, 16) + if length == 0: break + chunks.append(sys.stdin.buffer.read(length - 4)) + return b"".join(chunks) + + mode = "whole-file" + logfile = None + for arg in sys.argv[1:]: + if arg.startswith("--mode="): + mode = arg[7:] + elif arg.startswith("--log="): + logfile = open(arg[6:], "a") + + def log(msg): + if logfile: + logfile.write(msg + "\n") + logfile.flush() + + # Handshake + assert read_pkt() == "git-diff-client" + assert read_pkt() == "version=1" + read_pkt() + write_pkt("git-diff-server") + write_pkt("version=1") + write_flush() + while True: + p = read_pkt() + if p == "": break + write_pkt("capability=hunks") + write_flush() + + log("ready") + + while True: + cmd = None + pathname = None + while True: + p = read_pkt() + if p is None: sys.exit(0) + if p == "": break + if p.startswith("command="): cmd = p.split("=",1)[1] + if p.startswith("pathname="): pathname = p.split("=",1)[1] + if cmd is None: sys.exit(0) + old = read_content() + new = read_content() + old_first = old.split(b"\n")[0].decode(errors="replace") if old else "" + new_first = new.split(b"\n")[0].decode(errors="replace") if new else "" + log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}") + + if mode == "error": + write_flush() + write_pkt("status=error") + write_flush() + continue + + if mode == "abort": + write_flush() + write_pkt("status=abort") + write_flush() + continue + + if mode == "crash": + sys.exit(1) + + if cmd == "hunks": + if mode == "fixed-hunk": + write_pkt("hunk 5 2 5 2") + elif mode == "bad-hunk": + write_pkt("hunk 999 1 999 1") + elif mode == "bad-sync": + write_pkt("hunk 1 2 1 1") + elif mode == "overlap": + write_pkt("hunk 1 5 1 5") + write_pkt("hunk 3 2 3 2") + elif mode == "no-hunks": + pass + else: + ol = old.count(b"\n")
t/t4080-diff-process.sh line 140
Quoted 10 lines
t/t4080-diff-process.sh new file +10
-0+ nl = new.count(b"\n") + write_pkt(f"hunk 1 {ol} 1 {nl}") + write_flush() + write_pkt("status=success") + write_flush() + else: + write_flush() + write_pkt("status=error") + write_flush() + PYEOF
The existing pattern is to provide larger scripts as fixtures in associated t/tNNNN/ directories, not as heredoc, see e.g. t/t1509/prepare-chroot.sh. Writing scripts, especially lengthy ones, in heredoc strings makes it virtually impossible to use static code analysis or syntax highlighting to fend off banal errors.
Given the complexity of what t4080 tries to test (error, abort, crash, bad-sync, no-hunks, multiple files in one session, capability negotiation), it would unfortunately be infeasible to use test-tool pkt-line from a shell script implementing that diff.*.process protocol.
So I've spiked a demo how the test-tool diff-process-backend could look like (letting Opus do the menial typing, so that I can enjoy at least part of a sunny Sunday outside), which also passes the CI build and test:
https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6
That commit is of course not intended to be used as-is; Feel free to pick code parts of it and integrate them into your topic branch. Or write your own test-tool helper from scratch if that's more your jam.
Ciao,
Johannes
Quoted 424 lines
t/t4080-diff-process.sh new file +398
-0+ write_script diff-process-backend <<-SHEOF + exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@" + SHEOF +} + +BACKEND="./diff-process-backend" + +test_expect_success PYTHON 'setup' ' + setup_backend && + echo "*.c diff=cdiff" >.gitattributes && + git add .gitattributes && + + # boundary.c: 10 lines, changes at 5-6 and 9-10. + # Used by: hunk boundaries, error fallback, crash, bad hunks, overlap. + cat >boundary.c <<-\EOF && + line1 + line2 + line3 + line4 + OLD5 + OLD6 + line7 + line8 + OLD9 + OLD10 + EOF + git add boundary.c && + + # worddiff.c: single-line function, value changes 1 -> 999. + # Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat. + cat >worddiff.c <<-\EOF && + int value(void) { return 1; } + EOF + git add worddiff.c && + + # newfile.c: single-line function, value changes 42 -> 99. + # Used by: new file, --exit-code, multiple drivers. + cat >newfile.c <<-\EOF && + int new_func(void) { return 42; } + EOF + git add newfile.c && + + # logtest.c: single-line function for log/format-patch tests. + # Needs two commits so log -1 has a diff. + cat >logtest.c <<-\EOF && + int logfunc(void) { return 1; } + EOF + git add logtest.c && + + # two.c/one.c: two-file pair for error/abort/startup-failure tests. + cat >one.c <<-\EOF && + int first(void) { return 1; } + EOF + cat >two.c <<-\EOF && + int second(void) { return 2; } + EOF + git add one.c two.c && + + git commit -m "initial" && + + # Second commit for logtest.c (so log -1 has something to show). + cat >logtest.c <<-\EOF && + int logfunc(void) { return 2; } + EOF + git add logtest.c && + git commit -m "change logtest.c" && + + # Working tree modifications (not committed). + cat >boundary.c <<-\EOF && + line1 + line2 + line3 + line4 + NEW5 + NEW6 + line7 + line8 + NEW9 + NEW10 + EOF + + cat >worddiff.c <<-\EOF && + int value(void) { return 999; } + EOF + + cat >newfile.c <<-\EOF && + int new_func(void) { return 99; } + EOF + + cat >one.c <<-\EOF && + int first(void) { return 10; } + EOF + + cat >two.c <<-\EOF + int second(void) { return 20; } + EOF +' + +# +# Core behavior: the tool controls which lines are marked as changed. +# + +test_expect_success PYTHON 'diff process hunk boundaries affect output' ' + # The file has changes at lines 5-6 and 9-10, but fixed-hunk + # only reports lines 5-6 as changed. Lines 9-10 should not + # appear as changed in the output. + git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \ + diff boundary.c >actual && + test_grep "^-OLD5" actual && + test_grep "^-OLD6" actual && + test_grep "^+NEW5" actual && + test_grep "^+NEW6" actual && + test_grep ! "^-OLD9" actual && + test_grep ! "^-OLD10" actual && + test_grep ! "^+NEW9" actual && + test_grep ! "^+NEW10" actual +' + +test_expect_success PYTHON 'diff process works with new file' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- newfile.c >actual 2>stderr && + test_grep "return 99" actual && + test_grep "pathname=newfile.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process works with added file (empty old side)' ' + cat >added.c <<-\EOF && + int added(void) { return 1; } + EOF + git add added.c && + + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --cached -- added.c >actual 2>stderr && + test_grep "added" actual && + test_grep "pathname=added.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process skipped for binary files' ' + printf "\\0binary" >binary.c && + git add binary.c && + git commit -m "add binary" && + printf "\\0changed" >binary.c && + + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- binary.c >actual && + test_grep "Binary files" actual && + test_path_is_missing backend.log +' + +test_expect_success PYTHON 'diff process not consulted for unmatched driver' ' + echo "not tracked by cdiff" >unmatched.txt && + git add unmatched.txt && + git commit -m "add unmatched.txt" && + + echo "modified" >unmatched.txt && + + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- unmatched.txt >actual && + test_grep "modified" actual && + test_path_is_missing backend.log +' + +test_expect_success PYTHON 'multiple drivers use separate processes' ' + echo "*.h diff=hdiff" >>.gitattributes && + git add .gitattributes && + + cat >multi.h <<-\EOF && + int header(void) { return 1; } + EOF + git add multi.h && + git commit -m "add multi.h" && + + cat >multi.h <<-\EOF && + int header(void) { return 2; } + EOF + + rm -f backend-c.log backend-h.log && + git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \ + -c diff.hdiff.process="$BACKEND --log=backend-h.log" \ + diff -- newfile.c multi.h >actual 2>stderr && + test_grep "pathname=newfile.c" backend-c.log && + test_grep "pathname=multi.h" backend-h.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process works alongside textconv' ' + write_script uppercase-filter <<-\EOF && + tr "a-z" "A-Z" <"$1" + EOF + + cat >textconv.c <<-\EOF && + hello world + EOF + git add textconv.c && + git commit -m "add textconv.c" && + + cat >textconv.c <<-\EOF && + goodbye world + EOF + + rm -f backend.log && + git -c diff.cdiff.textconv="./uppercase-filter" \ + -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- textconv.c >actual 2>stderr && + # The diff process receives textconv-transformed (uppercase) content. + test_grep "pathname=textconv.c" backend.log && + test_grep "old=HELLO WORLD" backend.log && + test_grep "new=GOODBYE WORLD" backend.log && + test_must_be_empty stderr +' + +# +# Downstream features: word diff, log, equivalent files, exit code. +# + +test_expect_success PYTHON 'diff process with --word-diff' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --word-diff worddiff.c >actual 2>stderr && + test_grep "\[-1;-\]" actual && + test_grep "{+999;+}" actual && + test_grep "pathname=worddiff.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process works with git log -p' ' + # With no-hunks mode, the tool says the files are equivalent, + # so log -p should show the commit but no diff content. + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \ + log -1 -p -- logtest.c >actual 2>stderr && + test_grep "change logtest.c" actual && + test_grep ! "return 2" actual && + test_grep "command=hunks pathname=logtest.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process no hunks suppresses diff output' ' + cat >nohunks.c <<-\EOF && + int zero(void) { return 0; } + EOF + git add nohunks.c && + git commit -m "add nohunks.c" && + + cat >nohunks.c <<-\EOF && + int zero(void) { return 999; } + EOF + + git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \ + diff nohunks.c >actual && + test_must_be_empty actual +' + +test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' ' + git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \ + diff --exit-code nohunks.c +' + +test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' ' + test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \ + diff --exit-code newfile.c +' + +# +# Bypass mechanisms: flags and commands that skip the diff process. +# + +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --diff-algorithm=patience worddiff.c >actual && + test_grep "return 999" actual && + test_path_is_missing backend.log +' + +test_expect_success PYTHON 'diff process not used by --stat' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --stat worddiff.c >actual && + test_grep "worddiff.c" actual && + test_path_is_missing backend.log +' + +# +# Error handling and fallback. +# + +test_expect_success PYTHON 'diff process fallback on tool error status' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \ + diff boundary.c >actual 2>stderr && + # Fallback produces the full builtin diff (both change regions). + test_grep "^-OLD5" actual && + test_grep "^+NEW5" actual && + test_grep "^-OLD9" actual && + test_grep "^+NEW9" actual && + # Tool was contacted (it replied with error, not crash). + test_grep "command=hunks pathname=boundary.c" backend.log && + test_grep "diff process.*failed" stderr +' + +test_expect_success PYTHON 'diff process error keeps tool available for next file' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \ + diff -- one.c two.c >actual 2>stderr && + # Unlike abort, error keeps the tool available: both files + # are sent to the tool (and both fall back). + test_grep "pathname=one.c" backend.log && + test_grep "pathname=two.c" backend.log && + test_grep "return 10" actual && + test_grep "return 20" actual +' + +test_expect_success PYTHON 'diff process abort disables for session' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \ + diff -- one.c two.c >actual && + # Both files should still produce diff output via fallback. + test_grep "return 10" actual && + test_grep "return 20" actual && + # The tool aborts on the first file and git clears its + # capability. The second file never contacts the tool. + test_grep "pathname=one.c" backend.log && + test_grep ! "pathname=two.c" backend.log +' + +test_expect_success PYTHON 'diff process fallback on tool crash' ' + git -c diff.cdiff.process="$BACKEND --mode=crash" \ + diff boundary.c >actual 2>stderr && + test_grep "^-OLD5" actual && + test_grep "^+NEW5" actual && + test_grep "^-OLD9" actual && + test_grep "^+NEW9" actual && + # Crash is a communication failure, so a warning is emitted. + test_grep "diff process.*failed" stderr +' + +test_expect_success PYTHON 'diff process startup failure only warns once' ' + git -c diff.cdiff.process="/nonexistent/tool" \ + diff -- one.c two.c >actual 2>stderr && + # Both files produce diff output via fallback. + test_grep "return 10" actual && + test_grep "return 20" actual && + # Sentinel prevents repeated warnings: only one, not one per file. + test_grep "diff process.*failed" stderr >warnings && + test_line_count = 1 warnings +' + +test_expect_success PYTHON 'diff process fallback on bad hunks' ' + git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \ + diff boundary.c >actual 2>stderr && + test_grep "^-OLD5" actual && + test_grep "^+NEW5" actual && + test_grep "^-OLD9" actual && + test_grep "^+NEW9" actual && + # Invalid hunks are caught by xdiff validation, not the + # protocol layer, so no warning is emitted. + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' ' + cat >synctest.c <<-\EOF && + line1 + line2 + line3 + EOF + git add synctest.c && + git commit -m "add synctest.c" && + + cat >synctest.c <<-\EOF && + line1 + changed + line3 + EOF + + # bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new + # line as changed, leaving 1 unchanged old vs 2 unchanged new. + # The synchronization invariant fails and git falls back. + git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \ + diff synctest.c >actual 2>stderr && + test_grep "changed" actual +' + +test_expect_success PYTHON 'diff process fallback on overlapping hunks' ' + # boundary.c has 10 lines, so both hunks are in bounds + # but they overlap at lines 3-5, triggering the ordering check. + git -c diff.cdiff.process="$BACKEND --mode=overlap" \ + diff boundary.c >actual 2>stderr && + test_grep "NEW5" actual +' + +test_doneuserdiff.h +3
-0@@ -3,6 +3,7 @@#include "notes-cache.h" +struct diff_subprocess ;struct index_state ;struct repository ;@@ -33,6 +34,8 @@ struct userdiff_driver {int textconv_want_cache;const char *process;char *process_owned; +struct diff_subprocess *diff_subprocess; + unsigned diff_process_failed : 1; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,Signature · 1 line
gitgitgadget
Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process
Quoted 302 lines · Johannes Schindelin
Hi Michael,
I stumbled about this patch when it broke CI in Git for Windows, where we do _not_ use
NO_PYTHON, even though Python is unavailable in the build/test CI jobs. The existing tests handle this situation gracefully, this here patch does not:Quoted 6 lines · Michael Montalbo via GitGitGadget
t/t4080-diff-process.sh new file +0
-0t/t4080-diff-process.sh line 9
Quoted 9 lines
t/t4080-diff-process.sh new file +9
-0@@ -0,0 +1,538 @@ +#!/bin/sh + +test_description='diff process via long-running process' + +. ./test-lib.sh + +if test_have_prereq PYTHON +then + PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)When neither
python3norpythonare available (which is the case in the minimal Git for Windows SDK used in Git's CI runs), this fails underset -e. Before even running the first test case. Resulting in an unexpected TAP format error.Now, we could "fix" this by imitating what
lib-p4does (see
https://github.com/dscho/git/commit/bd0b5570c744f678911a67a62da63f30655f20d8
which demonstrates that it is indeed a work-around on Windows):-- snip --
t/t4080-diff-process.sh +3
-3@@ -4,9 +4,10 @@ test_description='diff process via long-running process' . ./test-lib.sh -if test_have_prereq PYTHON +if ! test_have_prereq PYTHON || ! test -x "$PYTHON_PATH" then - PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python) + skip_all='python interpreter not available' + test_done fi # -- snap --Of course, this uncovers _another_ problem with the Python script: It uses Python3-only
f"..."format strings, which cannot be handled by the Python2 to which thePYTHON_PATHvariable inlinux-TEST-varspoints. So this requires_anotherfollow-up (see also
https://github.com/dscho/git/commit/c12a9f4c80e5ce8db0fe370fac46fb45be2b775f):-- snip --
t/t4080-diff-process.sh +5
-4@@ -39,7 +39,8 @@ setup_backend () { def write_pkt(line): data = (line + "\n").encode() - sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data) + hdr = "{:04x}".format(len(data) + 4).encode() + sys.stdout.buffer.write(hdr + data) sys.stdout.buffer.flush() def write_flush():@@ -98,7 +99,8 @@ setup_backend () { new = read_content() old_first = old.split(b"\n")[0].decode(errors="replace") if old else "" new_first = new.split(b"\n")[0].decode(errors="replace") if new else "" - log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}") + log("command={} pathname={} old={} new={}".format( + cmd, pathname, old_first, new_first)) if mode == "error": write_flush()@@ -130,7 +132,7 @@ setup_backend () { else: ol = old.count(b"\n") nl = new.count(b"\n") - write_pkt(f"hunk 1 {ol} 1 {nl}") + write_pkt("hunk 1 {} 1 {}".format(ol, nl)) write_flush() write_pkt("status=success") write_flush() -- snap --And this is still not enough to make it work with Python2, see
https://github.com/dscho/git/actions/runs/27091523842/job/79955895737:-- snip --
[...]
+ git -c diff.cdiff.process=./diff-process-backend --mode=fixed-hunk diffboundary.cTraceback (most recent call last):
File "/__w/git/git/t/trashdirectory.t4080-diff-process/diff-process-backend.py", line 45, in <module>
assertread_pkt()== "git-diff-client"
File "/__w/git/git/t/trashdirectory.t4080-diff-process/diff-process-backend.py", line 4, inread_pkt
hdr =sys.stdin.buffer.read(4)
AttributeError: 'file' object has no attribute 'buffer'
-- snap --I have experienced similar patterns in my career, where a single decision required multiple follow-up fixes _just_ to avoid having to revert that decision. This kind of doubling down has never ended well.
Therefore I would like to take a step back, and ask: Is it _really_ a good idea to use Python here? Are we certain that we want to _require_ Python to run this test and skip it if Python isn't available (as is the case in the Windows-related parts of Git's very own CI) even if Python has nothing at all to do with the feature that is being tested?
I don't want to be doomed to repeat history, and we can very well learn e.g. from prior art in this very project, where the tests for the clean/smudge filters (which _also_ want to speak pkt-line over stdio) needlessly incurred Perl as a requirement to run the tests. It was Matheus's heroic work in
52917a998ef3a(t0021: implementation the rot13-filter.pl script in C, 2022-08-14) and4d1d843be7a15(tests: use the new C rot13-filter helper to avoid PERL prereq, 2022-08-14) that avoided that unnecessary prerequisite.Likewise, there is
test-tool pkt-lineintended for driving the pkt-line protocol via simple shell scripts.So the conscious project direction has been: fold pkt-line test backends into
test-tooland drop the scripting-language prereq. Reintroducing the same shape in Python would walk this back.Patrick's careful effort in
27bd8ee311719(Merge branch 'ps/fewer-perl', 2025-04-29) has been another clear sign that the Git project is actively _removing_ scripting-language dependencies from the build and test infrastructure, not adding new ones.The clear prior art in Git's own tests for what t4080 wants to do, as of today, is
t/helper/test-rot13-filter.c, which could be imitated here instead of (re-)introducing a dependency on a scripting language other than Unix shell in Git's test suite.The
PYTHONprereq exists in exactly five files today, allgit p4related (where Python is an intrinsic prerequisite given thatgit-p4.py_is_ written in Python):t/lib-git-p4.sh,t/t9802-git-p4-filetype.sh,t/t9810-git-p4-rcs.sh,t/t9835-git-p4-metadata-encoding-python2.sh, andt/t9836-git-p4-metadata-encoding-python3.sh.After
7cdbff14d482(remove merge-recursive-old, 2006-11-20), this here patch would be the first one, after almost 20 years, to re-introduce Python as a dependency outsidegit p4.And it would also be the first ever to embed a Python script as a heredoc:
Quoted 121 lines
t/t4080-diff-process.sh new file +121
-0+fi + +# +# A single parametric diff process. +# Usage: diff-process-backend --mode=<mode> [--log=<path>] +# +# Modes: +# whole-file - report all lines as changed (default) +# fixed-hunk - always report hunk 5 2 5 2 +# bad-hunk - report out-of-bounds hunk 999 1 999 1 +# bad-sync - report hunk with mismatched unchanged totals +# overlap - report two overlapping hunks +# no-hunks - return no hunks (files considered equivalent) +# error - return status=error for every request +# abort - return status=abort for every request +# crash - read one request then exit without responding +# +setup_backend () { + cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF + import sys, os + + def read_pkt(): + hdr = sys.stdin.buffer.read(4) + if len(hdr) < 4: return None + length = int(hdr, 16) + if length == 0: return "" + data = sys.stdin.buffer.read(length - 4) + return data.decode().rstrip("\n") + + def write_pkt(line): + data = (line + "\n").encode() + sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data) + sys.stdout.buffer.flush() + + def write_flush(): + sys.stdout.buffer.write(b"0000") + sys.stdout.buffer.flush() + + def read_content(): + chunks = [] + while True: + hdr = sys.stdin.buffer.read(4) + if len(hdr) < 4: break + length = int(hdr, 16) + if length == 0: break + chunks.append(sys.stdin.buffer.read(length - 4)) + return b"".join(chunks) + + mode = "whole-file" + logfile = None + for arg in sys.argv[1:]: + if arg.startswith("--mode="): + mode = arg[7:] + elif arg.startswith("--log="): + logfile = open(arg[6:], "a") + + def log(msg): + if logfile: + logfile.write(msg + "\n") + logfile.flush() + + # Handshake + assert read_pkt() == "git-diff-client" + assert read_pkt() == "version=1" + read_pkt() + write_pkt("git-diff-server") + write_pkt("version=1") + write_flush() + while True: + p = read_pkt() + if p == "": break + write_pkt("capability=hunks") + write_flush() + + log("ready") + + while True: + cmd = None + pathname = None + while True: + p = read_pkt() + if p is None: sys.exit(0) + if p == "": break + if p.startswith("command="): cmd = p.split("=",1)[1] + if p.startswith("pathname="): pathname = p.split("=",1)[1] + if cmd is None: sys.exit(0) + old = read_content() + new = read_content() + old_first = old.split(b"\n")[0].decode(errors="replace") if old else "" + new_first = new.split(b"\n")[0].decode(errors="replace") if new else "" + log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}") + + if mode == "error": + write_flush() + write_pkt("status=error") + write_flush() + continue + + if mode == "abort": + write_flush() + write_pkt("status=abort") + write_flush() + continue + + if mode == "crash": + sys.exit(1) + + if cmd == "hunks": + if mode == "fixed-hunk": + write_pkt("hunk 5 2 5 2") + elif mode == "bad-hunk": + write_pkt("hunk 999 1 999 1") + elif mode == "bad-sync": + write_pkt("hunk 1 2 1 1") + elif mode == "overlap": + write_pkt("hunk 1 5 1 5") + write_pkt("hunk 3 2 3 2") + elif mode == "no-hunks": + pass + else: + ol = old.count(b"\n")t/t4080-diff-process.sh line 140
Quoted 10 lines
t/t4080-diff-process.sh new file +10
-0+ nl = new.count(b"\n") + write_pkt(f"hunk 1 {ol} 1 {nl}") + write_flush() + write_pkt("status=success") + write_flush() + else: + write_flush() + write_pkt("status=error") + write_flush() + PYEOFThe existing pattern is to provide larger scripts as fixtures in associated
t/tNNNN/directories, not as heredoc, see e.g.t/t1509/prepare-chroot.sh. Writing scripts, especially lengthy ones, in heredoc strings makes it virtually impossible to use static code analysis or syntax highlighting to fend off banal errors.Given the complexity of what t4080 tries to test (error, abort, crash, bad-sync, no-hunks, multiple files in one session, capability negotiation), it would unfortunately be infeasible to use
test-tool pkt-linefrom a shell script implementing thatdiff.*.processprotocol.
Quoted 9 lines
So I've spiked a demo how the
test-tool diff-process-backendcould look like (letting Opus do the menial typing, so that I can enjoy at least part of a sunny Sunday outside), which also passes the CI build and test:
https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6That commit is of course not intended to be used as-is; Feel free to pick code parts of it and integrate them into your topic branch. Or write your own test-tool helper from scratch if that's more your jam.
Johannes, thank you for the great feedback. The historical context is really helpful and
the concerns you raise make a lot of sense. I will take a look at your spike and also work
on removing Python from the test.
Quoted 427 lines
Ciao,
JohannesQuoted 424 lines
t/t4080-diff-process.sh new file +398
-0+ write_script diff-process-backend <<-SHEOF + exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@" + SHEOF +} + +BACKEND="./diff-process-backend" + +test_expect_success PYTHON 'setup' ' + setup_backend && + echo "*.c diff=cdiff" >.gitattributes && + git add .gitattributes && + + # boundary.c: 10 lines, changes at 5-6 and 9-10. + # Used by: hunk boundaries, error fallback, crash, bad hunks, overlap. + cat >boundary.c <<-\EOF && + line1 + line2 + line3 + line4 + OLD5 + OLD6 + line7 + line8 + OLD9 + OLD10 + EOF + git add boundary.c && + + # worddiff.c: single-line function, value changes 1 -> 999. + # Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat. + cat >worddiff.c <<-\EOF && + int value(void) { return 1; } + EOF + git add worddiff.c && + + # newfile.c: single-line function, value changes 42 -> 99. + # Used by: new file, --exit-code, multiple drivers. + cat >newfile.c <<-\EOF && + int new_func(void) { return 42; } + EOF + git add newfile.c && + + # logtest.c: single-line function for log/format-patch tests. + # Needs two commits so log -1 has a diff. + cat >logtest.c <<-\EOF && + int logfunc(void) { return 1; } + EOF + git add logtest.c && + + # two.c/one.c: two-file pair for error/abort/startup-failure tests. + cat >one.c <<-\EOF && + int first(void) { return 1; } + EOF + cat >two.c <<-\EOF && + int second(void) { return 2; } + EOF + git add one.c two.c && + + git commit -m "initial" && + + # Second commit for logtest.c (so log -1 has something to show). + cat >logtest.c <<-\EOF && + int logfunc(void) { return 2; } + EOF + git add logtest.c && + git commit -m "change logtest.c" && + + # Working tree modifications (not committed). + cat >boundary.c <<-\EOF && + line1 + line2 + line3 + line4 + NEW5 + NEW6 + line7 + line8 + NEW9 + NEW10 + EOF + + cat >worddiff.c <<-\EOF && + int value(void) { return 999; } + EOF + + cat >newfile.c <<-\EOF && + int new_func(void) { return 99; } + EOF + + cat >one.c <<-\EOF && + int first(void) { return 10; } + EOF + + cat >two.c <<-\EOF + int second(void) { return 20; } + EOF +' + +# +# Core behavior: the tool controls which lines are marked as changed. +# + +test_expect_success PYTHON 'diff process hunk boundaries affect output' ' + # The file has changes at lines 5-6 and 9-10, but fixed-hunk + # only reports lines 5-6 as changed. Lines 9-10 should not + # appear as changed in the output. + git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \ + diff boundary.c >actual && + test_grep "^-OLD5" actual && + test_grep "^-OLD6" actual && + test_grep "^+NEW5" actual && + test_grep "^+NEW6" actual && + test_grep ! "^-OLD9" actual && + test_grep ! "^-OLD10" actual && + test_grep ! "^+NEW9" actual && + test_grep ! "^+NEW10" actual +' + +test_expect_success PYTHON 'diff process works with new file' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- newfile.c >actual 2>stderr && + test_grep "return 99" actual && + test_grep "pathname=newfile.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process works with added file (empty old side)' ' + cat >added.c <<-\EOF && + int added(void) { return 1; } + EOF + git add added.c && + + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --cached -- added.c >actual 2>stderr && + test_grep "added" actual && + test_grep "pathname=added.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process skipped for binary files' ' + printf "\\0binary" >binary.c && + git add binary.c && + git commit -m "add binary" && + printf "\\0changed" >binary.c && + + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- binary.c >actual && + test_grep "Binary files" actual && + test_path_is_missing backend.log +' + +test_expect_success PYTHON 'diff process not consulted for unmatched driver' ' + echo "not tracked by cdiff" >unmatched.txt && + git add unmatched.txt && + git commit -m "add unmatched.txt" && + + echo "modified" >unmatched.txt && + + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- unmatched.txt >actual && + test_grep "modified" actual && + test_path_is_missing backend.log +' + +test_expect_success PYTHON 'multiple drivers use separate processes' ' + echo "*.h diff=hdiff" >>.gitattributes && + git add .gitattributes && + + cat >multi.h <<-\EOF && + int header(void) { return 1; } + EOF + git add multi.h && + git commit -m "add multi.h" && + + cat >multi.h <<-\EOF && + int header(void) { return 2; } + EOF + + rm -f backend-c.log backend-h.log && + git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \ + -c diff.hdiff.process="$BACKEND --log=backend-h.log" \ + diff -- newfile.c multi.h >actual 2>stderr && + test_grep "pathname=newfile.c" backend-c.log && + test_grep "pathname=multi.h" backend-h.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process works alongside textconv' ' + write_script uppercase-filter <<-\EOF && + tr "a-z" "A-Z" <"$1" + EOF + + cat >textconv.c <<-\EOF && + hello world + EOF + git add textconv.c && + git commit -m "add textconv.c" && + + cat >textconv.c <<-\EOF && + goodbye world + EOF + + rm -f backend.log && + git -c diff.cdiff.textconv="./uppercase-filter" \ + -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- textconv.c >actual 2>stderr && + # The diff process receives textconv-transformed (uppercase) content. + test_grep "pathname=textconv.c" backend.log && + test_grep "old=HELLO WORLD" backend.log && + test_grep "new=GOODBYE WORLD" backend.log && + test_must_be_empty stderr +' + +# +# Downstream features: word diff, log, equivalent files, exit code. +# + +test_expect_success PYTHON 'diff process with --word-diff' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --word-diff worddiff.c >actual 2>stderr && + test_grep "\[-1;-\]" actual && + test_grep "{+999;+}" actual && + test_grep "pathname=worddiff.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process works with git log -p' ' + # With no-hunks mode, the tool says the files are equivalent, + # so log -p should show the commit but no diff content. + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \ + log -1 -p -- logtest.c >actual 2>stderr && + test_grep "change logtest.c" actual && + test_grep ! "return 2" actual && + test_grep "command=hunks pathname=logtest.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process no hunks suppresses diff output' ' + cat >nohunks.c <<-\EOF && + int zero(void) { return 0; } + EOF + git add nohunks.c && + git commit -m "add nohunks.c" && + + cat >nohunks.c <<-\EOF && + int zero(void) { return 999; } + EOF + + git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \ + diff nohunks.c >actual && + test_must_be_empty actual +' + +test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' ' + git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \ + diff --exit-code nohunks.c +' + +test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' ' + test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \ + diff --exit-code newfile.c +' + +# +# Bypass mechanisms: flags and commands that skip the diff process. +# + +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --diff-algorithm=patience worddiff.c >actual && + test_grep "return 999" actual && + test_path_is_missing backend.log +' + +test_expect_success PYTHON 'diff process not used by --stat' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --stat worddiff.c >actual && + test_grep "worddiff.c" actual && + test_path_is_missing backend.log +' + +# +# Error handling and fallback. +# + +test_expect_success PYTHON 'diff process fallback on tool error status' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \ + diff boundary.c >actual 2>stderr && + # Fallback produces the full builtin diff (both change regions). + test_grep "^-OLD5" actual && + test_grep "^+NEW5" actual && + test_grep "^-OLD9" actual && + test_grep "^+NEW9" actual && + # Tool was contacted (it replied with error, not crash). + test_grep "command=hunks pathname=boundary.c" backend.log && + test_grep "diff process.*failed" stderr +' + +test_expect_success PYTHON 'diff process error keeps tool available for next file' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \ + diff -- one.c two.c >actual 2>stderr && + # Unlike abort, error keeps the tool available: both files + # are sent to the tool (and both fall back). + test_grep "pathname=one.c" backend.log && + test_grep "pathname=two.c" backend.log && + test_grep "return 10" actual && + test_grep "return 20" actual +' + +test_expect_success PYTHON 'diff process abort disables for session' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \ + diff -- one.c two.c >actual && + # Both files should still produce diff output via fallback. + test_grep "return 10" actual && + test_grep "return 20" actual && + # The tool aborts on the first file and git clears its + # capability. The second file never contacts the tool. + test_grep "pathname=one.c" backend.log && + test_grep ! "pathname=two.c" backend.log +' + +test_expect_success PYTHON 'diff process fallback on tool crash' ' + git -c diff.cdiff.process="$BACKEND --mode=crash" \ + diff boundary.c >actual 2>stderr && + test_grep "^-OLD5" actual && + test_grep "^+NEW5" actual && + test_grep "^-OLD9" actual && + test_grep "^+NEW9" actual && + # Crash is a communication failure, so a warning is emitted. + test_grep "diff process.*failed" stderr +' + +test_expect_success PYTHON 'diff process startup failure only warns once' ' + git -c diff.cdiff.process="/nonexistent/tool" \ + diff -- one.c two.c >actual 2>stderr && + # Both files produce diff output via fallback. + test_grep "return 10" actual && + test_grep "return 20" actual && + # Sentinel prevents repeated warnings: only one, not one per file. + test_grep "diff process.*failed" stderr >warnings && + test_line_count = 1 warnings +' + +test_expect_success PYTHON 'diff process fallback on bad hunks' ' + git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \ + diff boundary.c >actual 2>stderr && + test_grep "^-OLD5" actual && + test_grep "^+NEW5" actual && + test_grep "^-OLD9" actual && + test_grep "^+NEW9" actual && + # Invalid hunks are caught by xdiff validation, not the + # protocol layer, so no warning is emitted. + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' ' + cat >synctest.c <<-\EOF && + line1 + line2 + line3 + EOF + git add synctest.c && + git commit -m "add synctest.c" && + + cat >synctest.c <<-\EOF && + line1 + changed + line3 + EOF + + # bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new + # line as changed, leaving 1 unchanged old vs 2 unchanged new. + # The synchronization invariant fails and git falls back. + git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \ + diff synctest.c >actual 2>stderr && + test_grep "changed" actual +' + +test_expect_success PYTHON 'diff process fallback on overlapping hunks' ' + # boundary.c has 10 lines, so both hunks are in bounds + # but they overlap at lines 3-5, triggering the ordering check. + git -c diff.cdiff.process="$BACKEND --mode=overlap" \ + diff boundary.c >actual 2>stderr && + test_grep "NEW5" actual +' + +test_doneuserdiff.h +3
-1@@ -3,6 +3,7 @@#include "notes-cache.h" +struct diff_subprocess ;struct index_state ;struct repository ;@@ -33,6 +34,8 @@ struct userdiff_driver {int textconv_want_cache;const char *process;char *process_owned; +struct diff_subprocess *diff_subprocess; + unsigned diff_process_failed : 1; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 , --gitgitgadget
Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process
Quoted 25 lines · Michael Montalbo
Quoted 13 lines · Johannes Schindelin
Hi Michael,
I stumbled about this patch when it broke CI in Git for Windows, where we do _not_ use
NO_PYTHON, even though Python is unavailable in the build/test CI jobs. The existing tests handle this situation gracefully, this here patch does not:
...
Given the complexity of what t4080 tries to test (error, abort, crash, bad-sync, no-hunks, multiple files in one session, capability negotiation), it would unfortunately be infeasible to usetest-tool pkt-linefrom a shell script implementing thatdiff.*.processprotocol.Quoted 9 lines
So I've spiked a demo how the
test-tool diff-process-backendcould look like (letting Opus do the menial typing, so that I can enjoy at least part of a sunny Sunday outside), which also passes the CI build and test:
https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6That commit is of course not intended to be used as-is; Feel free to pick code parts of it and integrate them into your topic branch. Or write your own test-tool helper from scratch if that's more your jam.
Quoted 5 lines
Johannes, thank you for the great feedback. The historical context is really helpful and
the concerns you raise make a lot of sense. I will take a look at your spike and also work
on removing Python from the test.
Another request.
Please do not force readers to scroll through a ~800 line message just to read only 5 lines of response from you. Keep relevant parts of the message you are responding to in your message to help readers understand the context in which your response was made, but trim everything else that is not relevant from your quote.
Thanks.
Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process
In reply to Johannes SchindelinQuoted 133 lines · Johannes Schindelin
Hi Michael,
I stumbled about this patch when it broke CI in Git for Windows, where we do _not_ use
NO_PYTHON, even though Python is unavailable in the build/test CI jobs. The existing tests handle this situation gracefully, this here patch does not:Quoted 6 lines · Michael Montalbo via GitGitGadget
t/t4080-diff-process.sh new file +0
-0t/t4080-diff-process.sh line 9
Quoted 9 lines
t/t4080-diff-process.sh new file +9
-0@@ -0,0 +1,538 @@ +#!/bin/sh + +test_description='diff process via long-running process' + +. ./test-lib.sh + +if test_have_prereq PYTHON +then + PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)When neither
python3norpythonare available (which is the case in the minimal Git for Windows SDK used in Git's CI runs), this fails underset -e. Before even running the first test case. Resulting in an unexpected TAP format error.Now, we could "fix" this by imitating what
lib-p4does (see
https://github.com/dscho/git/commit/bd0b5570c744f678911a67a62da63f30655f20d8
which demonstrates that it is indeed a work-around on Windows):-- snip --
t/t4080-diff-process.sh +3
-3@@ -4,9 +4,10 @@ test_description='diff process via long-running process' . ./test-lib.sh -if test_have_prereq PYTHON +if ! test_have_prereq PYTHON || ! test -x "$PYTHON_PATH" then - PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python) + skip_all='python interpreter not available' + test_done fi # -- snap --Of course, this uncovers _another_ problem with the Python script: It uses Python3-only
f"..."format strings, which cannot be handled by the Python2 to which thePYTHON_PATHvariable inlinux-TEST-varspoints. So this requires_anotherfollow-up (see also
https://github.com/dscho/git/commit/c12a9f4c80e5ce8db0fe370fac46fb45be2b775f):-- snip --
t/t4080-diff-process.sh +5
-4@@ -39,7 +39,8 @@ setup_backend () { def write_pkt(line): data = (line + "\n").encode() - sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data) + hdr = "{:04x}".format(len(data) + 4).encode() + sys.stdout.buffer.write(hdr + data) sys.stdout.buffer.flush() def write_flush():@@ -98,7 +99,8 @@ setup_backend () { new = read_content() old_first = old.split(b"\n")[0].decode(errors="replace") if old else "" new_first = new.split(b"\n")[0].decode(errors="replace") if new else "" - log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}") + log("command={} pathname={} old={} new={}".format( + cmd, pathname, old_first, new_first)) if mode == "error": write_flush()@@ -130,7 +132,7 @@ setup_backend () { else: ol = old.count(b"\n") nl = new.count(b"\n") - write_pkt(f"hunk 1 {ol} 1 {nl}") + write_pkt("hunk 1 {} 1 {}".format(ol, nl)) write_flush() write_pkt("status=success") write_flush() -- snap --And this is still not enough to make it work with Python2, see
https://github.com/dscho/git/actions/runs/27091523842/job/79955895737:-- snip --
[...]
+ git -c diff.cdiff.process=./diff-process-backend --mode=fixed-hunk diffboundary.cTraceback (most recent call last):
File "/__w/git/git/t/trashdirectory.t4080-diff-process/diff-process-backend.py", line 45, in <module>
assertread_pkt()== "git-diff-client"
File "/__w/git/git/t/trashdirectory.t4080-diff-process/diff-process-backend.py", line 4, inread_pkt
hdr =sys.stdin.buffer.read(4)
AttributeError: 'file' object has no attribute 'buffer'
-- snap --I have experienced similar patterns in my career, where a single decision required multiple follow-up fixes _just_ to avoid having to revert that decision. This kind of doubling down has never ended well.
Therefore I would like to take a step back, and ask: Is it _really_ a good idea to use Python here? Are we certain that we want to _require_ Python to run this test and skip it if Python isn't available (as is the case in the Windows-related parts of Git's very own CI) even if Python has nothing at all to do with the feature that is being tested?
I don't want to be doomed to repeat history, and we can very well learn e.g. from prior art in this very project, where the tests for the clean/smudge filters (which _also_ want to speak pkt-line over stdio) needlessly incurred Perl as a requirement to run the tests. It was Matheus's heroic work in
52917a998ef3a(t0021: implementation the rot13-filter.pl script in C, 2022-08-14) and4d1d843be7a15(tests: use the new C rot13-filter helper to avoid PERL prereq, 2022-08-14) that avoided that unnecessary prerequisite.Likewise, there is
test-tool pkt-lineintended for driving the pkt-line protocol via simple shell scripts.
Quoted 9 lines
So the conscious project direction has been: fold pkt-line test backends into
test-tooland drop the scripting-language prereq. Reintroducing the same shape in Python would walk this back.Patrick's careful effort in
27bd8ee311719(Merge branch 'ps/fewer-perl', 2025-04-29) has been another clear sign that the Git project is actively _removing_ scripting-language dependencies from the build and test infrastructure, not adding new ones.
Now I wonder if the extension / addition of more Perl test infra with my other series:
https://lore.kernel.org/git/[email protected]/T/
also goes against the project direction w.r.t. removing scripting languages. Maybe I should also re-evaluate the usage of Perl there. I am leveraging existing shell parsing logic written in Perl, but maybe the preference for Perl-based lint rules is a mistake and should be avoided.
Quoted 596 lines
The clear prior art in Git's own tests for what t4080 wants to do, as of today, is
t/helper/test-rot13-filter.c, which could be imitated here instead of (re-)introducing a dependency on a scripting language other than Unix shell in Git's test suite.The
PYTHONprereq exists in exactly five files today, allgit p4related (where Python is an intrinsic prerequisite given thatgit-p4.py_is_ written in Python):t/lib-git-p4.sh,t/t9802-git-p4-filetype.sh,t/t9810-git-p4-rcs.sh,t/t9835-git-p4-metadata-encoding-python2.sh, andt/t9836-git-p4-metadata-encoding-python3.sh.After
7cdbff14d482(remove merge-recursive-old, 2006-11-20), this here patch would be the first one, after almost 20 years, to re-introduce Python as a dependency outsidegit p4.And it would also be the first ever to embed a Python script as a heredoc:
Quoted 121 lines
t/t4080-diff-process.sh new file +121
-0+fi + +# +# A single parametric diff process. +# Usage: diff-process-backend --mode=<mode> [--log=<path>] +# +# Modes: +# whole-file - report all lines as changed (default) +# fixed-hunk - always report hunk 5 2 5 2 +# bad-hunk - report out-of-bounds hunk 999 1 999 1 +# bad-sync - report hunk with mismatched unchanged totals +# overlap - report two overlapping hunks +# no-hunks - return no hunks (files considered equivalent) +# error - return status=error for every request +# abort - return status=abort for every request +# crash - read one request then exit without responding +# +setup_backend () { + cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF + import sys, os + + def read_pkt(): + hdr = sys.stdin.buffer.read(4) + if len(hdr) < 4: return None + length = int(hdr, 16) + if length == 0: return "" + data = sys.stdin.buffer.read(length - 4) + return data.decode().rstrip("\n") + + def write_pkt(line): + data = (line + "\n").encode() + sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data) + sys.stdout.buffer.flush() + + def write_flush(): + sys.stdout.buffer.write(b"0000") + sys.stdout.buffer.flush() + + def read_content(): + chunks = [] + while True: + hdr = sys.stdin.buffer.read(4) + if len(hdr) < 4: break + length = int(hdr, 16) + if length == 0: break + chunks.append(sys.stdin.buffer.read(length - 4)) + return b"".join(chunks) + + mode = "whole-file" + logfile = None + for arg in sys.argv[1:]: + if arg.startswith("--mode="): + mode = arg[7:] + elif arg.startswith("--log="): + logfile = open(arg[6:], "a") + + def log(msg): + if logfile: + logfile.write(msg + "\n") + logfile.flush() + + # Handshake + assert read_pkt() == "git-diff-client" + assert read_pkt() == "version=1" + read_pkt() + write_pkt("git-diff-server") + write_pkt("version=1") + write_flush() + while True: + p = read_pkt() + if p == "": break + write_pkt("capability=hunks") + write_flush() + + log("ready") + + while True: + cmd = None + pathname = None + while True: + p = read_pkt() + if p is None: sys.exit(0) + if p == "": break + if p.startswith("command="): cmd = p.split("=",1)[1] + if p.startswith("pathname="): pathname = p.split("=",1)[1] + if cmd is None: sys.exit(0) + old = read_content() + new = read_content() + old_first = old.split(b"\n")[0].decode(errors="replace") if old else "" + new_first = new.split(b"\n")[0].decode(errors="replace") if new else "" + log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}") + + if mode == "error": + write_flush() + write_pkt("status=error") + write_flush() + continue + + if mode == "abort": + write_flush() + write_pkt("status=abort") + write_flush() + continue + + if mode == "crash": + sys.exit(1) + + if cmd == "hunks": + if mode == "fixed-hunk": + write_pkt("hunk 5 2 5 2") + elif mode == "bad-hunk": + write_pkt("hunk 999 1 999 1") + elif mode == "bad-sync": + write_pkt("hunk 1 2 1 1") + elif mode == "overlap": + write_pkt("hunk 1 5 1 5") + write_pkt("hunk 3 2 3 2") + elif mode == "no-hunks": + pass + else: + ol = old.count(b"\n")t/t4080-diff-process.sh line 140
Quoted 10 lines
t/t4080-diff-process.sh new file +10
-0+ nl = new.count(b"\n") + write_pkt(f"hunk 1 {ol} 1 {nl}") + write_flush() + write_pkt("status=success") + write_flush() + else: + write_flush() + write_pkt("status=error") + write_flush() + PYEOFThe existing pattern is to provide larger scripts as fixtures in associated
t/tNNNN/directories, not as heredoc, see e.g.t/t1509/prepare-chroot.sh. Writing scripts, especially lengthy ones, in heredoc strings makes it virtually impossible to use static code analysis or syntax highlighting to fend off banal errors.Given the complexity of what t4080 tries to test (error, abort, crash, bad-sync, no-hunks, multiple files in one session, capability negotiation), it would unfortunately be infeasible to use
test-tool pkt-linefrom a shell script implementing thatdiff.*.processprotocol.So I've spiked a demo how the
test-tool diff-process-backendcould look like (letting Opus do the menial typing, so that I can enjoy at least part of a sunny Sunday outside), which also passes the CI build and test:
https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6That commit is of course not intended to be used as-is; Feel free to pick code parts of it and integrate them into your topic branch. Or write your own test-tool helper from scratch if that's more your jam.
Ciao,
JohannesQuoted 424 lines
t/t4080-diff-process.sh new file +398
-0+ write_script diff-process-backend <<-SHEOF + exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@" + SHEOF +} + +BACKEND="./diff-process-backend" + +test_expect_success PYTHON 'setup' ' + setup_backend && + echo "*.c diff=cdiff" >.gitattributes && + git add .gitattributes && + + # boundary.c: 10 lines, changes at 5-6 and 9-10. + # Used by: hunk boundaries, error fallback, crash, bad hunks, overlap. + cat >boundary.c <<-\EOF && + line1 + line2 + line3 + line4 + OLD5 + OLD6 + line7 + line8 + OLD9 + OLD10 + EOF + git add boundary.c && + + # worddiff.c: single-line function, value changes 1 -> 999. + # Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat. + cat >worddiff.c <<-\EOF && + int value(void) { return 1; } + EOF + git add worddiff.c && + + # newfile.c: single-line function, value changes 42 -> 99. + # Used by: new file, --exit-code, multiple drivers. + cat >newfile.c <<-\EOF && + int new_func(void) { return 42; } + EOF + git add newfile.c && + + # logtest.c: single-line function for log/format-patch tests. + # Needs two commits so log -1 has a diff. + cat >logtest.c <<-\EOF && + int logfunc(void) { return 1; } + EOF + git add logtest.c && + + # two.c/one.c: two-file pair for error/abort/startup-failure tests. + cat >one.c <<-\EOF && + int first(void) { return 1; } + EOF + cat >two.c <<-\EOF && + int second(void) { return 2; } + EOF + git add one.c two.c && + + git commit -m "initial" && + + # Second commit for logtest.c (so log -1 has something to show). + cat >logtest.c <<-\EOF && + int logfunc(void) { return 2; } + EOF + git add logtest.c && + git commit -m "change logtest.c" && + + # Working tree modifications (not committed). + cat >boundary.c <<-\EOF && + line1 + line2 + line3 + line4 + NEW5 + NEW6 + line7 + line8 + NEW9 + NEW10 + EOF + + cat >worddiff.c <<-\EOF && + int value(void) { return 999; } + EOF + + cat >newfile.c <<-\EOF && + int new_func(void) { return 99; } + EOF + + cat >one.c <<-\EOF && + int first(void) { return 10; } + EOF + + cat >two.c <<-\EOF + int second(void) { return 20; } + EOF +' + +# +# Core behavior: the tool controls which lines are marked as changed. +# + +test_expect_success PYTHON 'diff process hunk boundaries affect output' ' + # The file has changes at lines 5-6 and 9-10, but fixed-hunk + # only reports lines 5-6 as changed. Lines 9-10 should not + # appear as changed in the output. + git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \ + diff boundary.c >actual && + test_grep "^-OLD5" actual && + test_grep "^-OLD6" actual && + test_grep "^+NEW5" actual && + test_grep "^+NEW6" actual && + test_grep ! "^-OLD9" actual && + test_grep ! "^-OLD10" actual && + test_grep ! "^+NEW9" actual && + test_grep ! "^+NEW10" actual +' + +test_expect_success PYTHON 'diff process works with new file' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- newfile.c >actual 2>stderr && + test_grep "return 99" actual && + test_grep "pathname=newfile.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process works with added file (empty old side)' ' + cat >added.c <<-\EOF && + int added(void) { return 1; } + EOF + git add added.c && + + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --cached -- added.c >actual 2>stderr && + test_grep "added" actual && + test_grep "pathname=added.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process skipped for binary files' ' + printf "\\0binary" >binary.c && + git add binary.c && + git commit -m "add binary" && + printf "\\0changed" >binary.c && + + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- binary.c >actual && + test_grep "Binary files" actual && + test_path_is_missing backend.log +' + +test_expect_success PYTHON 'diff process not consulted for unmatched driver' ' + echo "not tracked by cdiff" >unmatched.txt && + git add unmatched.txt && + git commit -m "add unmatched.txt" && + + echo "modified" >unmatched.txt && + + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- unmatched.txt >actual && + test_grep "modified" actual && + test_path_is_missing backend.log +' + +test_expect_success PYTHON 'multiple drivers use separate processes' ' + echo "*.h diff=hdiff" >>.gitattributes && + git add .gitattributes && + + cat >multi.h <<-\EOF && + int header(void) { return 1; } + EOF + git add multi.h && + git commit -m "add multi.h" && + + cat >multi.h <<-\EOF && + int header(void) { return 2; } + EOF + + rm -f backend-c.log backend-h.log && + git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \ + -c diff.hdiff.process="$BACKEND --log=backend-h.log" \ + diff -- newfile.c multi.h >actual 2>stderr && + test_grep "pathname=newfile.c" backend-c.log && + test_grep "pathname=multi.h" backend-h.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process works alongside textconv' ' + write_script uppercase-filter <<-\EOF && + tr "a-z" "A-Z" <"$1" + EOF + + cat >textconv.c <<-\EOF && + hello world + EOF + git add textconv.c && + git commit -m "add textconv.c" && + + cat >textconv.c <<-\EOF && + goodbye world + EOF + + rm -f backend.log && + git -c diff.cdiff.textconv="./uppercase-filter" \ + -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff -- textconv.c >actual 2>stderr && + # The diff process receives textconv-transformed (uppercase) content. + test_grep "pathname=textconv.c" backend.log && + test_grep "old=HELLO WORLD" backend.log && + test_grep "new=GOODBYE WORLD" backend.log && + test_must_be_empty stderr +' + +# +# Downstream features: word diff, log, equivalent files, exit code. +# + +test_expect_success PYTHON 'diff process with --word-diff' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --word-diff worddiff.c >actual 2>stderr && + test_grep "\[-1;-\]" actual && + test_grep "{+999;+}" actual && + test_grep "pathname=worddiff.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process works with git log -p' ' + # With no-hunks mode, the tool says the files are equivalent, + # so log -p should show the commit but no diff content. + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \ + log -1 -p -- logtest.c >actual 2>stderr && + test_grep "change logtest.c" actual && + test_grep ! "return 2" actual && + test_grep "command=hunks pathname=logtest.c" backend.log && + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process no hunks suppresses diff output' ' + cat >nohunks.c <<-\EOF && + int zero(void) { return 0; } + EOF + git add nohunks.c && + git commit -m "add nohunks.c" && + + cat >nohunks.c <<-\EOF && + int zero(void) { return 999; } + EOF + + git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \ + diff nohunks.c >actual && + test_must_be_empty actual +' + +test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' ' + git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \ + diff --exit-code nohunks.c +' + +test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' ' + test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \ + diff --exit-code newfile.c +' + +# +# Bypass mechanisms: flags and commands that skip the diff process. +# + +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --diff-algorithm=patience worddiff.c >actual && + test_grep "return 999" actual && + test_path_is_missing backend.log +' + +test_expect_success PYTHON 'diff process not used by --stat' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --stat worddiff.c >actual && + test_grep "worddiff.c" actual && + test_path_is_missing backend.log +' + +# +# Error handling and fallback. +# + +test_expect_success PYTHON 'diff process fallback on tool error status' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \ + diff boundary.c >actual 2>stderr && + # Fallback produces the full builtin diff (both change regions). + test_grep "^-OLD5" actual && + test_grep "^+NEW5" actual && + test_grep "^-OLD9" actual && + test_grep "^+NEW9" actual && + # Tool was contacted (it replied with error, not crash). + test_grep "command=hunks pathname=boundary.c" backend.log && + test_grep "diff process.*failed" stderr +' + +test_expect_success PYTHON 'diff process error keeps tool available for next file' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \ + diff -- one.c two.c >actual 2>stderr && + # Unlike abort, error keeps the tool available: both files + # are sent to the tool (and both fall back). + test_grep "pathname=one.c" backend.log && + test_grep "pathname=two.c" backend.log && + test_grep "return 10" actual && + test_grep "return 20" actual +' + +test_expect_success PYTHON 'diff process abort disables for session' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \ + diff -- one.c two.c >actual && + # Both files should still produce diff output via fallback. + test_grep "return 10" actual && + test_grep "return 20" actual && + # The tool aborts on the first file and git clears its + # capability. The second file never contacts the tool. + test_grep "pathname=one.c" backend.log && + test_grep ! "pathname=two.c" backend.log +' + +test_expect_success PYTHON 'diff process fallback on tool crash' ' + git -c diff.cdiff.process="$BACKEND --mode=crash" \ + diff boundary.c >actual 2>stderr && + test_grep "^-OLD5" actual && + test_grep "^+NEW5" actual && + test_grep "^-OLD9" actual && + test_grep "^+NEW9" actual && + # Crash is a communication failure, so a warning is emitted. + test_grep "diff process.*failed" stderr +' + +test_expect_success PYTHON 'diff process startup failure only warns once' ' + git -c diff.cdiff.process="/nonexistent/tool" \ + diff -- one.c two.c >actual 2>stderr && + # Both files produce diff output via fallback. + test_grep "return 10" actual && + test_grep "return 20" actual && + # Sentinel prevents repeated warnings: only one, not one per file. + test_grep "diff process.*failed" stderr >warnings && + test_line_count = 1 warnings +' + +test_expect_success PYTHON 'diff process fallback on bad hunks' ' + git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \ + diff boundary.c >actual 2>stderr && + test_grep "^-OLD5" actual && + test_grep "^+NEW5" actual && + test_grep "^-OLD9" actual && + test_grep "^+NEW9" actual && + # Invalid hunks are caught by xdiff validation, not the + # protocol layer, so no warning is emitted. + test_must_be_empty stderr +' + +test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' ' + cat >synctest.c <<-\EOF && + line1 + line2 + line3 + EOF + git add synctest.c && + git commit -m "add synctest.c" && + + cat >synctest.c <<-\EOF && + line1 + changed + line3 + EOF + + # bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new + # line as changed, leaving 1 unchanged old vs 2 unchanged new. + # The synchronization invariant fails and git falls back. + git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \ + diff synctest.c >actual 2>stderr && + test_grep "changed" actual +' + +test_expect_success PYTHON 'diff process fallback on overlapping hunks' ' + # boundary.c has 10 lines, so both hunks are in bounds + # but they overlap at lines 3-5, triggering the ordering check. + git -c diff.cdiff.process="$BACKEND --mode=overlap" \ + diff boundary.c >actual 2>stderr && + test_grep "NEW5" actual +' + +test_doneuserdiff.h +3
-1@@ -3,6 +3,7 @@#include "notes-cache.h" +struct diff_subprocess ;struct index_state ;struct repository ;@@ -33,6 +34,8 @@ struct userdiff_driver {int textconv_want_cache;const char *process;char *process_owned; +struct diff_subprocess *diff_subprocess; + unsigned diff_process_failed : 1; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 , --gitgitgadget
Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process
Quoted 12 lines · Michael Montalbo
Quoted 8 lines
So the conscious project direction has been: fold pkt-line test backends into
test-tooland drop the scripting-language prereq. Reintroducing the same shape in Python would walk this back.Patrick's careful effort in
27bd8ee311719(Merge branch 'ps/fewer-perl', 2025-04-29) has been another clear sign that the Git project is actively _removing_ scripting-language dependencies from the build and test infrastructure, not adding new ones.Now I wonder if the extension / addition of more Perl test infra with my other series:
Quoted 6 lines
https://lore.kernel.org/git/[email protected]/T/
also goes against the project direction w.r.t. removing scripting languages. Maybe I should also re-evaluate the usage of Perl there. I am leveraging existing shell parsing logic written in Perl, but maybe the preference for Perl-based lint rules is a mistake and should be avoided.
That sounds prudent (even though it is outside the scope of _this_ topic, of course).
Thanks.
Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process
In reply to Johannes SchindelinQuoted 5 lines · Johannes Schindelin
So the conscious project direction has been: fold pkt-line test backends into
test-tooland drop the scripting-language prereq. Reintroducing the same shape in Python would walk this back.
...
ThePYTHONprereq exists in exactly five files today, allgit p4
Quoted 8 lines
related (where Python is an intrinsic prerequisite given that
git-p4.py_is_ written in Python):t/lib-git-p4.sh,t/t9802-git-p4-filetype.sh,t/t9810-git-p4-rcs.sh,t/t9835-git-p4-metadata-encoding-python2.sh, andt/t9836-git-p4-metadata-encoding-python3.sh.
...
That commit is of course not intended to be used as-is; Feel free to pick code parts of it and integrate them into your topic branch. Or write your own test-tool helper from scratch if that's more your jam.
Showing better direction to new folks with such a clear thinking is very much appreciated. Even though it is natural and perfectly OK for tests that interacts with parts of Git that are written in these languages (e.g., we are OK for gitweb tests to require Perl), we should consciously keep ourselves clean and not adding unnecessary dependencies.
PATCH v3 5/6 diff: bypass diff process with --no-ext-diff and in format-patch
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Make --no-ext-diff disable diff.<driver>.process in addition to diff.<driver>.command. Although the two mechanisms work differently (command replaces Git's output, process feeds hunks back into the pipeline), both invoke external tools and --no-ext-diff means "no external tools."
Replace the OPT_BOOL for --ext-diff with an OPT_CALLBACK that sets both allow_external and no_diff_process, so a single option controls both. Passing --ext-diff explicitly clears no_diff_process, so a later --ext-diff overrides an earlier --no-ext-diff.
Disable the diff process unconditionally in format-patch so that generated patches are always based on the builtin diff algorithm and can be applied reliably by recipients who do not have the external tool.
Document that --diff-algorithm also bypasses the diff process, since it sets ignore_driver_algorithm which diff_process_fill_hunks already checks.
--- Documentation/diff-algorithm-option.adoc | 3 +++ Documentation/diff-options.adoc | 4 +++- builtin/log.c | 7 +++++++ diff.c | 16 ++++++++++++++-- diff.h | 4 +++- t/t4080-diff-process.sh | 16 ++++++++++++++++ 6 files changed, 46 insertions(+), 4 deletions(-)
Documentation/diff-algorithm-option.adoc +3 -0
@@ -18,3 +18,6 @@ For instance, if you configured the `diff.algorithm` variable to a non-default value and want to use the default one, then you have to use `--diff-algorithm=default` option. ++ +If you explicitly choose a diff algorithm, it also bypasses +`diff.<driver>.process` (see linkgit:gitattributes[5]).
Documentation/diff-options.adoc +3 -1
@@ -825,7 +825,9 @@ endif::git-format-patch[] to use this option with linkgit:git-log[1] and friends. `--no-ext-diff`:: - Disallow external diff drivers. + Disallow external diff helpers, including + `diff.<driver>.command` and `diff.<driver>.process` + (see linkgit:gitattributes[5]). `--textconv`:: `--no-textconv`::
builtin/log.c +7 -0
@@ -2213,6 +2213,13 @@ int cmd_format_patch(int argc,if (argc >1 )die (_ ("unrecognized argument: %s" ), argv[1 ]); +/* +* Disable diff.<driver>.process so that patches generated by +* format-patch are always based on the builtin diff algorithm +* and can be applied reliably. +*/ + rev.diffopt.flags.no_diff_process =1 ; +if (rev.diffopt.output_format &DIFF_FORMAT_NAME )die (_ ("--name-only does not make sense" ));if (rev.diffopt.output_format &DIFF_FORMAT_NAME_STATUS )
diff.c +14 -2
@@ -5913,6 +5913,17 @@ static int diff_opt_submodule(const struct option *opt,return 0 ; } +static int diff_opt_ext_diff (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; + +BUG_ON_OPT_ARG (arg); + options->flags.allow_external = !unset; + options->flags.no_diff_process = unset; +return 0 ; +} +static int diff_opt_textconv (const struct option *opt,const char *arg,int unset) {
@@ -6241,8 +6252,9 @@ struct option *add_diff_options(const struct option *opts, N_("exit with 1 if there were differences, 0 otherwise")),OPT_BOOL (0 ,"quiet" , &options->flags.quick,N_ ("disable all output of the program" )), -OPT_BOOL (0 ,"ext-diff" , &options->flags.allow_external, -N_ ("allow an external diff helper to be executed" )), +OPT_CALLBACK_F (0 ,"ext-diff" , options,NULL , +N_ ("allow an external diff helper to be executed" ), +PARSE_OPT_NOARG , diff_opt_ext_diff),OPT_CALLBACK_F (0 ,"textconv" , options,NULL ,N_ ("run external text conversion filters when comparing binary files" ), PARSE_OPT_NOARG, diff_opt_textconv),
diff.h +3 -1
@@ -173,7 +173,9 @@ struct diff_flags {*/ unsigned allow_external; -/** Disables diff.<driver>.process. */ +/** +* Disables diff.<driver>.process. Set by --no-ext-diff. +*/ unsigned no_diff_process;/**
t/t4080-diff-process.sh +16 -0
@@ -419,6 +419,22 @@ test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' ' test_path_is_missing backend.log ' +test_expect_success PYTHON 'diff process bypassed by --no-ext-diff' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --no-ext-diff worddiff.c >actual && + test_grep "return 999" actual && + test_path_is_missing backend.log +' + +test_expect_success PYTHON 'diff process not used by format-patch' ' + rm -f backend.log && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + format-patch -1 --stdout -- logtest.c >actual && + test_grep "return 2" actual && + test_path_is_missing backend.log +' + test_expect_success PYTHON 'diff process not used by --stat' ' rm -f backend.log && git -c diff.cdiff.process="$BACKEND --log=backend.log" \
Signature · 1 line
gitgitgadget
PATCH v3 6/6 blame: consult diff process for no-hunk detection
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
When a diff process is configured via diff.<driver>.process, consult it during blame's per-commit diffing. If the process returns no hunks for a commit's changes to a file, treat the commit as having no changes, causing blame to attribute lines to earlier commits.
The consultation happens at the pass_blame_to_parent() callsite using diff_process_fill_hunks(), matching how builtin_diff() in diff.c uses the same function. A new diff_hunks_xpp() variant accepts a pre-populated xpparam_t for this callsite, while the existing diff_hunks() retains its original signature and behavior. The copy-detection callsite is unaffected since it does not use the diff process.
The subprocess is long-running (one startup cost amortized across the blame traversal), but each commit in the file's history incurs a round-trip to the tool.
--- blame.c | 40 +++++++++++---- t/t4080-diff-process.sh | 106 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+), 9 deletions(-)
blame.c +31 -9
@@ -19,6 +19,8 @@#include "tag.h" #include "trace2.h" #include "blame.h" +#include "diff-process.h" +#include "xdiff-interface.h" #include "alloc.h" #include "commit-slab.h" #include "bloom.h"
@@ -314,17 +316,25 @@ static struct commit *fake_working_tree_commit(struct repository *r, -static int diff_hunks (mmfile_t *file_a,mmfile_t *file_b, -xdl_emit_hunk_consume_func_t hunk_func,void *cb_data,int xdl_opts) +static int diff_hunks_xpp (mmfile_t *file_a,mmfile_t *file_b, +xdl_emit_hunk_consume_func_t hunk_func, +void *cb_data,xpparam_t *xpp) { -xpparam_t xpp = {0 };xdemitconf_t xecfg = {0 };xdemitcb_t ecb = {NULL }; - xpp.flags = xdl_opts; xecfg.hunk_func = hunk_func; ecb.priv = cb_data; -return xdi_diff (file_a, file_b, &xpp, &xecfg, &ecb); +return xdi_diff (file_a, file_b, xpp, &xecfg, &ecb); +} + +static int diff_hunks (mmfile_t *file_a,mmfile_t *file_b, +xdl_emit_hunk_consume_func_t hunk_func,void *cb_data,int xdl_opts) +{ +xpparam_t xpp = {0 }; + + xpp.flags = xdl_opts; +return diff_hunks_xpp (file_a, file_b, hunk_func, cb_data, &xpp); }static const char *get_next_line (const char *start,const char *end)
@@ -1943,6 +1953,7 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb, struct blame_origin *parent, int ignore_diffs) {mmfile_t file_p, file_o; +xpparam_t xpp = {0 };struct blame_chunk_cb_data d;struct blame_entry *newdest =NULL ;
@@ -1961,10 +1972,21 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb, &sb->num_read_blob, ignore_diffs); sb->num_get_patch++; -if (diff_hunks (&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts)) -die ("unable to generate diff (%s -> %s)" , -oid_to_hex (&parent->commit->object.oid), -oid_to_hex (&target->commit->object.oid)); + xpp.flags = sb->xdl_opts; +/* +* If the diff process considers the files equivalent, +* skip the diff so blame looks past this commit. +*/ +if (diff_process_fill_hunks (&sb->revs->diffopt, target->path, + &file_p, &file_o, &xpp) + !=DIFF_PROCESS_EQUIVALENT ) { +if (diff_hunks_xpp (&file_p, &file_o, blame_chunk_cb, + &d, &xpp)) +die ("unable to generate diff (%s -> %s)" , +oid_to_hex (&parent->commit->object.oid), +oid_to_hex (&target->commit->object.oid)); + } +free (xpp.external_hunks);/* The rest are the same as the parent */ blame_chunk (&d.dstq, &d.srcq,INT_MAX , d.offset,INT_MAX ,0 , parent, target,0 );
t/t4080-diff-process.sh +106 -0
110 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Quoted 4 lines · Michael Montalbo via GitGitGadget
Language-aware diff tools (e.g., Difftastic) and format-specific analyzers can produce better line matching than Git's builtin diff algorithm, but diff.<driver>.command replaces Git's output entirely, losing downstream features like word diff, function context, color, and blame.
This seems to break CI on Windows; take a look at
https://github.com/git/git/actions/runs/26709491830/job/78717295153
for an example.
Thanks.
Quoted 8 lines · Junio C Hamano
Quoted 4 lines · Michael Montalbo via GitGitGadget
Language-aware diff tools (e.g., Difftastic) and format-specific analyzers can produce better line matching than Git's builtin diff algorithm, but diff.<driver>.command replaces Git's output entirely, losing downstream features like word diff, function context, color, and blame.
Quoted 7 lines
This seems to break CI on Windows; take a look at
https://github.com/git/git/actions/runs/26709491830/job/78717295153
for an example.
Thanks.
Thanks for the heads up. Interestingly, I think Windows exposed a latent issue with sub-process startup dying instead of erroring out when there is a space in the path.
I've submitted https://lore.kernel.org/git/[email protected]/T/#u
which I believe addresses the issue, however it seems like there are
some potentially
unrelated CI failures going on right now which is making verification
a bit harder.
v4 cover diff: add diff.<driver>.process for external hunk providers
In reply to Michael Montalbo via GitGitGadgetLanguage-aware diff tools (e.g., Difftastic) and format-specific analyzers can produce better line matching than Git's builtin diff algorithm, but diff.<driver>.command replaces Git's output entirely, losing downstream features like word diff, function context, color, and blame.
This series adds diff.<driver>.process, a long-running subprocess protocol that lets an external tool control which lines Git considers changed while Git handles all output formatting. The protocol follows filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation, one process per Git invocation.
The tool receives both file versions and returns changed regions (line ranges in the old and new file). Git validates and feeds them into the xdiff pipeline in place of the builtin diff algorithm. When the tool returns no hunks, Git treats the files as having no changes.
* Patch 1: xdiff plumbing for externally supplied hunks.
* Patch 2: diff.<driver>.process config key.
* Patch 3: refactor subprocess API to separate process lifecycle from hashmap management, since the diff process stores its subprocess on the userdiff driver rather than in a hashmap.
* Patch 4: the main feature.
* Patch 5: bypass knobs (--no-ext-diff, format-patch).
* Patch 6: blame integration so the tool can declare commits as having no changes.
Changes since v3:
* Replaced Python test backend with C test-tool helper (thanks to Johannes Schindelin).
* Added test coverage cases for deleted file, malformed hunk line, and missing capability.
* Fixed potential overflow in synchronization invariant check by counting from changed[] arrays instead of accumulating.
* Accept start=0 with count=0 in the hunk protocol, matching what git diff itself emits for empty file sides.
* Warn on external hunk validation failure with specific reasons (range exceeded, overlap, sync mismatch) to help tool authors debug their implementations.
* Test backend follows the same convention (start=0 when count=0 for empty file sides).
Michael Montalbo (6):
xdiff: support external hunks via xpparam_t
userdiff: add diff.<driver>.process config
sub-process: separate process lifecycle from hashmap management
diff: add long-running diff process via diff.<driver>.process
diff: bypass diff process with --no-ext-diff and in format-patch
blame: consult diff process for no-hunk detection
Documentation/config/diff.adoc | 5 + Documentation/diff-algorithm-option.adoc | 3 + Documentation/diff-options.adoc | 4 +- Documentation/gitattributes.adoc | 143 ++++++ Makefile | 2 + blame.c | 40 +- builtin/log.c | 7 + diff-process.c | 297 ++++++++++++ diff-process.h | 39 ++ diff.c | 29 +- diff.h | 5 + meson.build | 1 + sub-process.c | 28 +- sub-process.h | 9 +- t/helper/meson.build | 1 + t/helper/test-diff-process-backend.c | 299 ++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/meson.build | 1 + t/t4080-diff-process.sh | 553 +++++++++++++++++++++++ userdiff.c | 7 + userdiff.h | 5 + xdiff-interface.c | 7 +- xdiff/xdiff.h | 14 + xdiff/xdiffi.c | 123 ++++- xdiff/xprepare.c | 10 + xdiff/xprepare.h | 1 + 27 files changed, 1614 insertions(+), 21 deletions(-) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100644 t/helper/test-diff-process-backend.c create mode 100755 t/t4080-diff-process.sh
base-commit: ea97ad8d017de0c9037451a78008a0fd60abea0c
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/2120
Range-diff vs v3:
1: 13eb201d63 ! 1: 03f261dfe2 xdiff: support external hunks via xpparam_t
@@ xdiff/xdiff.h: typedef struct s_mmbuffer {
+/*
+ * Hunk descriptor for externally computed diffs.
-+ * Line numbers are 1-based, matching unified diff convention.
++ * Line numbers are 1-based; a start of 0 is accepted when
++ * count is 0 (empty file side, matching git diff output).
+ */
+struct xdl_hunk {
+ long old_start, old_count;
@@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
+{
+ size_t i;
+ long j, prev_old_end = 0, prev_new_end = 0;
-+ long total_old = 0, total_new = 0;
++ long changed_old = 0, changed_new = 0;
+
+ /*
+ * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
@@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
+ for (i = 0; i < nr_hunks; i++) {
+ struct xdl_hunk *h = &hunks[i];
+
-+ if (h->old_count < 0 || h->new_count < 0)
++ if (h->old_count < 0 || h->new_count < 0) {
++ warning("diff process hunk %"PRIuMAX": "
++ "negative count (old=%ld, new=%ld)",
++ (uintmax_t)(i + 1),
++ h->old_count, h->new_count);
+ return -1;
-+ if (h->old_start < 1 || h->new_start < 1)
++ }
++ if (h->old_start < 1 || h->new_start < 1) {
++ warning("diff process hunk %"PRIuMAX": "
++ "start must be >= 1 (old=%ld, new=%ld)",
++ (uintmax_t)(i + 1),
++ h->old_start, h->new_start);
+ return -1;
++ }
+
+ /*
+ * Range must fit: start + count - 1 <= nrec,
@@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
+ * start > nrec + 1 and allows start == nrec + 1
+ * (the position after the last line).
+ */
-+ if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1)
++ if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1) {
++ warning("diff process hunk %"PRIuMAX": "
++ "old range %ld+%ld exceeds %lu lines",
++ (uintmax_t)(i + 1),
++ h->old_start, h->old_count,
++ (unsigned long)xe->xdf1.nrec);
+ return -1;
-+ if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
++ }
++ if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1) {
++ warning("diff process hunk %"PRIuMAX": "
++ "new range %ld+%ld exceeds %lu lines",
++ (uintmax_t)(i + 1),
++ h->new_start, h->new_count,
++ (unsigned long)xe->xdf2.nrec);
+ return -1;
++ }
+
+ /* Ordering: no overlap with previous hunk (adjacent is OK) */
+ if (h->old_start < prev_old_end ||
-+ h->new_start < prev_new_end)
++ h->new_start < prev_new_end) {
++ warning("diff process hunk %"PRIuMAX": "
++ "overlaps with previous hunk",
++ (uintmax_t)(i + 1));
+ return -1;
++ }
+
+ for (j = 0; j < h->old_count; j++)
+ xe->xdf1.changed[h->old_start - 1 + j] = true;
@@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
+
+ prev_old_end = h->old_start + h->old_count;
+ prev_new_end = h->new_start + h->new_count;
-+ total_old += h->old_count;
-+ total_new += h->new_count;
+ }
+
+ /*
+ * Synchronization invariant: unchanged line counts must match.
+ * Otherwise xdl_build_script() would walk off one array.
++ *
++ * Count changed lines from the arrays rather than accumulating
++ * during the loop to avoid any overflow in the summation.
+ */
-+ if ((long)xe->xdf1.nrec - total_old !=
-+ (long)xe->xdf2.nrec - total_new)
++ for (j = 0; j < (long)xe->xdf1.nrec; j++)
++ if (xe->xdf1.changed[j])
++ changed_old++;
++ for (j = 0; j < (long)xe->xdf2.nrec; j++)
++ if (xe->xdf2.changed[j])
++ changed_new++;
++ if ((long)xe->xdf1.nrec - changed_old !=
++ (long)xe->xdf2.nrec - changed_new) {
++ warning("diff process: unchanged line count mismatch "
++ "(old: %ld unchanged, new: %ld unchanged)",
++ (long)xe->xdf1.nrec - changed_old,
++ (long)xe->xdf2.nrec - changed_new);
+ return -1;
++ }
+
+ return 0;
+}
2: 58f4763c63 = 2: 30617ee17b userdiff: add diff.<driver>.process config
3: d6c833dd42 ! 3: 459e485e6d sub-process: separate process lifecycle from hashmap management
@@ Commit message
and subprocess_stop() become thin wrappers that add hashmap
operations on top.
- No functional change for existing callers.
-
Signed-off-by: Michael Montalbo <[email protected]>
## sub-process.c ##
@@ sub-process.c: void subprocess_stop(struct hashmap *hashmap, struct subprocess_e
kill(entry->process.pid, SIGTERM);
finish_command(&entry->process);
+}
-+
+
+void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+{
+ if (!entry)
+ return;
-
++
+ subprocess_stop_command(entry);
hashmap_remove(hashmap, &entry->ent, NULL);
}
@@ sub-process.c: int subprocess_start(struct hashmap *hashmap, struct subprocess_e
+ int err;
+
+ err = subprocess_start_command(entry, cmd, startfn);
-+ if (err) {
++ if (err)
+ return err;
-+ }
+
+ hashmap_entry_init(&entry->ent, strhash(cmd));
hashmap_add(hashmap, &entry->ent);
4: d044fa0ee5 ! 4: 10b3980f59 diff: add long-running diff process via diff.<driver>.process
@@ Commit message
textconv-transformed content. The tool controls which lines
are marked as changed while the display shows the file content.
Patch output features (word diff, function context, color) work
- normally; summary formats like --stat use their own diff path
- and are not affected.
+ normally; --stat uses its own diff codepath and never consults
+ the diff process.
The handshake negotiates version=1 and capability=hunks. Per-file
requests send command=hunks, pathname, and both file contents as
@@ Commit message
"hunks populated" from "files equivalent" from "not applicable"
from "tool failure."
+ Helped-by: Johannes Schindelin <[email protected]>
Signed-off-by: Michael Montalbo <[email protected]>
## Documentation/config/diff.adoc ##
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
+at line 3 in the old file were replaced by 4 lines starting at
+line 3 in the new file. An `<old_count>` of 0 means no lines were
+removed (pure insertion); a `<new_count>` of 0 means no lines were
-+added (pure deletion).
++added (pure deletion). A start value of 0 is accepted when
++the corresponding count is 0 (e.g., `hunk 0 0 1 5` for a newly
++added file), matching what `git diff` itself emits for empty
++file sides.
+
+Lines are delimited by newlines. A file `"foo\nbar\n"` and a
+file `"foo\nbar"` both have 2 lines.
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
+packet: git< 0000
+-----------------------
+
-+If the tool returns invalid hunks (out of bounds, overlapping), Git
-+silently falls back to the builtin diff algorithm.
++If the tool returns invalid hunks (out of bounds, overlapping, or
++mismatched unchanged line counts), Git warns and falls back to the
++builtin diff algorithm.
+
+In case the tool cannot or does not want to process the content,
+it is expected to respond with an "error" status. Git warns and
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
## Makefile ##
+@@ Makefile: TEST_BUILTINS_OBJS += test-csprng.o
+ TEST_BUILTINS_OBJS += test-date.o
+ TEST_BUILTINS_OBJS += test-delete-gpgsig.o
+ TEST_BUILTINS_OBJS += test-delta.o
++TEST_BUILTINS_OBJS += test-diff-process-backend.o
+ TEST_BUILTINS_OBJS += test-dir-iterator.o
+ TEST_BUILTINS_OBJS += test-drop-caches.o
+ TEST_BUILTINS_OBJS += test-dump-cache-tree.o
@@ Makefile: LIB_OBJS += diff-delta.o
LIB_OBJS += diff-merges.o
LIB_OBJS += diff-lib.o
@@ diff-process.c (new)
+ if (errno || end == line || *end != '\0')
+ return -1;
+
++ /*
++ * git diff emits start=0 when count=0 (empty file side).
++ * Normalize to 1-based so downstream validation can assume start >= 1.
++ */
++ if (!hunk->old_count && !hunk->old_start)
++ hunk->old_start = 1;
++ if (!hunk->new_count && !hunk->new_start)
++ hunk->new_start = 1;
++
+ return 0;
+}
+
@@ meson.build: libgit_sources = [
'diffcore-break.c',
'diffcore-delta.c',
- ## t/.gitattributes ##
-@@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
- /t8005/*.txt eol=lf
- /t9*/*.dump eol=lf
- /t0040*.sh whitespace=-indent-with-non-tab
-+/t4080-diff-process.sh whitespace=-indent-with-non-tab
+ ## t/helper/meson.build ##
+@@ t/helper/meson.build: test_tool_sources = [
+ 'test-date.c',
+ 'test-delete-gpgsig.c',
+ 'test-delta.c',
++ 'test-diff-process-backend.c',
+ 'test-dir-iterator.c',
+ 'test-drop-caches.c',
+ 'test-dump-cache-tree.c',
+
+ ## t/helper/test-diff-process-backend.c (new) ##
+@@
++/*
++ * Test backend for the long-running diff process protocol
++ * (see diff-process.c and Documentation/gitattributes.adoc).
++ *
++ * Usage: test-tool diff-process-backend --mode=<mode> [--log=<path>]
++ *
++ * Implements the server side of the pkt-line handshake and a per-file
++ * response loop. The --mode= switch selects the response shape
++ * (success, error, abort, crash, malformed hunks).
++ *
++ * Per-file request from Git:
++ *
++ * packet: git> command=hunks
++ * packet: git> pathname=<path>
++ * packet: git> 0000
++ * packet: git> OLD_CONTENT
++ * packet: git> 0000
++ * packet: git> NEW_CONTENT
++ * packet: git> 0000
++ *
++ * Response varies by --mode (default: whole-file):
++ *
++ * whole-file packet: git< hunk 1 <old_lines> 1 <new_lines>
++ * fixed-hunk packet: git< hunk 5 2 5 2
++ * no-hunks (no hunk packets)
++ * bad-hunk packet: git< hunk 999 1 999 1
++ * bad-parse packet: git< garbage not a hunk
++ * bad-sync packet: git< hunk 1 2 1 1
++ * overlap packet: git< hunk 1 5 1 5
++ * packet: git< hunk 3 2 3 2
++ * no-cap (omits capability=hunks during handshake)
++ * error (status=error instead of status=success)
++ * abort (status=abort instead of status=success)
++ * crash exit(1) before sending any response
++ *
++ * All non-error/abort modes end with:
++ *
++ * packet: git< 0000
++ * packet: git< status=success
++ * packet: git< 0000
++ *
++ * Each request is logged to --log as:
++ *
++ * command=<cmd> pathname=<path> old=<first line> new=<first line>
++ */
++
++#include "test-tool.h"
++#include "pkt-line.h"
++#include "parse-options.h"
++#include "strbuf.h"
++
++static FILE *logfile;
++
++enum mode {
++ MODE_WHOLE_FILE,
++ MODE_FIXED_HUNK,
++ MODE_NO_HUNKS,
++ MODE_BAD_HUNK,
++ MODE_BAD_PARSE,
++ MODE_BAD_SYNC,
++ MODE_OVERLAP,
++ MODE_NO_CAP,
++ MODE_ERROR,
++ MODE_ABORT,
++ MODE_CRASH,
++};
++
++static enum mode parse_mode(const char *s)
++{
++ if (!strcmp(s, "whole-file"))
++ return MODE_WHOLE_FILE;
++ if (!strcmp(s, "fixed-hunk"))
++ return MODE_FIXED_HUNK;
++ if (!strcmp(s, "no-hunks"))
++ return MODE_NO_HUNKS;
++ if (!strcmp(s, "bad-hunk"))
++ return MODE_BAD_HUNK;
++ if (!strcmp(s, "bad-parse"))
++ return MODE_BAD_PARSE;
++ if (!strcmp(s, "bad-sync"))
++ return MODE_BAD_SYNC;
++ if (!strcmp(s, "overlap"))
++ return MODE_OVERLAP;
++ if (!strcmp(s, "no-cap"))
++ return MODE_NO_CAP;
++ if (!strcmp(s, "error"))
++ return MODE_ERROR;
++ if (!strcmp(s, "abort"))
++ return MODE_ABORT;
++ if (!strcmp(s, "crash"))
++ return MODE_CRASH;
++ die("unknown --mode=%s", s);
++}
++
++/*
++ * Read "key=value" packets up to a flush, capturing "command" and
++ * "pathname". Returns 1 if a request was read, 0 on EOF.
++ *
++ * The first packet uses the gentle variant so that a clean shutdown
++ * by Git (EOF) does not produce a spurious "the remote end hung up
++ * unexpectedly" on stderr. Subsequent packets use the non-gentle
++ * variant: once inside a request, truncation is a protocol violation
++ * and dying loudly is the correct response.
++ */
++static int read_request_header(char **command, char **pathname)
++{
++ int first = 1;
++ char *line;
++
++ *command = *pathname = NULL;
++ for (;;) {
++ const char *value;
++
++ if (first) {
++ if (packet_read_line_gently(0, NULL, &line) < 0)
++ return 0;
++ first = 0;
++ } else {
++ line = packet_read_line(0, NULL);
++ }
++ if (!line)
++ break;
++ if (skip_prefix(line, "command=", &value))
++ *command = xstrdup(value);
++ else if (skip_prefix(line, "pathname=", &value))
++ *pathname = xstrdup(value);
++ }
++ return 1;
++}
++
++static size_t count_lines(const struct strbuf *buf)
++{
++ size_t lines = 0;
++
++ for (size_t i = 0; i < buf->len; i++)
++ if (buf->buf[i] == '\n')
++ lines++;
++
++ return lines + (buf->len > 0 && buf->buf[buf->len - 1] != '\n');
++}
++
++static void send_status(const char *status)
++{
++ packet_flush(1);
++ packet_write_fmt(1, "%s\n", status);
++ packet_flush(1);
++}
++
++static void respond(enum mode mode,
++ const struct strbuf *old_buf,
++ const struct strbuf *new_buf)
++{
++ switch (mode) {
++ case MODE_ERROR:
++ send_status("status=error");
++ return;
++ case MODE_ABORT:
++ send_status("status=abort");
++ return;
++ case MODE_CRASH:
++ exit(1);
++ case MODE_FIXED_HUNK:
++ packet_write_fmt(1, "hunk 5 2 5 2\n");
++ break;
++ case MODE_BAD_HUNK:
++ packet_write_fmt(1, "hunk 999 1 999 1\n");
++ break;
++ case MODE_BAD_PARSE:
++ packet_write_fmt(1, "garbage not a hunk\n");
++ break;
++ case MODE_BAD_SYNC:
++ packet_write_fmt(1, "hunk 1 2 1 1\n");
++ break;
++ case MODE_OVERLAP:
++ packet_write_fmt(1, "hunk 1 5 1 5\n");
++ packet_write_fmt(1, "hunk 3 2 3 2\n");
++ break;
++ case MODE_NO_HUNKS:
++ break;
++ case MODE_NO_CAP:
++ case MODE_WHOLE_FILE: {
++ size_t old_lines = count_lines(old_buf);
++ size_t new_lines = count_lines(new_buf);
++ /*
++ * Match git diff output: start=0 when count=0
++ * (empty file side), 1 otherwise.
++ */
++ packet_write_fmt(1, "hunk %"PRIuMAX" %"PRIuMAX
++ " %"PRIuMAX" %"PRIuMAX"\n",
++ (uintmax_t)(old_lines ? 1 : 0),
++ (uintmax_t)old_lines,
++ (uintmax_t)(new_lines ? 1 : 0),
++ (uintmax_t)new_lines);
++ break;
++ }
++ }
++ send_status("status=success");
++}
++
++static void command_loop(enum mode mode)
++{
++ for (;;) {
++ char *command = NULL, *pathname = NULL;
++ struct strbuf obuf = STRBUF_INIT;
++ struct strbuf nbuf = STRBUF_INIT;
++
++ if (!read_request_header(&command, &pathname))
++ break; /* EOF: Git closed its end */
++
++ read_packetized_to_strbuf(0, &obuf, 0);
++ read_packetized_to_strbuf(0, &nbuf, 0);
++
++ if (logfile) {
++ fprintf(logfile,
++ "command=%s pathname=%s old=%.*s new=%.*s\n",
++ command ? command : "(none)",
++ pathname ? pathname : "(none)",
++ (int)(strchrnul(obuf.buf, '\n') - obuf.buf),
++ obuf.buf,
++ (int)(strchrnul(nbuf.buf, '\n') - nbuf.buf),
++ nbuf.buf);
++ fflush(logfile);
++ }
++
++ respond(mode, &obuf, &nbuf);
++
++ free(command);
++ free(pathname);
++ strbuf_release(&obuf);
++ strbuf_release(&nbuf);
++ }
++}
++
++static void handshake(enum mode mode)
++{
++ char *line;
++
++ line = packet_read_line(0, NULL);
++ if (!line || strcmp(line, "git-diff-client"))
++ die("bad welcome: '%s'", line ? line : "(eof)");
++ line = packet_read_line(0, NULL);
++ if (!line || strcmp(line, "version=1"))
++ die("bad version: '%s'", line ? line : "(eof)");
++ if (packet_read_line(0, NULL))
++ die("expected flush after version");
++
++ packet_write_fmt(1, "git-diff-server\n");
++ packet_write_fmt(1, "version=1\n");
++ packet_flush(1);
++
++ /* Drain capabilities advertised by Git */
++ while ((line = packet_read_line(0, NULL)))
++ ; /* drain */
++
++ /* Respond with our capabilities (or none for no-cap mode) */
++ if (mode != MODE_NO_CAP)
++ packet_write_fmt(1, "capability=hunks\n");
++ packet_flush(1);
++}
++
++static const char *const usage_str[] = {
++ "test-tool diff-process-backend --mode=<mode> [--log=<path>]",
++ NULL
++};
++
++int cmd__diff_process_backend(int argc, const char **argv)
++{
++ const char *mode_str = NULL, *log_path = NULL;
++ enum mode mode = MODE_WHOLE_FILE;
++ struct option options[] = {
++ OPT_STRING(0, "mode", &mode_str, "mode",
++ "response shape: whole-file (default), fixed-hunk,"
++ " no-hunks, bad-hunk, bad-sync, overlap, error,"
++ " abort, crash"),
++ OPT_STRING(0, "log", &log_path, "path",
++ "append per-request summary to this file"),
++ OPT_END()
++ };
++
++ argc = parse_options(argc, argv, NULL, options, usage_str, 0);
++ if (argc)
++ usage_with_options(usage_str, options);
++
++ if (mode_str)
++ mode = parse_mode(mode_str);
++
++ if (log_path) {
++ logfile = fopen(log_path, "a");
++ if (!logfile)
++ die_errno("failed to open log '%s'", log_path);
++ }
++
++ handshake(mode);
++ command_loop(mode);
++
++ if (logfile && fclose(logfile))
++ die_errno("error closing log");
++ return 0;
++}
+
+ ## t/helper/test-tool.c ##
+@@ t/helper/test-tool.c: static struct test_cmd cmds[] = {
+ { "date", cmd__date },
+ { "delete-gpgsig", cmd__delete_gpgsig },
+ { "delta", cmd__delta },
++ { "diff-process-backend", cmd__diff_process_backend },
+ { "dir-iterator", cmd__dir_iterator },
+ { "drop-caches", cmd__drop_caches },
+ { "dump-cache-tree", cmd__dump_cache_tree },
+
+ ## t/helper/test-tool.h ##
+@@ t/helper/test-tool.h: int cmd__csprng(int argc, const char **argv);
+ int cmd__date(int argc, const char **argv);
+ int cmd__delta(int argc, const char **argv);
+ int cmd__delete_gpgsig(int argc, const char **argv);
++int cmd__diff_process_backend(int argc, const char **argv);
+ int cmd__dir_iterator(int argc, const char **argv);
+ int cmd__drop_caches(int argc, const char **argv);
+ int cmd__dump_cache_tree(int argc, const char **argv);
## t/meson.build ##
@@ t/meson.build: integration_tests = [
@@ t/t4080-diff-process.sh (new)
+
+. ./test-lib.sh
+
-+if test_have_prereq PYTHON
-+then
-+ PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
-+fi
++# See t/helper/test-diff-process-backend.c for the backend implementation
++# and available --mode= options.
+
-+#
-+# A single parametric diff process.
-+# Usage: diff-process-backend --mode=<mode> [--log=<path>]
-+#
-+# Modes:
-+# whole-file - report all lines as changed (default)
-+# fixed-hunk - always report hunk 5 2 5 2
-+# bad-hunk - report out-of-bounds hunk 999 1 999 1
-+# bad-sync - report hunk with mismatched unchanged totals
-+# overlap - report two overlapping hunks
-+# no-hunks - return no hunks (files considered equivalent)
-+# error - return status=error for every request
-+# abort - return status=abort for every request
-+# crash - read one request then exit without responding
-+#
-+setup_backend () {
-+ cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
-+ import sys, os
-+
-+ def read_pkt():
-+ hdr = sys.stdin.buffer.read(4)
-+ if len(hdr) < 4: return None
-+ length = int(hdr, 16)
-+ if length == 0: return ""
-+ data = sys.stdin.buffer.read(length - 4)
-+ return data.decode().rstrip("\n")
-+
-+ def write_pkt(line):
-+ data = (line + "\n").encode()
-+ sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
-+ sys.stdout.buffer.flush()
-+
-+ def write_flush():
-+ sys.stdout.buffer.write(b"0000")
-+ sys.stdout.buffer.flush()
-+
-+ def read_content():
-+ chunks = []
-+ while True:
-+ hdr = sys.stdin.buffer.read(4)
-+ if len(hdr) < 4: break
-+ length = int(hdr, 16)
-+ if length == 0: break
-+ chunks.append(sys.stdin.buffer.read(length - 4))
-+ return b"".join(chunks)
-+
-+ mode = "whole-file"
-+ logfile = None
-+ for arg in sys.argv[1:]:
-+ if arg.startswith("--mode="):
-+ mode = arg[7:]
-+ elif arg.startswith("--log="):
-+ logfile = open(arg[6:], "a")
-+
-+ def log(msg):
-+ if logfile:
-+ logfile.write(msg + "\n")
-+ logfile.flush()
-+
-+ # Handshake
-+ assert read_pkt() == "git-diff-client"
-+ assert read_pkt() == "version=1"
-+ read_pkt()
-+ write_pkt("git-diff-server")
-+ write_pkt("version=1")
-+ write_flush()
-+ while True:
-+ p = read_pkt()
-+ if p == "": break
-+ write_pkt("capability=hunks")
-+ write_flush()
-+
-+ log("ready")
-+
-+ while True:
-+ cmd = None
-+ pathname = None
-+ while True:
-+ p = read_pkt()
-+ if p is None: sys.exit(0)
-+ if p == "": break
-+ if p.startswith("command="): cmd = p.split("=",1)[1]
-+ if p.startswith("pathname="): pathname = p.split("=",1)[1]
-+ if cmd is None: sys.exit(0)
-+ old = read_content()
-+ new = read_content()
-+ old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
-+ new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
-+ log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
-+
-+ if mode == "error":
-+ write_flush()
-+ write_pkt("status=error")
-+ write_flush()
-+ continue
-+
-+ if mode == "abort":
-+ write_flush()
-+ write_pkt("status=abort")
-+ write_flush()
-+ continue
-+
-+ if mode == "crash":
-+ sys.exit(1)
-+
-+ if cmd == "hunks":
-+ if mode == "fixed-hunk":
-+ write_pkt("hunk 5 2 5 2")
-+ elif mode == "bad-hunk":
-+ write_pkt("hunk 999 1 999 1")
-+ elif mode == "bad-sync":
-+ write_pkt("hunk 1 2 1 1")
-+ elif mode == "overlap":
-+ write_pkt("hunk 1 5 1 5")
-+ write_pkt("hunk 3 2 3 2")
-+ elif mode == "no-hunks":
-+ pass
-+ else:
-+ ol = old.count(b"\n")
-+ nl = new.count(b"\n")
-+ write_pkt(f"hunk 1 {ol} 1 {nl}")
-+ write_flush()
-+ write_pkt("status=success")
-+ write_flush()
-+ else:
-+ write_flush()
-+ write_pkt("status=error")
-+ write_flush()
-+ PYEOF
-+ write_script diff-process-backend <<-SHEOF
-+ exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
-+ SHEOF
-+}
-+
-+BACKEND="./diff-process-backend"
++BACKEND="test-tool diff-process-backend"
+
-+test_expect_success PYTHON 'setup' '
-+ setup_backend &&
++test_expect_success 'setup' '
+ echo "*.c diff=cdiff" >.gitattributes &&
+ git add .gitattributes &&
+
@@ t/t4080-diff-process.sh (new)
+ git add worddiff.c &&
+
+ # newfile.c: single-line function, value changes 42 -> 99.
-+ # Used by: new file, --exit-code, multiple drivers.
++ # Used by: modified file, --exit-code, multiple drivers.
+ cat >newfile.c <<-\EOF &&
+ int new_func(void) { return 42; }
+ EOF
@@ t/t4080-diff-process.sh (new)
+# Core behavior: the tool controls which lines are marked as changed.
+#
+
-+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
++test_expect_success 'diff process hunk boundaries affect output' '
+ # The file has changes at lines 5-6 and 9-10, but fixed-hunk
+ # only reports lines 5-6 as changed. Lines 9-10 should not
+ # appear as changed in the output.
@@ t/t4080-diff-process.sh (new)
+ test_grep ! "^+NEW10" actual
+'
+
-+test_expect_success PYTHON 'diff process works with new file' '
-+ rm -f backend.log &&
++test_expect_success 'diff process works with modified file' '
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ diff -- newfile.c >actual 2>stderr &&
+ test_grep "return 99" actual &&
@@ t/t4080-diff-process.sh (new)
+ test_must_be_empty stderr
+'
+
-+test_expect_success PYTHON 'diff process works with added file (empty old side)' '
++test_expect_success 'diff process works with added file (empty old side)' '
+ cat >added.c <<-\EOF &&
+ int added(void) { return 1; }
+ EOF
+ git add added.c &&
+
-+ rm -f backend.log &&
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ diff --cached -- added.c >actual 2>stderr &&
+ test_grep "added" actual &&
@@ t/t4080-diff-process.sh (new)
+ test_must_be_empty stderr
+'
+
-+test_expect_success PYTHON 'diff process skipped for binary files' '
++test_expect_success 'diff process works with deleted file (empty new side)' '
++ git add added.c &&
++ git commit -m "commit added.c" &&
++ git rm added.c &&
++
++ test_when_finished "rm -f backend.log" &&
++ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
++ diff --cached -- added.c >actual 2>stderr &&
++ test_grep "deleted file" actual &&
++ test_grep "pathname=added.c" backend.log &&
++ test_must_be_empty stderr
++'
++
++test_expect_success 'diff process skipped for binary files' '
+ printf "\\0binary" >binary.c &&
+ git add binary.c &&
+ git commit -m "add binary" &&
+ printf "\\0changed" >binary.c &&
+
-+ rm -f backend.log &&
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ diff -- binary.c >actual &&
+ test_grep "Binary files" actual &&
+ test_path_is_missing backend.log
+'
+
-+test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
++test_expect_success 'diff process not consulted for unmatched driver' '
+ echo "not tracked by cdiff" >unmatched.txt &&
+ git add unmatched.txt &&
+ git commit -m "add unmatched.txt" &&
+
+ echo "modified" >unmatched.txt &&
+
-+ rm -f backend.log &&
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ diff -- unmatched.txt >actual &&
+ test_grep "modified" actual &&
+ test_path_is_missing backend.log
+'
+
-+test_expect_success PYTHON 'multiple drivers use separate processes' '
++test_expect_success 'multiple drivers use separate processes' '
+ echo "*.h diff=hdiff" >>.gitattributes &&
+ git add .gitattributes &&
+
@@ t/t4080-diff-process.sh (new)
+ int header(void) { return 2; }
+ EOF
+
-+ rm -f backend-c.log backend-h.log &&
++ test_when_finished "rm -f backend-c.log backend-h.log" &&
+ git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
+ -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
+ diff -- newfile.c multi.h >actual 2>stderr &&
@@ t/t4080-diff-process.sh (new)
+ test_must_be_empty stderr
+'
+
-+test_expect_success PYTHON 'diff process works alongside textconv' '
++test_expect_success 'diff process works alongside textconv' '
+ write_script uppercase-filter <<-\EOF &&
+ tr "a-z" "A-Z" <"$1"
+ EOF
@@ t/t4080-diff-process.sh (new)
+ goodbye world
+ EOF
+
-+ rm -f backend.log &&
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.textconv="./uppercase-filter" \
+ -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ diff -- textconv.c >actual 2>stderr &&
@@ t/t4080-diff-process.sh (new)
+# Downstream features: word diff, log, equivalent files, exit code.
+#
+
-+test_expect_success PYTHON 'diff process with --word-diff' '
-+ rm -f backend.log &&
++test_expect_success 'diff process with --word-diff' '
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ diff --word-diff worddiff.c >actual 2>stderr &&
+ test_grep "\[-1;-\]" actual &&
@@ t/t4080-diff-process.sh (new)
+ test_must_be_empty stderr
+'
+
-+test_expect_success PYTHON 'diff process works with git log -p' '
++test_expect_success 'diff process works with git log -p' '
+ # With no-hunks mode, the tool says the files are equivalent,
+ # so log -p should show the commit but no diff content.
-+ rm -f backend.log &&
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+ log -1 -p -- logtest.c >actual 2>stderr &&
+ test_grep "change logtest.c" actual &&
@@ t/t4080-diff-process.sh (new)
+ test_must_be_empty stderr
+'
+
-+test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
++test_expect_success 'diff process no hunks suppresses diff output' '
+ cat >nohunks.c <<-\EOF &&
+ int zero(void) { return 0; }
+ EOF
@@ t/t4080-diff-process.sh (new)
+ test_must_be_empty actual
+'
+
-+test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
++test_expect_success 'diff process no hunks with --exit-code returns success' '
+ git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+ diff --exit-code nohunks.c
+'
+
-+test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
++test_expect_success 'diff process with --exit-code and hunks returns failure' '
+ test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
+ diff --exit-code newfile.c
+'
@@ t/t4080-diff-process.sh (new)
+# Bypass mechanisms: flags and commands that skip the diff process.
+#
+
-+test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
-+ rm -f backend.log &&
++test_expect_success 'diff process bypassed by --diff-algorithm' '
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ diff --diff-algorithm=patience worddiff.c >actual &&
+ test_grep "return 999" actual &&
+ test_path_is_missing backend.log
+'
+
-+test_expect_success PYTHON 'diff process not used by --stat' '
-+ rm -f backend.log &&
++test_expect_success 'diff process not used by --stat' '
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ diff --stat worddiff.c >actual &&
+ test_grep "worddiff.c" actual &&
@@ t/t4080-diff-process.sh (new)
+# Error handling and fallback.
+#
+
-+test_expect_success PYTHON 'diff process fallback on tool error status' '
-+ rm -f backend.log &&
++test_expect_success 'diff process fallback on tool error status' '
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+ diff boundary.c >actual 2>stderr &&
+ # Fallback produces the full builtin diff (both change regions).
@@ t/t4080-diff-process.sh (new)
+ test_grep "diff process.*failed" stderr
+'
+
-+test_expect_success PYTHON 'diff process error keeps tool available for next file' '
-+ rm -f backend.log &&
++test_expect_success 'diff process error keeps tool available for next file' '
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+ diff -- one.c two.c >actual 2>stderr &&
+ # Unlike abort, error keeps the tool available: both files
@@ t/t4080-diff-process.sh (new)
+ test_grep "pathname=one.c" backend.log &&
+ test_grep "pathname=two.c" backend.log &&
+ test_grep "return 10" actual &&
-+ test_grep "return 20" actual
++ test_grep "return 20" actual &&
++ test_grep "diff process.*failed" stderr
+'
+
-+test_expect_success PYTHON 'diff process abort disables for session' '
-+ rm -f backend.log &&
++test_expect_success 'diff process abort disables for session' '
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
-+ diff -- one.c two.c >actual &&
++ diff -- one.c two.c >actual 2>stderr &&
+ # Both files should still produce diff output via fallback.
+ test_grep "return 10" actual &&
+ test_grep "return 20" actual &&
+ # The tool aborts on the first file and git clears its
+ # capability. The second file never contacts the tool.
+ test_grep "pathname=one.c" backend.log &&
-+ test_grep ! "pathname=two.c" backend.log
++ test_grep ! "pathname=two.c" backend.log &&
++ test_must_be_empty stderr
+'
+
-+test_expect_success PYTHON 'diff process fallback on tool crash' '
++test_expect_success 'diff process fallback on tool crash' '
+ git -c diff.cdiff.process="$BACKEND --mode=crash" \
+ diff boundary.c >actual 2>stderr &&
+ test_grep "^-OLD5" actual &&
@@ t/t4080-diff-process.sh (new)
+ test_grep "diff process.*failed" stderr
+'
+
-+test_expect_success PYTHON 'diff process startup failure only warns once' '
++test_expect_success 'diff process startup failure only warns once' '
+ git -c diff.cdiff.process="/nonexistent/tool" \
+ diff -- one.c two.c >actual 2>stderr &&
+ # Both files produce diff output via fallback.
@@ t/t4080-diff-process.sh (new)
+ test_line_count = 1 warnings
+'
+
-+test_expect_success PYTHON 'diff process fallback on bad hunks' '
++
++test_expect_success 'diff process fallback on bad hunks' '
+ git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+ diff boundary.c >actual 2>stderr &&
+ test_grep "^-OLD5" actual &&
+ test_grep "^+NEW5" actual &&
+ test_grep "^-OLD9" actual &&
+ test_grep "^+NEW9" actual &&
-+ # Invalid hunks are caught by xdiff validation, not the
-+ # protocol layer, so no warning is emitted.
-+ test_must_be_empty stderr
++ test_grep "exceeds.*lines" stderr
+'
+
-+test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
++test_expect_success 'diff process fallback on mismatched unchanged totals' '
+ cat >synctest.c <<-\EOF &&
+ line1
+ line2
@@ t/t4080-diff-process.sh (new)
+ # The synchronization invariant fails and git falls back.
+ git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
+ diff synctest.c >actual 2>stderr &&
-+ test_grep "changed" actual
++ test_grep "changed" actual &&
++ test_grep "unchanged line count mismatch" stderr
+'
+
-+test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
++test_expect_success 'diff process fallback on overlapping hunks' '
+ # boundary.c has 10 lines, so both hunks are in bounds
+ # but they overlap at lines 3-5, triggering the ordering check.
+ git -c diff.cdiff.process="$BACKEND --mode=overlap" \
+ diff boundary.c >actual 2>stderr &&
-+ test_grep "NEW5" actual
++ test_grep "NEW5" actual &&
++ test_grep "overlaps with previous" stderr
++'
++
++test_expect_success 'diff process fallback on malformed hunk line' '
++ git -c diff.cdiff.process="$BACKEND --mode=bad-parse" \
++ diff boundary.c >actual 2>stderr &&
++ test_grep "^-OLD5" actual &&
++ test_grep "^+NEW5" actual
++'
++
++test_expect_success 'diff process skipped when tool omits capability' '
++ git -c diff.cdiff.process="$BACKEND --mode=no-cap" \
++ diff boundary.c >actual 2>stderr &&
++ test_grep "^-OLD5" actual &&
++ test_grep "^+NEW5" actual &&
++ test_must_be_empty stderr
+'
+
+test_done
5: f4fd9aa682 ! 5: 6ec6716ea4 diff: bypass diff process with --no-ext-diff and in format-patch
@@ Commit message
external tool.
Document that --diff-algorithm also bypasses the diff process,
- since it sets ignore_driver_algorithm which diff_process_fill_hunks
- already checks.
+ since it forces the builtin algorithm.
Signed-off-by: Michael Montalbo <[email protected]>
@@ diff.h: struct diff_flags {
/**
## t/t4080-diff-process.sh ##
-@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
+@@ t/t4080-diff-process.sh: test_expect_success 'diff process bypassed by --diff-algorithm' '
test_path_is_missing backend.log
'
-+test_expect_success PYTHON 'diff process bypassed by --no-ext-diff' '
-+ rm -f backend.log &&
++test_expect_success 'diff process bypassed by --no-ext-diff' '
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ diff --no-ext-diff worddiff.c >actual &&
+ test_grep "return 999" actual &&
+ test_path_is_missing backend.log
+'
+
-+test_expect_success PYTHON 'diff process not used by format-patch' '
-+ rm -f backend.log &&
++test_expect_success 'diff process not used by format-patch' '
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ format-patch -1 --stdout -- logtest.c >actual &&
+ test_grep "return 2" actual &&
+ test_path_is_missing backend.log
+'
+
- test_expect_success PYTHON 'diff process not used by --stat' '
- rm -f backend.log &&
+ test_expect_success 'diff process not used by --stat' '
+ test_when_finished "rm -f backend.log" &&
git -c diff.cdiff.process="$BACKEND --log=backend.log" \
6: 370e766978 ! 6: 3dadafa1bc blame: consult diff process for no-hunk detection
@@ Commit message
The consultation happens at the pass_blame_to_parent() callsite
using diff_process_fill_hunks(), matching how builtin_diff() in
diff.c uses the same function. A new diff_hunks_xpp() variant
- accepts a pre-populated xpparam_t for this callsite, while the
- existing diff_hunks() retains its original signature and behavior.
- The copy-detection callsite is unaffected since it does not use
- the diff process.
+ accepts a pre-populated xpparam_t so callers can pass external
+ hunks, while the existing diff_hunks() retains its original
+ signature and behavior. The copy-detection callsite is
+ unaffected since it does not use the diff process.
The subprocess is long-running (one startup cost amortized
across the blame traversal), but each commit in the file's
@@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
parent, target, 0);
## t/t4080-diff-process.sh ##
-@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
- test_grep "NEW5" actual
+@@ t/t4080-diff-process.sh: test_expect_success 'diff process skipped when tool omits capability' '
+ test_must_be_empty stderr
'
+#
+# Blame integration.
+#
+
-+test_expect_success PYTHON 'blame uses tool-provided hunks' '
++test_expect_success 'blame uses tool-provided hunks' '
+ cat >blame-hunk.c <<-\EOF &&
+ line1
+ line2
@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
+ test_grep "$CHANGE" line6
+'
+
-+test_expect_success PYTHON 'blame skips commits with no hunks from diff process' '
++test_expect_success 'blame skips commits with no hunks from diff process' '
+ cat >blame.c <<-\EOF &&
-+ int main(void)
-+ {
-+ return 0;
++ int main(void) {
++ return 0;
+ }
+ EOF
+ git add blame.c &&
@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
+ cat >blame.c <<-\EOF &&
+ int main(void)
+ {
-+ return 0;
++ return 0;
+ }
+ EOF
+ git add blame.c &&
@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
+ test_grep "$ORIG_COMMIT" with
+'
+
-+test_expect_success PYTHON 'blame --no-ext-diff bypasses diff process' '
-+ rm -f backend.log &&
++test_expect_success 'blame --no-ext-diff bypasses diff process' '
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+ blame --no-ext-diff blame.c >actual &&
+ # Without the process, blame attributes the reformat commit normally.
@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
+ test_path_is_missing backend.log
+'
+
-+test_expect_success PYTHON 'blame --no-ext-diff uses builtin hunks' '
++test_expect_success 'blame --no-ext-diff uses builtin hunks' '
+ # fixed-hunk mode would narrow blame to lines 5-6, but
+ # --no-ext-diff should bypass it and use the builtin diff.
-+ rm -f backend.log &&
++ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+ blame --no-ext-diff blame-hunk.c >actual &&
+ # Builtin diff attributes lines 9-10 to the change commit.Signature · 1 line
gitgitgadget
PATCH v4 1/6 xdiff: support external hunks via xpparam_t
From: Michael Montalbo <[email protected]>
Add two new xpparam_t fields (external_hunks, external_hunks_nr) that let callers supply pre-computed hunks. When set, xdl_diff() populates the changed[] arrays from these hunks instead of running the diff algorithm, then continues through compaction and emission as usual.
Validate supplied hunks before use: reject out-of-bounds line numbers, overlapping or out-of-order hunks, negative counts, and violations of the synchronization invariant (unchanged line counts must match between files). On validation failure, fall back to the builtin diff algorithm; this re-runs xdl_prepare_env() since the first call may have dirtied the changed[] arrays.
Skip trim_common_tail() in xdi_diff() when external hunks are present, since external hunks reference line numbers in the original content.
--- xdiff-interface.c | 7 ++- xdiff/xdiff.h | 14 ++++++ xdiff/xdiffi.c | 123 +++++++++++++++++++++++++++++++++++++++++++++- xdiff/xprepare.c | 10 ++++ xdiff/xprepare.h | 1 + 5 files changed, 152 insertions(+), 3 deletions(-)
xdiff-interface.c +6 -1
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t coif (mf1->size >MAX_XDIFF_SIZE || mf2->size >MAX_XDIFF_SIZE )return -1 ; -if (!xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT )) +/* +* External hunks reference line numbers in the original content; +* trimming the tail would change line counts and invalidate them. +*/ +if (!xpp->external_hunks && + !xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT ))trim_common_tail (&a, &b);return xdl_diff (&a, &b, xpp, xecfg, xecb);
xdiff/xdiff.h +14 -0
@@ -78,6 +78,16 @@ typedef struct s_mmbuffer {long size; } mmbuffer_t; +/* +* Hunk descriptor for externally computed diffs. +* Line numbers are 1-based; a start of 0 is accepted when +* count is 0 (empty file side, matching git diff output). +*/ +struct xdl_hunk { +long old_start, old_count; +long new_start, new_count; +}; + typedef struct s_xpparam { unsigned long flags;
@@ -88,6 +98,10 @@ typedef struct s_xpparam {/* See Documentation/diff-options.adoc. */ char **anchors;size_t anchors_nr; + +/* Externally computed hunks: bypass the diff algorithm. Owned by caller. */ +struct xdl_hunk *external_hunks; +size_t external_hunks_nr; } xpparam_t; typedef struct s_xdemitcb {
xdiff/xdiffi.c +121 -2
137 lines not on this page · raw message
xdiff/xprepare.c +10 -0
@@ -471,3 +471,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,return 0 ; } + +/* +* Reset the changed[] array so that no lines are marked as changed. +* Also clears the sentinel slots at changed[-1] and changed[nrec] +* that xdl_change_compact() relies on during backward scans. +*/ +void xdl_clear_changed (xdfile_t *xdf) +{ +memset (xdf->changed -1 ,0 , (xdf->nrec +2 ) *sizeof (bool )); +}
xdiff/xprepare.h +1 -0
@@ -28,6 +28,7 @@int xdl_prepare_env (mmfile_t *mf1,mmfile_t *mf2,xpparam_t const *xpp,xdfenv_t *xe);void xdl_free_env (xdfenv_t *xe); +void xdl_clear_changed (xdfile_t *xdf);
Signature · 1 line
gitgitgadget
PATCH v4 2/6 userdiff: add diff.<driver>.process config
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add the process field to struct userdiff_driver and teach the config parser to populate it from diff.<driver>.process.
--- userdiff.c | 7 +++++++ userdiff.h | 2 ++ 2 files changed, 9 insertions(+)
userdiff.c +7 -0
@@ -509,6 +509,13 @@ int userdiff_config(const char *k, const char *v) drv->algorithm = drv->algorithm_owned;return ret; } +if (!strcmp (type,"process" )) { +int ret; +FREE_AND_NULL (drv->process_owned); + ret =git_config_string (&drv->process_owned, k, v); + drv->process = drv->process_owned; +return ret; + }return 0 ; }
userdiff.h +2 -0
@@ -31,6 +31,8 @@ struct userdiff_driver {char *textconv_owned;struct notes_cache *textconv_cache;int textconv_want_cache; +const char *process; +char *process_owned; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,
Signature · 1 line
gitgitgadget
PATCH v4 3/6 sub-process: separate process lifecycle from hashmap management
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
subprocess_start() and subprocess_stop() couple two concerns:
managing a child process (setup, handshake, teardown) and managing a hashmap that indexes running processes by command string. The hashmap suits callers like convert.c where many files may share one filter process looked up by name, but callers that manage process lifetime through their own data structures do not need it.
Extract subprocess_start_command() and subprocess_stop_command() so callers can reuse the child process setup and handshake machinery without maintaining a hashmap. subprocess_start() and subprocess_stop() become thin wrappers that add hashmap operations on top.
--- sub-process.c | 28 +++++++++++++++++++++++----- sub-process.h | 9 ++++++++- 2 files changed, 31 insertions(+), 6 deletions(-)
sub-process.c +23 -5
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)return (len <0 ) ? len :0 ; } -void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry) +void subprocess_stop_command (struct subprocess_entry *entry) {if (!entry)return ;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry) entry->process.clean_on_exit =0 ;kill (entry->process.pid,SIGTERM );finish_command (&entry->process); +} +void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry) +{ +if (!entry) +return ; + +subprocess_stop_command (entry);hashmap_remove (hashmap, &entry->ent,NULL ); }
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)finish_command (process); } -int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd, +int subprocess_start_command (struct subprocess_entry *entry,const char *cmd,subprocess_start_fn startfn) {int err;
@@ -96,15 +103,26 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, coreturn err; } -hashmap_entry_init (&entry->ent,strhash (cmd)); - err =startfn (entry);if (err) {error ("initialization for subprocess '%s' failed" , cmd); -subprocess_stop (hashmap, entry); +subprocess_stop_command (entry);return err; } +return 0 ; +} + +int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd, +subprocess_start_fn startfn) +{ +int err; + + err =subprocess_start_command (entry, cmd, startfn); +if (err) +return err; + +hashmap_entry_init (&entry->ent,strhash (cmd));hashmap_add (hashmap, &entry->ent);return 0 ; }
sub-process.h +8 -1
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,*/ typedef int (*subprocess_start_fn )(struct subprocess_entry *entry); -/* Start a subprocess and add it to the subprocess hashmap. */ +/* Start a subprocess and run the startfn (typically handshake). */ +int subprocess_start_command (struct subprocess_entry *entry,const char *cmd, +subprocess_start_fn startfn); + +/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */ int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd,subprocess_start_fn startfn); +/* Kill a subprocess. */ +void subprocess_stop_command (struct subprocess_entry *entry); +/* Kill a subprocess and remove it from the subprocess hashmap. */ void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry);
Signature · 1 line
gitgitgadget
PATCH v4 4/6 diff: add long-running diff process via diff.<driver>.process
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add support for external diff processes that communicate via the long-running process protocol (pkt-line over stdin/stdout).
A diff process is configured per userdiff driver:
[diff "cdiff"]
process = /path/to/diff-tool
The tool provides custom line-matching: it receives file pairs and returns hunks that reference line numbers in the content. When textconv is also configured, the tool receives the textconv-transformed content. The tool controls which lines are marked as changed while the display shows the file content. Patch output features (word diff, function context, color) work normally; --stat uses its own diff codepath and never consults the diff process.
The handshake negotiates version=1 and capability=hunks. Per-file requests send command=hunks, pathname, and both file contents as packetized data. The tool responds with hunk lines and a status packet (success, error, or abort). On error, Git warns and falls back to the builtin diff algorithm for that file. On abort, Git silently falls back for the current file and stops sending further requests to the tool for the remainder of the session.
When the tool returns no hunks followed by status=success, Git treats the file as having no changes and produces no diff output. This also means --exit-code reports no changes for that file.
The subprocess is stored on the userdiff_driver struct and launched on first use. If the process fails to start, the handshake fails, or a communication error occurs mid-stream, the failure is cached on the driver to avoid retrying and re-warning on every subsequent file.
diff_process_fill_hunks() is the sole public entry point. It handles driver lookup, flag checks, subprocess management, and error reporting, returning an enum that lets callers distinguish "hunks populated" from "files equivalent" from "not applicable" from "tool failure."
--- Documentation/config/diff.adoc | 5 + Documentation/gitattributes.adoc | 143 +++++++++ Makefile | 2 + diff-process.c | 297 ++++++++++++++++++ diff-process.h | 39 +++ diff.c | 13 + diff.h | 3 + meson.build | 1 + t/helper/meson.build | 1 + t/helper/test-diff-process-backend.c | 299 ++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/meson.build | 1 + t/t4080-diff-process.sh | 432 +++++++++++++++++++++++++++ userdiff.h | 3 + 15 files changed, 1241 insertions(+) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100644 t/helper/test-diff-process-backend.c create mode 100755 t/t4080-diff-process.sh
Documentation/config/diff.adoc +5 -0
@@ -218,6 +218,11 @@ endif::git-diff[] Set this option to `true` to make the diff driver cache the text conversion outputs. See linkgit:gitattributes[5] for details. +`diff.<driver>.process`:: + The command to run as a long-running diff process that + provides hunks to Git's diff pipeline. + See linkgit:gitattributes[5] for details. + `diff.indentHeuristic`:: Set this option to `false` to disable the default heuristics that shift diff hunk boundaries to make patches easier to read.
Documentation/gitattributes.adoc +143 -0
149 lines not on this page · raw message
Makefile +2 -0
@@ -811,6 +811,7 @@ TEST_BUILTINS_OBJS += test-csprng.o TEST_BUILTINS_OBJS += test-date.o TEST_BUILTINS_OBJS += test-delete-gpgsig.o TEST_BUILTINS_OBJS += test-delta.o +TEST_BUILTINS_OBJS += test-diff-process-backend.o TEST_BUILTINS_OBJS += test-dir-iterator.o TEST_BUILTINS_OBJS += test-drop-caches.o TEST_BUILTINS_OBJS += test-dump-cache-tree.o
@@ -1140,6 +1141,7 @@ LIB_OBJS += diff-delta.o LIB_OBJS += diff-merges.o LIB_OBJS += diff-lib.o LIB_OBJS += diff-no-index.o +LIB_OBJS += diff-process.o LIB_OBJS += diff.o LIB_OBJS += diffcore-break.o LIB_OBJS += diffcore-delta.o
diff-process.c new file +297 -0
297 lines not on this page · raw message
diff-process.h new file +39 -0
@@ -0,0 +1,39 @@ +#ifndef DIFF_PROCESS_H +#define DIFF_PROCESS_H + +#include "xdiff/xdiff.h" + +struct diff_options ; + +enum diff_process_result { +DIFF_PROCESS_ERROR =-1 ,/* tool failure: warned, fell back */ +DIFF_PROCESS_OK =0 ,/* hunks populated in xpp */ +DIFF_PROCESS_SKIP ,/* no process configured: use builtin */ +DIFF_PROCESS_EQUIVALENT ,/* tool says files are equivalent */ +}; + +/* +* Consult the diff process configured for 'path' and populate +* xpp->external_hunks with the returned hunks. +* +* Handles driver lookup, flag checks (--no-ext-diff, +* --diff-algorithm), subprocess management, and error reporting. +* +* Returns DIFF_PROCESS_OK when hunks are populated in xpp. +* The caller owns xpp->external_hunks and must free() it. +* +* Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks +* (files are considered identical); caller should skip diff/blame. +* Returns DIFF_PROCESS_SKIP when no process applies; caller +* should use the builtin diff algorithm. +* Returns DIFF_PROCESS_ERROR on tool failure (already warned); +* caller should fall back to the builtin diff algorithm. +*/ +enum diff_process_result diff_process_fill_hunks ( +struct diff_options *diffopt, +const char *path, +const mmfile_t *file_a, +const mmfile_t *file_b, +xpparam_t *xpp); + +#endif /* DIFF_PROCESS_H */
diff.c +13 -0
@@ -25,6 +25,7 @@#include "utf8.h" #include "odb.h" #include "userdiff.h" +#include "diff-process.h" #include "submodule.h" #include "hashmap.h" #include "mem-pool.h"
@@ -4054,6 +4055,17 @@ static void builtin_diff(const char *name_a, xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; + +if (diff_process_fill_hunks (o, name_a, + &mf1, &mf2, &xpp) + ==DIFF_PROCESS_EQUIVALENT ) { +if (textconv_one) +free (mf1.ptr); +if (textconv_two) +free (mf2.ptr); + goto free_ab_and_return; + } + xecfg.ctxlen = o->context; xecfg.interhunkctxlen = o->interhunkcontext; xecfg.flags =XDL_EMIT_FUNCNAMES ;
@@ -4134,6 +4146,7 @@ static void builtin_diff(const char *name_a, }else if (xdi_diff_outf (&mf1, &mf2,NULL , fn_out_consume, &ecbdata, &xpp, &xecfg))die ("unable to generate diff for %s" , one->path); +free (xpp.external_hunks);if (o->word_diff)free_diff_words_data (&ecbdata);if (textconv_one)
diff.h +3 -0
@@ -173,6 +173,9 @@ struct diff_flags {*/ unsigned allow_external; +/** Disables diff.<driver>.process. */ +unsigned no_diff_process; +/** * For communication between the calling program and the options parser; * tell the calling program to signal the presence of difference using
meson.build +1 -0
@@ -328,6 +328,7 @@ libgit_sources = [ 'diff-merges.c', 'diff-lib.c', 'diff-no-index.c', + 'diff-process.c', 'diff.c', 'diffcore-break.c', 'diffcore-delta.c',
t/helper/meson.build +1 -0
@@ -12,6 +12,7 @@ test_tool_sources = [ 'test-date.c', 'test-delete-gpgsig.c', 'test-delta.c', + 'test-diff-process-backend.c', 'test-dir-iterator.c', 'test-drop-caches.c', 'test-dump-cache-tree.c',
t/helper/test-diff-process-backend.c new file +299 -0
299 lines not on this page · raw message
t/helper/test-tool.c +1 -0
@@ -22,6 +22,7 @@ static struct test_cmd cmds[] = { { "date", cmd__date }, { "delete-gpgsig", cmd__delete_gpgsig }, { "delta", cmd__delta }, + { "diff-process-backend", cmd__diff_process_backend }, { "dir-iterator", cmd__dir_iterator }, { "drop-caches", cmd__drop_caches }, { "dump-cache-tree", cmd__dump_cache_tree },
t/helper/test-tool.h +1 -0
@@ -15,6 +15,7 @@ int cmd__csprng(int argc, const char **argv);int cmd__date (int argc,const char **argv);int cmd__delta (int argc,const char **argv);int cmd__delete_gpgsig (int argc,const char **argv); +int cmd__diff_process_backend (int argc,const char **argv);int cmd__dir_iterator (int argc,const char **argv);int cmd__drop_caches (int argc,const char **argv);int cmd__dump_cache_tree (int argc,const char **argv);
t/meson.build +1 -0
@@ -512,6 +512,7 @@ integration_tests = [ 't4072-diff-max-depth.sh', 't4073-diff-stat-name-width.sh', 't4074-diff-shifted-matched-group.sh', + 't4080-diff-process.sh', 't4100-apply-stat.sh', 't4101-apply-nonl.sh', 't4102-apply-rename.sh',
t/t4080-diff-process.sh new file +432 -0
432 lines not on this page · raw message
userdiff.h +3 -0
@@ -3,6 +3,7 @@#include "notes-cache.h" +struct diff_subprocess ;struct index_state ;struct repository ;
@@ -33,6 +34,8 @@ struct userdiff_driver {int textconv_want_cache;const char *process;char *process_owned; +struct diff_subprocess *diff_subprocess; + unsigned diff_process_failed : 1; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,
Signature · 1 line
gitgitgadget
PATCH v4 5/6 diff: bypass diff process with --no-ext-diff and in format-patch
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Make --no-ext-diff disable diff.<driver>.process in addition to diff.<driver>.command. Although the two mechanisms work differently (command replaces Git's output, process feeds hunks back into the pipeline), both invoke external tools and --no-ext-diff means "no external tools."
Replace the OPT_BOOL for --ext-diff with an OPT_CALLBACK that sets both allow_external and no_diff_process, so a single option controls both. Passing --ext-diff explicitly clears no_diff_process, so a later --ext-diff overrides an earlier --no-ext-diff.
Disable the diff process unconditionally in format-patch so that generated patches are always based on the builtin diff algorithm and can be applied reliably by recipients who do not have the external tool.
Document that --diff-algorithm also bypasses the diff process, since it forces the builtin algorithm.
--- Documentation/diff-algorithm-option.adoc | 3 +++ Documentation/diff-options.adoc | 4 +++- builtin/log.c | 7 +++++++ diff.c | 16 ++++++++++++++-- diff.h | 4 +++- t/t4080-diff-process.sh | 16 ++++++++++++++++ 6 files changed, 46 insertions(+), 4 deletions(-)
Documentation/diff-algorithm-option.adoc +3 -0
@@ -18,3 +18,6 @@ For instance, if you configured the `diff.algorithm` variable to a non-default value and want to use the default one, then you have to use `--diff-algorithm=default` option. ++ +If you explicitly choose a diff algorithm, it also bypasses +`diff.<driver>.process` (see linkgit:gitattributes[5]).
Documentation/diff-options.adoc +3 -1
@@ -833,7 +833,9 @@ endif::git-format-patch[] to use this option with linkgit:git-log[1] and friends. `--no-ext-diff`:: - Disallow external diff drivers. + Disallow external diff helpers, including + `diff.<driver>.command` and `diff.<driver>.process` + (see linkgit:gitattributes[5]). `--textconv`:: `--no-textconv`::
builtin/log.c +7 -0
@@ -2217,6 +2217,13 @@ int cmd_format_patch(int argc,if (argc >1 )die (_ ("unrecognized argument: %s" ), argv[1 ]); +/* +* Disable diff.<driver>.process so that patches generated by +* format-patch are always based on the builtin diff algorithm +* and can be applied reliably. +*/ + rev.diffopt.flags.no_diff_process =1 ; +if (rev.diffopt.output_format &DIFF_FORMAT_NAME )die (_ ("--name-only does not make sense" ));if (rev.diffopt.output_format &DIFF_FORMAT_NAME_STATUS )
diff.c +14 -2
@@ -5936,6 +5936,17 @@ static int diff_opt_submodule(const struct option *opt,return 0 ; } +static int diff_opt_ext_diff (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; + +BUG_ON_OPT_ARG (arg); + options->flags.allow_external = !unset; + options->flags.no_diff_process = unset; +return 0 ; +} +static int diff_opt_textconv (const struct option *opt,const char *arg,int unset) {
@@ -6266,8 +6277,9 @@ struct option *add_diff_options(const struct option *opts, N_("exit with 1 if there were differences, 0 otherwise")),OPT_BOOL (0 ,"quiet" , &options->flags.quick,N_ ("disable all output of the program" )), -OPT_BOOL (0 ,"ext-diff" , &options->flags.allow_external, -N_ ("allow an external diff helper to be executed" )), +OPT_CALLBACK_F (0 ,"ext-diff" , options,NULL , +N_ ("allow an external diff helper to be executed" ), +PARSE_OPT_NOARG , diff_opt_ext_diff),OPT_CALLBACK_F (0 ,"textconv" , options,NULL ,N_ ("run external text conversion filters when comparing binary files" ), PARSE_OPT_NOARG, diff_opt_textconv),
diff.h +3 -1
@@ -173,7 +173,9 @@ struct diff_flags {*/ unsigned allow_external; -/** Disables diff.<driver>.process. */ +/** +* Disables diff.<driver>.process. Set by --no-ext-diff. +*/ unsigned no_diff_process;/**
t/t4080-diff-process.sh +16 -0
@@ -295,6 +295,22 @@ test_expect_success 'diff process bypassed by --diff-algorithm' ' test_path_is_missing backend.log ' +test_expect_success 'diff process bypassed by --no-ext-diff' ' + test_when_finished "rm -f backend.log" && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --no-ext-diff worddiff.c >actual && + test_grep "return 999" actual && + test_path_is_missing backend.log +' + +test_expect_success 'diff process not used by format-patch' ' + test_when_finished "rm -f backend.log" && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + format-patch -1 --stdout -- logtest.c >actual && + test_grep "return 2" actual && + test_path_is_missing backend.log +' + test_expect_success 'diff process not used by --stat' ' test_when_finished "rm -f backend.log" && git -c diff.cdiff.process="$BACKEND --log=backend.log" \
Signature · 1 line
gitgitgadget
PATCH v4 6/6 blame: consult diff process for no-hunk detection
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
When a diff process is configured via diff.<driver>.process, consult it during blame's per-commit diffing. If the process returns no hunks for a commit's changes to a file, treat the commit as having no changes, causing blame to attribute lines to earlier commits.
The consultation happens at the pass_blame_to_parent() callsite using diff_process_fill_hunks(), matching how builtin_diff() in diff.c uses the same function. A new diff_hunks_xpp() variant accepts a pre-populated xpparam_t so callers can pass external hunks, while the existing diff_hunks() retains its original signature and behavior. The copy-detection callsite is unaffected since it does not use the diff process.
The subprocess is long-running (one startup cost amortized across the blame traversal), but each commit in the file's history incurs a round-trip to the tool.
--- blame.c | 40 +++++++++++---- t/t4080-diff-process.sh | 105 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+), 9 deletions(-)
blame.c +31 -9
@@ -19,6 +19,8 @@#include "tag.h" #include "trace2.h" #include "blame.h" +#include "diff-process.h" +#include "xdiff-interface.h" #include "alloc.h" #include "commit-slab.h" #include "bloom.h"
@@ -314,17 +316,25 @@ static struct commit *fake_working_tree_commit(struct repository *r, -static int diff_hunks (mmfile_t *file_a,mmfile_t *file_b, -xdl_emit_hunk_consume_func_t hunk_func,void *cb_data,int xdl_opts) +static int diff_hunks_xpp (mmfile_t *file_a,mmfile_t *file_b, +xdl_emit_hunk_consume_func_t hunk_func, +void *cb_data,xpparam_t *xpp) { -xpparam_t xpp = {0 };xdemitconf_t xecfg = {0 };xdemitcb_t ecb = {NULL }; - xpp.flags = xdl_opts; xecfg.hunk_func = hunk_func; ecb.priv = cb_data; -return xdi_diff (file_a, file_b, &xpp, &xecfg, &ecb); +return xdi_diff (file_a, file_b, xpp, &xecfg, &ecb); +} + +static int diff_hunks (mmfile_t *file_a,mmfile_t *file_b, +xdl_emit_hunk_consume_func_t hunk_func,void *cb_data,int xdl_opts) +{ +xpparam_t xpp = {0 }; + + xpp.flags = xdl_opts; +return diff_hunks_xpp (file_a, file_b, hunk_func, cb_data, &xpp); }static const char *get_next_line (const char *start,const char *end)
@@ -1943,6 +1953,7 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb, struct blame_origin *parent, int ignore_diffs) {mmfile_t file_p, file_o; +xpparam_t xpp = {0 };struct blame_chunk_cb_data d;struct blame_entry *newdest =NULL ;
@@ -1961,10 +1972,21 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb, &sb->num_read_blob, ignore_diffs); sb->num_get_patch++; -if (diff_hunks (&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts)) -die ("unable to generate diff (%s -> %s)" , -oid_to_hex (&parent->commit->object.oid), -oid_to_hex (&target->commit->object.oid)); + xpp.flags = sb->xdl_opts; +/* +* If the diff process considers the files equivalent, +* skip the diff so blame looks past this commit. +*/ +if (diff_process_fill_hunks (&sb->revs->diffopt, target->path, + &file_p, &file_o, &xpp) + !=DIFF_PROCESS_EQUIVALENT ) { +if (diff_hunks_xpp (&file_p, &file_o, blame_chunk_cb, + &d, &xpp)) +die ("unable to generate diff (%s -> %s)" , +oid_to_hex (&parent->commit->object.oid), +oid_to_hex (&target->commit->object.oid)); + } +free (xpp.external_hunks);/* The rest are the same as the parent */ blame_chunk (&d.dstq, &d.srcq,INT_MAX , d.offset,INT_MAX ,0 , parent, target,0 );
t/t4080-diff-process.sh +105 -0
109 lines not on this page · raw message
Signature · 1 line
gitgitgadget
v5 cover diff: add diff.<driver>.process for external hunk providers
In reply to Michael Montalbo via GitGitGadgetLanguage-aware diff tools (e.g., Difftastic) and format-specific analyzers can produce better line matching than Git's builtin diff algorithm, but diff.<driver>.command replaces Git's diff output with the program's own output, so display features like word diff, function context, and color cannot operate on it; and because the program is consulted only for that patch output, blame, --stat, and git log -L fall back to Git's builtin line matching and cannot benefit from the tool at all.
This series adds diff.<driver>.process, a long-running subprocess protocol that lets an external tool control which lines Git considers changed while Git handles all output formatting. The protocol follows filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation, one process per Git invocation.
The tool receives both file versions and returns changed regions (line ranges in the old and new file). Git validates and feeds them into the xdiff pipeline in place of the builtin diff algorithm. When the tool returns no hunks, Git treats the files as having no changes, which propagates through patch output, the --stat summary, blame, and git log -L. The request also carries the two blobs' object names (old-oid/new-oid) so a tool can cache its analysis keyed on the pair.
* Patch 1: document how an external diff driver (diff.<driver>.command) relates to the rest of Git's diff features, so the contrast with the new process driver is clear.
* Patch 2: xdiff plumbing for externally supplied hunks.
* Patch 3: diff.<driver>.process config key.
* Patch 4: refactor subprocess API to separate process lifecycle from hashmap management, since the diff process stores its subprocess on the userdiff driver rather than in a hashmap.
* Patch 5: the main feature, including the old-oid/new-oid request metadata for blob-pair caching.
* Patch 6: bypass knobs (--no-ext-diff, format-patch).
* Patch 7: blame integration so the tool can declare commits as having no changes; introduces the shared xdi_diff_process() consult-then-diff helper that blame and git log -L both use.
* Patch 8: --stat/--numstat/--shortstat consult the tool, so the summary agrees with the patch output.
* Patch 9: git log -L range tracking consults the tool, so a reformat-only commit is dropped from the log rather than shown with an empty diff.
A "Which features consult the diff process" section in gitattributes(5) lays out, per feature, why each does or does not consult the process (patch output, blame, summary formats, and the -L line-range view do; pickaxe -G, patch-id, merge, range-diff, --check, and --raw do not, with reasons). Combined diffs (--cc) remain on the builtin algorithm and are noted as future work.
Changes since v4:
* New preparatory doc patch (patch 1): document how an external diff driver (diff.<driver>.command) relates to the rest of Git's diff features, so the contrast with the process driver added later in the series is explicit.
* New: --stat/--numstat/--shortstat now consult the process (patch 8). A file the tool reports as equivalent contributes no stat line, matching the empty patch git diff produces for it. Summary formats do not apply textconv, so the tool is fed raw content there, as the builtin --stat already counts raw lines.
* New: git log -L range tracking now consults the process (patch 9). Previously the tracking pass used the builtin diff while the displayed diff used the tool, so a reformat-only commit could be selected by tracking and then rendered with an empty diff. Tracking now agrees with the display and the commit is dropped.
* New: the request carries the old and new blob object names (old-oid/new-oid), so a tool can cache its line matching keyed on the blob pair. A side's oid is sent only when the tool receives that raw blob; it is omitted under textconv (which rewrites the bytes) and for a working-tree side with no stored object. This is where the process differs from diff.<name>.command, which never composes with textconv and so always feeds the raw blob its oid names.
* Refactor: blame and git log -L now consult the process through a single xdi_diff_process() helper instead of open-coding the consult-then-diff dance; builtin_diff() keeps its own path so it can short-circuit equivalent files before its funcname/word-diff setup. The whitespace bypass keys off the effective diff parameters (xpp), which removes blame's separate -w guard, and blame's diff_hunks() sheds an intermediate variant it no longer needs.
* Correctness: external-hunk validation now checks per-gap alignment, not just the total unchanged-line count. xdl_build_script() walks the two files in lockstep over unchanged lines, so a hunk set whose totals balance but whose per-gap line counts do not (e.g. hunk 1 1 3 1) previously passed validation and produced a corrupt diff, --stat, and blame attribution. Such responses are now rejected and Git falls back to the builtin diff.
* Robustness: hunk accumulation is bounded by the two files' combined line count, so a misbehaving tool that floods hunk lines cannot grow memory without bound before validation.
* Forward-compat: the hunk-line parser now ignores unknown trailing fields after the four counts, so a future protocol version can append a field without older clients rejecting it.
* Feature interactions: the whitespace-ignoring options (-w, --ignore-space-change, --ignore-blank-lines, ...), -I, and --anchored bypass the process (the tool is never told about them and could not honor them), and git blame -w does the same. A change that only adds or removes the trailing newline cannot be expressed as line hunks, so it also falls back to the builtin diff (preserving the "\ No newline at end of file" marker).
* Documentation: added the per-feature "Which features consult the diff process" section; documented that the process trusts the tool's notion of "unchanged" (it is not byte-validated, so like git diff -w such a patch may not apply against the old blob), that --exit-code/--quiet report success for tool- equivalent files, and that diff.<name>.command takes precedence when both it and .process are configured.
* Tests: t4080 grew coverage for the per-gap check, the hunk- flood cap, the whitespace/-w bypasses, the trailing-newline and added/deleted-file fallbacks, multi-hunk output through patch and --stat, --stat --exit-code, a mode-only change, and a mixed equivalent/changed multi-file diffstat.
Michael Montalbo (9):
gitattributes: document how external diff drivers relate to diff
features
xdiff: support external hunks via xpparam_t
userdiff: add diff.<driver>.process config
sub-process: separate process lifecycle from hashmap management
diff: add long-running diff process via diff.<driver>.process
diff: bypass diff process with --no-ext-diff and in format-patch
blame: consult diff process for no-hunk detection
diff: consult diff process for --stat counts
line-log: consult diff process for range tracking
Documentation/config/diff.adoc | 5 + Documentation/diff-algorithm-option.adoc | 3 + Documentation/diff-options.adoc | 4 +- Documentation/gitattributes.adoc | 274 +++++++ Makefile | 2 + blame.c | 24 +- builtin/log.c | 7 + diff-process.c | 529 +++++++++++++ diff-process.h | 75 ++ diff.c | 57 +- diff.h | 6 + line-log.c | 33 +- meson.build | 1 + sub-process.c | 28 +- sub-process.h | 9 +- t/helper/meson.build | 1 + t/helper/test-diff-process-backend.c | 381 +++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/meson.build | 1 + t/t4080-diff-process.sh | 950 +++++++++++++++++++++++ userdiff.c | 7 + userdiff.h | 5 + xdiff-interface.c | 7 +- xdiff/xdiff.h | 16 + xdiff/xdiffi.c | 84 +- xdiff/xprepare.c | 10 + xdiff/xprepare.h | 1 + 28 files changed, 2504 insertions(+), 18 deletions(-) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100644 t/helper/test-diff-process-backend.c create mode 100755 t/t4080-diff-process.sh
base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/2120
Range-diff vs v4:
-: ---------- > 1: 0fd994a3d3 gitattributes: document how external diff drivers relate to diff features
1: 03f261dfe2 ! 2: 2004502549 xdiff: support external hunks via xpparam_t
@@ Commit message
the diff algorithm, then continues through compaction and emission
as usual.
- Validate supplied hunks before use: reject out-of-bounds line
- numbers, overlapping or out-of-order hunks, negative counts, and
- violations of the synchronization invariant (unchanged line counts
- must match between files). On validation failure, fall back to
- the builtin diff algorithm; this re-runs xdl_prepare_env() since
- the first call may have dirtied the changed[] arrays.
+ Validate supplied hunks before use. Out-of-bounds line numbers,
+ overlapping or out-of-order hunks, and misaligned unchanged runs are
+ treated as a malformed tool response: xdl_populate_hunks_from_external()
+ warns, returns -1, and xdl_diff() falls back to the builtin diff
+ algorithm for that file. The run of unchanged lines between two hunks
+ (and before the first and after the last) must be the same length on
+ both sides; xdl_build_script() walks the two files in lockstep over
+ unchanged lines, so a balanced total is not enough. Non-negative
+ counts and 1-based starts are instead caller preconditions, checked
+ with BUG(), since the caller normalizes hunks before this point.
+
+ On rejection xdl_diff() frees the environment it prepared and falls
+ through to xdl_do_diff(), which prepares a fresh one for the builtin
+ pass.
Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
@@ xdiff/xdiff.h: typedef struct s_mmbuffer {
} mmbuffer_t;
+/*
-+ * Hunk descriptor for externally computed diffs.
-+ * Line numbers are 1-based; a start of 0 is accepted when
-+ * count is 0 (empty file side, matching git diff output).
++ * Hunk descriptor for externally computed diffs, in xdiff's own
++ * coordinates: line numbers are 1-based and a hunk's start is the
++ * first line it covers. A caller translates any external "empty side"
++ * idiom (such as git diff's start-0/count-0) to a 1-based start before
++ * handing hunks over.
+ */
+struct xdl_hunk {
+ long old_start, old_count;
@@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
-+ * bypassing the diff algorithm. Validates that hunks are in order,
-+ * non-overlapping, and within bounds.
++ * bypassing the diff algorithm. The caller normalizes and validates
++ * the hunks first (order, overlap, and lockstep alignment), so this
++ * only marks lines changed after asserting the memory-safety
++ * preconditions it depends on: non-negative counts and 1-based starts
++ * (checked with BUG()), and an in-bounds range (a silent -1 so the
++ * caller can fall back to the builtin diff rather than index changed[]
++ * out of range). Keeping this diagnostic-free leaves user-facing
++ * messages to the git layer.
+ *
-+ * Returns 0 on success, -1 on validation failure.
++ * Returns 0 on success, -1 if a hunk is out of range.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+ struct xdl_hunk *hunks,
+ size_t nr_hunks)
+{
+ size_t i;
-+ long j, prev_old_end = 0, prev_new_end = 0;
-+ long changed_old = 0, changed_new = 0;
++ long j;
+
+ /*
+ * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
@@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
+ for (i = 0; i < nr_hunks; i++) {
+ struct xdl_hunk *h = &hunks[i];
+
-+ if (h->old_count < 0 || h->new_count < 0) {
-+ warning("diff process hunk %"PRIuMAX": "
++ /*
++ * Non-negative counts and 1-based starts are caller
++ * preconditions (it normalizes hunks into xdiff coordinates
++ * before this point), so a violation is a bug, not a bad
++ * tool response.
++ */
++ if (h->old_count < 0 || h->new_count < 0)
++ BUG("external hunk %"PRIuMAX": "
+ "negative count (old=%ld, new=%ld)",
+ (uintmax_t)(i + 1),
+ h->old_count, h->new_count);
-+ return -1;
-+ }
-+ if (h->old_start < 1 || h->new_start < 1) {
-+ warning("diff process hunk %"PRIuMAX": "
-+ "start must be >= 1 (old=%ld, new=%ld)",
++ if (h->old_start < 1 || h->new_start < 1)
++ BUG("external hunk %"PRIuMAX": "
++ "start not 1-based (old=%ld, new=%ld)",
+ (uintmax_t)(i + 1),
+ h->old_start, h->new_start);
-+ return -1;
-+ }
+
+ /*
-+ * Range must fit: start + count - 1 <= nrec,
-+ * rewritten to avoid overflow. Same for both sides.
-+ *
-+ * When count is 0 (pure insert/delete) the check
-+ * reduces to 0 > nrec - start + 1, which rejects
-+ * start > nrec + 1 and allows start == nrec + 1
-+ * (the position after the last line).
++ * The caller validates ordering, overlap and lockstep
++ * alignment (and diagnoses a bad response). This is only a
++ * silent in-bounds guard so the marking loop cannot index
++ * changed[] out of range: start + count - 1 <= nrec,
++ * rewritten to avoid overflow. A count of 0 (pure
++ * insert/delete) allows start == nrec + 1, the position
++ * after the last line. On a miss, return -1 and let the
++ * caller fall back to the builtin diff.
+ */
-+ if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1) {
-+ warning("diff process hunk %"PRIuMAX": "
-+ "old range %ld+%ld exceeds %lu lines",
-+ (uintmax_t)(i + 1),
-+ h->old_start, h->old_count,
-+ (unsigned long)xe->xdf1.nrec);
-+ return -1;
-+ }
-+ if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1) {
-+ warning("diff process hunk %"PRIuMAX": "
-+ "new range %ld+%ld exceeds %lu lines",
-+ (uintmax_t)(i + 1),
-+ h->new_start, h->new_count,
-+ (unsigned long)xe->xdf2.nrec);
++ if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1 ||
++ h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
+ return -1;
-+ }
-+
-+ /* Ordering: no overlap with previous hunk (adjacent is OK) */
-+ if (h->old_start < prev_old_end ||
-+ h->new_start < prev_new_end) {
-+ warning("diff process hunk %"PRIuMAX": "
-+ "overlaps with previous hunk",
-+ (uintmax_t)(i + 1));
-+ return -1;
-+ }
+
+ for (j = 0; j < h->old_count; j++)
+ xe->xdf1.changed[h->old_start - 1 + j] = true;
+ for (j = 0; j < h->new_count; j++)
+ xe->xdf2.changed[h->new_start - 1 + j] = true;
-+
-+ prev_old_end = h->old_start + h->old_count;
-+ prev_new_end = h->new_start + h->new_count;
-+ }
-+
-+ /*
-+ * Synchronization invariant: unchanged line counts must match.
-+ * Otherwise xdl_build_script() would walk off one array.
-+ *
-+ * Count changed lines from the arrays rather than accumulating
-+ * during the loop to avoid any overflow in the summation.
-+ */
-+ for (j = 0; j < (long)xe->xdf1.nrec; j++)
-+ if (xe->xdf1.changed[j])
-+ changed_old++;
-+ for (j = 0; j < (long)xe->xdf2.nrec; j++)
-+ if (xe->xdf2.changed[j])
-+ changed_new++;
-+ if ((long)xe->xdf1.nrec - changed_old !=
-+ (long)xe->xdf2.nrec - changed_new) {
-+ warning("diff process: unchanged line count mismatch "
-+ "(old: %ld unchanged, new: %ld unchanged)",
-+ (long)xe->xdf1.nrec - changed_old,
-+ (long)xe->xdf2.nrec - changed_new);
-+ return -1;
+ }
+
+ return 0;
2: 30617ee17b = 3: 926cf01af6 userdiff: add diff.<driver>.process config
3: 459e485e6d = 4: 363d459ff6 sub-process: separate process lifecycle from hashmap management
4: 10b3980f59 ! 5: d003bc1f15 diff: add long-running diff process via diff.<driver>.process
@@ Commit message
textconv-transformed content. The tool controls which lines
are marked as changed while the display shows the file content.
Patch output features (word diff, function context, color) work
- normally; --stat uses its own diff codepath and never consults
- the diff process.
+ normally. A new "Which features consult the diff process"
+ documentation section lays out which features use the tool's hunks,
+ which compute independently, and why; the summary formats such as
+ --stat still use the builtin diff for now.
The handshake negotiates version=1 and capability=hunks. Per-file
- requests send command=hunks, pathname, and both file contents as
- packetized data. The tool responds with hunk lines and a status
- packet (success, error, or abort). On error, Git warns and falls
- back to the builtin diff algorithm for that file. On abort, Git
- silently falls back for the current file and stops sending further
- requests to the tool for the remainder of the session.
+ requests send command=hunks, pathname, the old and new blob object
+ names as old-oid/new-oid, and both file contents as packetized data.
+ The tool responds with hunk lines and a status packet (success,
+ error, or abort). On error, Git warns and falls back to the builtin
+ diff algorithm for that file. On abort, Git silently falls back for
+ the current file and stops sending further requests to the tool for
+ the remainder of the session.
+
+ old-oid/new-oid name the two blobs so a tool can cache its analysis
+ keyed on the pair. A side's oid is sent only when the content the
+ tool receives is that raw blob: it is omitted under textconv, which
+ rewrites the bytes, and for a working-tree side with no stored
+ object, so an oid that is sent always names the bytes the tool
+ receives. This is where the process protocol diverges from
+ diff.<driver>.command, which never composes with textconv (the
+ command replaces the whole diff and always gets the raw blob). Tools
+ ignore unknown request keys, so old tools skip them.
When the tool returns no hunks followed by status=success, Git
treats the file as having no changes and produces no diff output.
@@ Commit message
the failure is cached on the driver to avoid retrying and
re-warning on every subsequent file.
+ Git falls back to the builtin diff (rather than consulting the
+ tool) when an option the tool cannot honor is in effect: the
+ whitespace-ignoring flags, --ignore-blank-lines, -I<regex>, and
+ --anchored. The bypass keys off the effective diff parameters (xpp)
+ rather than diffopt, so a later caller whose flags live elsewhere is
+ covered uniformly. A change that only adds or removes the trailing
+ newline is likewise not expressible as hunks, so it too uses the
+ builtin diff. The hunk parser ignores unknown trailing fields on a
+ hunk line for response forward-compatibility.
+
+ Hunk accumulation is bounded by the combined byte count of the two
+ files, so a misbehaving tool that floods hunk lines cannot grow
+ memory without bound before validation runs.
+
diff_process_fill_hunks() is the sole public entry point. It
handles driver lookup, flag checks, subprocess management, and
error reporting, returning an enum that lets callers distinguish
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
+long-running process protocol (described in
+Documentation/technical/long-running-process-protocol.adoc).
+Unlike `diff.<name>.command`, which replaces Git's output entirely,
-+the diff process feeds results back into the standard pipeline.
++the diff process feeds results back into the standard pipeline. If
++both are configured for a path, `diff.<name>.command` takes precedence
++for the patch output it replaces; the summary formats, `git blame`,
++and `git log -L` never run the command and still consult the process.
+
+First, in `.gitattributes`, assign the `diff` attribute for paths.
+
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
+the process and performs the handshake. In the handshake, the welcome
+message sent by Git is "git-diff-client", only version 1 is supported,
+and the supported capability is "hunks" (the changed regions
-+described below).
++described below). The tool replies with "git-diff-server", the
++version it supports, and the capabilities it supports.
+
+For each file, Git sends a list of "key=value" pairs terminated with
+a flush packet, followed by the old and new file content as packetized
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
+-----------------------
+packet: git> command=hunks
+packet: git> pathname=path/file.c
++packet: git> old-oid=<hex>
++packet: git> new-oid=<hex>
+packet: git> 0000
+packet: git> OLD_CONTENT
+packet: git> 0000
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
+packet: git> 0000
+-----------------------
+
++The optional `old-oid` and `new-oid` keys give the object names of the
++old and new blobs, so a tool can cache its analysis keyed on the pair.
++A side's key is sent only when the content for that side is the raw
++blob it names: it is omitted when the content is textconv-transformed,
++and for a working-tree side that has no stored object. A tool that
++does not recognize these keys ignores them.
++
+The tool is expected to respond with zero or more hunk lines,
+a flush packet, and a status packet terminated with a flush packet.
+Each hunk line has the form:
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
+
+where `<old_start>` and `<old_count>` identify a range of lines in
+the old file, and `<new_start>` and `<new_count>` identify the
-+replacement range in the new file. Start values are 1-based and
-+counts are non-negative. Ranges must not extend beyond the end of
-+the file. For example, `hunk 3 2 3 4` means that 2 lines starting
-+at line 3 in the old file were replaced by 4 lines starting at
-+line 3 in the new file. An `<old_count>` of 0 means no lines were
-+removed (pure insertion); a `<new_count>` of 0 means no lines were
-+added (pure deletion). A start value of 0 is accepted when
-+the corresponding count is 0 (e.g., `hunk 0 0 1 5` for a newly
-+added file), matching what `git diff` itself emits for empty
-+file sides.
++replacement range in the new file. The four fields are separated by
++single spaces. Start values are 1-based and counts are non-negative.
++For example, `hunk 3 2 3 4` means that 2 lines starting at line 3 in
++the old file were replaced by 4 lines starting at line 3 in the new
++file. An `<old_count>` of 0 means no lines were removed (pure
++insertion); a `<new_count>` of 0 means no lines were added (pure
++deletion). For a side with a count of 0 (a pure insertion or
++deletion) the start is the 1-based line the change sits before,
++ranging from 1 to one past the last line (the line count plus 1, to
++place the change at the end of the file); like every start it must
++keep the unchanged runs aligned on both sides (see below), so for a
++given change it takes one specific value, not an arbitrary one. A
++start of 0 is also accepted and treated as 1, matching the
++empty-file-side form `git diff` emits (e.g. `hunk 0 0 1 5` for a newly
++added file). A nonzero range must not extend beyond the end of the
++file. Git ignores any extra
++whitespace-separated tokens after `<new_count>`, so a future protocol
++version can append fields to a hunk line (for example a "moved"
++marker) without older tools rejecting it.
+
+Lines are delimited by newlines. A file `"foo\nbar\n"` and a
+file `"foo\nbar"` both have 2 lines.
+
-+Hunks must be listed in order and must not overlap. Any line
-+not covered by a hunk is treated as unchanged, so the total
-+number of unchanged lines must be the same on both sides.
-+For example, if the old file has 10 lines and the hunks cover
-+4 of them (`old_count` values summing to 4), then 6 old lines
-+are unchanged. The new file must also have exactly 6 lines
-+not covered by hunks, so the `new_count` values must sum to
-+`new_file_lines - 6`.
++Hunks must be listed in order and must not overlap. Any line not
++covered by a hunk is treated as unchanged and is paired, in order,
++with the unchanged lines on the other side. Each run of unchanged
++lines between two hunks (and the run before the first hunk and
++after the last) must therefore be the same length on both sides,
++not merely equal in total. For the hunks `1 3 1 5` and `10 2 12 2`
++below, lines 4-9 of the old file and lines 6-11 of the new file are
++both the six unchanged lines between the two hunks. A response that
++balances only the total unchanged count but misaligns one of these
++runs is rejected, and Git falls back to the builtin diff.
++
++Git does not check that the lines a hunk leaves unchanged are
++byte-for-byte identical between the two sides; it pairs them by
++position and shows the new side as context. A tool may therefore
++report lines that differ textually (a pure reformatting, say) as
++unchanged, and the diff reflects that judgment. This is
++the point of a semantic backend, but it means a misbehaving tool can
++produce a diff whose context does not match the old blob; as with
++`git diff -w`, such a patch may not apply against the old content.
+
+-----------------------
+packet: git< hunk 1 3 1 5
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
+-----------------------
+
+If the tool responds with hunks and "success", Git marks those lines
-+as changed and feeds them into the standard diff pipeline. Patch
-+output features (word diff, function context, color) work normally.
-+Note that `--stat` and other summary formats use their own diff path
-+and are not affected by the diff process.
++as changed and feeds them into the standard diff pipeline. Git may
++still slide or regroup those changes against matching context for
++display, exactly as it compacts its own diffs, so the tool controls
++which lines are reported as changed, not the precise hunk boundaries.
++Patch output features (word diff, function context, color) work
++normally. Summary formats such as `--stat` still compute their counts
++with the builtin diff for now; see "Which features consult the diff
++process" below for the full picture and the reasoning behind it.
+
+If no hunk lines precede the flush, followed by "success", Git
-+treats the files as having no changes: `git diff` produces no output
-+and `git blame` skips the commit, attributing lines to earlier commits.
++treats the files as having no changes: `git diff` produces no output,
++`git diff --exit-code` and `--quiet` report success even though the
++stored blobs differ, and `git blame` skips the commit, attributing
++lines to earlier commits.
++The one exception is a change that only adds or removes the file's
++trailing newline: it cannot be expressed as line hunks, so when the
++line content otherwise matches Git keeps the builtin diff for that
++file (preserving the `\ No newline at end of file` marker) instead of
++treating the two sides as equal.
+
+-----------------------
+packet: git< 0000
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
+packet: git< 0000
+-----------------------
+
-+If the tool returns invalid hunks (out of bounds, overlapping, or
-+mismatched unchanged line counts), Git warns and falls back to the
-+builtin diff algorithm.
++If the tool returns well-formed but invalid hunks (out of bounds,
++overlapping, or with misaligned unchanged runs), Git warns and falls
++back to the builtin diff for that file; the tool stays available for
++subsequent files. A malformed hunk line, by contrast (bad syntax, a
++nonzero count paired with a start of 0, or more hunks than the file
++has lines), is a protocol violation: Git stops the process and does
++not send it further requests, as described below.
+
+In case the tool cannot or does not want to process the content,
+it is expected to respond with an "error" status. Git warns and
-+falls back to the builtin diff algorithm for this file. The tool
++falls back to the builtin diff algorithm for this file, treating any
++status other than "success" or "abort" the same way. The tool
+remains available for subsequent files.
+
+-----------------------
@@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
+forward-compatible. Future versions of Git may send additional
+`command=` values; tools that receive an unrecognized command should
+respond with `status=error` rather than terminating.
++
++Which features consult the diff process
++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
++
++The diff process answers a single question: given two blobs, which
++line ranges differ? Whether a particular feature consults it follows
++from whether that is the question the feature is really asking.
++
++Features that ask "which lines changed" use the tool's hunks in place
++of the builtin algorithm:
++
++- `git diff` patch output, together with everything layered on it:
++ word diff, function context (`-W`), `--color-moved`, the `@@` hunk
++ headers, and the `-L` line-range display. These operate on the
++ lines the patch step already emitted, so they reflect the tool's
++ hunks without any further negotiation.
++- `git blame`: a commit whose change the tool reports as equivalent is
++ skipped, and its lines are attributed to an earlier commit.
++
++Features that ask a different question do not consult the process, by
++design:
++
++- The pickaxe `-G<regex>` searches the textual diff for a pattern; it
++ asks "does this string appear in the diff," not "did these lines
++ change." (`-S` runs at an earlier stage and is likewise unaffected.)
++- `git patch-id` must produce a stable hash for `git rebase` and
++ cherry-pick detection; deriving it from a configured tool would make
++ equal patches hash differently from machine to machine.
++- The merge machinery (`git merge-tree`, `rerere`) computes merge
++ content and conflict signatures rather than display output, so the
++ tool's hunks must not alter its results.
++- `git range-diff` diffs patch text, not source blobs, so source-file
++ hunks do not apply to it.
++- `--check` reports whitespace errors in added lines using the builtin
++ diff's notion of which lines are added, not the tool's. It can
++ therefore flag (and exit non-zero on) a line the tool treats as
++ unchanged and that `git diff` shows as context. Whitespace breakage
++ is a property of the literal bytes, so `--check` keeps the builtin
++ partition deliberately; a future change could wire it to the tool if
++ matching `git diff` exactly became desirable.
++- `--raw`, `--name-only`, and `--name-status` compare object ids at
++ the tree level and never run a line-level diff at all.
++
++Some features ask "which lines changed" but still use the builtin
++algorithm for now, and may consult the process in a later change: the
++summary formats (`--stat`, `--numstat`, `--shortstat`); `git log -L`'s
++commit selection and parent range propagation (as distinct from its
++display, which is covered above); and combined diffs (`--cc` and merge
++diffs), whose protocol would have to be extended from a single old/new
++pair to one comparison per merge parent.
++
++`--diff-algorithm` bypasses the process entirely, for every feature
++listed above. The whitespace-ignoring options (`-w`,
++`--ignore-space-change`, `--ignore-blank-lines`, and the like),
++`-I<regex>`, and `--anchored` also bypass it for the affected files:
++the tool is never told about these options, so it could not honor
++them, and Git falls back to the builtin diff, which does.
+
Defining a custom hunk-header
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ diff-process.c (new)
+ * tool< capability=hunks / flush
+ *
+ * Per-file:
-+ * git> command=hunks / pathname=<path> / flush
++ * git> command=hunks / pathname=<path> / [old-oid=<hex>] / [new-oid=<hex>] / flush
+ * git> <old content packetized> / flush
+ * git> <new content packetized> / flush
+ * tool< hunk <old_start> <old_count> <new_start> <new_count>
@@ diff-process.c (new)
+#include "diff-process.h"
+#include "diff.h"
+#include "gettext.h"
++#include "hex.h"
+#include "repository.h"
+#include "sigchain.h"
+#include "userdiff.h"
@@ diff-process.c (new)
+ return packet_flush_gently(fd);
+}
+
-+static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
++/*
++ * A hunk in the diff process's presentation coordinates: the line
++ * numbering it reports over the protocol. Kept distinct from struct
++ * xdl_hunk (xdiff's coordinates) so that only translated hunks ever
++ * reach the diff algorithm; diff_process_hunk_to_xdl() is the single
++ * crossing point.
++ */
++struct diff_process_hunk {
++ long old_start, old_count;
++ long new_start, new_count;
++};
++
++/*
++ * Parse one non-negative decimal field of a hunk line into *out and
++ * advance *line past it. Fields must be plain decimal with no leading
++ * whitespace or sign (isdigit() takes an unsigned char to stay defined
++ * for high-bit bytes). The first three fields are followed by a single
++ * space; the last (is_last) is followed by end-of-string or a space.
++ * Trailing space-separated tokens after the last field are allowed and
++ * ignored, so a future protocol version can append fields (e.g. a
++ * "moved" marker) without older tools rejecting the line -- mirroring
++ * the request-side rule that tools ignore unknown keys.
++ */
++static int parse_hunk_field(const char **line, long *out, int is_last)
+{
++ const char *p = *line;
+ char *end;
+
-+ /*
-+ * Format: "hunk <old_start> <old_count> <new_start> <new_count>"
-+ * All numbers must be non-negative decimal with no leading
-+ * whitespace or sign characters.
-+ */
-+ if (!skip_prefix(line, "hunk ", &line))
-+ return -1;
-+
-+ if (!isdigit(*line))
++ if (!isdigit((unsigned char)*p))
+ return -1;
+ errno = 0;
-+ hunk->old_start = strtol(line, &end, 10);
-+ if (errno || end == line || *end++ != ' ')
++ *out = strtol(p, &end, 10);
++ if (errno || end == p)
+ return -1;
-+ line = end;
++ if (is_last) {
++ if (*end != '\0' && *end != ' ')
++ return -1;
++ } else {
++ if (*end != ' ')
++ return -1;
++ end++;
++ }
++ *line = end;
++ return 0;
++}
+
-+ if (!isdigit(*line))
++static int parse_hunk_line(const char *line,
++ struct diff_process_hunk *presented)
++{
++ /* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
++ if (!skip_prefix(line, "hunk ", &line))
+ return -1;
-+ errno = 0;
-+ hunk->old_count = strtol(line, &end, 10);
-+ if (errno || end == line || *end++ != ' ')
++ if (parse_hunk_field(&line, &presented->old_start, 0) ||
++ parse_hunk_field(&line, &presented->old_count, 0) ||
++ parse_hunk_field(&line, &presented->new_start, 0) ||
++ parse_hunk_field(&line, &presented->new_count, 1))
+ return -1;
-+ line = end;
++ return 0;
++}
+
-+ if (!isdigit(*line))
-+ return -1;
-+ errno = 0;
-+ hunk->new_start = strtol(line, &end, 10);
-+ if (errno || end == line || *end++ != ' ')
-+ return -1;
-+ line = end;
++/*
++ * Translate a hunk from the diff process's presentation coordinates
++ * into xdiff's.
++ *
++ * Protocol starts are already 1-based positions (the line a change
++ * sits before), the same numbering xdiff uses, so the only adjustment
++ * is for an empty file side: "git diff" addresses it with a start of 0
++ * and a count of 0 (e.g. "0 0 1 5" adds five lines to an empty old
++ * side), and since xdiff uses start-1 as an array index that 0 becomes
++ * 1 here. This is NOT the full inverse of xdl_emit_hunk_hdr()
++ * (xdiff/xutils.c): that emitter shifts a count-0 range to start-1 for
++ * the displayed "@@" header, but the protocol keeps the unshifted
++ * 1-based position for a mid-file insert or delete. This is the single
++ * point where presentation coordinates become xdiff coordinates, so
++ * xdl_populate_hunks_from_external() may assume 1-based starts.
++ *
++ * Returns -1 for a start of 0 paired with a nonzero count, which names
++ * no line in either coordinate system. (parse_hunk_line() already
++ * guarantees non-negative starts and counts.)
++ */
++static int diff_process_hunk_to_xdl(const struct diff_process_hunk *presented,
++ struct xdl_hunk *xdl)
++{
++ long old_start = presented->old_start;
++ long new_start = presented->new_start;
+
-+ if (!isdigit(*line))
++ if ((!old_start && presented->old_count) ||
++ (!new_start && presented->new_count))
+ return -1;
-+ errno = 0;
-+ hunk->new_count = strtol(line, &end, 10);
-+ if (errno || end == line || *end != '\0')
-+ return -1;
-+
-+ /*
-+ * git diff emits start=0 when count=0 (empty file side).
-+ * Normalize to 1-based so downstream validation can assume start >= 1.
-+ */
-+ if (!hunk->old_count && !hunk->old_start)
-+ hunk->old_start = 1;
-+ if (!hunk->new_count && !hunk->new_start)
-+ hunk->new_start = 1;
-+
++ if (!old_start)
++ old_start = 1;
++ if (!new_start)
++ new_start = 1;
++
++ xdl->old_start = old_start;
++ xdl->old_count = presented->old_count;
++ xdl->new_start = new_start;
++ xdl->new_count = presented->new_count;
+ return 0;
+}
+
@@ diff-process.c (new)
+ const char *path,
+ const char *old_buf, long old_size,
+ const char *new_buf, long new_size,
++ const struct object_id *oid_a,
++ const struct object_id *oid_b,
+ struct xdl_hunk **hunks_out,
+ size_t *nr_hunks_out)
+{
@@ diff-process.c (new)
+ int fd_in, fd_out;
+ struct strbuf status = STRBUF_INIT;
+ struct xdl_hunk *hunks = NULL;
++ struct diff_process_hunk presented;
+ struct xdl_hunk hunk;
+ size_t nr_hunks = 0, alloc_hunks = 0;
++ size_t max_hunks;
+ int len;
+ char *line;
+
@@ diff-process.c (new)
+
+ /* Send request */
+ if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
-+ packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
-+ packet_flush_gently(fd_in))
++ packet_write_fmt_gently(fd_in, "pathname=%s\n", path))
++ goto comm_error;
++ /*
++ * old-oid/new-oid let the tool key a cache on the blob pair. A
++ * side is sent only when its content is the raw blob (the caller
++ * passes NULL otherwise, e.g. for textconv'd content), so an oid
++ * that is present always names the bytes the tool receives.
++ */
++ if (oid_a &&
++ packet_write_fmt_gently(fd_in, "old-oid=%s\n", oid_to_hex(oid_a)))
++ goto comm_error;
++ if (oid_b &&
++ packet_write_fmt_gently(fd_in, "new-oid=%s\n", oid_to_hex(oid_b)))
++ goto comm_error;
++ if (packet_flush_gently(fd_in))
+ goto comm_error;
+
+ /* Send old file content */
@@ diff-process.c (new)
+ if (send_file_content(fd_in, new_buf, new_size))
+ goto comm_error;
+
++ /*
++ * Hunks are non-overlapping and each useful hunk covers at least
++ * one line, so a valid response cannot contain more hunks than the
++ * two files have lines, which is bounded by their byte sizes. Cap
++ * the accumulation accordingly so a misbehaving tool that floods
++ * hunk lines cannot drive unbounded memory growth before validation.
++ */
++ max_hunks = (size_t)old_size + (size_t)new_size + 1;
++
+ /* Read hunks until flush packet */
+ while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+ line) {
-+ if (parse_hunk_line(line, &hunk) < 0)
++ if (parse_hunk_line(line, &presented) < 0)
++ goto comm_error;
++ if (diff_process_hunk_to_xdl(&presented, &hunk) < 0)
++ goto comm_error;
++ if (nr_hunks >= max_hunks) {
++ warning(_("diff process '%s' sent too many hunks"
++ " for '%s'"), drv->process, path);
+ goto comm_error;
++ }
+ ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+ hunks[nr_hunks++] = hunk;
+ }
@@ diff-process.c (new)
+ return DIFF_PROCESS_ERROR;
+}
+
++/*
++ * Whether exactly one of the two blobs ends in a newline. A change
++ * that only adds or removes the trailing newline is not expressible as
++ * line hunks, so a tool comparing lines reports the files as equal.
++ */
++static int eof_newline_differs(const mmfile_t *a, const mmfile_t *b)
++{
++ int a_nl = a->size > 0 && a->ptr[a->size - 1] == '\n';
++ int b_nl = b->size > 0 && b->ptr[b->size - 1] == '\n';
++ return a_nl != b_nl;
++}
++
++/*
++ * Number of lines in a blob, matching xdiff's record count: one per
++ * newline, plus one more if the last line has no trailing newline.
++ */
++static long count_lines(const char *buf, long size)
++{
++ long lines = 0, i;
++
++ for (i = 0; i < size; i++)
++ if (buf[i] == '\n')
++ lines++;
++ if (size > 0 && buf[size - 1] != '\n')
++ lines++;
++ return lines;
++}
++
++/*
++ * Validate the tool's hunks (already in xdiff coordinates) against the
++ * two blobs before they bypass the diff algorithm. Each hunk must fit
++ * within its file, the hunks must be ordered and non-overlapping, and
++ * the unchanged run before each hunk (and after the last) must be the
++ * same length on both sides -- xdl_build_script() walks the two files
++ * in lockstep over unchanged lines, so a mismatched gap desynchronizes
++ * it and yields a corrupt diff even when the totals balance. This is
++ * the git layer's job so xdiff stays diagnostic-free; on a bad response
++ * we warn and the caller falls back to the builtin diff. Returns 0 if
++ * valid, -1 (after warning) otherwise.
++ */
++static int validate_external_hunks(const struct xdl_hunk *hunks, size_t nr,
++ long old_lines, long new_lines,
++ const char *process, const char *path)
++{
++ size_t i;
++ long prev_old_end = 0, prev_new_end = 0;
++
++ for (i = 0; i < nr; i++) {
++ const struct xdl_hunk *h = &hunks[i];
++
++ if (h->old_count > old_lines - h->old_start + 1 ||
++ h->new_count > new_lines - h->new_start + 1) {
++ warning(_("diff process '%s' returned a hunk past the "
++ "end of '%s'; using the builtin diff"),
++ process, path);
++ return -1;
++ }
++ if (h->old_start < prev_old_end || h->new_start < prev_new_end) {
++ warning(_("diff process '%s' returned overlapping hunks "
++ "for '%s'; using the builtin diff"),
++ process, path);
++ return -1;
++ }
++ if (h->old_start - prev_old_end != h->new_start - prev_new_end) {
++ warning(_("diff process '%s' returned hunks that leave "
++ "'%s' misaligned; using the builtin diff"),
++ process, path);
++ return -1;
++ }
++ prev_old_end = h->old_start + h->old_count;
++ prev_new_end = h->new_start + h->new_count;
++ }
++ if (old_lines - prev_old_end != new_lines - prev_new_end) {
++ warning(_("diff process '%s' returned hunks that leave '%s' "
++ "misaligned; using the builtin diff"),
++ process, path);
++ return -1;
++ }
++ return 0;
++}
++
+enum diff_process_result diff_process_fill_hunks(
+ struct diff_options *diffopt,
+ const char *path,
+ const mmfile_t *file_a,
+ const mmfile_t *file_b,
++ const struct object_id *oid_a,
++ const struct object_id *oid_b,
+ xpparam_t *xpp)
+{
+ struct userdiff_driver *drv;
@@ diff-process.c (new)
+ return DIFF_PROCESS_SKIP;
+ if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
+ return DIFF_PROCESS_SKIP;
++ /*
++ * Whitespace-ignoring, regex-ignore (-I) and anchored options
++ * change which lines count as different, but the tool is never
++ * told about them, so its hunks could not honor them. Rather
++ * than silently override the user's request, fall back to the
++ * builtin diff, which does honor these flags. Key this off xpp
++ * (the parameters this diff actually runs with) rather than
++ * diffopt, so a caller like blame that keeps its flags outside
++ * diffopt is covered without a separate guard of its own.
++ */
++ if ((xpp->flags & (XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES)) ||
++ xpp->ignore_regex_nr || xpp->anchors_nr)
++ return DIFF_PROCESS_SKIP;
+
+ drv = userdiff_find_by_path(diffopt->repo->index, path);
+ if (!drv || !drv->process)
@@ diff-process.c (new)
+ res = get_hunks(drv, path,
+ file_a->ptr, file_a->size,
+ file_b->ptr, file_b->size,
++ oid_a, oid_b,
+ &ext_hunks, &nr);
+ if (res == DIFF_PROCESS_OK) {
+ if (!nr) {
+ free(ext_hunks);
++ /*
++ * Zero hunks means the tool considers the line
++ * content identical, but it cannot express a
++ * trailing-newline-only change. When that is the
++ * actual difference, fall back to the builtin diff
++ * so the "\ No newline at end of file" marker is
++ * preserved instead of reporting the files equal.
++ */
++ if (eof_newline_differs(file_a, file_b))
++ return DIFF_PROCESS_SKIP;
+ return DIFF_PROCESS_EQUIVALENT;
+ }
++ if (validate_external_hunks(ext_hunks, nr,
++ count_lines(file_a->ptr, file_a->size),
++ count_lines(file_b->ptr, file_b->size),
++ drv->process, path) < 0) {
++ free(ext_hunks);
++ return DIFF_PROCESS_SKIP;
++ }
+ xpp->external_hunks = ext_hunks;
+ xpp->external_hunks_nr = nr;
+ return DIFF_PROCESS_OK;
@@ diff-process.h (new)
+#include "xdiff/xdiff.h"
+
+struct diff_options;
++struct object_id;
+
+enum diff_process_result {
-+ DIFF_PROCESS_ERROR = -1, /* tool failure: warned, fell back */
++ DIFF_PROCESS_ERROR = -1, /* failed; caller falls back to builtin */
+ DIFF_PROCESS_OK = 0, /* hunks populated in xpp */
-+ DIFF_PROCESS_SKIP, /* no process configured: use builtin */
++ DIFF_PROCESS_SKIP, /* process did not apply: use builtin */
+ DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
+};
+
@@ diff-process.h (new)
+ * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
+ * The caller owns xpp->external_hunks and must free() it.
+ *
-+ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks
-+ * (files are considered identical); caller should skip diff/blame.
++ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks and
++ * the blobs are not a trailing-newline-only change (files are
++ * considered identical); caller should skip diff/blame.
+ * Returns DIFF_PROCESS_SKIP when no process applies; caller
+ * should use the builtin diff algorithm.
+ * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
+ * caller should fall back to the builtin diff algorithm.
++ *
++ * oid_a/oid_b, when non-NULL, are sent to the tool as old-oid/new-oid
++ * so it can key a cache on the blob pair. Pass NULL for a side whose
++ * content is not the raw blob (e.g. textconv'd) or whose object name is
++ * unknown, so any oid that is sent always names the bytes the tool
++ * receives.
+ */
+enum diff_process_result diff_process_fill_hunks(
+ struct diff_options *diffopt,
+ const char *path,
+ const mmfile_t *file_a,
+ const mmfile_t *file_b,
++ const struct object_id *oid_a,
++ const struct object_id *oid_b,
+ xpparam_t *xpp);
+
+#endif /* DIFF_PROCESS_H */
@@ diff.c: static void builtin_diff(const char *name_a,
xpp.anchors = o->anchors;
xpp.anchors_nr = o->anchors_nr;
+
-+ if (diff_process_fill_hunks(o, name_a,
-+ &mf1, &mf2, &xpp)
++ /*
++ * Send the blob oids only for a side whose content is the
++ * raw blob: textconv rewrites the bytes, and a working-tree
++ * side has no stored oid, so pass NULL there rather than an
++ * oid that would not name what the tool receives.
++ */
++ if (diff_process_fill_hunks(o, name_a, &mf1, &mf2,
++ (textconv_one || !one->oid_valid) ? NULL : &one->oid,
++ (textconv_two || !two->oid_valid) ? NULL : &two->oid,
++ &xpp)
+ == DIFF_PROCESS_EQUIVALENT) {
+ if (textconv_one)
+ free(mf1.ptr);
@@ t/helper/test-diff-process-backend.c (new)
+ *
+ * packet: git> command=hunks
+ * packet: git> pathname=<path>
++ * packet: git> [old-oid=<hex>] (omitted for textconv/worktree)
++ * packet: git> [new-oid=<hex>]
+ * packet: git> 0000
+ * packet: git> OLD_CONTENT
+ * packet: git> 0000
@@ t/helper/test-diff-process-backend.c (new)
+ *
+ * Response varies by --mode (default: whole-file):
+ *
-+ * whole-file packet: git< hunk 1 <old_lines> 1 <new_lines>
++ * whole-file packet: git< hunk <1|0> <old_lines> <1|0> <new_lines>
++ * (start is 0 for an empty side, matching git diff)
+ * fixed-hunk packet: git< hunk 5 2 5 2
+ * no-hunks (no hunk packets)
+ * bad-hunk packet: git< hunk 999 1 999 1
+ * bad-parse packet: git< garbage not a hunk
+ * bad-sync packet: git< hunk 1 2 1 1
++ * bad-gap packet: git< hunk 1 1 3 1
++ * bad-start packet: git< hunk 0 1 1 1
++ * multi-hunk packet: git< hunk 5 2 5 2
++ * packet: git< hunk 9 2 9 2
++ * insert packet: git< hunk 3 0 3 2 (mid-file count-0 insertion)
++ * flood packet: git< hunk 1 1 1 1 (x100000)
+ * overlap packet: git< hunk 1 5 1 5
+ * packet: git< hunk 3 2 3 2
+ * no-cap (omits capability=hunks during handshake)
@@ t/helper/test-diff-process-backend.c (new)
+ * abort (status=abort instead of status=success)
+ * crash exit(1) before sending any response
+ *
-+ * All non-error/abort modes end with:
++ * All success modes (not error/abort/crash) end with:
+ *
+ * packet: git< 0000
+ * packet: git< status=success
@@ t/helper/test-diff-process-backend.c (new)
+ *
+ * Each request is logged to --log as:
+ *
-+ * command=<cmd> pathname=<path> old=<first line> new=<first line>
++ * command=<cmd> pathname=<path> old-oid=<hex> new-oid=<hex> old=<first line> new=<first line>
+ */
+
+#include "test-tool.h"
@@ t/helper/test-diff-process-backend.c (new)
+ MODE_BAD_HUNK,
+ MODE_BAD_PARSE,
+ MODE_BAD_SYNC,
++ MODE_BAD_GAP,
++ MODE_BAD_START,
++ MODE_MULTI_HUNK,
++ MODE_INSERT,
++ MODE_FLOOD,
+ MODE_OVERLAP,
+ MODE_NO_CAP,
+ MODE_ERROR,
@@ t/helper/test-diff-process-backend.c (new)
+ return MODE_BAD_PARSE;
+ if (!strcmp(s, "bad-sync"))
+ return MODE_BAD_SYNC;
++ if (!strcmp(s, "bad-gap"))
++ return MODE_BAD_GAP;
++ if (!strcmp(s, "bad-start"))
++ return MODE_BAD_START;
++ if (!strcmp(s, "multi-hunk"))
++ return MODE_MULTI_HUNK;
++ if (!strcmp(s, "insert"))
++ return MODE_INSERT;
++ if (!strcmp(s, "flood"))
++ return MODE_FLOOD;
+ if (!strcmp(s, "overlap"))
+ return MODE_OVERLAP;
+ if (!strcmp(s, "no-cap"))
@@ t/helper/test-diff-process-backend.c (new)
+ * variant: once inside a request, truncation is a protocol violation
+ * and dying loudly is the correct response.
+ */
-+static int read_request_header(char **command, char **pathname)
++static int read_request_header(char **command, char **pathname,
++ char **old_oid, char **new_oid)
+{
+ int first = 1;
+ char *line;
+
-+ *command = *pathname = NULL;
++ *command = *pathname = *old_oid = *new_oid = NULL;
+ for (;;) {
+ const char *value;
+
@@ t/helper/test-diff-process-backend.c (new)
+ *command = xstrdup(value);
+ else if (skip_prefix(line, "pathname=", &value))
+ *pathname = xstrdup(value);
++ else if (skip_prefix(line, "old-oid=", &value))
++ *old_oid = xstrdup(value);
++ else if (skip_prefix(line, "new-oid=", &value))
++ *new_oid = xstrdup(value);
+ }
+ return 1;
+}
@@ t/helper/test-diff-process-backend.c (new)
+ case MODE_BAD_SYNC:
+ packet_write_fmt(1, "hunk 1 2 1 1\n");
+ break;
++ case MODE_BAD_GAP:
++ /*
++ * Globally balanced (1 changed line on each side, so the
++ * total unchanged counts match) but the gap before the
++ * change differs between sides: old line 1 vs new line 3.
++ * Exercises the per-gap lockstep-alignment check.
++ */
++ packet_write_fmt(1, "hunk 1 1 3 1\n");
++ break;
++ case MODE_BAD_START:
++ /*
++ * A start of 0 is valid only for an empty (count 0) range;
++ * pairing it with a nonzero count names no line in either
++ * the protocol's or xdiff's coordinates, so the translation
++ * rejects it and git falls back to the builtin diff.
++ */
++ packet_write_fmt(1, "hunk 0 1 1 1\n");
++ break;
++ case MODE_MULTI_HUNK:
++ /*
++ * Two valid, non-overlapping, gap-aligned hunks. Exercises
++ * the accepting branch of the per-gap lockstep check with a
++ * non-zero previous-hunk end (the realistic two-region case).
++ */
++ packet_write_fmt(1, "hunk 5 2 5 2\n");
++ packet_write_fmt(1, "hunk 9 2 9 2\n");
++ break;
++ case MODE_INSERT:
++ /*
++ * A mid-file pure insertion (count 0 on the old side) in the
++ * protocol's 1-based-position form: 2 lines inserted before
++ * old line 3. Exercises the count-0 path, which uses the
++ * unshifted position (not git diff's "-3,0" display start).
++ */
++ packet_write_fmt(1, "hunk 3 0 3 2\n");
++ break;
++ case MODE_FLOOD: {
++ /*
++ * Emit far more hunks than any small file has lines, so Git
++ * trips its accumulation cap and falls back before reading
++ * them all.
++ */
++ int i;
++ for (i = 0; i < 100000; i++)
++ packet_write_fmt(1, "hunk 1 1 1 1\n");
++ break;
++ }
+ case MODE_OVERLAP:
+ packet_write_fmt(1, "hunk 1 5 1 5\n");
+ packet_write_fmt(1, "hunk 3 2 3 2\n");
@@ t/helper/test-diff-process-backend.c (new)
+{
+ for (;;) {
+ char *command = NULL, *pathname = NULL;
++ char *old_oid = NULL, *new_oid = NULL;
+ struct strbuf obuf = STRBUF_INIT;
+ struct strbuf nbuf = STRBUF_INIT;
+
-+ if (!read_request_header(&command, &pathname))
++ if (!read_request_header(&command, &pathname,
++ &old_oid, &new_oid))
+ break; /* EOF: Git closed its end */
+
+ read_packetized_to_strbuf(0, &obuf, 0);
@@ t/helper/test-diff-process-backend.c (new)
+
+ if (logfile) {
+ fprintf(logfile,
-+ "command=%s pathname=%s old=%.*s new=%.*s\n",
++ "command=%s pathname=%s old-oid=%s new-oid=%s"
++ " old=%.*s new=%.*s\n",
+ command ? command : "(none)",
+ pathname ? pathname : "(none)",
++ old_oid ? old_oid : "(none)",
++ new_oid ? new_oid : "(none)",
+ (int)(strchrnul(obuf.buf, '\n') - obuf.buf),
+ obuf.buf,
+ (int)(strchrnul(nbuf.buf, '\n') - nbuf.buf),
@@ t/helper/test-diff-process-backend.c (new)
+
+ free(command);
+ free(pathname);
++ free(old_oid);
++ free(new_oid);
+ strbuf_release(&obuf);
+ strbuf_release(&nbuf);
+ }
@@ t/helper/test-diff-process-backend.c (new)
+ enum mode mode = MODE_WHOLE_FILE;
+ struct option options[] = {
+ OPT_STRING(0, "mode", &mode_str, "mode",
-+ "response shape: whole-file (default), fixed-hunk,"
-+ " no-hunks, bad-hunk, bad-sync, overlap, error,"
-+ " abort, crash"),
++ "response shape (default whole-file);"
++ " see the file header for the full list of modes"),
+ OPT_STRING(0, "log", &log_path, "path",
+ "append per-request summary to this file"),
+ OPT_END()
@@ t/t4080-diff-process.sh (new)
+
+test_description='diff process via long-running process'
+
++TEST_PASSES_SANITIZE_LEAK=true
+. ./test-lib.sh
+
+# See t/helper/test-diff-process-backend.c for the backend implementation
@@ t/t4080-diff-process.sh (new)
+ EOF
+ git add logtest.c &&
+
-+ # two.c/one.c: two-file pair for error/abort/startup-failure tests.
++ # one.c/two.c: two-file pair for error/abort/startup-failure tests.
+ cat >one.c <<-\EOF &&
+ int first(void) { return 1; }
+ EOF
@@ t/t4080-diff-process.sh (new)
+ test_grep ! "^+NEW10" actual
+'
+
++test_expect_success 'diff process accepts valid multi-hunk output' '
++ # multi-hunk reports both changed regions (5-6 and 9-10) as two
++ # gap-aligned hunks. This exercises the accepting branch of the
++ # per-gap lockstep check (non-zero previous-hunk end) and must
++ # produce a correct two-region diff with the lines between the
++ # hunks kept as context.
++ git -c diff.cdiff.process="$BACKEND --mode=multi-hunk" \
++ diff boundary.c >actual 2>stderr &&
++ test_grep "^-OLD5" actual &&
++ test_grep "^+NEW5" actual &&
++ test_grep "^-OLD9" actual &&
++ test_grep "^+NEW9" actual &&
++ test_grep "^ line7" actual &&
++ test_grep "^ line8" actual &&
++ test_must_be_empty stderr
++'
++
++test_expect_success 'diff process accepts a mid-file count-0 insertion' '
++ # insert mode reports "hunk 3 0 3 2": a pure insertion (count 0 on
++ # the old side) in the protocol 1-based-position form. Exercises
++ # the count-0 hunk path that the other valid-hunk modes (full
++ # replacements, equal-count modifies) never hit. Empty stderr is
++ # the discriminator: a mishandled count-0 start would be rejected
++ # by the lockstep check and warn.
++ cat >insert.c <<-\EOF &&
++ a
++ b
++ c
++ d
++ e
++ EOF
++ git add insert.c &&
++ git commit -m "add insert.c" &&
++ cat >insert.c <<-\EOF &&
++ a
++ b
++ X
++ Y
++ c
++ d
++ e
++ EOF
++ git -c diff.cdiff.process="$BACKEND --mode=insert" \
++ diff insert.c >actual 2>stderr &&
++ test_grep "^+X" actual &&
++ test_grep "^+Y" actual &&
++ test_grep "^ c" actual &&
++ test_must_be_empty stderr
++'
++
+test_expect_success 'diff process works with modified file' '
+ test_when_finished "rm -f backend.log" &&
+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
@@ t/t4080-diff-process.sh (new)
+ diff --exit-code nohunks.c
+'
+
++test_expect_success 'diff process equivalent commit: --exit-code and --quiet agree' '
++ # A committed blob pair (not a worktree file) whose oids differ but
++ # the tool reports equivalent. --exit-code and --quiet must agree
++ # with the shown diff (empty) and report success, not fall back to
++ # the byte-level "oids differ" answer.
++ cat >ecq.c <<-\EOF &&
++ alpha
++ EOF
++ git add ecq.c &&
++ git commit -m "ecq v1" &&
++ cat >ecq.c <<-\EOF &&
++ beta
++ EOF
++ git add ecq.c &&
++ git commit -m "ecq v2" &&
++ git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
++ diff --exit-code HEAD^ HEAD -- ecq.c &&
++ git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
++ diff --quiet HEAD^ HEAD -- ecq.c
++'
++
++test_expect_success 'diff process falls back for trailing-newline-only change' '
++ test_when_finished "rm -f backend.log" &&
++ printf "a\nb\nc\n" >eofnl.c &&
++ git add eofnl.c &&
++ git commit -m "add eofnl.c" &&
++ printf "a\nb\nc" >eofnl.c &&
++ # Same lines, only the final newline removed. The tool reports
++ # no hunks (it sees identical lines), but that change is not
++ # expressible as hunks, so git falls back to the builtin diff
++ # rather than treating the files as equivalent.
++ git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
++ diff eofnl.c >actual 2>stderr &&
++ test_grep "No newline at end of file" actual &&
++ test_grep "pathname=eofnl.c" backend.log &&
++ test_must_be_empty stderr
++'
++
++test_expect_success 'diff process falls back for added file (empty old side)' '
++ test_when_finished "rm -f backend.log" &&
++ printf "x\ny\nz\n" >addnl.c &&
++ git add addnl.c &&
++ # The empty old side has no trailing newline while the new side
++ # does, so the newline fallback shows the addition rather than
++ # letting no-hunks suppress the whole new file.
++ git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
++ diff --cached addnl.c >actual 2>stderr &&
++ test_grep "^+x" actual &&
++ test_grep "pathname=addnl.c" backend.log &&
++ test_must_be_empty stderr
++'
++
+test_expect_success 'diff process with --exit-code and hunks returns failure' '
+ test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
+ diff --exit-code newfile.c
@@ t/t4080-diff-process.sh (new)
+ test_path_is_missing backend.log
+'
+
-+test_expect_success 'diff process not used by --stat' '
++test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
+ test_when_finished "rm -f backend.log" &&
-+ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
-+ diff --stat worddiff.c >actual &&
-+ test_grep "worddiff.c" actual &&
-+ test_path_is_missing backend.log
++ printf "a\nb\nc\n" >wsbypass.c &&
++ git add wsbypass.c &&
++ git commit -m "add wsbypass.c" &&
++ printf "a\n b \nc\n" >wsbypass.c &&
++ # The tool is never told about these options and could not honor
++ # them, so git bypasses the process for each (covering the whole
++ # XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES mask, not just -w).
++ for opt in -w -b --ignore-space-at-eol --ignore-blank-lines
++ do
++ rm -f backend.log &&
++ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
++ diff $opt wsbypass.c >actual 2>stderr &&
++ test_path_is_missing backend.log &&
++ test_must_be_empty stderr ||
++ return 1
++ done &&
++ # -w additionally suppresses the whitespace-only change via the
++ # builtin diff that now runs.
++ git -c diff.cdiff.process="$BACKEND" diff -w wsbypass.c >actual &&
++ test_must_be_empty actual
+'
+
+#
@@ t/t4080-diff-process.sh (new)
+ test_grep "^+NEW5" actual &&
+ test_grep "^-OLD9" actual &&
+ test_grep "^+NEW9" actual &&
-+ test_grep "exceeds.*lines" stderr
++ test_grep "hunk past the end" stderr
+'
+
+test_expect_success 'diff process fallback on mismatched unchanged totals' '
@@ t/t4080-diff-process.sh (new)
+ git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
+ diff synctest.c >actual 2>stderr &&
+ test_grep "changed" actual &&
-+ test_grep "unchanged line count mismatch" stderr
++ test_grep "misaligned" stderr
++'
++
++test_expect_success 'diff process fallback on misaligned hunk gap' '
++ # bad-gap reports hunk 1 1 3 1 on boundary.c: one changed line
++ # on each side, so the total unchanged counts match, but the
++ # unchanged run before the change differs (old line 1 vs new
++ # line 3). A global count check would accept this and emit a
++ # corrupt diff; the per-gap lockstep check rejects it and git
++ # falls back to the builtin algorithm.
++ git -c diff.cdiff.process="$BACKEND --mode=bad-gap" \
++ diff boundary.c >actual 2>stderr &&
++ # The builtin fallback shows both changed regions as additions
++ # (a corrupt-accepted hunk would show NEW5 only as context).
++ test_grep "^+NEW5" actual &&
++ test_grep "^+NEW9" actual &&
++ test_grep "misaligned" stderr
+'
+
+test_expect_success 'diff process fallback on overlapping hunks' '
+ # boundary.c has 10 lines, so both hunks are in bounds
-+ # but they overlap at lines 3-5, triggering the ordering check.
++ # but they overlap at lines 3-4, triggering the ordering check.
+ git -c diff.cdiff.process="$BACKEND --mode=overlap" \
+ diff boundary.c >actual 2>stderr &&
+ test_grep "NEW5" actual &&
-+ test_grep "overlaps with previous" stderr
++ test_grep "overlapping hunks" stderr
+'
+
+test_expect_success 'diff process fallback on malformed hunk line' '
@@ t/t4080-diff-process.sh (new)
+ test_grep "^+NEW5" actual
+'
+
-+test_expect_success 'diff process skipped when tool omits capability' '
-+ git -c diff.cdiff.process="$BACKEND --mode=no-cap" \
++test_expect_success 'diff process fallback on start 0 with nonzero count' '
++ # bad-start reports hunk 0 1 1 1. A start of 0 is valid only for
++ # an empty (count 0) range, so the presentation-to-xdiff
++ # translation rejects it and git falls back to the builtin diff
++ # instead of handing xdiff an out-of-range start.
++ git -c diff.cdiff.process="$BACKEND --mode=bad-start" \
+ diff boundary.c >actual 2>stderr &&
+ test_grep "^-OLD5" actual &&
+ test_grep "^+NEW5" actual &&
++ test_grep "diff process.*failed" stderr
++'
++
++test_expect_success 'diff process caps a flood of hunks and falls back' '
++ # flood emits far more hunks than the file has lines. Git must
++ # stop accumulating and fall back to the builtin diff rather than
++ # grow memory without bound.
++ git -c diff.cdiff.process="$BACKEND --mode=flood" \
++ diff boundary.c >actual 2>stderr &&
++ test_grep "^-OLD5" actual &&
++ test_grep "too many hunks" stderr
++'
++
++test_expect_success 'diff process skipped when tool omits capability' '
++ test_when_finished "rm -f backend.log" &&
++ git -c diff.cdiff.process="$BACKEND --mode=no-cap --log=backend.log" \
++ diff boundary.c >actual 2>stderr &&
++ # Builtin diff runs: all changes appear, including lines 9-10
++ # that a tool-provided hunk would have narrowed away.
++ test_grep "^-OLD5" actual &&
++ test_grep "^-OLD9" actual &&
++ # The process launched (creating the log) but was
++ # never sent a per-file request, so no hunks command is logged.
++ test_path_is_file backend.log &&
++ test_grep ! "command=hunks" backend.log &&
++ test_must_be_empty stderr
++'
++
++test_expect_success 'diff process receives old-oid and new-oid for a blob pair' '
++ test_when_finished "rm -f backend.log" &&
++ cat >oidpair.c <<-\EOF &&
++ int f(void) { return 1; }
++ EOF
++ git add oidpair.c &&
++ git commit -m "oidpair v1" &&
++ old=$(git rev-parse HEAD:oidpair.c) &&
++
++ cat >oidpair.c <<-\EOF &&
++ int f(void) { return 2; }
++ EOF
++ git add oidpair.c &&
++ git commit -m "oidpair v2" &&
++ new=$(git rev-parse HEAD:oidpair.c) &&
++
++ # Both sides are stored blobs, so their object names are sent.
++ git -c diff.cdiff.process="$BACKEND --log=backend.log" \
++ diff HEAD^ HEAD -- oidpair.c >actual 2>stderr &&
++ test_grep "old-oid=$old new-oid=$new" backend.log &&
++ test_must_be_empty stderr
++'
++
++test_expect_success 'diff process omits old-oid and new-oid for textconv content' '
++ test_when_finished "rm -f backend.log" &&
++ write_script oidcat <<-\EOF &&
++ cat "$1"
++ EOF
++ cat >oidtc.c <<-\EOF &&
++ alpha
++ EOF
++ git add oidtc.c &&
++ git commit -m "oidtc v1" &&
++ cat >oidtc.c <<-\EOF &&
++ beta
++ EOF
++ git add oidtc.c &&
++ git commit -m "oidtc v2" &&
++
++ # textconv rewrites the bytes, so the raw-blob object name that
++ # would otherwise identify each side is omitted.
++ git -c diff.cdiff.textconv="./oidcat" \
++ -c diff.cdiff.process="$BACKEND --log=backend.log" \
++ diff HEAD^ HEAD -- oidtc.c >actual 2>stderr &&
++ test_grep "pathname=oidtc.c" backend.log &&
++ test_grep "old-oid=(none) new-oid=(none)" backend.log &&
+ test_must_be_empty stderr
+'
+
5: 6ec6716ea4 ! 6: b2e80f014e diff: bypass diff process with --no-ext-diff and in format-patch
@@ Documentation/diff-options.adoc: endif::git-format-patch[]
`--textconv`::
`--no-textconv`::
+ ## Documentation/gitattributes.adoc ##
+@@ Documentation/gitattributes.adoc: display, which is covered above); and combined diffs (`--cc` and merge
+ diffs), whose protocol would have to be extended from a single old/new
+ pair to one comparison per merge parent.
+
+-`--diff-algorithm` bypasses the process entirely, for every feature
+-listed above. The whitespace-ignoring options (`-w`,
+-`--ignore-space-change`, `--ignore-blank-lines`, and the like),
++`--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
++for every feature listed above. The whitespace-ignoring options
++(`-w`, `--ignore-space-change`, `--ignore-blank-lines`, and the like),
+ `-I<regex>`, and `--anchored` also bypass it for the affected files:
+ the tool is never told about these options, so it could not honor
+ them, and Git falls back to the builtin diff, which does.
+
## builtin/log.c ##
@@ builtin/log.c: int cmd_format_patch(int argc,
if (argc > 1)
@@ diff.h: struct diff_flags {
- /** Disables diff.<driver>.process. */
+ /**
-+ * Disables diff.<driver>.process. Set by --no-ext-diff.
++ * Disables diff.<driver>.process. Set by --no-ext-diff and by
++ * format-patch.
+ */
unsigned no_diff_process;
@@ t/t4080-diff-process.sh: test_expect_success 'diff process bypassed by --diff-al
+ test_path_is_missing backend.log
+'
+
- test_expect_success 'diff process not used by --stat' '
+ test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
test_when_finished "rm -f backend.log" &&
- git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+ printf "a\nb\nc\n" >wsbypass.c &&
6: 3dadafa1bc ! 7: cf5bb8984a blame: consult diff process for no-hunk detection
@@ Commit message
commit as having no changes, causing blame to attribute lines
to earlier commits.
- The consultation happens at the pass_blame_to_parent() callsite
- using diff_process_fill_hunks(), matching how builtin_diff() in
- diff.c uses the same function. A new diff_hunks_xpp() variant
- accepts a pre-populated xpparam_t so callers can pass external
- hunks, while the existing diff_hunks() retains its original
- signature and behavior. The copy-detection callsite is
- unaffected since it does not use the diff process.
+ Introduce xdi_diff_process(), a process-aware xdi_diff() that
+ consults the process, runs xdiff on the tool's hunks or on the
+ builtin algorithm when it does not apply, frees the hunks, and
+ reports DIFF_PROCESS_EQUIVALENT (without running xdiff) so the caller
+ can drop or skip the change. It is the shared consult-then-diff path
+ for consumers that work on raw hunks: blame's pass_blame_to_parent()
+ uses it here, and git log -L reuses it later. builtin_diff() keeps
+ consulting the process directly, because it tests for equivalence
+ early, before its funcname-pattern and word-diff setup, so a
+ reformat-only file short-circuits without that work.
- The subprocess is long-running (one startup cost amortized
- across the blame traversal), but each commit in the file's
- history incurs a round-trip to the tool.
+ Blame's -w option is not communicated to the process and it could not
+ honor it, so blame must fall back to the builtin diff there. Because
+ blame keeps its whitespace flags in sb->xdl_opts rather than diffopt,
+ the process bypass keys off xpp (the flags the diff actually runs
+ with), which covers blame without a guard of its own.
+
+ The subprocess is long-running (one startup cost amortized across the
+ blame traversal), but each commit in the file's history incurs a
+ round-trip to the tool.
Signed-off-by: Michael Montalbo <[email protected]>
@@ blame.c
#include "alloc.h"
#include "commit-slab.h"
#include "bloom.h"
-@@ blame.c: static struct commit *fake_working_tree_commit(struct repository *r,
-
-
-
--static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
-- xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
-+static int diff_hunks_xpp(mmfile_t *file_a, mmfile_t *file_b,
-+ xdl_emit_hunk_consume_func_t hunk_func,
-+ void *cb_data, xpparam_t *xpp)
- {
-- xpparam_t xpp = {0};
- xdemitconf_t xecfg = {0};
- xdemitcb_t ecb = {NULL};
-
-- xpp.flags = xdl_opts;
- xecfg.hunk_func = hunk_func;
- ecb.priv = cb_data;
-- return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
-+ return xdi_diff(file_a, file_b, xpp, &xecfg, &ecb);
-+}
-+
-+static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
-+ xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
-+{
-+ xpparam_t xpp = {0};
-+
-+ xpp.flags = xdl_opts;
-+ return diff_hunks_xpp(file_a, file_b, hunk_func, cb_data, &xpp);
- }
-
- static const char *get_next_line(const char *start, const char *end)
@@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
struct blame_origin *parent, int ignore_diffs)
{
mmfile_t file_p, file_o;
+ xpparam_t xpp = {0};
++ xdemitconf_t xecfg = {0};
++ xdemitcb_t ecb = {NULL};
struct blame_chunk_cb_data d;
struct blame_entry *newdest = NULL;
@@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
sb->num_get_patch++;
- if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
-- die("unable to generate diff (%s -> %s)",
-- oid_to_hex(&parent->commit->object.oid),
-- oid_to_hex(&target->commit->object.oid));
+ xpp.flags = sb->xdl_opts;
++ xecfg.hunk_func = blame_chunk_cb;
++ ecb.priv = &d;
+ /*
-+ * If the diff process considers the files equivalent,
-+ * skip the diff so blame looks past this commit.
++ * Consult the diff process, then attribute the resulting chunks
++ * via blame_chunk_cb. It bypasses the process for the whitespace-
++ * ignoring options it cannot honor (they live in xpp.flags, which
++ * the consultation checks), and when the process reports the blobs
++ * equivalent it runs no diff, so blame passes this commit and looks
++ * past it. Look up the driver by the parent (old) path, as
++ * builtin_diff() does with name_a, so a renamed file resolves to the
++ * same driver across diff, blame, and line-log. Pass no
++ * old-oid/new-oid: blame diffs each blob pair once, so the tool gains
++ * nothing from a per-invocation cache key.
+ */
-+ if (diff_process_fill_hunks(&sb->revs->diffopt, target->path,
-+ &file_p, &file_o, &xpp)
-+ != DIFF_PROCESS_EQUIVALENT) {
-+ if (diff_hunks_xpp(&file_p, &file_o, blame_chunk_cb,
-+ &d, &xpp))
-+ die("unable to generate diff (%s -> %s)",
-+ oid_to_hex(&parent->commit->object.oid),
-+ oid_to_hex(&target->commit->object.oid));
-+ }
-+ free(xpp.external_hunks);
- /* The rest are the same as the parent */
- blame_chunk(&d.dstq, &d.srcq, INT_MAX, d.offset, INT_MAX, 0,
- parent, target, 0);
++ if (xdi_diff_process(&sb->revs->diffopt, parent->path,
++ &file_p, &file_o, NULL, NULL, &xpp, &xecfg, &ecb)
++ == DIFF_PROCESS_ERROR)
+ die("unable to generate diff (%s -> %s)",
+ oid_to_hex(&parent->commit->object.oid),
+ oid_to_hex(&target->commit->object.oid));
+
+ ## diff-process.c ##
+@@
+ #include "sub-process.h"
+ #include "pkt-line.h"
+ #include "strbuf.h"
++#include "xdiff-interface.h"
+ #include "xdiff/xdiff.h"
+
+ #define CAP_HUNKS (1u << 0)
+@@ diff-process.c: enum diff_process_result diff_process_fill_hunks(
+ }
+ return DIFF_PROCESS_SKIP;
+ }
++
++enum diff_process_result xdi_diff_process(
++ struct diff_options *diffopt,
++ const char *path,
++ mmfile_t *file_a,
++ mmfile_t *file_b,
++ const struct object_id *oid_a,
++ const struct object_id *oid_b,
++ xpparam_t *xpp,
++ xdemitconf_t *xecfg,
++ xdemitcb_t *ecb)
++{
++ enum diff_process_result res;
++
++ /*
++ * Consult the diff process, then run xdiff either constrained to
++ * the tool's hunks or, when the process does not apply, computing
++ * the diff itself as a fallback. EQUIVALENT short-circuits: the
++ * caller decides what "no change" means for it (drop the commit,
++ * skip the file, ...), so xdiff is not run.
++ *
++ * A SKIP/ERROR from the process just selects the builtin path
++ * (its warning, if any, was already emitted), so the result then
++ * reflects whether xdiff itself succeeded, not the process.
++ */
++ res = diff_process_fill_hunks(diffopt, path, file_a, file_b,
++ oid_a, oid_b, xpp);
++ if (res == DIFF_PROCESS_EQUIVALENT)
++ return res;
++
++ res = xdi_diff(file_a, file_b, xpp, xecfg, ecb) < 0
++ ? DIFF_PROCESS_ERROR : DIFF_PROCESS_OK;
++
++ FREE_AND_NULL(xpp->external_hunks);
++ xpp->external_hunks_nr = 0;
++ return res;
++}
+
+ ## diff-process.h ##
+@@ diff-process.h: enum diff_process_result diff_process_fill_hunks(
+ const struct object_id *oid_b,
+ xpparam_t *xpp);
+
++/*
++ * Process-aware xdi_diff(): consult the diff process for 'path', then
++ * run xdiff either constrained to the tool's hunks or computing the
++ * diff itself when the process does not apply or fails. Frees any
++ * hunks it obtained before returning.
++ *
++ * Returns DIFF_PROCESS_EQUIVALENT (without running xdiff) when the tool
++ * reports the blobs equal, so the caller can drop or skip the change;
++ * DIFF_PROCESS_OK when xdiff ran (on tool hunks or builtin); and
++ * DIFF_PROCESS_ERROR if xdiff itself errored.
++ *
++ * The caller fills xpp (flags, ignore_regex, anchors) and xecfg/ecb as
++ * for a direct xdi_diff() call. oid_a/oid_b are forwarded to
++ * diff_process_fill_hunks() (see there).
++ */
++enum diff_process_result xdi_diff_process(
++ struct diff_options *diffopt,
++ const char *path,
++ mmfile_t *file_a,
++ mmfile_t *file_b,
++ const struct object_id *oid_a,
++ const struct object_id *oid_b,
++ xpparam_t *xpp,
++ xdemitconf_t *xecfg,
++ xdemitcb_t *ecb);
++
+ #endif /* DIFF_PROCESS_H */
## t/t4080-diff-process.sh ##
-@@ t/t4080-diff-process.sh: test_expect_success 'diff process skipped when tool omits capability' '
+@@ t/t4080-diff-process.sh: test_expect_success 'diff process omits old-oid and new-oid for textconv content
test_must_be_empty stderr
'
@@ t/t4080-diff-process.sh: test_expect_success 'diff process skipped when tool omi
+ test_grep "$CHANGE" line9 &&
+ test_path_is_missing backend.log
+'
++
++test_expect_success 'blame -w bypasses diff process' '
++ test_when_finished "rm -f backend.log" &&
++ printf "alpha\nbeta\ngamma\n" >blamew.c &&
++ git add blamew.c &&
++ git commit -m "add blamew.c" &&
++ orig=$(git rev-parse --short HEAD) &&
++ printf "alpha\n beta \ngamma\n" >blamew.c &&
++ git commit -am "reindent beta" &&
++ reindent=$(git rev-parse --short HEAD) &&
++ # blame -w must ignore the whitespace-only change and attribute
++ # beta to the original commit, not the reindent commit. The tool
++ # is never told about -w, so blame must bypass it (not let tool
++ # hunks override -w).
++ git -c diff.cdiff.process="$BACKEND --mode=whole-file --log=backend.log" \
++ blame -w blamew.c >actual &&
++ sed -n "2p" actual >line2 &&
++ test_grep "$orig" line2 &&
++ test_grep ! "$reindent" line2 &&
++ test_path_is_missing backend.log
++'
+
test_done
-: ---------- > 8: c1d02d0e15 diff: consult diff process for --stat counts
-: ---------- > 9: c3c17ba8fc line-log: consult diff process for range trackingSignature · 1 line
gitgitgadget
PATCH v5 1/9 gitattributes: document how external diff drivers relate to diff features
From: Michael Montalbo <[email protected]>
The "Defining an external diff driver" section explains how to configure diff.<driver>.command but not how the driver relates to the rest of Git's diff machinery. In particular, the command only replaces the textual patch: word diff, function context, color, and the like cannot apply to its output, while the summary formats, blame, and git log -L do not run it at all and keep using the builtin diff.
Spell this out so the scope of an external diff driver is clear.
--- Documentation/gitattributes.adoc | 10 ++++++++++ 1 file changed, 10 insertions(+)
Documentation/gitattributes.adoc +10 -0
@@ -784,6 +784,16 @@ with the above configuration, i.e. `j-c-diff`, with 7 parameters, just like `GIT_EXTERNAL_DIFF` program is called. See linkgit:git[1] for details. +An external diff driver replaces the patch Git would otherwise +produce for the path: Git runs the command and shows its output in +place of its own. Output features that post-process Git's diff do +not apply to it; word diff, function context (`-W`), `--color-moved`, +and coloring all act on Git's builtin diff, not the driver's output. +The driver is consulted only when Git generates a textual patch. The +summary formats (`--stat`, `--numstat`, `--shortstat`, and +`--dirstat`), `git blame`, and `git log -L` do not run it and +continue to use Git's builtin diff. + If the program is able to ignore certain changes (similar to `git diff --ignore-space-change`), then also set the option `trustExitCode` to true. It is then expected to return exit code 1 if
Signature · 1 line
gitgitgadget
PATCH v5 2/9 xdiff: support external hunks via xpparam_t
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add two new xpparam_t fields (external_hunks, external_hunks_nr) that let callers supply pre-computed hunks. When set, xdl_diff() populates the changed[] arrays from these hunks instead of running the diff algorithm, then continues through compaction and emission as usual.
Validate supplied hunks before use. Out-of-bounds line numbers, overlapping or out-of-order hunks, and misaligned unchanged runs are treated as a malformed tool response: xdl_populate_hunks_from_external() warns, returns -1, and xdl_diff() falls back to the builtin diff algorithm for that file. The run of unchanged lines between two hunks (and before the first and after the last) must be the same length on both sides; xdl_build_script() walks the two files in lockstep over unchanged lines, so a balanced total is not enough. Non-negative counts and 1-based starts are instead caller preconditions, checked with BUG(), since the caller normalizes hunks before this point.
On rejection xdl_diff() frees the environment it prepared and falls through to xdl_do_diff(), which prepares a fresh one for the builtin pass.
Skip trim_common_tail() in xdi_diff() when external hunks are present, since external hunks reference line numbers in the original content.
--- xdiff-interface.c | 7 +++- xdiff/xdiff.h | 16 +++++++++ xdiff/xdiffi.c | 84 +++++++++++++++++++++++++++++++++++++++++++++-- xdiff/xprepare.c | 10 ++++++ xdiff/xprepare.h | 1 + 5 files changed, 115 insertions(+), 3 deletions(-)
xdiff-interface.c +6 -1
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t coif (mf1->size >MAX_XDIFF_SIZE || mf2->size >MAX_XDIFF_SIZE )return -1 ; -if (!xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT )) +/* +* External hunks reference line numbers in the original content; +* trimming the tail would change line counts and invalidate them. +*/ +if (!xpp->external_hunks && + !xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT ))trim_common_tail (&a, &b);return xdl_diff (&a, &b, xpp, xecfg, xecb);
xdiff/xdiff.h +16 -0
@@ -78,6 +78,18 @@ typedef struct s_mmbuffer {long size; } mmbuffer_t; +/* +* Hunk descriptor for externally computed diffs, in xdiff's own +* coordinates: line numbers are 1-based and a hunk's start is the +* first line it covers. A caller translates any external "empty side" +* idiom (such as git diff's start-0/count-0) to a 1-based start before +* handing hunks over. +*/ +struct xdl_hunk { +long old_start, old_count; +long new_start, new_count; +}; + typedef struct s_xpparam { unsigned long flags;
@@ -88,6 +100,10 @@ typedef struct s_xpparam {/* See Documentation/diff-options.adoc. */ char **anchors;size_t anchors_nr; + +/* Externally computed hunks: bypass the diff algorithm. Owned by caller. */ +struct xdl_hunk *external_hunks; +size_t external_hunks_nr; } xpparam_t; typedef struct s_xdemitcb {
xdiff/xdiffi.c +82 -2
98 lines not on this page · raw message
xdiff/xprepare.c +10 -0
@@ -471,3 +471,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,return 0 ; } + +/* +* Reset the changed[] array so that no lines are marked as changed. +* Also clears the sentinel slots at changed[-1] and changed[nrec] +* that xdl_change_compact() relies on during backward scans. +*/ +void xdl_clear_changed (xdfile_t *xdf) +{ +memset (xdf->changed -1 ,0 , (xdf->nrec +2 ) *sizeof (bool )); +}
xdiff/xprepare.h +1 -0
@@ -28,6 +28,7 @@int xdl_prepare_env (mmfile_t *mf1,mmfile_t *mf2,xpparam_t const *xpp,xdfenv_t *xe);void xdl_free_env (xdfenv_t *xe); +void xdl_clear_changed (xdfile_t *xdf);
Signature · 1 line
gitgitgadget
PATCH v5 3/9 userdiff: add diff.<driver>.process config
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add the process field to struct userdiff_driver and teach the config parser to populate it from diff.<driver>.process.
--- userdiff.c | 7 +++++++ userdiff.h | 2 ++ 2 files changed, 9 insertions(+)
userdiff.c +7 -0
@@ -509,6 +509,13 @@ int userdiff_config(const char *k, const char *v) drv->algorithm = drv->algorithm_owned;return ret; } +if (!strcmp (type,"process" )) { +int ret; +FREE_AND_NULL (drv->process_owned); + ret =git_config_string (&drv->process_owned, k, v); + drv->process = drv->process_owned; +return ret; + }return 0 ; }
userdiff.h +2 -0
@@ -31,6 +31,8 @@ struct userdiff_driver {char *textconv_owned;struct notes_cache *textconv_cache;int textconv_want_cache; +const char *process; +char *process_owned; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,
Signature · 1 line
gitgitgadget
PATCH v5 4/9 sub-process: separate process lifecycle from hashmap management
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
subprocess_start() and subprocess_stop() couple two concerns:
managing a child process (setup, handshake, teardown) and managing a hashmap that indexes running processes by command string. The hashmap suits callers like convert.c where many files may share one filter process looked up by name, but callers that manage process lifetime through their own data structures do not need it.
Extract subprocess_start_command() and subprocess_stop_command() so callers can reuse the child process setup and handshake machinery without maintaining a hashmap. subprocess_start() and subprocess_stop() become thin wrappers that add hashmap operations on top.
--- sub-process.c | 28 +++++++++++++++++++++++----- sub-process.h | 9 ++++++++- 2 files changed, 31 insertions(+), 6 deletions(-)
sub-process.c +23 -5
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)return (len <0 ) ? len :0 ; } -void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry) +void subprocess_stop_command (struct subprocess_entry *entry) {if (!entry)return ;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry) entry->process.clean_on_exit =0 ;kill (entry->process.pid,SIGTERM );finish_command (&entry->process); +} +void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry) +{ +if (!entry) +return ; + +subprocess_stop_command (entry);hashmap_remove (hashmap, &entry->ent,NULL ); }
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)finish_command (process); } -int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd, +int subprocess_start_command (struct subprocess_entry *entry,const char *cmd,subprocess_start_fn startfn) {int err;
@@ -96,15 +103,26 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, coreturn err; } -hashmap_entry_init (&entry->ent,strhash (cmd)); - err =startfn (entry);if (err) {error ("initialization for subprocess '%s' failed" , cmd); -subprocess_stop (hashmap, entry); +subprocess_stop_command (entry);return err; } +return 0 ; +} + +int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd, +subprocess_start_fn startfn) +{ +int err; + + err =subprocess_start_command (entry, cmd, startfn); +if (err) +return err; + +hashmap_entry_init (&entry->ent,strhash (cmd));hashmap_add (hashmap, &entry->ent);return 0 ; }
sub-process.h +8 -1
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,*/ typedef int (*subprocess_start_fn )(struct subprocess_entry *entry); -/* Start a subprocess and add it to the subprocess hashmap. */ +/* Start a subprocess and run the startfn (typically handshake). */ +int subprocess_start_command (struct subprocess_entry *entry,const char *cmd, +subprocess_start_fn startfn); + +/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */ int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd,subprocess_start_fn startfn); +/* Kill a subprocess. */ +void subprocess_stop_command (struct subprocess_entry *entry); +/* Kill a subprocess and remove it from the subprocess hashmap. */ void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry);
Signature · 1 line
gitgitgadget
PATCH v5 5/9 diff: add long-running diff process via diff.<driver>.process
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add support for external diff processes that communicate via the long-running process protocol (pkt-line over stdin/stdout).
A diff process is configured per userdiff driver:
[diff "cdiff"]
process = /path/to/diff-tool
The tool provides custom line-matching: it receives file pairs and returns hunks that reference line numbers in the content. When textconv is also configured, the tool receives the textconv-transformed content. The tool controls which lines are marked as changed while the display shows the file content. Patch output features (word diff, function context, color) work normally. A new "Which features consult the diff process" documentation section lays out which features use the tool's hunks, which compute independently, and why; the summary formats such as --stat still use the builtin diff for now.
The handshake negotiates version=1 and capability=hunks. Per-file requests send command=hunks, pathname, the old and new blob object names as old-oid/new-oid, and both file contents as packetized data. The tool responds with hunk lines and a status packet (success, error, or abort). On error, Git warns and falls back to the builtin diff algorithm for that file. On abort, Git silently falls back for the current file and stops sending further requests to the tool for the remainder of the session.
old-oid/new-oid name the two blobs so a tool can cache its analysis keyed on the pair. A side's oid is sent only when the content the tool receives is that raw blob: it is omitted under textconv, which rewrites the bytes, and for a working-tree side with no stored object, so an oid that is sent always names the bytes the tool receives. This is where the process protocol diverges from diff.<driver>.command, which never composes with textconv (the command replaces the whole diff and always gets the raw blob). Tools ignore unknown request keys, so old tools skip them.
When the tool returns no hunks followed by status=success, Git treats the file as having no changes and produces no diff output. This also means --exit-code reports no changes for that file.
The subprocess is stored on the userdiff_driver struct and launched on first use. If the process fails to start, the handshake fails, or a communication error occurs mid-stream, the failure is cached on the driver to avoid retrying and re-warning on every subsequent file.
Git falls back to the builtin diff (rather than consulting the tool) when an option the tool cannot honor is in effect: the whitespace-ignoring flags, --ignore-blank-lines, -I<regex>, and --anchored. The bypass keys off the effective diff parameters (xpp) rather than diffopt, so a later caller whose flags live elsewhere is covered uniformly. A change that only adds or removes the trailing newline is likewise not expressible as hunks, so it too uses the builtin diff. The hunk parser ignores unknown trailing fields on a hunk line for response forward-compatibility.
Hunk accumulation is bounded by the combined byte count of the two files, so a misbehaving tool that floods hunk lines cannot grow memory without bound before validation runs.
diff_process_fill_hunks() is the sole public entry point. It handles driver lookup, flag checks, subprocess management, and error reporting, returning an enum that lets callers distinguish "hunks populated" from "files equivalent" from "not applicable" from "tool failure."
--- Documentation/config/diff.adoc | 5 + Documentation/gitattributes.adoc | 249 +++++++++++ Makefile | 2 + diff-process.c | 491 ++++++++++++++++++++ diff-process.h | 49 ++ diff.c | 21 + diff.h | 3 + meson.build | 1 + t/helper/meson.build | 1 + t/helper/test-diff-process-backend.c | 381 ++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/meson.build | 1 + t/t4080-diff-process.sh | 645 +++++++++++++++++++++++++++ userdiff.h | 3 + 15 files changed, 1854 insertions(+) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100644 t/helper/test-diff-process-backend.c create mode 100755 t/t4080-diff-process.sh
Documentation/config/diff.adoc +5 -0
@@ -218,6 +218,11 @@ endif::git-diff[] Set this option to `true` to make the diff driver cache the text conversion outputs. See linkgit:gitattributes[5] for details. +`diff.<driver>.process`:: + The command to run as a long-running diff process that + provides hunks to Git's diff pipeline. + See linkgit:gitattributes[5] for details. + `diff.indentHeuristic`:: Set this option to `false` to disable the default heuristics that shift diff hunk boundaries to make patches easier to read.
Documentation/gitattributes.adoc +249 -0
255 lines not on this page · raw message
Makefile +2 -0
@@ -811,6 +811,7 @@ TEST_BUILTINS_OBJS += test-csprng.o TEST_BUILTINS_OBJS += test-date.o TEST_BUILTINS_OBJS += test-delete-gpgsig.o TEST_BUILTINS_OBJS += test-delta.o +TEST_BUILTINS_OBJS += test-diff-process-backend.o TEST_BUILTINS_OBJS += test-dir-iterator.o TEST_BUILTINS_OBJS += test-drop-caches.o TEST_BUILTINS_OBJS += test-dump-cache-tree.o
@@ -1140,6 +1141,7 @@ LIB_OBJS += diff-delta.o LIB_OBJS += diff-merges.o LIB_OBJS += diff-lib.o LIB_OBJS += diff-no-index.o +LIB_OBJS += diff-process.o LIB_OBJS += diff.o LIB_OBJS += diffcore-break.o LIB_OBJS += diffcore-delta.o
diff-process.c new file +491 -0
491 lines not on this page · raw message
diff-process.h new file +49 -0
49 lines not on this page · raw message
diff.c +21 -0
@@ -25,6 +25,7 @@#include "utf8.h" #include "odb.h" #include "userdiff.h" +#include "diff-process.h" #include "submodule.h" #include "hashmap.h" #include "mem-pool.h"
@@ -4055,6 +4056,25 @@ static void builtin_diff(const char *name_a, xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; + +/* +* Send the blob oids only for a side whose content is the +* raw blob: textconv rewrites the bytes, and a working-tree +* side has no stored oid, so pass NULL there rather than an +* oid that would not name what the tool receives. +*/ +if (diff_process_fill_hunks (o, name_a, &mf1, &mf2, + (textconv_one || !one->oid_valid) ?NULL : &one->oid, + (textconv_two || !two->oid_valid) ?NULL : &two->oid, + &xpp) + ==DIFF_PROCESS_EQUIVALENT ) { +if (textconv_one) +free (mf1.ptr); +if (textconv_two) +free (mf2.ptr); + goto free_ab_and_return; + } + xecfg.ctxlen = o->context; xecfg.interhunkctxlen = o->interhunkcontext; xecfg.flags =XDL_EMIT_FUNCNAMES ;
@@ -4135,6 +4155,7 @@ static void builtin_diff(const char *name_a, }else if (xdi_diff_outf (&mf1, &mf2,NULL , fn_out_consume, &ecbdata, &xpp, &xecfg))die ("unable to generate diff for %s" , one->path); +free (xpp.external_hunks);if (o->word_diff)free_diff_words_data (&ecbdata);if (textconv_one)
diff.h +3 -0
@@ -173,6 +173,9 @@ struct diff_flags {*/ unsigned allow_external; +/** Disables diff.<driver>.process. */ +unsigned no_diff_process; +/** * For communication between the calling program and the options parser; * tell the calling program to signal the presence of difference using
meson.build +1 -0
@@ -328,6 +328,7 @@ libgit_sources = [ 'diff-merges.c', 'diff-lib.c', 'diff-no-index.c', + 'diff-process.c', 'diff.c', 'diffcore-break.c', 'diffcore-delta.c',
t/helper/meson.build +1 -0
@@ -12,6 +12,7 @@ test_tool_sources = [ 'test-date.c', 'test-delete-gpgsig.c', 'test-delta.c', + 'test-diff-process-backend.c', 'test-dir-iterator.c', 'test-drop-caches.c', 'test-dump-cache-tree.c',
t/helper/test-diff-process-backend.c new file +381 -0
381 lines not on this page · raw message
t/helper/test-tool.c +1 -0
@@ -22,6 +22,7 @@ static struct test_cmd cmds[] = { { "date", cmd__date }, { "delete-gpgsig", cmd__delete_gpgsig }, { "delta", cmd__delta }, + { "diff-process-backend", cmd__diff_process_backend }, { "dir-iterator", cmd__dir_iterator }, { "drop-caches", cmd__drop_caches }, { "dump-cache-tree", cmd__dump_cache_tree },
t/helper/test-tool.h +1 -0
@@ -15,6 +15,7 @@ int cmd__csprng(int argc, const char **argv);int cmd__date (int argc,const char **argv);int cmd__delta (int argc,const char **argv);int cmd__delete_gpgsig (int argc,const char **argv); +int cmd__diff_process_backend (int argc,const char **argv);int cmd__dir_iterator (int argc,const char **argv);int cmd__drop_caches (int argc,const char **argv);int cmd__dump_cache_tree (int argc,const char **argv);
t/meson.build +1 -0
@@ -512,6 +512,7 @@ integration_tests = [ 't4072-diff-max-depth.sh', 't4073-diff-stat-name-width.sh', 't4074-diff-shifted-matched-group.sh', + 't4080-diff-process.sh', 't4100-apply-stat.sh', 't4101-apply-nonl.sh', 't4102-apply-rename.sh',
t/t4080-diff-process.sh new file +645 -0
645 lines not on this page · raw message
userdiff.h +3 -0
@@ -3,6 +3,7 @@#include "notes-cache.h" +struct diff_subprocess ;struct index_state ;struct repository ;
@@ -33,6 +34,8 @@ struct userdiff_driver {int textconv_want_cache;const char *process;char *process_owned; +struct diff_subprocess *diff_subprocess; + unsigned diff_process_failed : 1; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,
Signature · 1 line
gitgitgadget
PATCH v5 6/9 diff: bypass diff process with --no-ext-diff and in format-patch
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Make --no-ext-diff disable diff.<driver>.process in addition to diff.<driver>.command. Although the two mechanisms work differently (command replaces Git's output, process feeds hunks back into the pipeline), both invoke external tools and --no-ext-diff means "no external tools."
Replace the OPT_BOOL for --ext-diff with an OPT_CALLBACK that sets both allow_external and no_diff_process, so a single option controls both. Passing --ext-diff explicitly clears no_diff_process, so a later --ext-diff overrides an earlier --no-ext-diff.
Disable the diff process unconditionally in format-patch so that generated patches are always based on the builtin diff algorithm and can be applied reliably by recipients who do not have the external tool.
Document that --diff-algorithm also bypasses the diff process, since it forces the builtin algorithm.
--- Documentation/diff-algorithm-option.adoc | 3 +++ Documentation/diff-options.adoc | 4 +++- Documentation/gitattributes.adoc | 6 +++--- builtin/log.c | 7 +++++++ diff.c | 16 ++++++++++++++-- diff.h | 5 ++++- t/t4080-diff-process.sh | 16 ++++++++++++++++ 7 files changed, 50 insertions(+), 7 deletions(-)
Documentation/diff-algorithm-option.adoc +3 -0
@@ -18,3 +18,6 @@ For instance, if you configured the `diff.algorithm` variable to a non-default value and want to use the default one, then you have to use `--diff-algorithm=default` option. ++ +If you explicitly choose a diff algorithm, it also bypasses +`diff.<driver>.process` (see linkgit:gitattributes[5]).
Documentation/diff-options.adoc +3 -1
@@ -833,7 +833,9 @@ endif::git-format-patch[] to use this option with linkgit:git-log[1] and friends. `--no-ext-diff`:: - Disallow external diff drivers. + Disallow external diff helpers, including + `diff.<driver>.command` and `diff.<driver>.process` + (see linkgit:gitattributes[5]). `--textconv`:: `--no-textconv`::
Documentation/gitattributes.adoc +3 -3
@@ -1073,9 +1073,9 @@ display, which is covered above); and combined diffs (`--cc` and merge diffs), whose protocol would have to be extended from a single old/new pair to one comparison per merge parent. -`--diff-algorithm` bypasses the process entirely, for every feature -listed above. The whitespace-ignoring options (`-w`, -`--ignore-space-change`, `--ignore-blank-lines`, and the like), +`--no-ext-diff` and `--diff-algorithm` bypass the process entirely, +for every feature listed above. The whitespace-ignoring options +(`-w`, `--ignore-space-change`, `--ignore-blank-lines`, and the like), `-I<regex>`, and `--anchored` also bypass it for the affected files: the tool is never told about these options, so it could not honor them, and Git falls back to the builtin diff, which does.
builtin/log.c +7 -0
@@ -2217,6 +2217,13 @@ int cmd_format_patch(int argc,if (argc >1 )die (_ ("unrecognized argument: %s" ), argv[1 ]); +/* +* Disable diff.<driver>.process so that patches generated by +* format-patch are always based on the builtin diff algorithm +* and can be applied reliably. +*/ + rev.diffopt.flags.no_diff_process =1 ; +if (rev.diffopt.output_format &DIFF_FORMAT_NAME )die (_ ("--name-only does not make sense" ));if (rev.diffopt.output_format &DIFF_FORMAT_NAME_STATUS )
diff.c +14 -2
@@ -5948,6 +5948,17 @@ static int diff_opt_submodule(const struct option *opt,return 0 ; } +static int diff_opt_ext_diff (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; + +BUG_ON_OPT_ARG (arg); + options->flags.allow_external = !unset; + options->flags.no_diff_process = unset; +return 0 ; +} +static int diff_opt_textconv (const struct option *opt,const char *arg,int unset) {
@@ -6278,8 +6289,9 @@ struct option *add_diff_options(const struct option *opts, N_("exit with 1 if there were differences, 0 otherwise")),OPT_BOOL (0 ,"quiet" , &options->flags.quick,N_ ("disable all output of the program" )), -OPT_BOOL (0 ,"ext-diff" , &options->flags.allow_external, -N_ ("allow an external diff helper to be executed" )), +OPT_CALLBACK_F (0 ,"ext-diff" , options,NULL , +N_ ("allow an external diff helper to be executed" ), +PARSE_OPT_NOARG , diff_opt_ext_diff),OPT_CALLBACK_F (0 ,"textconv" , options,NULL ,N_ ("run external text conversion filters when comparing binary files" ), PARSE_OPT_NOARG, diff_opt_textconv),
diff.h +4 -1
@@ -173,7 +173,10 @@ struct diff_flags {*/ unsigned allow_external; -/** Disables diff.<driver>.process. */ +/** +* Disables diff.<driver>.process. Set by --no-ext-diff and by +* format-patch. +*/ unsigned no_diff_process;/**
t/t4080-diff-process.sh +16 -0
@@ -398,6 +398,22 @@ test_expect_success 'diff process bypassed by --diff-algorithm' ' test_path_is_missing backend.log ' +test_expect_success 'diff process bypassed by --no-ext-diff' ' + test_when_finished "rm -f backend.log" && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --no-ext-diff worddiff.c >actual && + test_grep "return 999" actual && + test_path_is_missing backend.log +' + +test_expect_success 'diff process not used by format-patch' ' + test_when_finished "rm -f backend.log" && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + format-patch -1 --stdout -- logtest.c >actual && + test_grep "return 2" actual && + test_path_is_missing backend.log +' + test_expect_success 'diff process bypassed under whitespace-ignoring flags' ' test_when_finished "rm -f backend.log" && printf "a\nb\nc\n" >wsbypass.c &&
Signature · 1 line
gitgitgadget
PATCH v5 7/9 blame: consult diff process for no-hunk detection
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
When a diff process is configured via diff.<driver>.process, consult it during blame's per-commit diffing. If the process returns no hunks for a commit's changes to a file, treat the commit as having no changes, causing blame to attribute lines to earlier commits.
Introduce xdi_diff_process(), a process-aware xdi_diff() that consults the process, runs xdiff on the tool's hunks or on the builtin algorithm when it does not apply, frees the hunks, and reports DIFF_PROCESS_EQUIVALENT (without running xdiff) so the caller can drop or skip the change. It is the shared consult-then-diff path for consumers that work on raw hunks: blame's pass_blame_to_parent() uses it here, and git log -L reuses it later. builtin_diff() keeps consulting the process directly, because it tests for equivalence early, before its funcname-pattern and word-diff setup, so a reformat-only file short-circuits without that work.
Blame's -w option is not communicated to the process and it could not honor it, so blame must fall back to the builtin diff there. Because blame keeps its whitespace flags in sb->xdl_opts rather than diffopt, the process bypass keys off xpp (the flags the diff actually runs with), which covers blame without a guard of its own.
The subprocess is long-running (one startup cost amortized across the blame traversal), but each commit in the file's history incurs a round-trip to the tool.
--- blame.c | 24 +++++++- diff-process.c | 38 ++++++++++++ diff-process.h | 26 +++++++++ t/t4080-diff-process.sh | 126 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 213 insertions(+), 1 deletion(-)
blame.c +23 -1
@@ -19,6 +19,8 @@#include "tag.h" #include "trace2.h" #include "blame.h" +#include "diff-process.h" +#include "xdiff-interface.h" #include "alloc.h" #include "commit-slab.h" #include "bloom.h"
@@ -1946,6 +1948,9 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb, struct blame_origin *parent, int ignore_diffs) {mmfile_t file_p, file_o; +xpparam_t xpp = {0 }; +xdemitconf_t xecfg = {0 }; +xdemitcb_t ecb = {NULL };struct blame_chunk_cb_data d;struct blame_entry *newdest =NULL ;
@@ -1964,7 +1969,24 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb, &sb->num_read_blob, ignore_diffs); sb->num_get_patch++; -if (diff_hunks (&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts)) + xpp.flags = sb->xdl_opts; + xecfg.hunk_func = blame_chunk_cb; + ecb.priv = &d; +/* +* Consult the diff process, then attribute the resulting chunks +* via blame_chunk_cb. It bypasses the process for the whitespace- +* ignoring options it cannot honor (they live in xpp.flags, which +* the consultation checks), and when the process reports the blobs +* equivalent it runs no diff, so blame passes this commit and looks +* past it. Look up the driver by the parent (old) path, as +* builtin_diff() does with name_a, so a renamed file resolves to the +* same driver across diff, blame, and line-log. Pass no +* old-oid/new-oid: blame diffs each blob pair once, so the tool gains +* nothing from a per-invocation cache key. +*/ +if (xdi_diff_process (&sb->revs->diffopt, parent->path, + &file_p, &file_o,NULL ,NULL , &xpp, &xecfg, &ecb) + ==DIFF_PROCESS_ERROR )die ("unable to generate diff (%s -> %s)" ,oid_to_hex (&parent->commit->object.oid),oid_to_hex (&target->commit->object.oid));
diff-process.c +38 -0
@@ -37,6 +37,7 @@#include "sub-process.h" #include "pkt-line.h" #include "strbuf.h" +#include "xdiff-interface.h" #include "xdiff/xdiff.h" #define CAP_HUNKS (1u << 0)
@@ -489,3 +490,40 @@ enum diff_process_result diff_process_fill_hunks( }return DIFF_PROCESS_SKIP ; } + +enum diff_process_result xdi_diff_process ( +struct diff_options *diffopt, +const char *path, +mmfile_t *file_a, +mmfile_t *file_b, +const struct object_id *oid_a, +const struct object_id *oid_b, +xpparam_t *xpp, +xdemitconf_t *xecfg, +xdemitcb_t *ecb) +{ +enum diff_process_result res; + +/* +* Consult the diff process, then run xdiff either constrained to +* the tool's hunks or, when the process does not apply, computing +* the diff itself as a fallback. EQUIVALENT short-circuits: the +* caller decides what "no change" means for it (drop the commit, +* skip the file, ...), so xdiff is not run. +* +* A SKIP/ERROR from the process just selects the builtin path +* (its warning, if any, was already emitted), so the result then +* reflects whether xdiff itself succeeded, not the process. +*/ + res =diff_process_fill_hunks (diffopt, path, file_a, file_b, + oid_a, oid_b, xpp); +if (res ==DIFF_PROCESS_EQUIVALENT ) +return res; + + res =xdi_diff (file_a, file_b, xpp, xecfg, ecb) <0 + ?DIFF_PROCESS_ERROR :DIFF_PROCESS_OK ; + +FREE_AND_NULL (xpp->external_hunks); + xpp->external_hunks_nr =0 ; +return res; +}
diff-process.h +26 -0
@@ -46,4 +46,30 @@ enum diff_process_result diff_process_fill_hunks(const struct object_id *oid_b, xpparam_t *xpp); +/* +* Process-aware xdi_diff(): consult the diff process for 'path', then +* run xdiff either constrained to the tool's hunks or computing the +* diff itself when the process does not apply or fails. Frees any +* hunks it obtained before returning. +* +* Returns DIFF_PROCESS_EQUIVALENT (without running xdiff) when the tool +* reports the blobs equal, so the caller can drop or skip the change; +* DIFF_PROCESS_OK when xdiff ran (on tool hunks or builtin); and +* DIFF_PROCESS_ERROR if xdiff itself errored. +* +* The caller fills xpp (flags, ignore_regex, anchors) and xecfg/ecb as +* for a direct xdi_diff() call. oid_a/oid_b are forwarded to +* diff_process_fill_hunks() (see there). +*/ +enum diff_process_result xdi_diff_process ( +struct diff_options *diffopt, +const char *path, +mmfile_t *file_a, +mmfile_t *file_b, +const struct object_id *oid_a, +const struct object_id *oid_b, +xpparam_t *xpp, +xdemitconf_t *xecfg, +xdemitcb_t *ecb); +#endif /* DIFF_PROCESS_H */
t/t4080-diff-process.sh +126 -0
130 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v5 8/9 diff: consult diff process for --stat counts
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
builtin_diff() already consults a configured diff.<driver>.process: a file the tool reports as equivalent emits no patch, and otherwise the tool's hunks drive the output. builtin_diffstat() ran its own xdiff and ignored the process, so "git diff --stat" still counted a byte-level change for a file that "git diff" showed as unchanged.
Consult diff_process_fill_hunks() before the stat xdiff, as builtin_diff() does. On DIFF_PROCESS_EQUIVALENT, skip the xdiff so the file keeps its zero inserted and deleted counts and the existing "nothing changed" pruning drops it, matching the empty patch. Otherwise the tool's hunks, or the builtin fallback, feed the counts through the shared xpparam_t.
Like the builtin summary path, builtin_diffstat() does not apply textconv, so the process is consulted on the raw blob content here, unlike builtin_diff() which sends textconv'd content. This keeps "git diff --stat" counting raw lines as it does today; the asymmetry between patch output and summary counts under textconv predates this change. Because the content is the raw blob, the stat path sends the blob object names to the tool (old-oid/new-oid) for any stored blob, where the patch path omits the oid under textconv.
Move the summary formats out of the "not yet wired" group of the "Which features consult the diff process" documentation and into the list of features that use the tool's hunks, noting the raw, non-textconv content they receive. Document that the line-counting --dirstat=lines follows these counts while the default --dirstat does not, and that summary formats and blame (only under --textconv) differ from patch output in whether they textconv the content the tool sees.
Add tests covering counts from the tool's hunks (--numstat, --shortstat), an equivalent file producing no stat line, --stat --exit-code, the raw non-textconv content the tool receives, a multi-file mix of equivalent and changed files, and a mode-only change.
--- Documentation/gitattributes.adoc | 33 +++++--- diff.c | 20 ++++- t/t4080-diff-process.sh | 130 +++++++++++++++++++++++++++++++ 3 files changed, 172 insertions(+), 11 deletions(-)
Documentation/gitattributes.adoc +23 -10
@@ -874,7 +874,10 @@ a flush packet, followed by the old and new file content as packetized data, each terminated with a flush packet. The pathname is relative to the repository root. When `diff.<name>.textconv` is also set, the tool receives the textconv-transformed content rather than the -raw blob. Git does not send binary files to the diff process. +raw blob, matching what the consuming feature itself diffs: patch +output is textconv'd, the summary formats (noted below) are not, and +`git blame` applies textconv only under `--textconv`. Git does not +send binary files to the diff process. ----------------------- packet: git> command=hunks
@@ -960,8 +963,8 @@ still slide or regroup those changes against matching context for display, exactly as it compacts its own diffs, so the tool controls which lines are reported as changed, not the precise hunk boundaries. Patch output features (word diff, function context, color) work -normally. Summary formats such as `--stat` still compute their counts -with the builtin diff for now; see "Which features consult the diff +normally, as do summary formats like `--stat`. Not every feature +consults the process, though; see "Which features consult the diff process" below for the full picture and the reasoning behind it. If no hunk lines precede the flush, followed by "success", Git
@@ -1040,6 +1043,17 @@ of the builtin algorithm: hunks without any further negotiation. - `git blame`: a commit whose change the tool reports as equivalent is skipped, and its lines are attributed to an earlier commit. +- `--stat`, `--numstat`, and `--shortstat`: the inserted and deleted + counts come from the tool's hunks, so a file the tool calls + equivalent contributes no stat line, matching the empty patch that + `git diff` produces for it. These summary formats do not apply + textconv (just as the builtin summary path does not), so the tool + is consulted on the raw blob content even when a `textconv` is also + configured for patch output; this mirrors how builtin `--stat` + already counts raw lines rather than the textconv'd view. The + line-counting `--dirstat=lines` uses these same counts; the default + `--dirstat`, which weighs byte changes, is computed on its own and + does not consult the tool. Features that ask a different question do not consult the process, by design:
@@ -1065,13 +1079,12 @@ design: - `--raw`, `--name-only`, and `--name-status` compare object ids at the tree level and never run a line-level diff at all. -Some features ask "which lines changed" but still use the builtin -algorithm for now, and may consult the process in a later change: the -summary formats (`--stat`, `--numstat`, `--shortstat`); `git log -L`'s -commit selection and parent range propagation (as distinct from its -display, which is covered above); and combined diffs (`--cc` and merge -diffs), whose protocol would have to be extended from a single old/new -pair to one comparison per merge parent. +Two cases ask "which lines changed" but still use the builtin +algorithm, and may consult the process in a later change: `git log +-L`'s commit selection and parent range propagation (as distinct from +its display, which is covered above), and combined diffs (`--cc` and +merge diffs), whose protocol would have to be extended from a single +old/new pair to one comparison per merge parent. `--no-ext-diff` and `--diff-algorithm` bypass the process entirely, for every feature listed above. The whitespace-ignoring options
diff.c +19 -1
@@ -4269,9 +4269,27 @@ static void builtin_diffstat(const char *name_a, const char *name_b, xecfg.ctxlen = o->context; xecfg.interhunkctxlen = o->interhunkcontext; xecfg.flags =XDL_EMIT_NO_HUNK_HDR ; -if (xdi_diff_outf (&mf1, &mf2,NULL , +/* +* Consult the diff process so --stat reflects the +* tool's view of which lines changed rather than the +* builtin line diff. --stat never applies textconv, so +* the tool is fed the same raw mmfiles the stat itself +* diffs (unlike builtin_diff, which consults the process +* on textconv'd content). +* When the tool reports the files as equivalent we skip +* xdiff entirely, leaving added and deleted at zero so +* the file is pruned below, just as builtin_diff() emits +* no patch for an equivalent file. +*/ +if (diff_process_fill_hunks (o, name_a, &mf1, &mf2, + one->oid_valid ? &one->oid :NULL , + two->oid_valid ? &two->oid :NULL , + &xpp) + !=DIFF_PROCESS_EQUIVALENT && +xdi_diff_outf (&mf1, &mf2,NULL , diffstat_consume, diffstat, &xpp, &xecfg))die ("unable to generate diffstat for %s" , one->path); +free (xpp.external_hunks);if (DIFF_FILE_VALID (one) &&DIFF_FILE_VALID (two)) { struct diffstat_file *file =
t/t4080-diff-process.sh +130 -0
142 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v5 9/9 line-log: consult diff process for range tracking
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
git log -L tracks line ranges by diffing each commit against its parent in collect_diff(). This pass used the builtin diff while the displayed diff (builtin_diff()) consults a configured diff.<driver>.process, so the two could disagree: a reformat-only commit selected by builtin tracking was then rendered with an empty diff because the tool reported the files equivalent.
Consult the process in collect_diff() too, mirroring the blame integration. When the tool reports the files equivalent, collect no ranges; the tracked range then maps across unchanged and the commit drops out of the log, matching what is displayed. Like the summary formats, the tracking pass diffs raw content, so the tool is consulted on the raw blobs here.
--- Documentation/gitattributes.adoc | 20 ++++++++++--------- line-log.c | 33 ++++++++++++++++++++++++++++---- t/t4080-diff-process.sh | 33 ++++++++++++++++++++++++++++++++ 3 files changed, 73 insertions(+), 13 deletions(-)
Documentation/gitattributes.adoc +11 -9
@@ -1037,12 +1037,16 @@ Features that ask "which lines changed" use the tool's hunks in place of the builtin algorithm: - `git diff` patch output, together with everything layered on it: - word diff, function context (`-W`), `--color-moved`, the `@@` hunk - headers, and the `-L` line-range display. These operate on the - lines the patch step already emitted, so they reflect the tool's - hunks without any further negotiation. + word diff, function context (`-W`), `--color-moved`, and the `@@` + hunk headers. These operate on the lines the patch step already + emitted, so they reflect the tool's hunks without any further + negotiation. - `git blame`: a commit whose change the tool reports as equivalent is skipped, and its lines are attributed to an earlier commit. +- `git log -L`: both the line-range display and the underlying range + tracking consult the tool, so a commit it reports as equivalent is + dropped from the log (its tracked range maps across unchanged) + rather than selected and then shown with an empty diff. - `--stat`, `--numstat`, and `--shortstat`: the inserted and deleted counts come from the tool's hunks, so a file the tool calls equivalent contributes no stat line, matching the empty patch that
@@ -1079,11 +1083,9 @@ design: - `--raw`, `--name-only`, and `--name-status` compare object ids at the tree level and never run a line-level diff at all. -Two cases ask "which lines changed" but still use the builtin -algorithm, and may consult the process in a later change: `git log --L`'s commit selection and parent range propagation (as distinct from -its display, which is covered above), and combined diffs (`--cc` and -merge diffs), whose protocol would have to be extended from a single +Combined diffs (`--cc` and merge diffs) ask "which lines changed" but +still use the builtin algorithm, and may consult the process in a +later change; their protocol would have to be extended from a single old/new pair to one comparison per merge parent. `--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
line-log.c +29 -4
@@ -7,11 +7,11 @@#include "tag.h" #include "tree.h" #include "diff.h" +#include "diff-process.h" #include "commit.h" #include "decorate.h" #include "repository.h" #include "revision.h" -#include "xdiff-interface.h" #include "strbuf.h" #include "line-log.h" #include "setup.h"
@@ -330,12 +330,15 @@ static int collect_diff_cb(long start_a, long count_a,return 0 ; } -static int collect_diff (mmfile_t *parent,mmfile_t *target,struct diff_ranges *out) +static int collect_diff (struct diff_options *diffopt,const char *path, +mmfile_t *parent,mmfile_t *target, +struct diff_ranges *out) {struct collect_diff_cbdata cbdata = {NULL };xpparam_t xpp;xdemitconf_t xecfg;xdemitcb_t ecb; +int ret =0 ;memset (&xpp,0 ,sizeof (xpp));memset (&xecfg,0 ,sizeof (xecfg));
@@ -345,7 +348,23 @@ static int collect_diff(mmfile_t *parent, mmfile_t *target, struct diff_ranges * xecfg.hunk_func = collect_diff_cb;memset (&ecb,0 ,sizeof (ecb)); ecb.priv = &cbdata; -return xdi_diff (parent, target, &xpp, &xecfg, &ecb); + +/* +* Consult the diff process so range tracking agrees with the +* diff that will be shown. When the tool reports the files as +* equivalent we collect no ranges, so the tracked range maps +* across unchanged and the commit drops out of the log, rather +* than being selected here but rendered with an empty diff by +* the process-aware builtin_diff(). Blob oids are not threaded to +* this path yet, so pass NULL and send no old-oid/new-oid (a later +* change can supply the pair, where they would let the tool cache +* across the range-tracking and display passes over the same +* commit). +*/ +if (xdi_diff_process (diffopt, path, parent, target, +NULL ,NULL , &xpp, &xecfg, &ecb) ==DIFF_PROCESS_ERROR ) + ret =-1 ; +return ret; }/*
@@ -927,7 +946,13 @@ static int process_diff_filepair(struct rev_info *rev, }diff_ranges_init (&diff); -if (collect_diff (&file_parent, &file_target, &diff)) +/* +* Select the driver by the old (parent) path, as builtin_diff() does +* with name_a, so a renamed file resolves to the same driver for +* range tracking as for the diff that is shown. +*/ +if (collect_diff (&rev->diffopt, pair->one->path, + &file_parent, &file_target, &diff))die ("unable to generate diff for %s" , pair->one->path);/* NEEDSWORK should apply some heuristics to prevent mismatches */
t/t4080-diff-process.sh +33 -0
@@ -914,4 +914,37 @@ test_expect_success 'blame -w bypasses diff process' ' test_path_is_missing backend.log ' +# +# Line-log (git log -L) range tracking. +# + +test_expect_success 'diff process drops equivalent commit from log -L' ' + test_when_finished "rm -f backend.log" && + cat >linelog.c <<-\EOF && + int tracked(void) { return 1; } + EOF + git add linelog.c && + git commit -m "add linelog.c" && + + cat >linelog.c <<-\EOF && + int tracked(void) { return 2; } + EOF + git commit -am "change tracked line" && + + # Builtin line tracking selects the change commit. + git log --no-ext-diff -L1,1:linelog.c --format="%s" >builtin && + test_grep "change tracked line" builtin && + + # With the tool reporting the change as equivalent, tracking + # drops the commit (the range maps across unchanged) instead of + # selecting it and rendering an empty diff. + git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \ + log -L1,1:linelog.c --format="%s" >actual && + test_grep ! "change tracked line" actual && + # The creating commit still appears, so the change commit was + # selectively dropped rather than the whole log going empty. + test_grep "add linelog.c" actual && + test_grep "command=hunks pathname=linelog.c" backend.log +' + test_done
Signature · 1 line
gitgitgadget
Quoted 8 lines · Michael Montalbo via GitGitGadget
A "Which features consult the diff process" section in gitattributes(5) lays out, per feature, why each does or does not consult the process (patch output, blame, summary formats, and the -L line-range view do; pickaxe -G, patch-id, merge, range-diff, --check, and --raw do not, with reasons). Combined diffs (--cc) remain on the builtin algorithm and are noted as future work.
Changes since v4:
This round does not play well with the mm/line-log-limited-ops topic, unfortunately, it seems.
Quoted 10 lines · Junio C Hamano
Quoted 7 lines · Michael Montalbo via GitGitGadget
A "Which features consult the diff process" section in gitattributes(5) lays out, per feature, why each does or does not consult the process (patch output, blame, summary formats, and the -L line-range view do; pickaxe -G, patch-id, merge, range-diff, --check, and --raw do not, with reasons). Combined diffs (--cc) remain on the builtin algorithm and are noted as future work.
Quoted 4 lines
Quoted 1 line
Changes since v4:
This round does not play well with the
mm/line-log-limited-opstopic, unfortunately, it seems.
Ah, thanks for the heads up and apologies for the hiccup. I will rebase this series on top of mm/line-log-limited-ops and make sure the two topics function together correctly.
v6 cover diff: add diff.<driver>.process for external hunk providers
In reply to Michael Montalbo via GitGitGadgetLanguage-aware diff tools (e.g., Difftastic) and format-specific analyzers can produce better line matching than Git's builtin diff algorithm, but diff.<driver>.command replaces Git's diff output with the program's own output, so display features like word diff, function context, and color cannot operate on it; and because the program is consulted only for that patch output, blame, --stat, and git log -L fall back to Git's builtin line matching and cannot benefit from the tool at all.
This series adds diff.<driver>.process, a long-running subprocess protocol that lets an external tool control which lines Git considers changed while Git handles all output formatting. The protocol follows filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation, one process per Git invocation.
The tool receives both file versions and returns changed regions (line ranges in the old and new file). Git validates and feeds them into the xdiff pipeline in place of the builtin diff algorithm. When the tool returns no hunks, Git treats the files as having no changes, which propagates through patch output, the --stat summary, blame, and git log -L. The request also carries the two blobs' object names (old-oid/new-oid) so a tool can cache its analysis keyed on the pair.
* Patch 1: document how an external diff driver (diff.<driver>.command) relates to the rest of Git's diff features, so the contrast with the new process driver is clear.
* Patch 2: xdiff plumbing for externally supplied hunks.
* Patch 3: diff.<driver>.process config key.
* Patch 4: refactor subprocess API to separate process lifecycle from hashmap management, since the diff process stores its subprocess on the userdiff driver rather than in a hashmap.
* Patch 5: the main feature, including the old-oid/new-oid request metadata for blob-pair caching.
* Patch 6: bypass knobs (--no-ext-diff, format-patch).
* Patch 7: blame integration so the tool can declare commits as having no changes; introduces the shared xdi_diff_process() consult-then-diff helper that blame and git log -L both use.
* Patch 8: --stat/--numstat/--shortstat consult the tool, so the summary agrees with the patch output.
* Patch 9: git log -L range tracking consults the tool, so a reformat-only commit is dropped from the log rather than shown with an empty diff.
A "Which features consult the diff process" section in gitattributes(5) lays out, per feature, why each does or does not consult the process (patch output, blame, summary formats, and the -L line-range view do; pickaxe -G, patch-id, merge, range-diff, --check, and --raw do not, with reasons). Combined diffs (--cc) remain on the builtin algorithm and are noted as future work.
Changes since v5:
* Changed series to be based on top of the in-flight mm/line-log-limited-ops:
https://lore.kernel.org/git/[email protected]/
* builtin_diffstat() now routes the process's hunks through that topic's -L line-range filter, so "git log -L --stat" scopes the tool's changed-line counts to the tracked range (patch 8, with a new t4080 test).
Michael Montalbo (9):
gitattributes: document how external diff drivers relate to diff
features
xdiff: support external hunks via xpparam_t
userdiff: add diff.<driver>.process config
sub-process: separate process lifecycle from hashmap management
diff: add long-running diff process via diff.<driver>.process
diff: bypass diff process with --no-ext-diff and in format-patch
blame: consult diff process for no-hunk detection
diff: consult diff process for --stat counts
line-log: consult diff process for range tracking
Documentation/config/diff.adoc | 5 + Documentation/diff-algorithm-option.adoc | 3 + Documentation/diff-options.adoc | 4 +- Documentation/gitattributes.adoc | 274 +++++++ Makefile | 2 + blame.c | 24 +- builtin/log.c | 7 + diff-process.c | 529 ++++++++++++ diff-process.h | 75 ++ diff.c | 84 +- diff.h | 6 + line-log.c | 33 +- meson.build | 1 + sub-process.c | 28 +- sub-process.h | 9 +- t/helper/meson.build | 1 + t/helper/test-diff-process-backend.c | 381 +++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/meson.build | 1 + t/t4080-diff-process.sh | 996 +++++++++++++++++++++++ userdiff.c | 7 + userdiff.h | 5 + xdiff-interface.c | 7 +- xdiff/xdiff.h | 16 + xdiff/xdiffi.c | 84 +- xdiff/xprepare.c | 10 + xdiff/xprepare.h | 1 + 28 files changed, 2566 insertions(+), 29 deletions(-) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100644 t/helper/test-diff-process-backend.c create mode 100755 t/t4080-diff-process.sh
base-commit: f67c51df064d2b64b257bd8c17d757cc0ce1b7fc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v6
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v6
Pull-Request: https://github.com/gitgitgadget/git/pull/2120
Range-diff vs v5:
1: 0fd994a3d3 = 1: b4a1ff4dea gitattributes: document how external diff drivers relate to diff features
2: 2004502549 = 2: ed2db0ac59 xdiff: support external hunks via xpparam_t
3: 926cf01af6 = 3: 115e31d806 userdiff: add diff.<driver>.process config
4: 363d459ff6 = 4: 850c7cbcf5 sub-process: separate process lifecycle from hashmap management
5: d003bc1f15 = 5: 4526809b7f diff: add long-running diff process via diff.<driver>.process
6: b2e80f014e = 6: 4795743ab9 diff: bypass diff process with --no-ext-diff and in format-patch
7: cf5bb8984a = 7: 00573a88a9 blame: consult diff process for no-hunk detection
8: c1d02d0e15 ! 8: 7e3ba56967 diff: consult diff process for --stat counts
@@ Commit message
Otherwise the tool's hunks, or the builtin fallback, feed the counts
through the shared xpparam_t.
+ Under -L, route the surviving hunks through the same line-range filter
+ builtin_diffstat() already uses for a tracked range, so a
+ process-provided diff is scoped to that range: "git log -L<range>
+ --stat" counts the tool's changed lines within the range rather than
+ the builtin line diff's.
+
Like the builtin summary path, builtin_diffstat() does not apply
textconv, so the process is consulted on the raw blob content here,
unlike builtin_diff() which sends textconv'd content. This keeps
@@ Commit message
Add tests covering counts from the tool's hunks (--numstat,
--shortstat), an equivalent file producing no stat line, --stat
--exit-code, the raw non-textconv content the tool receives, a
- multi-file mix of equivalent and changed files, and a mode-only
- change.
+ multi-file mix of equivalent and changed files, a mode-only change,
+ and a range-scoped --stat under "git log -L" that reflects the tool's
+ hunks.
Signed-off-by: Michael Montalbo <[email protected]>
@@ diff.c: static void builtin_diffstat(const char *name_a, const char *name_b,
xecfg.ctxlen = o->context;
xecfg.interhunkctxlen = o->interhunkcontext;
xecfg.flags = XDL_EMIT_NO_HUNK_HDR;
-- if (xdi_diff_outf(&mf1, &mf2, NULL,
+-
+- if (p->line_ranges) {
+- struct line_range_filter lr_filter;
+-
+- line_range_filter_init(&lr_filter, p->line_ranges,
+- diffstat_consume, diffstat);
+-
+- if (line_range_filter_diff(&lr_filter, &mf1, &mf2,
+- &xpp, &xecfg))
+ /*
+ * Consult the diff process so --stat reflects the
+ * tool's view of which lines changed rather than the
@@ diff.c: static void builtin_diffstat(const char *name_a, const char *name_b,
+ * xdiff entirely, leaving added and deleted at zero so
+ * the file is pruned below, just as builtin_diff() emits
+ * no patch for an equivalent file.
++ *
++ * Under -L, feed the tool's hunks through the same
++ * line-range filter the builtin stat uses, so a
++ * process-provided diff is scoped to the tracked range.
+ */
+ if (diff_process_fill_hunks(o, name_a, &mf1, &mf2,
+ one->oid_valid ? &one->oid : NULL,
+ two->oid_valid ? &two->oid : NULL,
+ &xpp)
-+ != DIFF_PROCESS_EQUIVALENT &&
-+ xdi_diff_outf(&mf1, &mf2, NULL,
- diffstat_consume, diffstat, &xpp, &xecfg))
- die("unable to generate diffstat for %s", one->path);
++ != DIFF_PROCESS_EQUIVALENT) {
++ if (p->line_ranges) {
++ struct line_range_filter lr_filter;
++
++ line_range_filter_init(&lr_filter, p->line_ranges,
++ diffstat_consume, diffstat);
++
++ if (line_range_filter_diff(&lr_filter, &mf1, &mf2,
++ &xpp, &xecfg))
++ die("unable to generate diffstat for %s",
++ one->path);
++ } else if (xdi_diff_outf(&mf1, &mf2, NULL, diffstat_consume,
++ diffstat, &xpp, &xecfg))
+ die("unable to generate diffstat for %s",
+ one->path);
+- } else if (xdi_diff_outf(&mf1, &mf2, NULL,
+- diffstat_consume, diffstat, &xpp, &xecfg))
+- die("unable to generate diffstat for %s", one->path);
++ }
+ free(xpp.external_hunks);
if (DIFF_FILE_VALID(one) && DIFF_FILE_VALID(two)) {
@@ t/t4080-diff-process.sh: test_expect_success 'diff process with --exit-code and
+ test_grep "2 deletions" actual
+'
+
++test_expect_success 'diff process scopes --stat to the tracked range under log -L' '
++ test_when_finished "rm -f backend.log" &&
++ cat >rangestat.c <<-\EOF &&
++ line1
++ line2
++ line3
++ line4
++ OLD5
++ OLD6
++ line7
++ line8
++ OLD9
++ OLD10
++ EOF
++ git add rangestat.c &&
++ git commit -m "add rangestat.c" &&
++
++ cat >rangestat.c <<-\EOF &&
++ line1
++ line2
++ line3
++ line4
++ NEW5
++ NEW6
++ line7
++ line8
++ NEW9
++ NEW10
++ EOF
++ git add rangestat.c &&
++ git commit -m "change rangestat.c" &&
++
++ # The file changes at lines 5-6 and 9-10, but fixed-hunk reports
++ # only 5-6. The builtin line diff counts both regions (4/4); the
++ # tool hunks flow through the same line-range filter the stat uses,
++ # so the range-scoped stat reflects the tool view instead (2/2).
++ git log --no-ext-diff -L1,10:rangestat.c --oneline --stat >builtin &&
++ test_grep "4 insertions(+), 4 deletions(-)" builtin &&
++
++ git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
++ log -L1,10:rangestat.c --oneline --stat >actual &&
++ test_grep "2 insertions(+), 2 deletions(-)" actual &&
++ test_grep ! "4 insertions" actual &&
++ test_grep "command=hunks pathname=rangestat.c" backend.log
++'
++
+test_expect_success 'diff process equivalent file makes --stat --exit-code succeed' '
+ # The tool reports worddiff.c equivalent, so --exit-code reports
+ # no change (0); the builtin diff would report a change (1). 9: c3c17ba8fc = 9: ffa6954d67 line-log: consult diff process for range tracking
Signature · 1 line
gitgitgadget
PATCH v6 1/9 gitattributes: document how external diff drivers relate to diff features
From: Michael Montalbo <[email protected]>
The "Defining an external diff driver" section explains how to configure diff.<driver>.command but not how the driver relates to the rest of Git's diff machinery. In particular, the command only replaces the textual patch: word diff, function context, color, and the like cannot apply to its output, while the summary formats, blame, and git log -L do not run it at all and keep using the builtin diff.
Spell this out so the scope of an external diff driver is clear.
--- Documentation/gitattributes.adoc | 10 ++++++++++ 1 file changed, 10 insertions(+)
Documentation/gitattributes.adoc +10 -0
@@ -784,6 +784,16 @@ with the above configuration, i.e. `j-c-diff`, with 7 parameters, just like `GIT_EXTERNAL_DIFF` program is called. See linkgit:git[1] for details. +An external diff driver replaces the patch Git would otherwise +produce for the path: Git runs the command and shows its output in +place of its own. Output features that post-process Git's diff do +not apply to it; word diff, function context (`-W`), `--color-moved`, +and coloring all act on Git's builtin diff, not the driver's output. +The driver is consulted only when Git generates a textual patch. The +summary formats (`--stat`, `--numstat`, `--shortstat`, and +`--dirstat`), `git blame`, and `git log -L` do not run it and +continue to use Git's builtin diff. + If the program is able to ignore certain changes (similar to `git diff --ignore-space-change`), then also set the option `trustExitCode` to true. It is then expected to return exit code 1 if
Signature · 1 line
gitgitgadget
PATCH v6 2/9 xdiff: support external hunks via xpparam_t
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add two new xpparam_t fields (external_hunks, external_hunks_nr) that let callers supply pre-computed hunks. When set, xdl_diff() populates the changed[] arrays from these hunks instead of running the diff algorithm, then continues through compaction and emission as usual.
Validate supplied hunks before use. Out-of-bounds line numbers, overlapping or out-of-order hunks, and misaligned unchanged runs are treated as a malformed tool response: xdl_populate_hunks_from_external() warns, returns -1, and xdl_diff() falls back to the builtin diff algorithm for that file. The run of unchanged lines between two hunks (and before the first and after the last) must be the same length on both sides; xdl_build_script() walks the two files in lockstep over unchanged lines, so a balanced total is not enough. Non-negative counts and 1-based starts are instead caller preconditions, checked with BUG(), since the caller normalizes hunks before this point.
On rejection xdl_diff() frees the environment it prepared and falls through to xdl_do_diff(), which prepares a fresh one for the builtin pass.
Skip trim_common_tail() in xdi_diff() when external hunks are present, since external hunks reference line numbers in the original content.
--- xdiff-interface.c | 7 +++- xdiff/xdiff.h | 16 +++++++++ xdiff/xdiffi.c | 84 +++++++++++++++++++++++++++++++++++++++++++++-- xdiff/xprepare.c | 10 ++++++ xdiff/xprepare.h | 1 + 5 files changed, 115 insertions(+), 3 deletions(-)
xdiff-interface.c +6 -1
@@ -143,7 +143,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t coif (mf1->size >MAX_XDIFF_SIZE || mf2->size >MAX_XDIFF_SIZE )return -1 ; -if (!xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT )) +/* +* External hunks reference line numbers in the original content; +* trimming the tail would change line counts and invalidate them. +*/ +if (!xpp->external_hunks && + !xecfg->ctxlen && !(xecfg->flags &XDL_EMIT_FUNCCONTEXT ))trim_common_tail (&a, &b);return xdl_diff (&a, &b, xpp, xecfg, xecb);
xdiff/xdiff.h +16 -0
@@ -78,6 +78,18 @@ typedef struct s_mmbuffer {long size; } mmbuffer_t; +/* +* Hunk descriptor for externally computed diffs, in xdiff's own +* coordinates: line numbers are 1-based and a hunk's start is the +* first line it covers. A caller translates any external "empty side" +* idiom (such as git diff's start-0/count-0) to a 1-based start before +* handing hunks over. +*/ +struct xdl_hunk { +long old_start, old_count; +long new_start, new_count; +}; + typedef struct s_xpparam { unsigned long flags;
@@ -88,6 +100,10 @@ typedef struct s_xpparam {/* See Documentation/diff-options.adoc. */ char **anchors;size_t anchors_nr; + +/* Externally computed hunks: bypass the diff algorithm. Owned by caller. */ +struct xdl_hunk *external_hunks; +size_t external_hunks_nr; } xpparam_t; typedef struct s_xdemitcb {
xdiff/xdiffi.c +82 -2
98 lines not on this page · raw message
xdiff/xprepare.c +10 -0
@@ -471,3 +471,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,return 0 ; } + +/* +* Reset the changed[] array so that no lines are marked as changed. +* Also clears the sentinel slots at changed[-1] and changed[nrec] +* that xdl_change_compact() relies on during backward scans. +*/ +void xdl_clear_changed (xdfile_t *xdf) +{ +memset (xdf->changed -1 ,0 , (xdf->nrec +2 ) *sizeof (bool )); +}
xdiff/xprepare.h +1 -0
@@ -28,6 +28,7 @@int xdl_prepare_env (mmfile_t *mf1,mmfile_t *mf2,xpparam_t const *xpp,xdfenv_t *xe);void xdl_free_env (xdfenv_t *xe); +void xdl_clear_changed (xdfile_t *xdf);
Signature · 1 line
gitgitgadget
PATCH v6 3/9 userdiff: add diff.<driver>.process config
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add the process field to struct userdiff_driver and teach the config parser to populate it from diff.<driver>.process.
--- userdiff.c | 7 +++++++ userdiff.h | 2 ++ 2 files changed, 9 insertions(+)
userdiff.c +7 -0
@@ -509,6 +509,13 @@ int userdiff_config(const char *k, const char *v) drv->algorithm = drv->algorithm_owned;return ret; } +if (!strcmp (type,"process" )) { +int ret; +FREE_AND_NULL (drv->process_owned); + ret =git_config_string (&drv->process_owned, k, v); + drv->process = drv->process_owned; +return ret; + }return 0 ; }
userdiff.h +2 -0
@@ -31,6 +31,8 @@ struct userdiff_driver {char *textconv_owned;struct notes_cache *textconv_cache;int textconv_want_cache; +const char *process; +char *process_owned; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,
Signature · 1 line
gitgitgadget
PATCH v6 4/9 sub-process: separate process lifecycle from hashmap management
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
subprocess_start() and subprocess_stop() couple two concerns:
managing a child process (setup, handshake, teardown) and managing a hashmap that indexes running processes by command string. The hashmap suits callers like convert.c where many files may share one filter process looked up by name, but callers that manage process lifetime through their own data structures do not need it.
Extract subprocess_start_command() and subprocess_stop_command() so callers can reuse the child process setup and handshake machinery without maintaining a hashmap. subprocess_start() and subprocess_stop() become thin wrappers that add hashmap operations on top.
--- sub-process.c | 28 +++++++++++++++++++++++----- sub-process.h | 9 ++++++++- 2 files changed, 31 insertions(+), 6 deletions(-)
sub-process.c +23 -5
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)return (len <0 ) ? len :0 ; } -void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry) +void subprocess_stop_command (struct subprocess_entry *entry) {if (!entry)return ;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry) entry->process.clean_on_exit =0 ;kill (entry->process.pid,SIGTERM );finish_command (&entry->process); +} +void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry) +{ +if (!entry) +return ; + +subprocess_stop_command (entry);hashmap_remove (hashmap, &entry->ent,NULL ); }
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)finish_command (process); } -int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd, +int subprocess_start_command (struct subprocess_entry *entry,const char *cmd,subprocess_start_fn startfn) {int err;
@@ -96,15 +103,26 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, coreturn err; } -hashmap_entry_init (&entry->ent,strhash (cmd)); - err =startfn (entry);if (err) {error ("initialization for subprocess '%s' failed" , cmd); -subprocess_stop (hashmap, entry); +subprocess_stop_command (entry);return err; } +return 0 ; +} + +int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd, +subprocess_start_fn startfn) +{ +int err; + + err =subprocess_start_command (entry, cmd, startfn); +if (err) +return err; + +hashmap_entry_init (&entry->ent,strhash (cmd));hashmap_add (hashmap, &entry->ent);return 0 ; }
sub-process.h +8 -1
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,*/ typedef int (*subprocess_start_fn )(struct subprocess_entry *entry); -/* Start a subprocess and add it to the subprocess hashmap. */ +/* Start a subprocess and run the startfn (typically handshake). */ +int subprocess_start_command (struct subprocess_entry *entry,const char *cmd, +subprocess_start_fn startfn); + +/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */ int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd,subprocess_start_fn startfn); +/* Kill a subprocess. */ +void subprocess_stop_command (struct subprocess_entry *entry); +/* Kill a subprocess and remove it from the subprocess hashmap. */ void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry);
Signature · 1 line
gitgitgadget
PATCH v6 5/9 diff: add long-running diff process via diff.<driver>.process
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Add support for external diff processes that communicate via the long-running process protocol (pkt-line over stdin/stdout).
A diff process is configured per userdiff driver:
[diff "cdiff"]
process = /path/to/diff-tool
The tool provides custom line-matching: it receives file pairs and returns hunks that reference line numbers in the content. When textconv is also configured, the tool receives the textconv-transformed content. The tool controls which lines are marked as changed while the display shows the file content. Patch output features (word diff, function context, color) work normally. A new "Which features consult the diff process" documentation section lays out which features use the tool's hunks, which compute independently, and why; the summary formats such as --stat still use the builtin diff for now.
The handshake negotiates version=1 and capability=hunks. Per-file requests send command=hunks, pathname, the old and new blob object names as old-oid/new-oid, and both file contents as packetized data. The tool responds with hunk lines and a status packet (success, error, or abort). On error, Git warns and falls back to the builtin diff algorithm for that file. On abort, Git silently falls back for the current file and stops sending further requests to the tool for the remainder of the session.
old-oid/new-oid name the two blobs so a tool can cache its analysis keyed on the pair. A side's oid is sent only when the content the tool receives is that raw blob: it is omitted under textconv, which rewrites the bytes, and for a working-tree side with no stored object, so an oid that is sent always names the bytes the tool receives. This is where the process protocol diverges from diff.<driver>.command, which never composes with textconv (the command replaces the whole diff and always gets the raw blob). Tools ignore unknown request keys, so old tools skip them.
When the tool returns no hunks followed by status=success, Git treats the file as having no changes and produces no diff output. This also means --exit-code reports no changes for that file.
The subprocess is stored on the userdiff_driver struct and launched on first use. If the process fails to start, the handshake fails, or a communication error occurs mid-stream, the failure is cached on the driver to avoid retrying and re-warning on every subsequent file.
Git falls back to the builtin diff (rather than consulting the tool) when an option the tool cannot honor is in effect: the whitespace-ignoring flags, --ignore-blank-lines, -I<regex>, and --anchored. The bypass keys off the effective diff parameters (xpp) rather than diffopt, so a later caller whose flags live elsewhere is covered uniformly. A change that only adds or removes the trailing newline is likewise not expressible as hunks, so it too uses the builtin diff. The hunk parser ignores unknown trailing fields on a hunk line for response forward-compatibility.
Hunk accumulation is bounded by the combined byte count of the two files, so a misbehaving tool that floods hunk lines cannot grow memory without bound before validation runs.
diff_process_fill_hunks() is the sole public entry point. It handles driver lookup, flag checks, subprocess management, and error reporting, returning an enum that lets callers distinguish "hunks populated" from "files equivalent" from "not applicable" from "tool failure."
--- Documentation/config/diff.adoc | 5 + Documentation/gitattributes.adoc | 249 +++++++++++ Makefile | 2 + diff-process.c | 491 ++++++++++++++++++++ diff-process.h | 49 ++ diff.c | 21 + diff.h | 3 + meson.build | 1 + t/helper/meson.build | 1 + t/helper/test-diff-process-backend.c | 381 ++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/meson.build | 1 + t/t4080-diff-process.sh | 645 +++++++++++++++++++++++++++ userdiff.h | 3 + 15 files changed, 1854 insertions(+) create mode 100644 diff-process.c create mode 100644 diff-process.h create mode 100644 t/helper/test-diff-process-backend.c create mode 100755 t/t4080-diff-process.sh
Documentation/config/diff.adoc +5 -0
@@ -218,6 +218,11 @@ endif::git-diff[] Set this option to `true` to make the diff driver cache the text conversion outputs. See linkgit:gitattributes[5] for details. +`diff.<driver>.process`:: + The command to run as a long-running diff process that + provides hunks to Git's diff pipeline. + See linkgit:gitattributes[5] for details. + `diff.indentHeuristic`:: Set this option to `false` to disable the default heuristics that shift diff hunk boundaries to make patches easier to read.
Documentation/gitattributes.adoc +249 -0
255 lines not on this page · raw message
Makefile +2 -0
@@ -811,6 +811,7 @@ TEST_BUILTINS_OBJS += test-csprng.o TEST_BUILTINS_OBJS += test-date.o TEST_BUILTINS_OBJS += test-delete-gpgsig.o TEST_BUILTINS_OBJS += test-delta.o +TEST_BUILTINS_OBJS += test-diff-process-backend.o TEST_BUILTINS_OBJS += test-dir-iterator.o TEST_BUILTINS_OBJS += test-drop-caches.o TEST_BUILTINS_OBJS += test-dump-cache-tree.o
@@ -1140,6 +1141,7 @@ LIB_OBJS += diff-delta.o LIB_OBJS += diff-merges.o LIB_OBJS += diff-lib.o LIB_OBJS += diff-no-index.o +LIB_OBJS += diff-process.o LIB_OBJS += diff.o LIB_OBJS += diffcore-break.o LIB_OBJS += diffcore-delta.o
diff-process.c new file +491 -0
491 lines not on this page · raw message
diff-process.h new file +49 -0
49 lines not on this page · raw message
diff.c +21 -0
@@ -25,6 +25,7 @@#include "utf8.h" #include "odb.h" #include "userdiff.h" +#include "diff-process.h" #include "submodule.h" #include "hashmap.h" #include "mem-pool.h"
@@ -4164,6 +4165,25 @@ static void builtin_diff(const char *name_a, xpp.ignore_regex_nr = o->ignore_regex_nr; xpp.anchors = o->anchors; xpp.anchors_nr = o->anchors_nr; + +/* +* Send the blob oids only for a side whose content is the +* raw blob: textconv rewrites the bytes, and a working-tree +* side has no stored oid, so pass NULL there rather than an +* oid that would not name what the tool receives. +*/ +if (diff_process_fill_hunks (o, name_a, &mf1, &mf2, + (textconv_one || !one->oid_valid) ?NULL : &one->oid, + (textconv_two || !two->oid_valid) ?NULL : &two->oid, + &xpp) + ==DIFF_PROCESS_EQUIVALENT ) { +if (textconv_one) +free (mf1.ptr); +if (textconv_two) +free (mf2.ptr); + goto free_ab_and_return; + } + xecfg.ctxlen = o->context; xecfg.interhunkctxlen = o->interhunkcontext; xecfg.flags =XDL_EMIT_FUNCNAMES ;
@@ -4208,6 +4228,7 @@ static void builtin_diff(const char *name_a, }else if (xdi_diff_outf (&mf1, &mf2,NULL , fn_out_consume, &ecbdata, &xpp, &xecfg))die ("unable to generate diff for %s" , one->path); +free (xpp.external_hunks);if (o->word_diff)free_diff_words_data (&ecbdata);if (textconv_one)
diff.h +3 -0
@@ -173,6 +173,9 @@ struct diff_flags {*/ unsigned allow_external; +/** Disables diff.<driver>.process. */ +unsigned no_diff_process; +/** * For communication between the calling program and the options parser; * tell the calling program to signal the presence of difference using
meson.build +1 -0
@@ -328,6 +328,7 @@ libgit_sources = [ 'diff-merges.c', 'diff-lib.c', 'diff-no-index.c', + 'diff-process.c', 'diff.c', 'diffcore-break.c', 'diffcore-delta.c',
t/helper/meson.build +1 -0
@@ -12,6 +12,7 @@ test_tool_sources = [ 'test-date.c', 'test-delete-gpgsig.c', 'test-delta.c', + 'test-diff-process-backend.c', 'test-dir-iterator.c', 'test-drop-caches.c', 'test-dump-cache-tree.c',
t/helper/test-diff-process-backend.c new file +381 -0
381 lines not on this page · raw message
t/helper/test-tool.c +1 -0
@@ -22,6 +22,7 @@ static struct test_cmd cmds[] = { { "date", cmd__date }, { "delete-gpgsig", cmd__delete_gpgsig }, { "delta", cmd__delta }, + { "diff-process-backend", cmd__diff_process_backend }, { "dir-iterator", cmd__dir_iterator }, { "drop-caches", cmd__drop_caches }, { "dump-cache-tree", cmd__dump_cache_tree },
t/helper/test-tool.h +1 -0
@@ -15,6 +15,7 @@ int cmd__csprng(int argc, const char **argv);int cmd__date (int argc,const char **argv);int cmd__delta (int argc,const char **argv);int cmd__delete_gpgsig (int argc,const char **argv); +int cmd__diff_process_backend (int argc,const char **argv);int cmd__dir_iterator (int argc,const char **argv);int cmd__drop_caches (int argc,const char **argv);int cmd__dump_cache_tree (int argc,const char **argv);
t/meson.build +1 -0
@@ -512,6 +512,7 @@ integration_tests = [ 't4072-diff-max-depth.sh', 't4073-diff-stat-name-width.sh', 't4074-diff-shifted-matched-group.sh', + 't4080-diff-process.sh', 't4100-apply-stat.sh', 't4101-apply-nonl.sh', 't4102-apply-rename.sh',
t/t4080-diff-process.sh new file +645 -0
645 lines not on this page · raw message
userdiff.h +3 -0
@@ -3,6 +3,7 @@#include "notes-cache.h" +struct diff_subprocess ;struct index_state ;struct repository ;
@@ -33,6 +34,8 @@ struct userdiff_driver {int textconv_want_cache;const char *process;char *process_owned; +struct diff_subprocess *diff_subprocess; + unsigned diff_process_failed : 1; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,
Signature · 1 line
gitgitgadget
PATCH v6 6/9 diff: bypass diff process with --no-ext-diff and in format-patch
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
Make --no-ext-diff disable diff.<driver>.process in addition to diff.<driver>.command. Although the two mechanisms work differently (command replaces Git's output, process feeds hunks back into the pipeline), both invoke external tools and --no-ext-diff means "no external tools."
Replace the OPT_BOOL for --ext-diff with an OPT_CALLBACK that sets both allow_external and no_diff_process, so a single option controls both. Passing --ext-diff explicitly clears no_diff_process, so a later --ext-diff overrides an earlier --no-ext-diff.
Disable the diff process unconditionally in format-patch so that generated patches are always based on the builtin diff algorithm and can be applied reliably by recipients who do not have the external tool.
Document that --diff-algorithm also bypasses the diff process, since it forces the builtin algorithm.
--- Documentation/diff-algorithm-option.adoc | 3 +++ Documentation/diff-options.adoc | 4 +++- Documentation/gitattributes.adoc | 6 +++--- builtin/log.c | 7 +++++++ diff.c | 16 ++++++++++++++-- diff.h | 5 ++++- t/t4080-diff-process.sh | 16 ++++++++++++++++ 7 files changed, 50 insertions(+), 7 deletions(-)
Documentation/diff-algorithm-option.adoc +3 -0
@@ -18,3 +18,6 @@ For instance, if you configured the `diff.algorithm` variable to a non-default value and want to use the default one, then you have to use `--diff-algorithm=default` option. ++ +If you explicitly choose a diff algorithm, it also bypasses +`diff.<driver>.process` (see linkgit:gitattributes[5]).
Documentation/diff-options.adoc +3 -1
@@ -825,7 +825,9 @@ endif::git-format-patch[] to use this option with linkgit:git-log[1] and friends. `--no-ext-diff`:: - Disallow external diff drivers. + Disallow external diff helpers, including + `diff.<driver>.command` and `diff.<driver>.process` + (see linkgit:gitattributes[5]). `--textconv`:: `--no-textconv`::
Documentation/gitattributes.adoc +3 -3
@@ -1073,9 +1073,9 @@ display, which is covered above); and combined diffs (`--cc` and merge diffs), whose protocol would have to be extended from a single old/new pair to one comparison per merge parent. -`--diff-algorithm` bypasses the process entirely, for every feature -listed above. The whitespace-ignoring options (`-w`, -`--ignore-space-change`, `--ignore-blank-lines`, and the like), +`--no-ext-diff` and `--diff-algorithm` bypass the process entirely, +for every feature listed above. The whitespace-ignoring options +(`-w`, `--ignore-space-change`, `--ignore-blank-lines`, and the like), `-I<regex>`, and `--anchored` also bypass it for the affected files: the tool is never told about these options, so it could not honor them, and Git falls back to the builtin diff, which does.
builtin/log.c +7 -0
@@ -2217,6 +2217,13 @@ int cmd_format_patch(int argc,if (argc >1 )die (_ ("unrecognized argument: %s" ), argv[1 ]); +/* +* Disable diff.<driver>.process so that patches generated by +* format-patch are always based on the builtin diff algorithm +* and can be applied reliably. +*/ + rev.diffopt.flags.no_diff_process =1 ; +if (rev.diffopt.output_format &DIFF_FORMAT_NAME )die (_ ("--name-only does not make sense" ));if (rev.diffopt.output_format &DIFF_FORMAT_NAME_STATUS )
diff.c +14 -2
@@ -6071,6 +6071,17 @@ static int diff_opt_submodule(const struct option *opt,return 0 ; } +static int diff_opt_ext_diff (const struct option *opt, +const char *arg,int unset) +{ +struct diff_options *options = opt->value; + +BUG_ON_OPT_ARG (arg); + options->flags.allow_external = !unset; + options->flags.no_diff_process = unset; +return 0 ; +} +static int diff_opt_textconv (const struct option *opt,const char *arg,int unset) {
@@ -6401,8 +6412,9 @@ struct option *add_diff_options(const struct option *opts, N_("exit with 1 if there were differences, 0 otherwise")),OPT_BOOL (0 ,"quiet" , &options->flags.quick,N_ ("disable all output of the program" )), -OPT_BOOL (0 ,"ext-diff" , &options->flags.allow_external, -N_ ("allow an external diff helper to be executed" )), +OPT_CALLBACK_F (0 ,"ext-diff" , options,NULL , +N_ ("allow an external diff helper to be executed" ), +PARSE_OPT_NOARG , diff_opt_ext_diff),OPT_CALLBACK_F (0 ,"textconv" , options,NULL ,N_ ("run external text conversion filters when comparing binary files" ), PARSE_OPT_NOARG, diff_opt_textconv),
diff.h +4 -1
@@ -173,7 +173,10 @@ struct diff_flags {*/ unsigned allow_external; -/** Disables diff.<driver>.process. */ +/** +* Disables diff.<driver>.process. Set by --no-ext-diff and by +* format-patch. +*/ unsigned no_diff_process;/**
t/t4080-diff-process.sh +16 -0
@@ -398,6 +398,22 @@ test_expect_success 'diff process bypassed by --diff-algorithm' ' test_path_is_missing backend.log ' +test_expect_success 'diff process bypassed by --no-ext-diff' ' + test_when_finished "rm -f backend.log" && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + diff --no-ext-diff worddiff.c >actual && + test_grep "return 999" actual && + test_path_is_missing backend.log +' + +test_expect_success 'diff process not used by format-patch' ' + test_when_finished "rm -f backend.log" && + git -c diff.cdiff.process="$BACKEND --log=backend.log" \ + format-patch -1 --stdout -- logtest.c >actual && + test_grep "return 2" actual && + test_path_is_missing backend.log +' + test_expect_success 'diff process bypassed under whitespace-ignoring flags' ' test_when_finished "rm -f backend.log" && printf "a\nb\nc\n" >wsbypass.c &&
Signature · 1 line
gitgitgadget
PATCH v6 7/9 blame: consult diff process for no-hunk detection
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
When a diff process is configured via diff.<driver>.process, consult it during blame's per-commit diffing. If the process returns no hunks for a commit's changes to a file, treat the commit as having no changes, causing blame to attribute lines to earlier commits.
Introduce xdi_diff_process(), a process-aware xdi_diff() that consults the process, runs xdiff on the tool's hunks or on the builtin algorithm when it does not apply, frees the hunks, and reports DIFF_PROCESS_EQUIVALENT (without running xdiff) so the caller can drop or skip the change. It is the shared consult-then-diff path for consumers that work on raw hunks: blame's pass_blame_to_parent() uses it here, and git log -L reuses it later. builtin_diff() keeps consulting the process directly, because it tests for equivalence early, before its funcname-pattern and word-diff setup, so a reformat-only file short-circuits without that work.
Blame's -w option is not communicated to the process and it could not honor it, so blame must fall back to the builtin diff there. Because blame keeps its whitespace flags in sb->xdl_opts rather than diffopt, the process bypass keys off xpp (the flags the diff actually runs with), which covers blame without a guard of its own.
The subprocess is long-running (one startup cost amortized across the blame traversal), but each commit in the file's history incurs a round-trip to the tool.
--- blame.c | 24 +++++++- diff-process.c | 38 ++++++++++++ diff-process.h | 26 +++++++++ t/t4080-diff-process.sh | 126 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 213 insertions(+), 1 deletion(-)
blame.c +23 -1
@@ -19,6 +19,8 @@#include "tag.h" #include "trace2.h" #include "blame.h" +#include "diff-process.h" +#include "xdiff-interface.h" #include "alloc.h" #include "commit-slab.h" #include "bloom.h"
@@ -1943,6 +1945,9 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb, struct blame_origin *parent, int ignore_diffs) {mmfile_t file_p, file_o; +xpparam_t xpp = {0 }; +xdemitconf_t xecfg = {0 }; +xdemitcb_t ecb = {NULL };struct blame_chunk_cb_data d;struct blame_entry *newdest =NULL ;
@@ -1961,7 +1966,24 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb, &sb->num_read_blob, ignore_diffs); sb->num_get_patch++; -if (diff_hunks (&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts)) + xpp.flags = sb->xdl_opts; + xecfg.hunk_func = blame_chunk_cb; + ecb.priv = &d; +/* +* Consult the diff process, then attribute the resulting chunks +* via blame_chunk_cb. It bypasses the process for the whitespace- +* ignoring options it cannot honor (they live in xpp.flags, which +* the consultation checks), and when the process reports the blobs +* equivalent it runs no diff, so blame passes this commit and looks +* past it. Look up the driver by the parent (old) path, as +* builtin_diff() does with name_a, so a renamed file resolves to the +* same driver across diff, blame, and line-log. Pass no +* old-oid/new-oid: blame diffs each blob pair once, so the tool gains +* nothing from a per-invocation cache key. +*/ +if (xdi_diff_process (&sb->revs->diffopt, parent->path, + &file_p, &file_o,NULL ,NULL , &xpp, &xecfg, &ecb) + ==DIFF_PROCESS_ERROR )die ("unable to generate diff (%s -> %s)" ,oid_to_hex (&parent->commit->object.oid),oid_to_hex (&target->commit->object.oid));
diff-process.c +38 -0
@@ -37,6 +37,7 @@#include "sub-process.h" #include "pkt-line.h" #include "strbuf.h" +#include "xdiff-interface.h" #include "xdiff/xdiff.h" #define CAP_HUNKS (1u << 0)
@@ -489,3 +490,40 @@ enum diff_process_result diff_process_fill_hunks( }return DIFF_PROCESS_SKIP ; } + +enum diff_process_result xdi_diff_process ( +struct diff_options *diffopt, +const char *path, +mmfile_t *file_a, +mmfile_t *file_b, +const struct object_id *oid_a, +const struct object_id *oid_b, +xpparam_t *xpp, +xdemitconf_t *xecfg, +xdemitcb_t *ecb) +{ +enum diff_process_result res; + +/* +* Consult the diff process, then run xdiff either constrained to +* the tool's hunks or, when the process does not apply, computing +* the diff itself as a fallback. EQUIVALENT short-circuits: the +* caller decides what "no change" means for it (drop the commit, +* skip the file, ...), so xdiff is not run. +* +* A SKIP/ERROR from the process just selects the builtin path +* (its warning, if any, was already emitted), so the result then +* reflects whether xdiff itself succeeded, not the process. +*/ + res =diff_process_fill_hunks (diffopt, path, file_a, file_b, + oid_a, oid_b, xpp); +if (res ==DIFF_PROCESS_EQUIVALENT ) +return res; + + res =xdi_diff (file_a, file_b, xpp, xecfg, ecb) <0 + ?DIFF_PROCESS_ERROR :DIFF_PROCESS_OK ; + +FREE_AND_NULL (xpp->external_hunks); + xpp->external_hunks_nr =0 ; +return res; +}
diff-process.h +26 -0
@@ -46,4 +46,30 @@ enum diff_process_result diff_process_fill_hunks(const struct object_id *oid_b, xpparam_t *xpp); +/* +* Process-aware xdi_diff(): consult the diff process for 'path', then +* run xdiff either constrained to the tool's hunks or computing the +* diff itself when the process does not apply or fails. Frees any +* hunks it obtained before returning. +* +* Returns DIFF_PROCESS_EQUIVALENT (without running xdiff) when the tool +* reports the blobs equal, so the caller can drop or skip the change; +* DIFF_PROCESS_OK when xdiff ran (on tool hunks or builtin); and +* DIFF_PROCESS_ERROR if xdiff itself errored. +* +* The caller fills xpp (flags, ignore_regex, anchors) and xecfg/ecb as +* for a direct xdi_diff() call. oid_a/oid_b are forwarded to +* diff_process_fill_hunks() (see there). +*/ +enum diff_process_result xdi_diff_process ( +struct diff_options *diffopt, +const char *path, +mmfile_t *file_a, +mmfile_t *file_b, +const struct object_id *oid_a, +const struct object_id *oid_b, +xpparam_t *xpp, +xdemitconf_t *xecfg, +xdemitcb_t *ecb); +#endif /* DIFF_PROCESS_H */
t/t4080-diff-process.sh +126 -0
130 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v6 8/9 diff: consult diff process for --stat counts
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
builtin_diff() already consults a configured diff.<driver>.process: a file the tool reports as equivalent emits no patch, and otherwise the tool's hunks drive the output. builtin_diffstat() ran its own xdiff and ignored the process, so "git diff --stat" still counted a byte-level change for a file that "git diff" showed as unchanged.
Consult diff_process_fill_hunks() before the stat xdiff, as builtin_diff() does. On DIFF_PROCESS_EQUIVALENT, skip the xdiff so the file keeps its zero inserted and deleted counts and the existing "nothing changed" pruning drops it, matching the empty patch. Otherwise the tool's hunks, or the builtin fallback, feed the counts through the shared xpparam_t.
Under -L, route the surviving hunks through the same line-range filter builtin_diffstat() already uses for a tracked range, so a process-provided diff is scoped to that range: "git log -L<range> --stat" counts the tool's changed lines within the range rather than the builtin line diff's.
Like the builtin summary path, builtin_diffstat() does not apply textconv, so the process is consulted on the raw blob content here, unlike builtin_diff() which sends textconv'd content. This keeps "git diff --stat" counting raw lines as it does today; the asymmetry between patch output and summary counts under textconv predates this change. Because the content is the raw blob, the stat path sends the blob object names to the tool (old-oid/new-oid) for any stored blob, where the patch path omits the oid under textconv.
Move the summary formats out of the "not yet wired" group of the "Which features consult the diff process" documentation and into the list of features that use the tool's hunks, noting the raw, non-textconv content they receive. Document that the line-counting --dirstat=lines follows these counts while the default --dirstat does not, and that summary formats and blame (only under --textconv) differ from patch output in whether they textconv the content the tool sees.
Add tests covering counts from the tool's hunks (--numstat, --shortstat), an equivalent file producing no stat line, --stat --exit-code, the raw non-textconv content the tool receives, a multi-file mix of equivalent and changed files, a mode-only change, and a range-scoped --stat under "git log -L" that reflects the tool's hunks.
--- Documentation/gitattributes.adoc | 33 ++++-- diff.c | 47 ++++++--- t/t4080-diff-process.sh | 176 +++++++++++++++++++++++++++++++ 3 files changed, 234 insertions(+), 22 deletions(-)
Documentation/gitattributes.adoc +23 -10
@@ -874,7 +874,10 @@ a flush packet, followed by the old and new file content as packetized data, each terminated with a flush packet. The pathname is relative to the repository root. When `diff.<name>.textconv` is also set, the tool receives the textconv-transformed content rather than the -raw blob. Git does not send binary files to the diff process. +raw blob, matching what the consuming feature itself diffs: patch +output is textconv'd, the summary formats (noted below) are not, and +`git blame` applies textconv only under `--textconv`. Git does not +send binary files to the diff process. ----------------------- packet: git> command=hunks
@@ -960,8 +963,8 @@ still slide or regroup those changes against matching context for display, exactly as it compacts its own diffs, so the tool controls which lines are reported as changed, not the precise hunk boundaries. Patch output features (word diff, function context, color) work -normally. Summary formats such as `--stat` still compute their counts -with the builtin diff for now; see "Which features consult the diff +normally, as do summary formats like `--stat`. Not every feature +consults the process, though; see "Which features consult the diff process" below for the full picture and the reasoning behind it. If no hunk lines precede the flush, followed by "success", Git
@@ -1040,6 +1043,17 @@ of the builtin algorithm: hunks without any further negotiation. - `git blame`: a commit whose change the tool reports as equivalent is skipped, and its lines are attributed to an earlier commit. +- `--stat`, `--numstat`, and `--shortstat`: the inserted and deleted + counts come from the tool's hunks, so a file the tool calls + equivalent contributes no stat line, matching the empty patch that + `git diff` produces for it. These summary formats do not apply + textconv (just as the builtin summary path does not), so the tool + is consulted on the raw blob content even when a `textconv` is also + configured for patch output; this mirrors how builtin `--stat` + already counts raw lines rather than the textconv'd view. The + line-counting `--dirstat=lines` uses these same counts; the default + `--dirstat`, which weighs byte changes, is computed on its own and + does not consult the tool. Features that ask a different question do not consult the process, by design:
@@ -1065,13 +1079,12 @@ design: - `--raw`, `--name-only`, and `--name-status` compare object ids at the tree level and never run a line-level diff at all. -Some features ask "which lines changed" but still use the builtin -algorithm for now, and may consult the process in a later change: the -summary formats (`--stat`, `--numstat`, `--shortstat`); `git log -L`'s -commit selection and parent range propagation (as distinct from its -display, which is covered above); and combined diffs (`--cc` and merge -diffs), whose protocol would have to be extended from a single old/new -pair to one comparison per merge parent. +Two cases ask "which lines changed" but still use the builtin +algorithm, and may consult the process in a later change: `git log +-L`'s commit selection and parent range propagation (as distinct from +its display, which is covered above), and combined diffs (`--cc` and +merge diffs), whose protocol would have to be extended from a single +old/new pair to one comparison per merge parent. `--no-ext-diff` and `--diff-algorithm` bypass the process entirely, for every feature listed above. The whitespace-ignoring options
diff.c +35 -12
55 lines not on this page · raw message
t/t4080-diff-process.sh +176 -0
188 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v6 9/9 line-log: consult diff process for range tracking
In reply to Michael Montalbo via GitGitGadgetFrom: Michael Montalbo <[email protected]>
git log -L tracks line ranges by diffing each commit against its parent in collect_diff(). This pass used the builtin diff while the displayed diff (builtin_diff()) consults a configured diff.<driver>.process, so the two could disagree: a reformat-only commit selected by builtin tracking was then rendered with an empty diff because the tool reported the files equivalent.
Consult the process in collect_diff() too, mirroring the blame integration. When the tool reports the files equivalent, collect no ranges; the tracked range then maps across unchanged and the commit drops out of the log, matching what is displayed. Like the summary formats, the tracking pass diffs raw content, so the tool is consulted on the raw blobs here.
--- Documentation/gitattributes.adoc | 20 ++++++++++--------- line-log.c | 33 ++++++++++++++++++++++++++++---- t/t4080-diff-process.sh | 33 ++++++++++++++++++++++++++++++++ 3 files changed, 73 insertions(+), 13 deletions(-)
Documentation/gitattributes.adoc +11 -9
@@ -1037,12 +1037,16 @@ Features that ask "which lines changed" use the tool's hunks in place of the builtin algorithm: - `git diff` patch output, together with everything layered on it: - word diff, function context (`-W`), `--color-moved`, the `@@` hunk - headers, and the `-L` line-range display. These operate on the - lines the patch step already emitted, so they reflect the tool's - hunks without any further negotiation. + word diff, function context (`-W`), `--color-moved`, and the `@@` + hunk headers. These operate on the lines the patch step already + emitted, so they reflect the tool's hunks without any further + negotiation. - `git blame`: a commit whose change the tool reports as equivalent is skipped, and its lines are attributed to an earlier commit. +- `git log -L`: both the line-range display and the underlying range + tracking consult the tool, so a commit it reports as equivalent is + dropped from the log (its tracked range maps across unchanged) + rather than selected and then shown with an empty diff. - `--stat`, `--numstat`, and `--shortstat`: the inserted and deleted counts come from the tool's hunks, so a file the tool calls equivalent contributes no stat line, matching the empty patch that
@@ -1079,11 +1083,9 @@ design: - `--raw`, `--name-only`, and `--name-status` compare object ids at the tree level and never run a line-level diff at all. -Two cases ask "which lines changed" but still use the builtin -algorithm, and may consult the process in a later change: `git log --L`'s commit selection and parent range propagation (as distinct from -its display, which is covered above), and combined diffs (`--cc` and -merge diffs), whose protocol would have to be extended from a single +Combined diffs (`--cc` and merge diffs) ask "which lines changed" but +still use the builtin algorithm, and may consult the process in a +later change; their protocol would have to be extended from a single old/new pair to one comparison per merge parent. `--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
line-log.c +29 -4
@@ -7,11 +7,11 @@#include "tag.h" #include "tree.h" #include "diff.h" +#include "diff-process.h" #include "commit.h" #include "decorate.h" #include "repository.h" #include "revision.h" -#include "xdiff-interface.h" #include "strbuf.h" #include "line-log.h" #include "setup.h"
@@ -330,12 +330,15 @@ static int collect_diff_cb(long start_a, long count_a,return 0 ; } -static int collect_diff (mmfile_t *parent,mmfile_t *target,struct diff_ranges *out) +static int collect_diff (struct diff_options *diffopt,const char *path, +mmfile_t *parent,mmfile_t *target, +struct diff_ranges *out) {struct collect_diff_cbdata cbdata = {NULL };xpparam_t xpp;xdemitconf_t xecfg;xdemitcb_t ecb; +int ret =0 ;memset (&xpp,0 ,sizeof (xpp));memset (&xecfg,0 ,sizeof (xecfg));
@@ -345,7 +348,23 @@ static int collect_diff(mmfile_t *parent, mmfile_t *target, struct diff_ranges * xecfg.hunk_func = collect_diff_cb;memset (&ecb,0 ,sizeof (ecb)); ecb.priv = &cbdata; -return xdi_diff (parent, target, &xpp, &xecfg, &ecb); + +/* +* Consult the diff process so range tracking agrees with the +* diff that will be shown. When the tool reports the files as +* equivalent we collect no ranges, so the tracked range maps +* across unchanged and the commit drops out of the log, rather +* than being selected here but rendered with an empty diff by +* the process-aware builtin_diff(). Blob oids are not threaded to +* this path yet, so pass NULL and send no old-oid/new-oid (a later +* change can supply the pair, where they would let the tool cache +* across the range-tracking and display passes over the same +* commit). +*/ +if (xdi_diff_process (diffopt, path, parent, target, +NULL ,NULL , &xpp, &xecfg, &ecb) ==DIFF_PROCESS_ERROR ) + ret =-1 ; +return ret; }/*
@@ -927,7 +946,13 @@ static int process_diff_filepair(struct rev_info *rev, }diff_ranges_init (&diff); -if (collect_diff (&file_parent, &file_target, &diff)) +/* +* Select the driver by the old (parent) path, as builtin_diff() does +* with name_a, so a renamed file resolves to the same driver for +* range tracking as for the diff that is shown. +*/ +if (collect_diff (&rev->diffopt, pair->one->path, + &file_parent, &file_target, &diff))die ("unable to generate diff for %s" , pair->one->path);/* NEEDSWORK should apply some heuristics to prevent mismatches */
t/t4080-diff-process.sh +33 -0
@@ -960,4 +960,37 @@ test_expect_success 'blame -w bypasses diff process' ' test_path_is_missing backend.log ' +# +# Line-log (git log -L) range tracking. +# + +test_expect_success 'diff process drops equivalent commit from log -L' ' + test_when_finished "rm -f backend.log" && + cat >linelog.c <<-\EOF && + int tracked(void) { return 1; } + EOF + git add linelog.c && + git commit -m "add linelog.c" && + + cat >linelog.c <<-\EOF && + int tracked(void) { return 2; } + EOF + git commit -am "change tracked line" && + + # Builtin line tracking selects the change commit. + git log --no-ext-diff -L1,1:linelog.c --format="%s" >builtin && + test_grep "change tracked line" builtin && + + # With the tool reporting the change as equivalent, tracking + # drops the commit (the range maps across unchanged) instead of + # selecting it and rendering an empty diff. + git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \ + log -L1,1:linelog.c --format="%s" >actual && + test_grep ! "change tracked line" actual && + # The creating commit still appears, so the change commit was + # selectively dropped rather than the whole log going empty. + test_grep "add linelog.c" actual && + test_grep "command=hunks pathname=linelog.c" backend.log +' + test_done
Signature · 1 line
gitgitgadget
v7 cover diff: add provider interface and initial providers
In reply to Michael Montalbo via GitGitGadgetEvery in-process diff in Git reduces, at one point, to a single
question: given two blobs and the settings the diff runs under, which line ranges changed? The answer is the diff's hunks: for each change, the position and length of the range on the old side and on the new. Each consumer asks in its own shape:
- blame diffs each suspect's blob against its parent's, taking only the coordinates through xdiff's hunk callback;
- the stat formats keep only the added and deleted counts;
- patch output emits from the hunks, with xdiff interleaving context and content around them;
- log -L maps the tracked range across each commit from the coordinates.
In every case the answer is computed the same way: load both blobs and run xdiff. That is the only source, so nothing that already holds the answer, or that would answer differently on purpose, can supply it instead. Sometimes that is what we want, which is why patch-id and format-patch stay on the builtin computation throughout: patch-id needs identical hashes on every machine, and a format-patch must apply for recipients who share none of the sender's configuration. Other times another source would be useful.
This RFC sketches a direction. The unified series shows one interface carrying two example providers and their interaction; it is not shaped to merge as one topic. If the direction holds, the work returns as separate reviewable series (see Roadmap). The two examples are demonstrations, each an RFC on its own: diff.<driver>.process, the RFC cooking as mm/diff-process-hunks, lets a configured external process answer with its own notion of which lines changed, and the diff-hunks store, new in this thread, remembers what xdiff computed and serves it back. One is authoritative and external, one a cache and in-process.
Three pieces:
- A hunk provider interface (diff-provider.h) is the point of the series. A provider is an alternate source for the answer: asked with the pair's object ids and the diff settings, before any blob is loaded, it may supply the hunks in place of the builtin computation. A miss falls through to that computation, and every answer passes one shared validity check first. The providers form a chain the repository owns, built on first consultation and released in repo_clear(), so provider state such as a running process never outlives its repository. Chain order is the authority, and the terminal provider is the builtin computation itself, so the interface never exists without an implementor: patch 02 ships it answering every request the way the consumers did before. A consumer states its request in one struct and reads one set of outcomes (answered, unanswered, or failed); it never names a provider, and a provider added later maps onto those outcomes inside the interface, so consumer code is written once. Because every diff now walks the chain even with no store or process configured, the default path was measured against the pre-series base and runs within noise (a 5000-commit log --stat and a long-history blame, ratio 1.00 either way).
- The diff-hunks store shows the non-authoritative side: an in-process cache at $GIT_DIR/objects/info/diff-hunks that may only reproduce the builtin diff, so serving from it never changes a command's output. It is read by default and written only when a repository owner opts in, warming it as a side effect of diff work the command already does:
GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null
A warmed store then serves the stat formats and blame from stored coordinates instead of a fresh diff: on git.git a 5000-commit log --stat runs about 1.9x faster, and blame reads the same entries opportunistically (full numbers in [1]). Its format and keying, what it may not serve, and how it handles corruption and staleness are in git-diff-hunks(1), gitformat-diff-hunks(5), and [2]. The interface point is small: a cache drops in as the provider that stands aside wherever an authoritative one answers.
- diff.<driver>.process shows the authoritative side: an external process, configured per driver, whose answers may deliberately differ from the builtin diff and outrank the store. Git asks it for a pair by object names alone, so it answers before any blob is read, which suits a cache or a process that fetches the blobs itself. Consulting is opt-in per command, following the allow_textconv precedent, and a pair the process cannot answer falls back to the builtin diff. The protocol, the per-command gate, how failures are handled, and the versioning that lets it grow are in gitattributes(5) and footnotes [3] and [4]. The interface point, again, is small: an external, authoritative provider joins the same chain ahead of the cache, and neither consumer learns it is there. A later content-carrying request would extend it to the pairs and consumers this identity-only form leaves on the builtin diff.
The series stops at the coordinates. A consumer that needs the changed text, such as patch output, would have only its hunk selection replaced, with xdiff still emitting content from the blobs; that machinery is the content enrichment sketched in the Roadmap. Establishing the framework on coordinates first keeps this series one design: the question, the interface, and two providers answering by identity.
Shape of the series:
01 documentation: how external diff drivers relate to the
features layered on the diff
02 the provider interface: the request and outcome types, the
emit entry point, the shared validity check, and the repository-owned chain with its terminal builtin provider
03 the store: on-disk format, library, and the diff-hunks command
04 recording: the stat walk computes, sums, and records
trim-stable pairs (writes gated off by default)
05 reading: the consult entry point and the store's registration
as a provider; the request gains the object ids and diff options
06 blame reading through the interface's emit path
07-09 process preparation: sub-process lifecycle split, a gentle
status read for an optional process, and the diff.<driver>.process config
10 the process provider, oid-only, at the head of the chain, with
the per-command gate; the request gains the path
Roadmap:
This RFC asks whether the direction is right, not for these ten patches to merge as one topic. If it holds, the work returns in reviewable pieces:
- the interface and the store (patches 01 through 06): a cache with measured numbers and no external-process machinery
- the process provider (patches 07 through 10) on the same interface
- the content enrichment (the content-carrying request, patch output and log -L consulting, and the xdiff machinery that feeds a provider's hunks into emission) once the identity-keyed framework settles.
Several design questions are left for those series.
mm/diff-process-hunks in seen would be dropped in favor of this thread and its split.
The series applies on the line-log topic (mm/line-log-limited-ops) rebased onto current master. The topic rewrites the same builtin_diffstat() region this series touches, and current master includes 061a68e443 (sub-process: use gentle handshake to avoid die() on startup failure), which this topic leans on: a process that dies during the handshake degrades to the builtin diff like every other failure. A trial merge against seen shows no interaction with other topics beyond the mm/diff-process-hunks replacement above.
The base (line-log topic on current master) and the full series are available at:
git fetch https://github.com/mmontalbo/git mm/line-log-stat-formats-followup git fetch https://github.com/mmontalbo/git mm/hunk-providers-oid-first
Changes since v6:
This is a restructuring, not an incremental reroll, so a range-diff against v6 is unreadable; the map of what changed:
- The series now leads with the hunk provider interface and brings the diff-hunks store in as its in-process implementation (patches 02 through 06, new to this thread). It keeps only the identity-keyed half of the external diff process protocol from mm/diff-process-hunks.
- v6's gitattributes documentation, sub-process split, and userdiff config return close to their v6 form as patches 01, 07, and 09. Patch 08 is new: a gentle status read so a protocol error in an optional process degrades to the builtin diff instead of dying.
- v6's protocol patch returns as patch 10, reduced to the oid-only request, consulting through the interface, and carrying a per-command gate (v6's bypass patch folds into it).
- v6's blame and stat consults return as identity-keyed consults (patches 05, 06, and 10); their content legs, along with v6's xdiff external-hunks machinery, content-carrying request, and line-log consult, are withheld for the content enrichment.
Footnotes:
[1] Store numbers, measured with hyperfine against the same build with
core.diffHunks=false. The warm is a full cold build of the store;
the blame speedup is file-dependent (see the coverage limitation):
git.git (82,912 commits, --all)
warm log --all --stat 20.9 s store 28 MB, verify 39 ms
log --stat -5000 1.91x (1.38 s -> 0.72 s)
blame diff.c 1.26x (509 ms -> 403 ms)
blame hit rate 54% (896 of 1653 pairs)
linux (1,445,548 commits, --all)
warm log --all --stat 714 s store 298 MB, verify 415 ms
log --stat -5000 1.43x (2.29 s -> 1.60 s)
blame kernel/sched/core.c 1.43x (1.59 s -> 1.11 s)
blame hit rate 74% (2414 of 3263 pairs)[2] The store is its own file because nothing existing is addressed by a blob pair: notes attach to single objects, commit-graph chunks to commits. One entry per pair, keyed by (old blob, new blob, xdl_opts) and recorded only when the pair's trimmed and untrimmed diffs agree, serves blame at zero context and the stat formats at any -U (divergent pairs are 0.4-0.5% of a warm and always compute). The writer fsyncs and commits atomically, and a reader bounds-checks every record and treats an unparsable file as absent; the trailing checksum is checked by git diff-hunks verify, not on every read, the same read-time trust the commit-graph and multi-pack-index take. There is deliberately no fsck integration, expiry, or background
maintenance: the store is derivable at any time, so the recovery path is git diff-hunks clear and a re-warm. New commits make it incomplete, not wrong; a later warm seeds from the file and pays only for what is new.
[3] Consulting the process is allowed per command, like textconv: git diff, git log and git show, and git blame consult it; the plumbing diff commands do not unless --ext-diff or --diff-process is given, and the interactive-patch machinery, format-patch, and range-diff stay builtin. Options the process is never told about select no process, and an object id is sent only when it names the exact bytes diffed (a pair under an active object replacement is not sent). The command comes from local configuration, as with filter.<name> .process: attributes select only a driver name, so cloning cannot cause a process to run. gitattributes(5) has the full gate.
[4] The protocol is versioned and capability-negotiated, and extends without breaking deployed processes: a process ignores request keys it does not know, Git ignores trailing tokens on a hunk line so fields can be appended, and new request forms arrive as capabilities a process may decline. Announcing a capability Git did not request aborts the command, the filter protocol's handshake rule. The content-carrying request is the natural first extension; markers for formatting-only changes and function or token boundaries are candidates beyond it.
Michael Montalbo (10):
gitattributes: document how external diff drivers relate to diff
features
diff: introduce a hunk provider interface
diff-hunks: add the store format, library, and command
diff: record precomputed hunks during stat output
diff: read precomputed hunks for stat output
blame: read precomputed hunks
sub-process: separate process lifecycle from hashmap management
sub-process: add a gentle status read
userdiff: add diff.<driver>.process config
diff: consult oid-only hunk providers via diff.<driver>.process
.gitignore | 1 + Documentation/Makefile | 1 + Documentation/config.adoc | 2 + Documentation/config/core.adoc | 10 +- Documentation/config/diff-hunks.adoc | 8 + Documentation/config/diff.adoc | 6 + Documentation/diff-algorithm-option.adoc | 3 + Documentation/diff-options.adoc | 15 +- Documentation/git-diff-hunks.adoc | 146 +++ Documentation/gitattributes.adoc | 171 ++++ Documentation/gitformat-diff-hunks.adoc | 129 +++ Documentation/meson.build | 2 + Makefile | 5 + blame.c | 81 +- builtin.h | 1 + builtin/blame.c | 9 +- builtin/diff-hunks.c | 53 ++ builtin/diff-tree.c | 3 + builtin/diff.c | 11 + builtin/log.c | 19 + chunk-format.c | 62 +- chunk-format.h | 14 + command-list.txt | 2 + diff-hunks.c | 1034 ++++++++++++++++++++++ diff-hunks.h | 141 +++ diff-process.c | 669 ++++++++++++++ diff-provider-internal.h | 130 +++ diff-provider.c | 190 ++++ diff-provider.h | 159 ++++ diff.c | 294 +++++- diff.h | 47 + environment.c | 1 + git.c | 1 + meson.build | 4 + odb.c | 2 + odb.h | 4 + range-diff.c | 6 + repo-settings.c | 1 + repo-settings.h | 1 + repository.c | 3 + repository.h | 8 + sub-process.c | 52 +- sub-process.h | 19 +- t/helper/meson.build | 1 + t/helper/test-diff-process-backend.c | 349 ++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/meson.build | 3 + t/perf/p4218-diff-hunks.sh | 48 + t/t4080-diff-process.sh | 593 +++++++++++++ t/t4220-diff-hunks.sh | 819 +++++++++++++++++ t/t4220/README | 55 ++ t/t4220/trim-divergent-new | 319 +++++++ t/t4220/trim-divergent-old | 316 +++++++ userdiff.c | 7 + userdiff.h | 2 + write-or-die.h | 7 +- xdiff-interface.h | 12 + 58 files changed, 5989 insertions(+), 64 deletions(-) create mode 100644 Documentation/config/diff-hunks.adoc create mode 100644 Documentation/git-diff-hunks.adoc create mode 100644 Documentation/gitformat-diff-hunks.adoc create mode 100644 builtin/diff-hunks.c create mode 100644 diff-hunks.c create mode 100644 diff-hunks.h create mode 100644 diff-process.c create mode 100644 diff-provider-internal.h create mode 100644 diff-provider.c create mode 100644 diff-provider.h create mode 100644 t/helper/test-diff-process-backend.c create mode 100755 t/perf/p4218-diff-hunks.sh create mode 100755 t/t4080-diff-process.sh create mode 100755 t/t4220-diff-hunks.sh create mode 100644 t/t4220/README create mode 100644 t/t4220/trim-divergent-new create mode 100644 t/t4220/trim-divergent-old
base-commit: 9a0c4701dcd5725c4184599322b52933ff5005ca
prerequisite-patch-id: 6270dea79c9f06530737cefa3e1a0a39a1be7877
prerequisite-patch-id: 46fcc16a7a2ed760a1134d2a92c87699f3ec7bdb
prerequisite-patch-id: c1e3da243003d060e429bc2196ae02b3453f01f9
prerequisite-patch-id: 4ad4e273494d4e8503706c21bfdc90a5d7ce116a
prerequisite-patch-id: f7fa1367756daafa83f4f030a5c7b6dc3dbb70d7
prerequisite-patch-id: 829e76c9fec655a07f9383086a35bff3290b1c74
prerequisite-patch-id: 5c5a0d61ae9b6d628d05f1eb5df046758f3111a8
Signature · 1 line
2.54.0
RFC PATCH v7 1/10 gitattributes: document how external diff drivers relate to diff features
The "Defining an external diff driver" section explains how to configure diff.<driver>.command but not how the driver relates to the rest of Git's diff machinery. In particular, the command only replaces the textual patch: word diff, function context, color, and the like cannot apply to its output, while the summary formats, blame, and git log -L do not run it at all and keep using the builtin diff.
Spell this out so the scope of an external diff driver is clear.
--- Documentation/gitattributes.adoc | 11 +++++++++++ 1 file changed, 11 insertions(+)
Documentation/gitattributes.adoc +11 -0
@@ -784,6 +784,17 @@ with the above configuration, i.e. `j-c-diff`, with 7 parameters, just like `GIT_EXTERNAL_DIFF` program is called. See linkgit:git[1] for details. +An external diff driver replaces the patch Git would otherwise +produce for the path: Git runs the command and shows its output in +place of its own. Output features that post-process Git's diff do +not apply to the driver's output; word diff, function context (`-W`), +`--color-moved`, and coloring all act on Git's builtin diff, not the +driver's output. +The driver is consulted only when Git generates a textual patch. The +summary formats (`--stat`, `--numstat`, `--shortstat`, and +`--dirstat`), `git blame`, and `git log -L` do not run it and +continue to use Git's builtin diff. + If the program is able to ignore certain changes (similar to `git diff --ignore-space-change`), then also set the option `trustExitCode` to true. It is then expected to return exit code 1 if
Signature · 1 line
2.54.0
RFC PATCH v7 2/10 diff: introduce a hunk provider interface
In reply to Michael MontalboTo learn which line ranges changed between two blobs, every consumer in the diff machinery loads both blobs and runs xdiff. There is no other way to supply that answer, even when it is known elsewhere: a cache may hold the ranges from the last time the pair was diffed, and a format-aware process may have its own idea of which lines changed. Either could answer from the blob object ids alone, but the loading and computing are hard-wired into each consumer, so such an answer has no place to enter.
Introduce the hunk provider interface, diff-provider.h, between asking the question and computing the answer. A provider answers a request made of the pair's identity, its blob object ids and the parameters that determine the diff. A provider is either authoritative, so its answer may deliberately differ from the builtin diff, or not, so its answer must reproduce the builtin result exactly. Every answer served from identity passes diff_provider_check_hunk() before a consumer sees
it: coordinates fit int32, hunks are ordered and non-overlapping, and the unchanged runs between them match on both sides. A failing answer is discarded and the pair falls through as unanswered.
Providers are repository-lifecycle objects. Each repository owns a chain of them, built on first consultation and released from repo_clear(), so a submodule gets its own providers and no provider state outlives the repository it serves. The chain has a fixed composition, and each provider gates itself per request, passing when it does not apply. Chain order is the authority: the first answer wins. A provider may instead refuse a pair whose request is shaped by parameters its recording key cannot express. After a refusal, no later provider answers the pair from identity, and the consumer must not record what it computes for it. The last provider is the builtin computation, the only one that computes rather than answering from identity, so a walk given a fill callback always ends in an answer, refusal or not.
The walk in diff-provider.c maps a provider's four dispositions (answer, pass, fail, refuse) onto the consumer-facing outcomes, and checks with BUG() that only the computing provider fails and that it passes on a walk with no fill callback. The implementor contract, the provider struct, its dispositions, and the shared check, lives in diff-provider-internal.h, as refs/refs-internal.h is to refs.h; consumers see only diff-provider.h.
The consumer surface is two types. struct diff_provider_request names what is diffed and under which parameters; each later commit that consults on more state adds the field it keys on (the object ids and diff options, then the path). enum diff_provider_outcome flattens two dependent axes into four points: the response state (answered, unanswered, failed) and, only when unanswered, whether the caller may record what it computes. The record rule rides in the outcome, not a separate flag, so -Wswitch forces every consumer to place the no-record arm. A provider added later maps onto these values inside the walk, so consumer code is written once.
diff_provider_emit_hunks() is the consumer entry: the caller states the request, a hunk callback, and a content-loading callback that reaches the terminal provider only when the ranges are computed. Blame's pass_blame_to_parent() is the first consumer, since it knows both blob ids before reading either blob; its loads move into the fill callback. With only the terminal provider registered, every request still computes, so behavior is unchanged. (Blame's -C/-M split detection diffs partial buffers with no blob identity and stays on xdi_diff().)
--- Makefile | 1 + blame.c | 36 +++++++-- diff-provider-internal.h | 122 +++++++++++++++++++++++++++++++ diff-provider.c | 154 +++++++++++++++++++++++++++++++++++++++ diff-provider.h | 130 +++++++++++++++++++++++++++++++++ meson.build | 1 + repository.c | 3 + repository.h | 8 ++ 8 files changed, 447 insertions(+), 8 deletions(-) create mode 100644 diff-provider-internal.h create mode 100644 diff-provider.c create mode 100644 diff-provider.h
Makefile +1 -0
@@ -1151,6 +1151,7 @@ LIB_OBJS += diff-delta.o LIB_OBJS += diff-merges.o LIB_OBJS += diff-lib.o LIB_OBJS += diff-no-index.o +LIB_OBJS += diff-provider.o LIB_OBJS += diff.o LIB_OBJS += diffcore-break.o LIB_OBJS += diffcore-delta.o
blame.c +28 -8
@@ -23,6 +23,7 @@#include "commit-slab.h" #include "bloom.h" #include "commit-graph.h" +#include "diff-provider.h" define_commit_slab(blame_suspects, struct blame_origin *);static struct blame_suspects blame_suspects;
@@ -1936,6 +1937,28 @@ static int blame_chunk_cb(long start_a, long count_a,return 0 ; } +struct blame_diff_fill_data { +struct blame_scoreboard *sb; +struct blame_origin *parent, *target; +int ignore_diffs; +}; + +/* +* Content load for diff_provider_emit_hunks(): runs when the diff is +* computed. +*/ +static int blame_diff_fill (void *data,mmfile_t *old_file,mmfile_t *new_file) +{ +struct blame_diff_fill_data *f = data; + +fill_origin_blob (&f->sb->revs->diffopt, f->parent, old_file, + &f->sb->num_read_blob, f->ignore_diffs); +fill_origin_blob (&f->sb->revs->diffopt, f->target, new_file, + &f->sb->num_read_blob, f->ignore_diffs); + f->sb->num_get_patch++; +return 0 ; +} +/* * We are looking at the origin 'target' and aiming to pass blame * for the lines it is suspected to its parent. Run diff to find
@@ -1945,9 +1968,11 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,struct blame_origin *target, struct blame_origin *parent, int ignore_diffs) { - mmfile_t file_p, file_o; struct blame_chunk_cb_data d;struct blame_entry *newdest =NULL ; +struct blame_diff_fill_data fill_data = { sb, parent, target, ignore_diffs }; +xpparam_t xpp = { .flags = sb->xdl_opts }; +struct diff_provider_request req = { .repo = sb->repo, .xpp = &xpp };if (!target->suspects)return ;/* nothing remains for this target */
@@ -1958,13 +1983,8 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb, d.ignore_diffs = ignore_diffs; d.dstq = &newdest; d.srcq = &target->suspects; -fill_origin_blob (&sb->revs->diffopt, parent, &file_p, - &sb->num_read_blob, ignore_diffs); -fill_origin_blob (&sb->revs->diffopt, target, &file_o, - &sb->num_read_blob, ignore_diffs); - sb->num_get_patch++; - -if (diff_hunks (&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts)) +if (diff_provider_emit_hunks (&req, blame_diff_fill, &fill_data, + blame_chunk_cb, &d) ==DIFF_PROVIDER_ERROR )die ("unable to generate diff (%s -> %s)" ,oid_to_hex (&parent->commit->object.oid),oid_to_hex (&target->commit->object.oid));
diff-provider-internal.h new file +122 -0
122 lines not on this page · raw message
diff-provider.c new file +154 -0
154 lines not on this page · raw message
diff-provider.h new file +130 -0
130 lines not on this page · raw message
meson.build +1 -0
@@ -356,6 +356,7 @@ libgit_sources = [ 'diff-merges.c', 'diff-lib.c', 'diff-no-index.c', + 'diff-provider.c', 'diff.c', 'diffcore-break.c', 'diffcore-delta.c',
repository.c +3 -0
@@ -5,6 +5,7 @@#include "odb.h" #include "odb/source.h" #include "config.h" +#include "diff-provider.h" #include "gettext.h" #include "object.h" #include "lockfile.h"
@@ -383,6 +384,8 @@ void repo_clear(struct repository *repo)FREE_AND_NULL (repo->submodule_prefix);FREE_AND_NULL (repo->ref_storage_payload); +diff_providers_clear (repo); +odb_free (repo->objects); repo->objects =NULL ;
repository.h +8 -0
@@ -7,6 +7,7 @@#include "environment.h" struct config_set ; +struct diff_provider ;struct git_hash_algo ;struct index_state ;struct lock_file ;
@@ -161,6 +162,13 @@ struct repository {/* Repository's remotes and associated structures. */ struct remote_state *remote_state; +/* +* The repository's diff hunk provider chain, NULL until the +* first consultation builds it (diff-provider.c); repo_clear() +* releases it. +*/ +struct diff_provider *diff_providers; +/* Repository's current hash algorithm, as serialized on disk. */ const struct git_hash_algo *hash_algo;
Signature · 1 line
2.54.0
RFC PATCH v7 3/10 diff-hunks: add the store format, library, and command
In reply to Michael MontalboBlame and "git log --stat" recover hunk coordinates by diffing blob pairs, and recompute them on every run. Add a cache of those coordinates at $GIT_DIR/objects/info/diff-hunks, beside the commit-graph, so a later run can look them up instead of decompressing the blobs and running xdiff again.
The store is a single chunk-format file (see gitformat-chunk(5)): an 8-byte header, a DHIX index of fixed-size entries sorted by key, a DHDT segment of hunk records, and a trailing hash checksum. An entry is keyed by the two blob object ids and the xdl_opts the pair was diffed under, so a stored result is served only where that exact key recurs, independent of path. A zero-context diff trims unchanged lines from hunk edges and can pick a different but equally valid set of hunks than an untrimmed diff, so a recording caller stores a pair only when its trimmed and untrimmed diffs are identical; such an entry answers any consumer at any context, and the rare divergent pair is always computed. Identical hunk blocks are interned once and shared across keys.
The library provides a reader (repo_diff_hunks_store and _replay, gated by core.diffHunks), loaded once and cached on the object database as the commit-graph is, and a writer that accumulates entries and flushes them in one atomic pass. An absent, corrupt, or disabled store reads as all misses. A record with no hunks is invalid too: replaying it would claim the pair equivalent, which the store never asserts, so it reads as a miss.
Ordinary reads are diagnostic-free. Loading parses the chunk table through read_table_of_contents_quiet(), new in chunk-format, which prints nothing on a malformed table and takes the repository's hash algorithm rather than the_hash_algo, so the file is bounds-checked under the algorithm it is keyed by.
The flush closes the repository's mmapped store and forgets that loading was attempted before committing the lockfile. A warming run that also reads may hold the file it is replacing mapped, and the rename must not land on a live mapping, which Windows refuses; a read after the flush then observes the committed file. commit-graph closes its graph before committing for the same reason.
Writing is off by default, enabled per run by GIT_DIFF_HUNKS_WRITE or persistently by diffHunks.write, the environment winning. A writer seeds from the existing store, so a flush merges rather than replaces. The seed's checksum is verified first: a corrupt store is discarded, not rewritten with a fresh checksum verify could no longer catch. An entry that fails the shared diff_provider_check_hunk() or names no blob is dropped with a warning, since it would only ever read as a miss. A seed that discarded or dropped anything forces the flush even when the warming run computed nothing new. The writer fsyncs through a new diff-hunks core.fsync component.
"git diff-hunks" inspects and manages the file: "verify" checks the checksum, chunk table, sort order, entry bounds, and every entry's hunk sequence against that shared check, so a store whose entries could only read as misses fails verify; "clear" removes the file. Later patches wire the readers and the writer into the diff and blame paths.
--- .gitignore | 1 + Documentation/Makefile | 1 + Documentation/config.adoc | 2 + Documentation/config/core.adoc | 10 +- Documentation/config/diff-hunks.adoc | 8 + Documentation/git-diff-hunks.adoc | 146 ++++ Documentation/gitformat-diff-hunks.adoc | 129 ++++ Documentation/meson.build | 2 + Makefile | 2 + builtin.h | 1 + builtin/diff-hunks.c | 53 ++ chunk-format.c | 62 +- chunk-format.h | 14 + command-list.txt | 2 + diff-hunks.c | 916 ++++++++++++++++++++++++ diff-hunks.h | 117 +++ environment.c | 1 + git.c | 1 + meson.build | 2 + odb.c | 2 + odb.h | 4 + repo-settings.c | 1 + repo-settings.h | 1 + write-or-die.h | 7 +- 24 files changed, 1467 insertions(+), 18 deletions(-) create mode 100644 Documentation/config/diff-hunks.adoc create mode 100644 Documentation/git-diff-hunks.adoc create mode 100644 Documentation/gitformat-diff-hunks.adoc create mode 100644 builtin/diff-hunks.c create mode 100644 diff-hunks.c create mode 100644 diff-hunks.h
.gitignore +1 -0
@@ -56,6 +56,7 @@ /git-diagnose /git-diff /git-diff-files +/git-diff-hunks /git-diff-index /git-diff-pairs /git-diff-tree
Documentation/Makefile +1 -0
@@ -33,6 +33,7 @@ MAN5_TXT += gitattributes.adoc MAN5_TXT += gitformat-bundle.adoc MAN5_TXT += gitformat-chunk.adoc MAN5_TXT += gitformat-commit-graph.adoc +MAN5_TXT += gitformat-diff-hunks.adoc MAN5_TXT += gitformat-index.adoc MAN5_TXT += gitformat-loose.adoc MAN5_TXT += gitformat-pack.adoc
Documentation/config.adoc +2 -0
@@ -472,6 +472,8 @@ include::config/credential.adoc[] include::config/diff.adoc[] +include::config/diff-hunks.adoc[] + include::config/difftool.adoc[] include::config/extensions.adoc[]
Documentation/config/core.adoc +9 -1
@@ -670,12 +670,13 @@ but risks losing recent work in the event of an unclean system shutdown. * `pack` hardens objects added to the repo in packfile form. * `pack-metadata` hardens packfile bitmaps and indexes. * `commit-graph` hardens the commit-graph file. +* `diff-hunks` hardens the diff-hunks store. * `index` hardens the index when it is modified. * `objects` is an aggregate option that is equivalent to `loose-object,pack`. * `reference` hardens references modified in the repo. * `derived-metadata` is an aggregate option that is equivalent to - `pack-metadata,commit-graph`. + `pack-metadata,commit-graph,diff-hunks`. * `committed` is an aggregate option that is currently equivalent to `objects`. This mode sacrifices some performance to ensure that work that is committed to the repository with `git commit` or similar commands
@@ -750,6 +751,13 @@ core.commitGraph:: to parse the graph structure of commits. Defaults to true. See linkgit:git-commit-graph[1] for more information. +core.diffHunks:: + If true, then Git will consult the diff-hunks store (if it + exists) to skip recomputing diff hunk coordinates in commands + such as `git log --stat` and linkgit:git-blame[1]. This controls + only reading; writing the store is controlled by `diffHunks.write`. + See linkgit:git-diff-hunks[1] for more information. Defaults to true. + core.useReplaceRefs:: If set to `false`, behave as if the `--no-replace-objects` option was given on the command line. See linkgit:git[1] and
Documentation/config/diff-hunks.adoc new file +8 -0
@@ -0,0 +1,8 @@ +diffHunks.write:: + If true, diff-producing commands (`git diff`, `git log`, + `git show`, or `git diff-tree` with a `--stat`, `--numstat`, or + `--shortstat` format) write the hunks + they compute to the diff-hunks store, filling it as a side effect. + The `GIT_DIFF_HUNKS_WRITE` environment variable overrides this for + a single invocation. Reading the store is controlled separately by + `core.diffHunks`. See linkgit:git-diff-hunks[1]. Defaults to false.
Documentation/git-diff-hunks.adoc new file +146 -0
146 lines not on this page · raw message
Documentation/gitformat-diff-hunks.adoc new file +129 -0
129 lines not on this page · raw message
Documentation/meson.build +2 -0
@@ -41,6 +41,7 @@ manpages = { 'git-describe.adoc' : 1, 'git-diagnose.adoc' : 1, 'git-diff-files.adoc' : 1, + 'git-diff-hunks.adoc' : 1, 'git-diff-index.adoc' : 1, 'git-diff-pairs.adoc' : 1, 'git-difftool.adoc' : 1,
@@ -175,6 +176,7 @@ manpages = { 'gitformat-bundle.adoc' : 5, 'gitformat-chunk.adoc' : 5, 'gitformat-commit-graph.adoc' : 5, + 'gitformat-diff-hunks.adoc' : 5, 'gitformat-index.adoc' : 5, 'gitformat-loose.adoc' : 5, 'gitformat-pack.adoc' : 5,
Makefile +2 -0
@@ -1159,6 +1159,7 @@ LIB_OBJS += diffcore-order.o LIB_OBJS += diffcore-pickaxe.o LIB_OBJS += diffcore-rename.o LIB_OBJS += diffcore-rotate.o +LIB_OBJS += diff-hunks.o LIB_OBJS += dir-iterator.o LIB_OBJS += dir.o LIB_OBJS += editor.o
@@ -1421,6 +1422,7 @@ BUILTIN_OBJS += builtin/credential.o BUILTIN_OBJS += builtin/describe.o BUILTIN_OBJS += builtin/diagnose.o BUILTIN_OBJS += builtin/diff-files.o +BUILTIN_OBJS += builtin/diff-hunks.o BUILTIN_OBJS += builtin/diff-index.o BUILTIN_OBJS += builtin/diff-pairs.o BUILTIN_OBJS += builtin/diff-tree.o
builtin.h +1 -0
@@ -175,6 +175,7 @@ int cmd_credential_store(int argc, const char **argv, const char *prefix, structint cmd_describe (int argc,const char **argv,const char *prefix,struct repository *repo);int cmd_diagnose (int argc,const char **argv,const char *prefix,struct repository *repo);int cmd_diff_files (int argc,const char **argv,const char *prefix,struct repository *repo); +int cmd_diff_hunks (int argc,const char **argv,const char *prefix,struct repository *repo);int cmd_diff_index (int argc,const char **argv,const char *prefix,struct repository *repo);int cmd_diff (int argc,const char **argv,const char *prefix,struct repository *repo);int cmd_diff_pairs (int argc,const char **argv,const char *prefix,struct repository *repo);
builtin/diff-hunks.c new file +53 -0
53 lines not on this page · raw message
chunk-format.c +47 -15
100 lines not on this page · raw message
chunk-format.h +14 -0
@@ -39,6 +39,20 @@ int read_table_of_contents(struct chunkfile *cf,int toc_length, unsigned expected_alignment); +/* +* Like read_table_of_contents(), for a reader that treats a malformed +* table as an absent file rather than reporting it: nothing is printed +* on failure, and the trailing-checksum bound is computed with the +* given hash algorithm instead of the_hash_algo. +*/ +int read_table_of_contents_quiet (struct chunkfile *cf, +const unsigned char *mfile, +size_t mfile_size, +uint64_t toc_offset, +int toc_length, +unsigned expected_alignment, +const struct git_hash_algo *algo); +#define CHUNK_NOT_FOUND (-2)/*
command-list.txt +2 -0
@@ -95,6 +95,7 @@ git-describe mainporcelain git-diagnose ancillaryinterrogators git-diff mainporcelain info git-diff-files plumbinginterrogators +git-diff-hunks plumbingmanipulators git-diff-index plumbinginterrogators git-diff-pairs plumbinginterrogators git-diff-tree plumbinginterrogators
@@ -223,6 +224,7 @@ gitfaq guide gitformat-bundle developerinterfaces gitformat-chunk developerinterfaces gitformat-commit-graph developerinterfaces +gitformat-diff-hunks developerinterfaces gitformat-index developerinterfaces gitformat-pack developerinterfaces gitformat-signature developerinterfaces
diff-hunks.c new file +916 -0
916 lines not on this page · raw message
diff-hunks.h new file +117 -0
117 lines not on this page · raw message
environment.c +1 -0
@@ -239,6 +239,7 @@ static const struct fsync_component_name { { "pack", FSYNC_COMPONENT_PACK }, { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "diff-hunks", FSYNC_COMPONENT_DIFF_HUNKS }, { "index", FSYNC_COMPONENT_INDEX }, { "objects", FSYNC_COMPONENTS_OBJECTS }, { "reference", FSYNC_COMPONENT_REFERENCE },
git.c +1 -0
@@ -566,6 +566,7 @@ static struct cmd_struct commands[] = { { "diagnose", cmd_diagnose, RUN_SETUP_GENTLY }, { "diff", cmd_diff, NO_PARSEOPT }, { "diff-files", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT }, + { "diff-hunks", cmd_diff_hunks, RUN_SETUP }, { "diff-index", cmd_diff_index, RUN_SETUP | NO_PARSEOPT }, { "diff-pairs", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT }, { "diff-tree", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT },
meson.build +2 -0
@@ -364,6 +364,7 @@ libgit_sources = [ 'diffcore-pickaxe.c', 'diffcore-rename.c', 'diffcore-rotate.c', + 'diff-hunks.c', 'dir-iterator.c', 'dir.c', 'editor.c',
@@ -633,6 +634,7 @@ builtin_sources = [ 'builtin/describe.c', 'builtin/diagnose.c', 'builtin/diff-files.c', + 'builtin/diff-hunks.c', 'builtin/diff-index.c', 'builtin/diff-pairs.c', 'builtin/diff-tree.c',
odb.c +2 -0
@@ -2,6 +2,7 @@#include "abspath.h" #include "commit-graph.h" #include "config.h" +#include "diff-hunks.h" #include "dir.h" #include "environment.h" #include "gettext.h"
@@ -1033,6 +1034,7 @@ void odb_close(struct object_database *o)for (source = o->sources; source; source = source->next)odb_source_close (source);close_commit_graph (o); +close_diff_hunks_store (o); }static void odb_free_sources (struct object_database *o)
odb.h +4 -0
@@ -8,6 +8,7 @@#include "thread-utils.h" struct cached_object_entry ; +struct diff_hunks_store ;struct list_objects_filter_options ;struct odb_source_inmemory ;struct packed_git ;
@@ -76,6 +77,9 @@ struct object_database {struct commit_graph *commit_graph; unsigned commit_graph_attempted : 1; /* if loading has been attempted */ +struct diff_hunks_store *diff_hunks_store; + unsigned diff_hunks_store_attempted : 1; /* if loading has been attempted */ +/* * This is meant to hold a *small* number of objects that you would * want odb_read_object() to be able to return, but yet you do not want
repo-settings.c +1 -0
@@ -77,6 +77,7 @@ void prepare_repo_settings(struct repository *r)repo_cfg_bool (r,"pack.usesparse" , &r->settings.pack_use_sparse,1 );repo_cfg_bool (r,"pack.usepathwalk" , &r->settings.pack_use_path_walk,0 );repo_cfg_bool (r,"core.multipackindex" , &r->settings.core_multi_pack_index,1 ); +repo_cfg_bool (r,"core.diffhunks" , &r->settings.core_diff_hunks,1 );repo_cfg_bool (r,"index.sparse" , &r->settings.sparse_index,0 );repo_cfg_bool (r,"index.skiphash" , &r->settings.index_skip_hash, r->settings.index_skip_hash);repo_cfg_bool (r,"pack.readreverseindex" , &r->settings.pack_read_reverse_index,1 );
repo-settings.h +1 -0
@@ -22,6 +22,7 @@ struct repo_settings {int core_commit_graph;int commit_graph_generation_version;int commit_graph_changed_paths_version; +int core_diff_hunks;int gc_write_commit_graph;int fetch_write_commit_graph;int command_requires_full_index;
write-or-die.h +5 -2
@@ -22,13 +22,15 @@ enum fsync_component {FSYNC_COMPONENT_INDEX =1 <<4 ,FSYNC_COMPONENT_REFERENCE =1 <<5 ,FSYNC_COMPONENT_OBJECT_MAP =1 <<6 , + FSYNC_COMPONENT_DIFF_HUNKS = 1 << 7, };#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ FSYNC_COMPONENT_PACK)#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) + FSYNC_COMPONENT_COMMIT_GRAPH | \ + FSYNC_COMPONENT_DIFF_HUNKS)#define FSYNC_COMPONENTS_DEFAULT ((FSYNC_COMPONENTS_OBJECTS | \ FSYNC_COMPONENTS_DERIVED_METADATA) & \
@@ -46,7 +48,8 @@ enum fsync_component { FSYNC_COMPONENT_COMMIT_GRAPH | \ FSYNC_COMPONENT_INDEX | \ FSYNC_COMPONENT_REFERENCE | \ - FSYNC_COMPONENT_OBJECT_MAP) + FSYNC_COMPONENT_OBJECT_MAP | \ + FSYNC_COMPONENT_DIFF_HUNKS)#ifndef FSYNC_COMPONENTS_PLATFORM_DEFAULT #define FSYNC_COMPONENTS_PLATFORM_DEFAULT FSYNC_COMPONENTS_DEFAULT
Signature · 1 line
2.54.0
RFC PATCH v7 4/10 diff: record precomputed hunks during stat output
In reply to Michael MontalboThe diff-hunks store has a writer, but nothing fills it. Teach builtin_diffstat() to do so: on a warming run (a writer is attached), a modified pair's stat is produced by collecting the pair's hunk coordinates instead of emitting text, the counts are summed from those hunks, and the pair is recorded. A run without a writer is unchanged, and nothing reads the store yet; the read side arrives next.
The store records one context-free entry per pair, and only for a trim-stable pair: one whose zero-context trimmed diff (what blame will read) and untrimmed diff (whose counts a nonzero-context stat matches) are identical. The warming path computes both and hands them to diff_hunks_writer_record_stable(), new here, which records only when they agree; a divergent pair is never recorded and every consumer computes it. The warming run displays the counts it shows a store-less
run: the trimmed ones, since xdi_diff trims at zero context, while the untrimmed counts serve only the stability comparison.
Not everything the stat path computes may be recorded. --ignore-blank-lines is part of the key, but it coalesces hunks differently between the text-emitting and coordinate-callback paths, so a recorded entry would not match a store-less run's --stat. -I patterns, --anchored, and break detection (-B) shape the diff outside the key entirely; the guard for those three sits in this consumer for now and moves into the store's own provider when it registers, next. A "log -L" range-scoped stat is not the whole-pair diff the key describes, so it does not record. Recording also requires both sides to be valid regular files whose blobs the key can name: a working-tree side, textconv output, or a gitlink has no usable id.
"git diff", "git log", "git show", and "git diff-tree" with the --stat, --numstat, and --shortstat formats attach a writer when writing is enabled and flush it when the traversal finishes, so a warming run such as
GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null
fills the cache as a side effect of the diff work the command already does. Writing is controlled by diffHunks.write and GIT_DIFF_HUNKS_WRITE.
Add the write half of t4220:
- ordinary commands never create the store, and creation is gated off by default, the environment overriding the config;
- a warming run builds a store that verifies, and a second refreshes it in place;
- a warming run displays parity at zero context on a trim-divergent pair, committed as a fixture (small synthetic pairs cannot diverge:
minimal diffs add and delete equal counts, and trimming preserves that);
- binary and mode-only pairs do not break the writer;
- a corrupt store is discarded at seed;
- verify and clear run against the files a warming run builds.
--- builtin/diff-tree.c | 3 + builtin/diff.c | 10 ++ builtin/log.c | 7 + diff-hunks.c | 32 ++++ diff-hunks.h | 18 ++- diff.c | 218 +++++++++++++++++++++---- diff.h | 17 ++ t/meson.build | 1 + t/t4220-diff-hunks.sh | 184 +++++++++++++++++++++ t/t4220/README | 55 +++++++ t/t4220/trim-divergent-new | 319 +++++++++++++++++++++++++++++++++++++ t/t4220/trim-divergent-old | 316 ++++++++++++++++++++++++++++++++++++ 12 files changed, 1150 insertions(+), 30 deletions(-) create mode 100755 t/t4220-diff-hunks.sh create mode 100644 t/t4220/README create mode 100644 t/t4220/trim-divergent-new create mode 100644 t/t4220/trim-divergent-old
builtin/diff-tree.c +3 -0
@@ -170,6 +170,8 @@ int cmd_diff_tree(int argc, opt->diffopt.rotate_to_strict =1 ; +diff_hunks_attach (&opt->diffopt); +/* * NOTE! We expect "a..b" to expand to "^a b" but it is * perfectly valid for revision range parser to yield "b ^a",
@@ -234,5 +236,6 @@ int cmd_diff_tree(int argc,diff_free (&opt->diffopt); } +diff_hunks_detach (&opt->diffopt);return diff_result_code (opt); }
builtin/diff.c +10 -0
@@ -568,6 +568,15 @@ int cmd_diff(int argc, } } +/* +* The hunk store is keyed by blob pair, so any diff whose +* file pairs carry known blob object IDs (tree-to-tree, +* index-to-tree) can consult the same entries that +* "git log --stat" and "git blame" use; pairs without known +* blobs bypass it at lookup time. +*/ +diff_hunks_attach (&rev.diffopt); +symdiff_prepare (&rev, &sdiff);for (i =0 ; i < rev.pending.nr; i++) {struct object_array_entry *entry = &rev.pending.objects[i];
@@ -648,6 +657,7 @@ int cmd_diff(int argc, result =diff_result_code (&rev);if (1 < rev.diffopt.skip_stat_unmatch)refresh_index_quietly (); +diff_hunks_detach (&rev.diffopt);release_revisions (&rev);object_array_clear (&ent);symdiff_release (&sdiff);
builtin/log.c +7 -0
@@ -693,8 +693,11 @@ int cmd_show(int argc, opt.tweak = show_setup_revisions_tweak;cmd_log_init (argc, argv, prefix, &rev, &opt, &cfg); +diff_hunks_attach (&rev.diffopt); +if (!rev.no_walk) { ret =cmd_log_walk (&rev); +diff_hunks_detach (&rev.diffopt);release_revisions (&rev);log_config_release (&cfg);return ret;
@@ -767,6 +770,7 @@ int cmd_show(int argc, } rev.diffopt.no_free =0 ; +diff_hunks_detach (&rev.diffopt);diff_free (&rev.diffopt);release_revisions (&rev);log_config_release (&cfg);
@@ -846,8 +850,11 @@ int cmd_log(int argc, opt.tweak = log_setup_revisions_tweak;cmd_log_init (argc, argv, prefix, &rev, &opt, &cfg); +diff_hunks_attach (&rev.diffopt); + ret =cmd_log_walk (&rev); +diff_hunks_detach (&rev.diffopt);release_revisions (&rev);log_config_release (&cfg);return ret;
diff-hunks.c +32 -0
@@ -651,6 +651,38 @@ int diff_hunks_writer_add(struct diff_hunks_writer *w,return 1 ; } +void diff_hunks_writer_record_stable (struct diff_hunks_writer *w, +const struct object_id *old_oid, +const struct object_id *new_oid, +int xdl_opts, +const struct precomputed_hunk *trimmed, +size_t nr_trimmed, +const struct precomputed_hunk *full, +size_t nr_full) +{ +size_t i; + +if (!w) +return ; +/* +* Record only a trim-stable pair, one whose trimmed and +* untrimmed diffs are identical, so the single entry answers +* any consumer at any context (see the top of this file). A +* pair where the two diffs differ is never recorded and every +* consumer computes it. +*/ +if (nr_trimmed != nr_full) +return ; +for (i =0 ; i < nr_trimmed; i++) +if (trimmed[i].old_start != full[i].old_start || + trimmed[i].old_count != full[i].old_count || + trimmed[i].new_start != full[i].new_start || + trimmed[i].new_count != full[i].new_count) +return ; +diff_hunks_writer_add (w, old_oid, new_oid, xdl_opts, + trimmed, nr_trimmed); +} +/* * Seed the writer with fname's entries so a rewrite preserves them, * setting *pruned when the rewrite will not carry the whole file
diff-hunks.h +17 -1
@@ -94,7 +94,8 @@ struct diff_hunks_writer *diff_hunks_writer_maybe_new(struct repository *r);* Record a blob pair's hunks as computed under xdl_opts; a later lookup * with a matching key is served these hunks. The caller must have * checked that the pair's trimmed and untrimmed diffs are identical -* (see the top of this file), so the entry answers at any context. +* (see the top of this file), so the entry answers at any context; +* diff_hunks_writer_record_stable() below performs that check. * NULL-safe. Returns 1 when the entry was recorded, 0 when the writer * refused it (no hunks, a null object id, or values the on-disk * 32-bit fields cannot hold).
@@ -106,6 +107,21 @@ int diff_hunks_writer_add(struct diff_hunks_writer *w,const struct precomputed_hunk *hunks, size_t nr_hunks); +/* +* Record the pair only if it is trim-stable: the recording caller +* hands over both the trimmed (xdi_diff) and untrimmed (xdl_diff) +* zero-context hunk sequences it computed, and the entry is added +* only when the two are identical. NULL-safe. +*/ +void diff_hunks_writer_record_stable (struct diff_hunks_writer *w, +const struct object_id *old_oid, +const struct object_id *new_oid, +int xdl_opts, +const struct precomputed_hunk *trimmed, +size_t nr_trimmed, +const struct precomputed_hunk *full, +size_t nr_full); +/* Flush the accumulated entries to the store and free the writer. NULL-safe. */ void diff_hunks_writer_finish (struct diff_hunks_writer *w);
diff.c +189 -29
245 lines not on this page · raw message
diff.h +17 -0
@@ -420,6 +420,13 @@ struct diff_options {*/ int max_depth;int max_depth_valid; + +/* +* Precomputed diff hunks (see diff-hunks.h). When hunks_writer is +* set (a warming run), diffstat records the hunks it computes; +* the writer is attached only for the stat output formats. +*/ +struct diff_hunks_writer *hunks_writer; };unsigned diff_filter_bit (char status);
@@ -668,6 +675,16 @@ void diffcore_fix_diff_index(void);int diff_queue_is_empty (struct diff_options *o);void diff_flush (struct diff_options *);void diff_free (struct diff_options *); + +/* +* Attach a diff-hunks writer to a diff producing a stat format, so a +* warming run records the hunks it computes; a no-op when writing is off +* or for other formats. Pair with diff_hunks_detach() once the diff is +* done. +*/ +void diff_hunks_attach (struct diff_options *o); +void diff_hunks_detach (struct diff_options *o); +void diff_warn_rename_limit (const char *varname,int needed,int degraded_cc);/* diff-raw status letters */
t/meson.build +1 -0
@@ -583,6 +583,7 @@ integration_tests = [ 't4216-log-bloom.sh', 't4217-log-limit.sh', 't4219-log-follow-merge.sh', + 't4220-diff-hunks.sh', 't4252-am-options.sh', 't4253-am-keep-cr-dos.sh', 't4254-am-corrupt.sh',
t/t4220-diff-hunks.sh new file +184 -0
184 lines not on this page · raw message
t/t4220/README new file +55 -0
55 lines not on this page · raw message
t/t4220/trim-divergent-new new file +319 -0
319 lines not on this page · raw message
t/t4220/trim-divergent-old new file +316 -0
316 lines not on this page · raw message
Signature · 1 line
2.54.0