diff: add diff.<driver>.process for external hunk providers
Subsystems (3)
RFC PATCH v7 5/10 diff: read precomputed hunks for stat output
In reply to Michael MontalboTeach builtin_diffstat() to consult the hunk provider interface through diff_provider_consult(), new here: the consult-only entry that answers without loading content or computing, so it never returns DIFF_PROVIDER_ERROR. On an answer, the summing callback accumulates the provided counts directly into the diffstat entry; the blobs were already loaded for the binary check, so an answer saves the diff run, not the content load (blame, taught next, skips its loads too). On an unanswered outcome it computes as before and, with a writer attached, records what it computed; on unanswered-no-record it computes without recording.
The provider behind the consult is the diff-hunks store, registered in front of the terminal builtin computation. Its consult serves a recorded pair through diff_hunks_replay(), which validates the sequence before any hunk reaches the callback, so direct accumulation is safe. The request gains the pair's object ids and the diff options read by the exclusions below. A side whose bytes are not a stored blob, such as a working-tree file or a gitlink, has a NULL id; the store passes it by and the terminal provider computes it. diff_provider_emit_hunks() walks the same chain, so blame's requests follow these rules the moment blame supplies identity. The walk also insists, as a BUG check, that a request's diff options belong to the repository whose chain it walks.
Each exclusion lives with the provider whose key cannot express it. -I patterns and --anchored shape the diff outside the store key, and break detection (-B) rescores the pair outside it; the store's consult maps all three to stop-no-record, so such a request is neither served nor recorded for any consumer. The consumer-side guard the recording commit carried for those three comes out here. The compile-time assert on xpparam_t's layout sits next to that decision, forcing an explicit keying decision whenever a diff parameter is added. The stat consumer keeps only the exclusion that is not about the key: --ignore-blank-lines is part of the key but coalesces hunks differently between the text-emitting and coordinate-callback paths, so the consumer returns before consulting. A "log -L" range-scoped stat neither reads nor records; the line-range filter computes it as before.
"git diff", "git log", "git show", and "git diff-tree" with the --stat, --numstat, and --shortstat formats consult the interface. Reading is controlled by core.diffHunks.
An answer is invisible in the output, so the store counts the pairs it serves and the consultations it cannot, and diff_hunks_read_stats() reports both; the stat path emits the hits as a trace2 "read-hits" datum for tests and tuning. The counters live on the store because only the store knows whether a consultation reached it, and none of its exclusion legs reaches the replay, so none counts as a miss.
Extend t4220 with the read half:
- output parity with and without the store, at several context lengths and both directions, and reversed pairs keying apart;
- the consultation made visible through the read-hits datum, and the trim-divergent pair correct at every context;
- the settings that must bypass the store doing so in both directions (-I, -B, --anchored, --ignore-blank-lines), asserted through the trace rather than output parity alone, which a coincidentally equal count could satisfy;
- a driver-forced algorithm keying apart rather than bypassing: it is part of the key, so a read under it misses the default entries and a warm records under its own.
A "log -L" range-scoped stat neither reads nor records.
--- builtin/log.c | 7 + diff-hunks.c | 88 +++++++++++- diff-hunks.h | 8 ++ diff-provider-internal.h | 7 + diff-provider.c | 62 +++++++-- diff-provider.h | 28 +++- diff.c | 86 ++++++++---- diff.h | 20 ++- t/t4220-diff-hunks.sh | 280 +++++++++++++++++++++++++++++++++++++++ 9 files changed, 539 insertions(+), 47 deletions(-)
builtin/log.c +7 -0
@@ -2225,6 +2225,13 @@ int cmd_format_patch(int argc,if (argc >1 )die (_ ("unrecognized argument: %s" ), argv[1 ]); +/* +* A patch generated by format-patch carries the builtin diffstat, +* not one served from a local store, so its counts do not depend +* on whether the sender warmed the store. +*/ + rev.diffopt.flags.no_precomputed_hunks =1 ; +if (rev.diffopt.output_format &DIFF_FORMAT_NAME )die (_ ("--name-only does not make sense" ));if (rev.diffopt.output_format &DIFF_FORMAT_NAME_STATUS )
diff-hunks.c +87 -1
123 lines not on this page · raw message
diff-hunks.h +8 -0
@@ -62,6 +62,14 @@ struct diff_hunks_store *repo_diff_hunks_store(struct repository *r);/* Free the repository's cached store, at object-database teardown. */ void close_diff_hunks_store (struct object_database *o); +/* +* Consultation counters for the repository's store: pairs the store +* served (hits) and pairs it was consulted for but could not serve +* (misses). Both zero when reading is disabled or no store exists. +*/ +void diff_hunks_read_stats (struct repository *r, +unsigned long *hits,unsigned long *misses); +/* * Replay the recorded hunks of an (old blob, new blob) pair diffed * under xdl_opts through hunk_func. The sequence is validated before
diff-provider-internal.h +7 -0
@@ -88,6 +88,13 @@ struct diff_provider {struct diff_provider *next; }; +/* +* The providers Git ships, besides the builtin computation that +* diff-provider.c holds itself. Each call returns a fresh provider +* for one repository's chain. +*/ +struct diff_provider *diff_hunks_store_provider_new (void ); +/* * Incremental well-formedness check for a provider-supplied hunk * sequence, shared by every provider. Each coordinate, and each
diff-provider.c +48 -14
97 lines not on this page · raw message
diff-provider.h +25 -3
@@ -26,6 +26,8 @@* sees it. */ +struct diff_options ; +struct object_id ;struct repository ;/*
@@ -89,15 +91,35 @@ enum diff_provider_outcome {* A consultation request. The interface consults providers from * these fields alone; no content is loaded before an answer. * -* repo owns the provider chain the request walks. xpp carries the -* parameters the diff runs with. Each provider gates itself on the -* fields that concern it. +* repo owns the provider chain the request walks. old_oid/new_oid +* name the blobs whose bytes are diffed; pass NULL for a side whose +* bytes are not a stored blob (a working-tree file, textconv output, +* a gitlink), so no provider answers from an id it cannot look up. +* diffopt carries the diff settings that live outside xpp; xpp +* carries the parameters the diff runs with. Each provider gates +* itself on the fields that concern it. */ struct diff_provider_request {struct repository *repo; +const struct object_id *old_oid; +const struct object_id *new_oid; +struct diff_options *diffopt;const xpparam_t *xpp; }; +/* +* Consult the providers for the request's pair without computing. +* On DIFF_PROVIDER_ANSWERED the hunks were emitted through hunk_cb +* (0-based emission coordinates, context 0) and were validated +* before the first callback ran, so a consumer may accumulate +* directly into its result. Never returns DIFF_PROVIDER_ERROR. +* The callback's return value is not consulted: emission of a +* validated answer has no error leg, so the callback must return 0. +*/ +enum diff_provider_outcome +diff_provider_consult (const struct diff_provider_request *req, +xdl_emit_hunk_consume_func_t hunk_cb,void *cb_data); +/* * Load the pair's content. Called at most once per request, only * when the ranges are computed rather than provided. The buffers
diff.c +62 -24
130 lines not on this page · raw message
diff.h +15 -5
@@ -206,6 +206,13 @@ struct diff_flags {unsigned suppress_diff_headers;unsigned dual_color_diffed_diffs;unsigned suppress_hunk_header_line_count; + +/* +* Do not serve the diffstat from the precomputed-hunks store. +* Set by format-patch so a generated patch carries the builtin +* counts and does not depend on the sender's local store state. +*/ +unsigned no_precomputed_hunks; }; static inline void diff_flags_or(struct diff_flags *a,
@@ -422,9 +429,11 @@ struct diff_options {int max_depth_valid;/* -* Precomputed diff hunks (see diff-hunks.h). When hunks_writer is -* set (a warming run), diffstat records the hunks it computes; -* the writer is attached only for the stat output formats. +* Precomputed diff hunks (see diff-hunks.h). diffstat consults the +* hunk provider interface before running xdiff, keyed by each file +* pair's blob object IDs. When hunks_writer is set (a warming run), +* diffstat also records the hunks it computes; the writer is +* attached only for the stat output formats. */ struct diff_hunks_writer *hunks_writer; };
@@ -679,8 +688,9 @@ void diff_free(struct diff_options*);/* * Attach a diff-hunks writer to a diff producing a stat format, so a * warming run records the hunks it computes; a no-op when writing is off -* or for other formats. Pair with diff_hunks_detach() once the diff is -* done. +* or for other formats. (Reading is separate: consumers consult the +* providers through diff_provider_consult(); see diff-provider.h.) Pair +* with diff_hunks_detach() once the diff is done. */ void diff_hunks_attach (struct diff_options *o);void diff_hunks_detach (struct diff_options *o);
t/t4220-diff-hunks.sh +280 -0
304 lines not on this page · raw message
Signature · 1 line
2.54.0
RFC PATCH v7 6/10 blame: read precomputed hunks
In reply to Michael MontalboBefore diffing a target blob against a parent, offer the pair's identity to the hunk provider interface. Blame's requests have gone through diff_provider_emit_hunks() since the interface arrived, but carried no identity, so nothing could answer them. Now blame fills in the pair's blob object ids and its diff options, and the chain serves the pair from the store, keyed by the ids and the request's xdiff flags, before the terminal provider falls back to fill-and-compute. Blame diffs at zero context, which is not part of the key. An answer replays the recorded hunks through blame_chunk_cb without loading either blob; a request carrying -I patterns or anchors is outside the key and always computes.
Blame withholds the identity where its diff is not the plain blob-pair diff the key describes: reverse blame, ignored revisions, textconv paths, and the working-tree or --contents pseudo-commit, whose blob is not a stored object. Those requests always compute. Whitespace and algorithm options such as -w instead change blame's xdl_opts, so the consult keys a different entry and misses a store warmed without them. Blame's default xdl_opts now come from DIFF_HUNKS_DEFAULT_XDL_OPTS, new here, which records the key-relevant defaults a diff_options-based consumer already carries (today the indent heuristic), so a default blame run and a default "log --stat" warming run share keys by construction.
"--show-stats" reports how many pairs the store served and how many consultations it could not, read from diff_hunks_read_stats(); the store counts its own consultations, so blame keeps no tally.
Extend t4220 with the blame side:
- parity for plain, --porcelain, and --incremental output, and hit and miss accounting across warming runs;
- the blame inputs that must bypass or miss the store: -w, indent heuristics, --reverse, textconv, -M/-C, and the --ignore-rev pass;
- rename and merge handling, and --contents;
- reading a truncated or corrupt store as absent, and a crafted zero-hunk record as a miss that verify flags.
Add p4218, measuring the cost of a warming run and the read speedups.
--- blame.c | 39 ++++ builtin/blame.c | 8 +- diff.h | 11 ++ t/meson.build | 1 + t/perf/p4218-diff-hunks.sh | 48 +++++ t/t4220-diff-hunks.sh | 355 +++++++++++++++++++++++++++++++++++++ 6 files changed, 461 insertions(+), 1 deletion(-) create mode 100755 t/perf/p4218-diff-hunks.sh
blame.c +39 -0
@@ -24,6 +24,7 @@#include "bloom.h" #include "commit-graph.h" #include "diff-provider.h" +#include "userdiff.h" define_commit_slab(blame_suspects, struct blame_origin *);static struct blame_suspects blame_suspects;
@@ -1937,6 +1938,24 @@ static int blame_chunk_cb(long start_a, long count_a,return 0 ; } +/* +* A hunk provider's key names the (old blob, new blob) pair and may only +* serve a diff whose result is determined by that pair and the xdiff +* settings. Textconv rewrites the buffers being diffed away from the +* blob contents the key names, so any origin whose path has a textconv +* driver must withhold the pair's identity. +*/ +static int blame_textconv_active (struct blame_scoreboard *sb, +const char *path) +{ +struct userdiff_driver *drv; + +if (!sb->revs->diffopt.flags.allow_textconv) +return 0 ; + drv =userdiff_find_by_path (sb->repo->index, path); +return drv && drv->textconv; +} +struct blame_diff_fill_data {struct blame_scoreboard *sb;struct blame_origin *parent, *target;
@@ -1973,6 +1992,7 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,struct blame_diff_fill_data fill_data = { sb, parent, target, ignore_diffs };xpparam_t xpp = { .flags = sb->xdl_opts };struct diff_provider_request req = { .repo = sb->repo, .xpp = &xpp }; +int provider_usable;if (!target->suspects)return ;/* nothing remains for this target */
@@ -1983,6 +2003,25 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb, d.ignore_diffs = ignore_diffs; d.dstq = &newdest; d.srcq = &target->suspects; +/* +* Offer the pair's identity only where blame's diff is the plain +* blob-pair diff the recording key describes; reverse blame, +* ignored revisions, and textconv paths withhold it and always +* compute. The working-tree/--contents pseudo-commit (marked by +* its null commit id) holds a blob that is not a stored object, +* so its pairs withhold identity too: no id may be sent that +* names bytes a provider cannot look up. +*/ + provider_usable = !sb->reverse && !ignore_diffs && + !is_null_oid (&target->commit->object.oid) && + !blame_textconv_active (sb, target->path) && + !blame_textconv_active (sb, parent->path); + +if (provider_usable) { + req.old_oid = &parent->blob_oid; + req.new_oid = &target->blob_oid; + } + req.diffopt = &sb->revs->diffopt;if (diff_provider_emit_hunks (&req, blame_diff_fill, &fill_data, blame_chunk_cb, &d) ==DIFF_PROVIDER_ERROR ) die("unable to generate diff (%s -> %s)",
builtin/blame.c +7 -1
@@ -15,6 +15,7 @@#include "hex.h" #include "commit.h" #include "diff.h" +#include "diff-hunks.h" #include "revision.h" #include "quote.h" #include "string-list.h"
@@ -1060,7 +1061,7 @@ int cmd_blame(int argc, parse_done:revision_opts_finish (&revs); no_whole_file_rename = !revs.diffopt.flags.follow_renames; - xdl_opts |= revs.diffopt.xdl_opts &XDF_INDENT_HEURISTIC ; + xdl_opts |= revs.diffopt.xdl_opts &DIFF_HUNKS_DEFAULT_XDL_OPTS ; revs.diffopt.flags.follow_renames =0 ; argc =parse_options_end (&ctx);
@@ -1315,9 +1316,14 @@ int cmd_blame(int argc,output (&sb, output_option);if (show_stats) { +unsigned long hunk_hits, hunk_misses; + +diff_hunks_read_stats (sb.repo, &hunk_hits, &hunk_misses);printf ("num read blob: %d\n" , sb.num_read_blob);printf ("num get patch: %d\n" , sb.num_get_patch);printf ("num commits: %d\n" , sb.num_commits); +printf ("num precomputed hits: %lu\n" , hunk_hits); +printf ("num precomputed misses: %lu\n" , hunk_misses); } cleanup:
diff.h +11 -0
@@ -231,6 +231,17 @@ static inline void diff_flags_or(struct diff_flags *a,#define DIFF_WITH_ALG (opts, flag) (((opts)->xdl_opts & ~XDF_DIFF_ALGORITHM_MASK) | XDF_##flag) +/* +* The xdl_opts bits git turns on by default that a from-scratch xdl_opts +* (git blame's own option parsing) does not set, and so must OR in to match +* a store warmed at the default diff settings; a diff_options-based consumer +* (diffstat) already has them in o->xdl_opts. Today this is only the indent +* heuristic. It does NOT cover a non-default diff.algorithm: a repo that +* configures one records under that algorithm, and a consumer keying without +* it misses (a lost hit, not wrong output). +*/ +#define DIFF_HUNKS_DEFAULT_XDL_OPTS XDF_INDENT_HEURISTIC +enum diff_words_type {DIFF_WORDS_NONE =0 ,DIFF_WORDS_PORCELAIN ,
t/meson.build +1 -0
@@ -1158,6 +1158,7 @@ benchmarks = [ 'perf/p4205-log-pretty-formats.sh', 'perf/p4209-pickaxe.sh', 'perf/p4211-line-log.sh', + 'perf/p4218-diff-hunks.sh', 'perf/p4220-log-grep-engines.sh', 'perf/p4221-log-grep-engines-fixed.sh', 'perf/p5302-pack-index.sh',
t/perf/p4218-diff-hunks.sh new file +48 -0
48 lines not on this page · raw message
t/t4220-diff-hunks.sh +355 -0
403 lines not on this page · raw message
Signature · 1 line
2.54.0
RFC PATCH v7 7/10 sub-process: separate process lifecycle from hashmap management
In reply to Michael Montalbosubprocess_start() and subprocess_stop() couple two concerns: managing a child process (setup, handshake, teardown) and managing a hashmap that indexes running processes by command string. The hashmap suits callers like convert.c where many files may share one filter process looked up by name, but callers that manage process membership under their own rules do not need the coupled operations.
Extract subprocess_start_command() and subprocess_stop_command() so callers can reuse the child process setup and handshake machinery without the map operations. subprocess_start() and subprocess_stop() become thin wrappers that add hashmap operations on top.
The diff process support added later in this series keeps its processes in a pool owned by a per-repository provider object, and an entry for a failed command must stay behind there so the command is not retried. That membership follows rules subprocess_start() and subprocess_stop() do not know. The pool therefore uses the _command variants for process lifecycle and manages its own map.
--- sub-process.c | 28 +++++++++++++++++++++++----- sub-process.h | 9 ++++++++- 2 files changed, 31 insertions(+), 6 deletions(-)
sub-process.c +23 -5
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)return (len <0 ) ? len :0 ; } -void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry) +void subprocess_stop_command (struct subprocess_entry *entry) {if (!entry)return ;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry) entry->process.clean_on_exit =0 ;kill (entry->process.pid,SIGTERM );finish_command (&entry->process); +} +void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry) +{ +if (!entry) +return ; + +subprocess_stop_command (entry);hashmap_remove (hashmap, &entry->ent,NULL ); }
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)finish_command (process); } -int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd, +int subprocess_start_command (struct subprocess_entry *entry,const char *cmd,subprocess_start_fn startfn) {int err;
@@ -96,15 +103,26 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, coreturn err; } -hashmap_entry_init (&entry->ent,strhash (cmd)); - err =startfn (entry);if (err) {error ("initialization for subprocess '%s' failed" , cmd); -subprocess_stop (hashmap, entry); +subprocess_stop_command (entry);return err; } +return 0 ; +} + +int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd, +subprocess_start_fn startfn) +{ +int err; + + err =subprocess_start_command (entry, cmd, startfn); +if (err) +return err; + +hashmap_entry_init (&entry->ent,strhash (cmd));hashmap_add (hashmap, &entry->ent);return 0 ; }
sub-process.h +8 -1
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,*/ typedef int (*subprocess_start_fn )(struct subprocess_entry *entry); -/* Start a subprocess and add it to the subprocess hashmap. */ +/* Start a subprocess and run the startfn (typically handshake). */ +int subprocess_start_command (struct subprocess_entry *entry,const char *cmd, +subprocess_start_fn startfn); + +/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */ int subprocess_start (struct hashmap *hashmap,struct subprocess_entry *entry,const char *cmd,subprocess_start_fn startfn); +/* Kill a subprocess. */ +void subprocess_stop_command (struct subprocess_entry *entry); +/* Kill a subprocess and remove it from the subprocess hashmap. */ void subprocess_stop (struct hashmap *hashmap,struct subprocess_entry *entry);
Signature · 1 line
2.54.0
RFC PATCH v7 8/10 sub-process: add a gentle status read
In reply to Michael Montalbosubprocess_read_status() reads "status=<key>" packets up to a flush with packet_read_line_gently(), which is gentle only about EOF. A malformed length header still dies inside pkt-line, and an empty packet is indistinguishable from the flush that ends the section. A protocol violation in a status section therefore either kills the whole command or silently truncates the section. That posture fits the filter protocol's callers, which treat their process as required infrastructure; the diff process consult added later in this series treats its process as optional, and any protocol error must degrade to the builtin diff rather than abort the command.
Add subprocess_read_status_gently(): the same status loop, reading through packet_read_with_status() with the gentle options, returning -1 on a truncated or malformed packet and on an empty packet where a status line or the terminating flush belongs. subprocess_read_status() and its callers are unchanged.
The handshake has its gentle counterpart in 061a68e443 (sub-process:
use gentle handshake to avoid die() on startup failure, 2026-06-01), which turned truncated handshake reads into error returns for every caller. This series' base includes that commit, so a process that dies during the handshake feeds the same non-fatal fallback as a status failure here, and an optional diff process degrades to the builtin diff on either kind of protocol error.
--- sub-process.c | 24 ++++++++++++++++++++++++ sub-process.h | 10 ++++++++++ 2 files changed, 34 insertions(+)
sub-process.c +24 -0
@@ -49,6 +49,30 @@ int subprocess_read_status(int fd, struct strbuf *status)return (len <0 ) ? len :0 ; } +int subprocess_read_status_gently (int fd,struct strbuf *status) +{ +for (;;) { +int pktlen =-1 ; +enum packet_read_status rs; +const char *value; + + rs =packet_read_with_status (fd,NULL ,NULL , packet_buffer, +sizeof (packet_buffer), &pktlen, +PACKET_READ_CHOMP_NEWLINE | +PACKET_READ_GENTLE_ON_EOF | +PACKET_READ_GENTLE_ON_READ_ERROR ); +if (rs ==PACKET_READ_FLUSH ) +return 0 ; +if (rs !=PACKET_READ_NORMAL || !pktlen) +return -1 ; +if (skip_prefix (packet_buffer,"status=" , &value)) { +/* the last "status=<foo>" line wins */ +strbuf_reset (status); +strbuf_addstr (status, value); + } + } +} +void subprocess_stop_command (struct subprocess_entry *entry) {if (!entry)
sub-process.h +10 -0
@@ -101,4 +101,14 @@ int subprocess_handshake(struct subprocess_entry *entry,int subprocess_read_status (int fd,struct strbuf *status); +/* +* Like subprocess_read_status(), but a malformed status section fails +* instead of dying: a truncated or malformed packet, and an empty +* packet where a status line or the terminating flush belongs, return +* -1 and leave the stream unusable. subprocess_read_status() cannot +* tell an empty packet from the flush that ends the section, and dies +* on a framing error inside packet_read_line_gently(). +*/ +int subprocess_read_status_gently (int fd,struct strbuf *status); +#endif
Signature · 1 line
2.54.0
RFC PATCH v7 9/10 userdiff: add diff.<driver>.process config
In reply to Michael MontalboAdd the process field to struct userdiff_driver and teach the config parser to populate it from diff.<driver>.process.
The field names a long-running hunk provider process. Nothing reads it yet: the consult, the protocol, and the documentation arrive with the next commit, which starts and pools processes keyed by this field's command string.
--- userdiff.c | 7 +++++++ userdiff.h | 2 ++ 2 files changed, 9 insertions(+)
userdiff.c +7 -0
@@ -509,6 +509,13 @@ int userdiff_config(const char *k, const char *v) drv->algorithm = drv->algorithm_owned;return ret; } +if (!strcmp (type,"process" )) { +int ret; +FREE_AND_NULL (drv->process_owned); + ret =git_config_string (&drv->process_owned, k, v); + drv->process = drv->process_owned; +return ret; + }return 0 ; }
userdiff.h +2 -0
@@ -31,6 +31,8 @@ struct userdiff_driver {char *textconv_owned;struct notes_cache *textconv_cache;int textconv_want_cache; +const char *process; +char *process_owned; };enum userdiff_driver_type {USERDIFF_DRIVER_TYPE_BUILTIN =1 <<0 ,
Signature · 1 line
2.54.0
RFC PATCH v7 10/10 diff: consult oid-only hunk providers via diff.<driver>.process
In reply to Michael MontalboThe provider chain so far holds the diff-hunks store in front of the terminal builtin computation. Open it to external processes: a pair on a path whose driver configures diff.<driver>.process is answered by a long-running process speaking a pkt-line protocol (following the filter process protocol), registered at the head of the chain and consulted before the store and before any blob is loaded.
The protocol starts with the smallest request that can carry an answer:
object names alone. A request is the pathname and the pair's old-oid/new-oid, with no content. The process answers with hunk lines, with a zero-hunk success that asserts the blobs equivalent (trailing newlines included), or with status=need-content, on which the pair falls through to the builtin answer. This serves the two shapes that need no content pushed to them: a cache keyed on the blob pair, and a process that fetches the blobs itself (for example over "git cat-file --batch"). A pair whose side is not a stored blob carries a NULL id; the provider sends no request and passes it. Because Git holds no content for the exchange, the answer is used as sent: hunks are validated for order, overlap, lockstep alignment, and magnitude, then replayed without the normalization xdiff applies to diffs it computes itself. The magnitude bound is the blobs' sizes, read from the object database without loading content: a blob of N bytes holds at most N lines.
Because the process's answer is authoritative, it outranks the store, and its head-of-chain position says so. A pair the process answers never reaches the store and is never recorded, so nothing it produces enters the store, which holds the builtin answer only. A request it does not answer, whether need-content, a missing capability, or a missing id, passes down the chain to the builtin answer, which is what the store serves, so the store may serve such a pair and a warming run may record it. Entries recorded before a process was configured are not purged; a pair the process answers ignores them, and "git diff-hunks clear" discards them.
The provider gates itself per request. The driver is looked up by the old-side path, so a renamed file resolves to the same driver, and by the repository-relative path, so a diff.relative run from a subdirectory names the pair the same way. Options the process is never told about select no process: the whitespace-ignoring options, -I, --anchored, and an algorithm forced by option or configuration (blame routes its algorithm through xdl_opts, so --histogram is covered). The request gains its last field, the path; the consumers change only by filling it, and neither names the process.
The provider's state is its repository's pool of running processes, keyed by the configured command, so drivers sharing a command share a process, a submodule speaks to its own, and releasing the provider (from repo_clear()) stops them. The pool owns a copy of each command string, so an entry outlives a config re-read. A command that fails stays as an entry that is not retried: its request and every later one pass, so the store may serve the path for the rest of the command.
A protocol error in a response never kills the command. The response is read through a packet reader gentle about framing, so an error takes one path: a single warning, the process stopped and marked failed, and the builtin diff for the rest of the command. That covers garbage bytes, a truncated response, an empty packet, a bare status, and an unrecognized status. Semantically invalid coordinates cost only their
pair: the response is drained, the pair is computed, and the process stays alive. A path the protocol cannot carry (an embedded newline, or one too long for a packet) falls back per path rather than costing the command its process. The handshake keeps one fatal check: a process that announces a capability Git did not request aborts the command, as the long-running filter protocol does.
Consulting is allowed per command, following the allow_textconv precedent. "git diff", "git log" and "git show", and "git blame" set allow_diff_process; the plumbing diff commands and the interactive-patch machinery never set it, so scripted and staging output stays builtin. The options adjust the flag:
- --no-ext-diff clears it and --ext-diff sets it;
- --diff-process and --no-diff-process set and clear it alone, leaving external diff drivers as they were;
- format-patch clears it unconditionally, so a generated patch applies for recipients without the process;
- range-diff passes --no-ext-diff to the "git log" it compares.
git blame and the summary formats consult the process. For blame, a pair reported equivalent emits no hunks, so the whole commit passes to its parent. In the stat formats such a pair sums to a zero-count entry, which the "nothing changed" rule omits, as under -w. The subprocess is
long-running: one startup cost across a traversal, one round-trip per consulted pair. Answers travel in struct xdl_hunk, new in xdiff-interface.h, holding xdiff's 1-based coordinates; nothing feeds them back to xdiff, since only coordinate consumers consult.
A content-carrying request is the natural extension: it would serve sides that are not stored blobs and processes that want content pushed to them, and bring patch output and log -L's range tracking to the same answer. As it stands, a process's answers show in blame and the summary formats while patch output stays builtin.
t4080 exercises the protocol, the per-command gate, and the error paths:
- each adversarial response shape warns and falls back to builtin, the request log proving which failures disable the process and which keep it alive (a malformed hunk line, coordinates past the blob size, a count overflowing strtol(), overlapping or misaligned hunks, an unrecognized status, a bare status, an empty packet, a mid-response crash, and raw garbage);
- a capability-less process and status=abort degrade without noise, and a failed start warns once and returns the path to the store;
- a trailing token on a hunk line is ignored, pinning field appendability;
- positive consults for git diff, git show, and diff-tree under --ext-diff and --diff-process; textconv output and gitlink sides are never identified; a diff.relative run consults by the repo-relative path;
- the equivalence answer is pinned from both consumers, and a warming run past a deferring process records the pair for a later read.
--- Documentation/config/diff.adoc | 6 + Documentation/diff-algorithm-option.adoc | 3 + Documentation/diff-options.adoc | 15 +- Documentation/gitattributes.adoc | 160 ++++++ Makefile | 2 + blame.c | 10 +- builtin/blame.c | 1 + builtin/diff.c | 1 + builtin/log.c | 11 +- diff-process.c | 669 +++++++++++++++++++++++ diff-provider-internal.h | 1 + diff-provider.c | 12 +- diff-provider.h | 39 +- diff.c | 42 +- diff.h | 9 + meson.build | 1 + range-diff.c | 6 + t/helper/meson.build | 1 + t/helper/test-diff-process-backend.c | 349 ++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/meson.build | 1 + t/t4080-diff-process.sh | 593 ++++++++++++++++++++ xdiff-interface.h | 12 + 24 files changed, 1916 insertions(+), 30 deletions(-) create mode 100644 diff-process.c create mode 100644 t/helper/test-diff-process-backend.c create mode 100755 t/t4080-diff-process.sh
Documentation/config/diff.adoc +6 -0
@@ -218,6 +218,12 @@ endif::git-diff[] Set this option to `true` to make the diff driver cache the text conversion outputs. See linkgit:gitattributes[5] for details. +`diff.<driver>.process`:: + The command to run as a long-running process that answers + which line ranges changed between two blobs. See + linkgit:gitattributes[5] for the protocol and when it is + consulted. + `diff.indentHeuristic`:: Set this option to `false` to disable the default heuristics that shift diff hunk boundaries to make patches easier to read.
Documentation/diff-algorithm-option.adoc +3 -0
@@ -18,3 +18,6 @@ For instance, if you configured the `diff.algorithm` variable to a non-default value and want to use the default one, then you have to use `--diff-algorithm=default` option. ++ +Explicitly choosing a diff algorithm on the command line also +bypasses `diff.<driver>.process` (see linkgit:gitattributes[5]).
Documentation/diff-options.adoc +14 -1
@@ -833,7 +833,20 @@ endif::git-format-patch[] to use this option with linkgit:git-log[1] and friends. `--no-ext-diff`:: - Disallow external diff drivers. + Disallow external diff drivers and processes, including + `diff.<driver>.command` and `diff.<driver>.process` + (see linkgit:gitattributes[5]). + +`--diff-process`:: +`--no-diff-process`:: + Allow (or forbid) consulting a diff process configured with + ++diff.++__<driver>__++.process++ (see linkgit:gitattributes[5]), + leaving external diff drivers unaffected. `git diff`, `git log`, + `git show`, and `git blame` allow consulting by default; the + plumbing diff commands forbid it unless this option or + `--ext-diff` is given. linkgit:git-format-patch[1] accepts the + option but ignores it: a generated patch is always based on the + builtin diff. `--textconv`:: `--no-textconv`::
Documentation/gitattributes.adoc +160 -0
166 lines not on this page · raw message
Makefile +2 -0
@@ -819,6 +819,7 @@ TEST_BUILTINS_OBJS += test-csprng.o TEST_BUILTINS_OBJS += test-date.o TEST_BUILTINS_OBJS += test-delete-gpgsig.o TEST_BUILTINS_OBJS += test-delta.o +TEST_BUILTINS_OBJS += test-diff-process-backend.o TEST_BUILTINS_OBJS += test-dir-iterator.o TEST_BUILTINS_OBJS += test-drop-caches.o TEST_BUILTINS_OBJS += test-dump-cache-tree.o
@@ -1151,6 +1152,7 @@ LIB_OBJS += diff-delta.o LIB_OBJS += diff-merges.o LIB_OBJS += diff-lib.o LIB_OBJS += diff-no-index.o +LIB_OBJS += diff-process.o LIB_OBJS += diff-provider.o LIB_OBJS += diff.o LIB_OBJS += diffcore-break.o
blame.c +9 -1
@@ -2010,17 +2010,25 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,* compute. The working-tree/--contents pseudo-commit (marked by * its null commit id) holds a blob that is not a stored object, * so its pairs withhold identity too: no id may be sent that -* names bytes a provider cannot look up. +* names bytes a process cannot look up. */ provider_usable = !sb->reverse && !ignore_diffs && !is_null_oid (&target->commit->object.oid) && !blame_textconv_active (sb, target->path) && !blame_textconv_active (sb, parent->path); +/* +* Look up the driver by the parent (old) path, as builtin_diff() +* does with name_a, so a renamed file resolves to the same driver +* across diff and blame. A process that reports a pair +* equivalent emits no hunks, so blame passes the whole commit +* through and looks past it. +*/ if (provider_usable) { req.old_oid = &parent->blob_oid; req.new_oid = &target->blob_oid; } + req.path = parent->path; req.diffopt = &sb->revs->diffopt;if (diff_provider_emit_hunks (&req, blame_diff_fill, &fill_data, blame_chunk_cb, &d) ==DIFF_PROVIDER_ERROR )
builtin/blame.c +1 -0
@@ -1025,6 +1025,7 @@ int cmd_blame(int argc,repo_init_revisions (the_repository, &revs,NULL ); revs.date_mode = blame_date_mode; revs.diffopt.flags.allow_textconv =1 ; + revs.diffopt.flags.allow_diff_process =1 ; revs.diffopt.flags.follow_renames =1 ; save_commit_buffer =0 ;
builtin/diff.c +1 -0
@@ -510,6 +510,7 @@ int cmd_diff(int argc,init_diffstat_widths (&rev.diffopt); rev.diffopt.flags.allow_external =1 ; rev.diffopt.flags.allow_textconv =1 ; + rev.diffopt.flags.allow_diff_process =1 ;/* If this is a no-index diff, just run it and exit there. */ if (no_index)
builtin/log.c +8 -3
@@ -209,6 +209,7 @@ static void cmd_log_init_defaults(struct rev_info *rev,init_diffstat_widths (&rev->diffopt); rev->diffopt.flags.recursive =1 ; rev->diffopt.flags.allow_textconv =1 ; + rev->diffopt.flags.allow_diff_process =1 ; rev->abbrev_commit = cfg->default_abbrev_commit; rev->show_root_diff = cfg->default_show_root; rev->subject_prefix = cfg->fmt_patch_subject_prefix;
@@ -2226,11 +2227,15 @@ int cmd_format_patch(int argc,die (_ ("unrecognized argument: %s" ), argv[1 ]);/* -* A patch generated by format-patch carries the builtin diffstat, -* not one served from a local store, so its counts do not depend -* on whether the sender warmed the store. +* Patches generated by format-patch must be based on the builtin +* diff so recipients without the store or the process can apply +* them, and so the emitted diffstat does not depend on the sender's +* local cache: the precomputed-hunks store is not consulted for the +* diffstat, and the diff process is not consulted even when +* --ext-diff enables the external diff command. */ rev.diffopt.flags.no_precomputed_hunks =1 ; + rev.diffopt.flags.allow_diff_process =0 ;if (rev.diffopt.output_format &DIFF_FORMAT_NAME )die (_ ("--name-only does not make sense" ));
diff-process.c new file +669 -0
669 lines not on this page · raw message
diff-provider-internal.h +1 -0
@@ -93,6 +93,7 @@ struct diff_provider {* diff-provider.c holds itself. Each call returns a fresh provider * for one repository's chain. */ +struct diff_provider *diff_process_provider_new (void );struct diff_provider *diff_hunks_store_provider_new (void );/*
diff-provider.c +7 -5
@@ -41,11 +41,11 @@ static struct diff_provider *builtin_provider_new(void)/* * The repository's chain, assembled on first walk. The composition -* is fixed, and the order is the authority resolution: the store is -* consulted before the builtin computation, the terminal provider, -* so the chain always ends in an implementor that can answer. -* Nothing is decided per repository here; each provider gates itself -* per request. +* is fixed, and the order is the authority resolution: the process +* outranks the store, and the builtin computation is the terminal +* provider, so the chain always ends in an implementor that can +* answer. Nothing is decided per repository here; each provider +* gates itself per request. */ static struct diff_provider *provider_chain (struct repository *r) {
@@ -53,6 +53,8 @@ static struct diff_provider *provider_chain(struct repository *r)if (*tail)return *tail; + *tail =diff_process_provider_new (); + tail = &(*tail)->next; *tail =diff_hunks_store_provider_new (); tail = &(*tail)->next; *tail =builtin_provider_new ();
diff-provider.h +23 -16
@@ -11,19 +11,23 @@* its content is loaded. * * A hunk provider answers a consumer's request from the pair's -* identity (its blob object ids) and the parameters that determine -* the diff; a request no provider answers falls through to the -* consumer's own computation. A provider is either authoritative for -* its requests, meaning its answer may deliberately differ from the -* builtin diff, or not, meaning its answer must reproduce the builtin -* result exactly. The interface resolves that authority through a -* provider chain owned by the repository, built on first consultation -* and released by repo_clear(): chain order is the resolution, and -* the builtin computation itself is the chain's terminal provider. A -* consumer never names a provider; it reads the outcome below. -* Every answer a provider serves from identity passes the shared -* coordinate check (diff-provider-internal.h) before any consumer -* sees it. +* identity, its blob object ids and the settings that determine the +* diff, before any content is loaded; a request no provider answers +* falls through to the consumer's own computation. Two providers implement this +* interface with different authority. The diff-hunks store +* (diff-hunks.h) is in-process and not authoritative: it may only +* reproduce the builtin result, so it never asserts a pair +* equivalent, and it stands aside wherever a process outranks it. A +* process configured in diff.<driver>.process (diff-process.c) is +* authoritative for its paths: its answer may deliberately differ +* from the builtin diff, including asserting a pair equivalent. The +* interface resolves that authority through a provider chain owned +* by the repository, built on first consultation and released by +* repo_clear(): chain order is the resolution, and the builtin +* computation itself is the chain's terminal provider. A consumer +* never names a provider; it reads the outcome below. Every answer a +* provider serves from identity passes the shared coordinate check +* (diff-provider-internal.h) before any consumer sees it. */ struct diff_options ;
@@ -95,14 +99,17 @@ enum diff_provider_outcome {* name the blobs whose bytes are diffed; pass NULL for a side whose * bytes are not a stored blob (a working-tree file, textconv output, * a gitlink), so no provider answers from an id it cannot look up. -* diffopt carries the diff settings that live outside xpp; xpp -* carries the parameters the diff runs with. Each provider gates -* itself on the fields that concern it. +* path names the file the pair is diffed as; a provider selected by +* path applies only where it is set. diffopt carries the diff +* settings that live outside xpp; xpp carries the parameters the +* diff runs with. Each provider gates itself on the fields that +* concern it. */ struct diff_provider_request {struct repository *repo;const struct object_id *old_oid;const struct object_id *new_oid; +const char *path;struct diff_options *diffopt;const xpparam_t *xpp; };
diff.c +38 -4
70 lines not on this page · raw message
diff.h +9 -0
@@ -173,6 +173,15 @@ struct diff_flags {*/ unsigned allow_external; +/** +* Allows diff.<driver>.process to be consulted. Set by the +* porcelain commands whose output may reflect a diff process +* (diff, log, show, blame) and by --ext-diff or --diff-process; +* plumbing does not set it by default, so its output stays +* builtin. Cleared by --no-ext-diff or --no-diff-process. +*/ +unsigned allow_diff_process; +/** * For communication between the calling program and the options parser; * tell the calling program to signal the presence of difference using
meson.build +1 -0
@@ -356,6 +356,7 @@ libgit_sources = [ 'diff-merges.c', 'diff-lib.c', 'diff-no-index.c', + 'diff-process.c', 'diff-provider.c', 'diff.c', 'diffcore-break.c',
range-diff.c +6 -0
@@ -52,6 +52,12 @@ static int read_patches(const char *range, struct string_list *list,int ret =-1 ; strvec_pushl(&cp.args, "log", "--no-color", "-p", + /* + * The patches being compared must be the builtin + * diff's: an external diff command or diff process + * could change either side of the comparison. + */ + "--no-ext-diff", "--reverse", "--date-order", "--decorate=no", "--no-prefix", "--submodule=short",/*
t/helper/meson.build +1 -0
@@ -12,6 +12,7 @@ test_tool_sources = [ 'test-date.c', 'test-delete-gpgsig.c', 'test-delta.c', + 'test-diff-process-backend.c', 'test-dir-iterator.c', 'test-drop-caches.c', 'test-dump-cache-tree.c',
t/helper/test-diff-process-backend.c new file +349 -0
349 lines not on this page · raw message
t/helper/test-tool.c +1 -0
@@ -22,6 +22,7 @@ static struct test_cmd cmds[] = { { "date", cmd__date }, { "delete-gpgsig", cmd__delete_gpgsig }, { "delta", cmd__delta }, + { "diff-process-backend", cmd__diff_process_backend }, { "dir-iterator", cmd__dir_iterator }, { "drop-caches", cmd__drop_caches }, { "dump-cache-tree", cmd__dump_cache_tree },
t/helper/test-tool.h +1 -0
@@ -15,6 +15,7 @@ int cmd__csprng(int argc, const char **argv);int cmd__date (int argc,const char **argv);int cmd__delta (int argc,const char **argv);int cmd__delete_gpgsig (int argc,const char **argv); +int cmd__diff_process_backend (int argc,const char **argv);int cmd__dir_iterator (int argc,const char **argv);int cmd__drop_caches (int argc,const char **argv);int cmd__dump_cache_tree (int argc,const char **argv);
t/meson.build +1 -0
@@ -518,6 +518,7 @@ integration_tests = [ 't4072-diff-max-depth.sh', 't4073-diff-stat-name-width.sh', 't4074-diff-shifted-matched-group.sh', + 't4080-diff-process.sh', 't4100-apply-stat.sh', 't4101-apply-nonl.sh', 't4102-apply-rename.sh',
t/t4080-diff-process.sh new file +593 -0
593 lines not on this page · raw message
xdiff-interface.h +12 -0
@@ -6,6 +6,18 @@struct object_database ; +/* +* Hunk descriptor for externally computed diffs, in xdiff's own +* coordinates: line numbers are 1-based and a hunk's start is the +* first line it covers. A caller translates any external "empty side" +* idiom (such as git diff's start-0/count-0) to a 1-based start before +* storing hunks in this struct. +*/ +struct xdl_hunk { +long old_start, old_count; +long new_start, new_count; +}; +/* * xdiff isn't equipped to handle content over a gigabyte; * we make the cutoff 1GB - 1MB to give some breathing
Signature · 1 line
2.54.0
Re: [RFC PATCH v7 10/10] diff: consult oid-only hunk providers via diff.<driver>.process
Quoted 4 lines · Michael Montalbo
(quoted diff) +3
-0+static long blob_line_cap(struct repository *r, const struct object_id *oid) +{ + unsigned long size;
Review
Quoted 10 lines
(quoted diff) +10
-0+ struct object_info oi = OBJECT_INFO_INIT; + + oi.sizep = &size; + if (odb_read_object_info_extended(r->objects, oid, &oi, + OBJECT_INFO_SKIP_FETCH_OBJECT) < 0) + return -1; + if (size > INT32_MAX) + return INT32_MAX; + return (long)size; +}
"unsigned long" for size breaks the build wherever size_t differs from unsigned long (32-bit platforms and 64-bit Windows):
diff-process.c:331:18: assignment to 'size_t *' {aka 'long long
unsigned int *'} from incompatible pointer type 'long unsigned int *' [-Wincompatible-pointer-types]
331 | oi.sizep = &size;
I'll fold this into a re-roll; in the meantime, squashing the following into "diff: consult oid-only hunk providers via diff.<driver>.process" on mm/diff-process-hunks resolves it.
-- >8 --
diff-process.c +0 -0
@@ -325,7 +325,7 @@ static int validate_external_hunks(const struct
xdl_hunk *hunks, size_t nr,
*/
static long blob_line_cap(struct repository *r, const struct object_id *oid) {
- unsigned long size;
+ size_t size;
struct object_info oi = OBJECT_INFO_INIT;
oi.sizep = &size;Re: [RFC PATCH v7 10/10] diff: consult oid-only hunk providers via diff.<driver>.process
In reply to Michael MontalboReview
Quoted 6 lines · Michael Montalbo
(quoted diff) +6
-0+static long blob_line_cap(struct repository *r, const struct object_id *oid) +{ + unsigned long size; + struct object_info oi = OBJECT_INFO_INIT; + + oi.sizep = &size;
This breaks the build on platforms whose ulong is narrower than size_t since June 21 due to c6a4629e32 (odb: use size_t for object_info.sizep and the size APIs, 2026-06-15), which changed the type of the '.sizep' member of 'struct object_info' from 'ulong' to 'size_t'.
Tentatively, I'll pile this on top of your series before rebuilding 'seen'; I do not know if there are further fallouts, though.
diff-process.c +1 -1
@@ -325,7 +325,7 @@ static int validate_external_hunks(const struct xdl_hunk *hunks, size_t nr,*/ static long blob_line_cap (struct repository *r,const struct object_id *oid) { -unsigned long size; +size_t size;struct object_info oi =OBJECT_INFO_INIT ; oi.sizep = &size;
Re: [RFC PATCH v7 0/10] diff: add provider interface and initial providers
In reply to Michael MontalboHi Michael
Quoted 13 lines · Michael Montalbo
Every in-process diff in Git reduces, at one point, to a single
question: given two blobs and the settings the diff runs under, which line ranges changed? The answer is the diff's hunks: for each change, the position and length of the range on the old side and on the new. Each consumer asks in its own shape:- blame diffs each suspect's blob against its parent's, taking only the coordinates through xdiff's hunk callback;
- the stat formats keep only the added and deleted counts;
- patch output emits from the hunks, with xdiff interleaving context and content around them;
- log -L maps the tracked range across each commit from the coordinates.
Quoted 8 lines
In every case the answer is computed the same way: load both blobs and run xdiff. That is the only source, so nothing that already holds the answer, or that would answer differently on purpose, can supply it instead. Sometimes that is what we want, which is why patch-id and format-patch stay on the builtin computation throughout: patch-id needs identical hashes on every machine, and a format-patch must apply for recipients who share none of the sender's configuration. Other times another source would be useful.
This explains the mechanics of the existing implementation, but doesn't really explain what the advantage of the change is beyond having a unified interface. What are things that are unlocked by this change? Are we doing it purely to cache the hunk headers or does it (as I suspect) unlock other features?
Quoted 9 lines
This RFC sketches a direction. The unified series shows one interface carrying two example providers and their interaction; it is not shaped to merge as one topic. If the direction holds, the work returns as separate reviewable series (see Roadmap). The two examples are demonstrations, each an RFC on its own: diff.<driver>.process, the RFC cooking as
mm/diff-process-hunks, lets a configured external process answer with its own notion of which lines changed, and the diff-hunks store, new in this thread, remembers what xdiff computed and serves it back. One is authoritative and external, one a cache and in-process.
What does it mean to be authoritative? Why isn't the xdiff code authoritative?
Quoted 12 lines
Three pieces:
- A hunk provider interface (
diff-provider.h) is the point of the series. A provider is an alternate source for the answer: asked with the pair's object ids and the diff settings, before any blob is loaded, it may supply the hunks in place of the builtin computation. A miss falls through to that computation, and every answer passes one shared validity check first. The providers form a chain the repository owns, built on first consultation and released inrepo_clear(), so provider state such as a running process never outlives its repository.
Quoted 1 line
Chain order is the authority,
I'm not sure what that phrase means
Quoted 10 lines
and the
terminal provider is the builtin computation itself, so the interface never exists without an implementor: patch 02 ships it answering every request the way the consumers did before. A consumer states its request in one struct and reads one set of outcomes (answered,
unanswered, or failed); it never names a provider, and a provider> added later maps onto those outcomes inside the interface, so consumer
code is written once. Because every diff now walks the chain even with no store or process configured, the default path was measured against the pre-series base and runs within noise (a 5000-commit log --stat and a long-history blame, ratio 1.00 either way).
So we can configure a provider for a particular file type via .gitattributes and diff.<driver>.process and then it is used automatically, if there is no provider configured we use xdiff?
Quoted 8 lines
- The diff-hunks store shows the non-authoritative side: an in-process cache at $GIT_DIR/objects/info/diff-hunks that may only reproduce the builtin diff, so serving from it never changes a command's output. It is read by default and written only when a repository owner opts in, warming it as a side effect of diff work the command already does:
GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null
Quoted 8 lines
A warmed store then serves the stat formats and blame from stored coordinates instead of a fresh diff: on git.git a 5000-commit log --stat runs about 1.9x faster, and blame reads the same entries opportunistically (full numbers in [1]). Its format and keying, what it may not serve, and how it handles corruption and staleness are in git-diff-hunks(1), gitformat-diff-hunks(5), and [2]. The interface point is small: a cache drops in as the provider that stands aside wherever an authoritative one answers.
I had a very quick look at the documentation in patch that implements this. Am I right in thinking it stores xdl_opts directly so that we cannot change the in-memory representation without breaking the cache? Are there plans to garbage collect cache entries when the corresponding blobs are removed?
Quoted 15 lines
- diff.<driver>.process shows the authoritative side: an external process, configured per driver, whose answers may deliberately differ from the builtin diff and outrank the store. Git asks it for a pair by object names alone, so it answers before any blob is read, which suits a cache or a process that fetches the blobs itself. Consulting is opt-in per command, following the
allow_textconvprecedent, and a pair the process cannot answer falls back to the builtin diff. The protocol, the per-command gate, how failures are handled, and the versioning that lets it grow are in gitattributes(5) and footnotes [3] and [4]. The interface point, again, is small: an external, authoritative provider joins the same chain ahead of the cache, and neither consumer learns it is there. A later content-carrying request would extend it to the pairs and consumers this identity-only form leaves on the builtin diff.
Quoted 1 line
The series stops at the coordinates.
Does this mean the offsets and lengths in the hunk header? I can't see any reference to "coordinates" in the existing code.
Quoted 6 lines
A consumer that needs the changed
text, such as patch output, would have only its hunk selection replaced, with xdiff still emitting content from the blobs; that machinery is the content enrichment sketched in the Roadmap. Establishing the framework on coordinates first keeps this series one design: the question, the interface, and two providers answering by identity.
This seems like an interesting proposal and the diff headers seem like a good place for initial phase to stop. It would be helpful to have a clearer explanation of the features this interface would support (i.e. what's the motivation for these changes) and a lot less jargon in the interface description.
Thanks
Phillip
Quoted 231 lines
Shape of the series:
01 documentation: how external diff drivers relate to the
features layered on the diff
02 the provider interface: the request and outcome types, the
emit entry point, the shared validity check, and the repository-owned chain with its terminal builtin provider
03 the store: on-disk format, library, and the diff-hunks command
04 recording: the stat walk computes, sums, and records
trim-stable pairs (writes gated off by default)
05 reading: the consult entry point and the store's registration
as a provider; the request gains the object ids and diff options
06 blame reading through the interface's emit path
07-09 process preparation: sub-process lifecycle split, a gentle
status read for an optional process, and the diff.<driver>.process config
10 the process provider, oid-only, at the head of the chain, with
the per-command gate; the request gains the pathRoadmap:
This RFC asks whether the direction is right, not for these ten patches to merge as one topic. If it holds, the work returns in reviewable pieces:
- the interface and the store (patches 01 through 06): a cache with measured numbers and no external-process machinery
- the process provider (patches 07 through 10) on the same interface
- the content enrichment (the content-carrying request, patch output and log -L consulting, and the xdiff machinery that feeds a provider's hunks into emission) once the identity-keyed framework settles.
Several design questions are left for those series.
mm/diff-process-hunksin seen would be dropped in favor of this thread and its split.The series applies on the line-log topic (
mm/line-log-limited-ops) rebased onto current master. The topic rewrites the samebuiltin_diffstat()region this series touches, and current master includes061a68e443(sub-process: use gentle handshake to avoiddie()on startup failure), which this topic leans on: a process that dies during the handshake degrades to the builtin diff like every other failure. A trial merge against seen shows no interaction with other topics beyond themm/diff-process-hunksreplacement above.The base (line-log topic on current master) and the full series are available at:
git fetch https://github.com/mmontalbo/git
mm/line-log-stat-formats-followupgit fetch https://github.com/mmontalbo/gitmm/hunk-providers-oid-firstChanges since v6:
This is a restructuring, not an incremental reroll, so a range-diff against v6 is unreadable; the map of what changed:
- The series now leads with the hunk provider interface and brings the diff-hunks store in as its in-process implementation (patches 02 through 06, new to this thread). It keeps only the identity-keyed half of the external diff process protocol from
mm/diff-process-hunks.
- v6's gitattributes documentation, sub-process split, and userdiff config return close to their v6 form as patches 01, 07, and 09. Patch 08 is new: a gentle status read so a protocol error in an optional process degrades to the builtin diff instead of dying.
- v6's protocol patch returns as patch 10, reduced to the oid-only request, consulting through the interface, and carrying a per-command gate (v6's bypass patch folds into it).
- v6's blame and stat consults return as identity-keyed consults (patches 05, 06, and 10); their content legs, along with v6's xdiff external-hunks machinery, content-carrying request, and line-log consult, are withheld for the content enrichment.Footnotes:
[1] Store numbers, measured with hyperfine against the same build with
core.diffHunks=false. The warm is a full cold build of the store; the blame speedup is file-dependent (see the coverage limitation): git.git (82,912 commits, --all) warm log --all --stat 20.9 s store 28 MB, verify 39 ms log --stat -5000 1.91x (1.38 s -> 0.72 s) blame diff.c 1.26x (509 ms -> 403 ms) blame hit rate 54% (896 of 1653 pairs) linux (1,445,548 commits, --all) warm log --all --stat 714 s store 298 MB, verify 415 ms log --stat -5000 1.43x (2.29 s -> 1.60 s) blame kernel/sched/core.c 1.43x (1.59 s -> 1.11 s) blame hit rate 74% (2414 of 3263 pairs)[2] The store is its own file because nothing existing is addressed by a blob pair: notes attach to single objects, commit-graph chunks to commits. One entry per pair, keyed by (old blob, new blob,
xdl_opts) and recorded only when the pair's trimmed and untrimmed diffs agree, serves blame at zero context and the stat formats at any -U (divergent pairs are 0.4-0.5% of a warm and always compute). The writer fsyncs and commits atomically, and a reader bounds-checks every record and treats an unparsable file as absent; the trailing checksum is checked by git diff-hunks verify, not on every read, the same read-time trust the commit-graph and multi-pack-index take. There is deliberately no fsck integration, expiry, or background
maintenance: the store is derivable at any time, so the recovery path is git diff-hunks clear and a re-warm. New commits make it incomplete, not wrong; a later warm seeds from the file and pays only for what is new.[3] Consulting the process is allowed per command, like textconv: git diff, git log and git show, and git blame consult it; the plumbing diff commands do not unless --ext-diff or --diff-process is given, and the interactive-patch machinery, format-patch, and range-diff stay builtin. Options the process is never told about select no process, and an object id is sent only when it names the exact bytes diffed (a pair under an active object replacement is not sent). The command comes from local configuration, as with filter.<name> .process: attributes select only a driver name, so cloning cannot cause a process to run. gitattributes(5) has the full gate.
[4] The protocol is versioned and capability-negotiated, and extends without breaking deployed processes: a process ignores request keys it does not know, Git ignores trailing tokens on a hunk line so fields can be appended, and new request forms arrive as capabilities a process may decline. Announcing a capability Git did not request aborts the command, the filter protocol's handshake rule. The content-carrying request is the natural first extension; markers for formatting-only changes and function or token boundaries are candidates beyond it.
Michael Montalbo (10):
gitattributes: document how external diff drivers relate to diff
features
diff: introduce a hunk provider interface
diff-hunks: add the store format, library, and command
diff: record precomputed hunks during stat output
diff: read precomputed hunks for stat output
blame: read precomputed hunks
sub-process: separate process lifecycle from hashmap management
sub-process: add a gentle status read
userdiff: add diff.<driver>.process config
diff: consult oid-only hunk providers via diff.<driver>.process.gitignore | 1 + Documentation/Makefile | 1 + Documentation/config.adoc | 2 + Documentation/config/core.adoc | 10 +- Documentation/config/diff-hunks.adoc | 8 + Documentation/config/diff.adoc | 6 + Documentation/diff-algorithm-option.adoc | 3 + Documentation/diff-options.adoc | 15 +- Documentation/git-diff-hunks.adoc | 146 +++ Documentation/gitattributes.adoc | 171 ++++ Documentation/gitformat-diff-hunks.adoc | 129 +++ Documentation/meson.build | 2 + Makefile | 5 + blame.c | 81 +- builtin.h | 1 + builtin/blame.c | 9 +- builtin/diff-hunks.c | 53 ++ builtin/diff-tree.c | 3 + builtin/diff.c | 11 + builtin/log.c | 19 + chunk-format.c | 62 +- chunk-format.h | 14 + command-list.txt | 2 + diff-hunks.c | 1034 ++++++++++++++++++++++ diff-hunks.h | 141 +++ diff-process.c | 669 ++++++++++++++ diff-provider-internal.h | 130 +++ diff-provider.c | 190 ++++ diff-provider.h | 159 ++++ diff.c | 294 +++++- diff.h | 47 + environment.c | 1 + git.c | 1 + meson.build | 4 + odb.c | 2 + odb.h | 4 + range-diff.c | 6 + repo-settings.c | 1 + repo-settings.h | 1 + repository.c | 3 + repository.h | 8 + sub-process.c | 52 +- sub-process.h | 19 +- t/helper/meson.build | 1 + t/helper/test-diff-process-backend.c | 349 ++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/meson.build | 3 + t/perf/p4218-diff-hunks.sh | 48 + t/t4080-diff-process.sh | 593 +++++++++++++ t/t4220-diff-hunks.sh | 819 +++++++++++++++++ t/t4220/README | 55 ++ t/t4220/trim-divergent-new | 319 +++++++ t/t4220/trim-divergent-old | 316 +++++++ userdiff.c | 7 + userdiff.h | 2 + write-or-die.h | 7 +- xdiff-interface.h | 12 + 58 files changed, 5989 insertions(+), 64 deletions(-)create mode 100644
Documentation/config/diff-hunks.adoccreate mode 100644Documentation/git-diff-hunks.adoccreate mode 100644Documentation/gitformat-diff-hunks.adoccreate mode 100644builtin/diff-hunks.c
create mode 100644diff-hunks.c
create mode 100644diff-hunks.h
create mode 100644diff-process.c
create mode 100644diff-provider-internal.hcreate mode 100644diff-provider.c
create mode 100644diff-provider.h
create mode 100644t/helper/test-diff-process-backend.ccreate mode 100755t/perf/p4218-diff-hunks.shcreate mode 100755t/t4080-diff-process.shcreate mode 100755t/t4220-diff-hunks.sh
create mode 100644 t/t4220/README
create mode 100644 t/t4220/trim-divergent-new create mode 100644 t/t4220/trim-divergent-oldbase-commit:
9a0c4701dcd5725c4184599322b52933ff5005ca
prerequisite-patch-id:6270dea79c9f06530737cefa3e1a0a39a1be7877
prerequisite-patch-id:46fcc16a7a2ed760a1134d2a92c87699f3ec7bdb
prerequisite-patch-id:c1e3da243003d060e429bc2196ae02b3453f01f9
prerequisite-patch-id:4ad4e273494d4e8503706c21bfdc90a5d7ce116a
prerequisite-patch-id:f7fa1367756daafa83f4f030a5c7b6dc3dbb70d7
prerequisite-patch-id:829e76c9fec655a07f9383086a35bff3290b1c74
prerequisite-patch-id:5c5a0d61ae9b6d628d05f1eb5df046758f3111a8
Re: [RFC PATCH v7 0/10] diff: add provider interface and initial providers
Quoted 3 lines · Phillip Wood
Hi Michael
Hi Phillip, apologies for the delayed response, and thank you for taking a look!
Quoted 23 lines · Michael Montalbo
Quoted 13 lines
Every in-process diff in Git reduces, at one point, to a single
question: given two blobs and the settings the diff runs under, which line ranges changed? The answer is the diff's hunks: for each change, the position and length of the range on the old side and on the new. Each consumer asks in its own shape:- blame diffs each suspect's blob against its parent's, taking only the coordinates through xdiff's hunk callback;
- the stat formats keep only the added and deleted counts;
- patch output emits from the hunks, with xdiff interleaving context and content around them;
- log -L maps the tracked range across each commit from the coordinates.Quoted 8 lines
In every case the answer is computed the same way: load both blobs and run xdiff. That is the only source, so nothing that already holds the answer, or that would answer differently on purpose, can supply it instead. Sometimes that is what we want, which is why patch-id and format-patch stay on the builtin computation throughout: patch-id needs identical hashes on every machine, and a format-patch must apply for recipients who share none of the sender's configuration. Other times another source would be useful.
Quoted 6 lines
This explains the mechanics of the existing implementation, but doesn't really explain what the advantage of the change is beyond having a unified interface. What are things that are unlocked by this change? Are we doing it purely to cache the hunk headers or does it (as I suspect) unlock other features?
After thinking more about this feedback...
Quoted 13 lines
Quoted 9 lines
This RFC sketches a direction. The unified series shows one interface carrying two example providers and their interaction; it is not shaped to merge as one topic. If the direction holds, the work returns as separate reviewable series (see Roadmap). The two examples are demonstrations, each an RFC on its own: diff.<driver>.process, the RFC cooking as
mm/diff-process-hunks, lets a configured external process answer with its own notion of which lines changed, and the diff-hunks store, new in this thread, remembers what xdiff computed and serves it back. One is authoritative and external, one a cache and in-process.What does it mean to be authoritative? Why isn't the xdiff code authoritative?
...
Quoted 3 lines
Quoted 1 line
Chain order is the authority,
I'm not sure what that phrase means
...and the concept I was trying to express here, I realize the abstraction introduced in the latest v7, a "diff provider", is not ultimately helpful and shoehorns several concepts together that should be addressed separately. The "chain order is authority" was (poorly) trying to describe that diff_providers occurring earlier in the list of providers would "override" the default "authority" of xdiff. The concept "authority" in these terms is unclear, though, it could mean "who gets to answer first" or "who gets to answer differently than xdiff." It also could paper over how each current caller uses xdiff in its own control flow with a "chain" of providers.
Quoted 10 lines
Quoted 9 lines
terminal provider is the builtin computation itself, so the interface never exists without an implementor: patch 02 ships it answering every request the way the consumers did before. A consumer states its request in one struct and reads one set of outcomes (answered,
unanswered, or failed); it never names a provider, and a provider> added later maps onto those outcomes inside the interface, so consumer
code is written once. Because every diff now walks the chain even with no store or process configured, the default path was measured against the pre-series base and runs within noise (a 5000-commit log --stat and a long-history blame, ratio 1.00 either way).
Quoted 4 lines
So we can configure a provider for a particular file type via .gitattributes and diff.<driver>.process and then it is used automatically, if there is no provider configured we use xdiff?
Yes, that was the original intent.
Quoted 17 lines
Quoted 8 lines
- The diff-hunks store shows the non-authoritative side: an in-process cache at $GIT_DIR/objects/info/diff-hunks that may only reproduce the builtin diff, so serving from it never changes a command's output. It is read by default and written only when a repository owner opts in, warming it as a side effect of diff work the command already does:
GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/nullQuoted 8 lines
A warmed store then serves the stat formats and blame from stored coordinates instead of a fresh diff: on git.git a 5000-commit log --stat runs about 1.9x faster, and blame reads the same entries opportunistically (full numbers in [1]). Its format and keying, what it may not serve, and how it handles corruption and staleness are in git-diff-hunks(1), gitformat-diff-hunks(5), and [2]. The interface point is small: a cache drops in as the provider that stands aside wherever an authoritative one answers.
Quoted 6 lines
I had a very quick look at the documentation in patch that implements this. Am I right in thinking it stores
xdl_optsdirectly so that we cannot change the in-memory representation without breaking the cache? Are there plans to garbage collect cache entries when the corresponding blobs are removed?
Yes, you are right regarding the in-memory representation breaking the cache. On a separate branch where I was working on diff-hunks more, I had a prune command that removed entries based on time as a maintenance task, but nothing more intelligent.
Quoted 15 lines
Quoted 15 lines
- diff.<driver>.process shows the authoritative side: an external process, configured per driver, whose answers may deliberately differ from the builtin diff and outrank the store. Git asks it for a pair by object names alone, so it answers before any blob is read, which suits a cache or a process that fetches the blobs itself. Consulting is opt-in per command, following the
allow_textconvprecedent, and a pair the process cannot answer falls back to the builtin diff. The protocol, the per-command gate, how failures are handled, and the versioning that lets it grow are in gitattributes(5) and footnotes [3] and [4]. The interface point, again, is small: an external, authoritative provider joins the same chain ahead of the cache, and neither consumer learns it is there. A later content-carrying request would extend it to the pairs and consumers this identity-only form leaves on the builtin diff.
Quoted 5 lines
Quoted 1 line
The series stops at the coordinates.
Does this mean the offsets and lengths in the hunk header? I can't see any reference to "coordinates" in the existing code.
Right, "coordinates" is a term I invented to refer to offsets and lengths, but I can see why it is unclear and do not think it needs to be coined. I will replace the use of that term with more explicit / already established terminology.
Quoted 7 lines
Quoted 6 lines
A consumer that needs the changed
text, such as patch output, would have only its hunk selection replaced, with xdiff still emitting content from the blobs; that machinery is the content enrichment sketched in the Roadmap. Establishing the framework on coordinates first keeps this series one design: the question, the interface, and two providers answering by identity.
Quoted 6 lines
This seems like an interesting proposal and the diff headers seem like a good place for initial phase to stop. It would be helpful to have a clearer explanation of the features this interface would support (i.e. what's the motivation for these changes) and a lot less jargon in the interface description.
I really appreciate your feedback on this. I spent some time thinking about the most straightforward way to express what I am going for. My goals are to:
- add a "cousin" interface to xdiff-interface that allows users to configure their own alternative to xdiff. Among other things, this would enable users to install their own more intelligent diffing mechanisms while composing (to some extent) with surrounding Git diff functionality rather than replacing it wholesale.
- still use xdiff in the end for its content rendering features so the alternative diff hunk provider composes with Git's existing diff features. Meaning, this "cousin" would stage hunks and then xdiff would render in the end.
- provide the additional feature of allowing current xdiff-interface callers to potentially operate with oid's only and no content loading to save time dealing with content when only hunk header info is needed.
- somehow make a consistent/easy interface for users to plug in their own xdiff alternatives, which has been via pkt-line.
I think returning to the v6 line of the topic for future re-rolls and focusing on the framing of a "cousin" interface to xdiff-interface, makes more sense, rather than the "diff provider" concept introduced in v7. Accordingly, I am inclined to eject the diff-hunks patches for now and focus on the "pkt-line xdiff-interface" line. The diff-hunks example was meant to show an in-process xdiff alternate / complement that cannot occur over pkt-line, but I think it makes a lot less sense to include in this RFC now that the "diff providers" concept would be dropped.
Quoted 4 lines
Thanks
Phillip
Thank you!
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Michael Montalbo via GitGitGadget
This series adds diff.<driver>.process, a long-running subprocess protocol that lets an external tool control which lines Git considers changed while Git handles all output formatting...
Now I'm realizing there are some diff flags that are not properly supported yet with an external diff provider. Flags like -L and the stat-related flags should probably behave like blame and consult the external diff process. In general, there should be a more explicit and comprehensive explanation for which flags interact with external diff processes and which do not, included principled reasons for why that is the case.