rev-list: add --missing=print-only mode
cover rev-list: add --missing=print-only mode
Hi,
At GitLab, Gitaly uses rev-list --missing=print to find missing objects in partial clones. The current output mixes present and missing objects together, with missing ones prefixed by '?', so Gitaly has to post-process the output to extract just the missing OIDs. Having a dedicated mode that outputs only the missing OIDs directly would simplify this.
This patch adds --missing=print-only which suppresses all regular object output and prints only the missing OIDs, one per line, without the '?' prefix. A missing_action_prints() helper consolidates the repeated three-way checks for MA_PRINT/MA_PRINT_INFO/MA_PRINT_ONLY.
The series is based on top of 9f223ef1c0 (Git 2.54-rc2).
CI: https://gitlab.com/gitlab-org/git/-/pipelines/2463294546
Siddharth Asthana (1):
rev-list: add --missing=print-only mode
Documentation/rev-list-options.adoc | 5 +++ builtin/rev-list.c | 49 ++++++++++++++++++++++------- t/t6022-rev-list-missing.sh | 26 +++++++++++++++ 3 files changed, 68 insertions(+), 12 deletions(-)
base-commit: 9f223ef1c026d91c7ac68cc0211bde255dda6199
Thanks
- Siddharth
PATCH 1/1 rev-list: add --missing=print-only mode
When working with partial clones, it's common to want just the list of missing objects. The current --missing=print mode does this but mixes present and missing objects together, with missing ones prefixed by '?'. Getting only the missing OIDs requires an extra pipe:
git rev-list --objects --all --missing=print | perl -ne 'print if s/^[?]//'
Add --missing=print-only which outputs only the missing object OIDs, one per line, without any prefix. This makes the above one-liner unnecessary and the output directly usable by downstream tools.
--- Documentation/rev-list-options.adoc | 5 +++ builtin/rev-list.c | 49 ++++++++++++++++++++++------- t/t6022-rev-list-missing.sh | 26 +++++++++++++++ 3 files changed, 68 insertions(+), 12 deletions(-)
Documentation/rev-list-options.adoc +5 -0
@@ -1056,6 +1056,11 @@ Unexpected missing objects will raise an error. The form `--missing=print` is like `allow-any`, but will also print a list of the missing objects. Object IDs are prefixed with a ``?'' character. + +The form `--missing=print-only` is like `print`, but will print ONLY the +missing objects (not the present ones), and without the ``?'' prefix. This +is useful for scripting, as a simpler alternative to +`--missing=print | sed -n 's/^?//p'`. ++ The form `--missing=print-info` is like `print`, but will also print additional information about the missing object inferred from its containing object. The information is all printed on the same line with the missing object ID in the
builtin/rev-list.c +37 -12
116 lines not on this page · raw message
t/t6022-rev-list-missing.sh +26 -0
@@ -198,6 +198,32 @@ do ' done +for obj in "HEAD~1" "HEAD~1^{tree}" "HEAD:1.t" +do + test_expect_success "rev-list --missing=print-only with missing $obj" ' + oid="$(git rev-parse $obj)" && + path=".git/objects/$(test_oid_to_path $oid)" && + + # Capture present OIDs before hiding anything. + git rev-list --objects --no-object-names HEAD ^$obj >present.raw && + + mv "$path" "$path.hidden" && + test_when_finished "mv $path.hidden $path" && + + git rev-list --missing=print-only --objects --no-object-names \ + HEAD >actual && + + # Only the missing OID should appear, without the "?" prefix. + grep "^$oid$" actual && + + # Present objects must NOT appear in the output. + while read present_oid + do + ! grep "^$present_oid$" actual || return 1 + done <present.raw + ' +done + test_expect_success "-z nul-delimited --missing" ' test_when_finished rm -rf repo &&
Signature · 1 line
2.53.0
Quoted 10 lines · Siddharth Asthana
When working with partial clones, it's common to want just the list of missing objects. The current --missing=print mode does this but mixes present and missing objects together, with missing ones prefixed by '?'. Getting only the missing OIDs requires an extra pipe:
git rev-list --objects --all --missing=print | perl -ne 'print if s/^[?]//'Add --missing=print-only which outputs only the missing object OIDs, one per line, without any prefix. This makes the above one-liner unnecessary and the output directly usable by downstream tools.
I'm a fan of this mode. It saves time dealing with the I/O.
builtin/rev-list.c static void missing_objects_map_entry_free(void *e) · line 112
Quoted 12 lines
static struct oidmap missing_objects; enum missing_action {builtin/rev-list.c +5
-4-MA_ERROR =0 ,/* fail if any missing objects are encountered */ -MA_ALLOW_ANY ,/* silently allow ALL missing objects */ -MA_PRINT ,/* print ALL missing objects in special section */ -MA_PRINT_INFO ,/* same as MA_PRINT but also prints missing object info */ +MA_ERROR =0 ,/* fail if any missing objects are encountered */ +MA_ALLOW_ANY ,/* silently allow ALL missing objects */ +MA_PRINT ,/* print ALL missing objects in special section */ +MA_PRINT_INFO ,/* same as MA_PRINT but also prints missing object info */ +MA_PRINT_ONLY ,/* print ONLY missing objects, without the "?" prefix */ MA_ALLOW_PROMISOR, /* silently allow all missing PROMISOR objects */
I'm not a fan of this adjustment of spacing on the comments, though I do see that MA_ALLOW_PROMISOR had already broken the pattern. MA_PRINT_ONLY would still work with the earlier comment alignment, though.
builtin/rev-list.c static void print_missing_object(struct missing_objects_map_entry *entry, · line 169
Quoted 4 lines
builtin/rev-list.c +3
-1-if (line_term) +if (arg_missing_action ==MA_PRINT_ONLY ) { +printf ("%s" ,oid_to_hex (&entry->entry.oid)); +putchar (line_term);
Is there a reason you didn't use a printf("%s%c") here to put the oid and line_term together?
Quoted 85 lines
builtin/rev-list.c +21
-6+return ; + }else if (line_term) {printf ("?%s" ,oid_to_hex (&entry->entry.oid)); - else + }else {printf ("%s%cmissing=yes" ,oid_to_hex (&entry->entry.oid), info_term); + }if (!print_missing_info) {putchar (line_term);@@ -209,6 +222,7 @@ static inline void finish_object__ma(struct object *obj, const char *name)case MA_PRINT :case MA_PRINT_INFO : +case MA_PRINT_ONLY :add_missing_object_entry (&obj->oid, name, obj->type);return ;@@ -246,6 +260,11 @@ static void show_commit(struct commit *commit, void *data)return ; } +if (arg_missing_action ==MA_PRINT_ONLY ) { +finish_commit (commit); +return ; + } +if (show_disk_usage) total_disk_usage +=get_object_disk_usage (&commit->object);@@ -384,6 +403,8 @@ static void show_object(struct object *obj, const char *name, void *cb_data)if (finish_object (obj, name, cb_data))return ;display_progress (progress, ++progress_counter); +if (arg_missing_action ==MA_PRINT_ONLY ) +return ;if (show_disk_usage) total_disk_usage +=get_object_disk_usage (obj);if (info->flags &REV_LIST_QUIET )@@ -525,6 +546,12 @@ static inline int parse_missing_action_value(const char *value)return 1 ; } +if (!strcmp (value,"print-only" )) { + arg_missing_action =MA_PRINT_ONLY ; + fetch_if_missing =0 ; +return 1 ; + } +if (!strcmp (value,"allow-promisor" )) { arg_missing_action =MA_ALLOW_PROMISOR ; fetch_if_missing =0 ;@@ -967,8 +994,7 @@ int cmd_rev_list(int argc,if (arg_print_omitted)oidset_init (&omitted_objects,DEFAULT_OIDSET_SIZE ); -if (arg_missing_action ==MA_PRINT || - arg_missing_action ==MA_PRINT_INFO ) { +if (missing_action_prints ()) {struct oidset_iter iter;struct object_id *oid;@@ -994,8 +1020,7 @@ int cmd_rev_list(int argc,printf ("~%s\n" ,oid_to_hex (oid));oidset_clear (&omitted_objects); } -if (arg_missing_action ==MA_PRINT || - arg_missing_action ==MA_PRINT_INFO ) { +if (missing_action_prints ()) {struct missing_objects_map_entry *entry;struct oidmap_iter iter;@@ -1011,7 +1036,7 @@ int cmd_rev_list(int argc,stop_progress (&progress); -if (revs.count) { +if (revs.count && arg_missing_action !=MA_PRINT_ONLY ) {if (revs.left_right && revs.cherry_mark)printf ("%d\t%d\t%d\n" , revs.count_left, revs.count_right, revs.count_same);else if (revs.left_right)
builtin/rev-list.c int cmd_rev_list(int argc, · line 1051
Quoted 6 lines
builtin/rev-list.c +1
-1@@ -1022,7 +1047,7 @@ int cmd_rev_list(int argc,printf ("%d\n" , revs.count_left + revs.count_right); } -if (show_disk_usage) +if (show_disk_usage && arg_missing_action !=MA_PRINT_ONLY )print_disk_usage (total_disk_usage);
I'm a little worried about all of these checks that need special-casing. These seem like options that are enabled by other options and could easily be grouped with the print-only setting.
If there is something wrong here where these need to be disabled, then we should update the documentation to say how this interacts with those options. And perhaps some warnings to say "these options are not compatible".
On that note: this patch is missing a document update.
Thanks,
-Stolee
Quoted 6 lines · Derrick Stolee
Quoted 5 lines · Siddharth Asthana
When working with partial clones, it's common to want just the list of missing objects. The current --missing=print mode does this but mixes present and missing objects together, with missing ones prefixed by '?'. Getting only the missing OIDs requires an extra pipe:
Quoted 7 lines
Quoted 5 lines
git rev-list --objects --all --missing=print | perl -ne 'print if s/^[?]//'
Add --missing=print-only which outputs only the missing object OIDs, one per line, without any prefix. This makes the above one-liner unnecessary and the output directly usable by downstream tools.
I'm a fan of this mode. It saves time dealing with the I/O.
Thanks!
Quoted 17 lines
Quoted 16 lines
static struct oidmap
missing_objects;
enummissing_action{
-MA_ERROR= 0, /* fail if any missing objects are encountered */ - MA_ALLOW_ANY, /* silently allow ALL missing objects */ - MA_PRINT, /* print ALL missing objects in special section */ - MA_PRINT_INFO, /* same asMA_PRINTbut also prints missing object info */
+MA_ERROR= 0, /* fail if any missing objects are encountered */ +MA_ALLOW_ANY, /* silently allow ALL missing objects */
+MA_PRINT, /* print ALL missing objects in special section */ +MA_PRINT_INFO, /* same asMA_PRINTbut also prints missing object info */
+MA_PRINT_ONLY, /* print ONLY missing objects, without the "?" prefix */
MA_ALLOW_PROMISOR, /* silently allow all missing PROMISOR objects */
Quoted 1 line
I'm not a fan of this adjustment of spacing on the comments,
You are right, unnecessary churn - will restore!
Quoted 11 lines
though I do see that
MA_ALLOW_PROMISORhad already broken the pattern.MA_PRINT_ONLYwould still work with the earlier comment alignment, though.Quoted 4 lines
- if (
line_term)
+ if (arg_missing_action==MA_PRINT_ONLY) { +printf("%s", oid_to_hex(&entry->entry.oid)); +putchar(line_term);Is there a reason you didn't use a
printf("%s%c")here to put the oid andline_termtogether?
I just followed the existing pattern in the same function where every other path uses a separate putchar(line_term). Happy to combine if you prefer.
Quoted 80 lines
Quoted 80 lines
+ return;
+ } else if (line_term) {
printf("?%s", oid_to_hex(&entry->entry.oid)); - else
+ } else {
printf("%s%cmissing=yes", oid_to_hex(&entry->entry.oid), info_term);
+ }if (!print_missing_info) {
putchar(line_term);(quoted diff) +0
-0@@ -209,6 +222,7 @@ static inline void finish_object__ma(struct object *obj, const char *name)case
MA_PRINT:
caseMA_PRINT_INFO:
+ caseMA_PRINT_ONLY:
add_missing_object_entry(&obj->oid, name, obj->type); return;builtin/rev-list.c +7
-0@@ -246,6 +260,11 @@ static void show_commit(struct commit *commit, void *data) return; } + if (arg_missing_action == MA_PRINT_ONLY) { + finish_commit(commit); + return; + } + if (show_disk_usage) total_disk_usage += get_object_disk_usage(&commit->object);@@ -384,6 +403,8 @@ static void show_object(struct object *obj, const char *name, void *cb_data) if (finish_object(obj, name, cb_data)) return; display_progress(progress, ++progress_counter); + if (arg_missing_action == MA_PRINT_ONLY) + return; if (show_disk_usage) total_disk_usage += get_object_disk_usage(obj);if (
info->flags&REV_LIST_QUIET)builtin/rev-list.c +9
-5@@ -525,6 +546,12 @@ static inline int parse_missing_action_value(const char *value) return 1; } + if (!strcmp(value, "print-only")) { + arg_missing_action = MA_PRINT_ONLY; + fetch_if_missing = 0; + return 1; + } + if (!strcmp(value, "allow-promisor")) { arg_missing_action = MA_ALLOW_PROMISOR; fetch_if_missing = 0;@@ -967,8 +994,7 @@ int cmd_rev_list(int argc, if (arg_print_omitted) oidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE); - if (arg_missing_action == MA_PRINT || - arg_missing_action == MA_PRINT_INFO) { + if (missing_action_prints()) { struct oidset_iter iter; struct object_id *oid;@@ -994,8 +1020,7 @@ int cmd_rev_list(int argc, printf("~%s\n", oid_to_hex(oid)); oidset_clear(&omitted_objects); } - if (arg_missing_action == MA_PRINT || - arg_missing_action == MA_PRINT_INFO) { + if (missing_action_prints()) { struct missing_objects_map_entry *entry; struct oidmap_iter iter;@@ -1011,7 +1036,7 @@ int cmd_rev_list(int argc, stop_progress(&progress); - if (revs.count) { + if (revs.count && arg_missing_action != MA_PRINT_ONLY) { if (revs.left_right && revs.cherry_mark) printf("%d\t%d\t%d\n", revs.count_left, revs.count_right, revs.count_same); else if (revs.left_right)@@ -1022,7 +1047,7 @@ int cmd_rev_list(int argc,
Quoted 8 lines
builtin/rev-list.c int cmd_rev_list(int argc, · line 1051
Quoted 5 lines
builtin/rev-list.c +1
-1printf("%d\n", revs.count_left + revs.count_right); } - if (show_disk_usage) + if (show_disk_usage && arg_missing_action != MA_PRINT_ONLY) print_disk_usage(total_disk_usage);I'm a little worried about all of these checks that need special-casing. These seem like options that are enabled
Phillip suggested making this a separate --missing-only flag that composes with existing --missing= modes - I think that's a better design. will work v2 around it.
Quoted 5 lines
by other options and could easily be grouped with the print-only setting.
If there is something wrong here where these need to be disabled, then we should update the documentation to say
Agreed - I will make --count and --disk-usage incompatible with --missing-only and die() with a clear message instead of silently suppressing them.
Quoted 4 lines
how this interacts with those options. And perhaps some warnings to say "these options are not compatible".
On that note: this patch is missing a document update.
There is a doc update in rev-list-options.doc, but it doesn't cover interactions with --count/--disk-usage. Will fix in v2.
Quoted 5 lines
Thanks,
-Stolee
Thanks,
Asthana
Quoted 8 lines · Siddharth Asthana
Quoted 7 lines · Derrick Stolee
Quoted 4 lines · Siddharth Asthana
- if (
line_term)
+ if (arg_missing_action==MA_PRINT_ONLY) { +printf("%s", oid_to_hex(&entry->entry.oid)); +putchar(line_term);
Quoted 9 lines
Quoted 2 lines
Is there a reason you didn't use a
printf("%s%c")here to put the oid andline_termtogether?I just followed the existing pattern in the same function where every other path uses a separate
putchar(line_term). Happy to combine if you prefer.Quoted 3 lines
Quoted 3 lines
+ } else {
printf("%s%cmissing=yes", oid_to_hex(&entry->entry.oid), info_term);
I was just looking at this line in the patch context as comparison, but if there are other patterns that prefer "printf(); putc();" then that's enough for me.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
- if (
show_disk_usage)
+ if (show_disk_usage&&arg_missing_action!=MA_PRINT_ONLY)print_disk_usage(total_disk_usage);I'm a little worried about all of these checks that need special-casing. These seem like options that are enabled
Phillip suggested making this a separate --missing-only flag that composes with existing --missing= modes - I think that's a better design. will work v2 around it.
That's a good idea. '--missing=' formats the output while your mode _filters_ the output. They are different things and worth different options.
Quoted 7 lines
Quoted 4 lines
how this interacts with those options. And perhaps some warnings to say "these options are not compatible".
On that note: this patch is missing a document update.
There is a doc update in rev-list-options.doc, but it doesn't cover interactions with --count/--disk-usage. Will fix in v2.
You're right. I thought I had looked for it in the patch but missed it somehow.
Thanks,
-Stolee
Quoted 10 lines · Siddharth Asthana
When working with partial clones, it's common to want just the list of missing objects. The current --missing=print mode does this but mixes present and missing objects together, with missing ones prefixed by '?'. Getting only the missing OIDs requires an extra pipe:
git rev-list --objects --all --missing=print | perl -ne 'print if s/^[?]//'Add --missing=print-only which outputs only the missing object OIDs, one per line, without any prefix. This makes the above one-liner unnecessary and the output directly usable by downstream tools.
Naming is a bit tough, as "print-only" sounds as if we're only printing them without doing anything else, but it doesn't quite convey the relation to non-missing objects. I don't really have a better suggestion though -- "print-exclusively" may convey the meaning a tiny bit better, but still suffers kind of the same issue.
Quoted 7 lines
builtin/rev-list.c +0
-0@@ -104,14 +104,22 @@ static void missing_objects_map_entry_free(void *e)static struct oidmap missing_objects;
builtin/rev-list.c static void missing_objects_map_entry_free(void *e) · line 111
Quoted 10 lines
builtin/rev-list.c +5
-4enum missing_action { -MA_ERROR =0 ,/* fail if any missing objects are encountered */ -MA_ALLOW_ANY ,/* silently allow ALL missing objects */ -MA_PRINT ,/* print ALL missing objects in special section */ -MA_PRINT_INFO ,/* same as MA_PRINT but also prints missing object info */ +MA_ERROR =0 ,/* fail if any missing objects are encountered */ +MA_ALLOW_ANY ,/* silently allow ALL missing objects */ +MA_PRINT ,/* print ALL missing objects in special section */ +MA_PRINT_INFO ,/* same as MA_PRINT but also prints missing object info */ +MA_PRINT_ONLY ,/* print ONLY missing objects, without the "?" prefix */
Makes me wonder whether we'll eventually also want to have MA_PRINT_INFO_ONLY.
builtin/rev-list.c static void missing_objects_map_entry_free(void *e) · line 116
Quoted 5 lines
builtin/rev-list.c +1
-0MA_ALLOW_PROMISOR, /* silently allow all missing PROMISOR objects */ };static enum missing_action arg_missing_action; +static inline int missing_action_prints (void )
How about naming this should_print_missing_object() instead? That gives the reader a bit more context.
builtin/rev-list.c int cmd_rev_list(int argc, · line 1042
Quoted 9 lines
builtin/rev-list.c +1
-1@@ -1011,7 +1036,7 @@ int cmd_rev_list(int argc,stop_progress (&progress); -if (revs.count) { +if (revs.count && arg_missing_action !=MA_PRINT_ONLY ) {if (revs.left_right && revs.cherry_mark)printf ("%d\t%d\t%d\n" , revs.count_left, revs.count_right, revs.count_same);else if (revs.left_right)
Not a fault of your patch, but I really feel like git-rev-list(1) is becoming more and more tangled. The fact that we have to add this check to so many different sites doesn't inspire confidence that we have indeed catched all of them that need this check.
It would be great if this was reworked a bit to become more obvious, but that's probably outside the scope of this patch series.
Quoted 21 lines
t/t6022-rev-list-missing.sh +13
-0@@ -198,6 +198,32 @@ do ' done +for obj in "HEAD~1" "HEAD~1^{tree}" "HEAD:1.t" +do + test_expect_success "rev-list --missing=print-only with missing $obj" ' + oid="$(git rev-parse $obj)" && + path=".git/objects/$(test_oid_to_path $oid)" && + + # Capture present OIDs before hiding anything. + git rev-list --objects --no-object-names HEAD ^$obj >present.raw && + + mv "$path" "$path.hidden" && + test_when_finished "mv $path.hidden $path" && + + git rev-list --missing=print-only --objects --no-object-names \
t/t6022-rev-list-missing.sh do · line 223
Quoted 10 lines
t/t6022-rev-list-missing.sh +10
-0+ HEAD >actual && + + # Only the missing OID should appear, without the "?" prefix. + grep "^$oid$" actual && + + # Present objects must NOT appear in the output. + while read present_oid + do + ! grep "^$present_oid$" actual || return 1 + done <present.raw
How many present object IDs do we have? I'm a bit worried that we now execute grep(1) hundreds of times. Can we maybe do some tricks with comm(1) instead?
Thanks!
Patrick
Quoted 16 lines · Patrick Steinhardt
builtin/rev-list.c static void missing_objects_map_entry_free(void *e) · line 111
Quoted 11 lines · Siddharth Asthana
static struct oidmap missing_objects; enum missing_action {builtin/rev-list.c +5
-4-MA_ERROR =0 ,/* fail if any missing objects are encountered */ -MA_ALLOW_ANY ,/* silently allow ALL missing objects */ -MA_PRINT ,/* print ALL missing objects in special section */ - MA_PRINT_INFO, /* same as MA_PRINT but also prints missing object info */ +MA_ERROR =0 ,/* fail if any missing objects are encountered */ +MA_ALLOW_ANY ,/* silently allow ALL missing objects */ +MA_PRINT ,/* print ALL missing objects in special section */ +MA_PRINT_INFO ,/* same as MA_PRINT but also prints missing object info */ + MA_PRINT_ONLY, /* print ONLY missing objects, without the "?" prefix */Makes me wonder whether we'll eventually also want to have
MA_PRINT_INFO_ONLY.
Perhaps we'd be better to add a "--missing-only" option that limits the output to missing objects? That would avoid the problem of "--missing=print-only" not really explaining what it does as well.
Quoted 19 lines
Quoted 19 lines
t/t6022-rev-list-missing.sh +19
-0+for obj in "HEAD~1" "HEAD~1^{tree}" "HEAD:1.t" +do + test_expect_success "rev-list --missing=print-only with missing $obj" ' + oid="$(git rev-parse $obj)" && + path=".git/objects/$(test_oid_to_path $oid)" && + + # Capture present OIDs before hiding anything. + git rev-list --objects --no-object-names HEAD ^$obj >present.raw && + + mv "$path" "$path.hidden" && + test_when_finished "mv $path.hidden $path" && + + git rev-list --missing=print-only --objects --no-object-names \ + HEAD >actual && + + # Only the missing OID should appear, without the "?" prefix. + grep "^$oid$" actual && + + # Present objects must NOT appear in the output.
Quoted 8 lines
t/t6022-rev-list-missing.sh do · line 223
Quoted 4 lines
t/t6022-rev-list-missing.sh +4
-0+ while read present_oid + do + ! grep "^$present_oid$" actual || return 1 + done <present.rawHow many present object IDs do we have? I'm a bit worried that we now execute grep(1) hundreds of times. Can we maybe do some tricks with comm(1) instead?
If we want to verify that it only prints a single oid then
echo $oid >expect && test_cmp expect actual
would be much simpler more helpful if the test fails
Thanks
Phil
Quoted 18 lines · Phillip Wood
Quoted 16 lines · Patrick Steinhardt
builtin/rev-list.c static void missing_objects_map_entry_free(void *e) · line 111
Quoted 11 lines · Siddharth Asthana
static struct oidmap missing_objects; enum missing_action {builtin/rev-list.c +5
-4-MA_ERROR =0 ,/* fail if any missing objects are encountered */ -MA_ALLOW_ANY ,/* silently allow ALL missing objects */ -MA_PRINT ,/* print ALL missing objects in special section */ - MA_PRINT_INFO, /* same as MA_PRINT but also prints missing object info */ +MA_ERROR =0 ,/* fail if any missing objects are encountered */ +MA_ALLOW_ANY ,/* silently allow ALL missing objects */ +MA_PRINT ,/* print ALL missing objects in special section */ +MA_PRINT_INFO ,/* same as MA_PRINT but also prints missing object info */ + MA_PRINT_ONLY, /* print ONLY missing objects, without the "?" prefix */Makes me wonder whether we'll eventually also want to have
MA_PRINT_INFO_ONLY.
Quoted 3 lines
Perhaps we'd be better to add a "--missing-only" option that limits the output to missing objects? That would avoid the problem of "--missing=print-only" not really explaining what it does as well.
Yes, I had the same thought.
Patrick
Quoted 18 lines · Phillip Wood
Quoted 17 lines · Patrick Steinhardt
Quoted 14 lines · Siddharth Asthana
static struct oidmap
missing_objects;
enummissing_action{
-MA_ERROR= 0, /* fail if any missing objects are encountered */ - MA_ALLOW_ANY, /* silently allow ALL missing objects */ - MA_PRINT, /* print ALL missing objects in special section */ - MA_PRINT_INFO, /* same asMA_PRINTbut also prints missing object info */
+MA_ERROR= 0, /* fail if any missing objects are encountered */ +MA_ALLOW_ANY, /* silently allow ALL missing objects */ +MA_PRINT, /* print ALL missing objects in special section */ +MA_PRINT_INFO, /* same asMA_PRINTbut also prints missing object info */
+MA_PRINT_ONLY, /* print ONLY missing objects, without the "?" prefix */
Quoted 4 lines
Quoted 2 lines
Makes me wonder whether we'll eventually also want to have
MA_PRINT_INFO_ONLY.Perhaps we'd be better to add a "--missing-only" option that limits the output to missing objects? That would avoid the problem of "--
Make sense, its orthogonal to --missing= and handles print-info-only for free.
Question though: should --missing-only without --missing=print (or print-info) be an errro? I am leaning towards requiring it explicitly so the behavior is always obvious.
Will rework v2 around this.
Quoted 32 lines
missing=print-only" not really explaining what it does as well.
Quoted 21 lines
Quoted 21 lines
t/t6022-rev-list-missing.sh +3
-0+for obj in "HEAD~1" "HEAD~1^{tree}" "HEAD:1.t" +do + test_expect_success "rev-list --missing=print-only with missing$obj" '
+ oid="$(git rev-parse $obj)" &&
+ path=".git/objects/$(test_oid_to_path $oid)" && +
+ # Capture present OIDs before hiding anything. + git rev-list --objects --no-object-names HEAD ^$objQuoted 1 line
present.raw &&
t/t6022-rev-list-missing.sh +11
-0+ + mv "$path" "$path.hidden" && + test_when_finished "mv $path.hidden $path" && + + git rev-list --missing=print-only --objects --no-object-names \ + HEAD >actual && + + # Only the missing OID should appear, without the "?" prefix. + grep "^$oid$" actual && + + # Present objects must NOT appear in the output.Quoted 8 lines
Quoted 4 lines
+ while read
present_oid
+ do
+ ! grep "^$present_oid$" actual || return 1 + done <present.rawHow many present object IDs do we have? I'm a bit worried that we now execute grep(1) hundreds of times. Can we maybe do some tricks with comm(1) instead?
Quoted 6 lines
If we want to verify that it only prints a single oid then
echo $oid >expect &&
test_cmp expect actualwould be much simpler more helpful if the test fails
yeah much cleaner, will use it. Thanks
Quoted 4 lines
Thanks
Phil
Thanks,
Asthana
Quoted 18 lines · Phillip Wood
Quoted 16 lines · Patrick Steinhardt
builtin/rev-list.c static void missing_objects_map_entry_free(void *e) · line 111
Quoted 11 lines · Siddharth Asthana
static struct oidmap missing_objects; enum missing_action {builtin/rev-list.c +5
-4-MA_ERROR =0 ,/* fail if any missing objects are encountered */ -MA_ALLOW_ANY ,/* silently allow ALL missing objects */ -MA_PRINT ,/* print ALL missing objects in special section */ - MA_PRINT_INFO, /* same as MA_PRINT but also prints missing object info */ +MA_ERROR =0 ,/* fail if any missing objects are encountered */ +MA_ALLOW_ANY ,/* silently allow ALL missing objects */ +MA_PRINT ,/* print ALL missing objects in special section */ +MA_PRINT_INFO ,/* same as MA_PRINT but also prints missing object info */ + MA_PRINT_ONLY, /* print ONLY missing objects, without the "?" prefix */Makes me wonder whether we'll eventually also want to have
MA_PRINT_INFO_ONLY.
Quoted 3 lines
Perhaps we'd be better to add a "--missing-only" option that limits the output to missing objects? That would avoid the problem of "--missing=print-only" not really explaining what it does as well.
The original --missing says what to do about missing objects. I find --missing=print-only to be surprising:
• Do print missing
• But also do not print present objects
Why would a --missing option dictate what to do about objects that are present (hide them)? That looks tacked-on.
--missing-only is better but has the same problem, IMO, of dictating how present objects should be treated.
From the cover letter:
CV> At GitLab, Gitaly uses rev-list --missing=print to find missing CV> objects in partial clones. The current output mixes present and missing CV> objects together, with missing ones prefixed by '?', so Gitaly has to CV> post-process the output to extract just the missing OIDs. Having a CV> dedicated mode that outputs only the missing OIDs directly would CV> simplify this.
This makes me think that I want:
1. I do want missing objects
2. I do not want present objects
Which makes me think that something like --present=no-print makes more sense.
Just speaking as a user and not looking at the code.
Quoted 1 line
[snip]
Quoted 6 lines · Patrick Steinhardt
Quoted 5 lines · Siddharth Asthana
When working with partial clones, it's common to want just the list of missing objects. The current --missing=print mode does this but mixes present and missing objects together, with missing ones prefixed by '?'. Getting only the missing OIDs requires an extra pipe:
Quoted 8 lines
Quoted 5 lines
git rev-list --objects --all --missing=print | perl -ne 'print if s/^[?]//'Add --missing=print-only which outputs only the missing object OIDs, one per line, without any prefix. This makes the above one-liner unnecessary and the output directly usable by downstream tools.
Naming is a bit tough, as "print-only" sounds as if we're only printing them without doing anything else, but it doesn't quite convey the
The name came from Christian's original suggesttion in the issue [1], but agreed it's ambiguous. Phillip's --missing-only approach solve this.
[1] https://gitlab.com/gitlab-org/git/-/work_items/80#note_464005298
Quoted 10 lines
relation to non-missing objects. I don't really have a better suggestion though -- "print-exclusively" may convey the meaning a tiny bit better, but still suffers kind of the same issue.
Quoted 6 lines
builtin/rev-list.c +0
-0@@ -104,14 +104,22 @@ static void missing_objects_map_entry_free(void *e)
Quoted 14 lines
builtin/rev-list.c static void missing_objects_map_entry_free(void *e) · line 111
Quoted 11 lines
builtin/rev-list.c +5
-4static struct oidmap missing_objects;enum missing_action { -MA_ERROR =0 ,/* fail if any missing objects are encountered */ -MA_ALLOW_ANY ,/* silently allow ALL missing objects */ -MA_PRINT ,/* print ALL missing objects in special section */ -MA_PRINT_INFO ,/* same as MA_PRINT but also prints missing object info */ +MA_ERROR =0 ,/* fail if any missing objects are encountered */ +MA_ALLOW_ANY ,/* silently allow ALL missing objects */ +MA_PRINT ,/* print ALL missing objects in special section */ +MA_PRINT_INFO ,/* same as MA_PRINT but also prints missing object info */ +MA_PRINT_ONLY ,/* print ONLY missing objects, without the "?" prefix */ Makes me wonder whether we'll eventually also want to have
MA_PRINT_INFO_ONLY.
Right - that's the strongest argument for --missing-only as a separate flag. Gets us that for free.
Quoted 9 lines
builtin/rev-list.c static void missing_objects_map_entry_free(void *e) · line 116
Quoted 6 lines
builtin/rev-list.c +1
-0MA_ALLOW_PROMISOR, /* silently allow all missing PROMISOR objects */ };static enum missing_action arg_missing_action; +static inline int missing_action_prints (void )How about naming this
should_print_missing_object()instead? That gives the reader a bit more context.
The function is a predicate on the mode, not on a specific object, so that name would be slightly misleading. But with --missing-only this helper might change shape anyway. WDYT?
Quoted 46 lines
builtin/rev-list.c int cmd_rev_list(int argc, · line 1042
Quoted 10 lines
builtin/rev-list.c +1
-1@@ -1011,7 +1036,7 @@ int cmd_rev_list(int argc,stop_progress (&progress); -if (revs.count) { +if (revs.count && arg_missing_action !=MA_PRINT_ONLY ) {if (revs.left_right && revs.cherry_mark)printf ("%d\t%d\t%d\n" , revs.count_left, revs.count_right, revs.count_same);else if (revs.left_right)Not a fault of your patch, but I really feel like git-rev-list(1) is becoming more and more tangled. The fact that we have to add this check to so many different sites doesn't inspire confidence that we have indeed catched all of them that need this check.
It would be great if this was reworked a bit to become more obvious, but that's probably outside the scope of this patch series.
Quoted 27 lines
t/t6022-rev-list-missing.sh +19
-0@@ -198,6 +198,32 @@ do ' done +for obj in "HEAD~1" "HEAD~1^{tree}" "HEAD:1.t" +do + test_expect_success "rev-list --missing=print-only with missing $obj" ' + oid="$(git rev-parse $obj)" && + path=".git/objects/$(test_oid_to_path $oid)" && + + # Capture present OIDs before hiding anything. + git rev-list --objects --no-object-names HEAD ^$obj >present.raw && + + mv "$path" "$path.hidden" && + test_when_finished "mv $path.hidden $path" && + + git rev-list --missing=print-only --objects --no-object-names \ + HEAD >actual && + + # Only the missing OID should appear, without the "?" prefix. + grep "^$oid$" actual && + + # Present objects must NOT appear in the output.
Quoted 8 lines
t/t6022-rev-list-missing.sh do · line 223
Quoted 4 lines
t/t6022-rev-list-missing.sh +4
-0+ while read present_oid + do + ! grep "^$present_oid$" actual || return 1 + done <present.rawHow many present object IDs do we have? I'm a bit worried that we now execute grep(1) hundreds of times. Can we maybe do some tricks with comm(1) instead?
Phillip's test_cmp approach is simpler, since we hide one object, the output should be exactly that OID. Will use that.
Thanks,
Asthana
Quoted 4 lines
Thanks!
Patrick
v2 cover rev-list: add --missing-only option to filter output
In reply to Siddharth AsthanaHi,
At GitLab, Gitaly uses rev-list --missing=print on partial clones to find missing objects. The output mixes present and missing objects and prefixes missing ones with '?', so we post-process it. --missing-only avoids that.
v1 added --missing=print-only as a --missing= mode. Review preferred a separate filter flag, so v2 adds --missing-only next to --missing=print or --missing=print-info.
Based on 1630431f32 (The 21st batch, 2026-08-31).
Changes from v1 include:
1. Separate --missing-only flag instead of print-only mode (Phillip, Patrick, Stolee).
2. Require --missing=print or --missing=print-info.
3. Die on --count / --disk-usage (Stolee).
4. No enum comment spacing churn.
5. print-info still prints path=/type=; only '?' is dropped.
6. test_cmp-based tests (Phillip).
7. Link to v1:
https://lore.kernel.org/git/[email protected]/
Thanks,
Siddharth
---
Siddharth Asthana (1):
rev-list: add --missing-only option to filter output
Documentation/rev-list-options.adoc | 13 ++++++++ builtin/rev-list.c | 42 ++++++++++++++++++++++--- t/t6022-rev-list-missing.sh | 49 +++++++++++++++++++++++++++++ 3 files changed, 99 insertions(+), 5 deletions(-)
Range-diff versus v1:
1: 7e8f1ad997 ! 1: 3c89e6bc38 rev-list: add --missing=print-only mode
@@ Metadata
Author: Siddharth Asthana <[email protected]>
## Commit message ##
- rev-list: add --missing=print-only mode
+ rev-list: add --missing-only option to filter output
- When working with partial clones, it's common to want just the list of
- missing objects. The current --missing=print mode does this but mixes
- present and missing objects together, with missing ones prefixed by '?'.
- Getting only the missing OIDs requires an extra pipe:
+ When working with partial clones, callers often need only the missing
+ object IDs. Today that means post-processing --missing=print to drop
+ present objects and strip the leading '?':
git rev-list --objects --all --missing=print | perl -ne 'print if s/^[?]//'
- Add --missing=print-only which outputs only the missing object OIDs, one
- per line, without any prefix. This makes the above one-liner unnecessary
- and the output directly usable by downstream tools.
+ Add --missing-only. Use it with --missing=print or --missing=print-info
+ to print only missing objects. --missing= still picks the format;
+ --missing-only only filters. The leading '?' is omitted. With
+ print-info, path= and type= are still shown.
+
+ Require --missing=print or --missing=print-info. Reject --count and
+ --disk-usage.
Signed-off-by: Siddharth Asthana <[email protected]>
## Documentation/rev-list-options.adoc ##
-@@ Documentation/rev-list-options.adoc: Unexpected missing objects will raise an error.
- The form `--missing=print` is like `allow-any`, but will also print a
- list of the missing objects. Object IDs are prefixed with a ``?'' character.
- +
-+The form `--missing=print-only` is like `print`, but will print ONLY the
-+missing objects (not the present ones), and without the ``?'' prefix. This
-+is useful for scripting, as a simpler alternative to
-+`--missing=print | sed -n 's/^?//p'`.
+@@ Documentation/rev-list-options.adoc: If some tips passed to the traversal are missing, they will be
+ considered as missing too, and the traversal will ignore them. In case
+ we cannot get their Object ID though, an error will be raised.
+
++`--missing-only`::
++ When used together with `--missing=print` or `--missing=print-info`,
++ suppress all output for present objects and print only the missing
++ ones. The selected `--missing=` format is preserved (so
++ `--missing=print-info` still emits `path=` / `type=` fields), but the
++ leading ``?'' prefix used by the non-`-z` forms is omitted. This is
++ useful for scripting, as a simpler and faster alternative to
++ post-processing the output of `--missing=print`.
++
- The form `--missing=print-info` is like `print`, but will also print additional
- information about the missing object inferred from its containing object. The
- information is all printed on the same line with the missing object ID in the
++This option is incompatible with `--count` and `--disk-usage`.
++It is an error to use `--missing-only` without `--missing=print` or
++`--missing=print-info`.
++
+ `--exclude-promisor-objects`::
+ (For internal use only.) Prefilter object traversal at
+ promisor boundary. This is used with partial clone. This is
## builtin/rev-list.c ##
-@@ builtin/rev-list.c: static void missing_objects_map_entry_free(void *e)
-
- static struct oidmap missing_objects;
- enum missing_action {
-- MA_ERROR = 0, /* fail if any missing objects are encountered */
-- MA_ALLOW_ANY, /* silently allow ALL missing objects */
-- MA_PRINT, /* print ALL missing objects in special section */
-- MA_PRINT_INFO, /* same as MA_PRINT but also prints missing object info */
-+ MA_ERROR = 0, /* fail if any missing objects are encountered */
-+ MA_ALLOW_ANY, /* silently allow ALL missing objects */
-+ MA_PRINT, /* print ALL missing objects in special section */
-+ MA_PRINT_INFO, /* same as MA_PRINT but also prints missing object info */
-+ MA_PRINT_ONLY, /* print ONLY missing objects, without the "?" prefix */
+@@ builtin/rev-list.c: enum missing_action {
MA_ALLOW_PROMISOR, /* silently allow all missing PROMISOR objects */
};
static enum missing_action arg_missing_action;
-
-+static inline int missing_action_prints(void)
++static int arg_missing_only;
++
++static inline int should_collect_missing(void)
+{
+ return arg_missing_action == MA_PRINT ||
-+ arg_missing_action == MA_PRINT_INFO ||
-+ arg_missing_action == MA_PRINT_ONLY;
++ arg_missing_action == MA_PRINT_INFO;
+}
-+
+
/* display only the oid of each object encountered */
static int arg_show_object_names = 1;
-
@@ builtin/rev-list.c: static void print_missing_object(struct missing_objects_map_entry *entry,
{
struct strbuf sb = STRBUF_INIT;
- if (line_term)
-+ if (arg_missing_action == MA_PRINT_ONLY) {
++ /*
++ * --missing-only filters present objects out of the walk output.
++ * It still uses the selected --missing= format for missing ones,
++ * except the human "?" prefix is omitted (script-friendly OIDs).
++ */
++ if (arg_missing_only && line_term)
+ printf("%s", oid_to_hex(&entry->entry.oid));
-+ putchar(line_term);
-+ return;
-+ } else if (line_term) {
++ else if (line_term)
printf("?%s", oid_to_hex(&entry->entry.oid));
-- else
-+ } else {
+ else
printf("%s%cmissing=yes", oid_to_hex(&entry->entry.oid),
- info_term);
-+ }
-
- if (!print_missing_info) {
- putchar(line_term);
-@@ builtin/rev-list.c: static inline void finish_object__ma(struct object *obj, const char *name)
-
- case MA_PRINT:
- case MA_PRINT_INFO:
-+ case MA_PRINT_ONLY:
- add_missing_object_entry(&obj->oid, name, obj->type);
- return;
-
@@ builtin/rev-list.c: static void show_commit(struct commit *commit, void *data)
return;
}
-+ if (arg_missing_action == MA_PRINT_ONLY) {
++ if (arg_missing_only) {
+ finish_commit(commit);
+ return;
+ }
@@ builtin/rev-list.c: static void show_object(struct object *obj, const char *name
if (finish_object(obj, name, cb_data))
return;
display_progress(progress, ++progress_counter);
-+ if (arg_missing_action == MA_PRINT_ONLY)
++ if (arg_missing_only)
+ return;
if (show_disk_usage)
total_disk_usage += get_object_disk_usage(obj);
if (info->flags & REV_LIST_QUIET)
-@@ builtin/rev-list.c: static inline int parse_missing_action_value(const char *value)
- return 1;
+@@ builtin/rev-list.c: int cmd_rev_list(int argc,
+ revs.exclude_promisor_objects = 1;
+ } else if (skip_prefix(arg, "--missing=", &arg)) {
+ parse_missing_action_value(arg);
++ } else if (!strcmp(arg, "--missing-only")) {
++ arg_missing_only = 1;
+ } else if (!strcmp(arg, "-z")) {
+ line_term = '\0';
+ info_term = '\0';
+ }
}
-+ if (!strcmp(value, "print-only")) {
-+ arg_missing_action = MA_PRINT_ONLY;
-+ fetch_if_missing = 0;
-+ return 1;
-+ }
++ if (arg_missing_only && !should_collect_missing())
++ die(_("--missing-only requires --missing=print or --missing=print-info"));
++
+ die_for_incompatible_opt2(revs.exclude_promisor_objects,
+ "--exclude_promisor_objects",
+ arg_missing_action, "--missing");
+@@ builtin/rev-list.c: int cmd_rev_list(int argc,
+ continue;
+ }
+
++ if (!strcmp(arg, "--missing-only"))
++ continue;
++
+ usage(rev_list_usage);
+
+ }
+@@ builtin/rev-list.c: int cmd_rev_list(int argc,
+ (revs.left_right || revs.cherry_mark))
+ die(_("marked counting and '%s' cannot be used together"), "--objects");
+
++ die_for_incompatible_opt2(arg_missing_only, "--missing-only",
++ revs.count, "--count");
++ die_for_incompatible_opt2(arg_missing_only, "--missing-only",
++ show_disk_usage, "--disk-usage");
+
- if (!strcmp(value, "allow-promisor")) {
- arg_missing_action = MA_ALLOW_PROMISOR;
- fetch_if_missing = 0;
+ save_commit_buffer = (revs.verbose_header ||
+ revs.grep_filter.pattern_list ||
+ revs.grep_filter.header_list);
@@ builtin/rev-list.c: int cmd_rev_list(int argc,
if (arg_print_omitted)
oidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE);
- if (arg_missing_action == MA_PRINT ||
- arg_missing_action == MA_PRINT_INFO) {
-+ if (missing_action_prints()) {
++ if (should_collect_missing()) {
struct oidset_iter iter;
struct object_id *oid;
@@ builtin/rev-list.c: int cmd_rev_list(int argc,
}
- if (arg_missing_action == MA_PRINT ||
- arg_missing_action == MA_PRINT_INFO) {
-+ if (missing_action_prints()) {
++ if (should_collect_missing()) {
struct missing_objects_map_entry *entry;
struct oidmap_iter iter;
-@@ builtin/rev-list.c: int cmd_rev_list(int argc,
-
- stop_progress(&progress);
-
-- if (revs.count) {
-+ if (revs.count && arg_missing_action != MA_PRINT_ONLY) {
- if (revs.left_right && revs.cherry_mark)
- printf("%d\t%d\t%d\n", revs.count_left, revs.count_right, revs.count_same);
- else if (revs.left_right)
-@@ builtin/rev-list.c: int cmd_rev_list(int argc,
- printf("%d\n", revs.count_left + revs.count_right);
- }
-
-- if (show_disk_usage)
-+ if (show_disk_usage && arg_missing_action != MA_PRINT_ONLY)
- print_disk_usage(total_disk_usage);
-
- cleanup:
## t/t6022-rev-list-missing.sh ##
@@ t/t6022-rev-list-missing.sh: do
@@ t/t6022-rev-list-missing.sh: do
+for obj in "HEAD~1" "HEAD~1^{tree}" "HEAD:1.t"
+do
-+ test_expect_success "rev-list --missing=print-only with missing $obj" '
++ test_expect_success "rev-list --missing-only with missing $obj" '
+ oid="$(git rev-parse $obj)" &&
+ path=".git/objects/$(test_oid_to_path $oid)" &&
+
-+ # Capture present OIDs before hiding anything.
-+ git rev-list --objects --no-object-names HEAD ^$obj >present.raw &&
-+
+ mv "$path" "$path.hidden" &&
+ test_when_finished "mv $path.hidden $path" &&
+
-+ git rev-list --missing=print-only --objects --no-object-names \
-+ HEAD >actual &&
-+
-+ # Only the missing OID should appear, without the "?" prefix.
-+ grep "^$oid$" actual &&
++ git rev-list --missing=print --missing-only --objects \
++ --no-object-names HEAD >actual &&
+
-+ # Present objects must NOT appear in the output.
-+ while read present_oid
-+ do
-+ ! grep "^$present_oid$" actual || return 1
-+ done <present.raw
++ echo $oid >expect &&
++ test_cmp expect actual
+ '
+done
++
++test_expect_success "--missing-only requires --missing=print or --missing=print-info" '
++ test_must_fail git rev-list --missing-only --objects HEAD 2>err &&
++ test_grep "requires --missing=print" err
++'
++
++test_expect_success "--missing-only is incompatible with --count" '
++ test_must_fail git rev-list --missing=print --missing-only \
++ --count --objects HEAD 2>err &&
++ test_grep "cannot be used together" err
++'
++
++test_expect_success "--missing-only is incompatible with --disk-usage" '
++ test_must_fail git rev-list --missing=print --missing-only \
++ --disk-usage --objects HEAD 2>err &&
++ test_grep "cannot be used together" err
++'
++
++test_expect_success "--missing-only works with --missing=print-info" '
++ oid="$(git rev-parse HEAD:1.t)" &&
++ path=".git/objects/$(test_oid_to_path $oid)" &&
++
++ mv "$path" "$path.hidden" &&
++ test_when_finished "mv $path.hidden $path" &&
++
++ git rev-list --missing=print-info --missing-only --objects \
++ --no-object-names HEAD >actual &&
++
++ # Filter keeps print-info fields; only the "?" prefix is dropped.
++ echo "$oid path=1.t type=blob" >expect &&
++ test_cmp expect actual
++'
+
test_expect_success "-z nul-delimited --missing" '
test_when_finished rm -rf repo &&base-commit: 1630431f326e15fcde608827b5ff38422528eb59
PATCH v2 1/1 rev-list: add --missing-only option to filter output
When working with partial clones, callers often need only the missing object IDs. Today that means post-processing --missing=print to drop present objects and strip the leading '?':
git rev-list --objects --all --missing=print | perl -ne 'print if s/^[?]//'
Add --missing-only. Use it with --missing=print or --missing=print-info to print only missing objects. --missing= still picks the format; --missing-only only filters. The leading '?' is omitted. With print-info, path= and type= are still shown.
Require --missing=print or --missing=print-info. Reject --count and --disk-usage.
---
Changes from v1 include:
1. Replace --missing=print-only mode with a separate --missing-only flag (Phillip, Patrick, Stolee).
2. Require --missing=print or --missing=print-info.
3. Die when combined with --count or --disk-usage (Stolee).
4. Keep enum comment alignment (no spacing churn).
5. Keep print-info path=/type=; only drop '?'.
6. Simpler tests with test_cmp (Phillip).
Documentation/rev-list-options.adoc | 13 ++++++++ builtin/rev-list.c | 42 ++++++++++++++++++++++--- t/t6022-rev-list-missing.sh | 49 +++++++++++++++++++++++++++++ 3 files changed, 99 insertions(+), 5 deletions(-)
Documentation/rev-list-options.adoc +13 -0
@@ -1083,6 +1083,19 @@ If some tips passed to the traversal are missing, they will be considered as missing too, and the traversal will ignore them. In case we cannot get their Object ID though, an error will be raised. +`--missing-only`:: + When used together with `--missing=print` or `--missing=print-info`, + suppress all output for present objects and print only the missing + ones. The selected `--missing=` format is preserved (so + `--missing=print-info` still emits `path=` / `type=` fields), but the + leading ``?'' prefix used by the non-`-z` forms is omitted. This is + useful for scripting, as a simpler and faster alternative to + post-processing the output of `--missing=print`. ++ +This option is incompatible with `--count` and `--disk-usage`. +It is an error to use `--missing-only` without `--missing=print` or +`--missing=print-info`. + `--exclude-promisor-objects`:: (For internal use only.) Prefilter object traversal at promisor boundary. This is used with partial clone. This is
builtin/rev-list.c +37 -5
102 lines not on this page · raw message
t/t6022-rev-list-missing.sh +49 -0
55 lines not on this page · raw message
base-commit: 1630431f326e15fcde608827b5ff38422528eb59
Signature · 1 line
2.54.0
Re: [PATCH v2 0/1] rev-list: add --missing-only option to filter output
In reply to Siddharth AsthanaQuoted 4 lines · Siddharth Asthana
At GitLab, Gitaly uses rev-list --missing=print on partial clones to find missing objects. The output mixes present and missing objects and prefixes missing ones with '?', so we post-process it. --missing-only avoids that.
Not an objection to the feature (as it can already be done with post-processing), but stepping back a bit, how is this list of missing objects used?
You know what objects you currently have. They refer to other objects, some of which you may lack. You can get a list of them, and then what? After fetching them from 'origin', you will have a new set of missing objects (e.g., after fetching a missing commit, its parents become known and missing; after fetching a missing tree, the blobs and trees contained in it become known and missing).
Fetching missing objects a batch at a time sounds like too much back-and-forth to peel the onion, if the process using '--missing-only' is trying to find out what it wants to fetch, and would be a rather inefficient way to backfill a shallow clone. So I wonder how this fits into the larger picture.
Thanks.
Re: [PATCH v2 0/1] rev-list: add --missing-only option to filter output
Quoted 10 lines · Junio C Hamano
Quoted 4 lines · Siddharth Asthana
At GitLab, Gitaly uses rev-list --missing=print on partial clones to find missing objects. The output mixes present and missing objects and prefixes missing ones with '?', so we post-process it. --missing-only avoids that.
Not an objection to the feature (as it can already be done with post-processing), but stepping back a bit, how is this list of missing objects used?
Thanks for asking
Quoted 9 lines
You know what objects you currently have. They refer to other objects, some of which you may lack. You can get a list of them, and then what? After fetching them from 'origin', you will have a new set of missing objects (e.g., after fetching a missing commit, its parents become known and missing; after fetching a missing tree, the blobs and trees contained in it become known and missing).
Quoted 5 lines
Fetching missing objects a batch at a time sounds like too much back-and-forth to peel the onion, if the process using '--missing-only' is trying to find out what it wants to fetch, and would be a rather inefficient way to backfill a shallow clone. So I wonder how this fits into the larger picture.
Agreed, we are not using it that way, and I would not want people to use it as a multi-round fetch loop either.
In Gitaly the main caller is transaction packing. When we commit a transaction we walk only the quarantine (plus the new ref tips) with something like:
git rev-list --objects --missing=print --stdin
- Objects that exist in the quarantine get packed and logged with the transaction.
- Objects that show up as missing (the '?' lines) are recorded as the transaction's dependencies. At apply time we check those OIDs already exit in the main repo and abort if they do not. We do not fetch them; they are expected to already be outside the quarantine.
So the missing list is a one-shot dependency set from a single walk, not iterative discovery / onion peeling. Today we parse the mixed output and strip the leading '?'. --missing-only is only so that path (and the old "count missing" demo) can get bare OIDs without post-processing.
Happy to clarify this in the commit message / docs in a v3 if useful.
Thanks,
Siddharth
Quoted 2 lines
Thanks.
Re: [PATCH v2 0/1] rev-list: add --missing-only option to filter output
Quoted 3 lines · Siddharth Asthana
Quoted 1 line
... So I wonder how this fits into the larger picture.
Happy to clarify this in the commit message / docs in a v3 if useful.
Yeah, strengthing the description of the motivation in the proposed log message is always a good way to sell your ware ;-).
v3 cover rev-list: add --missing-only option to filter output
In reply to Siddharth AsthanaHi,
v2 added --missing-only as a filter next to --missing=print / --missing=print-info. Junio asked how the missing list is used, so this v3 only updates the log message.
It is a one-shot walk. Gitaly uses --missing=print when packing a quarantine and treats '?' lines as objects that must already be in the main repo. Tests do the same (is this blob still missing). --missing-only is that list without the prefix, not a fetch loop.
Based on 1630431f32 (The 21st batch, 2026-08-31).
Changes from v2:
- Stronger motivation in the commit message (Junio).
- Link to v2:
https://lore.kernel.org/git/[email protected]/
Thanks,
Siddharth
---
Siddharth Asthana (1):
rev-list: add --missing-only option to filter output
Documentation/rev-list-options.adoc | 13 ++++++++ builtin/rev-list.c | 42 ++++++++++++++++++++++--- t/t6022-rev-list-missing.sh | 49 +++++++++++++++++++++++++++++ 3 files changed, 99 insertions(+), 5 deletions(-)
Range-diff versus v2:
1: 3c89e6bc38 ! 1: c751ff41a4 rev-list: add --missing-only option to filter output
@@ Commit message
git rev-list --objects --all --missing=print | perl -ne 'print if s/^[?]//'
+ This is for a one-shot walk, not a fetch loop. Callers already have
+ --missing=print and strip the leading '?'. Gitaly does that when packing
+ a quarantine: '?' lines are objects that must already exist in the main
+ repo. Tests do the same (is this blob still missing). --missing-only is
+ just that list without the prefix.
+
Add --missing-only. Use it with --missing=print or --missing=print-info to print only missing objects. --missing= still picks the format; --missing-only only filters. The leading '?' is omitted. With
base-commit: 1630431f326e15fcde608827b5ff38422528eb59
PATCH v3 1/1 rev-list: add --missing-only option to filter output
When working with partial clones, callers often need only the missing object IDs. Today that means post-processing --missing=print to drop present objects and strip the leading '?':
git rev-list --objects --all --missing=print | perl -ne 'print if s/^[?]//'
This is for a one-shot walk, not a fetch loop. Callers already have --missing=print and strip the leading '?'. Gitaly does that when packing a quarantine: '?' lines are objects that must already exist in the main repo. Tests do the same (is this blob still missing). --missing-only is just that list without the prefix.
Add --missing-only. Use it with --missing=print or --missing=print-info to print only missing objects. --missing= still picks the format; --missing-only only filters. The leading '?' is omitted. With print-info, path= and type= are still shown.
Require --missing=print or --missing=print-info. Reject --count and --disk-usage.
---
Changes from v2:
- Strengthen the motivation in the log message (Junio). This is a one-shot walk (Gitaly quarantine deps, tests), not a multi-round fetch loop.
Documentation/rev-list-options.adoc | 13 ++++++++ builtin/rev-list.c | 42 ++++++++++++++++++++++--- t/t6022-rev-list-missing.sh | 49 +++++++++++++++++++++++++++++ 3 files changed, 99 insertions(+), 5 deletions(-)
Documentation/rev-list-options.adoc +13 -0
@@ -1083,6 +1083,19 @@ If some tips passed to the traversal are missing, they will be considered as missing too, and the traversal will ignore them. In case we cannot get their Object ID though, an error will be raised. +`--missing-only`:: + When used together with `--missing=print` or `--missing=print-info`, + suppress all output for present objects and print only the missing + ones. The selected `--missing=` format is preserved (so + `--missing=print-info` still emits `path=` / `type=` fields), but the + leading ``?'' prefix used by the non-`-z` forms is omitted. This is + useful for scripting, as a simpler and faster alternative to + post-processing the output of `--missing=print`. ++ +This option is incompatible with `--count` and `--disk-usage`. +It is an error to use `--missing-only` without `--missing=print` or +`--missing=print-info`. + `--exclude-promisor-objects`:: (For internal use only.) Prefilter object traversal at promisor boundary. This is used with partial clone. This is
builtin/rev-list.c +37 -5
102 lines not on this page · raw message
t/t6022-rev-list-missing.sh +49 -0
55 lines not on this page · raw message
Range-diff:
1: 3c89e6bc38 ! 1: c751ff41a4 rev-list: add --missing-only option to filter output
@@ Commit message
git rev-list --objects --all --missing=print | perl -ne 'print if s/^[?]//'
+ This is for a one-shot walk, not a fetch loop. Callers already have
+ --missing=print and strip the leading '?'. Gitaly does that when packing
+ a quarantine: '?' lines are objects that must already exist in the main
+ repo. Tests do the same (is this blob still missing). --missing-only is
+ just that list without the prefix.
+
Add --missing-only. Use it with --missing=print or --missing=print-info to print only missing objects. --missing= still picks the format; --missing-only only filters. The leading '?' is omitted. With
base-commit: 1630431f326e15fcde608827b5ff38422528eb59
Signature · 1 line
2.54.0