Integrate commit-graph into 'fsck' and 'gc'

Derrick Stolee git 6 versions May 10, 2018 – Jun 8, 2018 In review

Get series with b4

b4 am [email protected]

Versions newest first · linked by author and title

v6 Latest Jun 8, 201821 patches 5 repliesIn review
v5 Jun 6, 201821 patches 6 repliesSuperseded
v4 Jun 4, 201821 patches 6 repliesSuperseded

Earlier versions · 3

Version	Posted	Patches	Replies	Status
v3	May 24, 2018	20 patches	48 replies	Superseded
v2	May 11, 2018	12 patches	18 replies	Superseded
v1	May 10, 2018	0 of 12 patches	6 replies	Superseded

Patches in v6 No applies recorded yet

#	Subject and files	Size	Replies
1/21	commit-graph: UNLEAK before die()`builtin/commit-graph.c`	+4 -1	0 replies
2/21	commit-graph: fix GRAPH_MIN_SIZE`commit-graph.c`	+3 -2	0 replies
3/21	commit-graph: parse commit from chosen graph`commit-graph.c`	+15 -3	0 replies
4/21	commit: force commit to parse from object database`commit.c` `commit.h`	+8 -2	0 replies
5/21	commit-graph: load a root tree from specific graph`commit-graph.c`	+9 -3	0 replies
6/21	commit-graph: add 'verify' subcommand`Documentation/git-commit-graph.txt` `builtin/commit-graph.c` `commit-graph.c` +2 more	+81 -0	0 replies
7/21	commit-graph: verify catches corrupt signature`t/t5318-commit-graph.sh`	+43 -0	0 replies
8/21	commit-graph: verify required chunks are present`commit-graph.c` `t/t5318-commit-graph.sh`	+38 -0	0 replies
9/21	commit-graph: verify corrupt OID fanout and lookup`commit-graph.c` `t/t5318-commit-graph.sh`	+58 -0	0 replies
10/21	commit-graph: verify objects exist`commit-graph.c` `t/t5318-commit-graph.sh`	+25 -0	0 replies
11/21	commit-graph: verify root tree OIDs`commit-graph.c` `t/t5318-commit-graph.sh`	+23 -1	0 replies
12/21	commit-graph: verify parent list`commit-graph.c` `t/t5318-commit-graph.sh`	+46 -0	1 reply
13/21	commit-graph: verify generation number`commit-graph.c` `t/t5318-commit-graph.sh`	+45 -0	0 replies
14/21	commit-graph: verify commit date`commit-graph.c` `t/t5318-commit-graph.sh`	+12 -0	0 replies
15/21	commit-graph: test for corrupted octopus edge`t/t5318-commit-graph.sh`	+10 -0	0 replies
16/21	commit-graph: verify contents match checksum`commit-graph.c` `t/t5318-commit-graph.sh`	+20 -2	0 replies
17/21	fsck: verify commit-graph`Documentation/git-fsck.txt` `builtin/fsck.c` `t/t5318-commit-graph.sh`	+32 -0	0 replies
18/21	commit-graph: use string-list API for input`builtin/commit-graph.c` `commit-graph.c` `commit-graph.h`	+23 ~~-38~~	0 replies
19/21	commit-graph: add '--reachable' option`Documentation/git-commit-graph.txt` `builtin/commit-graph.c` `commit-graph.c` +2 more	+47 -6	0 replies
20/21	gc: automatically write commit-graph files`Documentation/config.txt` `Documentation/git-gc.txt` `builtin/gc.c` +1 more	+33 -1	2 replies
21/21	commit-graph: update design document`Documentation/technical/commit-graph.txt`	+0 ~~-22~~	0 replies

Reviews trailers per patch

None of the 21 patches has a Reviewed-by or Acked-by of their own; the series has one for all of them.

Patch	Ramsay Jones	Ævar Arnfjörð Bjarmason	Szeder Gábor	Derrick Stolee
Series	Helped-by	Helped-by	Helped-by	R-b R-b
1/21
2/21
3/21
4/21
5/21
6/21	Helped-by
7/21			Helped-by
8/21
9/21
10/21
11/21
12/21
13/21
14/21
15/21
16/21
17/21
18/21
19/21
20/21		Helped-by
21/21

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v6 thread

Cover letter [PATCH v6 00/21] Integrate commit-graph into 'fsck' and 'gc'

Sorry for the fast rerolls, but Ævar found some runtime issues with the string-list use and '\*' as multiplication on some platforms. Thus, v4 and v5 are broken. I did get a repro of those issues using VSTS Linux build servers and checked that they are fixed in this version.

This version depends on 'next' and sb/object-store-alloc. To accommodate the repository arguments added by sb/object-store-alloc, the following diff occurs from the previous patch:

builtin/commit-graph.c +3 -2

index 76423b3fa5..c7d0db5ab4 100644
@@ -3,6 +3,7 @@
 #include "dir.h"
 #include "lockfile.h"
 #include "parse-options.h"
+#include "repository.h"
 #include "commit-graph.h"
 
 static char const * const builtin_commit_graph_usage[] = {

@@ -63,7 +64,7 @@ static int graph_verify(int argc, const char **argv)
        if (!graph)
                return 0;
 
-       return verify_commit_graph(graph);
+       return verify_commit_graph(the_repository, graph);
 }
 
 static int graph_read(int argc, const char **argv)

@@ -152,9 +153,9 @@ static int graph_write(int argc, const char **argv)
                return 0;
        }
 
+       string_list_init(&lines, 0);
        if (opts.stdin_packs || opts.stdin_commits) {
                struct strbuf buf = STRBUF_INIT;
-               string_list_init(&lines, 0);
 
                while (strbuf_getline(&buf, stdin) != EOF)
                        string_list_append(&lines, strbuf_detach(&buf, NULL));

commit-graph.c +3 -2

index 0d5adc8035..adf54e3fe7 100644
@@ -12,6 +12,7 @@
 #include "sha1-lookup.h"
 #include "commit-graph.h"
 #include "object-store.h"
+#include "alloc.h"
 
 #define GRAPH_SIGNATURE 0x43475048 /* "CGPH" */
 #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* "OIDF" */

@@ -866,7 +867,7 @@ static void graph_report(const char *fmt, ...)
 #define GENERATION_ZERO_EXISTS 1
 #define GENERATION_NUMBER_EXISTS 2
 
-int verify_commit_graph(struct commit_graph *g)
+int verify_commit_graph(struct repository *r, struct commit_graph *g)
 {
        uint32_t i, cur_fanout_pos = 0;
        struct object_id prev_oid, cur_oid, checksum;

@@ -948,7 +949,7 @@ int verify_commit_graph(struct commit_graph *g)
                hashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);
 
                graph_commit = lookup_commit(&cur_oid);
-               odb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());
+               odb_commit = (struct commit *)create_object(r, cur_oid.hash, alloc_commit_node(r));
                if (parse_commit_internal(odb_commit, 0, 0)) {
                        graph_report("failed to parse %s from object database",
                                     oid_to_hex(&cur_oid));

commit-graph.h +2 -1

index ee20f5e280..506cb45fb1 100644
@@ -2,6 +2,7 @@
 #define COMMIT_GRAPH_H
 
 #include "git-compat-util.h"
+#include "repository.h"
 #include "string-list.h"
 
 char *get_commit_graph_filename(const char *obj_dir);

@@ -53,6 +54,6 @@ void write_commit_graph(const char *obj_dir,
                        struct string_list *commit_hex,
                        int append);
 
-int verify_commit_graph(struct commit_graph *g);
+int verify_commit_graph(struct repository *r, struct commit_graph *g);
 
 #endif

t/t5318-commit-graph.sh +13 ~~-13~~

index b24e8b6689..9a0661983c 100755
@@ -33,8 +33,8 @@ test_expect_success 'create commits and repack' '
 '
 
 graph_git_two_modes() {
-       git -c core.commitGraph=true $1 >output
-       git -c core.commitGraph=false $1 >expect
+       git -c core.graph=true $1 >output
+       git -c core.graph=false $1 >expect
        test_cmp output expect
 }

@@ -282,17 +282,17 @@ GRAPH_CHUNK_LOOKUP_WIDTH=12
 GRAPH_CHUNK_LOOKUP_ROWS=5
 GRAPH_BYTE_OID_FANOUT_ID=$GRAPH_CHUNK_LOOKUP_OFFSET
 GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \
-                           1 \* $GRAPH_CHUNK_LOOKUP_WIDTH))
+                           1 * $GRAPH_CHUNK_LOOKUP_WIDTH))
 GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \
-                            2 \* $GRAPH_CHUNK_LOOKUP_WIDTH))
+                            2 * $GRAPH_CHUNK_LOOKUP_WIDTH))
 GRAPH_FANOUT_OFFSET=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \
-                      $GRAPH_CHUNK_LOOKUP_WIDTH \* $GRAPH_CHUNK_LOOKUP_ROWS))
-GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 \* 4))
-GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 \* 255))
-GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 \* 256))
-GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \* 8))
-GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \* 4 + 10))
-GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \* $NUM_COMMITS))
+                      $GRAPH_CHUNK_LOOKUP_WIDTH * $GRAPH_CHUNK_LOOKUP_ROWS))
+GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 * 4))
+GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 * 255))
+GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 * 256))
+GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8))
+GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 4 + 10))
+GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * $NUM_COMMITS))
 GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET
 GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))
 GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))

@@ -301,9 +301,9 @@ GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))
 GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12))
 GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))
 GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \
-                            $GRAPH_COMMIT_DATA_WIDTH \* $NUM_COMMITS))
+                            $GRAPH_COMMIT_DATA_WIDTH * $NUM_COMMITS))
 GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))
-GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 \* $NUM_OCTOPUS_EDGES))
+GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 * $NUM_OCTOPUS_EDGES))
 
 # usage: corrupt_graph_and_verify <position> <data> <string>
 # Manipulates the commit-graph file at the position

---

Derrick Stolee (21):
commit-graph: UNLEAK before die()
commit-graph: fix GRAPH_MIN_SIZE
commit-graph: parse commit from chosen graph
commit: force commit to parse from object database
commit-graph: load a root tree from specific graph
commit-graph: add 'verify' subcommand
commit-graph: verify catches corrupt signature
commit-graph: verify required chunks are present
commit-graph: verify corrupt OID fanout and lookup
commit-graph: verify objects exist
commit-graph: verify root tree OIDs
commit-graph: verify parent list
commit-graph: verify generation number
commit-graph: verify commit date
commit-graph: test for corrupted octopus edge
commit-graph: verify contents match checksum
fsck: verify commit-graph
commit-graph: use string-list API for input
commit-graph: add '--reachable' option
gc: automatically write commit-graph files
commit-graph: update design document

 Documentation/config.txt                 |  10 +-
 Documentation/git-commit-graph.txt       |  14 +-
 Documentation/git-fsck.txt               |   3 +
 Documentation/git-gc.txt                 |   4 +
 Documentation/technical/commit-graph.txt |  22 --
 builtin/commit-graph.c                   |  99 ++++++---
 builtin/fsck.c                           |  21 ++
 builtin/gc.c                             |   6 +
 commit-graph.c                           | 249 +++++++++++++++++++++--
 commit-graph.h                           |  11 +-
 commit.c                                 |   9 +-
 commit.h                                 |   1 +
 t/t5318-commit-graph.sh                  | 201 ++++++++++++++++++
 13 files changed, 572 insertions(+), 78 deletions(-)

Signature · 1 line

2.18.0.rc1

Read the cover letter in its thread